American Factfinder Pretesting Plan

The Census Bureau plans to conduct a usability test of the new American FactFinder (AFF) Web site under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). The objective of this research is to identify issues that are problematic and frustrating to the user and to compare the results to the original baseline usability testing that was conducted on the legacy AFF system in 2008. AFF is a free online tool that allows users to find, customize and download Census information. It is available to the public, and a multitude of diverse users search the site seeking a vast range of information.

Over the past year the developers of the AFF Web site have worked with stakeholders and the usability lab to address usability issues in their updates of the application. An update occurred in April of 2012. This study is intended to be a “follow-up baseline” on the latest version of the site to understand how it performs for users with respect to the legacy version. This is the second follow-up baseline study. The initial baseline study was submitted to OMB for approval in a letter dated October 9, 2008. The first follow-up to the baseline study was submitted to OMB for approval in a letter dated April 4, 2011.

During June, staff from the Census Bureau’s usability lab will interview 20 external participants from the Washington DC metropolitan area. We will recruit two different user groups: novice users of census data who have a minimum of one year of Internet experience and use the Internet at least three times a week to search for information; and expert users of census data who use census data for work purposes. Novice participants will be recruited from the Usability Lab database, which is composed of people from the metropolitan DC area who volunteered to participate after responding to a craigslist.com posting or an ad in a local newspaper. Expert users will be recruited both by the Usability Lab database, and by recommendations from contacts made by the Census Regional Offices or census partners. Participants will come to the Usability Lab at the Census Bureau for the study.

Participants will be given an initial questionnaire about their Internet experience and some demographic characteristics. Then each participant will be given a set of tasks. The study will use the same tasks and protocol that was used on the original baseline so that we will be able to compare the performance of the new site with respect to the performance of the old site¹. Tasks will be randomized such that no two participants will receive the tasks in the same order. Participants will be asked to think-aloud while they are working on the tasks. Participants will be prompted to think-aloud when they fall silent. Finally, participants will be asked to complete a questionnaire designed to measure their satisfaction with the new version of the AFF site. Subjective satisfaction ratings will be collected for such design elements as the layout of page, ease of finding information, and use of Census jargon. Participants will also provide feedback about the Web site during a debriefing at the conclusion of the session. A copy of the initial questionnaire, the satisfaction questionnaire, debriefing questionnaire and the task set are enclosed.

In this study we will also be collecting a new post-task satisfaction question after each task. The earlier two studies (original baseline in 2009 and first follow-up to baseline in 2011) did not collect this information so we will not use this measure to compare back but will use this measure to compare going forward. A copy of the post-task questionnaire is also enclosed.

Respondents will be informed that their involvement is voluntary and that the information they provide is confidential and will be seen only by employees involved in the research project. Participants will be videotaped and the recordings will be seen only by employees involved in the research project. Participants will be compensated $40.00 for their participation.

We estimate that users will spend one hour on average taking the study, including time spent working on the demographic and satisfaction questions, the tasks and the debriefing. Thus, the total estimated respondent burden for this test is 20 hours.

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Erica Olmsted-Hawala

Center for Survey Methods Research

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-4893

[email protected]

1 A few of the tasks have been updated with minor changes, such as date, to reflect a more recent time-period. A task that is not currently available on the new site has been re-written. This task will not compare going back to the original baseline but will be used going forward to future follow-up baseline studies. See the xls spreadsheet for a highlight of the tweaks to the tasks.