American FactFinder Website Testing Plan

The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). We will be conducting a new usability test on the American FactFinder (AFF) Web site. AFF is a free online tool that allows users to find, customize and download Census information. Testing will identify issues that are problematic and frustrating to the user.

This specific study will test new design modifications that the team is considering making on the application. One potential modification is the design of the headers and footers. Two different main start pages, an A design that incorporates new Census Bureau header and footer navigation features and a B design that is similar to the current design but takes on a few features of the Census Bureau header and footer. Census.gov development team has asked that all sites accessible through www.census.gov use the same Census.gov header and footer. The AFF team is concerned that users will experience confusion with the new header and footer because a) the header and footer fly open when a user moves the mouse over that area of the screen, even if the mouse movement is unintended, and b) some of the labels used in the headers and footers match existing navigation labels in AFF which may cause confusion during navigation tasks. Usability testing will provide data and insight into whether usability problems exist with respect to the A and B design and also whether one design performs better for users than the other.

A second objective of the research is to test whether proposed fixes and updates to three primary areas of the application work well for the users. The three areas and the updates that will be tested include:

• Guided Search (The team has modified the behavior of the Race/Ethnic group selection, added Race/Ethnic group topics to Race/Ethnic group step, added pseudo Race/Ethnic groups, and added year filter to search results

• Advanced Search (By Topic-removed Race/Ethnic group selection from topic list, alphabetized the topic list; By Population Group- modified the behavior of the Race/Ethnic group selection, added Race/Ethnic group topics to population group overlay, added pseudo Race/Ethnic groups,

• Download (Add download assistant path, added dataset selection option, added pre-packaged selection option)

The tasks were designed specifically to target the three new updated areas of the application as well as the A and B designs using the different Census.gov header and footers. The testing will be conducted in December of 2013 and January of 2013 on html medium-fidelity wireframes. A total of 30 interviews will be conducted, enough to notice if the A or B design works better for the users. Half of the participants will work with the A design and the other half will work with the B design. Participants will be randomly assigned to the A or B design. Tasks 4 through 8 will all be done on the B design.

The participants will be a mix of novice and expert users of census data. The recruiting of the participants will be contracted out to a recruitment firm. The contractor will use screening questionnaires that have been used by usability lab staff for prior AFF studies. In general for Census Bureau staff, it requires three screener conversations to recruit one participant. As well, each screener conversation lasts approximately three minutes. All participants will have a minimum of one year Internet experience and use the Internet at least three times a week to search for information. Participants will come to the Usability Lab at the Census Bureau, where Census Bureau usability staff will conduct the interviews. Participants will be compensated $40.00 for their participation.

Participants will complete an electronic initial questionnaire about their computer use, Internet experience and a paper questionnaire about their demographic characteristics. Then each participant will be given a set of tasks for the Web site. Participants will be asked to think aloud while they are working on the tasks, and will be prompted to think aloud when they fall silent. Participants will also provide feedback about the Web site after each task when they answer a post-task questionnaire. Participants will answer the same post-task satisfaction questionnaire after each task. (We are currently testing our wording of the post-task questionnaire so half of the participants will randomly receive the post-task satisfaction questionnaire version A and half will receive the post-task satisfaction questionnaire version B.) At the conclusion of the session, participants will be asked to complete a final satisfaction questionnaire designed to measure their overall satisfaction with the AFF site. Subjective satisfaction ratings will be collected for such design elements as the layout of page, ease of finding information, and use of Census jargon. Following the final satisfaction questionnaire, the test administrator will ask final debriefing questions about the user interface.

The following materials will be used in the study and are attached with this letter:

• Background questionnaire (To get understanding of users’ Internet and Computer experience)

• Demographic questionnaire

• Participant tasks (Created specifically to test the A/B design and the three updated areas of the application)

• Post-task questionnaire (versions A and B) (To get immediate task-based, user satisfaction of the application)

• Final satisfaction questionnaire (To get users’ overall satisfaction of the application)

• Debriefing questions (Allows for a back-and-forth, conversation between test administrator and user about topics related to application that was not yet covered)

• Protocol used for the study (To see when and how the research study will be run)

• Screenshots of the AFF application

We estimate that users will spend one hour on average taking the study, including time spent working on the demographic and satisfaction questions, the tasks and the debriefing. We estimate five minutes per case to screen and recruit respondents, for a total of 5 hours. Thus, the total estimated respondent burden for this study is 35 hours.

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Erica L. Olmsted-Hawala

Center for Survey Methods Research

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-4893

[email protected]