ACS Data Reliability Indicator Usability Interviewing Plan

The purpose of this letter is to inform you of our plans to conduct additional research under the generic clearance for questionnaire pre-testing research (OMB number 0607-0725). We will be conducting usability interviews to help determine which of three potential prototypes of a data reliability indicator for American Community Survey (ACS) data tables helps users find estimates and information about sampling error most efficiently, accurately, and with the most satisfaction. The prototypical tables will also be compared to a baseline version of the current table.

For this study, there are three different prototypical ACS data table types corresponding to different possible ways of displaying the data reliability indicator (two 4-level and one 3-level indicator) and one baseline table corresponding to the current version of the ACS data tables. Versions of these prototypes and the baseline table are available in Attachment A.

Between June and July 2009, staff from the Statistical Research Division’s usability lab will conduct a maximum of 60 usability interviews, 15 for each of the three prototypes and 15 for the baseline table.

Fifty participants will be randomly sampled from a sampling frame of 342 local Washington, D.C.-area journalists provided by the Census Bureau’s Public Information Office (PIO). From this sample of 50 journalists, the expected response rate is 40 participants (e.g., ten participants per prototype or baseline table). Twenty additional participants will be recruited through the Brookings Institute and the Council of Professional Associations on Federal Statistics (COPAFS). Participants will likely vary in their levels of familiarity with ACS and data; however, each test participant must self-report at least one year of prior experience in navigating different Websites.

Each participant will be randomly assigned to one of four conditions corresponding to each of the three table types (one of the three prototypical ACS data reliability indicators or the baseline table). The sessions will take place in the Census Bureau's usability lab. Each experimental session will include a test administrator and one participant. Participants will be asked to fill out an initial questionnaire with demographic questions on their age, sex, and education level. The questionnaire also contains questions about the respondents’ computer and Internet experience and is included as Attachment B. Participants will have no known cognitive or psychomotor disabilities.

At the beginning of the session, participants will be informed that their response is voluntary and that the information they provide is confidential and will be seen only by employees involved in the research project. Participants will be paid $40 for their assistance in this project.

The testing procedure will consist of asking the participant to complete a series of tasks using the ACS data tables (Attachment C). Each participant will only use one of the prototypes. The tasks are designed to capture the participant’s interaction with, and reactions to, the design and functionality of the ACS data confidence indicators. Except for the first task, which asks participants to report the first thing that they notice about the table, the tasks will be randomized for each participant. We will also be collecting eye-tracking data to examine which visual aspects of the login process were noticed and used by the respondents.

After the participant completes the tasks, he/she will be asked to complete a paper Questionnaire for User Interface Satisfaction (QUIS) that includes items worded for the ACS data reliability indicators context (Attachment D).

Participants will also be asked to provide a difficulty rating for each task on a short questionnaire, which can be found in Attachment E (which will be used for validation of the “easy” versus “hard” designation during analysis). The experimenter will hand the QUIS and the task difficulty rating questionnaire to the participant at the same time. After completing these forms, the test participants will be asked aloud some debriefing questions about their overall experience using the prototype ACS Data Confidence Indicator (Attachment F) and about their colorblindness status.

The estimated time for completion of the experimental session is one hour. Thus, the total estimated burden time for this research is a maximum of 60 hours.

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Kathleen T. Ashenfelter, Ph.D.

Center for Survey Methods Research

Room 5K020K

U.S. Census Bureau

Washington, D.C. 20233

(301) 763- 4922

[email protected]