Usability Interviewing Plan, ACS Data Reliability Indicator

The purpose of this letter is to inform you of our plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). We will be conducting a second round of usability interviews to help determine the usability of some prototypes of a data reliability indicator for American Community Survey (ACS) data tables. (Approval for the second round was submitted to OMB on May 29, 2009.) The goal is to determine which prototypical indicator helps users find estimates and information about sampling error most efficiently, accurately, and with the most satisfaction. The prototype tables will also be compared to their baseline counterparts. For this study, there are five different prototypical ACS data table types corresponding to different ACS data products – Data Profile, Selected Population Profile, Geographic Comparison, Detailed Table, and Subject Table. Each table will display a 3-level data reliability indicator. There will also be a “baseline” version of each table for comparison. For this round of testing, the baseline table is the version of the table that is currently believed to be the most promising based on the previous testing. Versions of these tables are available in Attachment A.

Between February and March 2010, staff from the Census Bureau’s usability lab will conduct a maximum of 40 usability interviews (8 per table type).

The participants will be novice ACS data users and will be recruited from email lists including the Association of Public Data Users (APDU), Census Information Centers (CIC), State Data Centers (SDC), the Census Advisory Committee, and the D.C. chapter of the American Association for Public Opinion Research (AAPOR). Participants may be affiliates of these groups and are not necessarily group members themselves. Local teachers will be recruited through Craig’s List and emails sent to the principals of the schools.

Each participant will be randomly assigned to conditions corresponding to each of the five table types, the placement of the data reliability indicator legend, the order of the colors in the data reliability indicator legend, and whether the indicator is in color or in grayscale. The design is detailed in Table 1. Each test participant will have at least one year of prior experience in navigating different Web sites. Participants will likely vary in their levels of familiarity with ACS data. The sessions will take place in the Census Bureau's usability lab. Each experimental session will include a test administrator and one participant. Participants will be asked to fill out an initial questionnaire with demographic questions on their age, sex, and education level. The questionnaire also contains questions about the respondents’ computer and Internet experience and is included as Attachment B.

Table 1 - Data Reliability Indicator Usability Testing – Round 3 (blank cells indicate baseline treatments)

Data Product	Testers	Indicator Treatment
		Placement (baseline: behind link)	Order (baseline: red first)	Color (baseline: color)
Data Profile	8	4 above table 4 behind link		2 no color 2 color 2 no color 2 color
Selected Pop Profile	8		4 green first 4 red first
Geographic Comparison	8		4 green first 4 red first
Detailed Table	8			4 no color 4 baseline
Subject Table	8	4 above table 4 behind link
TOTAL	40 testers

The testing procedure will consist of asking the participant to complete a series of tasks using the ACS data tables (Attachment C). The tasks will be slightly modified for each type of table so that they are applicable. Each participant will use only one of the prototypes. The tasks are designed to capture the participant’s interaction with, and reactions to, the design and functionality of the ACS data confidence indicators. Except for the first task, which asks participants to report the first thing that they notice about the table, the tasks will be randomized for each participant. We will also be collecting eye-tracking data to examine which visual aspects of the login process were noticed and used by the respondents.

After the participant completes the tasks, they will be handed a paper Questionnaire for User Interface Satisfaction (QUIS) that includes items worded for the ACS data reliability indicators context (Attachment D).

Participants will also be asked to provide a difficulty rating for each task on a short questionnaire, which can be found in Attachment E (which will be used for validation of the “easy” versus “hard” designation during analysis). The experimenter will hand the QUIS and the debriefing survey to the participant at the same time. After completing the tasks, the test participants will be asked aloud some debriefing questions about their overall experience using the prototype ACS Data Confidence Indicator (Attachment F).

Participants will be informed that their response is voluntary and that the information they provide is confidential and will be seen only by employees involved in the research project. Participants will be paid $40 for their assistance in this project, but only if they are not a federal employee.

The estimated time for completion of the experimental session is one hour. Thus, the total estimated burden time for this research is a maximum of 40 hours.

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Kathleen T. Ashenfelter, Ph.D.

Statistical Research Division

Center for Survey Methods Research

Room 5K020K

U.S. Census Bureau

Washington, D.C. 20233

(301) 763- 4922

[email protected]