Research Plan

The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). We will be conducting experimental research to evaluate alternative methods of collecting think-aloud (TA) protocols in usability studies. The objectives of this research are twofold: 1) to assess the methods currently in use at the Census Bureau, and 2) to add to the literature on TA in usability studies.

Collection of TA protocols is one of the primary tools used by usability professionals when conducting usability tests. However, there is no consistency in the approaches to collecting TA protocols from study to study. In some studies the participant is encouraged to “think out loud” while working on a task. Within this concurrent TA approach, there are many variations: from giving participants a practice session (or not), from test administrators sitting next to the participant (or not), and from minimal to active verbal probing by the test administrator. Still other practitioners use a retrospective TA where a participant talks only after the session is completed, typically while watching a video of their session

The obvious problems with usability professionals practicing different types of TA protocols are that not only might the results of a single study be called into question, but the replicability of that study and comparability across studies could also be in question. The planned research is an attempt to contribute to the knowledge about how best to conduct usability studies, so the Census Bureau can use the best methods in testing its Internet interfaces for surveys and Web sites.

We will conduct a split panel experiment on the TA protocol in usability studies. With a random assignment to condition, this study will be able to show whether or not the manipulation of the different conditions has a measurable effect. The experiment will include four panels or conditions, which demonstrate four different approaches to conducting TA in usability testing. These are the four conditions:

1. No probing words beyond “keep talking” reminders, with TA practice session before session begins;

2. Verbal feedback in form of “um-hum” or “un-hum” to keep participant talking, with TA practice session before session begins;

3. Interactive verbal feedback and probes (Test Administrator asks direct questions about different areas of Web site, asks direct questions about areas where user is having difficulty/is pausing/or is describing area as confusing or frustrating, gives help or assists when participant is struggling) with TA practice session before session begins; and

4. No TA, no probing or prompting; with TA practice session.

We plan to have one user group: general or novice participants. Key requirements for these respondents include the following:

• At least one year of Internet experience, with little or no exposure to the Census Web site or to Census terminology;

• Familiar and comfortable with searching for information on the Internet;

• Any amount of education; and

• Not a federal employee.

We will recruit from our internal database made up of respondents from the Washington DC metropolitan area who have volunteered to participate in a usability study at the Census Bureau. We will develop a scheme for randomizing assignment to conditions.

Between May and October 2008, staff from the Census Bureau’s usability lab will conduct a maximum of 100 testing sessions, each including a test administrator and one participant. We plan to run the sessions in the Census Bureau's usability lab. Participants will be asked to fill out an initial questionnaire with questions about their computer and Internet experience, age and education level. Then, depending on the condition, the participant will be told about the think-aloud protocol and asked to think-aloud (or not) as they work on eight typical tasks at Census.gov, the Census Bureau’s Web site. Again, depending on the condition, the test administrator will probe (or not) and give the participant reminders to think-aloud (or not). At the conclusion of the study, participants will be asked to fill out a final satisfaction questionnaire which is a modified version of the Questionnaire for User Interaction Satisfaction (QUIS). Copies of the initial and final questionnaire are enclosed, as well as a draft of the tasks respondents will be asked to complete.

Participants will be informed that their response is voluntary and that the information they provide is confidential and will be seen only by employees involved in the research project. We plan to provide a $40 monetary reimbursement to each participant who is not a government employee.

The estimated time for completion of the experimental session is one hour. Thus, the total estimated burden time for this research is a maximum of 100 hours.

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Erica L. Olmsted-Hawala

Center for Survey Methods Research

Room 5K104D

U.S. Census Bureau

Washington, D.C. 20233

(301) 763- 4893

[email protected]