2015 National Content Test Usability and Cognitive Testing

The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725).

This submission seeks approval for a segment of a large research contract, under which we are

developing and testing both English language and non-English language materials for various modes in preparation for various 2015, 2016 and 2017 Decennial Census Tests. The research is sponsored by the Decennial Management Division (DMD) at the Census Bureau and the ultimate goal is preparation for the 2020 Census.

The larger study consists of one focus group project and four cognitive interview projects, three of which include iterative rounds of testing. The research is being conducted by two contractors, RTI International and RSS and some testing is being done by Census Bureau staff.

This is the sixth OMB submission based on the larger research project and it is for Round 2 of internet instrument testing in Spanish and English.. We received approval in November, 2014 for Round 1 of NRFU (CAPI) testing and Round 1 of paper testing and in December, 2014 for Round 1 of internet testing. We received approval in February, 2015 for Round 2 of paper testing. We received approval for Round 1 of cognitive testing in five languages in March, 2015. Approval for focus group testing in 7 languages is pending. Those parts of the project are completed, currently underway or about to begin.

As a part of this current submission we are seeking approval for both internal, Census Bureau usability/cognitive interviews and contractor usability/cognitive interviews. The Census Bureau cases will test some additional materials and will include additional questionnaires for respondents to fill out, thus we are seeking approval for the contractor cases to take an average of 60 minutes, whereas the internal Census cases are projected to take 90 minutes.

We will be conducting a usability test on the Spanish and English versions of the 2015 National Content Test (NCT) online questionnaire. The 2015 NCT is a nationwide test with a Census Day of September 1, 2015. Different question versions for relationship, race, ethnicity, and roster building will be tested. New address fields for the Puerto Rico 2015 internet instrument will also be tested.

Usability testing will identify issues that are problematic and frustrating to the user; Spanish testing will also include cognitive testing of the translation. Testing will continue to focus on testing the screen design for different types of devices, specifically, the testing will focus on the screen design for mobile devices, the new question designs, and login screens.

The testing will be conducted in July and early August of 2015. Testing will take place at locations convenient for participants, including the usability lab at the U.S. Census Bureau for some of the internal Census Bureau cases,. Participants will use their own smartphones or tablets to complete the survey. A total of 20 English and 24 Spanish interviews will be conducted by the contractor and 21 English and 10 Spanish interviews will be conducted by Census Bureau staff.

Participants will be from the general public and for the English testing, we will emphasize recruiting people who are or who live with household members of Middle Eastern or North African descent, people with multi-racial backgrounds. We will also recruit a few large households, a few older respondents, and a few respondents born in Puerto Rico who speak English. We also are interested in respondents in households with unique situations (e.g., somebody at college, in jail, other temporary situations, etc.).

For the Spanish interviews, we will aim to recruit a variety of national origins and relationships within the households and we will aim to recruit some Puerto Rican Spanish speakers. If possible, there is also interest in recruiting people from Central or South America with indigenous heritage.

All participants will know how to use their smartphone or tablet to access the Internet. The contractor will compensate participants $40.00 for their participation in the 60 minute interview. The Census Bureau will compensate participants $60 for their participation in the 90 minute interview. Participants will be recruited using a combination of word-of-mouth, flyers, online and paper advertisements.

Participants will complete the 2015 NCT online survey using a think-aloud protocol. To complete the 2015 NCT, all participants, with the exception of Puerto Rican participants, will be given a paper with their login information containing a Census ID. Participants from Puerto Rico will be given the URL for the Puerto Rico online form so that they are able to interact with newly designed Puerto Rican address fields. After completing the online form, each participant will be asked debriefing questions about different screens.

Spanish speaking participants will complete the same tasks as English-speaking participants with two differences. First, they will complete the 2015 Census Test online survey in Spanish. Second, debriefing will be conducted in Spanish.

Below is a list of materials to be used in the current study. Included is a note on whether each attachment is new or has already been approved by OMB. In the cases where the documents were already approved we include a reference to the attachment number in the previously approved submission (originally submitted on 10/20/14).

1. Protocol used for the study (Enclosure 1)

2. PowerPoint slide shows with debriefing probes

   1. Debriefing questions and placeholder screen shots are included in the protocol. One final screens are available, we will update the PowerPoint file with those screens. (Enclosure 2 - all screen shots (with the exception of the new warning message and the relationship vignette screen) are concurrently being approved by OMB for the 2015 NCT. The probes are new.)

   2. For the online Puerto Rico form, participants will see both attached PowerPoint files. For the online stateside form, participants will not see the Puerto Rican address field PowerPoint. (Enclosure 3 - all screen shots are concurrently being approved by OMB for the 2015 NCT. The probes are new.)

List of materials already approved by OMB for the larger language research contract

3. Recruitment “teasers” to be used in advertising materials (already approved. File name from 10/20/14 submission: OMB1425_2015_Census_Test_cognitive_enc10.doc)

4. Recruitment flyers (already approved. File name from 10/20/14 submission: OMB1425_2015_Census_Test_cognitive_enc11.doc)

5. English recruitment screener (already approved. File name from 10/20/14 submission: OMB1425_2015_Census_Test_cognitive_enc8.doc)

6. Payment voucher (already approved. File name from 10/20/14 submission: OMB1425_2015_Census_Test_cognitive_enc9.doc)

List of materials already approved for Census Bureau use in usability testing projects

1. Background questionnaire (To get understanding of users’ mobile and Internet experience; (Previously approved by OMB)

2. Demographic questionnaire (Previously approved by OMB)

3. Satisfaction questionnaire (Previously approved by OMB)

RTI expects to screen 100-150 respondents to arrive at 44 completed cases. Screening takes approximately 6 minutes per case and an additional 1.5 minutes is required to call back and schedule selected participants. Thus the maximum burden for the 44 contractor cases is 60.1 hours total, 44 hours for interviewing and 16.1 hours for recruiting.

For internal Census cases, we estimate that users will spend 90 minutes on average taking the study, including time spent working on the demographic and satisfaction questions, the online survey and the debriefing, for a total of 46.5 hours. Respondents will be screened using the Census Bureau’s generic screener, thus the burden hours for screening are covered under a separate request.

The contact people for questions regarding data collection and statistical aspects of the design of this research are listed below:

Patricia Goerman

Center for Survey Methods Research

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-1819

[email protected]

Elizabeth Nichols (for Census portion)

Center for Survey Measurement

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-1724

[email protected]