OMB Memo

Generic Information Collection Request

Request: The U.S. Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). We propose to conduct the third round of testing of the Small-Scale Response Testing (SmaRT) Program. We are seeking approval for cognitive and usability testing conducted by Census Bureau staff.

Purpose: This project is modeled after the small-scale mailout testing program the Decennial directorate established in 2015 to prepare for the 2020 Census. The plan for the 2030 Census is to initiate and maintain an ongoing and iterative small-scale response testing (SmaRT) program by using a small, thoughtfully selected sample throughout the decade. This is the first round in the 2024 SmaRT test. Objectives regarding cognitive testing of mailing materials and usability testing of the Decennial online instrument (2024 Census Survey) are outlined below:

Mailing Materials

1. Identify major comprehension issues and review spontaneous feedback of the mailing materials focusing on:

   1. Tailored Contact Strategy

   2. Everyday Direct Mailer (EDDM) postcard

   3. Paper Fulfillment Only

      1. IVR text

2. Suggest revisions addressing comprehension issues

Decennial online instrument (2024 Census Survey)

1) Targeted Undercount

2) Determine if there are usability issues with “whole/partial roster” question based on targeted undercount questions

3) Determine any cognitive or usability issues with the new crowdsourcing vacancy questions at the end of the questionnaire

4) Find any usability problems which interfere with a respondent’s ability to answer the 2024 Census Survey questions accurately and with satisfaction

Population of Interest: The planned cognitive and usability evaluation will assess and improve the user experience for the general population, with a focus on oversampling a Historically Undercounted Population (HUP) group. (The HUP subteam identified group as Black or African American), as well as older adults (65 years and up) and the following household (HH) composition characteristics:

1. Unrelated HH composition

2. Related HH composition

3. Single person HH

4. Two person HH

5. Rural route address

6. Apartment dwellers

We will have both English speakers and monolingual Spanish speakers participate in testing. We are interested in recruiting participants with a diversity of ages, races, and gender identities. We are also interested in recruiting participants with a diversity of education levels.

Sample: We plan on testing with approximately 28 people total: fourteen English and fourteen Spanish speaking participants. We will aim to recruit participants that have not participated in previous testing for the decennial census; however, if we are struggling with meeting our recruitment goals, particularly for Spanish speaking participants, we may recruit participants who have participated in previous studies. We will recruit participants who have at least one year’s experience using the internet for tasks other than reading or writing email. Respondents that would be likely to complete the decennial census online will likely be familiar with the internet, so for this recruitment, familiarity with the internet is necessary. For the in-person sessions, participants will reside in the metropolitan DC area. The remote session participants can live anywhere in the U.S.

Language: Testing will be conducted in English and Spanish.

Timeline: Testing will be conducted in October and November 2023.

Method: Some sessions will be virtual with the test administrator and participant in different locations and others will be in person and will take place at locations convenient for participants. Remote participants will use their own laptop, desktop computer, or smartphone. In-person participants will use a Census-Bureau-issued laptop or their personal smartphone. Participants will be asked to think aloud as they read the mail materials and then work through the online instrument. We expect this study to take 90 minutes. Participants will be sent the privacy and consent information prior to the user session. Remote participants will be asked to join a 5-minute technology check session up to one week prior to the session to verify they can use the approved screensharing software Microsoft Teams. They will be asked to provide oral consent before they start the study.

Recruitment: Participants will be recruited using the approved generic screener questions.

Screens: Most of the questions to be tested are the same as those included in the 2020 Census, with the exception of four. Variable names for each new question in the spec, Enclosure 2, are included in parentheses. The first is a new address question (Residence_New). The second includes 2 versions a crowdsourcing vacancy question to identify vacant units (VAC_CROWD_1A/1B/2A/2B). The third is a question to determine if the roster of individuals includes everyone living at the residence (Whole/Partial) and the fourth is a targeted undercount question (Targeted_UC). The entire questionnaire is programmed in Qualtrics.

Protocol: The SmaRT testing of the mailing materials and the online instrument will employ a think-aloud protocol. Participants will be asked for their consent to participate in the study and given the privacy notice (Enclosure 1). In-person participants will be given the mail materials and remote participants will have the mail materials mailed to their homes (Enclosures 7A, 7B, 7C, 7D, 7E1, 7E2, 7F, 7G, 7H, 7I, 7J, 7K, 7L, 7M, 7N). All participants will be asked to read through them, one by one, while thinking aloud; then they will be asked a series of debriefing questions (Enclosures 3). They will then be asked to complete the 2024 Census Survey (online questionnaire) while thinking aloud (Enclosure 4). Then they will be asked satisfaction questions about their experience (Enclosure 5). Finally, they will answer a quizlette to assess their understanding of the census (Enclosure 6).

Below is a list of materials to be used in the study.

1. Specifications of the updated decennial census (Enclosure 2)

2. Protocol for cognitive testing of mailing materials including tasks & debriefing probes (Enclosure 3HUP or 3PQF)

3. Protocol for instrument testing and (Enclosure 4)

4. Satisfaction questions (Enclosure 5)

5. Quizlette (Enclosure 6)

6. Informed Consent and Privacy Notice (Enclosure 1)

7. Mailing materials (Enclosures 7A-7N)

Length of interview: The session should take 90 minutes for each of the 28 participants. We anticipate 5 minutes per participant for the technology check for remote participants.

Respondents will be screened using the Census Bureau’s generic screener. We estimate that we will screen 2-3 people for each successful recruit, resulting in 60 participants screened. The total estimated respondent burden for this request is 48 hours. See Table 1.

Table 1. Total Estimated Burden

Category of Respondent	No. of Respondents	Participation Time	Burden
Screening	60	5 minutes	5 hours
Tech check	10	5 minutes	1 hour
Usability Sessions	28	90 minutes	42 hours
Totals			48 hours

Use of Incentive: The total incentive for this project is $60. The test administrator will send the incentive money to the participant’s address using approved USPS priority mail. If the session is conducted in person, the test administrator will give the participant cash.

The contact person(s) for questions regarding data collection and statistical aspects of the design of this research is listed below:

Erica Olmsted

Center for Behavioral Science Methods

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-4893

[email protected]

Marcus Berger

Center for Behavioral Science Methods

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-8335

[email protected]

4