OMB Memo

2022 American Community Survey (ACS) Content Test Cognitive Interviews

Submitted Under Generic Clearance for Questionnaire Pretesting Research

Request: The Census Bureau, through a contract with RTI, plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). We will be conducting cognitive interviews to evaluate new and revised questions under consideration for the American Community Survey (ACS).

Purpose: In conjunction with OMB and stakeholders, the Census Bureau regularly re-evaluates the content of the ACS through Content Test cycles. At a high-level, this process involves:

1. Collecting proposals from federal agencies to modify existing questions or add questions on new topics,

2. Evaluating and selecting proposals based on merit,

3. Drafting question wording for selected proposals,

4. Cognitive testing of question wording,

5. Field testing through Content Tests,

6. Collecting public comments, and

7. Implementing approved changes to the production ACS.

The Census Bureau has already completed steps 1-3 in preparation for a Content Test in 2022. This submission is for the cognitive testing of question wording (step 4). There are 11 topics being tested, three of which are new to the ACS:

1. Household roster

2. Educational attainment

3. Health insurance coverage

4. Disability

5. Means of transportation to work

6. Income, weeks worked, and SNAP

7. Septic systems (new)

8. Electric vehicles (new)

9. Home heating fuel

10. Solar power (new)

11. Homeowners’ association fees

Cognitive testing will occur in three rounds. This submission is for the first two rounds of testing; a separate package will be submitted for the third round of testing at a later date.

Population of Interest: The ACS is sent to a nationally representative sample of households in the United States. The first two rounds of cognitive testing will focus on this population. The third round of cognitive testing will focus on Puerto Rico and group quarters, which are also sampled for the ACS.

Timeline: Two rounds of cognitive interviews will be conducted from July 2020 to May 2021. We plan to conduct a total of 335 interviews across the rounds, including 188 in the first round and 147 in the second round. If needed, up to 65 total additional interviews will be conducted in order to meet all recruitment requirements. This results in an upper limit of 400 interviews total for the two rounds. More detail on the breakdown of interviews by round and by other interview characteristics can be found in the Study and Recruitment Plan (see Enclosure 1).

Language: Interviews will be conducted in English (215 interviews) and Spanish (120 interviews). The additional 65 interviews may be conducted in English, Spanish, or both.

Sample: English interviews will target respondents who meet specific characteristics of interest for each of the topics being tested. For example, RTI will target respondents who have disabilities, have solar panels, or live in homes in which no one is related to each other. Some respondents will meet more than one recruitment criteria. A full list of characteristics by topic and the associated number of interviews for each can be found in Tables 2-3 and 2-4 in the Study and Recruitment Plan (see Enclosure 1) and screening questions associated with these topics can be found in Enclosure 2.

Recruiting efforts for Spanish interviews will focus primarily on obtaining diversity in Hispanic origin, with similar numbers of respondents of Mexican, Central American, South American, and Puerto Rican, Cuban, or Dominican origin being recruited (see Table 2-6 in Enclosure 1). An effort will be made to recruit a diverse set of respondents with the same characteristics as the English interviews to the maximum extent possible.

Recruitment: Respondents will be recruited through a variety of methods. RTI will use methods such as posting fliers at local community organizations, putting advertisements in community newspapers, and collaborating with staff at community-based organizations to identify potential respondents. RTI will also post advertisements on Craigslist.com and social media sites such as Facebook and Reddit. Finally, broadcast messages will be distributed to Census Bureau and RTI staff to leverage personal connections. A more detailed description of the recruitment methodology and copies of all recruitment materials can be found in Enclosures 3 and 4.

Method: Staff from RTI will conduct two rounds of up to 400 cognitive interviews across the two rounds. Interviews will be conducted remotely using audio or audio/visual software, or in-person in various cities and surrounding metropolitan areas across the country. Placeholder language to be included for remote interviews is provided in the materials. Areas will include Chicago, IL; Washington, D.C.; Research Triangle Park, NC; Saint Louis, MO; San Francisco, CA; Los Angeles, CA; Houston, TX; Miami, FL; New York, New York; Portland, OR; and Tucson/Phoenix, AZ. RTI will consistently monitor county-level COVID-19 trend data and local guidance throughout the data collection period to determine whether they feel safe sending interviewers out to do in-person interviews in a particular geography. If they do not feel safe doing so, only remote interviews will be offered for that geography. In instances where interviewers are approved to do in-person interviews, respondents will still be given the option to conduct the interview remotely.

In addition to testing in both English and Spanish, cognitive interviews will vary in several ways. In an effort to make recruitment more efficient and to shorten the length of the cognitive interviews, the 11 topics being tested have been divided into three groups. Respondents will be assigned to one group based on their responses to the screening questionnaire. We will shorten the ACS questionnaire for each of these groups so that no one group is asked to complete the entire ACS.

Group 1

• Household roster

• Septic systems

• Home heating fuel

• Solar power

• SNAP

Group 2

• Educational attainment

• Health insurance coverage

• Disability

Group 3

• Electric vehicles

• Homeowners’ association fees (Round 2 only)

• Means of transportation to work

• Income and weeks worked

Additionally, respondents will be assigned to one of three modes. We will test the paper questionnaire, as well as the computer-assisted interview (CAI) versions of the questions both with and without flashcards. The purpose of including two CAI modes is to test both in-person interviewing and telephone interviewing. In Round 1, we will test paper and CAI in English and CAI in Spanish. In Round 2, we will test all modes in both languages.

Respondents will be assigned to one of two versions of the questions in Round 1. One version of the questions will be selected for testing in Round 2 based on feedback from the first round. Finally, the homeowners’ association topic will only be tested in Round 2, because this topic was already extensively cognitively tested for the ACS in 2018. The suggested wording for this topic in this OMB package accounts for those findings.

Protocol: Respondents will first be asked to complete the ACS as they would if they received it at home or an interviewer had contacted them at home (see Enclosures 5-9 for paper questionnaires and CAI scripts). Researchers will observe respondents’ interactions with the survey and ask respondents probing questions such as how they came up with their answers and what certain terms and phrases mean to them. For a few topics, respondents will be asked to respond to hypothetical vignette scenarios. The interviews will conclude by asking respondents general debriefing probes about their experience. The introductory and concluding protocol text will be consistent across interviews and is shown in Enclosure 10. Enclosure 11 shows the variations on probing questions by group, mode, and version. Small changes to question wording and probing questions will be made between the first and second rounds of testing based on testing results.

Consent: We will inform respondents that their response is voluntary and that the information they provide is confidential and will be accessed only by employees involved in the research project. The consent form will also indicate that the respondent agrees that the interview can be audio and/or videotaped to facilitate analysis of the results (see Enclosure 12). Respondents can sign the consent form in-person for in-person interviews. For remote interviews, they can sign on their own print-out that is then securely emailed or mailed back to RTI or via a Qualtrics form that allows for signature. Respondents who do not consent to be video and/or audio-taped will still be allowed to participate.

Incentive: Respondents will receive $40 to offset the costs of participation in this research, such as childcare, internet or phone use, travel and parking.

Length of Interview: We estimate that each of the 400 interviews will take approximately one hour. This results in a burden of 400 hours total for the two rounds of cognitive testing.

The screening questions specific to this research will take approximately 17 minutes per person (see Enclosure 2). We estimate that we will screen 3 people for each successful recruit for each of the 400 interviews. Therefore, for the first round of interviewing, we estimate a total of 1,200 people screened for a total of 340 hours (1,200 people at 17 minutes each).

Thus, the total estimated burden for the two rounds of this research is 740 hours.

Table 1. Total Estimated Burden

Category of Respondent	No. of Respondents	Participation Time	Burden
Screening	1,200	17 minutes	340 hours
Cognitive Interviews	400	60 minutes	400 hours
Totals			740 hours

Below is a list of materials to be used in the current study:

Enclosure 1. Study and recruitment plan

Enclosure 2. Screening questionnaire

Enclosure 3. Recruitment methodology and ads

Enclosure 4. Email and text confirmation

Enclosure 5. ACS English Paper Questionnaire (Version 1)

Enclosure 6. ACS English Paper Questionnaire (Version 2)

Enclosure 7A and 7B. ACS English CAI Scripts (Group 1, Versions 1 and 2)

Enclosure 8A and 8B. ACS English CAI Scripts (Group 2, Versions 1 and 2)

Enclosure 9A and 9B. ACS English CAI Scripts (Group 3, Versions 1 and 2)

Enclosure 10. Protocol intro text

Enclosure 11. Protocol crosswalk

Enclosure 12. Consent form

The contact person for questions regarding data collection and the design of this research is listed below:

Jessica Holzberg

Center for Behavioral Science Methods

U.S. Census Bureau

Room 5K026B

Washington, D.C. 20233

(301) 763-2298

[email protected]

6