OMB Memo

2018 American Community Survey Respondent Burden Testing

Cognitive Testing of the “Ancestry” and “Class of Worker” Questions

on the American Community Survey

The U.S. Census Bureau in its continuing effort to reduce respondent burden on the American Community Survey (ACS) plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). Specifically, the U.S. Census Bureau will cognitively test interactions between the “Hispanic Origin and Race” questions with the “Ancestry” question and proposed changes to the “Class of Worker” question on the ACS. The revised question, developed under the auspices of the OMB Interagency Group on the American Community Survey and related subject-area subcommittees, is designed to facilitate respondent comprehension, reduce respondent burden, and where possible, streamline wording. For the “2018 ACS Respondent Burden Testing,” the U.S. Census Bureau has contracted with Westat, a statistical survey research corporation headquartered in Rockville, Maryland, to perform cognitive testing on the above mentioned topics.

The Census Bureau has proposed to revise the two separate questions on Hispanic origin and race on the 2020 ACS. The questions will be revised to have write-in fields for White and Black or African American; examples will be added for the categories White, Black or African American, and American Indian or Alaska Native; and revisions will be made to the examples for several categories. These changes are reflected in the “Questions Planned for the 2020 Census and the American Community Survey” (U.S. Census Bureau, 2018).¹ The ACS also asks a question later in the survey about ancestry. The expansion of the race question allows the Census Bureau to obtain detailed origin information for all household members, making it possible to consider whether the information already provided in response to the Hispanic origin and race questions may fulfill the data needs the ancestry question is intended to fill.

The 2016 Content Test included a race question very similar to what is proposed for the 2020 ACS. The 2016 Content Test cognitive testing as well as field testing indicated that some respondents found the ancestry question repetitive.² Additionally, analysis of the 2016 Content Test showed that between 72.5 percent and 74.0 percent of respondents provided the same response to the ancestry question that was reported in the race and Hispanic origin questions.³ The Census Bureau is conducting additional research to learn more about the perceived redundancies in the Hispanic origin/race and ancestry questions. This cognitive testing will help to understand whether or not the ancestry question remains to be perceived as repetitive, how respondents interpret the questions, and how information gathered from the Hispanic origin and race questions compares to information gathered from the ancestry question.

A revised Class of Worker (COW) question was tested as part of the 2016 ACS Content Test. The revised question grouped response categories under three general headings rather than one group list with no headings; added a checkbox for “Active Duty” military personnel; changed instructional text, question wording, and some category wording; and updated the question numbering. While the tested question was determined to be successful and will be implemented in the 2019 ACS, the rate that multiple checkboxes were marked increased (by four percentage points compared with the control version).⁴ The Census Bureau theorizes that the three general headings were interpreted by some respondents to mean they should mark one box under each heading. Respondents were expected to mark only one box for the whole question. For this round of cognitive testing, two versions of the COW question were developed to attempt to fix this issue. The focus of this cognitive testing is to understand how respondents interpret the headings and other elements of the question and why multiple responses are selected.

Cognitive Interviewing Period and Location: Cognitive interviewing will take place between May 29, 2018 and June 22, 2018, with recruiting beginning around May 21, 2018. Westat plans to conduct the interviews at its Rockville, Maryland location. If necessary, Westat will use additional locations in the Washington DC area to facilitate recruitment and interviews.

Conducting the Cognitive Interviews: Westat will perform 24 English-language cognitive interviews using a paper questionnaire (self-administered). For each interview, a respondent will answer a subset of the ACS questions. As part of the cognitive interview process, Westat interviewers will probe the respondents’ understanding and perception of the questions. Each cognitive interview will last approximately 60 minutes. The interviews will be audiotaped and may be subject to one-way observation from designated sworn employees of Westat and the Census Bureau.

Recruiting Requirements: Recruiting will take place between May 21, 2018 and June 18, 2018. All recruited respondents must be aged 18 and older and currently employed for pay. Respondents who reside in households with at least one other adult are preferred. In general, respondents are selected who also reflect a mix of age, sex, and level of education. We will also strive for each respondent to meet the requirements for both survey topics (see below).

• Ancestry: Respondents in specific race and ancestry groups were identified from research into the 2016 ACS Content Test where the answers in the questions were either likely to overlap completely or had specific differences that we hope to explore further, as follows:

• Respondents who are non-Hispanic white from any European ancestry

• Respondents who are black, African American, African, or Caribbean

• Respondents who are multiracial

• Respondents who are any of the following: Lebanese, Iranian, Syrian, Egyptian, Brazilian, Guyanese, Belizean, Jamaican, Haitian, Ethiopian, Nigerian, or other Caribbean or African heritage⁵

• Class of Worker: Respondents with the following characteristics were more likely to mark multiple boxes according to data from the 2016 ACS Content Test and are the focus of this round of cognitive testing:

• Respondents who work multiple jobs

• Respondents who are teachers from charter schools

• Respondents who are nurses (but not personal care aids; registered nurses preferred)

• Respondents who are self-employed, especially the following:

  • Owner/Operator truck drivers

  • Real estate brokers and sale agents

Recruiting Respondents: To recruit 24 respondents who meet the specified characteristics, Westat will use its own participant database as a starting point, and supplement it with recruits obtained through advertising. See the attached OMB-approved flyer template and the Craigslist ad template (Attachment B). If necessary, Westat will also use make use of its personal and business networks, using snowballing techniques where appropriate, and intercept interviewing (for example, going to a local truck stop).

Screening the Respondents: Westat will use a screener (see Attachment B) to screen potential respondents who respond to any of its outreach efforts. Respondents who are selected and complete an interview will receive $40 to offset the cost of participation (e.g., transportation, childcare costs). Westat will screen approximately 185 people to obtain 24 completed interviews. Screening will take approximately 5 minutes per person. Therefore, the initial screening of 185 people will take approximately 16 hours.

An interview will take approximately 60 minutes per respondent. Therefore, the total number of hours to interview 24 respondents will take 24 hours. The maximum burden is approximately 40 hours. The testing protocol is found in Attachment D.

Materials to be used for this Project:

Attachment A: ACS Cognitive Testing to Reduce the Burden and Difficulty of Questions: Research Goals, Recruiting Requirements, and Question Wording

Attachment B: ACS Cognitive Testing Recruitment Plan, Screener, Ad Template

Attachment C: ACS Cognitive Testing Consent Form

Attachment D: ACS Cognitive Testing Protocol Version 1

Attachment E: ACS Cognitive Testing Protocol Version 2

Attachment F: The American Community Survey, paper survey form ACS-1(X) ACOWP1

Attachment G: The American Community Survey, paper survey form ACS-1(X) ACOWP2V1

Attachment H: The American Community Survey, paper survey form ACS-1(X) ACOWP2V2

The contact person for questions regarding data collection and study design is:

Agnes Kee

American Community Survey Office

Room # 4K271

Washington, DC 20233

(301) 763-1516

[email protected]

1 https://www.census.gov/library/publications/2018/dec/planned-questions-2020-acs.html

2 https://www.census.gov/content/dam/Census/library/working-papers/2016/acs/2016_Westat_02.pdf

3 https://www.census.gov/library/working-papers/2017/acs/2017_Harth_01.html

4 https://www.census.gov/library/working-papers/2017/acs/2017_Martinez_01.html

5 Note that additional cognitive testing will be conducted by the Census Bureau’s in-house staff later in the year and will include other ancestry groups.