OMB Memo

American Community Survey (ACS) Respondent Perceptions of Burden Cognitive Interviews

Request: The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). We propose to conduct a cognitive pretesting evaluation of new questions about respondents’ perceptions of burden in the American Community Survey (ACS), conducted by staff in the Center for Survey Measurement (CSM).

Purpose: The ACS has been a “target of criticism for…excessive burden” and “generates a small but continuous stream of complaints to members of Congress” (National Academies of Sciences, Engineering, and Medicine, 2016). In response to these concerns, the American Community Survey Office (ACSO) has studied several possible methods of reducing burden, including but not limited to: softening the mandatory response messaging, reducing the number of Computer-Assisted Telephone Interview (CATI) and Computer-Assisted Personal Interview (CAPI) contact attempts, and removing questions from the survey. However, it is unclear if or how these changes have affected how burdened respondents feel after participation in the ACS.

We previously conducted focus groups in spring 2017 under the generic clearance for the collection of routine customer feedback (OMB number 0690-0030). The goal of these focus groups was to gather feedback from prior ACS respondents on 1) their overall level of perceived burden in being contacted by and responding to the ACS and 2) features of the ACS that contributed to or affected respondents’ level of perceived burden.

Using the feedback from prior respondents in the focus groups, we developed questions designed to capture respondents’ perceptions of burden in the ACS. The purpose of the current research is to test these questions. Following the testing, the American Community Survey Office (ACSO) may choose to further test or add questions about perceptions of burden to the production ACS.

Population of Interest: The ACS samples households from the general U.S. population. The planned cognitive pretesting evaluation will focus on assessing the experience of both prior ACS respondents and the general population.

Timeline: Testing will be conducted in two rounds, with the first round being conducted between September 2017-December 2017, and the second round being conducted between February-May 2018. Iterative changes may be made within rounds.

Language: Testing will be conducted in English only.

Method: Staff from CSM will conduct 96 cognitive interviews (48 in each round of testing). Approximately half of these 96 interviews will be conducted with prior ACS respondents, and the other half will be conducted with the general population. The protocol will differ slightly for each group of respondents, as described below.

Interviews will be held in the Washington, D.C. metropolitan area as well as in two to three other areas across the country. Selection of these additional areas will depend on the number of prior ACS respondent sample available.

Sample: For the prior ACS respondent interviews, Census Bureau staff will select households who responded to the ACS one to three months prior to the interviews in several areas of the country as noted above. If possible, we will attempt to recruit respondents on a rolling basis to ensure we are able to get participants who responded to the ACS very recently. We will exclude partial respondents.

For the general population interviews, we will use recruitment efforts as outlined below.

For both groups of respondents, we will attempt to recruit respondents from diverse demographic groups including race/ethnicity, age, gender, and household size to the maximum extent possible.

Recruitment: Prior ACS respondents will be recruited by telephone by CSM and/or staff in the Census Bureau contact centers. They may also receive an email reminder about the cognitive interview appointment after they agree to participate. We will attempt to recruit respondents who completed the ACS via a self-administered mode (mail or Internet) as well as those who completed the ACS via Computer Assisted Personal Interview (CAPI). We will provide the contact centers with the sample, a telephone script, and training. A copy of the script is included in Enclosure 1.

General population interviews will be recruited by CSM staff using a combination of word-of-mouth, fliers posted at local community organizations such as recreation centers, advertisements on Craigslist.com, and broadcast messages distributed through the Census Bureau’s daily all-staff broadcast emails. Recruitment advertisement text is shown in Enclosure 2. They will be screened using the Census Bureau generic pretesting screener with an additional question about who handles the mail for the household, as shown in Enclosure 3. In conjunction with existing screener questions about age, computer ownership, and internet use, this question will allow us to assign general population interviewees to their expected mode of response if they were to respond to the ACS.

This is a voluntary research study. Respondents can decline participation at any time. Interviews will be conducted at the Census Bureau headquarters and at locations convenient to interviewees such as libraries.

Protocol: Prior ACS respondents will be invited to participate in a 40-minute interview in which they will answer questions about their perceptions of burden in the ACS. They will also be debriefed about the contacts they received for the ACS and their experience completing the questionnaire. About half of these respondents will have completed the ACS via a self-administered mode (paper or internet), while the other half will have completed the ACS via Computer Assisted Personal Interview (CAPI).

General population respondents will be asked to complete the ACS prior to participating in an in-person interview. About half of these 48 respondents will be assigned to complete the ACS via a self-administered mode (most likely the paper questionnaire), while the other half will be assigned to complete the ACS via an interviewer-administered mode (most likely over the telephone). The ACS takes 40 minutes to complete, on average.

A few days after the respondent has completed the ACS, they will participate in a 35-minute in-person interview in which they will answer questions about their perceptions of burden in the ACS and be debriefed about their experience completing the questionnaire. These respondents will not be asked about contacts. If a general population respondent comes to an interview without completing the ACS, we will administer it during the interview.

The paper ACS questionnaire is shown in Enclosure 4, and the cognitive interview questions and protocol are shown in Enclosures 5 and 6. There are two versions of the protocol; the only difference between these versions is the order of the first two burden questions.

Consent: We will inform participants that their response is voluntary and that the information they provide is confidential and will be accessed only by employees involved in the research project. The consent form will also indicate that the respondent agrees that the interview can be audio and/or videotaped to facilitate analysis of the results (see Enclosure 7). Participants who do not consent to be video and/or audio-taped will still be allowed to participate.

Use of Incentive: We plan to offer an incentive of $40 to offset the costs of participation, such as travel and parking.

Below is a list of materials to be used in the current study:

1. Telephone recruitment script for prior ACS respondents (Enclosure 1)

2. Recruitment advertisement text (Enclosure 2)

3. Additional screener question (Enclosure 3)

4. Paper ACS questionnaire (Enclosure 4)

5. Questions about perceptions of burden (Enclosure 5)

6. Interview protocols (Enclosures 6A & 6B)

7. Consent form (Enclosure 7)

Length of interview: We estimate 40 minutes per respondent for the prior ACS respondents, and 75 minutes for general population respondents. The latter estimate includes time spent completing the ACS questionnaire prior to the cognitive interview. This results in a burden of 32 hours for prior ACS respondents and 60 hours for general population respondents, or 92 hours for both groups of respondents.

The pre-approved generic screening questionnaire will take approximately ten minutes per person, and the additional screening question specific to this research will take two minutes per person (see Enclosure 3).

We estimate that we will screen three people for each successful recruit for each of the 96 interviews. Therefore, we estimate a total of 288 people screened for a total of 9.6 hours (288 people at 2 minutes each). Thus, the total estimated burden for this research is 102 hours.

Table 1. Total Estimated Burden

Category of Respondent	No. of Respondents	Participation Time	Burden
Screening	288	2 minutes	9.6 hours
Cognitive Interviews – Prior ACS respondents	48	40 minutes	32 hours
Cognitive Interviews – General population respondents	48	75 minutes	60 hours
Totals			102 hours

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Jessica Holzberg
Center for Survey Measurement
U.S. Census Bureau
Washington, D.C. 20233
(301) 763-2298
[email protected]

2