OMB Memo

2020 Census Privacy and Confidentiality Evaluation Cognitive Interviews

Submitted Under Generic Clearance for Questionnaire Pretesting Research

Request: The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). We will be conducting cognitive interviews of survey questions for a 2020 Census evaluation on privacy and confidentiality concerns and supplemental pretesting research online using Qualtrics.

Purpose: Privacy and confidentiality has been at the forefront of concerns as the census moves online and increases reliance on administrative records. The Census Bureau has been conducting research on respondents’ privacy and confidentiality concerns with online response and administrative records use as the focus of one of the teams from the Research and Testing phase leading up to the 2020 Census. Thus far, much of this work has been hypothetical, with respondents asked how they would feel if x strategy were to be employed in the census. The 2020 Census provides an opportunity to evaluate respondent privacy and confidentiality concerns and their relationship to response mode, item non-response, and mismatches between administrative records and self-reported data in a decennial census environment.

This submission is for the pretesting of the questionnaire to be used in the 2020 Census Privacy and Confidentiality Evaluation, a follow-up study to the 2020 Census. The respondent-facing name for this study will be the 2020 Census Experience Follow-up Survey. Households that respond to the census will be eligible for selection into the study. Respondents will be asked about privacy and confidentiality concerns, opinions on administrative records, concerns about decennial census items, and related constructs. The study will be interviewer-administered in both telephone and in-person modes.

Population of Interest: Residents of the United States who are likely to respond to the 2020 Census.

Timeline: Cognitive interviews will be conducted between June and September 2019. Online responses will also be collected through Qualtrics during this time period.

Language: Interviews will be conducted in English only for the in-person testing, and in English and Spanish for the online testing.

Method: Staff from the Center for Behavioral Science Methods will conduct one round of cognitive interviews with 10 respondents, followed by a second round of cognitive interviews with 15 respondents. Interviews will be conducted in-person at the Census Bureau headquarters and at locations convenient to interviewees in the local metropolitan area (DC, Maryland, Virginia). Researchers will also conduct supplemental pretesting research online using Qualtrics, which will occur at the end of the first round of cognitive testing but prior to the second round of testing.

Sample and Recruitment: We will recruit respondents for the in-person cognitive testing using the methods described below. Our recruiting efforts will target respondents who are most likely to handle the mail and react to it in a household, as we expect these respondents are the most likely to respond to the 2020 Census (see Enclosure 1 for question text). We will also recruit respondents with diverse demographic characteristics and household composition to the maximum extent possible.

For the online testing, we generally plan to follow the same recruiting criteria as with the in-person interviews. However, we will also emphasize geographic diversity of respondents and may target respondents with characteristics we find difficult to recruit for in-person testing (e.g., respondents in highly complex households). For Spanish respondents, we will also screen for English language proficiency and national origin using questions from the previously approved generic screener for Spanish interviews.

Recruitment: For the in-person testing, respondents will be recruited through fliers posted at local community organizations, advertisements on Craigslist.com, and broadcast messages distributed through the Census Bureau’s daily online newsletter. All recruiting materials are attached (see Enclosure 2). For the online testing, Qualtrics will select the sample from their panel and will handle their recruitment.

Protocol: Recruited respondents for the in-person interviews will be asked to complete the 2020 Census questionnaire prior to participating in an in-person interview. About half of these 25 respondents will be assigned to complete the census via a self-administered mode (most likely online via an instrument programmed in Qualtrics), while the other half will be assigned to complete the census via an interviewer-administered mode (most likely over the telephone). The census takes 10 minutes to complete, on average. If a respondent comes to an interview without completing the census, we will administer it during the interview. The paper census questionnaire is shown in Enclosure 3.

A few days after the respondent has completed the census, they will participate in a 60-minute in-person interview in which they will be administered the 2020 Census Privacy and Confidentiality Evaluation questionnaire by the cognitive interviewer, and be asked to answer the questions as they would if they were at home (see Enclosure 4). Researchers will observe respondents’ reactions to the questions and ask probes as needed to determine how respondents came up with their answer and whether any of the questions were confusing or difficult to answer. We will also ask probes about particular words and phrases of interest, such as “differential privacy.” The interview will conclude by asking respondents general debriefing probes (see Enclosure 5).

In the online testing, respondents will be asked to complete a shortened version of the 2020 Census questionnaire in which they respond only for themselves and one other person (when applicable). Then, they will be asked a subset of the questions from the 2020 Census Privacy and Confidentiality Evaluation questionnaire. The selection of these questions will be driven by expert review of the questionnaire and the results of the first round of cognitive testing. Questions flagged as potentially problematic during expert review or problematic for respondents in the first round of cognitive testing will be selected. Follow-up probes will be embedded in the online instrument. We may also embed split-panel wording experiments to test multiple potential revisions to these questions. Results from the online testing will be used to inform the second round of cognitive testing. An example of how this may look is shown in Enclosure 6.

Half of respondents will receive a 2020 Census questionnaire that does not include the citizenship question and the other half will test a questionnaire that includes the citizenship question. We will begin testing with the half that does not include a citizenship question. If a final decision is made on the inclusion of the citizenship question during the testing period, we will continue with the draft materials that match what is to be included in the 2020 Census.

Consent: We will inform respondents that their response is voluntary and that the information they provide is confidential and will be accessed only by employees involved in the research project. The consent form for the in-person testing will also indicate that the respondent agrees that the interview can be audio and/or videotaped to facilitate analysis of the results (see Enclosure 7). Respondents who do not consent to be video and/or audio-taped will still be allowed to participate.

Incentive: Respondents in the in-person testing will receive $40 to offset the costs of participation in this research, such as travel and parking.

Respondents in the Qualtrics testing will receive an incentive from Qualtrics. Per Qualtrics, the exact amount and form that each respondent receives can vary depending on the participant’s profile, how they were recruited, and the form of incentives they have elected to receive (i.e. e-gift cards, points, cash, etc.). Generally speaking, respondents receive ~$1.00-$1.50 or a relatively equivalent value for completing a 15 to 20-minute consumer survey. 

Length of Interview: We estimate that each of the 25 in-person cognitive interviews will take approximately 70 minutes (10 minutes to complete the census at home plus a one-hour interview). This results in a burden of 30 hours.

The pre-approved generic screening questionnaire will take approximately ten minutes per person, and the additional screening questions specific to this research will take two minutes per person (see Enclosure 1). We estimate that we will screen three people for each successful recruit for each of the 25 interviews. Therefore, we estimate a total of 75 people screened for a total of 3 hours (75 people at 2 minutes each).

The online testing instrument will be administered to 200 respondents and will take approximately 15 minutes to complete. This results in a burden of 50 hours.

Thus, the total estimated burden for the first round of this research is 83 hours.

Table 1. Total Estimated Burden

Category	No. of Respondents	Participation Time	Burden
Screening	75	2 minutes	3 hours
Cognitive Interviews	25	70 minutes	30 hours
Qualtrics	200	15 minutes	50 hours
Totals			83 hours

Below is a list of materials to be used in the current study:

Enclosure 1. Additional screening questions
Enclosure 2. Recruitment materials

Enclosure 3. Draft 2020 Census paper questionnaire (with citizenship question)

Enclosure 3a. Draft 2020 Census paper questionnaire (without citizenship question)

Enclosure 4. Evaluation of Privacy and Confidentiality Concerns questionnaire (with citizenship questions)

Enclosure 4a. Evaluation of Privacy and Confidentiality Concerns questionnaire (without citizenship question)

Enclosure 5. Cognitive interview protocol

Enclosure 6. Example online testing protocol (with citizenship question)

Enclosure 6a. Example online testing protocol (without citizenship question)

Enclosure 7. Consent form

The contact person for questions regarding data collection and the design of this research is listed below:

Jessica Holzberg

Center for Behavioral Science Methods

U.S. Census Bureau

Room 5K026B

Washington, D.C. 20233

(301) 763-2298

[email protected]

3