Cog Testing - Protocol

Cognitive Testing of the 2020 Census Attitudes Survey – Form 1

(Internal name: 2020 Census Tracking Survey)

PARTICIPANT ID #: _________________________ DATE: ____ / ____ / 2019

INTERVIEWER’S NAME: _________________________________

START TIME: ______: ______ AM / PM

SECTION 1. INTRODUCTION

WELCOME PARTICIPANT AND THANK THEM FOR THEIR TIME.

FIELDWORK INTERVIEWERS: PLEASE MAKE SURE TO WEAR YOUR BADGE.

READ OR PARAPHRASE: Hello, my name is ________________, and I work for the Census Bureau. Thank you for agreeing to participate in our study.

INTERVIEWER: Introduce process informally and begin to establish rapport. Be sure to make the following points:

• The Census Bureau is working to develop new questions for a public opinion survey that will be conducted over the telephone and online in 2019 and 2020.

• Before surveys are conducted, it is important to try out questions with the help of people such as yourself. It is important to test the questions to make sure that they make sense, are easy to answer, and that everyone understands the questions the same way. The survey asks questions about your attitudes toward the Census, awareness about the Census and Census participation.

• If you agree to take part in this study, I will read you the questions as if I were an interviewer contacting you at your home. After I ask you all the survey questions, I will ask some follow-up questions about your answers and what you understood the question to mean. There are no right or wrong answers. Our purpose is not to compile information about you. Instead, your interview, along with those of others, will show us how to improve these questions for the survey.

• I want to emphasize that my only purpose today is to see how well the survey works for people. Any information that you choose to share will not be shared in reports in a way that identifies you.

• If you have any questions or concerns at any point during the process, or any other feedback, please just let me know.

SECTION 2. INTERVIEW CONSENT

PLACE THE CONSENT FORM IN FRONT OF PARTICIPANT

Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant, such as all information you provide is confidential and we won’t use your name or specific circumstances in any report. It also asks for your permission to have this session audio recorded so that I can concentrate on what you are saying rather than having to take extensive notes. Please ask me any questions you have about this document. Once you have finished reading the document, please sign and date it.

PARTICIPANT READS FORM THEN SIGNS AND DATES FORM

INTERVIEWER SIGNS AND DATES FORM

Thank you for signing the consent form. I will also sign this form to show that I agree to everything in the form. At the end of the interview, I’ll be sure to give you a copy for you to take with you.

IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION AUDIO-TAPED: I will now turn on the audio recorder.

Your participation in this interview is very important because it will help the Census Bureau with improving these questions. Your participation in this interview is voluntary, and you may decline to answer any question, or stop the interview at any time. We’re scheduled to be here for about 90 minutes, and I won’t keep you any longer. As a thank you for your help, we will give you $40 after the interview is completed.

Do you have any questions?

WRITE DOWN QUESTIONS

SECTION 3. THINK ALOUD PRACTICE

While I do care about your answers to the questions, I am more interested in the process you go through in your mind when you’re coming up with your answers. So as we go through the survey, I would like you to tell me everything that you are thinking. Please remember that I am interested in how the questions works for you, so there are no right or wrong answers. I did not write the survey so don’t feel like to have to hold back your opinion.

I know this process might seem odd, so let’s do a practice question to give you an idea of what we are looking for. Remember to think aloud as you answer.

How many windows are there in your home?

• [PROBE/ENCOURAGE AS NEEDED]

  • What are you thinking about?

  • How did you come up with your answer?

  • I noticed that you hesitated. Can you tell me what you were thinking?

Great. So as you go through the survey, I’d like you to try to keep thinking aloud just like that. I may remind you periodically to think out loud if you forget. Do you have any questions for me before we start?

INTERVIEWER NOTE: There are two versions of the protocol. The only difference between them is which version of a given question they see first. All respondents see all versions of the questions.

SECTION 4. ADMINISTER 2020 CENSUS ATTITUDES SURVEY

[INTERVIEWER READS EACH QUESTION ALOUD AS IF YOU WERE CONDUCTING A TELEPHONE INTERVIEW]

SECTION 5. PROBES

AWARENESS. CRAT’S RESEARCH QUESTIONS:

For all participants:

1. How do people interpret the term “2020 Census”?

2. How do respondents react to the different versions of the question?

For Spanish-speaking participants:

1. Are participants able to differentiate the translations for “some” and “a little”?

Now, I would like to ask you a few questions about some of the questions you just answered

1. AWARENESS1

Here is the first question I asked you. [HAND R PAPER WITH AWARENESS1]

1. Can you tell me a bit about how you came up with your answer for this question?

2. This question mentions the 2020 Census. Based on your understanding, what is the 2020 Census?

3. How easy or difficult was this question for you?

2. AWARENESS2.v1

Next, I asked this question [HAND R PAPER WITH AWARENESS2.v1]

1. Can you tell me more about how you came up with your answer for this question?

2. How easy or difficult was this question for you?

3. Now I’m going to show you another version of this question. [HAND R PAPER WITH AWARENESS2.v2] Are these questions asking you the same thing or something different? Can you tell me more about that?

4. [GESTURE TO BOTH VERSIONS] Which version of the question do you prefer?

Which version did R prefer?

___ Version 1 ____ Version 2 ____ R had no preference

5. [IF R HAD A PREFERENCE] Can you say more about why you prefer that version?

[PROBES FOR SPANISH ONLY]

6. What does the response option “some” mean to you in this question?

7. What does the response option “a little” mean to you in this question?

8. Can you tell me more about the difference between the response options “some” and a “little”?

“ASK ALL BEGINNING MARCH 2020.” CRAT’S RESEARCH QUESTIONS:

1. Who is the respondent thinking about when including “anyone in your household”?

“ASK ALL BEGINNING MARCH 2020”

Here is another question I asked [HAND R PAPER WITH “ASK ALL BEGINNING MARCH 2020”]

1. How did you come up with your answer for this question?

2. The question asked you about “you or anyone in your household,” who were you thinking of when you were answering this question?

4. 

   INTENT. CRAT’S RESEARCH QUESTIONS:

   1. What does the concept of “questions on the census form” mean to respondents?

   2. How do respondents react to the different versions of the question?

   IntentA/IntentB

Here is another question I asked [HAND R PAPER WITH IntentA]

1. How did you come up with your answer for this question?

2. What comes to mind when you see the phrase, “questions on the census form” in this question?

[FOR ENGLISH, SKIP TO IV.6]

[PROBES FOR SPANISH ONLY]

3. What does the response option “probably will” mean to you in this question?

4. What does the response option “might or might not” mean to you in this question?

5. What does the response option “probably will not” mean to you in this question?

[PROBES FOR ALL RESPONDENTS]

6. Now I’m going to show you another version of this question. [HAND R PAPER WITH IntentB] Are these questions asking you the same thing or something different? Can you tell me more about that?

7. [GESTURE TO BOTH VERSIONS] Which version of the question do you prefer?

Which version did R prefer?

___ Intent A ____ Intent B ____ R had no preference

8. [IF R HAD A PREFERENCE] Can you say more about why you prefer that version?

9. [POINT TO INTENTB] One version of the question asked you about, “someone in your household,” who were you thinking about when you [read/answered] this question?

5. MODE

MODE. CRAT’S RESEARCH QUESTIONS:

1. How easy or difficult is to answer this question?

Another question I asked was this [HAND R PAPER WITH MODE]

1. When I asked you this question earlier, you answered [FILL: the internet/a paper form]. Why do you think your household will be more likely to answer on [FILL: the internet/a paper form]?

2. How easy or difficult was this question for you?

6. 

   PERSONALLY COUNTED. CRAT’S RESEARCH QUESTIONS:

   1. What does “personally” mean to respondents in the context of the question.

   PERSONALLY COUNTED

I also asked you this question [HAND R PAPER WITH PERSONALLY_COUNTED]

1. In your own words, can you tell me what is this question asking?

2. What does the word “personally” mean to you in this question?

7. 

   CONFIDENTIALITY. CRAT’S RESEARCH QUESTIONS:

   1. What does “confidential” mean to respondents?

   2. [SPANISH] What is the difference between somewhat and not too concerned?

   CONFIDENTIALITY

Another question I asked was this [HAND R PAPER WITH CONFIDENTIALITY]

1. What does the word “confidential” mean to you in this question?

2. In your own words, what does the phrase “the Census Bureau will not keep answers confidential” mean to you in this question?”

8. 

   SHARE DATA. CRAT’S RESEARCH QUESTIONS:

   1. What do respondents have in mind when asked about “other government agencies”?

   
   
   

   SHARE_DATA

Another question I asked was this [HAND R PAPER WITH SHARE_DATA]

1. What concerns came to mind when I asked you this question, if any?

2. [ASK IF R IS EXTREMELY, VERY, OR SOMEWHAT ASK] Can you tell me about how you came up with your answer to this question?

3. In your own words, what does the phrase “share answers with other government agencies” mean to you in this question?

4. What government agencies were you thinking of when you were answering this question?

9. 

   USED_AGAINST. CRAT’S RESEARCH QUESTIONS:

   1. What does “used against you” mean to respondents?

   USED_AGAINST

Another question I asked was this [HAND R PAPER WITH USED_AGAINST]

1. What concerns came to mind when I asked you this question, if any?

2. [ASK IF R IS EXTREMELY, VERY, OR SOMEWHAT ASK] Can you tell me about how you came up with your answer to this question?

3. What does the phrase “used against you” mean to you in this question?

10. 

    UNAUTHORIZED_USE. CRAT’S RESEARCH QUESTIONS:

    1. What does “unauthorized” mean to respondents?

    UNAUTHORIZED_USE

Another question I asked was this [HAND R PAPER WITH UNAUTHORIZED_USE]

1. In your own words, what does the term “unauthorized person” mean to you in this question?

2. What concerns came to mind when I asked you this question, if any?

3. [ASK IF R IS EXTREMELY, VERY, OR SOMEWHAT ASK] Can you tell me about how you came up with your answer to this question?

WITHOUT DOCUMENTATION. CRAT’S RESEARCH QUESTIONS. What does without documentation mean to respondents?

11. 

    WITHOUT_DOCUMENTATION. CRAT’S RESEARCH QUESTIONS:

    1. What does “without documentation” mean to respondents?

    WITHOUT_DOCUMENTATION

Another question I asked was this [HAND R PAPER WITH WITHOUT_DOCUMENTATION]

1. What does the phrase “without documentation” mean to you in the context of this question?

12. MESSAGE AWARENESS

Another question I asked was this [HAND R PAPER WITH Msg_aware1]

1. In your own words, what is this question asking?

Next, I asked this question [HAND R PAPER WITH Msg_aware2]

2. In your own words, what is this question asking?

CITIZENSHIP. CRAT’S RESEARCH QUESTIONS:

1. How many people are expecting a citizenship question in the census?

13. CITIZENSHIP/ALT TO PRETEST

Another question I asked was this [HAND R PAPER WITH CITIZENSHIP]

1. How did you come up with your answer to this question?

2. Can you explain in your own words what is this question asking?

3. How easy or difficult was it to answer to this question?

4. Now I’m going to show you another version of this question. [HAND R PAPER WITH ALT TO PRETEST] Are these questions asking you the same thing or something different? Can you tell me more about that?

5. [GESTURE TO BOTH VERSIONS] Which version of the question do you prefer?

Which version did R prefer?

___ CITIZENSHIP ____ ALT TO PRETEST ____ R had no preference

6. [IF R HAD A PREFERENCE] Can you say more about why you prefer that version?

14. ADHOC_GOVSHUT

Another question I asked was this [HAND R PAPER WITH ADHOC_GOVSHUT]

1. How did you come up with your answer for this question?

2. In your own words, what is this question asking?

15. TREND ITEMS

Another question I asked was this [HAND R PAPER WITH USE_FED_STATS]

1. In your own words, what does the term “federal statistics” mean to you?

Then I asked this [HAND R PAPER WITH TRUST_FED_STATS]

2. In your own words, what does the phrase “trust in the federal statistics” mean to you in this question?

16. HOUSEHOLD SIZE

Another question I asked was this [HAND R PAPER WITH HH_SIZE]

1. Who were you thinking of when you were answering this question?

2. Did you include people who are temporarily away from your household?

[IF YES:] Can you tell me who is temporarily away from your household?

3. Did you include babies? What about older children?

SECTION 6. DEBRIEFING

Those are all the questions I have for you.

Is there anything else you would like to tell us that you have not had a chance to mention yet?

Thank you for participating in this study. Your feedback has been very helpful.

TURN OFF TAPE RECORDER AND GIVE INCENTIVE. HAVE R SIGN THE PAYMENT VOUCHER.

Here is your money for helping me today. I need you to sign this payment voucher so the government knows that I gave you the money.

MAKE SURE TO GIVE A COPY OF THE CONSENT FORM (NOT THE ONE SIGNED BY THE RESPONDENT) TO THE RESPONDENT AT THE END OF THE INTERVIEW.