2 Cognitive Interviewing Protocol

ATTACHMENT III: COGNITIVE INTERVIEWING PROTOCOL

1. PRE-INTERVIEW ACTIVIES

1. Introduction

Hello, my name is ______ and I work for the Census Bureau. Thank you very much for helping us out today. [Make small talk; ask about any trouble finding the interview location or, if in their home, make some compliments and thank them for allowing you to come to their home].

[If applicable: Before we get started let me mention that there are some observers watching the interview in another room. They are only watching to get an idea of how the questions in the survey we are going to talk about are working.]

Let me start by telling you a little bit about what we're doing here. Every 10 years the Census Bureau conducts a head count and it is extremely important that this be as accurate as possible. So we turn to people like you to find out if our questions make sense and are easy to understand and answer.

2. Task

I'll ask you to go through the Census form and fill it out as if it was the real thing, in 2020. But I'm mainly interested in your interpretations and reactions to the questions, and how you decided on your answers. So after you’re done filling out the form, we’ll go through it one section at a time.

There are no right or wrong answers, we just want to know your thoughts and reactions to the questions to help us make them better so that we can learn how to make the questions work for everyone.

3. Confidentiality

The things we will talk about in the interview today will only be used in our research to help us improve the survey. Your name will not be attached to anything you say. Direct quotes may be used in research papers and professional presentations, but your name and any names you might mention today will never be used in our reports and presentations. Your participation in this study is completely voluntary. You can refuse to answer or skip over any particular questions. If at any time you decide you do not want to go on, that is your choice and you may stop. Please feel free to ask me any questions at any time.

4. Audio Recording

I'd like to ask for your permission to audio record the session today. The main reason we record these interviews is so that we don't have to rely on notes or our memories later. This allows me to concentrate on what you're saying during the interview. What you say on the recording may be used only for research purposes and cannot be told to anybody else not working on this project except as required by law (20 U.S.C., § 9573). Do you have any questions about the recording?

• AFTER ANSWERING QUESTIONS AND GIVING FURTHER EXPLANATION, CONTINUE.

• IF THE PARTICIPANT DOES NOT WANT TO BE RECORDED BUT IS WILLING TO DO THE INTERVIEW, CONTINUE

• IS NO LONGER INETERESTED IN PARTICIPATING, THANK THE PARTICIPANT FOR HIS/HER TIME AND END THE INTERVIEW.

5. Written Consent Form

Before we start, I’m going to give you a written consent form.

HAND THE INFORMED CONSENT DOCUMENT TO THE PARTICIPANT

This document explains the purpose of the study and how we will protect your information.

[After respondent signs]: Thank you for signing the consent form. I will also sign this form to show that I agree to everything in the form.

• ASK PARTICIPANT IF (S)HE HAS ANY QUESTIONS ABOUT THE CONSENT FORM

• IF PARTICIPANT AGREES, OBTAIN THE RESPONDENT’S SIGNATURE; INTERVIEWER SHOULD ALSO SIGN THE FORM

• TAKE A SECOND COPY AND BOTH PARTICIPANT AND INTERVIEWER SIGN; GIVE ONE COPY TO THE PARTICIPANT TO KEEP

6. Verbal Consent on Recording

IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION AUDIO-TAPED:

I’m going to turn on the recorder now, and once it is on, I’m going to ask for your permission to record today, and then we will get started.

TURN ON TAPE RECORDER, ASK FOR PERMISSION TO RECORD ON THE TAPE, AND BEGIN THE INTERVIEW.

OK, let’s begin.

7. Mode Assignment (Paper, Internet Self-Response or NRFU)

Did you participate in the 2010 Census?

• Yes  Do you remember if you…

(a) mailed back the form

(b) answered the questions over the phone or

(c) a Census-taker came to your house or

(d) something else?

 Use answers from main screener, Q16a, to identify ISR versus paper mode

• No  NRFU mode

• DK/Don’t remember  NRFU mode

Q16 from Main Screener: Do you have at least one year of Internet experience?

• Yes  16a

• No (skip to next question)  PAPER mode

16a. How often do you use the Internet either on a computer or on a mobile device like a

smartphone or tablet?...Several times a day, about once a day, a few times a week, a few times a month, or a few times a year?

• Several times a day  ISR mode

• About once a day  ISR mode

• A few times a week  ISR mode

• A few times a month  PAPER mode

• A few times a year  PAPER mode

• Not at all [do not read to respondent]  PAPER mode

PAPER MODE

2. MAIN INTERVIEW

1. Let’s start with the number of people you wrote down in Question 1; you recorded [X].

• How did you come up with that number?

• Did you have any questions about who to include? [why/why not]

• Did you have questions about who to exclude? [why/why not]

• At the beginning, the set of instructions asks you to count “people living in this house…” Can you tell me how you interpreted the word “living”?

• The set of instructions and Question 1 are worded like this (read below). How do you interpret each of those phrases? Do they mean the same thing to you or something different?

  • Instructions: “count the people living in this house.”

  • Question 1: “How many people were living or staying in this house…”

• [Unless the respondent talks about the instructions, hold off discussing those until you’ve gone through all the names of individuals listed on the form]

2. Question 2 asks about “additional” people.

   • What was your first reaction to this question?

   • How did you decide to check the box(se) that you checked?

   • [Probe on each response category checked]

     1. Meaning of terms/category

     2. How they decided it applied to their household

     3. Why they did not initially include that/those person/people in Q1

   • [Probe on meaning of terms/category of each response category not checked]

   • Let’s talk about the phrase “staying here on April 1, 2020”

     1. Can you tell me, in your own words, what that phrase means? What are some examples of what you think we mean by someone “staying here on April 1, 2020”?

     2. Now let’s talk about the date.

        1. April 1 is what we call “Census Day.” We need to count everyone once and only once, in the right place. To do that, we pick a day so we can get a snapshot of the population at a particular point in time.

        2. Next year when we do the Census for real, we’ll be out in the field collecting data for about a 2-month window, before and after April 1.

        3. Now that I’ve explained that, would you answer Q1 or Q2 any differently? [probe on how their notion of “staying here on April 1” relates to their answers to Q1 and Q2].

   • Did you notice you could check more than one box?

3. Now let’s talk about each person you listed on the form. First was [yourself/NAME].

• Why did you list that person as “Person 1”?

• Did you have any doubts about that?

• [Open probing on who Person 1 is, if it’s not the respondent]

  • Is the person a relative, landlord, etc?

  • Why does the respondent consider Person 1 “living or staying” at the residence?

• Was there another person you considered listing as Person 1?

• Let’s talk about the instructions in Q5:

  • Phrase “someone living here who pays the rent or owns this residence”?

  • How did you decide if someone who owns/rents “lives” at the “residence”?

  • If you decided no one who lives there own/rents, how did you decide who to select as “Person 1”?

4. How about Person 2?

• How did you decide who to list here?

• [open probing on who this person is to the respondent and why the respondent considered him/her to be “living” at the residence]

• How did you decide on your answer for Q2?

• What about the phrase “usually live or stay” with regard to [Person 2] – did you have any questions about what that meant?

• [If Person 2 sometimes stays somewhere else] Where does [Person 2] sometimes stay?

• [go thru response categories one at a time; ask respondent to say what these places mean to them]

• How did you decide what to answer for this question?

• [If respondent is NOT Person 1]: Do you think [Person 1] would have answered this question the same way you did? That is, you selected [category X] for how Person 2 is related to [Person 1]. How certain do you feel that if [Person 1] was filling out the form, they would also select [category X]?

5. Now let’s talk about Person 3. [Repeat protocol for Person 2]

6. Now let’s talk about Person 4. [Repeat protocol for Person 2]

7. Now let’s talk about Person 5. [Repeat protocol for Person 2]

8. After you finished answering questions for each person 1-6 about (sex, age, race), there was an instruction at the bottom right (shown below).

• Did you notice this instruction?

• Did you remember how many people you had listed in Question 1, or did you have to go back and look?

9. Now let’s talk about Person 6. [Repeat protocol for Person 2]

10. Now let’s talk about the instructions for what to do if the number of people listed in Q1 was 7 or more. These instructions were on the bottom right corner of the page about Person 6.

• Did you notice the instruction?

• Did you go back and look at what you wrote down in Q1?

• When you first wrote your answer to Q1, did you feel fairly certain or did you have some doubts?

• As you were writing down Persons 2-6, did you change your mind about who you should have included in the count you recorded in Q1? [Probe for whether they thought they should have included anyone they initially left out of the count, or whether they counted someone they later thought should not have been included].

• If your answer to Q1 was 7 or more, did you know where to record them?

• For Person 7 how did you decide what to answer for “Related to Person 1?”

• And how about for Person 8? (How did you decide what to answer for “Related to Person 1?”)

• And for Person 9?

• And Person 10?

• [If Q1 was more than 10 people]: What did you think when you got to Person 10 on the form?

11. Now let’s go back to the very beginning and talk about the instructions.

• Are there any babies or young children who sometimes stay at the residence?

• [Probe on respondents’ understanding/definition of each phrase]

  • “Count the people living in this…”

  • “live and sleep here most of the time”

  • “without a permanent place to live”

  • Let’s look at the that second half of the instructions –

    • institutions

    • college

    • Armed Forces

    • nursing home

    • jail

    • prison

    • detention facility

Did this list make you think of any other institutions? [probe for which ones; why].

12. Vignettes: Now let’s talk about other situations – not your own household. Let’s go back to the flyer you may have seen advertising the study, where we said:

You may be eligible if you live …

• with at least 4 people OR

• with 3 or more generations (e.g., grandparents, parents, grandchildren) OR

• with distant relatives, such as nephews/nieces, cousins, great grandchildren, etc. OR

• where not everyone is related to each other OR

• where someone has moved in or out recently

• Can you think of any household that meets at least one of these descriptions – any neighbors, friends, family, etc?

• [if so] I’m going to give you another blank Census form and ask you to pretend you are filling it out for that household. First try to think of who within that household would be the one to fill out the form for everyone. Then try to put yourself in their shoes and go through the form, ok?

• [On blank form, highlight the following questions and ask them to answer these (and skip the rest)]:

  • Q1, Q2, Q5

  • Person2-6: Q1-Q3

  • Persons 7-10 (all Qs)

  • What would they do if there were more than 10 people?

NOTE: If there’s still time in the interview, we could administer some hypothetical vignettes. Based on the literature (Schwede et al 2018 AAPOR poster) the “top 3” household profiles where most young children were initially left off the form are:

• households with nonrelatives (26%)

• multi-generation households (21%)

• family households with other relatives (15%)

Based on O’Hare and Pollard presentation (Southern Demographic Association October 2018), the top independent variables for households with omitted young children:

• Young children living with nonrelatives

• Large households (7+ people)

• Linguistically isolated

• Child living with Grandparent

• Mobility

• Percent minority families

The scenarios below capture some of these characteristics. We could tweak, invent others, etc.

1. Woman in her 40s lives with her two teenage children, and her mother. They have an extra room and they are renting it out to someone they found on Craigslist, and the renter lives there with her baby.

2. Married couple in their 60s live with their daughter (in her 20s). Daughter’s friend, also in her 20s, has a toddler. Friend and toddler have been staying there for the past few months and will stay there until they can find a place of their own that they can afford.

3. Man in his 60s lives with his elderly parents, and with his son (in his 30s) and his son’s girlfriend. The girlfriend has a toddler, and on weekends the toddler stays with her biological father, who lives a few blocks away.

4. Married couple in their 30s live with their two young daughters, and their elderly father. The husband’s cousin, who just graduated from college, moved out of her college housing and is staying with them with her baby rent-free while she completes a 6-month internship. The cousin doesn’t know where she and her baby will live when the internship is over.

13. Finally, these are the questions we ask about each person listed on the form.

[not sure how to go about these probes – ask at general level, not specific to any one hh member, or pick a person, or what. It’ll be too long/too much to go thru each hh member for each question…thoughts?]

Page 20 of 20