Protocol V1

OY2 ACS Respondent Burden Testing

Cognitive Interview Protocol: Version 1

Introduction

Hello, my name is ______________ and I work for Westat, a survey research company in Rockville, Maryland. Thank you for taking the time to participate in this research study.

Westat is working to develop new questions for the American Community Survey, which is sponsored by the U.S. Census Bureau. Before surveys are conducted, it’s important to try out questions with the help of people such as yourself. The survey asks questions about your household, yourself, and other people who live with you.

It is important that the questions make sense, are easy to answer, and that everyone understands the questions the same way. If you agree to take part in this study, I will hand you a questionnaire to answer. As you go along, I will ask you to explain what you are thinking and how you are coming up with your answers. There are no right or wrong answers. Our purpose is not to compile information about you. Instead, your interview along with those of others will show us how to improve these questions for a later survey.

The information collected in this interview is covered under OMB # 0607-0725.

Informed Consent

Before we get started, there are a few things I should mention. This is a research project, and your participation is voluntary. If you prefer not to answer any questions just say so, and you can go on to the next one. It’s also okay if you change your mind after starting and would rather not participate.

All your answers, everything you say, will be kept confidential. We will not use your name in any reports. We may use quotes in our report that you provide during the interview, but never in a way that would identify you. The interview will take about an hour and you will receive $40. We will also need to audio record our conversation. This helps me so I can listen to what you are saying and won’t have to take a lot of detailed notes while you are talking; it will also help when we write up a summary of this interview. Only project staff will have access to the recording and other project materials, and those materials will be stored according to Title 13 requirements for protecting the identity of individual respondents. {Finally, some of the researchers developing the questions are here today observing our interview to learn if there are things that might need to be changed.}

HAND CONSENT FORMS TO RESPONDENT, This form contains all of the things I just told you about your rights in this interview. Please read it over and sign both copies if you are willing to take part in the study.

HAVE R SIGN TWO CONSENT FORMS, KEEP ONE AND GIVE ONE TO RESPONDENT.

TURN ON RECORDER. The date and time is ____________. Now that the recorder is running, let me ask again, is it okay with you if we record this interview?

Protocol

In a moment, I am going to hand you the questionnaire. I’d like you to fill it out as if you received it in the mail at home. If you had received this form at home, your address would be located on the front of the questionnaire in the white area. Some questions ask about “this building.” For those, please continue to imagine you’re filling the survey out at home.

As you fill it out, I’d like you to think out loud as you come up with your answers to the questions. While I am interested in your answers, I am even more interested in the process you go through when you answer the questions. So, as much as possible, I’d like you to think aloud for me; meaning, talk me through your thought process as you answer each question. To help make this clearer, I’m going to give you an example.

If a question asked how many times I’ve eaten out at a restaurant in the past week, I would think out loud about my answer by saying:

“I’m going to assume that ‘restaurant’ includes places like fast food as well as full-service restaurants. I wonder if I should include drive-through places? I’m thinking maybe yes since I did eat there and it seems like the question wants to know where I ate. So I ate out on Sunday night, and then I went through a drive through to get coffee and breakfast on Wednesday, and then I ate out on both Friday and Saturday nights. So I guess I’m going to say four times.”

So I’d like you to do something like that with many of the questions on the survey. I’ll let you know when to start and stop doing it. Sometimes I’ll remind you to think aloud if you are not doing it as much as I’d like you to.

In addition, after you answer some of the survey questions, I will stop to talk with you about the questions you just answered. Then you will continue on with the survey and when you’re finished, we’ll talk about a few more questions. Before we begin, do you have any questions about these instructions?

HAND QUESTIONNAIRE TO RESPONDENT: Please go ahead and start filling out the questionnaire. Again, remember to read the questions out loud and tell me everything you’re thinking as you decide on your answers.

AS RESPONDENT ANSWERS Q5 AND Q6 FOR ANY RESPONDENT:

• MARK HERE IF R NOTICES OR VERBALIZES NOTICING THE WRITE-IN SPACES FOR THE BLACK AND WHITE CATEGORIES OF THE RACE QUESTION

• MARK HERE IF R CHECKS THE BOX FOR THE BLACK OR WHITE RACE CATEGORY BUT DOES NOT WRITE IN A DETAILED ORIGIN FOR EITHER

BEFORE RESPONDENT BEGINS HOUSING SECTION SAY: For the questions on this page, you don’t need to think aloud while answering.

STOP RESPONDENT AT END OF HOUSING SECTION (BOTTOM OF PAGE 5).

SELECTION OF PERSONS TO PROBE ON:

WRITE RESPONDENT’S FIRST NAME AS PERSON #1 BELOW.

ASK: Please tell me which adults ages 18 + in the household work for pay, either full-time or part-time.

IF NONE, ASK IF ANY ADULTS AGES 18+ HAVE WORKED FOR PAY IN PAST 5 YRS

IF NONE, ASK IF ANY TEENS AGES 15+ WORK FOR PAY.

IF NONE, SKIP TO THE 3 PRIORITY RULES BELOW

CHECK QUESTIONS 2 AND 5 ON ROSTER FOR THOSE WHO WORK/HAVE WORKED IN PAST 5 YEARS.

SELECT PERSONS 2-3 FROM AMONG THOSE WHO WORK/HAVE WORKED IN PAST 5 YEARS ACCORDING TO THE FOLLOWING PRIORITY:

1. Select someone who is a different race/ethnicity than Person 1.

2. Select someone who is unrelated to Person 1.

3. If all working persons are the same race/ethnicity and related to Person 1, select any.

Person 1 First Name _________________________

Person 2 First Name _________________________

Person 3 First Name _________________________

BEFORE RESPONDENT BEGINS DETAILED PERSON QUESTIONS: Now when filling out these next questions, please think aloud again, like you were doing before. Please continue until I ask you to stop again. Please answer Person 1 questions about yourself, Person 2 questions about (PERSON 2 NAME), and [IF PERSON 3 CHOSEN ABOVE: Person 3 questions about (PERSON 3 NAME).]

WHEN FINISHED, ADMINISTER FIRST SET OF PROBES.

HISPANIC ORIGIN AND RACE: Q5 and Q6, PAGES 2-4

Research Questions:

What were respondents’ overall reactions to the Hispanic origin and race questions?

Were they confused, or have any difficulty reporting for the Hispanic origin or race questions?

For the race question, did they notice they should provide detailed groups under the White and Black categories?

Let’s look back at questions 5 and 6 about your race and origin.

[IF THERE WAS ANY CONFUSION RAISED IN THE THINKALOUD THAT MERITS MORE PROBING] When you were answering these questions, you [SUMMARIZE PROBLEM OR QUESTION RAISED DURING THINK ALOUD]. Tell me more about that.

IF NOTHING WRITTEN IN: Did you notice the areas to write in your origin?

What made you decide {to write/not to write} in an answer?

PERSON 2 PROBES (PAGE 2)

[IF THERE WAS ANY CONFUSION RAISED IN THE THINKALOUD THAT MERITS MORE PROBING] When you were answering these questions, you [SUMMARIZE PROBLEM OR QUESTION RAISED DURING THINK ALOUD]. Tell me more about that.

What made you decide {to write/not to write} in an answer for (PERSON 2)?

PERSON 3 PROBES (PAGE 3)

[IF THERE WAS ANY CONFUSION RAISED IN THE THINKALOUD THAT MERITS MORE PROBING] When you were answering these questions, you [SUMMARIZE PROBLEM OR QUESTION RAISED DURING THINK ALOUD]. Tell me more about that.

What made you decide {to write/not to write} in an answer for (PERSON 3)?

ANCESTRY: q12, PAGES 8, 12, 16

Research Questions:

How do respondents react and respond to the ancestry question after answering the Hispanic origin and race questions with write-in areas for White and Black added?

What were respondents’ overall reactions to the ancestry question?

Were they confused, or did they have any difficulty reporting?

Did their response to one question affect their response to the others?

Why did respondents answer the ancestry question differently from or similarly to the Hispanic race and origin questions?

PERSON 1 PROBES

[IF ANSWER WAS WRITTEN IN ON Q5/Q6:] How did you decide what to write in for this question compared to what you decided to write in for questions 5/6 on page 2 of the questionnaire?

Do you think this question is asking something different or the same as questions 5 and 6? What makes you say it’s {different/the same}?

IF NEEDED: Can you tell me what you know about your ancestry or ethnic origin?

IF NEEDED: How do you think race is different, if at all, from ancestry or ethnic origin?

PERSON 2 PROBES (IF PERSON 2 IS OF DIFFERENT ANCESTRY THAN PERSON 1, PAGE 12)

How did you decide what to write in for this question compared to what you decided to write in for questions 5 and 6?

IF NEEDED: Can you tell me more about [PERSON 2]’s ancestry or ethnic origin?

PERSON 3 PROBES (IF PERSON 3 IS OF DIFFERENT ANCESTRY THAN PERSON 1 OR PERSON 2, PAGE 16)

How did you decide what to write in for this question compared to what you decided to write in for questions 5 and 6?

IF NEEDED: Can you tell me more about [PERSON 3]’s ancestry or ethnic origin?

HAND RESPONDENT SECOND PART OF QUESTIONNAIRE: Now I’d like you to finish filling out the questionnaire. (READ AS APPROPRIATE BASED ON WHETHER OTHERS IN HH ARE WORKING: For this part, person 2 is [PERSON 2] and person 3 is [PERSON 3]. As you did before, please think aloud as you are answering the questions.

AS RESPONDENT ANSWERS Q41 FOR PERSON 1:

• MARK HERE IF R SEEMS TO HAVE TAKEN A LONG TIME TO ANSWER

• MARK HERE IF R MARKED MORE THAN ONE BOX

CLASS OF WORKER: Q41, PAGES 11, 15, 19

Research Questions:

What message do the headers (version 1) convey to the respondent?

Are respondents confused by the question or the response options?

Do respondents understand they should only select one response?

How do self-employed respondents categorize themselves?

If a respondent works more than one job, how do they answer?

Do respondents read and understand the instructions on how to answer?

Do respondents correctly categorize their main or most recent employment?

Do respondents understand the purpose of the question?

PERSON 1 (Q41, PAGE 11)

CHECK RESPONDENT DATA SHEET: IF R IS SELF-EMPLOYED, USE “YOUR BUSINESS” IN THE FIRST PROBE AND LISTEN TO CONFIRM THAT R IS SELF-EMPLOYED AND NOT WORKING FOR SOMEONE ELSE WHO IS SELF-EMPLOYED

Before we look at any of the questions you answered, please describe for me in your own words {your business/the type of employer you work for}.

Now let’s focus on Question 41. What do you think this question is asking?

IF SELECTED MORE THAN ONE CATEGORY: I see you marked {read category 1} and {read category 2}. Can you tell me how you decided to select both of those?

Did you notice the green headers above the response choices?

What do you think is the purpose of the green headers in the question?

IF R NOTICED GREEN HEADERS, Would you say the green headers helped you answer the question, confused you about how to answer the question, or made no difference in how you answered the question? What makes you say they {helped you/confused you/didn’t make a difference}?

CHECK RESPONDENT DATA SHEET - IF RESPONDENT WORKS AT MORE THAN ONE JOB: How did you decide which job to answer about for this question?

COMPARE ALTERNATE VERSION (Q41):

I’d like to present you with another way the question might be asked. (HAVE R ANSWER ALTERNATE Q41)

How would you answer this other version of the question?

[SHOW R BOTH VERSIONS OF QUESTION 41] Does one of these seem easier to answer than the other? What makes you say that?

PERSON 2 PROBES (Q41, PAGE 15)

How did you decide what answer to pick for (PERSON 2)’s type of employer?

IF SELECTED MORE THAN ONE CATEGORY: I see you marked {read category 1} and {read category 2}. Can you tell me how you decided to select both of those?

PERSON 3 PROBES (Q41, PAGE 19)

How did you decide what answer to pick for (PERSON 3)’s type of employer?

IF SELECTED MORE THAN ONE CATEGORY: I see you marked {read category 1} and {read category 2}. Can you tell me how you decided to select both of those?

Closing and Incentive

IF OBSERVERS ARE PRESENT, CHECK TO SEE IF THEY HAVE FURTHER QUESTIONS.

Those are all the questions I have for you. Is there anything we haven't discussed that you would like to mention?

DISCUSS ANY RESPONDENT COMMENTS.

Thank you for your time.

STOP TAPE RECORDER.
GIVE INCENTIVE AND HAVE RESPONDENT SIGN RECEIPT.

OY2 ACS Respondent Burden Testing – Cognitive Interview Protocol Version 1 14