Round 2 Protocol Version 2

Cognitive Testing of American Community Survey (ACS) Respondent Comment/Feedback and Ancestry

PARTICIPANT ID #: _________________________ DATE: ____ / ____ / _____

INTERVIEWER’S NAME:_________________________________________________

START TIME: ______: ______ AM / PM

Introduction

Hello, my name is ________________, and I work for the Census Bureau. Thank you for agreeing to participate in our study. Let me start by telling you a little about what we will be doing today. The U.S. Census Bureau counts the population in the United States and also conducts various kinds of surveys. Today we will be trying out some new questions for one of these surveys called the American Community Survey, or the ACS.

Before questions are added to surveys, it’s important to try out questions with the help of people such as yourself. It is important that the questions make sense, are easy to answer, and that everyone understands the questions the same way.

If you agree to take part in this study, I will ask you to answer the questions as if they were a part of the ACS. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I will ask you some questions about your answers, or about the questions themselves. There are no right or wrong answers. Our purpose is not to compile information about you. Instead, your interview, along with those of others, will show us how to improve these questions for the survey.

Your participation in this interview is voluntary, and you may decline to answer any question at any time.

IF OBSERVER PRESENT: I also want to let you know that this interview is being observed by (DESCRIBE OBSERVER’S ROLE: e.g., someone who works on the American Community Survey). They are observing the interview because they are also interested in how the questions work for people like you.

Informed Consent

• HAND RESPONDENT COPY OF THE CONSENT FORM.

Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant, such as that all information you provide is confidential and we won’t use your name in any report. It also asks for your permission to have this session audio (IF RELEVANT: and video) recorded so that I can concentrate on what you are saying rather than having to take extensive notes. Please ask me any questions you have about this document. Once you have finished reading the document, please sign and date it.

• PARTICIPANT READS FORM THEN SIGNS AND DATES FORM.

Thank you for signing the consent form. I will also sign this form to show that I agree to everything in the form.

• [INTERVIEWER SIGNS AND DATES FORM.]

Do you have any questions about the process?

IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION RECORDED:

• AUDIO RECORDING ONLY: I will now turn on the recorder. TURN ON RECORDER.

• VIDEO AND AUDIO RECORDING: I will need to leave the room for a moment to turn on the video recorder. I will be right back. (GO TO OBSERVATION ROOM AND TURN ON, THEN RETURN). Thank you for your patience. I will now also turn on this audio recorder, so that I have a back-up in case the video recording does not work. (TURN ON RECORDER)

ACS Administration

I am going to give you a copy of the American Community Survey. Please go through and answer these questions as you would if this form had been mailed to your home. I would also like you to think aloud as you answer the questions. Tell me everything you are thinking about as you go about answering each question. I may remind you to think aloud as we go through the survey questions.

I find it helps people to get the hang of thinking aloud while they are answering a question if they do a practice question first. Let’s begin with a practice question. Remember to try to think aloud as you answer.

How many windows are there in the place where you live? _____________

If needed: Try to visualize the place where you live, and think about how many windows there are in that place. As you count up the windows, tell me what you are seeing and thinking about.

• How did you come up with that answer?

• Tell me more about that. Why did you say [ANSWER]?

• I noticed that you hesitated. Tell me what you were thinking.

Okay, let’s move on.

So I’d like you to do something like that with many of the questions on the survey. I’ll let you know when to start and stop doing it. Sometimes I’ll remind you to think aloud if you are not doing it as much as I’d like you to. Do you have any questions before you start?

HAND ACS QUESTIONNAIRE TO RESPONDENT: Please go ahead and start filling out the questionnaire. Remember to tell me everything you are thinking as you decide on your answers.

FOR THINK-ALOUD SECTIONS, ENCOURAGE RESPONDENTS TO THINK ALOUD AS NEEDED THROUGHOUT INTERVIEW:

• Remember to tell me what you what are thinking about as you answer these questions.

• How did you come up with your answer?

	Person 1	Person 2	Person 3
Did R notice or verbalize noticing the write-in spaces for Black and White categories of the race question?
Did R check the box for the Black or White category but didn’t write in a detailed origin for either?

BEFORE RESPONDENT BEGINS HOUSING SECTION SAY: For the questions on this page, you don’t need to think aloud while answering.

Please pause when you get to the part of the form that says “Stop here.” I will tell you what to do next. LET RESPONDENT GO THROUGH ROSTER AND HOUSING QUESTIONS.

ALL RESPONDENTS

ONCE RESPONDENT HAS ANSWERED ROSTER AND HOUSING QUESTIONS, STOP AND COUNT HOUSEHOLD MEMBERS (AFTER HOUSING QUESTION 23 ON PAPER QUESTIONNAIRE.

IF MORE THAN 3 HOUSEHOLD MEMBERS (INCLUDING RESPONDENT):

SELECTION OF PERSONS TO PROBE ON:

WRITE RESPONDENT’S FIRST NAME AS PERSON #1 BELOW.

SELECT PERSONS 2-3 ACCORDING TO THE FOLLOWING CRITERIA:

1. Select someone who is a different race/ethnicity than Person 1.

2. Select someone who is unrelated to Person 1.

3. Select someone who is extended family to Person 1.

4. If no one meets criteria 1-3, select anyone.

Person 1 First Name _________________________

Person 2 First Name _________________________

Person 3 First Name _________________________

BEFORE RESPONDENT BEGINS DETAILED PERSON QUESTIONS: Now when filling out these next questions, please think aloud again, like you were doing before. Please continue until I ask you to stop again. Please answer Person 1 questions about yourself, Person 2 questions about (PERSON 2 NAME), and [IF PERSON 3 IS CHOSEN ABOVE: Person 3 questions about (PERSON 3 NAME).]

(ALLOW UP TO 35 MINUTES FOR INTRODUCTION AND ACS ADMINISTRATION. IF RESPONDENT IS TAKING TOO LONG, ENSURE THAT THEY HAVE AT LEAST ANSWERED ALL QUESTIONS FOR PERSON 1 AND 2, THEN SAY: In order to make sure we end this interview on time, you don’t need to complete the form for (Person 3). Please complete the form as if there were only 2 people in your household.

• MARK HERE IF THE RESPONDENT LOOKS AT THE BACK PAGE

Respondent Comment/Feedback (Question Administration and Probing)

IF FEEDBACK ITEM WAS FILLED IN DURING ADMINISTRATION, CONTINUE WITH “VERSION ON THE QUESTIONNAIRE.”

IF FEEDBACK ITEM WAS NOT FILLED IN DURING ADMINISTRATION:

DIRECT RESPONDENT TO FEEDBACK ITEM AT THE END OF THE QUESTIONNAIRE IF THEY ARE NOT ON THE PAGE. WHEN THEY ARE ON THE PAGE WITH THE FEEDBACK ITEM, CONTINUE BELOW.

• Now I would like you to take a moment and review this page. Imagine you had just finished completing the ACS at home. Please tell me what you would do.

  • IF R INDICATES THEY WOULD ANSWER FEEDBACK ITEM BUT DOESN’T FILL IT OUT: If you would answer this item POINTING TO FEEDBACK ITEM, please write down what you would say.

Version on the Questionnaire:

• In your own words, what is this asking you to do? POINT TO FEEDBACK ITEM

• IF FEEDBACK ITEM IS FILLED: How did you decide what to write here?

• IF FEEDBACK ITEM IS NOT FILLED: How did you decide to leave this blank? What kind of comments do you think would be appropriate to write here?

FROM THIS POINT ON IN PROTOCOL UNTIL COMPARING THE TWO PAPER VERSIONS, ONLY ALLOW RESPONDENT TO LOOK AT ONE VERSION. THEY WILL HAVE THE OPPORTUNITY LATER TO COMPARE.

TAKE BACK QUESTIONNAIRE.

Alternate Version:

HAND RESPONDENT ALTERNATE VERSION. I would like you to take a moment and review this page. Please do not fill this out.

• In your own words, what is this asking you to do? POINT TO FEEDBACK ITEM

• What kind of comments do you think would be appropriate to write here?

• IF NEEDED: Did you notice the black heading with the word “thank you”? (POINT TO BLACK HEADING)

• IF NEEDED: What do you think of this?

Comparing the two versions:

HAND RESPONDENT THE QUESTIONNAIRE (MAKE SURE THEY ARE ON THE PAGE WITH THE FEEDBACK ITEM).

• Thinking about the two versions of this page you saw today, (POINT TO EACH OF THE TWO VERSIONS), what do you see as the difference, if any, between these pages?

• What does the phrase “your experience” mean to you? POINT TO THE PHRASE “YOUR EXPERIENCE” ON THE APPROPRIATE FEEDBACK ITEM

• Do you think the amount of write-in space that is provided on these two versions is appropriate? IF NEEDED: Is there not enough space to provide feedback? Is there too much space? Or is it just the right amount?

• Do you think these are required items or optional items to fill out?

• IF RESPONDENT THINKS IT’S REQUIRED: Would you write anything if it were optional?

• Which one do you prefer, if any?

• If needed: Tell me more about that

• Is there one that you think is easier to answer than the other?

  • If needed: Tell me more about that

Debriefing Questions (ACS Respondent Comment Feedback)

1. Each of the versions you saw today had a space to provide comments. If you wanted to give feedback about the ACS, would you prefer to do it here, or in some other way? [MAKE SURE TO CONFIRM WITH RESPONDENT THE MODE THEY WOULD WANT TO PROVIDE FEEDBACK]

If needed: By sending an email or giving feedback online? By calling someone?

2. IF FEEDBACK ITEM FILLED OUT: Would you expect to receive a response from the Census Bureau about what you wrote here? IF FEEDBACK ITEM NOT FILLED OUT: Do you think other people who write something here would expect a response from the Census Bureau acknowledging to what they wrote? IF YES TO EITHER QUESTION: Can you tell me more about that? IF NEEDED: What type of response would you expect? Would you expect a letter or phone call in return? Would you expect the Census Bureau to take action on your comment?

3. What kinds of things do you think other people might write if they were asked to answer any of these statements?

If needed: Do you think people would write positive or negative comments here? Do you think people would use this space to complain about the ACS? Do you think people would interpret this item as a way of opting out of the survey?

Race and Ancestry (Probing)

HISPANIC ORIGIN AND RACE: Q5 and Q6, PAGES 2-4

Research Questions:

What were respondents’ overall reactions to the Hispanic origin and race questions?

Were they confused, or have any difficulty reporting for the Hispanic origin or race questions?

For the race question, did they notice they should provide detailed groups under the White and Black categories?

PERSON 1 PROBES

Now I’m interested in your reactions to some of the questions on the survey. Let’s look back at questions 5 and 6 about your race and origin.

IF THERE WAS ANY CONFUSION RAISED IN THE THINK ALOUD THAT MERITS MORE PROBING: When you were answering these questions, you (SUMMARIZE PROBLEM OR QUESTION RAISED DURING THINK ALOUD). Tell me more about that.

IF NOTHING WRITTEN IN: Did you notice the areas to write in your origin?

What made you decide {to write/not to write} in an answer?

PERSON 2 PROBES

IF THERE WAS ANY CONFUSION RAISED IN THE THINK ALOUD THAT MERITS MORE PROBING: When you were answering these questions, you (SUMMARIZE PROBLEM OR QUESTION RAISED DURING THINK ALOUD). Tell me more about that.

What made you decide {to write/not to write} in an answer for (PERSON 2)?

PERSON 3 PROBES

IF THERE WAS ANY CONFUSION RAISED IN THE THINKALOUD THAT MERITS MORE PROBING: When you were answering these questions, you (SUMMARIZE PROBLEM OR QUESTION RAISED DURING THINK ALOUD). Tell me more about that.

What made you decide {to write/not to write} in an answer for (PERSON 3)?

ANCESTRY: q12, PAGES 8, 12, 16

Research Questions:

How do respondents react and respond to the ancestry question after answering the Hispanic origin and race questions with write-in areas for White and Black added?

What were respondents’ overall reactions to the ancestry question?

Were they confused, or did they have any difficulty reporting?

Did their response to one question affect their response to the others?

Why did respondents answer the ancestry question differently from or similarly to the Hispanic race and origin questions?

PERSON 1 PROBES

IF ANSWER WAS WRITTEN IN ON Q5/Q6: How did you decide what to write in for this question compared to what you decided to write in for questions 5/6 on page 2 of the questionnaire?

Do you think this question is asking something different or the same as questions 5 and 6? What makes you say it’s {different/the same}?

IF NEEDED: Can you tell me what you know about your ancestry or ethnic origin?

IF NEEDED: How do you think race is different, if at all, from ancestry or ethnic origin?

PERSON 2 PROBES (IF PERSON 2 IS OF DIFFERENT ANCESTRY THAN PERSON 1, PAGE 12)

IF ANSWER WAS WRITTEN IN ON Q5/Q6: How did you decide what to write in for this question compared to what you decided to write in for questions 5 and 6?

IF NEEDED: Can you tell me more about [PERSON 2]’s ancestry or ethnic origin?

PERSON 3 PROBES (IF PERSON 3 IS OF DIFFERENT ANCESTRY THAN PERSON 1 OR PERSON 2, PAGE 16)

IF ANSWER WAS WRITTEN IN ON Q5/Q6: How did you decide what to write in for this question compared to what you decided to write in for questions 5 and 6?

IF NEEDED: Can you tell me more about [PERSON 3]’s ancestry or ethnic origin?

Debriefing Questions (Overall)

1. Was there anything else that stood out to you about your experience that we didn’t talk about today?

Those are all of the questions that I had for you today. Thank you very much for your participation.

TURN OFF THE RECORDER.

Here is an envelope containing your $40, and a voucher form that verifies I gave you the money. Please complete the information in the highlighted areas and sign and date.

HAND THE CASH INCENTIVE TO THE PARTICIPANT AND COLLECT VOUCHER FORM.

11