Cognitive Testing of ACS Burden Questions (Version B)
PARTICIPANT
ID #:
_________________________
DATE: ____ / ____ / _____
INTERVIEWER’S
NAME:_________________________________________________
START
TIME: ______:
______ AM / PM
Introduction
Hello, my name is ________________, and I work for the Census Bureau. Thank you for agreeing to participate in our study. Let me start by telling you a little about what we will be doing today. The U.S. Census Bureau counts the population in the United States and also conducts various kinds of surveys.
Before surveys are conducted, it’s important to try out questions with the help of people such as yourself. It is important that the questions make sense, are easy to answer, and that everyone understands the questions the same way. Today we will be trying out some questions for a national survey called the American Community Survey.
If you agree to take part in this study, I will ask you to answer the questions as if this were the actual survey. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I will ask you some questions about your answers, or about the questions themselves. There are no right or wrong answers. Our purpose is not to compile information about you. Instead, your interview, along with those of others, will show us how to improve these questions for the survey.
Your participation in this interview is voluntary, and you may decline to answer any question at any time.
[IF OBSERVER PRESENT]: I also want to let you know that this interview is being observed by [DESCRIBE OBSERVER’S ROLE: e.g., someone who works on the American Community Survey]. They are observing the interview because they are also interested in how the questions work for people like you.
Informed Consent
[HAND RESPONDENT COPY OF THE CONSENT FORM.]
Before
we start, I would like you to read over the document in front of you.
This document explains a little bit about this interview and provides
information about your rights as a participant, such as that all
information you provide is confidential and we won’t use your
name in any report. It also asks for your permission to have this
session audio (IF
RELEVANT: and
video) recorded so
that I can concentrate on what you are saying rather than having to
take extensive notes. Please ask me any questions you have about
this document. Once you have finished reading the document, please
sign and date it.
[PARTICIPANT READS FORM THEN SIGNS AND DATES FORM.]
Thank
you for signing the consent form. I will also sign this form to show
that I agree to everything in the form.
[INTERVIEWER SIGNS AND DATES FORM.]
Do you have any questions about the process?
[IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION TAPED] I will now turn on the recorder.
[TURN ON RECORDER]
CONTINUE TO PRIOR RESPONDENTS BELOW OR SKIP TO NEW RESPONDENTS
Prior Respondents: Questionnaire Check
Because it may have been a little while since you answered the American Community Survey, I have a paper copy of the survey here. You may have answered the survey on paper, or you may have done it online, or with an interviewer who came to your home. You may not have answered every question. Please take a minute to flip through this copy of the survey and refresh your memory.
Before we begin, I want to verify that you completed the American Community Survey. Is that correct?
IF YES: Great. Now we are going to go through some additional American Community Survey questions. First, please answer this question. [HAND RESPONDENT COMPLETION TIME QUESTION]. AFTER COMPLETE: Thank you. For this next part of the interview, we are going to do something a little bit different. SKIP TO THINK-ALOUD PRACTICE, PAGE 5
IF NO OR NOT SURE: We will need to complete the American Community Survey before we continue. This will take 20-40 minutes. SKIP TO ACS ADMINISTRATION, PAGE 3
New Respondents: Questionnaire Check
Before we begin, I want to verify that you completed the American Community Survey. Is that correct? IF ASSIGNED MODE WAS PAPER, ASK RESPONDENT TO SHOW YOU.
IF YES: Great. Now we are going to go through some additional American Community Survey questions. First, please answer this question. [HAND RESPONDENT COMPLETION TIME QUESTION]. AFTER COMPLETE: Thank you. For this next part of the interview, we are going to do something a little bit different. SKIP TO THINK-ALOUD PRACTICE, PAGE 5
IF NO OR NOT SURE: We will need to complete the American Community Survey before we continue. This will take 20-40 minutes. CONTINUE TO ACS ADMINISTRATION BELOW
ACS Administration (if needed)
CONTINUE TO PAPER ADMINISTRATION BELOW OR SKIP TO CAPI ADMINISTRATION (PAGE 4) BASED ON EXPECTED OR ASSIGNED MODE.
IF EXPECTED MODE WAS INTERNET (PRIOR RESPONDENTS), ADMINISTER PAPER.
IF ASSIGNED MODE WAS CATI (NEW RESPONDENTS), ADMINISTER CAPI.
PAPER ADMINISTRATION
I am going to give you a paper copy of the American Community Survey. Please go through and answer these questions as you would if this form had been mailed to your home. Please pause when you get to the part of the form that says “Stop here.” I will tell you what to do next. LET RESPONDENT GO THROUGH ROSTER AND HOUSING QUESTIONS.
ONCE RESPONDENT HAS ANSWERED ROSTER AND HOUSING QUESTIONS, STOP AND COUNT HOUSEHOLD MEMBERS.
IF 3 OR FEWER HOUSEHOLD MEMBERS (INCLUDING RESPONDENT): Thank you. You can continue answering the American Community Survey as you would if this form had been mailed to your home.
IF MORE THAN 3 HOUSEHOLD MEMBERS (INCLUDING RESPONDENT): Thank you. If this were the real American Community Survey, you would answer the next questions about [NAMES OF ALL HOUSEHOLD MEMBERS]. However, in order to make sure we end this interview on time, please answer the remaining questions about [NAMES AS ASSIGNED BELOW] only.
ASSIGN 2 HOUSEHOLD MEMBERS TO ANSWER ABOUT IN ADDITION TO SELF:
FIRST PRIORITY: EXTENDED FAMILY MEMBER OR UNRELATED HOUSEHOLD MEMBER
SECOND PRIORITY: ONE OF EACH IF THERE ARE MORE THAN ONE IN PRIORITY #1
THIRD PRIORITY: IMMEDIATE RELATIVE
(ALLOW UP TO 45 MINUTES FOR INTRODUCTION AND ACS ADMINISTRATION. IF RESPONDENT IS TAKING TOO LONG, ENSURE THAT THEY HAVE AT LEAST ANSWERED ALL QUESTIONS FOR PERSON 1 AND 2, THEN SAY: In order to make sure we end this interview on time, we are going to move to the next part of the interview. CONTINUE TO “AFTER ADMINISTRATION COMPLETE” INSTRUCTIONS BELOW.)
AFTER ADMINISTRATION COMPLETE: Thank you. Please answer this question as well. [HAND RESPONDENT COMPLETION TIME QUESTION]. AFTER RESPONDENT COMPLETES: Great, thank you. Now we are going to go through some additional American Community Survey questions. For this next part of the interview, we are going to do something a little bit different.
CONTINUE TO THINK-ALOUD PRACTICE, PAGE 5
CAPI ADMINISTRATION
I am going to ask you questions from the American Community Survey. Please answer these questions as if I were an interviewer who had come to your home. ASK RESPONDENT ROSTER AND HOUSING QUESTIONS.
ONCE RESPONDENT HAS ANSWERED ROSTER AND HOUSING QUESTIONS, STOP AND COUNT HOUSEHOLD MEMBERS.
IF 3 OR FEWER HOUSEHOLD MEMBERS (INCLUDING RESPONDENT), CONTINUE ADMINISTERING SURVEY.
IF MORE THAN 3 HOUSEHOLD MEMBERS (INCLUDING RESPONDENT): Thank you. If this were the real American Community Survey, you would answer the next questions about [NAMES OF ALL HOUSEHOLD MEMBERS]. However, in order to make sure we end this interview on time, please answer the remaining questions about [NAMES AS ASSIGNED BELOW] only.
ASSIGN 2 HOUSEHOLD MEMBERS TO ANSWER ABOUT IN ADDITION TO SELF:
FIRST PRIORITY: EXTENDED FAMILY MEMBER OR UNRELATED HOUSEHOLD MEMBER
SECOND PRIORITY: ONE OF EACH IF THERE ARE MORE THAN ONE IN PRIORITY #1
THIRD PRIORITY: IMMEDIATE RELATIVE
(ALLOW UP TO 45 MINUTES FOR INTRODUCTION AND ACS ADMINISTRATION. IF RESPONDENT IS TAKING TOO LONG, ENSURE THAT THEY HAVE AT LEAST ANSWERED ALL QUESTIONS FOR PERSON 1 AND 2, THEN SAY: In order to make sure we end this interview on time, we are going to move to the next part of the interview. CONTINUE TO “AFTER ADMINISTRATION COMPLETE” INSTRUCTIONS BELOW.)
AFTER ADMINISTRATION COMPLETE, NOTE END TIME OF ACS INTERVIEW: ______. Great, thank you. Now we are going to go through some additional American Community Survey questions. For this next part of the interview, we are going to do something a little bit different.
CONTINUE TO THINK-ALOUD PRACTICE BELOW
Think-Aloud Practice
I will be asking you to answer some questions, but I’d also like you to think aloud as you answer the questions. Tell me everything you are thinking about as you go about answering each question. I may remind you to think aloud as we go through the survey questions.
Let’s begin with a practice question. Remember to try to think aloud as you answer.
Practice Question 1. How many windows are there in the place where you live? _____________
IF NEEDED: Try to visualize the place where you live, and think about how many windows there are in that place. As you count up the windows, tell me what you are seeing and thinking about.
PROBE IF NEEDED:
How did you come up with that answer?
Tell me more about that. Why did you say [ANSWER]?
I noticed that you hesitated. Tell me what you were thinking.
Okay,
let’s move on. CONTINUE
TO OVERALL BURDEN
______________________________________________________________________
Overall Burden (Question Administration and Probing)
INTERVIEWER: NOTE VERSION ASSIGNMENT B FOR INTERVIEW. RESPONDENTS WILL BE ASKED AND PROBED ABOUT VERSION B, AND THEN ASKED AND PROBED ABOUT VERSION A.
IF PAPER or internet, self-administer:
We are going to start with the first question. [HAND TO RESPONDENT] Please answer this question as you would if this form had been mailed to your home, and remember to think aloud as you answer.
HAVE RESPONDENT COMPLETE FIRST QUESTION on paper
if capi, interviewer administer:
I am going to ask you one question to start. Please answer this question as if I were an interviewer who had come to your home, and remember to think aloud as you answer.
read FIRST question to respondent and record answer on paper
NOTE ANY PROBLEMS YOU HAVE ADMINISTERING THE QUESTION TO THE RESPONDENT, SUCH AS DIFFICULTY READING THE RESPONSE OPTIONS
ENCOURAGE RESPONDENTS TO THINK ALOUD DURING ADMINISTRATION OF THE QUESTION:
Remember to tell me what you what are thinking about as you answer this question.
How did you come up with your answer?
AFTER QUESTION ADMINISTRATION:
Thank you. Before we move on, I would like to talk a little bit more about how you came up with your answer so I can see how we can improve this question.
IF PAPER: ALLOW RESPONDENT TO KEEP PAPER.
IF CATI/CAPI: HAND QUESTION SERIES TO RESPONDENT. Here is the answer I recorded for this question.
IF RESPONDENT DOES NOT THINK ALOUD, ASK HOW THEY CAME UP WITH THEIR ANSWER BEFORE EXISTING PROBES.
1B. [VERSION B] How much of a hassle was the American Community Survey for you?
|
Version
B Probes:
What words or phrases would you use to describe your experience taking the American Community Survey?
What does “a hassle” mean to you?
What are some other words or phrases that mean the same thing as “a hassle”?
CARD SORT #1: PICK UP ACTIVITY CARDS, SHUFFLE, AND HAND TO RESPONDENT. Next, we are going to do something a little bit different. I am going to hand you a stack of cards. Each card has a different kind of activity listed on it. I would like you to place these cards into the four categories of the question you just answered. [LAY OUT CARDS WITH RESPONSE CATEGORIES AND GESTURE TO]: Thinking about when you complete these activities yourself, put the activities that you think are very much a hassle in this pile, put the activities that you think are somewhat of a hassle in this pile, and so on. Remember, there are no right or wrong answers.
[GIVE RESPONDENT A FEW MINUTES TO COMPLETE TASK]
[AFTER RESPONDENT COMPLETES CARD SORT]: Thank you.
Card Sort #1 Probes:
IF RESPONDENT WANTS TO CHANGE CARD SORT DURING PROBING: Because it’s important for us to see how you think about the categories, I ask that you please don’t go back and change your answers. We can talk about any answers you would like to change later in the interview.
Can you tell me how did you decide what to put in each category?
If needed: What do they have in common?
Now I’d like us to look at each pile individually.
START WITH FIRST RESPONSE OPTION IN HASSLE QUESTION:
IF MORE THAN ONE CARD: Looking at the cards in pile [FIRST RESPONSE OPTION], how would you order these cards from more or less of a hassle within this category?
IF NOT MORE THAN ONE CARD, MOVE TO NEXT CATEGORY
[REPEAT FOR EACH PILE WITH MORE THAN ONE CARD IN RESPONSE OPTION ORDER]
I am going to take a photo of the cards so that I remember where you placed the activities. **TAKE PHOTO OF CARD SORT.**
When you first answered this question “How much of a hassle was the American Community Survey for you?”, you selected [RESPONSE OPTION]. Are there any other activities that you can think of that you would also consider to be [RESPONSE OPTION], other than the ones you saw on the cards?
IF RESPONDENT WANTED TO CHANGE CARD SORT DURING PROBING: Earlier, there was [a card / some cards] that you wanted to move. Can you tell me more about that?
ALTERNATE VERSION OF BURDEN QUESTION:
PAPER: [HAND RESPONDENT VERSION A] Now I’d like you to take a moment and read over this question. Please answer it thinking about your experience with the American Community Survey, and remember to think aloud.
HAVE RESPONDENT COMPLETE ALTERNATE QUESTION on paper
CATI: Now I’d like to ask a second question. Please answer it thinking about your experience with the American Community Survey, and remember to think aloud.
read ALTERNATE question to respondent and record answer on paper
NOTE ANY PROBLEMS YOU HAVE ADMINISTERING THE QUESTION TO THE RESPONDENT, SUCH AS DIFFICULTY READING THE RESPONSE OPTIONS
ENCOURAGE RESPONDENTS TO THINK ALOUD DURING ADMINISTRATION OF THE QUESTION:
Remember to tell me what you what are thinking about as you answer this question.
How did you come up with your answer?
AFTER QUESTION ADMINISTRATION:
Thank you. Before we move on, I would like to talk a little bit more about how you came up with your answer so I can see how we can improve this question.
IF PAPER: ALLOW RESPONDENT TO KEEP PAPER.
IF CATI/CAPI: HAND QUESTION SERIES TO RESPONDENT. Here is the answer I recorded for this question.
IF RESPONDENT DOES NOT THINK ALOUD, ASK HOW THEY CAME UP WITH THEIR ANSWER BEFORE EXISTING PROBES.
1A. [VERSION A] How burdensome was the American Community Survey for you?
|
Version
A Probes:
What does “burdensome” mean to you?
What are some other words or phrases that mean the same thing as “burdensome”?
What do you see as the difference between these two questions [POINT TO BOTH VERSIONS]?
Is one easier to answer than the other?
CARD SORT #2: PICK UP ACTIVITY CARDS, SHUFFLE, AND HAND BACK TO RESPONDENT. I would like you to look at the stack of cards again and sort the cards into four different piles. This time, the categories will be the four categories of the second question you answered: [LAY OUT CARDS WITH RESPONSE CATEGORIES AND READ THEM AS THEY ARE PLACED]. Thinking again about when you complete these activities yourself, put the activities that you think are very burdensome in this pile, put the activities that you think are moderately burdensome in this pile, and so on. Remember, there are no right or wrong answers.
[GIVE RESPONDENT A FEW MINUTES TO COMPLETE TASK]
[AFTER RESPONDENT COMPLETES CARD SORT]: Thank you.
Card Sort #2 Probes:
IF RESPONDENT WANTS TO CHANGE CARD SORT DURING PROBING: Because it’s important for us to see how you think about the categories, I ask that you please don’t go back and change your answers. We can talk about any answers you would like to change later in the interview.
Can you tell me how did you decide what to put in each category?
If needed: What do they have in common?
Now I’d like us to look at piles individually.
START WITH FIRST RESPONSE OPTION IN BURDENSOME QUESTION:
IF MORE THAN ONE CARD: Looking at the cards in the pile [FIRST RESPONSE OPTION], how would you order these cards from more to less burdensome within this category?
IF NOT MORE THAN ONE CARD, MOVE TO NEXT CATEGORY
[REPEAT FOR EACH PILE WITH MORE THAN ONE CARD IN RESPONSE OPTION ORDER]
I am going to take a photo of the cards so that I remember where you placed the activities. **TAKE PHOTO OF CARD SORT.**
When you first answered this question “How burdensome was the American Community Survey for you?”, you selected [RESPONSE OPTION]. Are there any other activities that you can think of that you would also consider to be [RESPONSE OPTION], other than the ones you saw on the cards?
IF RESPONDENT WANTED TO CHANGE CARD SORT DURING PROBING: Earlier, there was [a card / some cards] that you wanted to move. Can you tell me more about that?
Thank you. Let’s continue with some additional questions.
Burden Question Series (Question Administration and Probing)
CONTINUE IN SAME MODE OF ADMINISTRATION. HAVE RESPONDENT ANSWER ALL REMAINING QUESTIONS, AND THEN PROBE.
QUESTIONS 2 AND 3 ARE FOR PRIOR ACS RESPONDENTS ONLY.
IF PAPER or internet:
Here are the next questions I would like you to answer. [HAND TO RESPONDENT] Please answer these questions as you would if this form had been mailed to your home, and remember to think aloud as you answer.
HAVE RESPONDENT COMPLETE REMAING QUESTIONS on paper
if capi, interviewer administer:
Please answer these next questions as if I were an interviewer who had come to your home, and remember to think aloud as you answer these questions.
read REMAINING questions to respondent and record answerS on paper
NOTE ANY PROBLEMS YOU HAVE ADMINISTERING THE QUESTION TO THE RESPONDENT, SUCH AS DIFFICULTY READING THE RESPONSE OPTIONS
ENCOURAGE RESPONDENTS TO THINK ALOUD DURING ADMINISTRATION OF THE QUESTIONS:
Remember to tell me what you what are thinking about as you answer these questions.
How did you come up with your answer?
AFTER QUESTION ADMINISTRATION:
Thank you. Now I would like to talk a little bit more about how you came up with your answers, so I can see how we can improve these questions.
IF PAPER: ALLOW RESPONDENT TO KEEP PAPER.
IF CATI/CAPI: HAND QUESTION SERIES TO RESPONDENT. Here are the answers I recorded for these questions.
IF RESPONDENT DOES NOT THINK ALOUD, ASK HOW THEY CAME UP WITH THEIR ANSWERS BEFORE EXISTING PROBES.
IF NEW RESPONDENT, SKIP TO QUESTION 4 ON PAGE 11. IF PRIOR RESPONDENT, ASK ABOUT QUESTIONS 2 AND 3 TOGETHER.
|
|
Questions 2 and 3 Probes:
If needed: Tell me what you remember about being contacted for the American Community Survey.
If not apparent from respondent narrative: Do you remember about how many times were you contacted overall?
What does “appropriate” mean to you in question 3?
Would you have preferred to have been contacted in a different way?
NOTE ANY COMMENTS ON THE TONE OR TIMING OF THE CONTACT ATTEMPTS, MESSAGING
|
Question 4 Probes:
What does “effort” mean to you in question 4?
IF COMPLETED ACS PRIOR TO INTERVIEW: Did you consult any records?
IF COMPLETED ACS IN INTERVIEW: Would you have consulted any records if you were at home when completing the ACS?
IF COMPLETED ACS PRIOR TO INTERVIEW: Did you ask someone else in your household to help you complete the American Community Survey? Tell me more about that.
IF COMPLETED ACS IN INTERVIEW: If you had completed this survey at home, would you have asked someone else in your household to help you complete the American Community Survey? Tell me more about that.
IF COMPLETED ACS PRIOR TO INTERVIEW: Did you take any steps to make sure that the American Community Survey was a real survey?
If needed: Were you concerned that the survey may have been a scam?
If yes: What steps did you take? (If needed: Did you look up the survey online, or call the Census Bureau?)
IF COMPLETED ACS IN INTERVIEW: If you had received this survey at your home, would you have taken any steps to make sure that the American Community Survey was a real survey?
If needed: Would you be concerned that the survey may be a scam?
If yes: What steps would you take? (If needed: Would you look up the survey online, or call the Census Bureau?)
You said that you put [RESPONSE OPTION] into answering the American Community Survey. What other types of activities can you think of that you would also consider to require [RESPONSE OPTION]?
|
|
Question 6 Probes:
[IF SOMEWHAT OR VERY DIFFICULT]: Do you remember any particular questions that were difficult to answer?
7A. Would you say it is very invasive, or somewhat invasive?
|
Question 7 Probes:
What does “invading your privacy” mean to you in this question?
What does “respecting your privacy” mean to you in this question?
Do you remember any particular questions that you felt invaded your privacy?
Debriefing Questions
We are almost finished with the interview.
PRIOR RESPONDENTS ONLY: Thinking generally about the American Community Survey from the first time you were contacted until you finished answering the survey questions, what stood out to you the most about your experience?
Thinking about all of the other questions on this page besides the first two that we talked about [POINT TO BURDENSOME AND HASSLE QUESTIONS], which question(s) you answered today are the most important in capturing how you felt about your overall experience with the American Community Survey?
Was there anything else that stood out to you about your experience that we didn’t talk about today?
Thinking about these last questions that you answered today [POINT TO BURDEN SERIES], were there any questions you think some people might find difficult to answer?
Did you have any other thoughts about the last few questions we talked about today [POINT TO BURDEN SERIES]?
Those are all of the questions that I had for you today. Thank you very much for your participation.
[TURN OFF THE TAPE RECORDER.]
Here is an envelope containing your $40, and a voucher form that verifies I gave you the money. Please complete the information in the highlighted areas and sign and date.
[HAND THE CASH INCENTIVE TO THE PARTICIPANT AND COLLECT VOUCHER FORM.]
Cards:
1. The American Community Survey
2. Doing taxes
3. Moving
4. Dealing with health insurance
5. Handling an emergency
6. Getting a passport or driver’s license
7. Planning a major event
8. Paying bills
9. Commuting
10. Applying to jobs
11. Going to the bank
12. Cleaning
13. Voting
14. Grocery shopping
15. Exercising
16. Filling out forms at the doctor’s office
17. Eating
18. Watching TV
19. Going on vacation
20. Spending time with friends
21. Spending time on hobbies
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Jessica Holzberg (CENSUS/CSM FED) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |