2020 Census
Thank you for your time today. My name is XX and I work at [the Human Factors and Usability group at/RTI International on behalf of/RSS on behalf of] the U.S. Census Bureau. I will be working with you today. We evaluate how easy or difficult Census products are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes live to a much larger group.
Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to take a video of the device as you use it and record the audio from this session to get an accurate record of your feedback. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session.
[Hand consent form; give time to read and sign; sign own name and date, start recording.]
Thank you.
Ok great. Today, we have about 5 or 6 different tasks we are going to ask you to do.
The first task is to complete two questionnaires. They collect information about you and how often you use different electronic devices. These questionnaires are not part of the questionnaire that you are going to fill out in a moment. We use information from these for our documentation.
[CSM: Have participant fill out the demographic questionnaire and the background mobile questionnaire as needed. You can leave the room and watch from observation room when they are ready to begin again if in Census lab. If off site—simply move away from where they are answering so they don’t feel like we are looking over their shoulder.]
Thank you.
Now, you will complete the main task.
Today you will be helping us evaluate the design of the 2020 Census by completing it using this laptop/your phone/your tablet.
If you run into any difficulties as you work on the questions, please don’t blame yourself. Any difficulties are the result of the design of the questionnaire, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants to help improve the questionnaire. I did not create the questionnaire, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make the questionnaire work well for everyone.
Think Aloud: I want you to answer the questions exactly the way you would if you were at your home, but with one major difference. I would like you to think aloud as you answer the questions. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.
Practice: Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?
[If respondent does not adequately complete the think aloud, model it for them to make sure they understand, i.e. say something like, “So for instance, if I live in an apartment I might think out loud and say, okay, I have one window in my bedroom, one window in my living room, and I have a little window in my front door, but it looks out onto a hallway. Should I count that? I think I shouldn’t. Okay, I have two windows then. Now an explanation like that helps me understand how you came up with your answer, and it lets me know that some people might think a window to the inside of a building counts, and some people might not count that. With this information we could go back to the researchers and give them ideas on how to improve the question.”]
Great that’s what I want you to do throughout our session. I will remind you to think aloud if you get quiet.
At the end of the session I will have some questions for you about your experience as you worked on specific screens and how satisfied you were with the questionnaire. We will open up some of the screens and talk about them.
Eye tracking Census staff: [Ensure we have an external audio recording device plugged into the laptop. (Mic and Web cameras work) If we eye-track we need to turn off Camtasia]
We are also going to record where you look on the screen as we work on our study. Now we will do a simple task that will allow the computer to find your eyes. To do this, please follow the dot with your eyes.
[Calibrate the participants’ eyes]
ID LOG-IN:
[Hand participant the mailing materials with their ID. If laptop, please start at www.google.com]
This is an example of mailing materials you would receive at your home if you were selected for the 2020 Census. Please take a moment to read it. Notice that the address is not your real address. If you were to receive the survey at your home, the mailing materials you would get would have your real address. You will have to pretend that this [letter/card] came to your address and the address in the survey is your address.
That is the only part of the questionnaire that is pretend. Please answer the survey questions as they apply to you in your real life.
NON-ID:
[Hand participant the paper/login card with URL and if laptop, please start at www.google.com]
We would like you to pretend that you heard about the 2018 Census Test/2020 Census on the radio and wanted to log onto the URL you heard to complete your census survey. This is the URL [point to the URL on the paper].
Now go online and complete the online census questionnaire. You should answer the survey questions as they apply to you in your real life.
[CENSUS INTERVIEWERS: After final SUBMIT is clicked, stop eye tracking and start Camtasia.]
Satisfaction questions:
Now I would like you to answer these questions about how satisfied you were with the questionnaire you just completed. (Administer satisfaction questionnaire.)
Knowledge check: [Read aloud or administer on paper. If needed, reassure respondent that there are no right or wrong answers and their answers will help us make sure that Census materials are easy for people to use.] The next set of questions are about your experience with the online census questionnaire and website. We will ask you about what you expected the online census website to be able to do and what you thought the text on the website meant. Do your best to answer the questions. There are no right or wrong answers. If you do not know the answer, it’s okay to [IF ADMINISTERING VERBALLY: say you don’t know / IF HAVING RESPONDENT COMPLETE INDEPENDENTLY ON PAPER: select that as your choice].
Do you think you are required by law to complete the Census?
Yes
No
I don’t know
{English interviews only} Do you think the online census questionnaire is available in languages other than English?
Yes
No
I don’t know
If you exit the questionnaire and log back in later to finish, do you think the answers you already entered will be saved?
Yes
No
I don’t know
If you don’t know the answer to a question on the Census, do you think you can leave it blank and move to the next question?
Yes
No
I don’t know
If you have people renting a room in your home who are not related to you, do you think you should include those renters on the census questionnaire?
Yes
No
I don’t know
Do you think one person can answer the census questionnaire for everyone in the household?
Yes
No
I don’t know
Do you think someone can select more than one race on the census questionnaire?
Yes
No
I don’t knowg
Do you think you should list someone on your Census form if they move into your home in mid-April while you are filling out the online Census questionnaire?
Yes
No
I don’t know
Do you think you need your Census ID to access the census questionnaire online?
Yes
No
I don’t know
In your opinion, how many reminders to answer the census should be mailed to people’s homes (like letters and postcards)?
Only 1
2
3
4
5
6
More than 7
Did you notice if the website you saw today had instructions on how to find your Census ID?
Yes
No
I don’t know
Did you notice if the website or census questionnaire that you saw today included an explanation about the confidentiality of your responses?
Yes
No
I don’t know
[Only complete the vignettes if you have enough time and if the situation didn’t come up spontaneously during the interview. To save time, consider priming a case on a laptop to show the date of birth screen for Jamie Doe.]
1st vignette: Date of birth screen –
We want to get some more feedback on the date of birth question. I’d like to tell you about a pretend situation. [Hand respondent a paper with vignette 1 written down and either read it aloud to them or have them read it aloud to you.]
“You recently had a new roommate move in named Jamie Doe. You don’t know Jamie’s age but Jamie recently graduated from college.”
Please show me what you would do to answer this question for Jamie and tell me why you are answering this. [Show them the primed date of birth screen for Jamie Doe. If they do not access help, watch what they do and then go back to the screen and say: “Please access the help for this question and let me know what that information means to you.”]
2nd vignette: Exit and Resume
Okay, now I’d like you to pretend that you had to leave the survey to do something else. What would you do? Go ahead and do that.
[If they say they would just leave it on the laptop, say “Let’s pretend you want to shut your laptop down for the night, now what would you do?” Observe to see if they use the log-out button. If they are going to shut the laptop down without logging out, stop them before they turn the laptop off.]
After they completed the task: If you came back to complete the survey at a later time do you think the information you entered previously would still be there? Why do you think that?
What would you do if it was not there and you had to start from scratch?
Debriefing:
Now we are going to look at some of the screens. [Open up the Debriefing slides and go through them as time allows. You can skip any screens that they did not receive. Now is the time to follow up on any observations you made or ask them if they have any comments on the screens. Treat these comments separately from any spontaneous comments during the session during analysis of findings. CSM puts these probes into PowerPoint in the Notes section and then prints off a copy to take notes on during the interview.]
On each screen:
Do you have any comments?
Any screen that caused user issues based on test administrator observing interview and noting down hesitations, laughter, changing answers, requests for assistance, etc.:
Tell me more about your experience with this screen.
Landing Page:
Look over the page and let me know anything that comes to mind.
[POINT TO THE MIDDLE SECTION: Why should I respond?/What to expect?] Did you read these paragraphs on this screen?
If Yes: What did you think this message was about?
If No: Please take a moment and read it now. What do you think this message is about?
Login Page:
Do you have any comments about this screen?
If non-ID case: Where would you find the Census ID? [Show subsequent screens if they said they would click on any of the links. Ask, “Do you have any comments about this screen?]
If non-ID case: [Show CAPTCHA screen] Do you have any comments about this screen?
If non-English case: Did you see the English button at the top? Do you have any comments on this? Was it easy or difficult to find [TARGET] language? Why?
Optional: [Point to “Log in Failed. Please try again” edit message or “Your account has been temporarily locked. You will be able to access the survey in 30 minutes” edit message.] What would you do if you encountered this screen?
Residence:
If non-ID case: Tell me more about your experience with this screen. Do you have any comments on the way it is laid out?
If non-ID case in applicable non-English language: If you were to write your own address, not the address on the Login Card, would you write your address in English or in <TARGET LANGUAGE>? Why?
If ID case in applicable non-English language: You did not see this screen, but some people are shown this screen when they fill out the questionnaire. Would you write your address in English or in <TARGET LANGUAGE> here? Why?
Rural_Route & PR Address (Applicable non-English cases)
You did not see these screens before but please take a look. What do you think they mean by “Rural Route Address”?
Homeless
You did not see this screen before but please take a look. What are they asking here? [IF NEEDED: Can you give me an example?]
Name and address
Do you have any comments on this screen?
If non-English and provided the name in TARGET LANGUAGE (or using diacritics), instead of English: Can you tell me how you decided to write your name in TARGET LANGUAGE?
Optional: [POINT TO INSTRUCTION, “We will only contact you…”] Did you read this instruction? What does the instruction mean in your own words?
Optional: Is that the phone number most likely to reach you? [If no, probe more about why R didn’t give a phone # (s)he answers.]
Optional: Is that the email address you are most likely to check? [If no, probe more about why R didn’t give an email (s)he checks often.]
Demographic Dashboard
Any comments on this page?
Was the order of the people listed on your questionnaire what you expected, or were you expecting a different order? Can you tell me more about that? [Remind them of the order if necessary]
Popcount:
Optional if they added people later: I believe you entered x people at this question, can you tell me a little more about why you did not include y or z at this question but you added them later?
Help text: [Point to “why we ask” paragraph at bottom.] Please take a moment to read this part now. Tell me what you think about this. Do you have any comments?
People (applicable non-English cases):
Do you have any comments on this screen?
If R had any issues with providing names: Please tell me about your experience with this screen.
If R provided names in TARGET LANGUAGE instead of English: How did you decide to enter the names in [TARGET LANGAUGE]?
[Point to the instruction about providing names] Did you notice this instruction when you were completing the survey? What does it mean in your own words?
Undercount 1 (NEW LANGUAGES ONLY: TG, PL, FR, HC, PT, JP):
What do you think they mean by 'foster children'?
Can you give me an example of 'people without a permanent place to live'?
Tenure:
Optional: Tell me more about why you chose [their answer].
What does the phrase “include home equity loans” mean to you in this question?
If you were completing the Census at your house, do you think you would need to mail any information to the Census Bureau after reading this question?
Relationship (NEW LANGUAGES ONLY: TG, PL, FR, HC, PT, JP):
Do you have any comments on this screen?
Did you have any difficulty finding the correct relationship?
Please read through these response options. Is there anything that is unclear to you? What is it? What do you think it means?
If not mentioned by R: What do you think they mean by ‘unmarried partner’? Does the translation sound natural in [TARGET LANGUAGE]? What about “roommate or housemate”?
Date of Birth:
Were there any household members whose age you didn’t know?
If so, what did you do? (Fill in what you knew? Leave the question blank? Making something up?)
Hispanic origin and then Race:
What were you thinking about when you answered this question [show Hispanic origin]?
How about this question [show race]?
Do you have any comments on the instructions, the question or answer options?
Optional: What word would you use to describe these choices?
Optional: What does “Select all boxes that apply.” mean to you?
Optional: What does “Note, you may report more than one group.” mean to you?
Optional , if R entered detailed origin(s) in the text box(es): I noticed that you typed in XXX (and YYY) when you were completing the questionnaire. Could you tell me more about your experience?
Optional , if R did not left detailed origin(s) in the text box(es) blank: I noticed that you did not enter any information in these boxes (after you marked your answer). Could you tell me more about your experience? (If not mentioned, point to “Enter, for example,…” instruction relevant to answers R marked: Did you notice this instruction when you complete this screen? Could you tell me what this means to you?)
Optional: Do you have any comments on the question?
Optional: What word would you use to describe these choices?
Optional: If someone identified with more than one of these groups, could they choose more than one or could they just choose one?
Optional: What should they do if their group was not listed?
Citizenship:
What were you thinking about when you answered this question?
Do you have any comments on the instructions, the question or answer options?
Were there any household members whose citizenship or year of naturalization you didn’t know?
Overcount:
Optional: What time period were you thinking of when you answered this question?
Optional: Verify or probe that they do not have any other address where they could be counted.
NEW LANGUAGES ONLY: TG, PL, FR, HC, PT, JP: What does “nursing home” mean to you in this question?
NEW LANGUAGES ONLY: TG, PL, FR, HC, PT, JP: What do you think they mean here by 'at a seasonal or second residence'? Can you give me an example?
Mailing materials:
English/Spanish cases: Now I’m going to have you read some letters that we are sending to people to tell them to participate in the census . There are 5 pieces of mail that we want feedback on. I’m going to give you these in order of when they would arrive, but look at the date of each of the letters to get a sense of when they would come to your door. Please think-aloud as you look at each letter.
Other languages: Now I’m going to have you read a letter that we are thinking sending to people to tell them to participate in the census. Please think-aloud as you look at the letter.
[Show Internet-first or Choice, but not both. Try to get an equal number of people to see each version. If possible, have the letters in the envelopes.]
Internet-first:
D-LF1: [If ID case: This is the letter that you used to log in to this survey at the beginning. / If non ID Case: The Census Bureau usually sends out a letter like this to each household to ask for participation.] About 80 percent households in the U.S. would receive this letter. These households would be in areas that are likely to have Internet access. [Allow them to read and wait to probe until they look up.] Do you have any comments?
What would you do if you were to receive this in your mail? [If needed: Do you think this letter would motivate you to take some action? What action would that be? If you received a letter like this at home by mail, do you think you would type the internet address and the User ID to access the survey? Why or why not?]
If non-English case: What about other [TARGET LANGUAGE] speakers? Do you think they would try to log in the survey to participate in the Census by reading this letter? Why or why not?
If non-English case: The initial letter from the Census Bureau will be only in English as you see. Is there any way to make this easier for [TARGET LANGUAGE] speakers to participate in the Census using this letter? [If needed: How can we improve this letter?]
D-LF2 [English and Spanish only]: Then a few days later, this letter would come. [Allow them to read and wait to probe until they look up.] Do you have any comments? What would you do if you were to receive this in your mail?
D-PF3 [English and Spanish only]: Now, if you had not answered on the Internet you would receive this postcard. [Allow them to read and wait to probe until they look up.] Do you have any comments? What would you do if you were to receive this in your mail?
D-LF4 [English and Spanish only]: Now, if you had still not answered on the Internet you would receive this package. [Allow them to read and wait to probe until they look up.] Do you have any comments? What would you do if you were to receive this in your mail?
D-P5 [English and Spanish only]: Now, if you had still not answered you would receive this postcard. [Allow them to read and wait to probe until they look up.] Do you have any comments? What would you do if you were to receive this in your mail?
Choice:
D-LC1: [If ID case: This is the letter that you used to log in to this survey at the beginning. / If non ID Case: The Census Bureau usually sends out a letter like this to each household to ask for participation.] About 20 percent of households in the U.S. would receive this letter. These households would be in areas that are not likely to have Internet access in their homes based on previous data. [Allow them to read and wait to probe until they look up.] Do you have any comments?
What would you do if you were to receive this in your mail? [If needed: Do you think this letter would motivate you to take some action? What action would that be? If you received a letter like this at home by mail, do you think you would type the internet address and the User ID to access the survey? Why or why not?]
If non-English case: What about other [TARGET LANGUAGE] speakers? Do you think they would try to log in the survey to participate in the Census by reading this letter? Why or why not?
If non-English case: The initial letter from the Census Bureau will be only in English as you see. Is there any way to make this easier for [TARGET LANGUAGE] speakers to participate in the Census using this letter? [If needed: How can we improve this letter?]
D-LC2 [English and Spanish only]: Then a few days later, this letter would come. [Allow them to read and wait to probe until they look up.] Do you have any comments? What would you do if you were to receive this in your mail?
D-PC3 [English and Spanish only]: Now, if you had not answered on the Internet or by mail you would receive this postcard. [Allow them to read and wait to probe until they look up.] Do you have any comments? What would you do if you were to receive this in your mail?
D-LC4 [English and Spanish only]: Now, if you had still not answered you would receive this package. [Allow them to read and wait to probe until they look up.] Do you have any comments? What would you do if you were to receive this in your mail?
D-P5 [English and Spanish only]: Now, if you had still not answered you would receive this postcard. [Allow them to read and wait to probe until they look up.] Do you have any comments? What would you do if you were to receive this in your mail?
CLOSING
Is there anything else you’d like to mention that we haven’t talked about?
Thank you very much for your help today. {Pay respondent and have him/her sign a receipt. Turn off recording devices.}
Optional Note-Taking Template: P#___
TA:______________________
Date: _____________________
Location: Headquarters / Other :________________________
Device: _____________________
WiFi or Dataplan
Browser type used:_____________________________________
Other browsers on device:________________________________
(Circle) ID / NonID-
# people:
Any multi-racial: Yes No
Any other residences: Yes No
Any edit messages:
Language toggle (if applicable):
R toggled independently / Interviewer had to direct R to toggle / Interviewer had to complete language toggle for R
Details:
Vignette 1: DOB
Notes:
Administered it / Did not administer it / Happened during “live” survey
Entered mm= dd= yyyy=
Edit appeared: Yes/ No
Read edit: Y/N/ Not sure
Entered age / Age calculated / No age
Accessed help / Did not access help spontaneously and had to be told to access help
Comments on help:
Vignette 2: Log-Out Task
Administered it / Did not administer it / Happened during “live” survey
What happened:
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Erica Olmsted Hawala (CENSUS/CSM FED) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-21 |