Form 2016 Census Test p 2016 Census Test p 2016 Census Test protocol

2016 Census Test

Thank you for your time today. My name is XX and I work with the Human Factors and Usability group at the Census Bureau. I will be working with you today. In this lab, we evaluate how easy or difficult Census products are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes live to a much larger group.

Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to take a video of the device as you use it and record the audio from this session to get an accurate record of your feedback. Only those of us connected with the project will review the tape and it will be used solely for research purposes. Your name will not be associated with the tape or any of the other data collected during the session.

[Hand consent form; give time to read and sign; sign own name and date, start recording.]

Thank you.

Ok great.

I have two questionnaires that I’d like you to fill out. They collect information about you and how often you use different electronic devices.

[Have participant fill out the demographic questionnaire and the background mobile questionnaire. You can leave the room and watch from observation room when they are ready to begin again. If off site—simply move away from where they are answering so they don’t feel like we are looking over their shoulder.]

Thank you.

Today you will be helping us evaluate the design of the 2015 Census Test by completing it using [your smartphone/ your tablet/ this laptop].

If you run into any difficulties as you work on the survey, please don’t blame yourself. Any difficulties are the result of the design of the Web survey, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants to help improve the survey. I did not create the online survey, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make the online survey work well for everyone.

Think Aloud: I want you to answer the questions exactly the way you would if you were at your home, but with one major difference. I would like you to think aloud as you answer the questions. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.

Practice: Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?

Great that’s what I want you to do throughout our session. I will remind you to think aloud if you get quiet.

At the end of the session I will have some questions for you about your experience as you worked on specific screens and how satisfied you were with the survey. We may open up some of the screens and talk about them.

[Eye tracking for census staff cases]

We are also going to record where you look on the screen as we work on our study. Now we will do a simple task that will allow the computer to find your eyes. To calibrate your eyes, please follow the dot with your eyes.

[Calibrate the participants’ eyes]

[interviewer: if with ID]

[Hand participant the mailing materials with their ID]

This is an example of mailing materials you would receive at your home if you were selected for the 2016 Census Test. Please take a moment to read it. Notice that the address is not your real address. If you were to receive the survey at your home, the mailing materials you would get would have your real address. Since we cannot replicate that for the lab setting, you will have to pretend that this letter/card came to your address and the address in the survey is your address.

That is the only part of the study that is pretend. Please answer the survey questions as they apply to you in your real life.

Laptop/Desktop: We will bring up the online questionnaire for you.

[interviewer: if without an ID]

[Hand participant the paper with URL]

This is an example of mailing materials you would receive at your home if you were selected for the 2016 Census Test. Please take a moment to read it. Now go online and complete the online census questionnaire. You may answer the survey questions as they apply to you in your real life.

Laptop/Desktop with eye tracking: We will bring up the online questionnaire for you.

STOP Respondent after entered demographics for themselves and ask them to pretend that they had to leave the instrument to do something else.

Once they are exited, then ask them to resume. Interviewer instruction: You may either text them or type the correct URL for them if they had difficulty in the first round.

After final SUBMIT is clicked, have participant answer satisfaction questions.

Now I would like you to answer these satisfaction questions. (Hand paper questionnaire to participant.)

We have a few more tasks for you to complete.

Only complete this next task if you have enough time and if this situation didn’t come up during the interview.

We would like to get more feedback on the date of birth question. Now log back in with this ID and this PIN. The instrument should take you to the age and date of birth screen.

Is there any relative or friend of yours who you know, but you don’t know their complete date of birth or their age? What age information do you know about [that person]?

(If they have someone in mind for whom they do not know their birth date or age)

Now let’s look at the screen. Please pretend that [name of person] name is Jamie Doe. Tell me in your own words what you would do on this screen for that person.

Now go ahead and do that.

(If edit comes up and after person reads message.) Keep talking.

(If they cannot think of a person for whom they don’t know their age or date of birth):

You recently had a new roommate move in named Jamie Doe. You don’t know Jamie’s age but Jamie recently went out to celebrate her birthday this month. She also recently graduated from college.

Please show me what you would do to answer this question for Jamie and tell me why you are answering this way.

The next task is on the laptop. We are going to go over each of the screens you saw.

Debriefing: Ask all the probes using a power point slide with those screens.

Is there anything else you’d like to mention that we haven’t talked about?

[NON-ENGLISH INTERVIEWS ONLY] Thanks, we’re almost finished. I have just a few more questions for you.

Language fluency	Very Well	Well	Not well	Not at all
DB1. How well do you speak English?
DB2. How well do you read English?
DB3. How well do you write in English?
DB4. How well do you understand movies and videos in English?

Language use	1.Almost never	2.Some-times	3.Often	4.Almost always
DA1. How often do you think in English? Would you say…
DA2. How often do you speak English at home? Would you say…
DA3. How often do you speak English with your friends? Would you say…

Probes on DB1:

1. You answered that you speak English [R answer DB1]. Can you tell me more about why you answered the way you did?

2. When you answered how well you speak English, did you take accent into account?

3. When you answered how well you speak English, did you take reading into account?

4. IF R ANSWERED WELL AT DB1: What would you have to be able to do in order to answer “very well”?

5. ALL RS: For you, what is the difference between speaking English “well” and speaking “very well”? (If needed: What can people who speak ‘very well’ do that people who only speak ‘well’?]

6. (IF THEY REPORT ANY ENGLISH PROFICIENCY) At what age did you start speaking English?

What country were you born in?

When a form is available in both English and [Spanish/Chinese/Korean], in which language do you normally complete it? 

1. In Spanish / Chinese / Korean

2. In English

At what age did you start speaking Spanish?

Now that you have filled out the census form, what do you think is the most important use of the Census data?

How do you think the census data may benefit your community? And do you see any benefit of the census data at the personal level?

P#___

TA:______________________

Date: _____________________

Location: Headquarters / Other :________________________

Device: _____________________

WiFi or Dataplan

Browser type used:_____________________________________

Other browsers on device:________________________________

(Circle) ID / NonID- Coverage version:

Popcount (Circle) Y / N

Relationship version:

Accessing URL (if only on smartphone or tablet – for laptop, eye tracking is already set up the URL):

Used search / typed in URL

# attempts (when they hit enter, or when they chose a website from a search):___

# of incorrect sites accessed:____________

#spelling errors made:______________________

TA assist: Y / N

Oral comments (circle): None Other: (Describe)

Login: # attempts with submit: ______________

# of self-corrections before submit: ____________

TA assist: Y / N

Oral comments (circle): None Other: (Describe)

Re-entry: : ______________

Problems Y/ N : Describe

Used (circle): PIN or Verification question

TA assist: Y / N

Problems Y/ N

Email or phone number

Email provided Y / N Comments:

Phone # provided: Y / N Comments

Spontaneous comments on instruction (circle): None Confidentiality Other: (Describe)

Roster

# hhld members:

Any errors:

Spontaneous comments (circle): None Repetitive Comments on examples. Other: (Describe)

Popcount

Did count = # of people listed Y / N

Demographics:

Tenure and reference person answer:

Description of household (circle): Children Unrelated Unmarried Married Same-sex

Relationship Oral comments (circle): None Comments on examples. Other: (Describe)

Description of races selected:

Did anyone choose multiple or single

Race Oral comments (circle): None Comments on examples. Other: (Describe)

Race edit: Y / N

If Y: what was outcome:

How many ethnicities chosen:

Ethnicity edit: Y / N

Spontaneous comments:

AGE and DOB

Knew information / Made information up or guessed / Left blank / Other

Describe:

Get age edit: Y / N

If Y: what was outcome:

Spontaneous comment on baby instruction for age Y / N

Other Comments

Over Count Q

Anyone have another place: Y / N

Was it listed Y / N

Describe

Any errors in entering address info

Oral comments (circle): None Repetitive Comments on examples. Other: (Describe)

Submit screen

Spontaneous comments

12