2015 National Content Test Usability and Cognitive Testing

2015 National Content Test

Thank you for your time today. My name is XX and I work with (the Census Bureau/RTI International/RSS). I will be working with you today. Our job is to evaluate how easy or difficult Census products are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes live to a much larger group.

Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to take a video of the device as you use it and record the audio from this session to get an accurate record of your feedback. Only those of us connected with the project will review the tape and it will be used solely for research purposes. Your name will not be associated with the tape or any of the other data collected during the session.

[Hand consent form; give time to read and sign; sign own name and date, start recording.]

Thank you.

Ok great.

[Spanish and English] I now have two questionnaires that I’d like you to fill out. One collects some information on your demographics and one collects information about how often you use different electronic devices.

[FOR CENSUS CASES: Have participant fill out the demographic questionnaire and the background mobile questionnaire. You can leave the room and watch from observation room when they are ready to begin again. If off site—simply move away from where they are answering so they don’t feel like we are looking over their shoulder.]

Thank you.

Today you will be helping us evaluate the design of the 2015 National Content Test by completing it using [your smartphone/ your tablet/ this laptop].

If you run into any difficulties as you work on the survey, please don’t blame yourself. Any difficulties are the result of the design of the Web survey, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants to help improve the survey. I did not create the online survey, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make the online survey work well for everyone.

Think Aloud: I want you to answer the questions exactly the way you would if you were at your home, but with one major difference. I would like you to think aloud as you answer the questions. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.

Practice: Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?

Great, that’s what I want you to do throughout our session. I will remind you to think aloud if you get quiet.

At the end of the session I will have some questions for you about your experience as you worked on specific screens and how satisfied you were with the survey. We may open up some of the screens and talk about them.

[CENSUS ONLY: Eye tracking]

We are also going to record where you look on the screen as we work on our study. Now we will do a simple task that will allow the computer to find your eyes. To calibrate your eyes, please follow the dot with your eyes.

[Calibrate the participants’ eyes]

[interviewer: if with ID]

[Hand participant the paper with their ID]

This is an example of mailing materials you would receive at your home if you were selected for the 2015 National Content Test. Notice that the address is not your real address. If you were to receive the survey at your home, the mailing materials you would get would have your real address. Since we cannot replicate that for the lab setting, please pretend that is your address.

That is the only part of the study that is pretend. You may answer the survey questions as they apply to you in your real life.

[interviewer: if Puerto Rican]

[Hand participant the paper with URL]

For this study, we are interested in how the Puerto Rican online survey works for people. Using this website, we would like you to complete the Puerto Rican form online. You will have to pretend that you still live in Puerto Rico. Can you remember the address of the last place you lived at while in Puerto Rico? Okay great, please pretend that you still live at that address.

That is the only part of the study that is pretend. You may answer the survey questions as they apply to you in your real life.

[CENSUS CASES ONLY: After final SUBMIT is clicked, have participant answer satisfaction questions.]

Now I would like you to answer these satisfaction questions. (Hand paper questionnaire to participant.)

We have a few more tasks for you to complete.

We would like to get more feedback on the date of birth question. Now log back in with this ID and this PIN. The instrument should take you to the age and date of birth screen.

Is there any relative or friend of yours who you know, but you don’t know their complete date of birth or their age? What age information do you know about [that person]?

(If they have someone in mind for whom they do not know their birth date or age)

Now let’s look at the screen. Please pretend that [name of person] name is Jamie Doe. Tell me in your own words what you would do on this screen for that person.

Now go ahead and do that.

(If edit comes up and after person reads message.) Keep talking.

(If they cannot think of a person for whom they don’t know their age or date of birth):

You recently had a new roommate move in named Jamie Doe. You don’t know Jamie’s age but Jamie recently went out to celebrate her birthday this month. She also recently graduated from college.

Please show me what you would do to answer this question for Jamie and tell me why you are answering this way.

The next task is on the laptop. We are going to go over each of the screens you saw.

Debriefing: Ask all the probes using a power point slide with those screens.

 

[Census and RTI Spanish cases only

[SPANISH INTERVIEWS ONLY, CENSUS AND RTI]

Thank you. We’re almost finished. I have just a few more questions for you.

Language fluency	Very well	well	Not very well	Not at all
How well do you speak English?	1	2	3	4
How well do you read in English?	1	2	3	4
How well do you write in English?	1	2	3	4
How well do you understand movies and videos in English?	1	2	3	4

Language use	Almost never	sometimes	Frequently	Almost always
How often do you think in English? Would you say …	1	2	3	4
How often do you speak English at home? Would you say...	1	2	3	4
How often do you speak English with your friends? ¿Would you say… How often do you speak English at work? Would you say…	1	2	3	4

Final probe for Spanish speakers; In what language do you normally fill out survey? In Spanish or English?

P#___

TA:______________________

Notes pages for interview summaries

Date: _____________________

Location: Headquarters / Other :________________________

Device: _____________________

WiFi or Dataplan

Browser type used:_____________________________________

Other browsers on device:________________________________

(Circle) ID / NonID- Puerto Rico only

Race version:

Ethnicity version: (Circle) Checkbox / Write in

Popcount (Circle) Y / N

Roster: (Circle) one at a time / 6 on screen

Accessing URL:

Used search / typed in URL

# attempts (when they hit enter, or when they chose a website from a search):___

# of incorrect sites accessed:____________

#spelling errors made:______________________

TA assist: Y / N

Oral comments (circle): None Other: (Describe)

Login: # attempts with submit: ______________

# of self-corrections before submit: ____________

TA assist: Y / N

Oral comments (circle): None Other: (Describe)

Email or phone number

Email provided Y / N Comments:

Phone # provided: Y / N Comments

Spontaneous comments on instruction (circle): None Confidentiality Other: (Describe)

Roster

# hhld members:

Any errors:

Spontaneous comments (circle): None Repetitive Comments on examples. Other: (Describe)

Popcount

Did count = # of people listed Y / N

Demographics:

Tenure and reference person answer:

Description of household (circle): Children Unrelated Unmarried Married Same-sex

Relationship Oral comments (circle): None Comments on examples. Other: (Describe)

Description of races selected:

Did anyone choose multiple or single

Race Oral comments (circle): None Comments on examples. Other: (Describe)

Race edit: Y / N

If Y: what was outcome:

How many ethnicities chosen:

Use predictive text Y / N / NA

Ethnicity edit: Y / N

Spontaneous comments:

AGE and DOB

Knew information / Made information up or guessed / Left blank / Other

Describe:

Get age edit: Y / N

If Y: what was outcome:

Spontaneous comment on baby instruction for age Y / N

Other Comments

Over Count Q

Anyone have another place: Y / N

Was it listed Y / N

Describe

Any errors in entering address info

Oral comments (circle): None Repetitive Comments on examples. Other: (Describe)

Submit screen

Spontaneous comments