Online ACS Mobile Device

Usability Testing Protocol

2015 American Community Survey Internet Instrument Usability Test – Focus on Mobile/Tablet: Mobile Friendly design

Thank you for your time today. My name is XX and I work here with the Human Factors and Usability group. I will be working with you today. In this lab, we evaluate how easy or difficult Census products are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it.

Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to videotape the session to get an accurate record of your feedback. Only those of us connected with the project will review the tape and it will be used solely for research purposes. Your name will not be associated with the tape or any of the other data collected during the session.

[Hand consent form; give time to read and sign; sign own name and date, start recording.]

Thank you.

If you run into any difficulties as you work on the survey, please don’t blame yourself. Any difficulties are the result of the design of the Web survey, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants to help improve the survey. I did not create the online survey, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make the online survey work well for everyone.

Think Aloud: I want you to answer the questions exactly the way you would if you were at your home, but with one major difference. I would like you to think aloud as you answer the questions. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.

Practice: Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?

Great that’s what I want you to do throughout our session. I will remind you to think aloud if you get quiet.

At the end of the session I will have some questions for you about your experience as you worked on specific screens and how satisfied you were with the survey. We may open up some of the screens and talk about them.

[Eye tracking]

We are also going to record where you look on the screen as we work on our study. Now we will do a simple task that will allow the computer to find your eyes. I am going to have you position yourself in front of this stand keeping your arms outside of these bars here. The eye tracking devices is here and we need to keep the area from the device to your eyes clear.

[Indicate the bars, the and how they need to position their arms]

To calibrate your eyes, please look at this plate here. Look at the middle of the plate at the number 5. Now I am going to ask you to look at the different numbers, starting with 1. Look at 1. Now look at 2. Not look at 3. Now look at 4. Now look again at 5. Good.

[Calibrate the participants’ eyes]

Ok great. I now have a demographic questionnaire that I’d like you to fill out.

[Hand the demographic questionnaire to participant and wait in the room while they fill it out and return it to you]

Thank you. We are ready to begin the main portion of the session. You will help us evaluate the design of the American Community Survey by completing it using your mobile computing device / tablet.

[Hand participant the paper with their ID]

This is the paper that you will use when you fill out the American Community Survey online. Here is your User ID. If you were to receive the survey at your home, the mailing materials you would get would have your real address. Since we cannot replicate that for the lab setting, all participants will use the same address. For the purposes of this study, please pretend that your address is:

101 Main Street

Anytown, MD 20000

That is the only part of the study that is pretend. You may answer the survey questions as they apply to you in your real life.

[If in usability lab at Census Bureau HQ] Now I am going to go around to the other room and do a sound check. While I am doing that, please take a moment to complete this Background Questionnaire. This gives us information about your computer use and Internet experience. I am going to leave, but we will still be able to communicate through a series of microphones and speakers. Do you have any questions?

[Open background questionnaire on computer and Internet experience. Leave room. Once in control room do a sound check and start the eye-tracking software: Tobii Studio.]

[If in library say:] Now I am going to work on making sure my equipment is ready and working. While I am doing that, please take a moment to complete this Background Questionnaire. This gives us information about your computer use and Internet experience.

[Open background questionnaire on computer and Internet experience. Move to back desk away from the participant and do a sound check and start the eye-tracking software: Tobii Studio.]

Now it is time to complete the American Community Survey online. Remember to answer the questions as accurately as possible based on your living situation.

PIN screen LOG-OUT LOG Back in During Session

At the detailed person question for the 1^st or 2^nd person in the household -- somewhere in the detailed person questions—approx. around Q.18-21]

[Tell participant:]

Now I want you to imagine that you had to go somewhere and needed to save and continue answering the questions later. What would you do?

Okay now you need to log back in. Please do that now.

[Need participant to save and log out and then log in again, will have to help them if they cannot do it by themselves.]

After final SUBMIT is clicked, have participant answer satisfaction questions.

Now I would like you to answer these satisfaction questions.

Debriefing

Now I’ll ask you some debriefing questions that I have for you.