Protocol

Usability Testing Protocol

MATERIALS NEEDED FOR INTERVIEW

• Interviewer Protocol Script

• Consent Form

• Invitation Letter with Username ID

• Satisfaction Survey

• Incentive

• Payment Receipt

• Pens and Paper

• Webcam with gooseneck clamp

• Laptop

• Extra Equipment and Charging Cords

INTERVIEWER USABILITY TESTING SCRIPT

Consent – Oral – Consent Form – Census Laptop Camtasia

Thank you for your time today. My name is [XX] and I work with the Human Factors and Usability Group at the U.S. Census Bureau. I will be working with you today. In this lab, we evaluate how easy or difficult our websites are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes live to a much larger group.

Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to take a video recording of your computer as you use it along with the audio from this session to get an accurate record of your feedback. Your name will not be associated with the recording or any of the other data collected during the session.

[Hand CBAMS consent form; give time to read and sign; sign own name and date. Show participants where the camera is and what part of the screen will be recorded start recording.]

START RECORDING – Census laptop

OK great. Thank you.

In today’s session you will help us evaluate the design of a Web survey that we are working on called the Census Attitudes Survey. You will do this by completing it using [your computer/this laptop/your smartphone/your tablet]. We are scheduled to be here for about 90 minutes, and I won’t keep you any longer. As a thank you for your help, we will give you $40 at the end of the session.

The survey is still under development. If you run into any difficulties as you work in the web instrument, please don’t blame yourself. Any difficulties are the result of the design of the web instrument, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants, to help improve the web instrument. I did not create it, so don’t feel like you have to hold back your thoughts to be polite to me. We appreciate your help so we can make the web instrument work well for everyone.

I want you to respond to the web instrument exactly the way you would if I wasn’t here, but with one major difference. I would like you to think aloud as you work within the instrument. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.

Practice Think Aloud - Oral

Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?

[Probe if they fall silent; if they just give a number ask them to do it again but this time so that you can understand how they came up with the number; after they finish the think-aloud exercise, explain why we are asking them to do this, e.g., if the Census was interested in the number of windows, which we are not, but if we were, and someone was looking at their sliding glass door and wondered aloud that they didn’t know if they should include it or not, them saying those words would clue us in that we needed to fix our question, make it clear that sliding glass doors should be included, etc.

Great that’s what I want you to do throughout our session. I will remind you to think aloud if you get quiet by saying something like “keep talking.”

At the end of the session, I will have some additional questions about how satisfied you were with the survey and we will go back and look at some of the survey screens and talk about them.

Do you have any questions for me before we begin?

Interview – Starting with Invitation letter

[Hand participant the mailing materials (invitation letter with their user ID).]

This is an example of mailing materials you would receive at your home if you were selected for the Census Attitudes Survey. Please take a moment to read it.

{PAUSE until they finish reading it}

Please go ahead and do what the letter says to get started. Answer any survey questions as they apply to you.

Remember that I am interested in what you have to say as you go about interacting with the web instrument. Now you can get started.

[IF SPANISH INTERVIEW, MAKE SURE THEY COMPLETE THE SURVEY IN SPANISH. LET THEM FIND THE LANGUAGE TOGGLE ON THEIR OWN. IF THEY DON’T AND BEGIN FILLING OUT THE QUESTIONS IN ENGLISH, STOP THEM AND TELL THEM THE SURVEY IS AVAILABLE IN SPANISH AND ASK IF THEY CAN FIND A WAY TO CHANGE THE LANGUAGE TO SPANISH. IF THEY CAN’T, CHANGE THE LANGUAGE FOR THEM.]

Probes that will be used in study

• Keep talking

• Um-hum?

• What are you thinking?

• What are you including?

• What would you do if I wasn’t here

[AT SOME POINT BEFORE THE END OF THE SURVEY STOP THE PARTICIPANT TO COMPLETE A LOGOUT/LOGIN TASK.]

Okay, at this point I would like you to pretend that you have to stop in the middle of the survey, for instance to run an errand, and then come back to complete it later. What would you do in this situation?

[IF THEIR RESPONSE DID NOT INVOLVE LOGGING OUT OF THE SURVEY, INSTRUCT THEM TO DO SO.]

Great, now let’s pretend you have returned to complete the survey. Please log back in to continue.

[IF THEY DID NOT WRITE DOWN PIN, PROVIDE IT FOR THEM SO THAT THEY CAN LOG BACK IN]

Satisfaction Questions – Paper form

Now I would like you to answer these questions about the web instrument.

[Hand the satisfaction questionnaire to the participant.]

General Debriefing Questions – Oral

Now I have a few questions about the survey.

• In your own words, tell me the purpose of this survey.

• As you completed the survey, did you ever find yourself wanting more information about a question or about the purpose of the survey?

• Did you run into any difficulties as you completed the survey?

• [If true:] You may have noticed that there was no button to use to go to a previous screen. What did you think about that?

• You may have noticed that there was a PSB logo on the screen. What did you think about that?

Thank you.

General Debriefing Questions – Oral with mailing materials

[Hand the mailing materials to the participant.]

Now I have a few questions about the survey.

Do you have any comment on these materials?

If this came to your home, what would your initial reaction be to receiving this in the mail?

Are there any indications to you that these materials are actually from the Census or does anything indicate that it could be from something other than a legitimate source? What are those things/indicators? Would you verify that the letter was legitimate before completing the survey? (If yes) How would you do that?

Additional Debriefing Questions – Oral with debriefing slides (if time permits)

[Open up screenshots of survey on laptop.]

We are going to go over [each/some] some of the screens you saw.

[Interviewer instruction on each screen: Do you have any comments about this screen? [IF P HAD PROBLEMS] Tell me more about your experience with this screen. Administer probes in the Powerpoint presentation containing screenshots.]

Is there anything else you’d like to mention that we haven’t talked about?

Finishing up session – Oral and Incentive letter

[Give the respondent incentive envelope [$40], and thank them for their time and effort. Collect materials and leave or escort respondent out of the interviewing room.]

Okay, that’s all that I have today. Many thanks for your work on this project. The last thing I need you to do is sign this form that says I am giving you $40.

3