Protocol

Formative Testing Protocol

MATERIALS NEEDED FOR INTERVIEW

• Interviewer Protocol Script

• Consent Form

• Electronic pre-questionnaire

• Participant tasks and randomization order

• Electronic SUS

• Laptop

• Incentive

• Payment Receipt

• Pens and Paper

Usability Testing Script

Thank you for your time today. My name is XX and I work with the Human Factors and Usability Group at the U.S. Census Bureau. I will be working with you today. In this lab, we evaluate how easy or difficult it is to use our online website and web surveys. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes live to a much larger group.

[Participant was emailed consent form]. Prior to the session, we emailed you a consent form, which explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to take a recording as you work with our online tool. We do this to get an accurate record of your feedback. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session. Do you give your consent?

Thank you.

OK great.

Today you will be helping us evaluate the design of a website that we are working on developing. I will give you a set of tasks and you will use the website to try to find information to help you answer the tasks. If you run into any difficulties as you work on the site, please don’t blame yourself. Any difficulties are the result of the design of the site, not your skills or abilities. We are going to use your comments and experiences as well as the comments and experiences of other participants to help improve the Website. I did not create the site, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make the website work well for everyone.

Think Aloud: I want you to answer the task questions exactly the way you would if you were at your home, but with one major difference. I would like you to think aloud as you answer the questions. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each task.

Practice Think Aloud

Let’s do a practice task before we start: Please think aloud as you answer the question, how many windows are in your home?

[Probe if they fall silent; if they just give a number ask them to do it again but this time so that you can understand how they came up with the number; after they do the think aloud explain why we are asking them to do this, e.g., if the Census was interested in the number of windows, which we are not, but if we were, and someone was looking at their sliding glass door and wondered aloud that they didn’t know if they should include it or not, them saying those words would clue us in that we needed to fix our question, make it clear that sliding glass doors should be included, etc. Probes should be kept to a minimum – trying to not interfere with the timing data, so used sparingly: Keep talking…; Um-hum?; What are you thinking?; Tell me more…

Great that’s what I want you to do throughout our session. I will remind you to think aloud if you get quiet.

At the end of the session I will have some questions for you about your experience as you worked on the tasks and how satisfied you were with the website. We may open up some of the screens and talk about them.

Interview – Starting with First Task

[Open demographics and mobile experience questionnaires.]

Before we start, I have two questionnaires that I’d like to go through with you. They collect information about you and how often you use different electronic devices and your demographics. I will read the questions and you can respond orally and I will record your answers on the computer. Do you have any questions?

[Read the background and demographic questionnaire and the mobile experience questionnaire and record the answers]

[When participant has completed the questionnaires say the following:]

We also emailed you a list of tasks that we’d like you to complete. Please open that document on your computer. [The TA can also text the task questions using the SfB chat feature].

Please share your screen by clicking the computer icon on the bottom of the screen, and selecting ‘present desktop’. We will begin each task by having you read the task question out loud. As you work, remember to talk to me about what you are thinking and feeling. Click on the page where you would click to find the information you are looking for. Once you have found the information you are looking for please state your answer aloud. For example, say, “My answer is ---” or “This is my final answer.” After each task, I will return you to the page where I would like you to begin the next task.

I want to let you know that we only have an hour for this session so I may need to move you on to the next task depending on what we have learned and what we still need to learn about the application. Also, the application is still in development and we may come across some examples where it isn’t functioning correctly and that is okay we will just do the best we can with where the application is at.

[When all tasks are finished open the System Usability Scale (SUS)]

Now I’d like you to ask you some questions about your experience using this tool.

[Ask SUS questions and record answers. Then ask Debriefing questions]

Timing on individual tasks

After 5 minutes – Ask Participant:

Do you feel like you are closer or further from the answer?

IF they offer that they are far from their answer you can move them on: Say

Due to time we will move on to the next task.

If they say they are close – allow them to try for two additional minutes.

After 7 minutes – if Participant is not close to an answer say–

Thank you for your work on this task. Due to time constraints, I would like to move us onto the next task.

After 7 minutes if participant is basically ready to give the answer say:

Okay due to time I need to get your answer to the task and move onto the next task.

Debriefing Probes

[If participant marked any SUS responses either on the low or high end, probe:] Tell me what you were thinking about when you marked this question a XXX.?

Overall, what was your impression of the online tool?

Debriefing on Individual Web components/screens

[Open the tool….]

Do you have any comments on this screen?

the main screen:

the list of results screen:

a results table screen:

a map screen:

[If there were issues with it:] Tell me more about your experience with this screen.

Is there anything else you’d like to mention that we haven’t talked about?

[Point out to the participant the Dataset area of the tool]

Determine what you would find in each dataset (open filter and go down list, ask if results were expected.) What types of data do you think would be collected in the county business patterns?

(Potentially this is an expert user task)

[Point out to the participant the Survey / Program area of the tool]

Determine what you would find in each Survey/Program (open filter and go down list, ask if results were expected.)

Finishing up session

[IF NOT A FEDERAL EMPLOYEE and the respondent emailed a signed voucher]

[Tell the respondent that you will send the cash via certified mail or USPS priority since you have received their signed voucher via email and thank them for their time and effort.]

[IF NOT A FEDERAL EMPLOYEE and signing online using online Qualtrics tool]. The last thing I need you to do is sign this form that says I am giving you $40 as a thank you for your participation.

I am also going to email you a packet that walks through each of the tasks in case you are interested. In the future, would you be willing to answer a few questions about your thoughts on these materials via email? Okay, that’s all that I have today. Many thanks for your work on this project.