Core UX Protocol

Usability Testing Protocol - English

MATERIALS NEEDED FOR INTERVIEW

• [Interviewer Protocol Script]

• [Laptop and Snagit recording software]

• [URL link to CPS survey]

• [Satisfaction & Debriefing questions – in Qualtrics link]

• [XLS spreadsheet for note taking]

• [MS Teams]

URLS (mobile/PC):

Qualtrics URL:

INTERVIEWER USABILITY TESTING SCRIPT

Thank you for your time today. My name is XX and I work at the U.S. Census Bureau. I will be working with you today.

The main part of this session is for you to answer some questions about a survey that we are working on at the Census Bureau. I want you to respond to the questions naturally, but with one major difference; and that is to have you think aloud as you do it. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question. So, anything that comes to your mind about the screens, the designs or the questions, the words on the screen- share those thoughts with me. If you fall silent, I may remind you to think aloud by saying “keep talking” or “what are you thinking?”

Practice Think Aloud

Let’s do a practice question before we start: Please think aloud as you answer the question, “How many windows are in your home?”

[Probe if they fall silent; if they just give a number ask them to do it again but this time so that you can understand how they came up with the number. After they finish the think-aloud exercise, explain why we are asking them to do this, e.g., if the Census was interested in the number of windows, and someone was looking at their sliding glass door and wondered aloud that they didn’t know if they should include it or not, them saying those words would clue us in that we needed to fix this question, make it clear that sliding glass doors should be included, etc.]

Probes to use in study

• Keep talking

• Um-hum?

• What are you thinking?

Great. That is what I want you to do throughout our session. I will remind you to think aloud if you get quiet.

Okay, so here’s what we will work on today. Your first task will be to answer survey questions as they apply to your real life. Then, I will have you answer a brief satisfaction questionnaire about the screens you saw. Lastly, I will ask you some final questions about your experience and will show you some online screens to get your feedback, and that should be it. Do you have any questions for me?

Recording

[Participant was emailed consent form]. We emailed you information about the session today and your rights as a participant. As a reminder, we are recording these sessions. I just need to capture your consent orally. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session. Do you give your consent?

OK, great. So now I am going to turn on the recording device and capture your oral consent on the recording:

[Turn on Camtasia or Snagit] Please state, “I give consent to be audio recorded and my screen to be recorded.”

Screen Sharing

So, just like we practiced in the tech check – I am going to need you to go ahead and share your screen with us. Make sure you have closed any windows you do not want us to record, as the recording is turned on.

Now - do you see the share icon? It might look like a square with an arrow pointing up. Go ahead and click on that.

[If able to see screen] Okay great I can see your screen. Now we can get started with the task.

[If unable to see screen will need to troubleshoot it]

Task 1: Starting on the rostering for their personal living situation

[TA: Start timing for the task once rostering page has loaded. Note all actions and comments users make during this task]

Okay, so we would like you to answer these questions as you would if you had gone online and were answering it in your home.

[TA: paste URL into chat window]

I’ll have you go ahead and click on this link that I am pasting to you in the chat window.

[If necessary:] Go ahead and answer the questions as the materials suggest you should do.

[If necessary:] Please remember to talk me through what you are thinking.

[TA: Note where user pauses and if they seem to stumble over any instructions or any of the questions.]

[TA: After they click the final next button to the end of the survey, open the satisfaction questions and say]:

Okay I am going to open up some questions for you to gather your impressions of the screens you just worked on. You can continue to think aloud for me.

[TA Paste the Qualtrics satisfaction questionnaire into teams chat window; Note Taker: continue to note any feedback or comments they speak aloud. They may say something that we will refer back to during debriefing section- if so, make a note of that]

Debriefing Probes:

[TA: If using Qualtrics for any images to show user - paste Qualtrics link into chat window at appropriate time during session.]

[PROBES WILL BE SELECTED FROM THE COGNITIVE INTERVIEW PROTOCOL BASED ON THE RESULTS OF COGNITIVE TESTING]

Were there any other parts of the survey you did that were confusing or difficult?

Do you have any additional thoughts on the screens we looked at today that you would like to share?

Is there anything else you’d like to mention that we haven’t talked about?

Let me turn to the team and see if they have any questions…

Finishing up session

Okay, that’s it for today. Now I just need you to fill out this screen with your address and where you would like the cash mailed.

[If necessary] Is this address where would you like the cash mailed? [ if no, enter correct address.]

3