Round 1 UX Protocol

Usability Testing Round 1 Protocol - English

MATERIALS NEEDED FOR INTERVIEW

• [Interviewer Protocol Script]

• [Laptop and Snagit recording software]

• [URL link to CPS survey]

• [Satisfaction & Debriefing questions – in Qualtrics link]

• [XLS spreadsheet for note taking]

• [MS Teams]

URLS (mobile/PC):

Qualtrics URL:

INTERVIEWER USABILITY TESTING SCRIPT

My name is [INTERVIEWER NAME] and I will be conducting our research session today. I work for [CONTRACTOR], a survey research company, and I am talking to you on behalf of the U.S. Census Bureau and the Bureau of Labor Statistics. <I also have my colleague [COLLEAGUE/OBSERVER NAME] who will be [taking notes for us today/observing our session].>

Note this is the first of the two research sessions you will participate with us.

The purpose of today’s session is to help us see how well our questions work in an online survey. We’re interested in how people understand these questions and how you come up with your answers. We are not evaluating you, so there are no right or wrong answers. We are looking to see what works and what we can improve.

You may run into difficulties as you work on the survey. Any difficulties are the result of the design of the web survey, not your skills or abilities. We are going to use your comments and experiences as well as those of other participants to improve the survey. I did not create the online survey, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make the online survey work well for everyone.

Before we start, I want to talk about the consent form I shared with you via email. Did you have a chance to review and sign this form? [IF NO: Can you please open the link to this form in your email?] I just want to review some of the important information that was included in that consent document:

• With your permission, I would like to video- and audio-record our conversation. This will allow me to concentrate on what you are saying instead of taking a lot of notes while you are talking. It will also help me write a summary of the interview.

• All your responses and everything you say will be kept strictly confidential, and only researchers working on this project will see your answers or hear the recording. Your name will not be used in any of our reports. Nothing you tell us today will be shared with other federal agencies or with anyone else.

• Your participation is voluntary, and you may stop at any time. You may choose not to answer any questions you don’t want to answer.

• This study will include two interviews, the one today and then a second one a few months from now. Each interview will take about 1 hour.

• The form also has the contact name and telephone number of the project director, should you have any additional questions about the study. It also contains information about how to get in touch with the Institutional Review Board representative if you have any questions about your rights as a research participant in this study.

• As a thank you for your time today, you will receive a $50 electronic gift card immediately after we complete the interview. When you complete the second interview in a few months, you will receive $60.

[IF OBSERVER IS PRESENT: I also wanted to let you know that this interview is being observed by someone from the Census Bureau/someone from the Bureau of Labor Statistics/someone else who works on this study. They are observing the interview because they are also interested in learning how our surveys work, and whether we need to change something that does not work.]

Recording

I just need to capture your consent orally. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session. Do you give your consent?

OK, great. So now I am going to turn on the recording device and capture your oral consent on the recording:

[Turn on Snagit or Logitech] Do you consent to participate in the interview today, and for the interview to be recorded and my screen to be recorded?

Practice Think Aloud

The main part of this session is for you to answer some questions about a survey that we are working on at the Census Bureau. I want you to respond to the questions naturally, but with one major difference; and that is to have you think aloud as you do it. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question. So, anything that comes to your mind about the screens, the designs or the questions, the words on the screen- share those thoughts with me. If you fall silent, I may remind you to think aloud by saying “keep talking” or “what are you thinking?”

Let’s do a practice question before we start: Please think aloud as you answer the question, “How many windows are in your home?”

[Probe if they fall silent; if they just give a number ask them to do it again but this time so that you can understand how they came up with the number. After they finish the think-aloud exercise, explain why we are asking them to do this, e.g., if the Census was interested in the number of windows, and someone was looking at their sliding glass door and wondered aloud that they didn’t know if they should include it or not, them saying those words would clue us in that we needed to fix this question, make it clear that sliding glass doors should be included, etc.]

Probes to use in study

• Keep talking

• Um-hum?

• What are you thinking?

Great. That is what I want you to do throughout our session. I will remind you to think aloud if you get quiet.

Okay, so here’s what we will work on today. Your first task will be to answer survey questions as they apply to your real life. Then, I will have you answer a brief satisfaction questionnaire about the screens you saw. Lastly, I will ask you some final questions about your experience and will show you some online screens to get your feedback, and that should be it. Do you have any questions for me?

Screen Sharing

I am going to need you to go ahead and share your screen with us. Make sure you have closed any windows you do not want us to record, as the recording is turned on.

[If they are computer] If you don’t mind, would you tell us what type of computer you are using today? [If needed] Is this Microsoft, Apple, etc.?

[If they are mobile] If you don’t mind, would you tell us what type of phone you are using today?

Now – do you see the share icon?

Desktop/Laptop Case: It might look like a square with an arrow pointing up. Go ahead and click on that.

Mobile case:

• In the Teams App click the button with the three dots at the bottom next to the red hang-up button. At the top row you will see “Share” with a square and an arrow pointing up. You can click that button. Then select “Share screen”. Make sure “Audio” is toggled on.

• There will be a “Screen Broadcast” box the opens and you can select “Start Broadcast”

• It will count down and it will be sharing when the red dot is blinking. Now to go to the link you just opened you can click anywhere outside of the Screen Broadcast box.

[If able to see screen] Okay great I can see your screen. Now we can get started with the task.

[If unable to see screen will need to troubleshoot it]

Task 1: CPS Core Interview

Desktop/Laptop cases: Explain to the respondent that typically they’d receive a letter to ask them to fill out the instrument. Share letter for login information. Provide User ID separately from your list.

Mobile cases: Provide the link to respondent and User ID in the chat.

[TA: paste URL into chat window]

[If necessary:] Go ahead and answer the questions as the materials suggest you should do.

[If necessary:] Please remember to talk me through what you are thinking

[TA: Start timing for log in page once that is loaded. Note all actions and comments users make during the log in. End timer once they have logged into the core instrument]

Note, please write down the email address they use so you have it for the second interview in case participant struggles logging in. The participant must provide their real email address for the purposes of this study.

[TA: Start timing for the task once rostering page has loaded. Note all actions and comments users make during this task]

Okay, so we would like you to answer these questions as you would if you had gone online and were answering it in your home.

[If necessary:] Please remember to talk me through what you are thinking

Note to interviewers, there will be a pretend address in the instrument. Have participants say “Yes” to these address verification questions.

[TA: Note where user pauses and if they seem to stumble over any instructions or any of the questions.]

Question/Page	Observations
Anyone mentions using multiple modes to log in the survey (e.g. Says something like “let me check my email on my phone really quick.”)
Write down on what’s happening on the roster screen (adding, removing, updating after they have added)
Date of birth questions
Write down their reactions to questions with many/long response options (e.g., race, relationship, education, looking for work)
Question about “looking for work” (please mention if participant sees screen and note observations)
Questions related to proxy household members
Any questions that use he/she pronouns rather than names of household members
Any questions that has response options in the question wording
Nativity Drop Down (United States right now is at the bottom)

Satisfaction Questionnaire:

[TA: After they click the final next button to the end of the survey, say]:

Okay I am going to send you a link to answer some question on your experience with the survey you just completed. You can continue to think aloud for me.

[TA Paste the Qualtrics satisfaction questionnaire into teams chat window; Note Taker: continue to note any feedback or comments they speak aloud. They may say something that we will refer back to during debriefing section- if so, make a note of that]

Debriefing Probes:

[TA: If using Qualtrics for any images to show user - paste Qualtrics link into chat window at appropriate time during session.]

Thank you very much for answering those questions.

Now I have some questions about the survey questions you just answered.

I’m going to now share you a link with the screens you saw during the interview. Note, you don’t have to answer on screen as I am just asking about your thoughts. This will show fictional names in the instrument.

Overall survey

1. Tell me what you thought about answering this survey on your [computer/mobile phone].

2. [If needed] What did you like about completing this survey on your [computer/mobile phone]? What did you dislike about completing this survey on your [computer/mobile phone]?

3. [If needed] Did you find this survey easy or difficult to complete? Can you tell me more about that? What did you find easy? What did you find difficult?

Letter questions [if completed interview on computer]

4. Before you started the survey, you reviewed a letter that invites people like yourself to complete a survey. What did you think of this letter?

5. Was it clear what you had to do next after reading? [If needed] Can you please tell me more about that?

6. [If needed] What did you like about this letter? [If needed] What did you dislike about this letter?

Log in questions

7. Next, I’m going to talk about the log in process. If I were not here, do you think you would have continued to use your [phone/computer] to log in or would you have used multiple devices?

   1. As you noticed, you had to enter your email address, then pull up a passcode from your email, and then enter a User ID. What did you think about this?

   2. How do these steps compare to other web sites or applications you have logged into?

8. Next, I’m going to show you some screens that you saw before you started the survey. You will see these screens one by one.

Note, we will show one screen at a time and these questions will repeat for each screen. Interviewer will show image 1, ask all of the below questions, then image 2, repeat questions, etc.

1. What did you think about this screen?

2. Was it clear what you had to do for this screen?

3. [If needed] What did you find easy about this? Tell me more about that.

4. [If needed] What did you find difficult about this? Tell me more about that.

Image 1

Image 2

Image 3

Image 4

Roster Questions

[SHOW ROSTER QUESTIONS]

Image 5

9. When you were filling in who lives in your household, can you tell me more what you were thinking as you were answering these questions?

1. What did you think of the process of adding the people to your household on the [computer/phone]? Was it clear how to add them? Can you tell me more about that? [If needed] Do you feel you were able to add all of the members in your household to this survey?

2. [if needed] Please probe based on what your observations were on the roster screen during the interview.

Demographic Questions

[Show date of birth question]

Image 6

10. [For everyone] For each household member, you answered their date of birth/age, what did you think about this process?

    1. [if mobile] What did you think of the spin wheels used to select month, day, and year?

Probe on one of relationship and education (question with a long list). If they appeared to struggle with one of these two items, probe on that specific item. If there was no struggle, choose any one to probe on.

11. [For all household members] You answered questions about each person’s [relationship/education]. Can you tell me more what you were thinking when you were answering these questions? Did you think you were able to find an answer for each member of the household?

Image 7

Image 8

Work questions

12. In your own words, tell me about your [employment/job searching/retirement].

13. In your own words, tell me as much as you can about [insert name of other household member] [employment/job searching/retirement]. (Note to interviewer, repeat for each household member of working age)

    1. You answered a series of questions about whether each person in your household worked for either pay or profit. Were you able to answer about each member of your household? Did you feel confident about your answers on each member of your household?

14. [If participant saw “looking for work” screen] I am showing you this question now that you answered for [yourself/NAME]. Can you tell me more about what you were thinking as you answered this?

    1. [If mobile] Do you think you were able to select the answer that best matched the household member’s situation?

Note to OMB, we will be populating this section based on results from cognitive testing that is currently in field and any further instrument development between now and when usability testing will begin.

Image 9

Specific Questions

[PROBE ON ANY SPECIFIC QUESTIONS FROM CPS CORE INSTRUMENT WHERE THE PARTICIPANT HESITATED OR HAD ISSUES FILLING IN INFORMATION.]

I noticed for this question you answered [INSERT ANSWER]. Can you tell me more what you were thinking?

I noticed you hesitated when answering this question. Can you tell me more about that?

Debriefing

Were there any other parts of the survey you did that were confusing or difficult?

Do you have any additional thoughts on the screens we looked at today that you would like to share?

Is there anything else you’d like to mention that we haven’t talked about?

[IF OBSERVERS: Let me turn to the team and see if they have any questions…]

Finishing up session

TA: End Recording.

Okay, that’s it for today. Thanks so much for all your answers. Now I will just stop the recording and then I will confirm with you the email you would like us to send your digital gift cards.

[If participant planned to be scheduled for Round 2] I would like to schedule a date and time for the next interview in [insert month]. Please confirm email and phone number for them and gain permission to text for future round.

[If Round 2 is full] If there is an opening for the next round, we will reach out to you potentially in [insert month]. Please confirm email and phone number for them and gain permission to text for future round.

9