Round 2 UX Protocol

Usability Testing Round 2 Protocol - English

MATERIALS NEEDED FOR INTERVIEW

• [Interviewer Protocol Script]

• [Laptop and Snagit recording software]

• [URL link to CPS survey]

• [Satisfaction & Debriefing questions – in Qualtrics link]

• [XLS spreadsheet for note taking]

• [MS Teams]

URLS (mobile/PC):

Qualtrics URL:

INTERVIEWER USABILITY TESTING SCRIPT

My name is [INTERVIEWER NAME] and I will be conducting our research session today. I work for [CONTRACTOR], a survey research company, and I am talking to you on behalf of the U.S. Census Bureau and the Bureau of Labor Statistics. <I also have my colleague [COLLEAGUE/OBSERVER NAME] who will be [taking notes for us today/observing our session].>

Note this is the second of the two research sessions you will participate with us.

The purpose of today’s session is to help us see how well our questions work in an online survey. We’re interested in how people understand these questions and how you come up with your answers. We are not evaluating you, so there are no right or wrong answers. We are looking to see what works and what we can improve.

You may run into difficulties as you work on the survey. Any difficulties are the result of the design of the web survey, not your skills or abilities. We are going to use your comments and experiences as well as those of other participants to improve the survey. I did not create the online survey, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make the online survey work well for everyone.

Before we start, I want to talk about the consent form I shared with you via email. Did you have a chance to review and sign this form? [IF NO: Can you please open the link to this form in your email?] I just want to review some of the important information that was included in that consent document:

• With your permission, I would like to video- and audio-record our conversation. This will allow me to concentrate on what you are saying instead of taking a lot of notes while you are talking. It will also help me write a summary of the interview.

• All your responses and everything you say will be kept strictly confidential, and only researchers working on this project will see your answers or hear the recording. Your name will not be used in any of our reports. Nothing you tell us today will be shared with other federal agencies or with anyone else.

• Your participation is voluntary, and you may stop at any time. You may choose not to answer any questions you don’t want to answer.

• This study is the second interview. This interview will take about 1 hour.

• The form also has the contact name and telephone number of the project director, should you have any additional questions about the study. It also contains information about how to get in touch with the Institutional Review Board representative if you have any questions about your rights as a research participant in this study.

• As a thank you for your time today, you will receive $60.

[IF OBSERVER IS PRESENT: I also wanted to let you know that this interview is being observed by someone from the Census Bureau/someone from the Bureau of Labor Statistics/someone else who works on this study. They are observing the interview because they are also interested in learning how our surveys work, and whether we need to change something that does not work.]

Recording

I just need to capture your consent orally. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session. Do you give your consent?

OK, great. So now I am going to turn on the recording device and capture your oral consent on the recording:

[Turn on Snagit or Logitech] Do you consent to participate in the interview today, and for the interview to be recorded and my screen to be recorded?

Practice Think Aloud

The main part of this session is for you to answer some questions about a survey that we are working on at the Census Bureau. I want you to respond to the questions naturally, but with one major difference; and that is to have you think aloud as you do it. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question. So, anything that comes to your mind about the screens, the designs or the questions, the words on the screen- share those thoughts with me. If you fall silent, I may remind you to think aloud by saying “keep talking” or “what are you thinking?”

Let’s do a practice question before we start: Please think aloud as you answer the question, “How many windows are in your home?”

[Probe if they fall silent; if they just give a number ask them to do it again but this time so that you can understand how they came up with the number. After they finish the think-aloud exercise, explain why we are asking them to do this, e.g., if the Census was interested in the number of windows, and someone was looking at their sliding glass door and wondered aloud that they didn’t know if they should include it or not, them saying those words would clue us in that we needed to fix this question, make it clear that sliding glass doors should be included, etc.]

Probes to use in study

• Keep talking

• Um-hum?

• What are you thinking?

Great. That is what I want you to do throughout our session. I will remind you to think aloud if you get quiet.

Okay, so here’s what we will work on today. Your first task will be to answer survey questions as they apply to your real life. Then, I will have you answer a brief satisfaction questionnaire about the screens you saw. Lastly, I will ask you some final questions about your experience and will show you some online screens to get your feedback, and that should be it. Do you have any questions for me?

Screen Sharing

I am going to need you to go ahead and share your screen with us. Make sure you have closed any windows you do not want us to record, as the recording is turned on.

[If they are computer] If you don’t mind, would you tell us what type of computer you are using today? [If needed] Is this Microsoft, Apple, etc.?

[If they are mobile] If you don’t mind, would you tell us what type of phone you are using today?

Now - do you see the share icon?

Desktop/Laptop Case: It might look like a square with an arrow pointing up. Go ahead and click on that.

Mobile case:

• In the Teams App click the button with the three dots at the bottom next to the red hang-up button. At the top row you will see “Share” with a square and an arrow pointing up. You can click that button. Then select “Share screen”. Make sure “Audio” is toggled on.

• There will be a “Screen Broadcast” box the opens and you can select “Start Broadcast”

• It will count down and it will be sharing when the red dot is blinking. Now to go to the link you just opened you can click anywhere outside of the Screen Broadcast box.

[If able to see screen] Okay great I can see your screen. Now we can get started with the task.

[If unable to see screen will need to troubleshoot it]

Task 1: CPS Core Interview

Desktop/Laptop cases: Explain to the respondent that typically they’d receive a letter to ask them to fill out the instrument. Share letter for login information. Provide User ID separately from your list.

Mobile cases: Provide the link to respondent and User ID in the chat.

[TA: paste URL into chat window]

[If necessary:] Go ahead and answer the questions as the materials suggest you should do.

[If necessary:] Please remember to talk me through what you are thinking.

[TA: Start timing for log in page once that is loaded. Note all actions and comments users make during the log in. End timer once they have logged into the core instrument]

Note, please have the first email address on hand in case they struggle logging in.

[TA: Start timing for the task once rostering page has loaded. Note all actions and comments users make during this task]

Okay, so we would like you to answer these questions as you would if you had gone online and were answering it in your home.

Note to interviewers, there will be a pretend address in the instrument. Have participants say “Yes” to these address verification questions.

[TA: Note where user pauses and if they seem to stumble over any instructions or any of the questions.]

Question/Page	Observations
Anyone mentions using multiple modes to log in the survey
Write down on what’s happening on the roster screen (adding, removing, updating after they have added)
(If seen) Date of birth questions
(If seen) Write down their reactions to questions with many/long response options (e.g., race, relationship, education, looking for work)
Question about “looking for work” (please mention if participant sees screen and note observations)
Questions related to proxy household members
Any questions that use he/she pronouns rather than names of household members
Any questions that has response options in the question wording
Nativity Drop Down (United States right now is at the bottom)

Satisfaction Questionnaire:

[TA: After they click the final next button to the end of the survey, say]:

Okay I am going to send you a link to answer some question on your experience with the survey you just completed. You can continue to think aloud for me.

[TA Paste the Qualtrics satisfaction questionnaire into teams chat window; Note Taker: continue to note any feedback or comments they speak aloud. They may say something that we will refer back to during debriefing section- if so, make a note of that]

Debriefing Probes:

[TA: If using Qualtrics for any images to show user - paste Qualtrics link into chat window at appropriate time during session.]

Thank you very much for answering those questions.

Now I have some questions about the survey questions you just answered.

I’m going to now share a link with the screens you saw during the interview. Note, you don’t have to answer on screen as I am just asking about your thoughts. This will show fictional names in the instrument.

Overall survey

1. Tell me what you thought about answering this survey on your [computer/mobile phone].

2. [If needed] What did you like about completing this survey on your [computer/mobile phone]? What did you dislike about completing this survey on your [computer/mobile phone]?

3. [If needed] Did you find this survey easy or difficult to complete? Can you tell me more about that? What did you find easy? What did you find difficult?

Letter questions [if completed interview on computer]

4. Before you started the survey, you reviewed a letter that invites people like yourself to complete a survey. What did you think of this letter?

5. Was it clear what you had to do next after reading? [If needed] Can you please tell me more about that?

6. [If needed] What did you like about this letter? [If needed] What did you dislike about this letter?

Log in questions

7. Next, I’m going to talk about the log in process. If I were not here, do you think you would have continued to use your [phone/computer] to log in or would you have used multiple devices?

1. [If needed] As you noticed, you had to enter your email address, then pull up a passcode from your email, and then enter a User ID. What did you think about this?

[If needed] How do these steps compare to other web sites or applications you have logged into?

8. Next, I’m going to show you some screens that you saw before you started the survey. You will see these screens one by one.

Note, we will show one screen at a time and these questions will repeat for each screen. Interviewer will show image 1, ask all of the below questions, then image 2, repeat questions, etc.

1. What did you think about this screen?

2. Was it clear what you had to do for this screen?

3. [If needed] What did you find easy about this? Tell me more about that.

4. [If needed] What did you find difficult about this? Tell me more about that.

Image 1

Image 2

Image 3

Roster Questions

9. When you were answering the questions about who still lives in your household, can you tell me what you were thinking?

2. [If needed] Did you find these questions easy or difficult to answer? Can you tell me more about that?

3. [If needed, repeat for each household member change they did] I noticed that you had [added/edited/removed] a household member, [insert name]. Can you tell me more about that?

4. [if needed] Please probe based on what your observations were on the roster screen during the interviewer.

For this next part of the study, I am going to send you a link to the survey again. Please use this information to log in.

[After participant logs in]

Vignette:

(Note to OMB these tasks may change in wording depending on what the instrument is but this is an idea of what we’re trying to measure. The Image below is a draft with test data.)

Task 1: I want you to add someone in your household named Joey Doe. Joey Doe is White, Non-Hispanic, 15 years, and is not working for pay or profit. Please show me what you would do and think aloud for me.

Task 2: Jane Doe recently moved out of the household to go to college so they no longer live in the household. Please show me what you would do and think aloud for me.

Task 3: In the last survey, you had indicated that Mary Doe’s birthday was November 18^th, 2011. However, you now realize you had put the wrong birth date and Mary’s birthday is November 18^th, 2012. Please tell me what you would do so the information in the survey is correct.

Demographic Questions

10. a. [If added a household member] For [insert name], you answered their date of birth/age, what did you think about this process? [if mobile] What did you think of the spin wheels used to select month, day, and year?

Probe on one of relationship and education (question with a long list). If they appeared to struggle with one of these two items, probe on that specific item. If there was no struggle, choose any one to probe on.

2. [If added a household member] You answered questions about [name’s] [relationship/education]. Can you tell me more what you were thinking when you were answering these questions? Did you think you were able to find an answer for [insert name]?

(Note to OMB: We may revise the instrument depending on cognitive testing results and as the web instrument is developed.)

Work questions

11. [If needed] In your own words, tell me about your [employment/job searching/retirement].

12. [If needed] In your own words, tell me as much as you can about [insert name] [employment/job searching/retirement]. (Note to interviewer, repeat for each household member of working age)

13. You may have noticed that some of the questions included prior answers from the first survey you had completed. What did you think about this?

1. [If needed] What did you find helpful about this?

2. Was there anything you did not find helpful about this?

1. Probe if they struggle answering [employment/job searching/retirement] questions for at least one of the household members.

(Note to OMB: We will likely add questions in this section based on cognitive testing results and as the web instrument is developed. The following examples will give you an example of the dependent data used and the types of information we are trying to collect from participants in the second interview.)

Dependent Data:

[Show the following screens if someone in the household had reported employment last time]

For the next part, I’m going to share some examples of questions you had seen. These screens contained information you had provided during your first interview.

14. Did you have any initial thoughts when you first saw this screen for [yourself/household member]?

1. Was it helpful or unhelpful to have the name have who [you/household member] worked for?

15. Repeat as necessary.

Tell me more about what you were thinking about when you saw this question for [yourself/household member’s name]?

5. Can you tell me a little bit about the activities and duties for [your/household member’s name] job now?

6. Can you tell me a little bit about the activities and duties for [your/household member’s name] job last month?

7. Do you think you were able to accurately report an answer for [yourself/household member’s name]?

Specific Questions

[PROBE ON ANY SPECIFIC QUESTIONS FROM CPS CORE INSTRUMENT]

I noticed for this question you answered [INSERT ANSWER]. Can you tell me more what you were thinking?

I noticed you hesitated when answering this question. Can you tell me more about that?

Debriefing

Were there any other parts of the survey you did that were confusing or difficult?

Do you have any additional thoughts on the screens we looked at today that you would like to share?

Is there anything else you’d like to mention that we haven’t talked about?

Let me turn to the team and see if they have any questions…

Finishing up session

TA: End Recording.

Okay, that’s it for today. Thanks so much for all your answers. Now I will just stop the recording and then I will confirm with you the email you would like us to send your digital gift cards.

10