Protocol

Vacant Crowdsourcing Instrument Study

MATERIALS NEEDED FOR INTERVIEW

• Interviewer Protocol

• Consent Form

• Mailing Materials

• Laptop with the Confidence-in-accuracy and Satisfaction Survey

• Demographic Questionnaire and Mobile Use Questionnaire

• Task sheet

• Incentive

• Payment Receipt

• Pens and Paper

• Tobii x260

• Laptop

• Audio Recorder

• iPhone

• Extra Equipment and Charging Cords

INTERVIEWER USABILITY TESTING SCRIPT

Hello, my name is XX and I work for the U.S. Census Bureau. Thank you for agreeing to help us today. Let me start by telling you a little about what we will be doing today. In this lab, we evaluate how easy or difficult our various products are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes live to a much larger group.

Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to take a video recording of this session along with audio to get an accurate record of your feedback. Your name will not be associated with the recording or any of the other data collected during the session.

[Hand VCIS consent form; give time to read and sign; sign own name and date. Show participants where the camera is and what part of the screen will be recorded. Turn on audio recorder and start video recording.]

Okay great.

Thank you.

There are several parts to today’s session.

The first task is to complete this questionnaire. It collects background information about you. Please fill out this information.

[Have participant fill out the usability background materials: demographic questionnaire on the laptop —simply move away from where they are answering so they don’t feel like we are looking over their shoulder.]

Thank you. The next part of today’s session is for you to complete this Mobile Devise Use Questionnaire. It collects information on your internet usage. Please fill out this information.

[Have participant fill out the Mobile Devise Use Questionnaire]

The next part of today’s session is for you to go through pieces of mail that might be mailed to you in 2020. While you are looking at the materials I give you, I’d like you to tell me what’s going through your mind. This could include what information you’re seeing in each material, questions that come to your mind, and reactions you’re having. I’m going to ask you to treat each one as you would if it came to your home.

Practice Think Aloud

Let’s do a practice question before we start. Remember to think aloud as you answer

How many windows are there in the place where you live?

_____________ WINDOWS

If needed: Try to visualize the place where you live, and think about how many windows there are in that place. As you count up the windows, tell me what you are seeing and thinking about.

PROBES:

• How did you come up with that answer?

• Tell me more about that. Why did you say [ANSWER]?

• I noticed that you hesitated. Tell me what you were thinking

Great that’s what I want you to do throughout our session. I will remind you to think aloud if you get quiet. Do you have any questions about the process?

Cognitive Testing

Okay, let’s get started.

Letter 1

Imagine you received this letter at your house. [Give respondent either original or alternate version of the letter.] Please take a moment to read it, remembering to think aloud. You can also let me know if anything stands out to you in the letter.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does P read letter; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

2. Does P comment on the part about gathering information on vacant homes?

___ Yes ___ No ___ Not sure

3. Does P comment on the part about saving taxpayer money?

___ Yes ___ No ___ Not sure

4. Does P comment on how to respond to the survey?

___ Yes ___ No ___ Not sure

5. Does P comment on the confidentiality?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this mailing:

Other notes:

Probes

1. What do you think the main point of this mailing is?

2. What do you think you would have to do for the study?

3. Based on reading the letter, how long do you think the study will take to complete?

4. Is there any part of the letter that is confusing or unclear?

5. What would you have done next after reading it? [If needed] Tell me more about that.

6. [If necessary] After getting this letter, would you participate in the study? Why/why not?

7. [If necessary] How would you respond to the survey?

8. If you couldn’t respond by Internet, what would you do?

9. What does the word “vacant” mean to you in this sentence? [POINT TO THE SENTENCE “For this brief study, we are gathering information about vacant homes near where you live.”]

10. What does the phrase “near where you live” mean to you in this sentence? [POINT TO THE SENTENCE “For this brief study, we are gathering information about vacant homes near where you live.”]

11. In your own words, what is the last sentence in the first paragraph saying? [POINT TO THE SENTENCE “Your participation could help save taxpayer money…]

12. How would you identify these vacant homes nearby? [If needed] Do you think you would have to walk around near you home?

13. Now I would like to present to you another version of this letter. [Present original or alternate version] Please take a few moments to look at the call-out box. [After participant indicates they looked at the box] Which of these two versions do you prefer? [If needed] Can you tell me more about that?

[Take the letter back]

Postcard 2

Thank you. About a few days after you get the first letter, you would receive this postcard. [Give Postcard #2] Please take a moment to read it, remembering to think aloud.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does P read postcard; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

2. Does P comment on the part about gathering information on vacant homes?

___ Yes ___ No ___ Not sure

3. Does P comment on the part about saving taxpayer money?

___ Yes ___ No ___ Not sure

4. Does P comment on how to respond to the survey?

___ Yes ___ No ___ Not sure

5. Does P comment on the confidentiality?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this postcard:

Other notes:

Probes

1. What do you think the main point of this mailing is?

2. Is there any part of the postcard that is confusing or unclear?

3. What would you have done next after reading it? [If needed] Tell me more about that.

4. In your own words, what is the last sentence in the first paragraph saying? [POINT TO THE SENTENCE “By answering a few simple questions about the vacant homes near where you live, future censuses could be less costly for taxpayers.”]

[Take the postcard back]

Letter 3

If you had not completed the study, about two weeks after receiving the post card, you would receive this reminder letter. [Give Letter #3] Please take a moment to read it, remembering to think aloud.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does P read letter; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

2. Does P comment on the part about gathering information on vacant homes?

___ Yes ___ No ___ Not sure

3. Does P comment on the part about saving taxpayer money?

___ Yes ___ No ___ Not sure

4. Does P comment on how to respond to the survey?

___ Yes ___ No ___ Not sure

5. Does P comment on the confidentiality?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this postcard:

Other notes:

Probes

1. What do you think the main point of this mailing is?

2. What would you have done next after reading it? [If needed] Tell me more about that.

[Take the letter back]

Postcard 4

If you had not completed the study, about two weeks after receiving the second letter, you would receive another post card [Give Postcard #4.] Please take a moment to read it, remembering to think aloud.

OBSERVE (DO NOT READ TO RESPONDENT):

1. Does P read postcard; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

2. Does P comment on the part about gathering information on vacant homes?

___ Yes ___ No ___ Not sure

3. Does P comment on the part about saving taxpayer money?

___ Yes ___ No ___ Not sure

4. Does P comment on how to respond to the survey?

___ Yes ___ No ___ Not sure

5. Does P comment on the confidentiality?

___ Yes ___ No ___ Not sure

Note any questions R asked/comments R made about this mailing:

Other notes:

Probes

1. What do you think the main point of this mailing is?

2. What would you have done next after reading it? [If needed] Tell me more about that.

[Take the postcard back]

Debriefing Questions (Cognitive Testing)

1. Did you find any of the letters or postcards we looked at today confusing or unclear?

2. Did you have any other thoughts about these mailing materials?

Usability Testing

For the next part of today’s session, you will be helping us evaluate the design of different online maps we are working on developing. If you run into any difficulties as you work on the application, please don’t blame yourself. Any difficulties are the result of the design of the application, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants to help improve the application. I did not create the application, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make the application work well for everyone.

Eyetracking

We are also going to record where you look on the screen as we work on our study. Now we will do a simple task that will allow the computer to find your eyes. To calibrate your eyes, please follow the dot with your eyes.

[Calibrate the participants’ eyes]

Tasks

[Set the task questions and questionnaire by participant.]

[Open mobile devise use questionnaire on the participant’s computer. Go into control room.]

We will begin each task by having you read the task question out loud. As you work, remember to talk to me about what you are thinking and feeling. Once you feel you have completed the task, please let us know. After each task, I will ask you to complete two questionnaires before moving on to the next task.

[Administer tasks]

[After participant completes each task]

Now I’d like you to answer two questions.

[Administer the certainty-in-accuracy question and the satisfaction question]

[After participant answers both questions]

Now I’d like you to do the next task.

[Repeat until participant completes all four tasks and post-task questions.]

[Go back into participant’s testing room. Then ask debriefing questions.]

Preferred Map Prototype

Now I will show you different ways of displaying addresses. Each of these images uses an example address. Please imagine seeing your home and your neighbors’ homes on this type of display.

[Show participant each of the 6 map displays on mobile device or laptop.]

Please rank these different types of displays in the order of your preference [for locating your home and your neighbors’ home], with the display you like the most ranked number 1, and the type you like the least ranked number 6. I will write down the order.

[Record order on Preference questionnaire]

[After participant completes question]

1. Can you tell me more what you were thinking when you answered this question?

Debriefing Questions

Now I have some other questions:

1. When was the last time you used a map on your computer or smartphone?

   1. Within the last week, including today

   2. More than a week ago, but less than a month ago

   3. In the last 1-3 months

   4. More than 3 months ago

   5. I don’t use maps on a computer or smartphone [SKIP QUESTIONS 2, 3, and 4]

2. What did you use the map for (e.g. driving directions)?

3. How comfortable are you using maps on your computer or smartphone?

   1. Extremely comfortable

   2. Very comfortable

   3. Somewhat comfortable

   4. A little comfortable

   5. Not at all comfortable

4. In the last few months, have you used maps mostly on your smartphone, mostly on your computer, or equally on a smartphone and computer?

   1. Mostly on my smartphone

   2. Mostly on my computer

   3. Equally on a smartphone and computer

   4. I only own a smartphone

   5. I only own a computer

Now I would like to show you the map that you saw earlier:

[Open up Map XX]

1. Earlier, you had indicated that it was [fill in based on participant’s answer to task difficulty question] to find your home using this map. Can you tell me more about that?

2. [IF AT LEAST ONE UNIT SELECTED WAS VACANT] Earlier you were asked to identify the vacant units in your neighborhood, and you answered that it was [fill in based on participant’s answer to task difficulty question] to find these units using this map. Can you tell me more about that?

3. [ALL] What does the word “vacant” mean to you?

4. [IF AT LEAST ONE UNIT SELECTED WAS VACANT] How did you know that the residential unit(s) you selected in your neighborhood were vacant? How did you know which residential units were not vacant? We noticed that you were not that sure, etc. can you tell me more about that. Note: The participant may have answered these already in question 2.

5. [IF NO UNITS SELECTED AS VACANT] How did you determine that none of the residential units in your neighborhood were vacant? How would you know if a housing unit was vacant?

6. For these tasks, how easy or difficult was it for you to zoom in and zoom out to find the residential units of interest? If needed: Can you tell me more about that.

Additional Debriefing Questions

Is there anything else you’d like to mention that we haven’t talked about?

Finishing up session

[Give the respondent incentive envelope [$40], and thank them for their time and effort. Collect materials and leave or escort respondent out of the interviewing room.]

Okay, that’s all that I have today. Many thanks for your work on this project. The last thing I need you to do is sign this form that says I am giving you $40.

13