D Protocol

Protocol: COMPASS NRFU Usability/Cognitive Evaluation

Thank you for your time today. My name is XX and I work here with the XXXX. I will be working with you today. We evaluate how easy or difficult Census products are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it.

Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to record the session to get an accurate record of your feedback. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session.

[Hand consent form; give time for participant to read and sign; sign own name and date, start recording.]

Thank you.

Ok great. I now have a demographic questionnaire and a questionnaire about mobile experience that I’d like you to fill out.

[Have the participant complete the Mobile experience Questionnaires on the computer.

Thank you.

I am going to give you a little background about what you will be working on today. As you may know the Census Bureau counts people in the U.S. Population. This is normally done with a survey that comes in the mail or an online survey. If the household does not fill out the survey, then the Census Bureau sends an interviewer to the home to ask the survey questions. These questions consist of things like “What are the names of people living in your household? How many people live in your household? What are their names? , etc. So today we need your help to evaluate the design of a mobile app that is going to be used by Census interviewers to collect the household information from a person when they are right at their house.

• Participant acting as Interviewer:

To do this we will have you use the mobile app to complete a household interview. For the interview, I will play the part of the person who answers the door and answers your questions and you will play the part of the census interviewer who uses the app on a smartphone (that I will give you in a moment) to ask the census questions.

If you run into any difficulties as we work on the app, please don’t blame yourself. Any difficulties are the result of the design of the app, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants to help improve the application. I did not create the app, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make sure that it works well for everyone.

Throughout the session I will have some questions for you about your experience using the app and at the end I would like to know about your overall satisfaction using the app.

As you work on the app I would like for you to do three things:

1. First ask the question as it is worded on the instrument. (I will give you a brief training on this)

2. Enter my responses (as I play the part of the person who living at the household). (Again I will give you a brief training on how to do this).

3. Tell me if the question doesn’t make sense or if you think a question should be re-worded and why.

As you go about doing these three things it may help you to think aloud. In other words, I would like you to tell me everything that you are thinking, doing, and feeling as you go about using the app to complete each interview.

Practice: Since thinking aloud may not be common while working with a smartphone, let’s do a practice before we start: Please think aloud as you answer the question, how many windows are in your home?

Great that’s what I want you to do throughout our session. I will remind you to think aloud if you get quiet.

Now it is time to do a brief training on how to conduct an interview.

So, you are going to play the role of census interviewer. Your job is to conduct census interviews. I will play the role of each person you interview. In real life, you would be going to a different household address to conduct each interview.

TRAINING (Only administered for participants acting as Interviewers)

Now, I will give you a brief training on how to conduct an interview with the NRFU questionnaire.

The text on the screen will appear in different formats to help you understand what you are to say and do. This reference sheet <show text treatment sheet> explains the different formats. You can always refer back to this sheet. The text formats are the following:

[Reference sheet will include the following information]

• Black text: The words you need to say to the person you are interviewing will always be in black, for example, “I’m here to complete a Census questionnaire….”

• Red italics: Text that refers to actions you need to carry out appears in red italics, for example, “Hand respondent Information Sheet and point to Confidentiality Notice….”

• Blue text: Text that appears in blue refers to instructions you need to follow, for example, “Start with the person who….”

• Gray italics: Text that appears in gray refers to areas of information that you need to provide, for example, “My name is <Your name>”

• White text: Text that appears in white refers to response options to the interview questions and other general information that you will use.

Use these different text formats to guide you when conducting the interview. It’s okay if you need to refer to the text treatment sheet later on.

The screen provides you with the option to continue or quit the interview. Please select Continue.

The top of the screen contains the navigation tools for the NRFU instrument. You can select the forward gray arrow to proceed to the next question or select the back arrow to revisit a previous question. Please select the forward arrow. You can also swipe to move forward and back.

The next screen asks you to identify the way in which you are attempting to contact the household at this address. All of the interviews that you conduct today will be personal visits. A personal visit involves visiting the household and interviewing a household member or a proxy respondent (i.e. a neighbor). Please select Personal visit  Forward arrow / or swipe.

The following screen asks you to identify the address you are attempting

Select Attempting Census Address.

And tap forward arrow or swipe forward.

The next screen is the result of your contact attempt after you have arrived at the address and knocked on the door. It provides an introduction to the respondent. For today’s interviews please use your name in the introduction. It also tells you in red italics to hand me the Information Sheet. You will be instructed later in the interview to remind me (the person living at the household) to refer back to the Information sheet.

A likely result of your contact attempt is, “Yes, correct address.”

I want to point out the gray up button at the bottom of the screen>>. Selecting the << gray up button>> makes additional options become visible or hides them.

Now Please select “Yes correct address” and tap gray forward arrow or swipe forward.

Ok, lets end here. So we’ve practiced going through some of the interview screens.

Do you have any questions ?

As you go through the NRFU questionnaire, I will provide responses to the questions while playing the role of the person you are interviewing. At other times, I will stop you and demonstrate how to use the app. If you have any questions, it’s fine to ask.

We will now begin the session. During these interviews, I will no long be the trainer. I only will play the role of the person you are interviewing. Therefore, I will not be available to answer questions about how to use the application. If time permits, you will conduct 4 interviews.

As I mentioned before, we are not evaluating you in any way. Rather, we are assessing the user-friendliness of this device, including the questionnaire.

I want to remind you that your job with me today is to

1. Ask the question as it is worded on the instrument

2. Enter my responses (as I play the part of the person living in the household), and

3. Tell me if the question doesn’t make sense or if you think a question should be re-worded and why.

Lastly, remember to think aloud as you do the interview so we can understand what the issues are.

[Select the appropriate interview script and begin evaluation interviews]

Now it is time to use the COMPASS application.

Let’s begin.

[Hand the participant the first Interview scenario and have them read aloud]

[After all interview scenarios are completed- administer SUS satisfaction questionnaire.

{Administer debriefing items. }

• Participant acting as respondent:

What I’ll ask you to do today is listen to the survey questions I ask and answer them just as you would if you were at your home and an interviewer knocked on the door.

After you complete the questionnaire, I will ask you to tell me a bit more about your answers to the Census questionnaire. Some of the things I will ask you may sound a little strange. For example, I might ask you what a certain word means to you or how you came up with your answer.

Again, the reason for this is to see how people understand each question in the questionnaire. Even though some of the things I will ask you may sound like I am giving you a test, there are no right or wrong answers. We are interested in how the questions in the questionnaire work for you.

Do you have any questions before we begin?

[ Administer the entire NRFU instrument without probing.

During the interview, note Respondent (R) questions or signs of difficulty.

After the interview, review R responses and administer scripted retrospective probes.]

Now it is time to use the COMPASS application.

Let’s begin.

{Administer debriefing items. }