Protocol

FNS Testing Protocol

MATERIALS NEEDED FOR INTERVIEW

• Interviewer Protocol Script

• Consent Form

• Demographic Questionnaire

• Mobile Use Questionnaire

• Mailing materials

• QUIS Survey location

• Laptop or Mobile Phone

• Incentive

• Payment Receipt

• Pens and Paper

• Extra Equipment –e.g., microphone for laptop studies, charging cords

• Location of extra UserIDs if using

Protocol

Thank you for your time today. My name is XX and I work with the Human Factors and Usability Group at the U.S. Census Bureau. I will be working with you today. In our group, we evaluate how easy or difficult it is to use web surveys. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes out to a much larger group.

Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to take a video recording of the device as you use it along with the audio from this session to get an accurate record of your feedback. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session.

[Hand consent form; give time to read and sign; sign own name and date. Show participants where the camera is and what part of the screen will be recorded start recording.]

Thank you.

Start Recording (Camtasia)

OK great.

I have two questionnaires that I’d like you to fill out. They collect information about you and how often you use different electronic devices.

[Have participant fill out the background and demographic questionnaire and the mobile experience questionnaire. You can leave the room and watch from observation room when they are ready to begin again. If off site—simply move away from where they are answering so they don’t feel like we are looking over their shoulder. Make sure they hit the SUBMIT button. C:\Projects\FNS ]

Thank you.

Today you will be helping us evaluate the design of a form that we are working on for the U.S. Department of Agriculture, the Food and Nutrition Service. The Supplemental Nutrition Assistance Program or Food Stamps. You will be helping us evaluate the form that must be filled out in order to determine eligibility for receiving the SNAP benefits. You will do this by completing it using [your smartphone/this laptop].

If you run into any difficulties as you work on the online form, please don’t blame yourself. Any difficulties are the result of the design of the form, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants that work with us on this project to help improve the form. I did not create the form, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make it work well for everyone.

[Think aloud] I want you to answer the questions exactly the way you would if you were at your home, but with one major difference. I would like you to think aloud as you answer the questions. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.

Practice Think Aloud

Because thinking aloud is not something that we do every day, let’s do a practice before we start: Please think aloud as you answer the question, how many windows are in your home?

[Probe if they fall silent; if they just give a number ask them to do it again but this time so that you can understand how they came up with the number; after they do the think aloud explain why we are asking them to do this, e.g., if the Census was interested in the number of windows, which we are not, but if we were, and someone was looking at their sliding glass door and wondered aloud that they didn’t know if they should include it or not, them saying those words would clue us in that we needed to fix our question, make it clear that sliding glass doors should be included, etc.

Probes that will be used in study

• Keep talking

• Um-hum?

• What are you thinking?]

Great that’s what I want you to do throughout our session. I will remind you to think aloud if you get quiet.

At the end of the session I will have some questions for you about your experience as you worked on specific screens and how satisfied you were with the online form. We may open up some of the screens and talk about them.

Eye tracking – for laptop cases

[TOBII ET set up with bar on laptop]

While you are performing the task, we will record where you look on the screen. In order to accurately record where your eyes look, we would like you to do a simple task that will allow the computer to find your eyes. To get the best possible data, first please situate yourself in a position that is comfortable for you. On the next screen, you will see a red dot moving across the screen. Please follow the red ball with your eyes as it moves and keep your eyes on it as it pauses in each corner of the screen.

Ready?

[Calibrate the P’s eyes, check the calibration quality, repeat calibration until the calibration quality is satisfactory – e.g., green markers tightly on the gray balls]

Interview – Starting with online form

[Hand participant the URL. Make certain to write down the PIN or any information that user may need later in survey]

Imagine that you wanted to apply for the SNAP (Food Stamps) benefits. What would you do?

Please answer the online form questions as they apply to you in your real life.

Remember that I am interested in what you have to say as you go about answering the online form questions.

[Note: you will need to redirect them to the correct URL. Don’t redirect them until after they attempt navigate to the correct state website for the online SNAP form.

https://insert it here ]

Interruption – Log out vignette

[STOP Respondent after the mid point of the form. Ask them to pretend that they had to leave and to do something else.]

Imagine you had to go somewhere and were not able to finish answering the questions at this time. What would you do?

[Once they are exited, then ask them to resume.]

Okay now I’d like you to imagine that you are ready to continue working on the online form. What would you do?

[Interviewer instruction: You may either text them or type the correct URL for them if they had difficulty in the start of the session. Note what participant does if they don’t have their PIN Give the respondent their PIN if they didn’t have it]

FAQ Interruption

[Interruption for FAQ or Help. After they have completed aprox ¾ the questionnaire]

Okay now I would like you to imagine you had a question about this online form. You wanted to learn more about why the form is asking for your social security number. What would you do?

You have some concerns about whether the information you enter into the form will be kept safe. What would you do to learn more about how the information will be kept secure?

[First wait to see if user can find the answer on their own. If it appears user does not see/think of using the FAQ can probe, such as, “Is there any other way you could find this information?” “Is there any place on the website that would give you this information.”]

[Once they find the information, see if they can return to the survey without assistance.]

[After final SUBMIT is clicked, have participant answer satisfaction questions.]

Satisfaction Questions

Now I would like you to answer these satisfaction questions.

[Open the QUIS on the laptop and have participant answer the questions.]

Okay great. Now the next task is on the laptop. We are going to go over some of the screens you saw.

Vignettes

1. Log out and log back in – during session

2. Question about the FAQ – during session

3. Seeing different (less frequented pathing) – after SATQ

Placeholder vignette – if there is a question from team about a less traveled path in the form – we will ask them to imagine a scenario and answer the question based on this scenario

Debriefing on Individual Screens

[Open the ppt screenshots Ask the probes using the power point slide deck with screen shots. Show respondents the screen.]

Do you have any comments on this screen?

[If there were issues with it:] Tell me more about your experience with this screen.

[On screens where there were instructions –]

Did you read the instructions? What did they mean to you?

Additional Debriefing Questions

[If participant marked any QUIS responses either on the low or high end, probe:]

Tell me what you were thinking about when you marked this question a XXX.

Overall, what was your impression of the online form? If you were applying for SNAP benefits, would you do it online? Tell me about that.

Is there anything else you’d like to mention that we haven’t talked about?

Question about sharing data across agencies:

Government agencies collect information when people fill out forms for programs. One example is the information given when someone fills out an application for food assistance benefits. They also collect data using surveys, like the Census that is taken every ten years. How would you feel if the information that agencies get for programs was shared with other government agencies instead of asking for information directly, like for the Census? Tell me more about why you answered that way?

Is filling out a form like this, for getting food assistance, the same as filling out a Census form or other survey? What makes them the same or different?

Finishing up session

[Give the respondent incentive envelope [$40 incentive], and thank them for their time and effort. Collect materials and escort respondent out of the interviewing room.]

Okay, that’s all that I have today. Many thanks for your work on this project. The last thing I need you to do is sign this form that says I am giving you $40 as a thank you for your participation.

5