Protocol

School Meals Application Usability Testing Protocol

Participant ID #: ________________________

Interview Date: ____ / ____ / ________

Interviewer initials: _____

Start Time: ____________ AM / PM End Time: ____________ AM / PM

Introduction

Thank you for your time today. My name is [INTERVIEWER NAME] and I work with the Human Factors and Usability Group at the U.S. Census Bureau. I will be working with you today. In our group, we evaluate how easy or difficult it is to use online web surveys, tools, and applications. What works well we keep, however, when potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes out to a much larger group.

Today you will be helping us evaluate the design of a website used to submit an application for the National School Lunch Program. You will do this by completing the application using [your tablet/your smartphone/this laptop]. Any information that you choose to share will not be shared with anyone else, including your child’s school and the National School Lunch Program. Your [child/children]’s participation in the school meals program won’t change because of anything you report. The website being tested is only a prototype therefore the information you enter into the application is not being stored or sent anywhere.

If you run into any difficulties as you work on the application, please don’t blame yourself. Any difficulties are the result of the design of the online application, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants that work with us on this application to help improve it. I did not create the website, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make the website work well for everyone.

Informed Consent

• HAND RESPONDENT A COPY OF THE CONSENT FORM.

Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant, such as that all information you provide is confidential and we won’t share your name or any other personal information. It also asks for your permission to have this session audio recorded and to capture video from the screen as you work. Please ask me any questions you have about this document. Once you have finished reading the document, please sign and date it.

• PARTICIPANT READS FORM THEN SIGNS AND DATES FORM.

• INTERVIEWER SIGNS AND DATES FORM.

• START CAMTASIA RECORDING (If P consented)

I will now start recording the session. Your participation in this interview is voluntary, and you may decide not to answer any question you don’t want to, or stop the testing at any time. This session is scheduled for about 60 minutes. As a thank you for your help, we will give you $40 after the testing is completed.

Before we begin, do you have any questions about the process?

Demographic and Mobile Questionnaires

I am going to go around to the other room and do a sound check. While I am doing that, I have two questionnaires that I’d like you to fill out. They collect information about you and how often you use different electronic devices. You can start these now while I am doing the sound check. I am going to leave but we will still be able to communicate through microphones and speakers. Do you have any questions?

• HAVE PARTICIPANT FILL OUT DEMOGRAPHIC AND MOBILE EXPERIENCE QUESTIONNAIRES.

• MOVE TO CONTROL ROOM

• MAKE SURE THEY HAVE HIT THE ‘SUBMIT’ BUTTON ON QUESTIONNAIRES BEFORE BEGINNING NEXT PART.

Think Aloud Practice and Protocol

During the test session while you are performing the tasks, I’d like you to “think aloud.” You just speak aloud what you are thinking at that moment. Because thinking aloud is not something that we do every day, let’s do a practice before we start: Please think aloud as you answer the question, how many windows are in your home?

• Probe if they fall silent; if they just give a number ask them to do it again but this time so that you can understand how they came up with the number; after they do the think aloud explain why we are asking them to do this, e.g., if the Census was interested in the number of windows, which we are not, but if we were, and someone was looking at their sliding glass door and wondered aloud that they didn’t know if they should include it or not, them saying those words would clue us in that we needed to fix our question, make it clear that sliding glass doors should be included, etc.

• Probes that will be used in study

• Keep talking

• Um-hum?

• What are you thinking?

• What are you including?

Great, that’s what I want you to do throughout our session. I will remind you to think aloud if you get quiet.

At the end of the session I will have some questions for you about your experience and how satisfied you were with the online application.

Now I will bring up the website.

Interview

• OPEN BROWSER AND BRING UP WEBSITE. URL WILL BE SAVED IN BROWSER FAVORITES FOR EASY NAVIGATION https://www.fns.usda.gov/school-meals/web-based-prototype/docs/index.html#/

• HAVE RESPONDENT COMPLETE THE ENTIRE APPLICATION WHILE THINKING ALOUD. ENCOURAGE HIM/HER TO THINK ALOUD AS NEEDED.

Again, as you go through the application, I’d like you to say out loud what you are thinking about. This will give me a better idea of how the application is working for you. Do you have any questions for me before we begin?

Ok. Please start filling out the application.

User Interaction Satisfaction Questionnaire

• AFTER PARTICIPANT HAS COMPLETED THE APPLICATION, OPEN THE USER INTERACTION SATISFACTION QUESTIONNAIRE

Now I’d like you to fill out a Satisfaction Questionnaire. When you are done I will come around to ask you some questions about the Web site.

Debriefing Wrap-up Questions

Thank you so much. Before we wrap things up I just have a few debriefing questions:

Question 1: Do you have any sources of income that you did not report because the application did not ask about it, or because you could not find a place to report it?

Question 2: If you were at home filling this out, would you have done anything differently?

Question 3: Would you have consulted additional documents or information sources prior to submitting the application?

Question 4: Would you likely have completed the application in one sitting, or would you have taken a break and come back later? If so, why?

Closeout and Respondent Payment

I want to thank you very much for your participation.

I also want to reassure you that none of the personal information you shared with me today will be used in any way other than to help us improve the application. No one other than my fellow researchers will see anything you provided to me. We will not share it with anyone, including your child’s school and the National School Lunch Program.

I will now give you $40 and ask you to sign a receipt form verifying that you received the money.

• STOP CAMTASIA. HAND THE CASH INCENTIVE TO THE PARTICIPANT.