Appendix V Protocol for Lab-Based Usability Testing

Appendix V: Protocol for Lab-Based Usability Testing

MATERIALS NEEDED FOR SESSION

• Interviewer Protocol Script

• Debriefing guide for field data entry (Day 1-3)

• Debriefing guide of usability session

• Three use cases (emailed to participant ahead of time)

• Food items to be used for data entry tasks (will be delivered to participant’s home in advance)

• Electronic satisfaction questionnaire

• Participant’s smartphone

• Participant’s laptop or PC

• Pens and Paper

Introduction Script

Thank you for your time today. My name is XX and I am a member of the study team. I will be working with you today to conduct a laboratory usability evaluation for the FoodLogger.

On Day 1, you read the Disclaimer and signed a written consent form to participate in this lab-based usability evaluation. I would like to refresh your memory about the main points: The two documents explain the purpose of the study and your rights as a participant. The written consent also informs you that we would like to take an audio-and-video recording as you are working with the FoodLogger on your phone today. The purpose for the recording is to keep records of your using FoodLogger during the test so that we can conduct further analysis after today’s session. Only those of us who are involved in the project will have access to the recording, and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session.

Today you will be helping us evaluate the FoodLogger. You have already been using FoodLogger for a few days, so today we want to ask you about your experiences and observe you using FoodLogger to report food information. We are going to use your comments and experiences as well as the comments and experiences of other participants to help improve FoodLogger. I did not create the app, so don’t feel like you should hold back your thoughts to be polite to me. We appreciate your help so we can make FoodLogger work well for every respondent. Do you have any questions before we begin?

Usability Session

Great! For our session today, we want to observe you using FoodLogger to report food items that have been delivered to you. I will give you a set of scenarios of food events and you will use FoodLogger on your phone to enter the information about those events. If you run into any difficulties as you work on FoodLogger, please don’t blame yourself. Any difficulties are the result of FoodLogger’s design, not your skills or abilities.

Thank you.

Practice Think Aloud

I want you to work through these food event scenarios as you would if I were not here, but with one major difference. I would like you to “think aloud” as you are using FoodLogger. I am interested in what you do on FoodLogger, but I am also interested in the process you go through in your mind when you use FoodLogger to enter food information. I would like you to tell me everything that you are thinking and feeling as you go about working on each scenario. Let me show you how I would think aloud: For example, if I were to think aloud while I’m grocery shopping, it might sound like this: ”I start my shopping in the produce section because that’s the first section I encounter. I check my list to see what I need. After I found the produce I want to buy, I move to the deli counter because that’s next to the produce section.”

Let’s do a practice “think aloud” before we start: Please think aloud as you answer this question: How many phone apps did you use today?

[Probe if they fall silent; if they just give a number ask them to do it again but this time so that you can understand how they came up with the number; after the think-aloud explain why we are asking them to do this, e.g., if the Census was interested in the number of apps, which we are not, but if we were, and someone was looking at their phone and wondered aloud if they should include making a call as using an app, them saying those words would clue us in that we needed to fix our question, make it clear that making a phone call should be included, etc. Probes should be kept to a minimum – trying to not interfere with the timing data, so used sparingly: Keep talking…; Um-hum?; What are you thinking?; Tell me more…]

Great. That’s what I want you to do throughout this session. I will remind you to think aloud if you get quiet.

At the end of the session I will have some questions for you about your experience as you worked on the scenarios. We may re-open some of the screens and talk about them.

Set up smartphone screen sharing

In order to observe how you work with FoodLogger, we need to install MS Teams on your smartphone. Let me walk you through the installation steps…

[Refer to Appendix Y. Make sure you are seeing participant’s smartphone screen]

Interview – Starting with First Scenario

We will be working on two/three scenarios today.

I emailed/mailed you a document with each scenario. Open that document and read the first one, then gather the food items on the list. Once you have the food items ready, read the scenario aloud. You will use that information, along with the food in front of you, to report each scenario into the FoodLogger on your phone. As you work, remember to talk to me about what you are thinking and feeling. Once you have finished entering the information, please tell us. For example, say, “I’m done” or “I’m finished with this one.”

I want to let you know that we have a limited amount of time for this session so I may need to move you on to the next scenario depending on what we have learned and what we still need to learn about the application.

You may begin, remembering to think aloud.

[After completing FAH use case, repeat the process with each subsequent use case. Say to respondent “Okay now we’re going to get set up for the next scenario. Please gather the food items needed. Once you have them ready, please read the scenario aloud and then begin.” When all tasks are finished, send the link to the satisfaction questionnaire (Appendix W)]

Now I’d like you to fill out a satisfaction questionnaire. I’m sharing a link to it in the chat of our Teams meeting. You can click on it and follow the instructions on the screen.

When you are done, I will ask you some questions about what you have done today.

[Ask debriefing questions (Appendix V)]

Timing on Individual Cases

After 10 minutes for the FAH use case; After 7 minutes for the FAFH use case; After 5 minutes for the school meal use case – Ask Participant:

Do you feel like you are close to being done reporting for this scenario?

If they offer that they are far from being done, you can move them on, say:

Due to time we will move on to the next scenario.

If they say they are close – allow them to work for two additional minutes.

After 15 minutes for the FAH use case; After 10 minutes for the FAFH use case; After 7 minutes for the school meal use case – if participant is not close to an answer, say:

Thank you for your work on this scenario. Due to time constraints, I would like to move us onto the next scenario.

Debriefing Probes

[If participant marked any satisfaction questionnaire responses either on the low or high end, probe:] Tell me what you were thinking about when you marked this question XXX?

[Use Appendix V for further debriefing questions]

Debriefing on Field Data Entry (Days 1-3)

I have a few more questions to ask you about your experiences in enter food information in the past three days.

[Use Appendix M for field data entry debriefing]

Finishing the Session

Okay, that’s all that I have for you today. Please continue to use FoodLogger and enter all food that you get until your seven-day study period is over. We will meet you again in four days for one last debriefing and to confirm your incentive. Many thanks for your help with this project.