Round-2&3 Protocol

Appendix O: Round 2&3 Protocol

Thank you for agreeing to participate in this study. My name is XX and I am a member of the study team. I’ll be working with you today on this study. The purpose of this study is to test a web-based survey instrument that will be used for the national survey of Americans’ food acquisitions and purchases. The survey will help us better understand what kind of food Americans get and how they get it and will help make policies such that all Americans can afford to get healthy food. The survey is to be conducted by the Census Bureau on behalf of U.S. Department of Agriculture.

In today’s session, we will evaluate how easy or difficult it is to use the web instrument for people like you. Your experiences will help us improve the instrument before it is used by a larger group of respondents. Today, we will ask you (and your child) to use the instrument to report some food items acquired in differnt scenarios, then we will ask you a few questions about your experience with completing these tasks. Upon completion of today’s session, each of you will receive $75(for Round 2)/$100(for Round 3) for your participation. Do you have any questions before we move on?

<Answer participants’ questions if any>

Great. Now, I would like you to read this Disclaimer. It explains the purpose of this study and your rights as a participant. It also informs you that we would like to take an audio-and-video recording of our meetings during the study. The purpose for the recording is to get an accurate record of your performance and feedback so that we can best improve the web instrument. Only those of us who are involved in this study will have access to the recording and it will be strictly used for research purposes. Your name will not be associated with the recording or any of the other data collected during the study period.

<Wait until the participants completes reading the Disclaimer>

Do you have any questions about the Disclaimer?

< Answer participants’ questions if any>

Great, now I would like you (and your child, if applicable) to read this Consent Form and sign the form if you agree with it.

<After the Consent Forms are sighed>

Thank you. Now I would like you to complete this demographic questionnaire.

<Wait until the participants completes the demographic questionnaire>

Thank you both.

<If a child is participating, speak to the primary respondent, otherwise, skip two sentences below>

I will work with you first, then your child.

<Speak to the other household member>

XXX, you can take a break now. I will call you when I am done with < primary respondent >

<Back to the primary respondent>

You will first access to the web instrument and then report the food items. Before you do that, I’d like to teach you a method called “Think Aloud.”

<Practice Think Aloud>

I want you to complete the tasks as if you are alone by yourself, but with one major difference. I would like you to “think aloud” as you are performing the tasks: Speak out aloud everything that you are thinking and feeling as you go about working with the web instrument . I am interested in the process you go through in your mind when you are using the web instrument . Let me show you how I would think aloud: For example, if I were to think aloud while I’m grocery shopping, it might sound like this: “I start my shopping in the produce section because that’s the first section I encounter. I check my list to see what I need. After I found the vegetable I want to buy, I move to the deli counter because that’s next to the produce section.”

Let’s do a practice: Please think aloud as you answer this question: How many phone apps did you use today?

[HINT: Probe if they fall silent. If they just give a number ask them to do it again but this time so that you can understand how they came up with the number. Probes should be kept to a minimum – trying to not interfere with the timing data, so used sparingly: Keep talking…; Um-hum?; What are you thinking?; Tell me more…]

Great. That’s what I want you to do throughout this session. I will remind you to think aloud if you get quiet.

At the end of the session, I will have some questions for you about your experience of using the web instrument.

<Set up smartphone screen sharing>

In order to observe how you use the web instrument, I’d like you to share your phone screen on the MS Teams.

<Assist the P to set up screensharing if needed>

<Once seeing participant’s smartphone screen, put the link to the web instrument in the chat>

Great. Now we are ready to start the tasks. If you run into any difficulties as you work with the web instrument, please don’t blame yourself. Any difficulties are the result of web instrument itself, not your skills or abilities. Please open the chat and tap the link I just put there. Let me know when you see the web instrument on the screen.

<Wait until the P has the web instrument on screen>

<Start screen recording>

<Start Task 1 - Training>

Now I’d like you watch the video and learn how to use the web instrument. Please Think Aloud while watching the video.

<Observe P’s performance of watching the video>

<Once P finishes watching video, put the link to the training quiz in the chat>

Great job. Now I’d like you to tap the link I’ve just put in the Chat and take the quiz.

<Wait until the P completes the quiz>

<Start Task 2 – FAH use case>

Thank you. Now let’s do the second task. Please read aloud this task instruction.

<Hand P the Use Case 1 – FAH>

<After P completes reading the use case, put the grocery bag on the table>

Here are the groceries you just got, I’d like you to report them now. Keep Thinking Aloud.

<Observe P’s performance of FAH Use Case>

<Wait until the P completes FAH Use Case>

<Start Task 3 – FAFH use case>

Thank you. Now let’s do the third task. Please read aloud this task instruction.

<Hand P the Use Case 2 – FAFH>

<After P completes reading the use case, put the food image pakage on the table>

Here are the foods you remember ordered, I’d like you to report them now. Keep Thinking Aloud.

<Observe P’s performance of FAFH Use Case >

<Wait until the P completes FAH Use Case>

<Start Task 4 – School Meal use case. This use case is to be performed by PR who is proxy reporting for school children, otherwise skip this task and jump to School Child Reporting or PR debriefing accordingly>

Thank you. Now let’s do the next task. Please read aloud this task instruction.

<Hand P the Use Case 3 – School Meal, proxy>

<After P completes reading the instruction, say:>

Please do what the instruction says. Keep Thinking Aloud

<Observe P’s performance of School Meal Use Case >

<Once P completes all the tasks, put the link to the satisfaction questionnaire in the chat>

Great job. Now I’d like you to tap the link I’ve just put in the Chat and answer that survey as well.

<Wait until the P completes the satisfaction questionnaire>

Thank you. Now I’d like you to stop sharing your phone screen.

<School Child Reporting - If there is a school child reporting, proceed to the following. Otherwise, skip the school child reporting part, and jump to PR debriefing>

Can you help get your child so that I can work with him/her on reporting? Then you can take a short break before we re-convene.

<Turn to the school child>

Hello, thank you for your patience. Let’s first practice Think Aloud.

<Repeat the Think Aloud Practice part on p3>

< Repeat setting up screenshare on p3>

<Once seeing participant’s smartphone screen, put the link to the web instrument in the chat>

Great. Now we are ready to start the tasks. If you run into any difficulties as you work with the web instrument, please don’t blame yourself. Any difficulties are the result of web instrumentitself, not your skills or abilities. Please open the chat and tap the link I just put there. Let me know when you see the web instrument on the screen.

<Wait until the P has the web instrument on screen>

<Start Task 4 – School Meal use case. This use case is to be performed by the school child. If it’s not applicable, skip this task>

Thank you. Now let’s do the next task. Please read aloud this task instruction.

<Hand P the Use Case 3 – School Meal, self report>

<After P completes reading the instruction, say:>

Please do what the instruction says. Keep Thinking Aloud

<Observe P’s performance of School Meal Use Case >

<Once P completes all the tasks, put the link to the satisfaction questionnaire in the chat>

Great job. Now I’d like you to tap the link I’ve just put in the Chat and answer that survey as well.

<Wait until the P completes the satisfaction questionnaire>

Thank you. Now I’d like you to stop sharing your phone screen.

<Wait until the P stops sharing screen>

Now I’d like to ask you a few questions about your experience with completing these surveys:

<Conduct an ad-hoc debriefing based on observation, also probe on extreme responses to the satisfaction questionnaire>

Thank you. You have completed this session. Can you please ask <primary respondent> to join the meeting again?

<Speak to the primary respondent>

Hello, thanks for your patience!

<PR debriefing>

Now I’d like to ask you a few questions about your experience with completing these surveys:

<Conduct an ad-hoc debriefing based on observation, also probe on extreme responses to the satisfaction questionnaire>

<Closing interview - This part is for Round 3 only. For Round 2, skip it and jump to incentive voucher portion>

Now, I’d like you to answer a few more questions regarding your food reporting experience. (if the screen is not shared) Please share your screen with me again. Tap the button with the label of “Closing Interview,” and follow the instructions on the screen. Keep Thinking Aloud.

<Observe P’s performance of doing the Closing Interview until it is completed>

<Conduct an ad-hoc debriefing based on observation>

<Stop screen recording>

<Put the link to the incentive voucher in the Chat>

Thank you. You have completed this session. Now, I’d like you to sign the incentive voucher. Please open the Chat and tap the link to the incentive voucher, fill in all the information, and sign.

<Wait for the incentive voucher to be signed, then give the incentive cash to PR in person>

Thank you again for your participation!

TA Notes:

Timing on Individual Cases

After 10 minutes for the FAH use case; After 7 minutes for the FAFH use case; After 5 minutes for the school meal use case – Ask Participant:

Do you feel like you are close to being done reporting for this scenario?

If they offer that they are far from being done, you can move them on, say:

Due to time we will move on to the next scenario.

If they say they are close – allow them to work for two additional minutes.

After 15 minutes for the FAH use case; After 10 minutes for the FAFH use case; After 7 minutes for the school meal use case – if participant is not close to an answer, say:

Thank you for your work on this scenario. Due to time constraints, I would like to move us onto the next scenario.

7