Round-1 Protocol

Appendix E: Round-1 Protocol

Thank you for agreeing to participate in this study. My name is XX and I am a member of the study team. I’ll be working with you today on this study. The purpose of this study is to test a survey instrument on a smartphone that will be used for a national survey of Americans’ food acquisitions and purchases. The survey will help us better understand what kind of food Americans get and how they get it and will help make policies to ensure that all Americans can afford to get healthy food. The survey is to be conducted by the U.S. Department of Agriculture.

In today’s session, we will evaluate how easy or difficult it is to answer the survey for people like you. Your experiences will help us improve the survey instrument before it is used by a larger group of respondents. Today, we will ask you and your household member to complete some questionnaires which are the initial part of the National Food Study, then we will ask you a few questions about your experience with completing these questionnaires. Upon completion of today’s session, [each of] you will receive $50 for your participation. Do you have any questions before we move on?

<Answer participants’ questions if any>

Great. Now, I would like you to read this consent form. It explains the purpose of this study and your rights as a participant. It also informs you that we would like to take an audio-and-video recording of our meetings during the study. The purpose for the recording is to get an accurate record of your feedback so that we can best improve the survey instrument. Only those of us who are involved in this study will have access to the recording and it will be strictly used for research purposes. Your name will not be associated with the recording or any of the other information collected during the study.

<Wait until the participants completes reading the Consent Form>

Do you have any questions about the Consent Form?

< Answer participants’ questions if any>

Great, now I would like you to sign this Consent Form.

<After the Consent Forms are sighed>

Thank you. Now I would like you to complete this demographic questionnaire.

<Wait until the participants complete the demographic questionnaire>

Thank you both.

<Speak to the primary respondent>

I will work with you first, then <the other household-member participant>

<Speak to the other household member>

XXX, you can take a break now. I will call you when I am done with < primary respondent >

<Back to the primary respondent>

You will first access the survey instrument and then follow what is said on the screens to complete the questionnaires. Before you do that, I’d like to teach you a method called “Think Aloud.”

<Practice Think Aloud>

I want you to complete the questionnaire as if you are alone by yourself, but with one major difference. I would like you to “think aloud” as you are answering the questionnaires: Speak out aloud everything that you are thinking and feeling as you go about working on the questionnaires. I am interested in the process you go through in your mind when you answer the questionnaires. Let me show you how I would think aloud: For example, if I were to think aloud while I’m grocery shopping, it might sound like this: “I start my shopping in the produce section because that’s the first section I encounter. I check my list to see what I need. After I find the vegetable I want to buy, I move to the deli counter because that’s next to the produce section.”

Let’s do a practice: Please think aloud as you answer this question: How many phone apps did you use today?

[HINT: Probe if they fall silent. If they just give a number ask them to do it again but this time so that you can understand how they came up with the number. Probes should be kept to a minimum – trying to not interfere with the timing data, so use sparingly: Keep talking…; Um-hum?; What are you thinking?; Tell me more…]

Great. That’s what I want you to do throughout this session. I will remind you to think aloud if you get quiet.

At the end of the session, I will have some questions for you about your experience of answering these questionnaires.

<Set up smartphone screen sharing>

In order to observe how you complete these questionnaires, I’d like you to share your phone screen on the MS Teams.

<Assist the P to set up screensharing if needed>

<Once seeing participant’s smartphone screen, put the link to the web instrument in the chat>

Great. Now we are ready to complete the questionnaires. If you run into any difficulties as you work with the survey instrument, please don’t blame yourself. Any difficulties are the result of instrument itself, not your skills or abilities. We are not testing you. You are helping us to test the instrument.

Please open the chat and tap the link I just put there. Let me know when you see the questionnaire on the screen.

<Wait until the P has the instrument on screen>

<Start screen recording>

Remember to think aloud as you answer the questions on the online form. If you fall silent, I may remind you to think aloud by saying “keep talking” or “what are you thinking?” <Observe P’s performance>

<Make note if they had any difficulty with the address screen.

Make note if they had any difficulty with the new matrix on who will be at home this week.

• Make note of who they said would be there and who would be away and if anyone was a visitor.

Make note if they had any difficulty with the proxy questions. What did they comment about who should/should not answer the questions for themselves

Make note if they had any difficulty with the food sharing group (FSG) screens.

• Make note of who at their address they included in their FSG.

• Were there multiple FSG in the household?

Make note if they had any difficulty with the SNAP/WIC series.

TA will use emerging probes during session as needed:

• "What were you thinking about as you answered this question?"

• "Tell me more about that."

"What does this phrase/word mean to you?">

<Once P completes the questionnaire, put the link to the satisfaction and debriefing questionnaire in the chat

URL for Satisfaction/Debriefing/:

https://research.rm.census.gov/xxxxxxxx >

Great job. Now I’d like you to tap the link I’ve just put in the Chat and answer that survey as well.

<Wait until the P completes the satisfaction questionnaire>

Thank you. Now I’d like you to stop sharing your phone screen and then take a short break before we re-convene. Can you help get your <spouse/child/whoever> so that I can work with him/her?

<Turn to the 2^nd household member>

Hello, thank you for your patience. Let’s first practice Think Aloud.

<Repeat the Think Aloud Practice part on p3>

< Repeat setting up screenshare on p3>

<Once seeing participant’s smartphone screen, put the link to the web instrument in the chat>

Great. Now we are ready to complete the questionnaire. Please open the chat and tap the link I just put there. Let me know when you see the questionnaire on the screen.

<Wait until the P has the web instrument on screen>

Now please follow the instructions on the screen to answer the questionnaire. Remember to think aloud as you answer the questions on the online form. If you fall silent, I may remind you to think aloud by saying “keep talking” or “what are you thinking?”

<Observe P’s performance, probe as necessary and fit>

<Once P completes all the questionnaires, put the link to the satisfaction questionnaire in the chat>

Great job. Now I’d like you to tap the link I’ve just put in the Chat and answer that survey as well.

<Wait until the P completes the satisfaction questionnaire>

Thank you. Now I’d like you to stop sharing your phone screen.

<Wait until the P stops sharing screen>

Now I’d like to ask you a few questions about your experience with completing these surveys:

<Conduct an ad-hoc debriefing based on observation, also probe on extreme responses to the satisfaction questionnaire>

<For any question they struggled with during the session you could debrief this way:

Take a look at this screen. Any thoughts when you saw this?>

<Question about roster series A1e-g and A2 Note to TA: Research question is to get an understanding of whether the respondent understands the rostering concept: What were you thinking about as you answered these questions?>

<Question about A3 – Did you have any thoughts about answering the question “Does {name} contribute financially to this household’s expenses?”

Note to TA: this question location is not fixed yet. It came towards the beginning of the session after they answered about who would be at their home this week. Perhaps it needs to come later closer to the income series of questions. Note if participants mention it was out of place.>

<Question about Food Sharing Group A3D_0 -A3D2a: Research question is to get an understanding of whether the respondent understands the concept of a food sharing group as we have written it in the questionnaire?

When you answered this question what were you thinking of? What does food shaing here mean to you? What did it make you think of? Can you share about who is in your hhd? Who is in your food sharing group? Are they the same people? How did you know to include or exclude [XX insert name of person they included or excluded ] in your food sharing group? How did you decide who should be included here and who should not?

Note to TA: We may like to show them examples of what a food sharing group looks like, and what it doesn't look like.>

Question about B5. This week and next week [is NAME] attending classes for kindergarten, elementary or grade school, middle school or junior high school, high school, or some other school (college, technical school, etc.)?:

How would you answer this question if you/your child had one week in school and one week in spring break?>

<Question about P6 on height and weight. The next questions are about {your/NAME’s} height and weight. In what units will you report {your/NAME’s} height?

What does "Units” mean to you in this question? What if it said “The next questions are about {your/NAME’s} height and weight.” How do you report your height>

<Question about Proxy reporting – so section before A3B. Note to TA: The research question is to get a better understanding of how respondent understands the proxy series. Does the PR understand the concept of proxy reporting?

What were you thinking when you had to answer these questions about [XX, YY, ZZ] in your household?

• You said XX would report for themselves / that you would report for XX. Can you tell me what you were thinking when you marked that?

• How did you decide that XX would complete their own survey / that they would answer the survey/food log for XX

• [If they said they would answer the food log for XX in their household] Do you plan to report for XX for the entire period (e.g., 4, 5 or 7 days?) How long did you think you would be reporting for them?>

Thank you. You have completed this session. Can you please ask <primary respondent> to join the meeting again?

<Speak to the primary respondent>

Hello, thanks for your patience!

Now I’d like to ask you a few questions about your experience with completing these surveys:

<Conduct an ad-hoc debriefing based on observation, also probe on extreme responses to the satisfaction questionnaire>

<Stop screen recording>

<Put the link to the incentive voucher in the Chat>

Thank you. You have completed this session. Now, I’d like you to sign the incentive voucher. Please open the Chat and tap the link to the incentive voucher, fill in all the information, and sign.

<Wait for the incentive voucher to be signed>

Great. You will receive the incentives via the USPS mail. Thank you again for your participation!

6