F1. Pre-test Findings Memo

Appendix F1

Pre-test Findings Memo

This page has been left blank for double-sided copying

To: Anita Singh

From: Kim McDonald, Emma Wells, Dory Thrasher, Julie Hartnack, and Jodie Davis

Date: 8/02/2023

Subject: SNAP Mobile Payment Pilots Evaluation: Pre-Test Results

This memo summarizes our findings from pre-testing four data collection instruments for the SNAP Mobile Payment Pilots Evaluation: the client experience survey, the survey of non-adopters, the key informant interview discussion guide, and the retailer interview discussion guide. Section I presents findings from the survey pre-tests with SNAP participants, Section II describes findings from the interview discussion guide pre-tests and Section III describes next steps.

I. SNAP Participant Surveys

The goal of the survey pre-test was to assess survey length and flow, respondents’ interpretation of questions, and the comprehensiveness of response options. We describe how we revised the Client Experience Survey and Survey of Non-Adopters based on observations from pre-tests conducted in three States (California, New York, and Oregon), two modes (self-administered and interviewer-administered), and two languages (English and Spanish). First, we describe the pre-test procedures and sample; then we review the findings and suggested modifications from the pre-test.

A. Pre-test procedures and sample

Survey structure and modifications for pre-testing. Questions in Section A and Section E are the same for both surveys, whereas Sections B, C, and D are tailored to specific respondent populations (Exhibit 1).

Exhibit 1. Sections by survey type

Section	Client Experience Survey	Survey of Non-Adopters
Section A: Screener	Screen for survey administration	Screen for survey administration
Section B: Awareness and Engagement with Pilot	Experiences setting up and using SNAP mobile payments	Familiarity with and interest in SNAP mobile payments
Section C: Barriers to Use and Feedback	Challenges encountered using SNAP mobile payments	Hesitations, challenges, and other opinions about SNAP mobile payments
Section D: Respondent Characteristics	Respondent characteristics	Respondent characteristics and cell phone capability
Section E: End	Collect contact information for incentives	Collect contact information for incentives

The option to redeem SNAP benefits by using an electronic benefit transfer (EBT) card on a mobile device (SNAP mobile payments) is not yet available. However, using and setting up mobile payments with a debit or credit card is a widely used payment option that works similarly to how SNAP mobile payments would work. To make the survey clearer for pre-test participants, the study team modified the surveys in the pre-test to use the term “mobile payments” instead of “SNAP mobile payments.” The study team also removed several references to SNAP that could have confused pre-test participants responding to questions about mobile payments in general. Some of the survey changes that were made for pre-testing are shown in Exhibit 2.

Exhibit 2. Example survey modifications for pre-test

Question	Revised pre-test question
B1 (Client Experience Survey)	How did you hear about SNAP mobile payments? Choose all that apply. By SNAP mobile payments, we mean using a mobile device to pay for your [SNAP] purchases in person at a store. Friends or neighbors Store outreach (for example, signs posted in the store, an information booth, a mailed flier, or a store app notification) Your local SNAP agency Social media Another way (specify)
C5 (Client Experience Survey)	What made it difficult to pay using SNAP mobile payments at checkout? Choose all that apply. You had technical issues or difficulty logging into the [SNAP app] You had technical issues or difficulty logging into your mobile wallet or payment app (for example, Google Wallet, Apple Wallet, or Samsung Wallet) You had technical issues with your phone You had connectivity or network issues while making the payment You had trouble using a second form of payment for non-SNAP eligible items The store register would not accept the payment Another reason (specify)

Pre-test sample. Pre-test participants were recruited by study team staff in June 2023. The study team administered the pre-test in three States: California, Oregon, and New York. These States were selected to help test questions that might differ by State, including those with references to State-specific SNAP apps, and to allow for in-person administration by the study team. In the three States, study team staff reached out to organizations that serve adults similar to those in the study population for the SNAP Mobile Payment Pilots Evaluation, including two food pantries, one local farmers’ market, and a local college. In coordination with these organizations, the study team invited individuals to participate in a pre-test interview with Mathematica. All pre-test participants were screened to ensure they were receiving SNAP benefits at the time of the pre-test interview and to confirm their use or non-use of mobile payments.

The Client Experience Survey was pre-tested with 9 participants who had used mobile payments in a store to buy food (8 in English and one in Spanish), and the Survey of Non-Adopters was fielded with nine pre-test participants who had not (8 in English and one in Spanish).

Pre-test procedures. Pre-test interviews were conducted in person or by telephone. To mirror self-administration of the survey by web, 11 participants filled out a printed version of the survey. These respondents self-administered the questionnaire by reading each question to themselves and circling or highlighting their answers. To replicate interviewer administration of the survey by telephone, interviewers on the study team read survey questions aloud to 7 participants and asked them to respond verbally. Exhibit 3 shows the breakdown of pre-test administration mode by survey type.

Exhibit 3. Pre-test administration mode by survey type

	Self-administered (paper)	Interviewer-administered (verbal)	Total
Client Experience Survey	7	2	9
Survey of Non-Adopters	4	5	9
Total	11	7	18

Before beginning each pre-test, the interviewer explained the context of the survey to help clarify to participants why they were being asked questions about both mobile payments and SNAP. Exhibit 4 includes the introductory text that interviewers read to pre-test participants.

Exhibit 4. Pre-test introduction

• Next year, five states will let people use their phones or smartwatches to pay with SNAP, either by using an app to generate a QR code or by adding an EBT card to a mobile wallet like Apple Pay or Google Pay. We’re going to be calling this SNAP mobile payments, and we’ll be surveying people who used or didn’t use this option in those five states.

• Because SNAP mobile payments are not available yet, this survey will ask you about mobile payments in general instead of SNAP mobile payments.

• The goal of this conversation is to improve the quality of the survey, not to collect actual answers to the questions.

• After you finish taking the survey, I am going to go through the survey questions with you to get your feedback on how they are worded to make sure we wrote questions that are understandable and that the answers make sense.

• We also want to test how long the survey takes, so if questions come up as you are taking the survey, please wait to ask them until the end.

Following administration of the survey, the interviewer asked the respondent to go back through the survey to discuss specific questions, asking a series of cognitive interview prompts to elicit feedback. These questions deepened our understanding of how respondents answered the survey, the challenges they encountered, and any suggestions they had for improvement.

The study team conducted two of the pre-test interviews in Spanish to obtain feedback on the Spanish translation. Interviews took between 20 to 30 minutes, and pre-test respondents were given or sent a $20 Visa gift card to thank them for their participation.

B. Pre-test findings

Overall, respondents thought the survey was straightforward and easy to understand. Respondents said the flow between sections made sense, and they thought the survey length was appropriate. Next, we discuss findings on respondent burden and recommendations for item-specific revisions.

Respondent burden. The average time it took to finish the survey across modes, locations, and languages was 4.8 minutes. However, there was a difference between the time it took to complete surveys administered on paper (self-administered) and surveys administered verbally (interview-administered). Self-administered surveys averaged 3.5 minutes, and interviewer-administered surveys averaged 6.6 minutes. A few factors account for this difference. First, participants were more inclined to ask questions during the interviewer-administered surveys despite the study team’s instruction to hold questions until the end. In addition, one interviewer-administered survey took 10 minutes and another took eight minutes, whereas none of the others took longer than six minutes. With those outliers removed, the average interviewer-administered survey time was 5.7 minutes. Therefore, although the time varied by administration mode, the overall burden of the pre-tested instruments was consistent with our five-minute estimate.

Item-specific revisions. Exhibit 5 describes the proposed modifications based on pre-test findings. These modifications will apply to both the English and Spanish surveys.

Exhibit 5. Survey modifications from pre-test (English and Spanish)

Survey	Item(s)	Feedback	Revision
Introduction
Both (Client Experience and Non-Adopters)	The U.S. Department of Agriculture, Food and Nutrition Service (FNS), is sponsoring a new study…	One respondent said we could be clearer about what FNS does and how it is related to SNAP.	Added explanation that FNS is “the government agency that administers SNAP.”
Section A
Both	A0. Some stores in your area will now let you use your mobile device (like your phone or your smartwatch) to pay for food with your [STATE SNAP] card.	Some respondents commented that we could be clearer in describing mobile payments. For example, some phones do not have the capability to run apps or make mobile payments, which could cause confusion.	Added “smart” before “phone” to clarify which phones support mobile payments and state SNAP apps.
Both	A2. Are you the main food shopper for your household?	Some respondents wondered if roommates would count as part of their household or were otherwise confused about the meaning of “household.”	Revised wording to clarify that we are asking about “SNAP household” instead of “household.”
Both		Many respondents were the only member of their household.	Added “or yourself” to account for individuals who are the only member of their household.
Section B
Both	B1 (Client Experience), B2, B6 (Non-Adopter) How did you hear about SNAP mobile payments? Choose all that apply.	One respondent understood the question, but commented that friends, family, neighbors, SNAP agencies, and even stores could all have social media.	Revised wording of question to ask “where” participants heard about SNAP mobile payments instead of “how.” Removed “social media” because this option is a method of communication, whereas the other options represent entities that share information.
Both		Many respondents noted that “family” was missing as an option.	Modified Option 1 (“friends or neighbors”) to include “family”
Both		One respondent thought they could probably find out about SNAP mobile payments from their SNAP caseworker. Another respondent said it would be helpful to fill in specific information instead of saying “local SNAP agency.”	Revised wording in Option 3 to clarify that “SNAP agency” includes “SNAP caseworker” Replace Option 3, “local SNAP agency” with “SNAP agency ([STATE SNAP])” to fill SNAP name by state.
Both		One respondent found out about mobile payments through community outreach and thought this option would be even more applicable to SNAP Mobile Payments.	Added “community outreach.”
Section C
Both	C1, C1b (Client Experience), C2, C2b (Non-Adopter) On a scale of 1-4, how easy was it to set up SNAP mobile payments on your mobile device?	Several respondents were confused because the question mentioned a scale of 1 to 4 but did not spell out where the answer choices fall on the scale.	Added “where 1 is very difficult, and 4 is very easy” to clarify how the response options correspond to the scale.
Both	C3 (Client Experience), C4 (Non-Adopter) How long did it take you to set up for SNAP mobile payments?	Most respondents remarked that it took them less than five minutes to set up for mobile payments.	Changed to an open field question. Setting up for SNAP mobile payments may differ from setting up for mobile payments in general, especially because the involvement of the SNAP app may add extra steps. Because we do not know exactly what this process will involve and time ranges may vary widely, we recommend transitioning to an open field response to allow participants to share more detailed and accurate responses.
Both	C5 (Client Experience), C1, C6 (Non-Adopter) What made it difficult/why were you not able to pay using SNAP mobile payments at checkout?	One respondent was confused about the difference between Option 2 (“you had technical difficulties with your phone”) and Option 3 (“you had connectivity or network issues while making the payment”). Another respondent noted that Option 4 (“connectivity or network issues”) would be clearer if it specified that this means Wi-Fi or cell service–related challenges.	Removed former Option 3 (“You had technical difficulty with your phone”). This item is redundant with the other options, and any other situations can be covered through “other.” Added “internet” and “cellular” to what is now Option 3.
Non-Adopter	C5. Why haven’t you used your mobile device to pay for your SNAP food purchases at a store?	One respondent noted that multiple options could apply.	Added “Choose all that apply.”
Both	C9 (Client Experience), C7 (Non-Adopter) Comparison between physical EBT card and SNAP mobile payments	Several respondents commented that they neither agreed nor disagreed and wished they had a middle option. Some respondents also noted that the question structure was confusing.	Restructured question to ask respondents to choose between a physical EBT card and SNAP mobile payment. Added a neutral “They are the same” option.
Both	C9e (Client Experience), C7e (Non-Adopter) Which option is more secure from theft or skimming…	Multiple respondents noted the threat of theft, which they also called skimming.	Added question “Which is/would be more secure from theft or skimming: a physical EBT card or SNAP mobile payment?”
Section D
Both	D1 (Client Experience), D2 (Non-Adopter) How comfortable are you using technology in your everyday life?	Several respondents thought there should be a middle option.	Added “neither comfortable nor uncomfortable” as an option.

II. Key Informant Interview Discussion Guide

This section summarizes our findings from pre-testing the draft implementation interview guides for key informants and retailers. The goal of the pre-test was to assess interview guide length and flow, understand respondents’ interpretation of questions, and gauge the comprehensiveness of the guide. We describe how we revised the Key Informants Interview Guide and the Retailer Interview Guide based on suggestions from pre-tests with staff involved in the pilots from the Illinois Department of Human Services staff, Aldi, and Schnuck’s Markets. First, we describe the pre-test procedures; then present the findings.

A. Pre-test respondents and procedures

Pre-test respondents. Pre-test participants were recruited by study team staff in June and July 2023. The study team administered the pre-tests with likely key informants in one pilot State—Illinois—to ensure the interview guides were comprehensive of activities the State Agency and their retailer partners expect to complete during the pilot planning and implementation. The study team invited staff from the Illinois Department of Human Services (IDHS) who are involved in the pilot planning to participate in the pre-test and requested their help identifying retailer partners. We the reached out to corporate-level contacts at Aldi and Schnuck’s Markets and requested their participation in a pre-test. We asked our corporate contacts to invite store managers or other key staff to join the conversation. Store managers and other corporate staff involved with payment technologies participated in the pre-test with Schnuck’s Market, and one corporate-level staff person participated from Aldi.

Pre-test guides.

Key informant interview guide. The key informant interview guide includes over 100 questions that cover all key informant respondents and three rounds of data collection. The guide will ultimately be tailored to ask only questions relevant for the respondent’s role in the pilot and round of data collection and will take no longer than an hour.

Retailer interview guide. The retailer interview guide is designed for use with corporate or store-level staff at participating pilot retailers. The interview guide includes fewer questions than the key informant interview guide and is designed to take no longer than 30 minutes with any respondent.

Example pre-testing questions

• What feedback do you have on the questions in this section?

• What did you think of the organization and flow of the questions?

• Could any questions be eliminated due to being repetitive or unapplicable?

• Are there any questions or topics you found sensitive?

Pre-test procedures. We took slightly different approaches to pre-testing the key informant interview guide and the retailer interview guide. We asked IDHS staff to review the full interview guide and consider recommendations for improving content or clarity. We then held a one-hour virtual meeting to debrief and ask targeted questions. We did not ask retailer staff to review any materials ahead of a 45-minute virtual meeting to review the questions together and ask for targeted feedback.

All virtual meetings were held in June and July 2023. In each virtual meeting we began by explaining the context of the Mobile Payments Pilot Evaluation, the pre-testing process, and the structure of the interview guides. We clarified that we were seeking feedback on the structure of the interview guides, the content of the questions, and the likelihood that the ultimate respondents would be able to answer the questions. We asked questions about each section and sub-section of the interview guide during the virtual meeting (see pop-out box for examples). We took notes throughout the conversations and recorded the meetings.

B. Pre-test findings

General acceptability: Overall, participants expressed that the interview guides were comprehensive and questions were straightforward. Respondents made several suggestions for improving the comprehension of certain sections and questions. For example, IDHS suggested adding another sentence to the introduction of the key informant interview to provide more context for the type of work Mathematica typically does for FNS. In addition, IDHS suggested adding more information about the reference period for the specific interview, given the key informant interviews will occur several times over a period of years. During discussions about interview guide administration, some retailers raised concerns that customer-facing workers, such as cashiers and clerks, might not have the knowledge to answer certain questions. Our respondents suggested that certain questions, particularly those about the retailer’s technology systems and customer shopping patterns would be better answered by managers and corporate-level staff with access to store data.

Language choice: During the discussion of the store environment, retailers expressed a preference for using the term "EBT" instead of "SNAP" in the text, noting that most in-store language, include point-of-sale systems, use “EBT” to refer to the payment type.

Item-specific revisions. Exhibit 6 describes the proposed modifications based on pre-test findings.

Exhibit 6. Questions modifications from pre-test

Discussion Guide Section	Question(s)	Feedback	Revision
Key Informant Interview Guide
Section B	Question 7. Are these formal or informal partnerships?	One respondent stated that “Partners could be contracted in another way, but still be an advisor on this.”	Revised text to include “contracted or noncontracted.”
Section C	Question 14. Were new staff hired specifically for the pilot?	One respondent suggested we ask about whether any staff have been added over time or whether they felt there were additional staff needs	Revised text to include new question text: “Have you added any new staff to the pilot [since the last interview]?” and an additional question “Do you feel additional staff are needed that were not hired or identified?”
Section D	Question 44 and 45. What steps did you take with your State’s mobile application developer to prepare to offer NFC/QR code technology?	Respondents are not planning to include QR codes in their State’s pilot.	Noted in parentheses for interviewer that these questions are only to be asked of States implementing this technology in their pilot program.
Section F	Operations (Round 2 only)	One respondent thought that the guide does not include enough questions about EBT testing at the retailer level.	Added probe to question 47, “Did you offer training to retailers that included walking through the app as if they were a customer?” Added additional question about testing with retailers. “Will retailers get to test out the payment method so they know how it works/what the experience is like?”
Retailer Interview Guide
Introduction	“…what strategies were used to ensure program integrity…”	Retailers stated that the average check/clerk may not understand the term “program integrity.”	Added text in parentheses that says “or payment security.”
Section B	Question 7. Did the store offer mobile payment methods (for non-SNAP purchases) prior to your involvement in the SNAP mobile payment pilot?	One respondent commented that the term EBT was used by staff in-stores instead of the term SNAP. Likewise, respondents suggested asking this question of both in-store and corporate level staff.	Revised to reference “SNAP/EBT” throughout the discussion guide. Add interviewer notes throughout guide indicating what questions to ask of in-store and corporate staff
		Respondents stated that mobile payment needs to be defined in this question.	Defined mobile payment in parentheses in text “for example, Apple Pay, Google Pay, etc.” We also suggest adding this text to the introduction to define the term up-front.
Section C	Question 12. What did your store need to do to prepare to offer SNAP mobile payments?	Some respondents wondered if in-store cashiers or clerks would know how to answer this question.	If a respondent does not know the answer to a question, we will instruct interviewers to reiterate that questions may be skipped during the interview. Added an interviewer note to ask this question of in-store only if relevant.
Section D	Question 23. Has your store experienced any challenges with mobile payments processing incorrectly?	A few respondents noted that complaints could also go to the customer satisfaction department	Added the question: “Has your store you received any complaints about mobile payment processing?”
Section E	Question 28. How difficult would it be to roll out mobile payments to stores in the rest of the State?	Some respondents were not sure whether this question would be relevant for store clerks and managers (in their stores).	If a respondent does not know the answer to a question, we will instruct interviewers to reiterate that questions may be skipped during the interview. Added interviewer notes throughout guide indicating what questions to ask of in-store and corporate staff.
	Question 29. What advice would you give other retailers that want to implement mobile payments?	Some respondents were not sure whether this question would be relevant for store clerks and managers (in their stores).	If a respondent does not know the answer to a question, we will instruct interviewers to reiterate that questions may be skipped during the interview. Added interviewer notes throughout guide indicating what questions to ask of in-store and corporate staff.

Respondent burden. Retailer staff suggested that the estimated duration of the interview would range from 15 to 30 minutes, depending on the level of engagement from the interviewee. (We did not ask IDHS staff to estimate the time for key informant interviews since the pre-test covered all questions in the guide but only selected questions will be asked in each interview round.)

III. Next Steps

The study team has revised the surveys, key informant interview guide, and retailer interview guide to reflect the changes described in this memo. We have had both revised surveys translated into Spanish. Finalized surveys and interview guides will be included in the final OMB package.