C1. Partner Key Informant Interview Guide

Appendix C1

Partner Key Informant Interview Guide

This page has been left blank for double-sided copying.

SNAP Mobile Payment Pilots Evaluation

Key Informant Interviews: Master Discussion Guide

Notes to site visitors

Site visit teams will tailor this master protocol by identifying the roles of each respondent before the site visit and determining which of the following questions apply (and should be asked) of each. No respondent will be asked all of these questions. In addition, any information that may have been learned from document reviews or conversations with the State SNAP agency and partners during prior conversations should be documented and confirmed during the site visits. The questions in this master guide apply to all pilots, unless otherwise noted.

Respondents to be interviewed will include staff from: State SNAP agencies (pilot director and other staff) and partners, including, EBT processors, payment providers, and technology developers. Retailers will be interviewed using a separate guide.

We will conduct three rounds of site visits using the questions included below. The first round will focus on the planning phase of the pilot. The second round will examine early and ongoing implementation and operations for each pilot. The third round will focus on the close-out of the pilot and discuss overall impressions, lessons learned, and challenges encountered. We will tailor the questions in the protocol by round to ask respondents about events as they happen (or to reflect on what previously happened since the last visit) and to determine how their opinions or approaches changed over time.

Exhibit 1. Discussion guide topics by respondent type and site visit round

	State SNAP Agency Staff	Partners
		EBT Processorsa	Payment Providersb	Technology developersc
A. Background	All	All	All	All
B. Partnerships	1	NA	NA	NA
C. Staffing	All	All	All	All
D. Planningd	1	1	1	1
E. Early Implementation	2	2	2	2
F. Operationse	2	2	2	2
G. Late implementation and closeout	3	3	3	3
H. Costs	3	NA	NA	NA
I. Experiences and context	All	All	All	All
J. Lessons learned and scalability	3	3	3	3

^a EBT Processors produce the EBT mobile apps, which will be used to generate QR codes and/or will be a component of setting up NFC technologies for some pilots. [Add examples from pilot document review].

^bPayment providers include mobile wallet platforms such as Apple Pay, Google Pay, or Samsung Pay.

^cTechnology developers include any organization involved in creating technology to support the acceptance and processing of SNAP mobile payments. For example, token service providers and [add examples from pilot document review].

^dAll staff will be asked about planning activities, but only State SNAP agency staff will be asked about pilot design, activities with partners, and outreach and education preparation. All program integrity questions are for State SNAP agency staff with the exception of one question for partners.

^eAll staff will be asked about technical assistance, but only State SNAP agency staff will be asked about how SNAP participants would learn about, prepare to use, and use mobile payments, and program performance and integrity.

NA = not applicable.

Introduction

My name is [X], and I work for Mathematica, an independent research firm. As you might know, we are evaluating the Supplemental Nutrition Assistance Program (SNAP) Mobile Payment Pilot for the Food and Nutrition Service (FNS) of the U.S. Department of Agriculture. Pilots are being conducted in five States that allow SNAP participants to use mobile payments as an alternate option to a physical electronic benefit transfer (EBT) card. For the purposes of this study, we are interested in understanding how State SNAP agencies and partners designed and implemented the pilots, how SNAP participants adopted mobile payments, and what strategies were used to ensure program integrity.

My colleagues and I are currently visiting the five pilot States to collect information from a wide range of partners involved in the pilot. We are learning about the implementation of the pilot programs at three different stages, as well as the challenges you encountered and lessons learned during the implementation and operation of the pilot. Thank you for taking time to speak with us today. Your perspective and insights on these issues will be very helpful to the study.

Your participation in this interview is voluntary. We will never use your name in a report and all of our notes and recordings will be stored safely and destroyed at the end of the study. We will not share the information you provide outside of the study team, including your supervisor and state-level staff. You may refuse to answer any question and may stop the interview at any time. There will not be any penalities if you refuse to participate in part or full. There are no known risks to participating.

We will take notes over the course of the interview so that we can remember the information we collect. We will use this information in our reports to FNS, describing the range of responses expressed by staff. The reports might list the names of organizations that contributed information, but we will not quote you or anyone by name or title. I would also like to record our discussion so I can listen to it later when I write up my notes. No one outside the immediate research team at Mathematica will listen to the recording. We will destroy the recording after the study is complete. If you want to say something that you do not want recorded, please let me know and I will be glad to pause the recorder. Do you have any objections to being part of this interview or to my recording our discussion?

I expect our conversation will take about 60 minutes. First, do you have any questions for me about the project in general or what we will be discussing today?

Public Burden Statement

This information is being collected to assist the Food and Nutrition Service in evaluating the Supplemental Nutrition Assistance Program (SNAP) Mobile Payment Pilot (MPP). This is a voluntary collection and FNS will use the information to evaluate the implementation, adoption, and scalability of the SNAP MPP. This collection does request personally identifiable information under the Privacy Act of 1974. According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-XXXX. The time required to complete this information collection is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, 1320 Braddock Place, 5th Floor, Alexandria, VA 22314 ATTN: PRA (0584-XXXX). Do not return the completed form to this address.

Privacy Act Statement

Authority: This information is being collected under the authority of Section 4006(e) of the Agricultural Act of 2018. Disclosure of the information is voluntary.

Purpose: The information is being collected to evaluate the implementation, adoption, and scalability of Supplemental Nutrition Assistance Program (SNAP) Mobile Payment Pilot projects.

Routine Use: Information may be disclosed for any of the routine uses listed in the published System of Record notice titled FNS-8 USDA/FNS Studies and Reports published in the Federal Register April 25, 1991, Volume 56, Number 80 (pages 19078–19080).

Disclosure: Providing the information on this form is voluntary. There are no penalties for nonresponse.

1. Background (All rounds, unless known from prior round)

Respondents: All staff

1. What is your official job title or position?

2. What are your primary responsibilities? What are your responsibilities related to the pilot?

3. [For partners only] What services does your organization provide? What is your organization’s role in the pilot?

2. Partnerships (Round 1, select questions Round 3)

Respondents: State SNAP agency staff

4. What partner organizations (partners) are involved in the pilot?

Probes:

• What are each of their roles?

• How do the involved entities or their roles differ?

5. How and why were these partners selected?

Probes:

• Do these relationships predate the pilot or are they new?

• Did any partnerships fail to materialize? If so, why?

• Did any partners drop out of the pilot? If so, why?

• Were any added later in the pilot planning period? If so, why and when?

6. How did you recruit partners?

7. Are these formal or informal partnerships? For instance, are they contracted or non-contracted partners?

Probes:

• What kinds of partnership agreements are in place for the pilot?

• Does the State SNAP agency have data sharing agreements (or memoranda of understanding [MOUs]) in place with each partner and retailer? If not, why not? Has that caused any issues in cooperation?

8. What are the main strengths to the current organization and partnering structure?

9. Are there any weaknesses of the current organization and partnering structure? Describe.

10. [Round 3 only] Have any partnerships changed over time? How and why? Were any partnerships only needed for start up of the pilot, for example? (Round 3 only)

3. Staffing (All rounds, unless known from prior round)

Respondents: All staff

11. Who is leading the planning, development, and implementation of the pilot? What is their role?

12. Which staff within your organization work on the pilot?

13. What roles do key staff members play? Have these roles changed over time?

14. Were new staff hired specifically for the pilot [since the last interview]? If so, for which roles?

15. Do you feel additional staff are needed that were not hired or identified?

4. Planning (Round 1 only)

Respondents: All staff

Planning activities [All staff]

16. Please describe the planning process, including the types of activities that occurred during this phase.

Probes:

• Which organizations or staff were involved in the planning process?

• What were the roles of each organization?

• How did individuals and organizations communicate during the planning phase (telephone calls, in-person meetings, email, and so on)?

• How frequently did communication occur?

17. What was the timeline for planning activities?

18. What are your goals for the planning process and for the pilot?

19. What were the most critical decisions in the planning of the pilot? Why were these decisions critical?

Probes:

• When were these decisions made?

• Which individuals and organizations participated in making these decisions?

• Were the views of all organizations considered during the decision-making process? If not, describe why.

20. What activities did your organization undertake to prepare for new [program integrity/security] concerns introduced by mobile payment technologies?

21. Were the planning processes perceived as effective by your organization and other partners? If not, describe why. (For example, effective in terms of staffing, communication, resources, and timelines?)

22. What aspects of the planning process were most successful?

23. What challenges were encountered during the planning process?

Probes:

• Did the pilot encounter any delays? If so, why?

• What strategies were used to address these challenges?

24. What would you change about the planning process if you could?

Pilot design [State SNAP agency staff]

25. How did you identify the geographic area where you offered mobile payments?

26. How did you identify the retailers to partner with?

27. How did you identify the technology that you would use to accept and process mobile payments?

Probes:

• What considerations informed your decision?

• If the approach changed following the award, why did you make this change?

28. Please describe how a SNAP participant will use the mobile payment technology from the point of taking out their phone to pay.

Probes:

• NFC: Mobile wallet

• QR: Mobile application

• Staffed register versus a self-checkout

29. What would a SNAP participant need to do to prepare to use the mobile payment method before getting to the register?

30. What security measures will exist at each step?

31. Who in the SNAP household would be able to use the mobile payment method? How will you ensure users are authorized? If not everyone in the household, why is that the design?

32. How would a SNAP participant split payments between EBT and another form of payment, if needed for non-eligible items?

33. What changes were needed to create or update policies and practices to support the pilot?

Probes:

• How did you update policies to prepare for new program integrity concerns around mobile payment?

• How did you update eligibility policies to prepare for informing participants of mobile payments?

Program integrity [State SNAP agency staff, select questions for partners]

34. Did your agency/organization need to update or change any fraud or program integrity monitoring systems or indicators to plan for the availability of SNAP mobile payments?

Probes:

• Outlier detection techniques

• Other analytics to detect potential fraud

35. What staff were involved in this work? What are their roles?

36. What types of analyses or audits did the State agency perform before the pilot to help in defining pilot-specific metrics of fraud?

37. Did the State agency request analyses of fraud risks by partners (technology vendors, mobile application providers, etc.)? If so, how were those analyses conducted?

38. What technological approaches has the State agency considered for authenticating users, and how did you determine the fraud risks of those approaches?

39. Were differences in data security methods between available mobile payment systems a factor when the State agency was deciding which system to implement?

40. [Partners only] What policies and strategies has the State agency considered for safeguarding SNAP customers’ personal identifiable information at each stage in the process (i.e., registration, point of transaction, post-transaction, and storage) against being appropriated?

41. What fraud mitigation steps are you planning to incorporate into the testing and rollout of the integration of mobile payment technology into the SNAP EBT system?

Probes:

• Were discussions of fraud prioritized during specific steps of the planning period? What were those steps and what did discussions involve?

Activities with partners [State SNAP agency staff]

42. What steps did you take with your technology developer to prepare to offer mobile payments?

43. What plans did you make for testing the mobile payment method?

Probes:

• What activities does testing involve?

• Who is be involved?

• What plans do you have for user acceptance testing?

• What are the plans for ongoing troubleshooting/technical support after implementation launch?

44. [If State is using NFC] What steps did you take with your State’s EBT processing vendor to prepare to offer NFC? mobile payments?

Probes:

• Updates to the contract language

• Updates to processing specifications

• Other

45. [If State is using QR codes] What steps did you take with your State’s mobile application developer to prepare to offer QR code technology?

Probes:

• Updates to the contract language

• Updates to processing specifications

• Other

46. What steps did you take to conduct outreach to [Apple Pay/Google Pay/Master Card/ other payment processing vendor] to prepare to offer mobile payments?

47. What steps did you take to prepare participating retailers to offer mobile payments?

Probes:

• Did you provide training or other resources to retailers? If so, what information did that [training/resource] include?

• Did you offer training to retailers that included walking through the app as if they were a customer?

48. Were there key steps retailers needed to complete to prepare to accept EBT mobile payments? If so, what were those steps?

Probes:

• Training

• Point-of-sale system technology (and associated ‘bridge’ technology connecting the payment method to the point-of-sale system)

• Ongoing support resources

Outreach and education preparation [State SNAP agency staff]

49. How have you prepared to conduct outreach and education activities about mobile payment availability to SNAP households?

Probes:

• Outreach through retailers, partners, existing State SNAP agency channels

• Content of messaging (or examples they can share)

• Ongoing support resources

• Support for participants whose primary language is not English (limited English proficiency)

50. How do you ensure SNAP participants understand how to add the mobile payment method to their mobile devices?

51. How do you ensure SNAP participants understand which retailers in their area were participating?

52. What benefits do you communicate to SNAP participants about SNAP mobile payment?

Probes:

• Security/privacy

• Ease of use across household food shoppers

• Convenience

5. Early Implementation (Round 2 only)

Respondents: All staff

53. Please describe the process used to roll out the pilot.

Probes:

• What types of activities occurred during the early stages of implementation?

• Which activities occurred later in the pilot’s implementation?

• What was the timeline for these activities?

• Was the timeline followed as planned?

• Were any major changes needed after assessing early implementation activities?

• Did activities or timelines differ by location, payment method, or retailer?

• Did the implementation encounter any delays? If so, what were the reasons for those delays?

54. Which staff were responsible for oversight of the pilot roll-out and by what process?

55. How does the State SNAP agency communicate with partners and retailers?

Probes:

• Discuss the types of and reasons for communication.

• Have there been any communication challenges?

56. Were any changes made to the pilot design in the later stage of planning that affected implementation? If so, what changes and why were they needed?

57. Did staff receive training related to the pilot? If so, what information did the training include?

Probes:

• Who conducted the training?

• What was the content?

• How long was each training session?

• Do you believe the initial level of training was adequate? If not, please describe what additional training was needed.

• What ongoing training and support is available?

58. What worked well during the early implementation process?

59. What challenges were encountered during the early implementation process? What strategies were used to address these challenges?

6. Operations (Round 2 only)

Respondents: All staff (see sub-sections)

Pathways to using mobile payments [State SNAP agency staff]

60. After implementation, describe how a SNAP participant [in the pilot area] could learn about mobile payment methods and eligible retailers.

Probes:

• What is the content of messages about mobile payments?

• How are messages distributed? Where are messages located?

• Does this vary by the source of outreach?

61. What has worked particularly well related to outreach? Which methods appeared to have the best results?

62. What challenges has the pilot encountered related to outreach?

63. Did you receive any feedback from clients or the community on the outreach approaches used? Have strategies changed over time as a result of this? How?

64. How does the SNAP participant prepare to use mobile pay?

Probes:

• How do they set up the payment method on their phone? Please walk through each step of setting up the payment method. (For example, install an app, enter information, accept permissions, etc.)

• After a participant is certified for SNAP, how soon can they add mobile pay to their phone?

• If a participant wanted to set up the mobile payment method at the register, could they do so easily? How long would it take and how many steps would they have to complete for setup?

• Who in the SNAP household could use the mobile payment method?

• How would they share the payment method? How would it be limited to authorized users only?

65. How does a SNAP participant receive assistance if they have difficulty setting up the mobile payment method on their phone? How would they learn about the availability of that assistance?

66. What challenges has the pilot encountered related to adding mobile payments to phones?

67. What common barriers do SNAP participants experience in learning about mobile payment availability? In adopting mobile payment?

68. Do these barriers vary by any participant characteristics, such as age, address, or type of technology used (such as QR versus NFC)?

Use of mobile payment [State SNAP agency staff]

69. Please describe how a SNAP participant uses the mobile payment technology from the point of taking out their phone to pay.

70. What security measures exist at each step?

Probes:

• Multi-factor authentication, tokenization, pin use

71. How would a SNAP participant split payments between EBT and another card if needed for non-eligible items?

72. Would a SNAP participant likely experience any issues with switching between use of the physical EBT card and the mobile payment method interchangeably?

73. What would happen if the physical EBT card were lost? Would the participant be able to continue use of mobile payment?

74. What would happen if a SNAP participant attempted to use SNAP mobile pay at a non-participating retailer (Walmart, for example) that had mobile payment technologies available?

75. Are there any resources to assist a SNAP participant at the register if they are unable to use the mobile payment method?

Probes:

• What would happen if they did not have their EBT card with them?

• Educational resources

• Retailer assistance (cashier or manager)

76. What challenges has the pilot encountered related to mobile payment use at the register?

Probes:

• What are common barriers to using mobile payment at the register?

77. Have there been any changes to the EBT app interface to support mobile payment?

Probes:

• Changes to make it easier to set up mobile payment?

• Changes to make it easier for participants to track spending?

78. Are there any EBT app features a SNAP participant would be more likely to interact with as a result of using the mobile payment method?

79. If a SNAP participant has added mobile payment to their phone but has not used it, is there a way for the State SNAP agency to remind them of its availability? For example, a push notification within an app?

80. Do these barriers vary by any participant characteristics, such as age, address, or type of technology used (such as QR versus NFC)?

Technical assistance [All staff]

81. What type of technical assistance or support resources does [your agency/organization] offer for the ongoing implementation of mobile payments?

Probes:

• Hotline (telephone) or live chat (mobile/online)

• Online ticketing system

• Frequently asked question resources

• How do retailers notify the State SNAP agency?

• How do SNAP participants notify the State SNAP agency?

82. What steps does your agency/organization take if there is a technological problem with the processing or acceptance of mobile payments? How do you determine the source of the problem?

83. How do you communicate with other partners about technological issues?

84. How do you communicate with retailers about technological issues?

85. How do you communicate with SNAP participants about technological issues?

86. How often have technical assistance requests been submitted? By retailers? By clients? Does it vary by retailer, location, type of issue?

87. Will retailers get to test out the payment method so they know how it works/what the experience is like?

Program performance and integrity [State SNAP agency – management or data staff, EBT processors, mobile application developers, payment platforms]

88. What processes or systems are in place to measure aspects of pilot performance (for example, outreach, adoption, use, repeat use, technical assistance needs, and so on)?

89. How is pilot performance monitored?

Probes:

• Who tracks, collects, reports, and monitors the data (describe which staff participate in each)?

• How is the performance of each partner monitored?

90. How does the State SNAP agency monitor mobile payments for fraud or misuse? Does this vary from typical methods of monitoring for fraud or misuse?

Next are a series of questions about the data collected during each step of the set up and use of mobile payments.

91. Registration/setup: What data are collected during registration or setup of the mobile payment method?

• How are the data transmitted (encrypted, plain text, etc.)?

• Where are the data sent (kept on mobile devices, sent to the cloud, etc.)?

• Where are the data stored (mobile device, cloud, etc.)?

• How are the data stored (encrypted, not encrypted, etc.)?

• Are the data shared with any other entity? If they are shared, how does a third-party receive and store the data?

• What are the differences between data handling of Apple Pay, Google Pay, and Samsung Pay, etc.?

92. Transaction: What data are sent during a transaction (device ID, POS number, transaction ID, etc.)?

• How are the data transmitted (tokenized, encrypted, etc.)?

• Where do the data go (payment device only, payment device to the cloud, etc.)?

• Are any data stored, if so, how is it stored (unencrypted or encrypted, and if encrypted, in what manner)?

• What transaction data are stored? (Items bought, SKU numbers, transaction ID, transaction amount, date of transaction, posting date of transaction, device information used for transaction)

• Who has access to the data (State SNAP agency, partners, users, retailers, etc.)?

93. User interface: Can only one person register a SNAP account on a mobile device (one-to-one)?

• Can multiple family members register a single SNAP account on many devices (one-to-many)?

• Are account balance updates real-time?

• What is the hierarchy for account management (rights, rules, etc.)?

94. Fraud indicators: What approach was used to prevent fraudulent activity?

• What, if any, processes or indicators were implemented specifically to detect fraud conducted via mobile payments?

• Does the state monitor for spikes in purchases by individual customers compared to pre-pilot usage patterns?

• To what entities was identified fraud reported?

• What information was provided to support the cases that were reported?

• What actions were taken in response to the cases that were reported?

95. To what extent has the pilot met the program integrity goals it established during the planning phase?

7. Late implementation and closeout (Round 3 only)

Respondents: All staff

96. What types of activities occurred during the later stages of implementation?

Probes:

• What was the timeline for these activities? Was the timeline followed as planned?

• Were any major changes needed during [the period since we last met]?

• Did changes differ by location, payment method, or retailer?

97. Which staff were responsible for oversight of the pilot during [the period since we last met]?

98. How did they monitor the implementation?

Probes:

• Outreach and education

• Use

• Retailers

• Program integrity

99. Were there any changes to how you communicate with [the State SNAP agency/partners and retailers]? Discuss the types of and reasons for changes in communication. Have there been any communication challenges?

100. Were any changes made to the pilot design in the later stage of implementation? How did these changes differ from the earlier approach?

Probes:

• Mobile payment method (NFC/QR)

• EBT processing, mobile application, payment providers

• Retailers

• Outreach to SNAP participants

8. Costs (Round 3 only)

Respondents: State agency staff

101. The evaluation team has been collecting cost data about the pilot since you began the planning period. We would like to review a summary of the cost data the State agency has submitted during the pilot planning and implementation to date, including total costs, costs for planning versus implementation, and costs by type (such as direct labor versus indirect costs). Please review these numbers and assess whether they seem generally on target with your understanding of pilot costs for planning and implementation.

Probes:

• How did the State agency estimate costs for the planning and implementation of the pilot early on or during the proposal phase?

• Is the total actual cost more or less expensive than you expected or estimated?

• Were there unexpected costs?

• What cost more than anticipated? Less?

102. What costs would you expect to incur in scaling the pilot – to more retailers, or to statewide implementation?

103. Are there any costs that you expect would be significantly more pronounced if the pilots were scaled to more retailers or statewide? Why?

104. To what extent are you able to adjust the contracted budgets with partners based on needs for changes or technical assistance?

Probes:

• Vendors

• For example, EBT processing, mobile application provider, others?

105. What are the main sources of funding for the implementation of mobile pay?

Probes:

• Federal

• State

106. Is performance of your contracted partners tied to funding? If so, how do you monitor the performance of your contracted services providers? Please describe.

107. How often (for example, monthly or quarterly) do you receive and process invoices for services from your partners?

108. Have State budget concerns affected the implementation of the pilot (such as budget issues, hiring freezes, furloughs, or limiting overtime)? How was implementation affected? What strategies were used to minimize the effect of these challenges?

9. Experiences and context (All rounds)

Respondents: All staff

109. What has been the staff response to the pilot? How do you think the pilot went? What aspects of the pilot had a positive effect on service delivery? What aspects had a negative effect on service delivery?

110. Do you feel the pilot is meeting the goals identified during the planning phase? [Prefill with goals outlined during the planning phase interviews in Round 1, question 17.]

111. Were there concurrent policy or political changes that might have affected the mobile payment pilot? Please describe.

112. Are there any state or local factors in [STATE] that might be contributing to the pilot’s success or challenges?

113. Were there economic factors in [STATE] that might have affected the pilot? Please describe.

10. Lessons learned and scalability (Round 3 only)

Respondents: All staff

114. What would you do differently if you could implement the pilot again?

115. What are the three most important lessons you learned while planning the pilot? What about while implementing and operating the pilot?

116. What partnerships were successful for the implementation of the pilot? What partnerships were challenging? Why?

117. What conditions or resources would have to be maintained for the availability of mobile payments to continue successfully in the future?

118. What changes would have to be made to operate mobile payments successfully in the future?

119. Could this program be continued with the current level of funding allocated to this effort? Could you leverage other funds to continue?

120. [If not Statewide] How difficult would it be to roll out mobile payments to the rest of the State? How difficult would it be to add more retailers?

121. What advice would you give other States that want to implement mobile payments?