Evaluation of FY 2019 Summer Electronic Benefit Transfer (EBT) Demonstrations - State and Tribal Grantees and SFAs (1 of 3))

Attachment B.1: Grantee Interview Protocol

OMB Control No.: 0584-0524

Expiration Date: 12/31/2022

Evaluation of 2019-2022 Summer EBT Project

Interview with Grantees

[Note to site visitors:] Unless otherwise noted, all questions will be asked of a single respondent only.

INTRODUCTION

My name is [X], from Abt Associates.

As you may know, Abt is working to evaluate the 2019-2022 Summer EBT Grant Projects for the Food and Nutrition Service of USDA. My colleagues and I are contacting each of the four grantees and collecting information from a wide range of stakeholders involved in the project to learn about the process of implementation, the challenges you encountered, successes you’ve experienced and lessons learned.

I want to start by thanking you for taking time to speak with us today. Your perspective and insights on these issues are very helpful. Our reports to FNS will describe the range of responses expressed by staff, and may list the names of agencies and partners who contributed information, but we will not quote you or anyone by name or title. However, because of the relatively small number of organizations participating in the study, there is a possibility that a response could be correctly attributed to you. I expect our conversation will take approximately 90 minutes.

Do you agree to participate in the interview as described? Please respond with “yes” or “no.”

YES  Great, thank you

NO  Do you have questions about the evaluation or the interview that I can answer?

Permission to Record: In order to ensure that we fully and accurately capture the points raised during today’s discussion, we would like to digitally record this conversation. Please note that the interviews will remain private. There is a small risk of loss of privacy, but the research team has taken many steps to reduce this risk. Your identity and any information attributable to you will not be released to anyone outside of the research team except as otherwise required by law. The recording will NOT be shared with FNS, your school or school district, or anyone outside the research team, unless you give us permission to do so. We will only use the recording to ensure accuracy of the transcription. Any identifying information, such as your name or anyone else’s name that may be mentioned, will be omitted from the final transcript. No names will appear in the final report given to FNS. The recording of your interview will be deleted at the end of the study, after all data have been analyzed.

The Food and Nutrition Service (FNS) is conducting this study to obtain information about the experiences of the Summer Electronic Benefit Transfer (EBT) Project by grantees, retailers, school districts and participants.  Participation in this study is voluntary and the information collected will be used to understand how Summer EBT projects are implemented in varying settings. This study does not collect personally identifiable information. Information collected in this study will be kept private to the full extent permitted by law. According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number.  The valid OMB control number for this information collection is 0584-0524.  The time required to provide this information collection is estimated to average 90 minutes per response, including the time to review instructions, search existing data resources, gather and maintain the data needed, and complete and review the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to:  USDA/Food and Nutrition Service, Office of Policy Support, 1320 Braddock Place, 5th Floor, Alexandria, VA 22314 ATTN:  PRA (0584-0524).

Do we have your permission to participate in this interview?

Yes  Thank you. Do you have any questions before we begin?

No  Okay. Thank you for your time.

Do we have your permission to record this interview?

Yes  Thank you. Do you have any questions before we begin?

No  Okay. That is not a problem at all. We would like to ask for your patience, as we will need to take a little more time to note your responses by hand to ensure we accurately capture your insights. Do you have any questions before we begin?

First, do you have any questions for me about the project in general or what we will be discussing today?

A. BACKGROUND ON RESPONDENT

I’d like to start with just a question about your background and role in the project.

A.1 What are your responsibilities related to Summer EBT?

B. implementation PROCESSES, CHALLENGES, AND SUCCESSES

Now I’d like to ask a series of questions about different aspects of your approach to and experience with implementation and populations served.

Characteristics of service area and population

B.1 Based on your experience, can you describe your typical Summer EBT client? How does the typical client vary by service area?

B.2 [For Cherokee Nation lead with “At this point…”] Do you think the process for selecting the populations to serve was effective? Would you change anything moving forward?

B.3 [For Cherokee Nation lead with “At this point…”] When selecting populations to serve, what about the selection process worked and what did not work?

B.4 What do you wish had been considered when selecting service areas?

B.5 Who would need to be involved in selecting service areas if the project were a permanent Federal program administered at the state or tribal level? Are there certain agencies or partners that would need to be involved?

B.6 What areas and/or populations would you choose to serve if Summer EBT was a permanent program? How would your service area change? What areas would you choose if you could not operate statewide?

Impacts on Other Nutrition Assistance Programs

The next few questions focus on other summer nutrition assistance programs.

B.7 Prior to the implementation of Summer EBT, how would you describe a family’s typical experience in regards to access to food during the summer? In other words, if Summer EBT did not exist, how would families access food/what would they do? What have they done in the past?

B.8 What other types of nutrition assistance programs for children in the summer do you know of that are accessible in your Summer EBT service areas? Probe for: SFSP, SSO

B.9 How does access to these programs differ between customized service areas? Does one type of area have better access than the other?

B.10 Do you think there is any overlap in service areas or children served between Summer EBT and other summer nutrition assistance programs? If yes, how could Summer EBT be targeted to reduce overlap with other options for summer nutrition assistance?

B.11 If Summer EBT was permanently authorized, how best could it be operated to reduce overlap with other options for summer nutrition assistance?

B.12 If Summer EBT was a permanent program, what role do you envision for your organization/agency in planning and selecting service areas?

Challenges and Successes during Implementation

Now I’d like to ask questions about implementation challenges and successes.

B.13 We know that grantees can face many challenges when implementing Summer EBT. Please tell me about the challenges you encountered this summer in implementation. We are particularly interested in those that may have delayed project implementation.

Probe for the following topics if not raised by the respondent:

1. How about challenges leveraging or obtaining the resources required, such as personnel, partnerships, information technology, data, and others.

2. Challenges related to working with households?

3. Challenges that result in delays in implementation of your project?

4. Challenges related to your partnerships?

5. [For Cherokee Nation only.] Challenges specifically related to building a new program?

[For Cherokee Nation only.] – Otherwise Skip to B16

B.14 If you could go back to the time prior to implementation, what would you do differently?

B.14a What happened that was unanticipated and how did you deal with those unanticipated issues?

B.15 What worked well this year that you will implement again next year?

B.15a What did not work well that you will not include next year?

Skip to B18

B.16 How did you address these challenges? What suggestions do you have for ways to do things differently to make implementation smoother?

B.17 If Summer EBT was a permanent program, would these issues still pose a challenge? Which challenges would be resolved?

B.18 What new challenges would you would anticipate arising if Summer EBT was a permanent program?

B.19 If Summer EBT was a permanent program, how would you use your implementation resources differently?

B.20 What additional resources might you require if Summer EBT was a permanent program?

B.21 What would you say are your biggest successes with implementing Summer EBT? And what factors contributed to them?

Partnerships

The next few questions focus on partnerships.

B.22 Who have you partnered with to implement the Summer EBT project? What is the extent of their involvement?

Probe for:

1. Other state/local agencies

2. Vendors

3. Community-based organizations

4. Schools

5. Other partners

B.23 What challenges did you face when coordinating with these partners?

B.24 What successes did you have? What would you say contributed to those?

B.25 If Summer EBT was a permanent program would you change your partnerships and the role of any partners? Specifically: [Ask each sub-question below.]

1. How might that affect the agency/entity responsible for program administration?

2. How might it lead to partners operating differently?

3. Would you need to establish new partnerships? If so, with whom and what would their roles be?

B.26 Now I’d like you to consider what would be required to expand to a larger service area. What would you change with the structure of your project or how you operate Summer EBT? Specifically: [Ask each sub-question below.]

1. Would you change which agency or entity has primary responsibility for program administration?

2. How would you make changes within the agency/entity responsible for program administration?

3. How would your partners operate differently?

4. Which types of new partnerships would you need to establish? With whom and what would their roles be?

5. What new systems or additional resources would you need?

Identifying Eligible Children

Now I’d like to ask questions regarding your process for identifying children eligible to participate in Summer EBT.

B.27 What is your process for identifying children eligible for Summer EBT?

Probe for:

1. SFA involvement

2. Coordination between state and tribal departments to produce and verify lists of eligible children

3. Integration of IT systems

4. Mailing, emailing, texts

5. Other

B.28 Please describe the process of delivering benefits to children – from issuance to card delivery to card use.

B.29 If Summer EBT was a permanent program, how would these procedures change, if at all?

Probe for:

1. Staffing and IT changes to handle state/tribal wide program to determine child eligibility

2. Other/new systems considered/developed

3. Other agencies need to be involved – IT, HHS, multiple tribal organizations and agencies

Benefit Errors

Now I’d like to ask a few questions about errors that may occur with benefits. The study is seeking to learn from the work of the demonstration projects; please note we are asking to learn more and the study is not an audit of any kind. Benefit errors can occur when food benefits are delivered to and/or redeemed by ineligible individuals, when food benefits are not delivered to eligible individuals, or when an incorrect benefit amount has been issued to eligible individuals. We know that your agency, like all grantees, try to limit the occurrence of these errors, and we’re interested in learning about approaches to minimizing them.

B.30 What types of errors have you experienced with the project?

B.31 Which types of errors are more common?

B.32 Which types of errors are harder to manage?

B.33 What are things to consider that could result in benefit issuance, delivery or redemption errors?

Probe for:

1. New systems needed to process and communicate data

2. How systems interface

3. New relationships

4. Communications across different agencies/partners that have different systems, priorities, roles, language

B.34 What processes, if any, have been put in place to prevent or detect benefit issuance, delivery or redemption errors?

B.35 What are the challenges related to preventing these kinds of errors?

B.36 What recommendations would you offer for how to prevent these kinds of errors? Are there specific changes to systems or processes you would recommend?

B.37 Based on what your agency has learned regarding benefit issuance, delivery or redemption errors, what will change in future project years?

B.38 If Summer EBT became a permanent program, what would your agency do to promote a high level of integrity in Summer EBT?

Challenging-to-Serve Populations

These next few questions ask about populations that are challenging to serve. . [Note to site visitor: This may be when grantee brings in someone within in their agency with SFSP background.]

B.39 Which populations are especially challenging to serve through traditional summer meals programs (SFSP)?

Probe for:

1. Low income children living in areas that are not low income (i.e., not area eligible for SFSP)

2. Degree of rurality of families – rural and very rural families

3. Children who attend Community Eligibility Provision (CEP) schools

B.40 Which populations are especially challenging to serve through Summer EBT?

Probe for:

1. Degree of rurality of families – rural and very rural families

2. Children who attend Community Eligibility Provision (CEP) schools

3. Children who change households or whose households move and whose current contact is not readily available

B.41 Why is it challenging to serve these populations?

B.42 What are the greatest challenges in serving families in rural and very rural areas?

B.43 What has the project done to serve these challenging populations?

C. beneficiaries’ use of benefits

I’d like to ask a few questions about how your project administers benefits and what factors affect how participants use their benefits.

C.1 [Food Package model only] How do you customize food packages for Summer EBT participants?

Probe for:

1. How do you determine which foods to offer?

2. How do you use feedback from participants?

3. How do you use data on participants’ use of benefits?

4. How – and how often - do you adjust food packages?

C.2 [Ask only of grantees that have implemented Summer EBT during the previous summer] Do you think that participants use more of their benefits if they participate in Summer EBT during a previous summer? Why or why not?

[If the grantee uses the Debit Card model, continue to C.3. Otherwise, skip to C.4]

C.3 You said your Summer EBT benefits are administered through the SNAP program. Why did you decide to administer benefits through the SNAP Program?

C.4 Do you co-load Summer EBT benefits on cards with any other benefits? Which ones?

C.5 What are the benefits and challenges of co-loading Summer EBT with other benefits?

C.6 Do you rely on active or passive consent when enrolling households into Summer EBT? Why did you decide on an ACTIVE/PASSIVE consent model? What are some of the benefits and drawbacks of this consent model? How do you think benefit use differs under either the passive or active consent model?

C.7 When thinking about enrolling households in future years, are you considering any changes to ACTIVE/PASSIVE consent? If yes, why are you considering the change?

C.8 If Summer EBT was a permanent program, would you make the decision to use active or passive consent? Why?

C.9 How would you describe the customer service supports that you provide to participating households? How do you think these supports affect participants’ use of benefits?

C.10 What other factors might affect participants’ redemption and exhaustion of benefits?

Probe for:

1. Integration of stakeholders – particularly schools – at the local level

2. Proximity to retailers or other retailer-related factors

3. Accessibility issues for participants with disabilities (mobility)

4. Allowable foods in food package sites

5. Summer EBT co-loaded with other benefits

6. Other

C.11 Based on what your agency has learned about the use of benefits and feedback from stakeholders (partners, retailers, parents/caregivers), what would you like to change in future project years?

C.12 If Summer EBT became a permanent program, what would you suggest changing from what you described above?

D. Implementation Costs

Now I would like to turn to some questions related to the cost of implementing the project. For these questions, please answer to the best of your ability.

D.1 To date, how do project costs compare with the amounts you had proposed and budgeted? In which cost categories? Have you spent more than expected? Where have you spent less than expected?

What do you think accounts for these differences?

D.2 What unexpected costs, if any, have you encountered? How large were they? Again, I don’t need precise amounts.

D.3 Based on your experiences in the last year, how have you adjusted your budget for next year?

Probe for:

1. What costs did you add? What costs did you remove?

2. For what reasons?

D.4 What differences have you experienced, if any, between the costs to operate Summer EBT in customized service areas?

Probe for:

1. Rural vs. Very Rural?

2. Coordinating with partners?

3. Recruiting participants?

4. Mailing/shipping costs?

5. Other?

D.5 [For grantees that implemented in a previous summer] For service areas that did not exist in 2018, based on what you learned from existing service areas, what did you do differently this year in these new service areas in terms of costs?

D.6 If Summer EBT were a permanent program, how would you expect your costs to change?

Probe for:

1. Cost structure

2. Reductions due to economies of scale

3. Other

D.7 Assuming permanent authorization, how could you change your approach to implementation in terms of costs? What activities would you eliminate or avoid? Which activities could you change for cost-saving?

Probe for:

1. Change your partners

2. Investing in new systems or resources

3. Using In-kind resources – such as tribal offices or school spaces

4. Other

CLOSING

What else do you think is important for FNS to know about the [STATE/ITO]’s [SUMMER EBT PROJECT NAME] that we did not ask about?

Thank you for your time and helpful feedback. The information you have shared will be valuable to our team as we look across states and localities for themes and ideas that we can share with FNS.

What questions do you have for me before we finish?

END OF INTERVIEW