State Agencies, Grantees and SFA

OMB Clearance Number: _0584 -0559 Expiration Date: xx/xx/xxxx

Public reporting burden for this collection of information is estimated to average (60 minutes) per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: U.S. Department of Agriculture, Food and Nutrition Services, Office of Research and Analysis, 3101 Park Center Drive, Room 1014, Alexandria, VA 22302 ATTN: PRA (0584-0559*). Do not return the completed form to this address.

Evaluation of the SEBT for Children

Implementation Study Protocols

Government Entities

Telephone Interview Phone Calls

February 25, 2014

INTRODUCTION

My name is [X], from [Abt Associates/Mathematica Policy Research].

As you may know, Abt and Mathematica are working together to evaluate the Summer EBT for Children demonstrations for the Food and Nutrition Service of USDA. My colleagues and I are conducting telephone interviews with a wide range of stakeholders involved in the demonstration in order to learn about the process of implementation, the challenges you encountered, and lessons learned. I want to start by thanking you for taking time to speak with us today. Your perspective and insights on these issues are very helpful.

Our reports to FNS will describe the range of responses expressed by staff, and may list the names of agencies and partners who contributed information, but we will not quote you or anyone by name or title. However, because of the relatively small number of organizations participating in the study, there is a possibility that a response could be correctly attributed to you.

I expect our conversation will take approximately 60 minutes.

[OPTIONAL IF INTERVIEWER CHOOSES TO RECORD:] I want to be sure I am keeping track of everything you are saying. May I record our discussion so that I can listen to it later when I write up my notes? No one outside of our research team will have access to the recording. [IF YES:] Thank you. It will be helpful if you speak up, speak clearly, and speak one at a time. [IF NO:] That’s no problem. I’ll take notes as you talk, but I may sometimes need to ask you to slow down or repeat so that I can get all the information.

First, do you have any questions for me about the project in general or what we will be discussing today?

NOTE TO INTERVIEWERS: THIS GUIDE IS ORGANIZED ACCORDING TO TOPIC AREAS, TO BE ASKED OF SPECIFIC RESPONDENTS ACCORDING TO THE MASTER TOPIC LIST. HOWEVER, IT IS EXPECTED THAT SOME OF THE RESPONDENTS WILL NOT BE KNOWLEDGABLE ABOUT TOPICS ASSIGNED TO THEM, DEPENDING UPON THE SPECIFIC DEMONSTRATION. IF A RESPONDENT DOES NOT HAVE KNOWLEDGE OF THE TOPIC, SKIP TO THE NEXT TOPIC ASSIGNED TO HIM/HER.

SECTION C: Basic parameters of the summer EBT MODEL

C.1. Services beyond the EBT benefit. Since the last site visit, have there been any changes to the supplementary services offered to Summer EBT for Children households, such as nutrition education? If so, please explain. What has been the reaction of households?

SECTION D: Project Organization and Management

D.1. Organizational structure to administer the demonstration. Did the mix of organizations involved in the demonstration or the organizational structure change since the last interview? If so, how and why? Does this include both organizations that are receiving grant funds and those that are not? What were the main advantages to the organizational structure used in [STATE]? What aspect(s) would you recommend that other demonstration programs adopt? Why?

D.2. Communication between grantees and key partners. Thinking back over the course of the spring and summer, have there been any challenges to maintaining communication with key partner agencies or organizations? In what ways and for what areas could communications be improved? What forms of communication and collaboration have been most helpful? In what ways?

D.3. Changes in staffing and staff turnover. How has the staffing structure changed, if at all, since the last interview? How and why? If there were temporary staff hired for the demonstration, when were/will they be terminated? Has there been other staff turnover? In what positions and for what reasons? What were the effects of this turnover? Overall, where were the strengths and challenges of staffing for the demonstration?

SECTION E: Involving Local Agencies and Community Organizations

E.1. Successes and challenges of the partnership. Thinking about local agencies and community organizations that served as partners but did not receive grant funds, what were the most effective elements of the partnership, and why? What have you learned about establishing and maintaining these partnerships? What do you view as the key logistical considerations that other state grantees and their partners would need to consider, if they were to implement a similar type of partnership? What has worked best in these partnerships? What partnership aspects could be improved? From which partners would you have liked more involvement?

E.2. Relationship of Summer EBT for Children with SFSP. Overall, how would you describe the relationship between the agencies operating [SEBTC PROGRAM NAME] and the agencies operating the SFSP, at the State and local levels? In what ways did [SEBTC PROGRAM NAME] affect attitudes of parents and children toward the SFSP? Was the SFSP more or less successful in the project site than in previous years, and why?

SECTION F: Coordination with EBT Vendor

1. EBT system preparations. Since the last interview, have there been any changes to the vendor’s EBT system or third-party processor (TPP) systems for [SEBTC PROGRAM NAME]? Who was involved with this process? How long did this take? What activities took the most effort? What issues were encountered in completing implementation? What operational issues have been encountered? What changes are you considering for next year? What recommendations about vendor EBT system or TPP system design or operations do you have for other sites operating Summer EBT for Children?

2. Point of sale (POS) system preparations. Since the last interview, have there been any changes to the POS system programming or data loads for [SEBTC PROGRAM NAME]? Who was involved with this process? How long did this take? What activities took the most effort? What issues were encountered in completing implementation? What operational issues have been encountered? What changes are you considering for next year? What recommendations about POS system design or operations do you have for other sites operating Summer EBT for Children?

3. Automated customer service system preparations. Since the last interview, have there been any changes to the interactive voice response (IVR) system, customer service scripts, or user website for [SEBTC PROGRAM NAME]? Who was involved with this process? How long did this take? What activities took the most effort? What issues were encountered in completing implementation? How well do the automated customer service systems (interactive voice response (IVR) system, customer service scripts, user website) meet customer needs? What operational issues have been encountered? What changes are you considering for next year? What recommendations about automated customer service systems do you have for other sites operating Summer EBT for Children?

SECTION G: Identifying Eligible Children and Households

[NO QUESTIONS FROM THIS SECTION WILL BE ASKED DURING SEPTEMBER]

SECTION H: OUTREACH AND Consent for the Demonstration
and Evaluation

[NO QUESTIONS FROM THIS SECTION WILL BE ASKED DURING SEPTEMBER]

SECTION I: Informing households about Random Assignment Results AND ENROLLING THEM IN [SEBTC PROGRAM NAME]

I.1. Enrolling households in the [SEBTC PROGRAM NAME] database. Since the last interview, have there been any changes to the structure and processing of the household database for [SEBTC PROGRAM NAME]? Why were changes needed? Who was involved with this process? How long did this take? What activities took the most effort? What issues were encountered? What operational issues have been encountered? What changes are you considering for next year? What recommendations about tracking and managing household data do you have for other sites operating Summer EBT for Children?

I.2. Technical support to site operations. Have there been any changes in technical support for site-level operations, such as issuing cards and resolving problems requiring in-person contact? Since the last interview, what issues were encountered? What changes are you considering for next year? What recommendations about technical support for site operations do you have for other sites operating Summer EBT for Children?

SECTION K: replacing ebt cards and other participant support

I’d like to turn now to some operational questions about customer service for [SEBTC PROGRAM NAME] households, focusing on the time since our last interview.

K.1. Extent of card replacements. During the late summer months, how often did you have to replace EBT cards that were lost, stolen, or damaged? What were the most common reasons? Were there any issues with the process for replacing the cards? Did households seem to understand the process? How long did it take for households to receive replacement cards? Would you make any changes to the process for the next summer?

K.2. Other participant support provided. Were there any changes to participant supports? If so, how and why?

SECTION L: preventing and detecting abuse

L.1. Process for preventing and detecting abuse. Did the process work as expected? Were there any changes in late summer to the process for detecting and preventing abuse of [SEBTC PROGRAM NAME] funds? If so, how and why?

L.2. Extent of reported fraud and abuse. How many reports of fraud or abuse were received? What were the main types of reports or allegations? How many were substantiated? How many claims were established, and in what amount? How much was recovered? What other enforcement actions were taken against households, retailers, or others? What lessons did you learn from this process about assuring the integrity of Summer EBT for Children?

SECTION M: DEMONSTRATION PHASE-OUT

M.1. Process for expunging funds—SNAP approach. What process was used to expunge unused benefits cards? Was this similar to the process used for SNAP when the cards are inactive? When did this happen? Did the timeline change? If so, why? Is the process complete? If so, when did you complete it? If not, when do you anticipate finishing? What happened to unused funds? Did households receive any notification or reminders near the end of the summer that the benefits would be forfeited if not used? If so, how and when?

M.2. Expiration of benefits—WIC approach. Were there any changes to the rules for expiration of benefits? If so, how and why?

M.3. Deactivation of cards. Were EBT cards deactivated, and if so how was this done? Did households receive any notification or reminders near the end of the summer that the demonstration was ending and cards would become inactive? If so, how and when? Will households that participated in the summer keep the same card for use in future summers or will they be issued a new card?

M.4. Challenges with closing out SEBT for Children accounts. Did the process for closing out accounts work as anticipated? Were there any problems with the process? Did you have to change the process from what you planned? What were the responses of households? Is there anything you would do differently for the next summer?

SECTION N: training for schools AND OTHER COMMUNITY PARTNERS

[NO QUESTIONS FROM THIS SECTION WILL BE ASKED DURING SEPTEMBER]

SECTION O: training for parents on [SEBTC PROGRAM NAME] benefits

[NOTE TO VISITOR: ASK FOLLOWING QUESTIONS IF SITE HAD HOTLINE OR OTHER SUPPORTS FOR HOUSEHOLDS]

O.1. Extent of additional support provided to parents. How often did parents use the customer service number? How long were wait times (can you track)? Did questions arise from these calls that you would consider including in the overall parent training? How many parents used the website (can you track)? Have there been any issues with the process? What feedback did you get about participant support or the effectiveness of the various types of media/training? Were there any issues with the overall customer support process? Would you make any changes to the process for the next summer?

SECTION P: training and support for retailers

[NOTE TO VISITOR: ASK FOLLOWING QUESTIONS IF RETAILER TRAINING AND OTHER SUPPORT PROVIDED]

P.1. Extent of retailer training provided. Was the level of retailer training for the demonstration appropriate? How much follow-up training/support was needed? Overall, was more training needed/offered? Was the type of training appropriate? How did the retailers respond to the training? Did any issues arise that were not anticipated or addressed through training (stocking issues due to high demand, concerns/reactions from retailers about the new program, problems with cashiers recognizing and accepting cards)? Would you suggest making changes to the training (i.e., content, timing, length, location) for the next summer?

P.2. Extent of additional retailer support provided. How often did retailers use the customer service number for questions related to [SEBTC PROGRAM NAME]? Did questions arise from these calls that you would consider including in the overall retailer training? Were there any issues with the customer service process for retailers? Would you make any changes to the process for the next summer?

P.3. Retailer response to structure of training. Since the last interview, was any additional training for [SEBTC PROGRAM NAME] provided to the stores that you represent? If so, who provided it, and what type of training occurred (e.g., information materials distribution, one-on-one, group, computer)? What was the focus of the training (e.g., information about the [SEBTC PROGRAM NAME], the EBT look and use, the new WIC package)? How long was the training? How accessible and effective was it? Was there any follow-up to the training? If so, what and how?

P.4. Retailer response to usefulness of training. Overall, how well did training prepare stores for the demonstration, both inside the demonstration site and beyond? Did the training/materials cover all that retailers needed to know? Was the type of training appropriate? What changes would you suggest the state makes to the training in the future (i.e., content, timing, length, location)? Did store managers provide additional training to staff/cashiers? What type of training did they conduct? What information did they share? Were there any problems with staff understanding the program or allowing households to use their new cards? [For WIC sites] Did the special [SEBTC PROGRAM NAME] version of the WIC food package cause any problems for you or your systems? Was follow-up training needed during the demonstration period?

P.5. Retailer response to additional support. How often did stores need to contact the retailer customer service line with questions or problems related to [SEBTC PROGRAM NAME]? Was the level of service better then, the same as, or worse than for regular [SNAP/WIC] calls, in terms of wait times and quality of service? What recommendations do you have, if any, for improving retailer customer service for [SEBTC PROGRAM NAME]?

SECTION R: Successes, Challenges, and Solutions

R.1. Goals. Did [STATE] have to adjust its goals or expectations during the course of demonstration implementation? If so, what specifically changed? Why was this necessary? How was this accomplished?

R.2. Perceived impacts of the demonstration. To what extent do you believe the demonstration affected children’s food security during the summer? To what extent did it affect child nutrition during the summer? To what extent did it affect household food security? To what extent did it affect new applications for [SNAP or WIC]? To what extent did it affect applications for NSLP or SBP? To what extent did it affect other summer feeding programs such as the SFSP, or emergency food services such as food banks and kitchens? Did they experience an increase or decrease in demand for food? Are effects more apparent for certain groups of populations? If the [STATE] implemented services beyond the EBT benefit, to what extent does it appear that these services affected children’s outcomes?

R.3. Successes. What have been the greatest successes of the demonstration? What factors contributed significantly to this success? What, if anything, could have been done differently to achieve greater improvements in children’s food security during the summer?

R.4. Challenges. What have been the biggest challenges to meeting the goals of the demonstration? Have these challenges been resolved? If so, how? If not, why? What could [STATE] done different to meet its goals more effectively?

SECTION S: Feasibility of Continuing and Replicating DemonstrationS

S.1. Conditions needed to continue the demonstration. What conditions would need to be maintained for the demonstration to continue successfully next year? What program components or implementation practices would need to be maintained? What changes, if any, would need to be made in order to operate successfully next year?

S.2. Conditions needed for replication in other sites. What state and local conditions are critical to replicate the demonstration in other areas of the country? What program components or implementation practices are essential for replication? Are there practices you would advise other states and local areas to avoid? What advice would you give to other states and local areas interested in replication of the demonstration?

SECTION T: Implementation and Operational Costs

Now I would like to turn to some questions related to the cost of implementing the demonstration. I have copies of the updated grant expenditure data that you provided before the telephone call and would like to review them with you in detail.

T.1. Review of grant expenditures. [REVIEW COST FORMS WITH APPROPRIATE STAFF.] Before we get into the details, what is the timeline for finalizing grant expenditures? What kinds of changes, if any, do you expect from the most recent data? Please briefly describe the responsibilities of each staff member listed. What staff title applies to that person (for example, senior administrator, project manager, IT/MIS manager, IT/MIS developer, data entry, nutrition program specialist, administrative support)? Please describe what is included in the other direct costs by category.

T.2. Start-up versus on-going costs. What types of activities are covered by your start-up expenditures? Over what period do they pertain? How did you determine what activities would be considered start-up expenditures? If the demonstration were to continue for another year, would any of these costs be repeated next spring? If so, which ones? What types of activities were covered by on-going expenditures? Over what period do they pertain? If the demonstration were to continue for another year, would all of these costs be repeated next spring and summer? If not, which will not be needed?

T.3. Sufficiency of funds. Do you believe that your organization had sufficient funds through your grant to implement the [SEBTC PROGRAM NAME] benefits successfully? If not, how much additional funding did you need? What would you have used those funds for?

T.4. Deviations from grant budget. Since we last talked, were there variations between your approved grant budget and your actual spending? Across what categories? In what amounts? Can you explain how and why resources were not spent as expected? [NOTE: REVIEW BUDGETS AND EXPENDITURES BEFORE VISIT TO TAILOR QUESTIONS AND EMPHASIZE THIS IS NOT AN AUDIT.]

T.5. Matched, volunteer and donated resources [PARTNERS NOT RECEIVING GRANT FUNDS]. What resources were used to support implementation of the grant? How much was used? How were these resources used? Are volunteers involved in providing services? If so, can you estimate how much volunteer time was spent? What was the role of these volunteers? Were any other resources donated for the demonstration (e.g. office space, materials)? If so, what were they, who donated them, and how much was used?

T.6. Staff time spent on [SEBTC PROGRAM NAME]. [ASK OF EACH RESPONDENT AT GRANTEE AND SUBCONTRACTORS SEPARATELY] When did you begin working on [SEBTC PROGRAM NAME]? What are your specific responsibilities? What percentage of your time do you typically spend on the program? Has this changed over time? How did the percentage compare before the school year ended compared to after the school year ended? How much of your time is charged to the grant? How much is paid by other funding sources? Do you work overtime hours on the project? If so, how much is paid versus unpaid?

SECTION U: CLOSING

Is there anything you think is important for the FNS to know about the [STATE]’s [SEBTC PROGRAM NAME] demonstration that we did not ask about?

Thank you for your time and helpful feedback. The information you have shared will be valuable to our team as we look across states and localities for themes and ideas that we can share with FNS.

Are your any questions you have for me before we finish?

END OF INTERVIEW

5