State Agencies, Grantees and SFA

S

OMB No.: 0584-0559

Expiration Date: 3/31/2014

EBTC Evaluation

September 2011 Telephone Interviews

Interview with Grantee and Major Partners

[Note to site visitors:] Unless otherwise noted, all questions will be asked of a single respondent only. Site visitors will identify the roles of respondents during the site visit planning process and determine which respondent should address each major topic area. The site visit guide also indicates questions within each topic area that should be asked of more than one respondent.

INTRODUCTION

My name is [X], from [Abt Associates/Mathematica Policy Research].

As you may know, Abt and Mathematica are working together to evaluate the Summer EBT for Children demonstrations for the Food and Nutrition Service of USDA. My colleagues and I are visiting each of the five sites and collecting information from a wide range of stakeholders involved in the demonstration in order to learn about the process of implementation, the challenges you encountered, and lessons learned. I want to start by thanking you for taking time to speak with us today. Your perspective and insights on these issues are very helpful.

For stakeholders not receiving grant funds: Your participation is voluntary, and your responses will be kept confidential to the extent provided by law. You may refuse to answer any question and may stop the interview at any time.

Our reports to FNS will describe the range of responses expressed by staff, and may list the names of agencies and partners who contributed information, but we will not quote you or anyone by name or title. However, because of the relatively small number of organizations participating in the study, there is a possibility that a response could be correctly attributed to you.

I expect our conversation will take approximately [XX] minutes.

[OPTIONAL IF INTERVIEWER CHOOSES TO RECORD:] I want to be sure I am keeping track of everything you are saying. May I record our discussion so that I can listen to it later when I write up my notes? No one outside of our research team will have access to the recording. [IF YES:] Thank you. It will be helpful if you speak up, speak clearly, and speak one at a time. [IF NO:] That’s no problem. I’ll take notes as you talk, but I may sometimes need to ask you to slow down or repeat so that I can get all the information.

First, do you have any questions for me about the project in general or what we will be discussing today?

A. BACKGROUND ON RESPONDENT

[Ask this section of all new respondents]

A.1 What organization do you work for?

A.2 What services does [ORGANIZATION] provide?

A.3 What is your position? What are your day-to-day responsibilities?

A.4 How long have you worked for [ORGANIZATION]?

A.5 When did you begin working on [SEBTC PROGRAM NAME]?

A.6 What are your specific responsibilities?

B. Basic parameters of the summer EBT MODEL

Services beyond the EBT benefit.

B.1 Since the last site visit, have there been any changes to the supplementary services offered to Summer EBT for Children households, such as nutrition education?

[If so, probe:]

- What types of changes?

- For what reasons?

- What has been the reaction of households?

C. Project Organization and Management

Organizational structure to administer the demonstration.

C.1 Did the mix of organizations involved in the demonstration or the organizational structure change since the last interview?

[If so, probe:]

- How and why?

- Does this include both organizations that are receiving grant funds and those that are not?

C.2 What were the main advantages to the organizational structure used in [STATE]?

C.3 What aspect(s) would you recommend that other demonstration programs adopt? Why?

Communication between grantees and key partners. [Ask of grantee and all major partners.]

C.4 Thinking back over the course of the spring and summer, have there been any challenges to maintaining communication with key partner agencies or organizations?

C.5 In what ways and for what areas could communications be improved?

C.6 What forms of communication and collaboration have been most helpful? In what ways?

Changes in staffing and staff turnover. [Ask of grantee and all major partners.]

C.7 How has the staffing structure changed, if at all, since the last interview? How and why?

C.8 If there were temporary staff hired for the demonstration, when were/will they be terminated?

C.9 Has there been other staff turnover?

[If so, probe:]

- In what positions?

- For what reasons?

- What were the effects of this turnover?

C.10 Overall, where were the strengths and challenges of staffing for the demonstration?

D. Involving Local Agencies and Community Organizations

Successes and challenges of the collaboration.

D.1 Thinking about local agencies and community organizations that served as collaborating agencies but did not receive grant funds, what were the most effective elements of these collaborations, and why?

D.2 What have you learned about establishing and maintaining these collaborations?

D.3 What do you view as the key logistical considerations that other state grantees and their partners would need to consider, if they were to implement a similar type of collaboration?

D.4 What has worked best in these collaborations?

D.5 What collaboration aspects could be improved?

D.6 From which collaborators would you have liked more involvement?

D.7 Would you add additional or change collaborators in the following year?

Relationship of Summer EBT for Children with SFSP.

D.8 Overall, how would you describe the relationship between the agencies operating [SEBTC PROGRAM NAME] and the agencies operating the SFSP, at the State and local levels?

D.9 Did you learn that [SEBTC PROGRAM NAME] affected attitudes of parents and children toward the SFSP? In what ways?

D.10 Was the SFSP more or less successful in the project site than in previous years? In what ways? Why?

E. Coordination with EBT Vendor

EBT system preparations.

E.1 Since the last interview, have there been any changes to the vendor’s EBT system or third-party processor (TPP) systems for [SEBTC PROGRAM NAME]? Who was involved with this process?

[If so, probe:]

- How long did this take?

- What activities took the most effort?

- What issues were encountered in completing implementation?

- What operational issues have been encountered?

- What changes are you considering for next year?

- What recommendations about vendor EBT system or TPP system design or operations do you have for other sites operating Summer EBT for Children?

Point of sale (POS) system preparations.

E.2 Since the last interview, what steps in the implementation of changes to the POS system programming or data loads for [SEBTC PROGRAM NAME] took place?

[Probe:]

- Who was involved with this process?

- How long did this take?

- What activities took the most effort?

- What issues were encountered in completing implementation?

- What operational issues have been encountered?

- What changes are you considering for next year?

- What recommendations about POS system design or operations do you have for other states operating Summer EBT for Children?

Automated customer service system preparations.

E.3 Since the last interview, what steps in the implementation of changes to the interactive voice response (IVR) system, customer service scripts, or user website needed to implement for [SEBTC PROGRAM NAME] took place?

[Probe:]

- Who was involved with this process?

- How long did this take?

- What activities took the most effort?

- What issues were encountered in completing implementation?

- How well do the automated customer service systems (interactive voice response (IVR) system, customer service scripts, user website) meet customer needs?

- What operational issues have been encountered?

- What changes are you considering for next year?

- What recommendations about automated customer service systems do you have for other sites operating Summer EBT for Children?

F. replacing ebt cards and other participant support

I’d like to turn now to some operational questions about customer service for [SEBTC PROGRAM NAME] households, focusing on the time since our last interview.

Extent of card replacements.

F.1 During the late summer months, how often did you have to replace EBT cards that were lost, stolen, or damaged?

F.2 What were the most common reasons?

F.3 Were there any issues with the process for replacing the cards?

F.4 Did households seem to understand the process?

F.5 How long did it take for households to receive replacement cards?

F.6 Would you make any changes to the process for the next summer?

Other participant support provided.

F.7 Were there any changes to participant supports (such as 800 numbers, local information lines, a website)? If so, how and why?

G. preventing and detecting abuse

Process for preventing and detecting abuse.

G.1 Did the process for preventing and detecting abuse work as expected?

[If not, probe:]

- What appears to have failed?

- For what reasons?

- What changes would you make to the process for the next summer?

G.2 Were there any changes in late summer to the process for detecting and preventing abuse of [SEBTC PROGRAM NAME] funds?

[If so, probe:]

- What types of changes?

- For what reasons?

Extent of reported fraud and abuse.

G.3 How many reports of fraud or abuse were received?

[Probe:]

- What were the main types of reports or allegations?

- How many were substantiated?

- How many claims were established, and in what amount?

- How much was recovered?

- What other enforcement actions were taken against households, retailers, or others?

G.4 What lessons did you learn from this process about assuring the integrity of Summer EBT for Children?

H. DEMONSTRATION PHASE-OUT

Process for expunging funds—SNAP approach.

H.1 What process is being or was used to expunge unused benefits cards?

[Probe:]

- Was this similar to the process used for SNAP when the cards are inactive?

- When did this happen?

- Did the timeline change? If so, why?

H.2 Is the process complete?

[If so] When did you complete it?

[If not] When do you anticipate finishing?

H.3 What happened to unused funds?

H.4 Did households receive any notification or reminders near the end of the summer that the benefits would be forfeited if not used?

[If so, probe:]

- How were they informed?

- When?

- What was the reaction of households?

Expiration of benefits—WIC approach.

H.5 Were there any changes to the rules for expiration of benefits?

[If so, probe:]

- How?

- Why?

Deactivation of cards.

H.6 Were EBT cards deactivated?

[If so, probe:]

- How was this done?

- When was it done?

- What was the reason for deactivating cards?

H.7 Did households receive any notification or reminders near the end of the summer that the demonstration was ending and cards would become inactive?

[If so, probe:]

- How were they informed?

- When?

- What was the reaction of households?

H.8 Will households that participated in the summer keep the same card for use in future summers or will they be issued a new card?

Challenges with closing out SEBT for Children accounts.

H.9 Did the process for closing out accounts work as anticipated?

[If not, probe:]

- What did not go as planned?

- What were the problems with the process?

- Were they resolved? How?

H.10 What were the responses of households?

[Probe:]

- Did you encounter any households that were unaware that their benefits were ending?

- How many?

- Did households raise any other issues?

H.11 Is there anything you would do differently for the next summer?

I. training for parents on [SEBTC PROGRAM NAME] benefits

[NOTE TO VISITOR: ASK FOLLOWING QUESTIONS IF SITE HAD HOTLINE OR OTHER SUPPORTS FOR HOUSEHOLDS]

Extent of additional support provided to parents.

I.1 How often did households use your customer support systems (account information, customer service number, website)? Were you able to track this?

I.2 What are common questions from households?

[Probe:]

- Account balances

- Eligible foods

- Expiration of benefits

- Other issues

I.3 Have there been any issues with the participant support process?

I.4 Did you receive any feedback on the effectiveness of the various types of media training? Do households seem to understand the process?

I.5 Would you make any changes to the process for the next summer?

J. training and support for retailers

[NOTE TO VISITOR: ASK FOLLOWING QUESTIONS IF RETAILER TRAINING AND OTHER SUPPORT PROVIDED]

Extent of retailer training provided.

J.1 Was the level and type of retailer training provided through the demonstration appropriate?

J.2 How much follow-up training/support was needed?

J.3 Overall, was more training needed/offered?

J.4 How did the retailers respond to the training?

J.5 Did any issues arise that were not anticipated or addressed through training?

[Probe:]

- Stocking issues due to high demand

- Concerns/reactions from retailers about the new program

- Problems with cashiers recognizing and accepting cards

- Other issues?

J.6 Would you suggest making changes to the training (i.e., content, timing, length, location) for the next summer?

Extent of additional retailer support provided.

J.7 How often did retailers use the customer service number? Were you able to track this?

J.8 What were the common issues they raised?

J.9 Did questions arise from these calls that you would consider including in the retailer training?

J.10 Were there any issues with the customer service process?

J.11 Would you make any changes to the process for the next summer?

K. Successes, Challenges, and Solutions

Goals.

K.1 Did [STATE] have to adjust its goals or expectations during the course of demonstration implementation?

[If so, probe:]

- What specifically changed?

- Why was this necessary?

- How was this accomplished?

Perceived impacts of the demonstration. [Ask of grantee and all major partners.]

K.2 To what extent do you believe the demonstration has affected children’s food security during the summer?

K.3 To what extent has it affected child nutrition during the summer?

K.4 To what extent has it affected household food security?

K.5 To what extent has it affected new applications for [SNAP or WIC]?

K.6 To what extent has it affected other summer feeding programs such as the SFSP, or emergency food services such as food banks and kitchens? Did they experience an increase or decrease in demand for food?

K.7 Are effects more apparent for certain groups of populations? [If so,] Which groups?

K.8 If [STATE] implemented services beyond the EBT benefit, to what extent does it appear that these services affected children’s outcomes?

K.9 What events during the summer outside of the demonstration may have affected its impacts?

Successes. [Ask of grantee and all major partners.]

K.10 What have been the greatest successes of the demonstration?

K.11 What factors contributed significantly to this success?

K.12 What, if anything, could have been done differently to achieve greater improvements in children’s food security?

Challenges. [Ask of grantee and all major partners.]

K.13 What have been the biggest challenges to meeting the goals of the demonstration?

K.14 Have these challenges been resolved?

- If so, how?

- If not, why?

K.15 How have these challenges affected the effectiveness of the demonstration?

K.16 What could the [GRANTEE] or other agencies have done differently to meet demonstration goals more effectively?

L. Feasibility of Continuing and Replicating DemonstrationS

Conditions needed to continue the demonstration. [Ask of grantee and all major partners.]

L.1 What conditions would need to be maintained for the demonstration to continue successfully next year?

L.2 What program components or implementation practices would need to be maintained?

L.3 What changes, if any, would need to be made in order to operate successfully next year?

Conditions needed for replication in other sites. [Ask of grantee and all major partners.]

L.4 What state and local conditions are critical to replicate the demonstration in other areas of the country?

L.5 What program components or implementation practices are essential for replication?

L.6 Are there practices you would advise other states and local areas to avoid?

[If so, probe:]

- What are they?

- Why would you suggest avoiding them?

L.7 What advice would you give to other states and local areas interested in replication of the demonstration?

M. Implementation and Operational Costs

Now I would like to turn to some questions related to the cost of implementing the demonstration. I have copies of the updated grant expenditure data that you provided before the telephone call and would like to review them with you in detail.

Review of grant expenditures. [Review cost forms with appropriate staff at the grantee and each major partner receiving grant funds.]

M.1 Before we get into the details, what is the timeline for finalizing grant expenditures?

M.2 What kinds of changes, if any, do you expect from the most recent data?

M.3 Can you please briefly describe the responsibilities of each staff member listed?

M.4 What staff title applies to each person (for example, senior administrator, project manager, IT/MIS manager, IT/MIS developer, data entry, nutrition program specialist, administrative support)?

M.5 Can you please describe what is included in the other direct costs by category?

T.2. Start-up versus on-going costs. [Ask grantee and each major partner receiving grant funds]

M.6 Since the last visit, have any additional activities been considered start-up expenditures?

[Probe:]

- What types of activities?

- Over what period do they pertain?

- If the demonstration were to continue for another year, would any of these costs be repeated next spring? If so, which ones?

M.7 Since the last visit, what types of activities were covered by on-going expenditures?

[Probe:]

- Over what period do they pertain?

- If the demonstration were to continue for another year, would all of these costs be repeated next spring and summer? If not, which will not be needed?

Sufficiency of funds. [Ask grantee and each major partner receiving grant funds]

M.8 Do you believe that your organization had sufficient funds through your grant to implement the [SEBTC PROGRAM NAME] benefits successfully?

[If not, probe:]

- How much additional funding did you need?

- What would you have used those funds for?

Deviations from grant budget. [Ask grantee and each major partner receiving grant funds]

M.9 Since we last talked, were there variations between your approved grant budget and your actual spending?

[If yes, probe:]

- Across what categories?

- In what amounts?

- Can you explain how and why resources were not spent as expected?

Matched, volunteer and donated resources [Ask grantee and each major partner receiving grant funds].

M.10 Since the last site visit, have any resources been used beyond the grant to support implementation of the grant?

[If so, probe:]

• What was the source and how much was used?

• How were these resources used?

M.11 Since the last site visit, have volunteers been involved in providing services?

[If so, probe:]

• Can you estimate how much volunteer time was spent?

• What was the role of these volunteers?

M.12 Since the last site visit, were any other resources donated for the demonstration (e.g. office space, materials)?

[If so, probe:]

• What were they?

• Who donated them?

• How much was used?

Matched, volunteer and donated resources [Ask of major partners not receiving grant funds].

M.13 Since the last site visit, what resources have been used to support implementation of the grant?

[Probe:]

• How much was used?

• How were these resources used?

M.14 Since the last site visit, have volunteers been involved in providing services?

[If so, probe:]

• Can you estimate how much volunteer time was spent?

• What was the role of these volunteers?

M.15 Since the last site visit, were any other resources donated for the demonstration (e.g. office space, materials)?

[If so, probe:]

• What were they?

• Who donated them?

• How much was used?

Staff time spent on [SEBTC PROGRAM NAME]. [Ask of each respondent at grantee and all major partners]

M.16 What percentage of your time do you typically spend on the program?

[Probe:]

• Has this changed over time?

• How much of your time is charged to the grant?

• How much is paid by other funding sources?

M.17 Do you work overtime hours on the project?

[If so, probe:]

• How much is paid versus unpaid?

CLOSING

Is there anything you think is important for the FNS to know about the [STATE]’s [SEBTC PROGRAM NAME] demonstration that we did not ask about?

Thank you for your time and helpful feedback. The information you have shared will be valuable to our team as we look across states and localities for themes and ideas that we can share with FNS.

Are your any questions you have for me before we finish?

END OF INTERVIEW

25