State Agencies, Grantees and SFA

SEBTC Evaluation

July 2011 Site Visits

School Food Authority Interview

INTRODUCTION

My name is [X], from [Abt Associates/Mathematica Policy Research].

As you may know, Abt and Mathematica are working together to evaluate the Summer EBT for Children demonstrations for the Food and Nutrition Service of USDA. My colleagues and I are visiting each of the five sites and collecting information from a wide range of stakeholders involved in the demonstration in order to learn about the process of implementation, the challenges you encountered, and lessons learned. I want to start by thanking you for taking time to speak with us today. Your perspective and insights on these issues are very helpful.

For stakeholders not receiving grant funds: Your participation is voluntary, and your responses will be kept confidential to the extent provided by law. You may refuse to answer any question and may stop the interview at any time.

Our reports to FNS will describe the range of responses expressed by staff, and may list the names of agencies and partners who contributed information, but we will not quote you or anyone by name or title. However, because of the relatively small number of organizations participating in the study, there is a possibility that a response could be correctly attributed to you.

I expect our conversation will take approximately 90 minutes.

[OPTIONAL IF INTERVIEWER CHOOSES TO RECORD:] I want to be sure I am keeping track of everything you are saying. May I record our discussion so that I can listen to it later when I write up my notes? No one outside of our research team will have access to the recording. [IF YES:] Thank you. It will be helpful if you speak up, speak clearly, and speak one at a time. [IF NO:] That’s no problem. I’ll take notes as you talk, but I may sometimes need to ask you to slow down or repeat so that I can get all the information.

First, do you have any questions for me about the project in general or what we will be discussing today?

A. BACKGROUND ON RESPONDENT

A.1 What SFA do you work for?

A.2 What is your position? What are your day-to-day responsibilities?

A.3 How long have you worked for [ORGANIZATION]?

A.4 When did you begin working on [SEBTC PROGRAM NAME]?

A.5 What are your specific responsibilities?

B. Local Context

Food environment.

B.1 Since the last site visit, has the food environment in your local area changed (such as the number of [SNAP/WIC] retailers, the types of retailers, or local access to those retailers)?

[If so, probe:]

• In what ways?

• For what reasons?

• Are more or fewer resources available to local households?

• What types?

Summer Food Service Program (SFSP).

B.2 Has there been any change this summer in the number or types of Summer Food Service Program sites available in your local area compared to summer 2010?

[If so, probe:]

• What changed?

• For what reasons?

• Are more or fewer summer feeding sites operating?

• Are more or fewer children being served?

• Are the hours, days, or weeks of operation different?

Other existing summer feeding options.

B.3 Since summer 2010, has there been any change in other summer feeding options (such as Seamless Summer Feeding Program, NSLP, kids' cafes, Child and Adult Care Food Programs) were available in your local area?

[If so, probe:]

• What changed?

• For what reasons?

• Are more or fewer summer feeding options for children?

• Are more or fewer children being served?

• Are the hours, days, or weeks of operation different?

Gaps in summer food services.

B.4 Do gaps exist this summer in the availability of summer food services for children? How large is the unmet need?

B.5 What are reasons for the gaps in services?

[Probe:]

• Limited funding

• Not enough sites

• Poor transportation to sites

• Not open the full summer

• Limited outreach

• Lack of awareness

• Stigma

• Limited hours or days of operation

• Parents concerns’ about neighborhood safety

• Lack of program activities

• Other reasons?

C. Basic parameters of the summer EBT MODEL

Level of EBT benefits.

C.1 Now that the benefits have been administered, what has been the reaction of parents/households to the $60 per child monthly benefit level specified by FNS?

[Probe:]

• Do they believe the level is too high, too low, or appropriate?

C.2 What do households/parents think of the fact that children of all ages are receiving the same amount of benefits and that multiple children in a household each receive the full benefit (no accounting for economies of scale)?

Mix of benefits in WIC sites.

C.3 Now that the benefits have been administered, what has been the reaction of parents to the WIC foods selected for the demonstration?

[Probe:]

• Overall mix of foods

• Split between food and cash voucher for fruits and vegetables

• Same types and amounts of food for children of all ages

D. Project Organization and Management

Organizational structure to administer the demonstration.

D.1 What are your and/or your SFAs responsibilities as they relate to [SEBTC PROGRAM NAME]?

D.2 Have your responsibilities changed since our last site visit?

[If so, probe:]

• How have your responsibilities changed?

• Why did they change?

Agreements between organizations.

D.3 Has the agreement or contracts between your SFA and [GRANTEE] changed since our last visit?

[If so, probe:]

• How?

• Why?

• Can we get copies of the new agreement?

Staffing structure.

D.4 How many staff members from the SFA are currently involved in demonstration activities?

D.5 What percentage of time do staff members dedicate to the demonstration?

D.6 Has there been staff turnover?

[If so, probe:]

• In what positions and for what reasons?

• What were the effects of this turnover?

D.7 Were any new staff members hired since the last visit for the demonstration? Which ones?

Communication between grantees and key partners.

D.8 How often and for what reasons do you communication with [GRANTEE]? What topics are discussed?

D.9 Have there been challenges to maintaining communication?

[If so, probe:]

• In what ways and for what areas could communications be improved?

D.10 What forms of communication and collaboration have been most helpful? In what ways?

Successes and challenges of the partnerships

D.11 What have been the most effective elements of your partnership with [GRANTEE], and why? What has worked best in this partnership?

D.12 What aspects of this partnership could be improved?

D.13 What have you learned about establishing and maintaining this partnership?

E. training and technical assistance

for schools AND OTHER COMMUNITY PARTNERS

Types and extent of training and TA.

E.1 Did you receive any training and technical assistance related to the demonstration since the last site visit?

[If not, ask:]

• What types of information or training would have been helpful?

[If so, ask the remainder of this section:]

E.3 What training and TA did you receive?

Experiences providing training.

E.4 What type of information or training did you receive (e.g., information materials distribution, one-on-one, group, computer)?

[If you received written information, probe:]

• What was the focus of the materials (e.g., the use of benefits and EBT cards, outreach and education to eligible households)?

• When did you receive them?

• Was the level of information sufficient to understand the program and complete your role?

[If you received training, ask the remaining questions in this section:]

E.5 What was the focus of the training (e.g., developing lists of eligible children, the use of benefits and EBT cards, outreach and education to eligible households)?

E.6 Who conducted the training?

E.7 How long was the training?

E.8 Was there any follow-up training provided?

E.9 Was the level of training sufficient to understand the program and complete your role in the demonstration?

E.10 Was the type of training appropriate?

E.11 Would you suggest conducting more/less of certain types?

E.12 What changes would you suggest the state makes to the training in the future (i.e., content, timing, length, location)?

F. OUTREACH AND Consent for the Demonstration
and Evaluation

F.1 Was the SFA involved in conducting outreach efforts or monitoring the consent process?

[If so, continue with this section.]

Outreach and recruitment efforts.

F.2 Since the last interview, were any additional outreach efforts conducted to inform parents/caregivers about the demonstration?

[If so, probe:]

• What methods were used (e.g., mail, flyers, radio, television)?

• What were the main messages in outreach materials?

• Can we get copies of these materials?

• How widely were materials and information distributed?

• Are they different for households enrolled in SNAP/WIC versus those that are not?

• What organizations were responsible for outreach?

• Were materials distributed in multiple languages and/or were media efforts targeted to subpopulations of non-English speaking households? If so, what languages?

Consent collection process.

[NOTE TO VISITOR, ASK FOLLOWING SECTIONS ON CONSENT IF THE CONSENT PROCESS WAS NOT YET COMPLETE AT THE TIME OF THE LAST SITE VISIT]

F.3 When did process of obtaining consent from households end in the spring?

F.4 Did the process go as expected? [If not,] how was it different than expected?

F.5 Did it take less or more time than expected?

Tracking and managing consent distribution and receipt.

F.6 Was the SFA involved in tracking and managing consent distributions?

F.7 If so, how did you track how and when letters went out and were sent back?

[Probe:]

• How were returned letters handled, if addresses are incorrect?

• How were with incomplete or missing information handled?

• How were the data to track the return of consent forms stored and cleaned?

• How was the data completed by parents/guardians on consent forms stored and cleaned?

• Who has access to the data?

Staff responsibilities related to consent.

F.8 Was the SFA involved in managing the production, distribution, and collection of consents?

F.9 Was the SFA responsible for tracking information about consent mailings and returns (such as the number of letters that have been mailed out, returned as undeliverable, positive responses [active consent only], refusal?

Consent return rates [IF ACTIVE CONSENT REQUIRED] for active sites.

[Confirm # of consent packages sent in the initial mailing.]

F.10 What percentage of households that were sent mailings has returned forms with complete information? What were common problems with how the forms were filled out?

F.11 What percentage returned forms with incomplete or missing information?

F.12 What percentage of letters was returned undeliverable?

F.13 [Ask if there was a second mailing for the consent process.] For the second mailing, what percentage of households returned forms with complete information? What were common problems with how the forms were filled out?

F.14 Approximately how many parents called with questions or concerns?

F.15 What percentage of households received multiple contacts?

Consent return rates [IF PASSIVE CONSENT REQUIRED].

F.16 [Confirm # of consent packages sent in the initial mailing.]

F.17 What percentage of households that were sent mailings opted out or responded that they did not want to be part of the demonstration?

F.18 Was there any effort to encourage these households to reconsider?

F.19 What percentage of letters was returned undeliverable? Was an attempt made to determine the correct address and then mail the consent forms again?

F.20 [Ask if there was a second mailing for the consent process], For the second mailing, what percentage of households returned forms with complete information? What were common problems with how the forms were filled out?

F.21 Approximately how many parents called with questions or concerns?

Maximizing consent response, [IF ACTIVE CONSENT REQUIRED].

F.22 Since the last visit, what did you do to maximize response rates from households?

Probe:]

• Did anyone try to contact nonresponsive households?

• Who tried to contact them and how (second letters, phone calls)?

• Were specific households targeted?

• Did you try to contact households more than once?

• What was the reaction of households that were reached?

F.23 For how many weeks did you attempt to contact nonresponsive households before finalizing the list of consenting households?

F.24 What challenges have you faced in encouraging parents to apply for the benefit? Did you overcome them and how?

F.25 What kind of technical assistance or support would have been most useful?

F.26 How successful were efforts to contact nonresponsive households?

F.27 What was the final number of consenting households?

Transfer of information.

F.28 How was contact information on consenting households transferred confidentially between the SFA, other leading agencies and the evaluation team?

F.29 What information was sent to whom?

F.30 What challenges did you encountered to this transfer of information?

F.31 What types of support would have helped overcome these challenges?

G. Informing households about Random Assignment Results AND ENROLLING THEM IN [SEBTC PROGRAM NAME]

[NOTE TO VISITOR, ASK FOLLOWING QUESTIONS IF NOTIFICATION PROCESS WAS NOT YET COMPLETE AT THE TIME OF THE LAST SITE VISIT]

G.1 Is the SFA involved in notifying households of their selection for the program? [If so, continue with this section.]

Informing households that were not selected for [SEBTC PROGRAM NAME].

G.2 Did you notify both families who were and were not selected to get the Summer EBT for Children benefit?

[If yes:]

• What information was provided?

• Were they given materials about other summer feeding options, [SNAP/WIC], nutrition education, or other topics? If so, what was provided? [GET COPIES OF MATERIALS]

G.3 What was parents’ response to these materials?

[Probe:]

• Did you receive any complaints about households not being selected to receive benefits?

• If so, how many?

• Who responded to these households and how?

Informing households that were selected for [SEBTC PROGRAM NAME].

G.4 We will ask more detailed questions about the materials later, but can you provide a brief description of the materials sent to households that were selected?

G.5 What was parents’ response to these materials?

[Probe:]

• Did you receive comments that households were pleased to be accepted? If so how many?

Enrolling households in the [SEBTC PROGRAM NAME] database.

G.6 Since the last site visit, were additional steps needed to officially enrolled households assigned to the demonstration in the program?

[Probe:]

• What database are you using?

• Why did you choose this database?

• Did you have to make any modifications to the database for this demonstration?

• What child or household information had to be entered?

• Who was responsible for entering this data?

• How long did it take?

G.7 Since the last visit, did you encounter any new challenges while enrolling households?

[If so, probe:]

• Can you describe them?

• Were you able to resolve these challenges? How?

• What would you do differently in the future if the program were continued?

SECTION H: training of households and other participant support

H.1 Was the SFA involved in training households or providing them with support once benefits were administered? [If so, continue with this section.]

Training for parents.

H.2 Beyond the information letter that was sent to parents about participating in the [SEBTC PROGRAM NAME] program, did parents in the demonstration sites receive training or additional information packets?

H.3 Did materials or training differ for certain populations (e.g., [SNAP/WIC] recipients versus non-recipients)?

Content of materials and training.

[Note to visitors: Please get copies of all materials.]

H.4 What is the focus of the materials or training?

[Probe:]

• Information about the [SEBTC PROGRAM NAME]

• What the EBT card is and how to use it (including activation)

• What types of foods are allowed, locations for accessing benefits

• Who should receive the benefit (children)

H.5 Do the materials or training provide broader information about eligibility for other programs (e.g., SNAP, WIC, summer food programs in the community)?

H.6 Do the materials or training address potential concerns about immigration status and benefit receipt?

H.7 Does the training include information on nutrition education (if so, we will talk in more detail about this later)?

Format of training.

H.8 How are materials or training being provided (e.g., information packets, one-on-one, classes, computer, web-based)?

H.9 Who prepared and distributed materials?

[Ask the following questions if there are group training sessions being conducted]

H.10 Who is conducting training?

H.11 Where are trainings being conducted?

H.12 How large are the classes?

H.13 How long do the training classes last?

H.14 How many training sessions will be held? And how often?

H.15 Are materials and training distributed in multiple languages? Which languages will be included?

H.16 Is follow-up training planned?

Partner agencies’ experiences with training.

H.17 Was the type of training appropriate?

H.18 Was the level of training appropriate? Would you suggest conducting more/less of certain types?

H.19 How much follow-up training/support was needed?

H.20 Overall, was more training needed/offered?

H.21 For classes, was the size and turnout about what you thought it would be?

H.22 Where were the classes located? Were those locations convenient for parents/guardians?

H.23 When were the classes held? Were the times convenient for parents/guardians?

H.24 How did the parents respond to the training?

H.25 Would you suggest making changes to the training (i.e., content, timing, length, location) for the next summer?

Participant support.

H.26 Is there a contact number for households to call if they have questions about the program, benefits, or card?

[Probe:]

• Is this contact number the same as [SNAP/WIC]?

• Who are they contacting (e.g., eligibility worker, CBO, vendor, automated)?

• What are the hours of operation for the customer service number?

• Have customer service representatives been trained on how to respond to questions?

H.27 What happens if a customer service representative cannot answer a question?

[Probe:]

• Is there a local contact number? Who answers that line?

• Is there any face-to-face support offered? Who provides the assistance?

H.28 Is there a website or webpage dedicated to [SEBTC PROGRAM NAME] information?

[If so:]

• How is access controlled?

Extent of participant support provided.

H.28 How often do households use the additional supports (account information, customer service, website)? Were you able to track this?

H.29 What are common questions from households?

[Probe:]

• Account balances

• Eligible foods

• Expiration of benefits

• Other issues?

H.30 Have there been any issues with the participant support process?

H.31 Did you receive any feedback on the effectiveness of the various types of media training? Do households seem to understand the process?

H.32 Would you make any changes to the process for the next summer?

I. Successes, Challenges, and Solutions

Perceived impacts of the demonstration.

I.1 To what extent do you believe the demonstration has affected children’s food security during the summer?

I.2 To what extent has it affected child nutrition during the summer?

I.3 To what extent has it affected household food security?

I.4 To what extent has it affected new applications for [SNAP or WIC]?

I.5 To what extent has it affected other summer feeding programs such as the SFSP, or emergency food services such as food banks and kitchens? Did they experience an increase or decrease in demand for food?

I.6 Are effects more apparent for certain groups of populations? [If so,] Which groups?

I.7 If [STATE] implemented services beyond the EBT benefit, to what extent does it appear that these services affected children’s outcomes?

I.8 What events during the summer outside of the demonstration may have affected its impacts?

Successes.

I.9 Thus far, what have been the greatest successes of the demonstration?

I.10 What factors contributed significantly to this success?

I.11 What, if anything, could have been done differently to achieve greater improvements in children’s food security?

Challenges.

I.12 Thus far, what have been the biggest challenges to meeting the goals of the demonstration?

I.13 Have these challenges been resolved?

• If so, how?

• If not, why?

I.14 What could the [GRANTEE] or other agencies have done differently to meet demonstration goals more effectively?

I.15 How have these challenges affected the effectiveness of the demonstration?

J. Feasibility of Continuing and Replicating DemonstrationS

Conditions needed to continue the demonstration.

J.1 What conditions would need to be maintained for the demonstration to continue successfully next year?

J.2 What program components or implementation practices would need to be maintained?

J.3 What changes, if any, would need to be made in order to operate successfully next year?

Conditions needed for replication in other sites.

J.4 What state and local conditions are critical to replicate the demonstration in other areas of the country?

J.5 What program components or implementation practices are essential for replication?

J.6 Are there practices you would advise other states and local areas to avoid?

[If so, probe:]

• What are they?

• Why would you suggest avoiding them?

J.7 What advice would you give to other states and local areas interested in replication of the demonstration?

K. Implementation and Operational Costs

Now I would like to turn to some questions related to the cost of implementing the demonstration.

Review of grant expenditures. [Review cost forms if SFA receiving grant funds]

K.1 Can you please briefly describe the responsibilities of each staff member listed?

K.2 What staff title applies to each person (for example, senior administrator, project manager, IT/MIS manager, IT/MIS developer, data entry, nutrition program specialist, administrative support)?

K.3 Can you please describe what is included in the other direct costs by category?

Start-up versus on-going costs. [Ask if SFA receiving grant funds]

K.4 What types of activities are covered by your start-up expenditures?

[Probe:]

• Over what period do they pertain?

• How did you determine what activities would be considered start-up expenditures?

• If the demonstration were to continue for another year, would any of these costs be repeated next spring? If so, which ones?

K.5 What types of activities were covered by on-going expenditures?

[Probe:]

• Over what period do they pertain?

• If the demonstration were to continue for another year, would all of these costs be repeated next spring and summer? If not, which will not be needed?

Sufficiency of funds. [Ask if SFA receiving grant funds]

K.6 Do you believe that your organization had sufficient funds through your grant to implement the [SEBTC PROGRAM NAME] benefits successfully?

[If not, probe:]

• How much additional funding did you need?

• What would you have used those funds for?

Accounting systems. [Ask if SFA receiving grant funds]

K.7 Did you need to adapt your accounting systems to meet the grant requirements or evaluation cost data collection needs?

[If so, probe:]

• Please describe how.

• Did these changes influence other aspects of your agency’s work?

• Will these revised procedures be used within the organization beyond the grant?

Tracking expenditures. [Ask if SFA receiving grant funds]

K.8 Did you experience any challenges in tracking grant expenditures over the course of the start-up period? If so, please describe.

K.9 Over the course of the summer? If so, please describe.

Deviations from grant budget. [Ask if SFA receiving grant funds]

[NOTE TO VISITORS: REVIEW BUDGETS AND EXPENDITURES BEFORE VISIT TO TAILOR QUESTIONS AND EMPHASIZE THIS IS NOT AN AUDIT.]

K.10 Were there variations between your approved grant budget and your actual spending?

[If yes, probe:]

• Across what categories?

• In what amounts?

• Can you explain how and why resources were not spent as expected?

Matched, volunteer and donated resources [Ask if SFA receiving grant funds].

K.11 Have any resources been used beyond the grant to support implementation of the grant?

[If so, probe:]

• What was the source and how much was used?

• How were these resources used?

K.12 Are volunteers involved in providing services?

[If so, probe:]

• Can you estimate how much volunteer time was spent?

• What was the role of these volunteers?

K.13 Were any other resources donated for the demonstration (e.g. office space, materials)?

[If so, probe:]

• What were they?

• Who donated them?

• How much was used?

Matched, volunteer and donated resources [Ask if SFA is NOT receiving grant funds].

K.14 What resources have been used to support implementation of the grant?

[Probe:]

• How much was used?

• How were these resources used?

K.15 Are volunteers involved in providing services?

[If so, probe:]

• Can you estimate how much volunteer time was spent?

• What was the role of these volunteers?

K.16 Were any other resources donated for the demonstration (e.g. office space, materials)?

[If so, probe:]

• What were they?

• Who donated them?

• How much was used?

Staff time spent on [SEBTC PROGRAM NAME]. [Ask of each respondent at the SFA]

K.17 What percentage of your time do you typically spend on the program?

[Probe:]

• Has this changed over time?

• How much of your time is charged to the grant?

• How much is paid by other funding sources?

K.18 Do you work overtime hours on the project?

[If so, probe:]

• How much is paid versus unpaid?

CLOSING

Is there anything you think is important for the FNS to know about the [STATE]’s [SEBTC PROGRAM NAME] demonstration that we did not ask about?

Thank you for your time and helpful feedback. The information you have shared will be valuable to our team as we look across states and localities for themes and ideas that we can share with FNS.

Are your any questions you have for me before we finish?

END OF INTERVIEW

16