State Agencies, Grantees and SFA

S

OMB No.: 0584-0559

Expiration Date: 3/31/2014

EBTC Evaluation

July 2011 Site Visits

Interview with Grantee and Major Partners

[Note to site visitors:] Unless otherwise noted, all questions will be asked of a single respondent only. Site visitors will identify the roles of respondents during the site visit planning process and determine which respondent should address each major topic area. The site visit guide also indicates questions within each topic area that should be asked of more than one respondent.

INTRODUCTION

My name is [X], from [Abt Associates/Mathematica Policy Research].

As you may know, Abt and Mathematica are working together to evaluate the Summer EBT for Children demonstrations for the Food and Nutrition Service of USDA. My colleagues and I are visiting each of the five sites and collecting information from a wide range of stakeholders involved in the demonstration in order to learn about the process of implementation, the challenges you encountered, and lessons learned. I want to start by thanking you for taking time to speak with us today. Your perspective and insights on these issues are very helpful.

For stakeholders not receiving grant funds: Your participation is voluntary, and your responses will be kept confidential to the extent provided by law. You may refuse to answer any question and may stop the interview at any time.

Our reports to FNS will describe the range of responses expressed by staff, and may list the names of agencies and partners who contributed information, but we will not quote you or anyone by name or title. However, because of the relatively small number of organizations participating in the study, there is a possibility that a response could be correctly attributed to you.

I expect our conversation will take approximately [XX] minutes.

[OPTIONAL IF INTERVIEWER CHOOSES TO RECORD:] I want to be sure I am keeping track of everything you are saying. May I record our discussion so that I can listen to it later when I write up my notes? No one outside of our research team will have access to the recording. [IF YES:] Thank you. It will be helpful if you speak up, speak clearly, and speak one at a time. [IF NO:] That’s no problem. I’ll take notes as you talk, but I may sometimes need to ask you to slow down or repeat so that I can get all the information.

First, do you have any questions for me about the project in general or what we will be discussing today?

A. BACKGROUND ON RESPONDENT

[Ask this section of all new respondents]

A.1 What organization do you work for?

A.2 What services does [ORGANIZATION] provide?

A.3 What is your position? What are your day-to-day responsibilities?

A.4 How long have you worked for [ORGANIZATION]?

A.5 When did you begin working on [SEBTC PROGRAM NAME]?

A.6 What are your specific responsibilities?

b. Demonstration Planning

Goals of the demonstration.

B.1 Since the last site visit in [MONTH], have the goals of the demonstration changed?

[If so, probe:]

- How?

- For what reasons?

- What new goals do you hope to achieve?

- To what extent are the goals quantifiable?

- How will they be measured?

C. Local Context

Food environment.

C.1 Since the last site visit, has the food environment in the demonstration area changed (such as the number of [SNAP/WIC] retailers, the types of retailers, or local access to those retailers)?

[If so, probe:]

- In what ways?

- For what reasons?

- Are more or fewer resources available to local households?

- What types?

Summer Food Service Program (SFSP).

C.2 Has there been any change this summer in the number or types of Summer Food Service Program sites available in the demonstration area compared to summer 2010?

[If so, probe:]

- What changed?

- For what reasons?

- Are more or fewer summer feeding sites operating?

- Are more or fewer children being served?

- Are the hours, days, or weeks of operation different in any of the sites?

Other existing summer feeding options.

C.3 Since summer 2010, has there been any change in other summer feeding options (such as Seamless Summer Feeding Program, NSLP, kids' cafes, Child and Adult Care Food Programs) were available in the target communities?

[If so, probe:]

- What changed?

- For what reasons?

- Are more or fewer summer feeding options for children?

- Are more or fewer children being served?

- Are the hours, days, or weeks of operation different?

Gaps in summer food services.

C.4 Do gaps exist this summer in the availability of summer food services for children? How large is the unmet need?

C.5 What are reasons for the gaps in services?

[Probe:]

- Limited funding

- Not enough sites

- Poor transportation to sites

- Not open the full summer

- Limited outreach

- Lack of awareness

- Stigma

- Limited hours or days of operation

- Parents concerns’ about neighborhood safety

- Lack of program activities

- Other reasons?

D. Basic parameters of the summer EBT MODEL

EBT cards.

D.1 Since the last site visit, have there been changes to the approach for the distribution of EBT cards? If so, please explain.

Level of EBT benefits.

D.2 Now that the EBT cards have been sent to households, has there been a reaction to the $60 per child monthly benefit level specified by FNS? What has been their reaction?

[Probe:]

• Do they believe the level is too high, too low, or appropriate?

D.3 Have there been questions or concerns about the program from Households? If so, what have they been?

D.4 What do households/parents think of the fact that children of all ages are receiving the same amount of benefits and that multiple children in a household each receive the full benefit (no accounting for economies of scale)?

Mix of benefits in WIC sites.

D.5 Now that the benefits have been administered, what has been the reaction of parents to the WIC foods selected for the demonstration?

[Probe:]

-Overall mix of foods

-Split between food and cash voucher for fruits and vegetables

-Same types and amounts of food for children of all ages

Timeframe of EBT benefits.

D.6 Since the last site visit, have there been any changes to the schedule for issuing EBT benefits?

[If so, probe:]

- Can you explain the changes?

- Why were the changes made?

- What has been the reaction of households?

Services beyond the EBT benefit.

D.7 Since the last site visit, have there been any changes to the supplementary services offered to Summer EBT for Children households, such as nutrition education?

[If so, probe:]

- Can you describe those changes?

-Why were the changes made?

- What has been the reaction of households?

E. Project Organization and Management

Organizational structure to administer the demonstration.

E.1 Has the overall organizational structure for administering the demonstration changed since the last site visit?

[If so, probe:]

- Can you please describe how?

E.2 Are any new organizations participating?

[If so, probe:]

- What are their responsibilities?

- How were they selected?

E.3 Did any organizations drop out of the demonstration?

[If so, probe:]

- Why?

- Which organizations have taken over their responsibilities?

E.4 What are the main advantages to the current organizational structure?

[Probe:]

- What aspect(s) would you recommend that other demonstration programs adopt?

- Why?

Agreements between organizations.

E.5 Have agreements between [GRANTEE] and [EACH PARTICIPATING ORGANIZATION] changed since the last visit?

[If so, probe:]

- How

- Why?

- Can we have copies of the new agreements?

Staffing structure [Ask of grantee and all major partners.]

E.6 What is your current staffing structure for administering demonstration services?

[Probe:]

- How many staff members are involved?

- What roles do key staff members play? What type of backgrounds and qualifications do they have?

- What roles do support staff play? What type of backgrounds and qualifications do they have?

- Are there shared responsibilities across organizations? Who leads or coordinates them?

E.7 What percentage of time do staff members dedicate to the demonstration?

E.8 Has there been staff turnover?

[If so, probe:]

- In what positions and for what reasons?

- What were the effects of this turnover?

E.9 Were any new staff members hired since the last visit for the demonstration? Which ones?

Communication between grantees and key partners.

E.10 What levels of staff communicate across the lead agencies?

E.11 How often? For what reasons? What topics are discussed?

E.12 In what format are meetings held (large group meetings, conference calls, one-on-one with each partner)?

E.13 Have there been challenges to maintaining communication? In what ways and for what areas could communications be improved?

E.14 What forms of communication and collaboration have been most helpful? In what ways?

F. Involving Local Agencies and Community Organizations

Identification of collaborating agencies.

F.1 Has there been any change in the involvement of local agencies and community organizations that do not receive grant funds?

[Probe:]

- SFSP sites and other summer feeding programs

- Emergency food providers

- Hunger advocacy groups

- Other faith or community-based organizations

[If so, probe:]

- What changes were made?

- Why?

- At this stage, which partners play the most prominent roles in the demonstration?

Roles and responsibilities of collaborating agencies.

F.2 Have there been any changes to the roles and responsibilities of collaborating agencies involved at the time of the last interview?

[If so, probe:]

- Can you describe those changes?

F.3 If you have involved new collaborating agencies, what are the roles and responsibilities of each new partner for the demonstration?

[Probe:]

- Play advisory role in planning and decision making

- Volunteer resources and staff time for training and activities

- Provide complementary services to beneficiaries

- Provide outreach and referral

- Something else?

Collaborative agreements.

F.4 Has there been any change to the agreements made with participating organizations? If so, what changed? Why? [Can we get copies?]

Continued collaboration.

F.5 What levels of staff communicate across agencies/organizations? How often? For what reasons? In what way (large group meetings, conference calls, one-on-one with each partner)? Have there been challenges to maintaining collaboration? What forms of communication and collaboration have been most helpful?

Successes and challenges of the partnership.

F.6 What are the most effective elements of the collaboration, and why?

F.7 What have you learned about establishing and maintaining these collaborations?

F.8 What lessons would you share with other grantees?

F.9 What collaboration aspects could be improved?

F.10 From which collaborators would you have liked more involvement?

G. OUTREACH AND Consent for the Demonstration
and Evaluation

[NOTE TO VISITOR, ASK FOLLOWING QUESTIONS IF OUTREACH AND CONSENT PROCESS WAS NOT YET COMPLETE AT THE TIME OF THE LAST SITE VISIT]

Outreach and recruitment efforts.

G.1 Since the last interview, have there been any additional outreach efforts to inform parents/caregivers about the demonstration?

[If so, probe:]

- What methods were used (e.g., mail, flyers, radio, television)?

- What were the main messages in outreach materials?

- Can we get copies of these materials?

- How widely were materials and information distributed?

- Are they different for households enrolled in SNAP/WIC versus those that are not?

- What organizations were responsible for outreach?

- Were materials distributed in multiple languages and/or were media efforts targeted to subpopulations of non-English speaking households? If so, what languages?

Consent collection process.

[NOTE TO VISITOR, ASK FOLLOWING SECTIONS ON CONSENT IF THE CONSENT PROCESS WAS NOT YET COMPLETE AT THE TIME OF THE LAST SITE VISIT]

G.2 When did process of obtaining consent from households end in the spring?

G.3 Did the process go as expected? [If not,] how was it different than expected?

G.4 Did it take less or more time than expected?

Tracking and managing consent distribution and receipt.

G.5 Since the last visit, were there any changes to how you tracked when letters went out and were sent back?

[Probe:]

- How were returned letters handled, if addresses are incorrect?

- How were with incomplete or missing information handled?

- How were the data to track the return of consent forms stored and cleaned?

- How was the data completed by parents/guardians on consent forms stored and cleaned?

- Who has access to the data?

Staff responsibilities related to consent.

G.6 Since the last interview, have there been any changes to who had primary responsibility for managing the production, distribution, and collection of consents?

G.7 Since the last interview, have there been any changes to who was responsible for tracking information about consent mailings and returns (such as the number of letters that have been mailed out, returned as undeliverable,, positive responses [active consent only], refusal?

Consent return rates [IF ACTIVE CONSENT REQUIRED] for active sites

[Confirm # of consent packages sent in the initial mailing.]

G.8 What percentage of households that were sent mailings have returned forms with complete information? What were common problems with how the forms were filled out?

G.9 What percentage returned forms with incomplete or missing information?

G.10 What percentage of letters were returned undeliverable?

G.11 [Ask if there was a second mailing for the consent process], For the second mailing, what percentage of households returned forms with complete information? What were common problems with how the forms were filled out?

G.12 Approximately how many parents called with questions or concerns?

G.13 What percentage of households received multiple contacts?

Consent return rates [IF PASSIVE CONSENT REQUIRED]

[Confirm # of consent packages sent in the initial mailing.]

G.14 What percentage of households that were sent mailings opted out or responded that they did not want to be part of the demonstration?

G.15 Was there any effort to encourage these households to reconsider?

G.16 What percentage of letters were returned undeliverable? Was an attempt made to determine the correct address and then mail the consent forms again?

G.17 [Ask if there was a second mailing for the consent process], For the second mailing, what percentage of households returned forms with complete information? What were common problems with houw the forms were filled out?

G.18 Approximately how many parents called with questions or concerns?

Maximizing consent response, [IF ACTIVE CONSENT REQUIRED].

G.19 Since the last visit, what did you do to maximize response rates from households?

[Probe:]

- Did anyone try to contact nonresponsive households?

- Who tried to contact them and how (second letters, phone calls)?

- Were specific households targeted?

- Did you try to contact households more than once?

- What was the reaction of households that were reached?

G.20 For how many weeks did you attempt to contact nonresponsive households before finalizing the list of consenting households?

G.21 What challenges have you faced in encouraging parents to apply for the benefit? Did you overcome them and how?

G.22 What kind of technical assistance or support would have been most useful?

G.23 How successful were efforts to contact nonresponsive households?

G.24 What was the final number of consenting households?

G.25 How well did the transfor of information to the evaluator work? Were there challenges? What types of support would have helped overcome these challenges?

H. Informing households about Random Assignment Results

AND ENROLLING THEM IN [SEBTC PROGRAM NAME]

[NOTE TO VISITOR, ASK FOLLOWING QUESTIONS IF NOTIFICATION PROCESS WAS NOT YET COMPLETE AT THE TIME OF THE LAST SITE VISIT]

Informing households that were not selected for [SEBTC PROGRAM NAME].

I.1 Did you notify both families who were and were not selected to get the Summer EBT for Children benefit?

[If yes:]

- What information was provided?

- Were they given materials about other summer feeding options, [SNAP/WIC], nutrition education, or other topics? If so, what was provided? [GET COPIES OF MATERIALS]

I.2 What was parents’ response to these materials?

[Probe:]

- Did you receive any complaints about households not being selected to receive benefits?

- If so, how many?

- Who responded to these households and how?

Informing households that were selected for [SEBTC PROGRAM NAME].

I.3 We will ask more detailed questions about the materials later, but can you provide a brief description of the materials sent to households that were selected?

I.4 What was parents’ response to these materials?

[Probe:] Did you receive comments that households were pleased to be accepted? If so how many?

Enrolling households in the [SEBTC PROGRAM NAME] database.

I.5 Since the last site visit, were additional steps needed to officially enrolled households assigned to the demonstration in the program?

[Probe:]

- Did you have to make any modifications to the database for this demonstration?

- What child or household information had to be entered?

- Who was responsible for entering this data?

- How long did it take?

I.6 Since the last visit, did you encounter any new challenges while enrolling households?

[If so, probe:]

- Can you describe them?

- Were you able to resolve these challenges? How?

- What would you do differently in the future if the program were continued?

J. COORDINATION WITH EBT VENDOR

EBT system preparations.

J.1 Since the last interview, what steps in the implementation of changes to the vendor’s EBT system or third-party processor (TPP) systems for [SEBTC PROGRAM NAME] have taken place?

[Probe:]

- Who was involved with this process?

- How long did this take?

- What activities took the most effort?

- What issues were encountered in completing implementation?

- What operational issues have been encountered?

- What changes are you considering for next year?

EBT system interfaces.

J.2 Since the last interview, what changes were made to system interfaces with the [SEBTC PROGRAM NAME] eligibility or certification system?

[Probe:]

- What changes were made to the interfaces with FNS databases?

- Who was involved in this process?

- How long did this take?

- What activities took the most effort?

- What issues were encountered?

- What changes are you considering for next year?

EBT system reconciliation.

J.3 Since the last interview, what changes were made to the settlement and reconciliation of the [SEBTC PROGRAM NAME] EBT system?

[Probe:]

- Who was involved in this process?

- How long did it take?

- What activities took the most effort?

- What issues were encountered?

- What changes are you considering for next year?

EBT retailer coordination.

J.4 Since the last interview, what changes were made to your agreements with EBT retailers and third party processors?

[Probe:]

- Who was involved in this process?

- How long did it take?

- What activities took the most effort?

- What issues were encountered?

- What changes are you considering for next year?

Point of sale (POS) system preparations.

J.5 Since the last interview, what steps in the implementation of changes to the POS system programming or data loads for [SEBTC PROGRAM NAME] took place?

[Probe:]

- If programming or data loads were required, were you able to do this remotely?

- Who was involved with this process?

- How long did this take?

- What activities took the most effort?

- What issues were encountered in completing implementation?

- What operational issues have been encountered?

- What changes are you considering for next year?

Automated customer service system preparations.

J.6 Since the last interview, what steps in the implementation of changes to the interactive voice response (IVR) system, customer service scripts, or user website needed to implement for [SEBTC PROGRAM NAME] took place?

[Probe:]

- Who was involved with this process?

- How long did this take?

- What activities took the most effort?

- What issues were encountered in completing implementation?

- How well do the automated customer service systems (interactive voice response (IVR) system, customer service scripts, user website) meet customer needs?

- What operational issues have been encountered?

- What changes are you considering for next year?

K. distributing ebt cards¹

EBT card distribution.

K.1 Were new EBT cards distributed according to plan (method, time frame)?

[If not, probe:] What deviations were there and why?

Additional materials sent with EBT cards.

K.2 Did the distribution of materials with EBT cards proceed according to plan? [If not, probe:] What deviation was there in content or media, and why? [GET COPIES]

K.3 What was the participant response to the materials (questions, concerns, kudos)?

Challenges with EBT card distribution.

K.4 Were there any problems with the method for distributing the EBT cards to households?

K.5 Were there problems with the cards not reaching the families (due to lack of contact information, families moved, incorrect addresses, etc.)?

[If so, probe:]

- How often did this occur?

- How was returned mail handled (due to incorrect addresses, moves, etc.)?

- How many households with returned mail subsequently got cards?

- If cards had to be picked up, what was the no-show rate?

K.6 Were any changes in the process needed to address the distribution issues?

K.7 Did the changes resolve the problems?

K.8 Would you suggest changes to the process for next summer?

Challenges with EBT card activation.

K.9 Were there any problems with households’ ability to activate new EBT cards?

[If so, probe:]

- What types of problems?

- How common were they?

K.10 Were there any problems with loading additional benefits onto existing EBT cards?

[If so, probe:]

- What types of problems?

- How common were they?

K.11 Did the overall process work as expected?

K.12 Would you suggest changes to the process for the next summer?

Household responses to EBT card activation.

K.13 Did households generally understand the program and their benefits?

[Probe:]

- What the new benefits were?

- How much their benefit was?

- When the benefits would be available?

- How to activate new cards?

- How to use their cards, if they had not used EBT before?

K.14 [For sites with new cards] Were there any issues that arise as a result of distributing new cards?

[Probe:]

- Was the branding on the card clear (for households, retailers)?

- Did retailers have any confusion about how to process the cards?

K.15 [For sites that loaded benefits onto existing cards] Were there any issues that arose as a result of having benefits from multiple programs on the same cards?

[Probe:]

- Did households understand what portion of their benefits were from each program?

- Did households understand what benefits were being used first?

- Did households understand when benefits from each program would expire?

- Did retailers have any confusion about how to process the cards?

K.16 Would you suggest making any changes to the EBT cards for the next summer?

Process for replacing cards.

K.17 If the household’s EBT card is lost, stolen or damaged, what is the process for replacement?

[Probe:]

- Is the process the same as [SNAP/WIC]? If not, what is different?

- Who does the household contact?

L. training of households and other participant support

[NOTE TO VISITOR: ASK FOLLOWING QUESTIONS IF SITE HAD HOTLINE OR OTHER SUPPORTS FOR HOUSEHOLDS]

Training for parents.

L.1 Beyond the information letter that was sent to parents about participating in the [SEBTC PROGRAM NAME] program, did parents in the demonstration sites receive training or additional information packets?

L.2 Did materials or training differ for certain populations (e.g., [SNAP/WIC] recipients versus non-recipients)?

Content of materials and training.

[Note to visitors: Please get copies of all materials.]

L.3 What is the focus of the materials or training?

[Probe:]

- Information about the [SEBTC PROGRAM NAME]

- What the EBT card is and how to use it (including activation)

- What types of foods are allowed, locations for accessing benefits

- Who should receive the benefit (children)

L.4 Do the materials or training provide broader information about eligibility for other programs (e.g., SNAP, WIC, summer food programs in the community)?

L.5 Do the materials or training address potential concerns about immigration status and benefit receipt?

L.6 Does the training include information on nutrition education (if so, we will talk in more detail about this later)?

Format of training.

L.7 How are materials or training being provided (e.g., information packets, one-on-one, classes, computer, web-based)?

L.8 Who prepared and distributed materials?

[Ask the following questions if there are group training sessions being conducted]

L.9 Who is conducting training?

L.10 Where are trainings being conducted?

L.11 How large are the classes?

L.12 How long do the training classes last?

L.13 How many training sessions will be held? And how often?

L.14 Are materials and training distributed in multiple languages? Which languages will be included?

L.15 Is follow-up training planned?

Partner agencies’ experiences with training.

L.16 Was the type of training appropriate?

L.17 Was the level of training appropriate? Would you suggest conducting more/less of certain types?

L.18 How much follow-up training/support was needed?

L.19 Overall, was more training needed/offered?

L.20 For classes, was the size and turnout about what you thought it would be?

L.21 Where were the classes located? Were those locations convenient for parents/guardians?

L.22 When were the classes held? Were the times convenient for parents/guardians?

L.23 How did the parents respond to the training?

L.24 Would you suggest making changes to the training (i.e., content, timing, length, location) for the next summer?

Extent of participant support provided.

[Ask if respondent indicated there were additional supports for households if they had questions about the program]

L.25 How often do households use the additional supports (account information, customer service number, website)? Were you able to track this?

L.26 What are common questions from households?

[Probe:]

- Account balances

- Eligible foods

- Expiration of benefits

- Other issues?

L.27 Have there been any issues with the participant support process?

L.28 Did you receive any feedback on the effectiveness of the various types of media training? Do households seem to understand the process?

L.29 Would you make any changes to the process for the next summer?

M. preventing and detecting abuse

Process for preventing and detecting abuse.

M.1 What is the process for detecting and preventing abuse of [SEBTC PROGRAM NAME] funds?

[Probe:]

- Is this the same as for [SNAP/WIC]?

- What type of monitoring system does the state have in place to detect abuses (e.g., computer programs that monitor EBT transactions at retailers)?

- Is there a system for fraud reporting and claims?

M.2 How are claims and suspected fraud investigated?

Extent of reported fraud and abuse.

M.3 Has the process worked as expected?

M.4 Have you had to make any changes to the process to better detect and prevent abuse?

M.5 Have you received reports of fraud or abuses?

[If so, probe:]

- Have your monitoring systems detected abuses by retailers or households?

- What kinds of fraud and abuse have been identified?

- How prevalent has fraud and abuse been among the demonstration population?

N. Expunging funds

Timing and process for expunging funds – SNAP approach.

N.1 When will unused funds be expunged from accounts for [SEBTC PROGRAM NAME]?

N.2 What process is being used to expunge funds from the cards?

[Probe:]

- Is this similar to the process used for SNAP when the cards are inactive?

- What happens to unused funds?

- What notification or reminders will households receive before their benefits are expunged?

Expiration of benefits – WIC approach.

N.3 What are the rules for expiration of [SEBTC PROGRAM NAME] benefits?

[Probe:]

- Are the rules the same as for WIC?

- How does the State account for expired benefits?

- Do households receive any notification or reminders before their benefits expire?

Challenges with expunging funds.

N.4 Has the process for expunging funds/benefit expiration worked as anticipated?

N.5 Have there been any problems with expunging funds for the [SEBTC PROGRAM NAME] benefits?

N.6 Did you have to change the process after implementation?

N.7 Is there anything you would do differently for the next summer?

Deactivating SEBTC cards.

N.8 Will [SEBTC PROGRAM NAME] cards be deactivated? If so, when?

O. training and technical assistance

for schools AND OTHER COMMUNITY PARTNERS

[NOTE TO VISITOR: ASK FOLLOWING QUESTIONS IF TRAINING AND OTHER SUPPORT FOR SCHOOLS AND OTHER COMMUNITY PARTNERS PROVIDED]

Types and extent of training and TA.

O.1 Did plans for providing training and TA to schools and community partners change since the last site visit?

[If so, probe:]

- How (e.g., format, organizations involved, number or types of staff included, timing)?

- Why were these changes made?

O.2 What training and TA were provided and to whom?

Experiences providing training.

O.3 Was the type of training or TA appropriate?

O.4 Was the level of training or TA appropriate? Would you suggest conducting more/less of certain types?

O.5 How much follow-up training/support was needed?

O.6 Overall, was more training or TA needed/offered?

O.7 How did the school staff respond to the training or TA?

O.8 Would you suggest making changes to the training or TA for next summer in terms of content, timing, length, or location?

P. training and support for retailers

[NOTE TO VISITOR: ASK FOLLOWING QUESTIONS IF RETAILER TRAINING AND OTHER SUPPORT PROVIDED]

Retailer training.

P.1 Since the last interview, has information/training been provided to retailers?

P.2 Have all [SNAP/WIC] retailers in the demonstration areas received information/training?

P.3 [For WIC sites] Have the changes to the WIC food package for [SEBTC PROGRAM NAME] households required additional retailer training?

Type of material or training.

P.4 What was the focus of the materials or training (e.g., information about the [SEBTC PROGRAM NAME], the look of the EBT card, use of the EBT card, the new WIC package)? What type of materials or training will be provided (e.g., information packets, one-on-one, group, computer, web-based)?

P.5 Who prepared and distributed materials?

P.6 Are materials or training offered in multiple languages? Which languages are included?

P.7 Were face-to-face trainings conducted at the retailer’s location or with multiple retailers?

P.8 Who conducted the training?

P.9 How long did training classes last?

P.10 How many trainings were there?

P.11 What retailer personnel were invited to the training?

P.12 What type of information was discussed?

P.13 How much training has been conducted so far? Please describe what has been done so far and what is planned for the future.

P.14 Is follow-up training planned after retailers receive the initial training or as new staff members join the retailers?

Extent of retailer training provided.

P.15 Was the type of training appropriate?

P.16 Was the level of retailer training appropriate?

P.17 How much follow-up training/support was needed?

P.18 Overall, was more training needed/offered?

P.19 How did the retailers respond to the training?

P.20 Did any issues arise that were not anticipated or addressed through training?

[Probe:]

- Stocking issues due to high demand

- Concerns/reactions from retailers about the new program

- Problems with cashiers recognizing and accepting cards

- Other issues?

P.21 Would you suggest making changes to the training (i.e., content, timing, length, location) for the next summer?

Additional retailer support.

P.22 Will retailers be expected to use the existing EBT retailer service number?

[Probe:]

- Who answers this number (vendor personnel, automated, State, etc.)?

- What are the hours of operation?

P.23 Will there be a special customer service/contact number for retailer questions about SEBTC?

[Probe:]

- Who will answer this number (vendor personnel, automated, State, etc.)?

- What will be the hours of operation for the special SEBTC retailer service number?

P.24 Are any other supports planned for retailers? If so, please describe them.

Extent of additional retailer support provided.

P.25 How often did retailers use the customer service number? Were you able to track this?

P.26 What were the common issues they raised?

P.27 Did questions arise from these calls that you would consider including in the retailer training?

P.28 Were there any issues with the customer service process?

P.29 Would you make any changes to the process for the next summer?

Q. NUTRITION EDUCATION

[SEBTC PROGRAM NAME] nutrition education.

Q.1 Since the last site visit, what nutrition education services have been or are being offered to household members as part of the demonstration? [NOTE TO SITE VISITOR, IF NO ADDITIONAL SERVICES WERE OFFERED, SKIP TO Q.10]

Q.2 Were these nutrition education services designed specifically for the demonstration? Or are they part of on-going education offered in the local area? Or both?

Q.3 What organization(s) is/are responsible for developing the nutrition education approach and materials?

Q.4 Are comparable activities being provided across the entire demonstration area? If there is variation across local areas or agencies, please describe that variation.

Q.5 Who is being targeted for nutrition education?

[Probe:]

• Are only households that receive the Summer EBT benefit participating or are households who do not receive the benefit also participating in nutrition education?

• Are subgroups of individuals targeted for particular education components (for example children only, parents/guardians but not children, etc.)?

Q.6 What is the structure of the services (e.g. written materials versus training)?

[If training, then probe:]

• Group or one-on-one?

• How long are the classes?

• How often are they offered?

Q.7 What is the focus of the nutrition education?

[Probe: What are some examples of handouts or lessons covered?]

• Stretching your shopping dollar

• (for WIC sites only) List of SEBTC foods available, portion sizes allowed

• Fruits and vegetables

• Whole grains

• Eating seasonal foods

• Farmer’s markets (if able to use card there)

• Using low-fat or fat-free milk

• Healthy snacks for kids

• Introducing new foods

Promoting nutrition education services.

Q.8 Since the last site visit, what activities have been used to promote the awareness and use of nutrition education among targeted households and children (for example, have you distributed information, provided support services, or made referrals)? Why did you to select these promotion activities? [IF NO NEW ACTIVITIES, SKIP TO Q.10.]

Q.9 Do you perceive these activities as helpful in encouraging targeted households to understand or take advantage of the education services?

Other nutrition education services offered in local areas.

Q.10 Since the last interview, have other nutrition education services become available to local residents in the target areas? [IF NO NEW ACTIVITIES, SKIP TO Q.15.]

Q.11 What organization(s) offer those services (such as (regular) WIC, cooperative extension, SNAP program, hospitals, Head Start, community groups, schools)?

Q.12 Do available services vary across local areas? If so, how?

Q.13 How many and what types of residents tend to receive these services?

Q.14 What is the focus of the nutrition education?

[Probe:]

• Stretching your shopping dollar

• Fruits and vegetables

• Whole grains

• Eating seasonal foods

• Farmer’s markets (if able to use card there)

• Using low-fat or fat-free milk

• Healthy snacks for kids

• Introducing new foods

Attitudes towards [SEBTC PROGRAM NAME] nutrition education.

Q.15 What has been the reaction of the staff towards the nutrition education component of the demonstration (e.g., receptive, enthusiastic, indifferent, etc.)?

Q.16 What have participants’ reactions been?

[Probe:]

- Are there particular components or modes of education that they prefer?

- If so, which ones and for what reasons?

Successes and challenges of [SEBTC PROGRAM NAME] nutrition education.

[ASK IF THERE IS A SEBTC NUTRITION EDUCATION COMPONENT]

Q.17 How successful has this nutrition education component been in meeting your goals?

Q.18 What has been successful about the planning process and delivery of education? What factors have led to these successes?

Q.19 What challenges have you faced? What are you and local agencies doing to overcome these challenges?

Q.20 What kinds of improvements could be made in the future?

R. Successes, Challenges, and Solutions

Goals.

R.1 Did [STATE] have to adjust its goals or expectations during the course of demonstration implementation?

[If so, probe:]

- What specifically changed?

- Why was this necessary?

- How was this accomplished?

Perceived impacts of the demonstration. [Ask of grantee and all major partners.]

R.2 To what extent do you believe the demonstration has affected children’s food security during the summer?

R.3 To what extent has it affected child nutrition during the summer?

R.4 To what extent has it affected household food security?

R.5 To what extent has it affected new applications for [SNAP or WIC]?

R.6 To what extent has it affected other summer feeding programs such as the SFSP, or emergency food services such as food banks and kitchens? Did they experience an increase or decrease in demand for food?

R.7 Are effects more apparent for certain groups of populations? [If so,] Which groups?

R.8 If [STATE] implemented services beyond the EBT benefit, to what extent does it appear that these services affected children’s outcomes?

R.9 What events during the summer outside of the demonstration may have affected its impacts?

Successes. [Ask of grantee and all major partners.]

R.10 Thus far, what have been the greatest successes of the demonstration?

R.11 What factors contributed significantly to this success?

R.12 What, if anything, could have been done differently to achieve greater improvements in children’s food security?

Challenges. [Ask of grantee and all major partners.]

R.13 Thus far, what have been the biggest challenges to meeting the goals of the demonstration?

R.14 Have these challenges been resolved?

- If so, how?

- If not, why?

R.15 What could the [GRANTEE] or other agencies have done differently to meet demonstration goals more effectively?

R.16 How have these challenges affected the effectiveness of the demonstration?

S. Feasibility of Continuing and Replicating DemonstrationS

Conditions needed to continue the demonstration. [Ask of grantee and all major partners.]

S.1 What conditions would need to be maintained for the demonstration to continue successfully next year?

S.2 What program components or implementation practices would need to be maintained?

S.3 What changes, if any, would need to be made in order to operate successfully next year?

Conditions needed for replication in other sites. [Ask of grantee and all major partners.]

S.4 What state and local conditions are critical to replicate the demonstration in other areas of the country?

S.5 What program components or implementation practices are essential for replication?

S.6 Are there practices you would advise other states and local areas to avoid?

[If so, probe:]

- What are they?

- Why would you suggest avoiding them?

S.7 What advice would you give to other states and local areas interested in replication of the demonstration?

T. Implementation and Operational Costs

Now I would like to turn to some questions related to the cost of implementing the demonstration. I brought copies of the grant expenditure data that you provided before the site visit and would like to review them with you in detail.

Review of grant expenditures. [Review cost forms with grantee and all major partners]

T.1 Can you please briefly describe the responsibilities of each staff member listed?

T.2 What staff title applies to each person (for example, senior administrator, project manager, IT/MIS manager, IT/MIS developer, data entry, nutrition program specialist, administrative support)?

T.3 Can you please describe what is included in the other direct costs by category?

Start-up versus on-going costs. [Ask grantee and each major partner receiving grant funds]

T.4 What types of activities are covered by your start-up expenditures?

[Probe:]

- Over what period do they pertain?

- How did you determine what activities would be considered start-up expenditures?

- If the demonstration were to continue for another year, would any of these costs be repeated next spring? If so, which ones?

T.5 What types of activities were covered by on-going expenditures?

[Probe:]

- Over what period do they pertain?

- If the demonstration were to continue for another year, would all of these costs be repeated next spring and summer? If not, which will not be needed?

Sufficiency of funds. [Ask grantee and each major partner receiving grant funds]

T.6 Do you believe that your organization had sufficient funds through your grant to implement the [SEBTC PROGRAM NAME] benefits successfully?

[If not, probe:]

- How much additional funding did you need?

- What would you have used those funds for?

Accounting systems. [Ask grantee and each major partner receiving grant funds]

T.7 Did you need to adapt your accounting systems to meet the grant requirements or evaluation cost data collection needs?

[If so, probe:]

- Please describe how.

- Did these changes influence other aspects of your agency’s work?

- Will these revised procedures be used within the organization beyond the grant?

Tracking expenditures. [Ask grantee and each major partner receiving grant funds]

T.8 Did you experience any challenges in tracking grant expenditures over the course of the start-up period? If so, please describe.

T.9 Over the course of the summer? If so, please describe.

Deviations from grant budget. [Ask grantee and each major partner receiving grant funds]

[NOTE TO VISITORS: REVIEW BUDGETS AND EXPENDITURES BEFORE VISIT TO TAILOR QUESTIONS AND EMPHASIZE THIS IS NOT AN AUDIT.]

T.10 Were there variations between your approved grant budget and your actual spending?

[If yes, probe:]

- Across what categories?

- In what amounts?

- Can you explain how and why resources were not spent as expected?

Matched, volunteer and donated resources [[Ask grantee and each major partner receiving grant funds].

T.11 Have any resources been used beyond the grant to support implementation of the grant?

[If so, probe:]

• What was the source and how much was used?

• How were these resources used?

T.12 Are volunteers involved in providing services?

[If so, probe:]

• Can you estimate how much volunteer time was spent?

• What was the role of these volunteers?

T.13 Were any other resources donated for the demonstration (e.g. office space, materials)?

[If so, probe:]

• What were they?

• Who donated them?

• How much was used?

Matched, volunteer and donated resources [Ask of major partners not receiving grant funds].

T.14 What resources have been used to support implementation of the grant?

[Probe:]

• How much was used?

• How were these resources used?

T.15 Are volunteers involved in providing services?

[If so, probe:]

• Can you estimate how much volunteer time was spent?

• What was the role of these volunteers?

T.16 Were any other resources donated for the demonstration (e.g. office space, materials)?

[If so, probe:]

• What were they?

• Who donated them?

• How much was used?

Staff time spent on [SEBTC PROGRAM NAME]. [Ask of each respondent at grantee and all major partners]

T.17 What percentage of your time do you typically spend on the program?

[Probe:]

• Has this changed over time?

• How much of your time is charged to the grant?

• How much is paid by other funding sources?

T.18 Do you work overtime hours on the project?

[If so, probe:]

• How much is paid versus unpaid?

CLOSING

Is there anything you think is important for the FNS to know about the [STATE]’s [SEBTC PROGRAM NAME] demonstration that we did not ask about?

Thank you for your time and helpful feedback. The information you have shared will be valuable to our team as we look across states and localities for themes and ideas that we can share with FNS.

Are your any questions you have for me before we finish?

END OF INTERVIEW

1 Note to visitor: if process for distributing EBT cards, replacing cards, or other EBT operations is specified in proposal, confirm and clarify as needed.

54