Business- Profit & Not -for- Profit

Process Study Topics for Private Entities*
Topic	EBT Vendor	CBOs	Retailers
Plan demonstration and prepare application		
Identify eligible children and households	
Inform selected parents/caretakers	
Distribute SEBT for Children cards		
Training for parents	
Training for/support for retailers			
Other training and public information			
Project organization and management		
Replace cards/other participant support		
Prevent/detect losses and abuse			
Involve local agencies or community organizations		
Successes, challenges and solutions			
Feasibility of continuing and replicating demonstrations			
Implementation and operational costs		

*Not all topics will be asked at each wave of the interview (April visits, July visits, September telephone interviews).

Please see individual guides.

OMB Clearance Number: Expiration Date:

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is xxxxxx. The time required to complete this information collection is estimated to average 90 minutes per response. If you have any comments concerning the accuracy of time estimates or suggestions for improving this form, please contact: U. S. Department of Agriculture, Food and Nutrition Service, Office of Research & Analysis, Alexandria, VA 22302.

Evaluation of the SEBT for Children

Implementation Study Protocols

Private Entities

April Site Visits

February 3, 2011

Introduction

My name is [X], from [Abt Associates/Mathematica Policy Research].

As you may know, Abt and Mathematica are working together to evaluate the Summer EBT for Children demonstrations for the Food and Nutrition Service of USDA. My colleagues and I are visiting each of the five sites and collecting information from a wide range of stakeholders involved in the demonstration in order to learn about the process of implementation, the challenges you encountered, and lessons learned. I want to start by thanking you for taking time to speak with us today. Your perspective and insights on these issues are very helpful.

Our reports to FNS will describe the range of responses expressed by staff, and may list the names of agencies and partners who contributed information, but we will not quote you or anyone by name or title. However, because of the relatively small number of organizations participating in the study, there is a possibility that a response could be correctly attributed to you.

I expect our conversation will take approximately [XX] minutes.

[OPTIONAL IF INTERVIEWER CHOOSES TO RECORD:] I want to be sure I am keeping track of everything you are saying. May I record our discussion so that I can listen to it later when I write up my notes? No one outside of our research team will have access to the recording. [IF YES:] Thank you. It will be helpful if you speak up, speak clearly, and speak one at a time. [IF NO:] That’s no problem. I’ll take notes as you talk, but I may sometimes need to ask you to slow down or repeat so that I can get all the information.

First, do you have any questions for me about the project in general or what we will be discussing today?

NOTE TO INTERVIEWERS: THIS GUIDE IS ORGANIZED ACCORDING TO TOPIC AREAS, TO BE ASKED OF SPECIFIC RESPONDENTS ACCORDING TO THE MASTER TOPIC LIST. HOWEVER, IT IS EXPECTED THAT SOME OF THE RESPONDENTS WILL NOT BE KNOWLEDGABLE ABOUT TOPICS ASSIGNED TO THEM, DEPENDING UPON THE SPECIFIC DEMONSTRATION. IF A RESPONDENT DOES NOT HAVE KNOWLEDGE OF THE TOPIC, SKIP TO THE NEXT TOPIC ASSIGNED TO HIM/HER.

Demonstration Planning and the Application Process

A.1. Impetus for applying. Why did [STATE] choose to apply for the demonstration grant? What were the perceived opportunities of this grant? What were the main challenges posed by the requirements of the RFA?

<<If respondent does not know, skip remaining questions in this section.>>

A.2. Goals of the demonstration. What goals do you hope to achieve? To what extent are the goals quantifiable? How will they be measured?

A.3. Organizations involved in application process. For what reasons did [AGENCY NAME] emerge as the lead agency for the grant? What is the agency’s previous experience with child nutrition or summer feeding programs for children? What other state agencies were involved in the application process? What were their roles? What was the nature and extent of collaboration between agencies? What local school food authorities (SFAs) were involved in the application? What was their role? Was your EBT vendor involved? In what way were any retailers involved? Were state or local advocacy, or community-based organizations involved? In what roles?

A.4. Selection of local area. How was the local area selected for involvement? What is the need for this program in the selected area? Were other areas considered? Why were they not included in the proposal (i.e why were they not selected by the State to be the demo area)?

A.5. The planning process and issues addressed. Who was involved in the planning process? What were the roles of the key players? What mechanisms were used during this planning process (e.g., workgroups, steering committees, focus groups)? Over what period of time did the planning process occur? What were the main issues of focus and/or concern during the planning process? What challenges did you encounter during the planning and application process? How were they addressed?

Local Context

B.1a. Food environment for WIC sites. How many WIC retailers are there in the demonstration area?¹ How many are large retailers such as national or regional franchises? Are there farmer’s markets in the area? Do they accept WIC paper vouchers? If so, what proportion? Do they accept WIC EBT cards? If so, what proportion? Are there any community gardens? Do local stores such as drugstores and bodegas offer any fresh foods? Do they accept WIC EBT cards? Are there any barriers for some local residents to access [SNAP/WIC] retailers? If so, are there agencies that offer services to help with this? What services do they offer (such as help with transportation, ability to order groceries for delivery to neighborhood centers)? Are there food banks in the area? If so, how large and how frequently are they open? Are there churches or other religious organizations that provide food emergency food services? Are there kitchens or other sources of emergency meals?

B.1b. Food environment for SNAP sites. How many SNAP retailers are there in the demonstration area?² How many are large retailers such as national or regional franchises? Are there farmer’s markets in the area? Do they accept SNAP EBT cards? Are there community gardens? Do local stores such as drugstores and bodegas offer any fresh foods? Are there any barriers for some local residents to access [SNAP/WIC] retailers? If so, are there agencies that offer services to help with this? What services do they offer (such as help with transportation, ability to order groceries for delivery to neighborhood centers)? Are there food banks in the area? If so, how large and how frequently are they open? Are there churches or other religious organizations that provide food outreach? Are there kitchens or other sources of emergency meals?

B.2. Local schools.

How many elementary, middle, and high schools are in the demonstration area? Are there any charter schools or private schools in the area that operate the NSLP? If so, how many and for what grade levels? How many students eligible for free/reduced-price meals are enrolled? Are they included in the demonstration?

B.3. Summer Food Service Program (SFSP).Were there summer feeding programs available in the demonstration area in 2010?³ How many SFSP sites operated in the target area? What types of sites operated? How many children were served? How many and what types of organizations (SFA, government, non-profit) sponsor the SFSP and how many feeding sites did they run? How many weeks were the programs open? What kinds of issues arose during the summer of 2010?

B.4. Other existing summer feeding options. What other summer feeding options (such as Seamless Summer Feeding Program, NSLP, kids' cafes, Child and Adult Care Food Programs) were available in the summer of 2010 in the target communities? Please describe them. What types and how many organizations administered these programs? How many children were typically served by each?

B.5. Gaps in summer food services. Do gaps exist in the availability of summer food services for children? How large is the unmet need? What are reasons for the gaps in services (e.g., limited funding, not enough sites, poor transportation to sites, not open the full summer, limited outreach, lack of awareness, stigma, limited hours or days of operation, parents concerns’ about neighborhood safety, lack of program activities)?

Basic parameters of the summer EBT MODEL

[NOTE TO VISITOR: We will get more details on these features later in the protocol. This section is intended to understand the basic model as context for the remainder of the interview.]

C.1. Branding. Does the demonstration have a “brand” or name for outreach purposes? Why was that name selected? Did you develop a logo? What does the logo represent?

C.2. Eligibility. Are all children who are eligible for FRP meals included in the lottery for summer EBT benefits? If not, what additional eligibility rules or restrictions are being implemented? How and why were these rules or restrictions determined?

C.3. EBT cards. Are new EBT cards being designed for the demonstration or did [STATE] plan to use existing EBT cards? For households already receiving benefits from [SNAP/WIC], will they receive a new card for the summer benefit or will the summer benefit be added to the balance on their existing card? Why did [STATE] choose this method (e.g., easier administratively, cost savings, easier for households)? If a single card is used for multiple programs, are the [SEBTC PROGRAM NAME] funds distinct? Will households with multiple children be given a single summer benefit card or will they be given the option to receive one card per child? Why did [STATE] choose this method?

C.4. Level of EBT benefits. What has been the reaction of participating organizations to the $60 per child monthly benefit level specified by FNS? Do they believe the level is too high, too low, or appropriate? How are benefits being prorated for partial months during the summer? Why was this approach taken? How have participating organizations reacted to children of all ages receiving the same amount of benefits and that multiple children in a household each receive the full benefit (no accounting for economies of scale)?

C.4a. Mix of benefits in WIC sites. What has been the reaction of participating organizations to the WIC package designed for the summer demonstration? Do you feel the split between food and cash voucher was appropriate? Do you feel the mix of acceptable foods was appropriate?

C.5. Timeframe of EBT benefits. During what calendar period will EBT benefits be active (e.g., day after school ends in spring through day before school begins in fall)? What factors were considered in making this decision? When will monthly benefits issued (e.g., first day of the month vs. staggered schedule)? What implications might this have for implementation? What implications might it have for households accessing benefits?

C.6. Services beyond the EBT benefit. Are families offered any services, such as nutrition education, beyond the EBT benefit? If so, we will talk about them in more detail later, but can you give me a brief description?

Project Organization and Management

D.1. Organizational structure to administer the demonstration. What is the overall organizational structure for administering the demonstration? (Please include only those organizations that are receiving grant funds. We will discuss other partners separately.) What agencies, organizations, and partners are involved? What are the responsibilities of each of these entities, as they relate to [SEBTC PROGRAM NAME]? Has the mix of organizations or the organizational structure changed since the grant application? If so, how and why? Have the responsibilities of each organization changed from what was proposed in the grant application? If so, how and why? What are the main advantages to the current organizational structure?

D.2. Agreements between organizations. Are there agreements or contracts in place between [GRANTEE] and [EACH PARTICIPATING ORGANIZATION]? If so, do these agreements or contracts identify the goals, objectives, roles, and responsibilities as they pertain to the demonstration? Have these agreements been formalized? If yes, how so? [GET COPIES OF CONTRACTS or MOUs]

D.3. Staffing structure. What is the current staffing structure for administering demonstration services, within and across each organization? How many staff members are involved at each organization at the state level? At the local level? What roles do key staff members play? What type of backgrounds and qualifications do they have? What roles do support staff play? What type of backgrounds and qualifications do they have? Are there shared responsibilities across organizations? Who leads or coordinates them? Were any new staff members hired as a result of the demonstration? If so, which ones? What new skills are required of existing staff to carry out demonstration services? Did these staff receive training to carry out their new responsibilities?

D.4. Changes in staffing and staff turnover. Has the staffing structure changed since the inception of the grant? If so, how and why? Has there been staff turnover? In what positions and for what reasons? What were the effects of this turnover?

D.5. Communication between grantees and key partners. What levels of staff communicate across the lead agencies? How often? For what reasons? What topics are discussed? In what format are meetings held (large group meetings, conference calls, one-on-one with each partner)? What percentage of time do staff members dedicate to the activities of the partnership? Have there been challenges to maintaining communication? In what ways and for what areas could communications be improved? What forms of communication and collaboration have been most helpful? In what ways?

Involving Local Agencies and Community Organizations

E.1. Identification of partners. What local agencies and community organizations that do not receive grant funds are partners in the demonstration (e.g., SFSP sites and other summer feeding programs, emergency food providers, hunger advocacy groups, other faith or community-based organizations)? Which partners play the most prominent roles in the demonstration? How did you identify these partners? Did you consider or approach others? Why are they not involved? Did any planned partnerships fail to materialize? Which ones? Why weren’t these partnerships realized?

E.2. Roles and responsibilities of prominent partners. What are the roles and responsibilities of each partner for the demonstration? For example, does the partner play an advisory role in planning and decision making, volunteer resources and staff time for training and activities, provide complementary services to beneficiaries, provide outreach and referral, or something else? For prominent partners, describe the roles in more detail. What do you believe are the key benefits of the partnership for the project, the participating households, and the partners?

E.3. Initiation of the partnerships. Did you have working partnerships with these local organizations before this demonstration grant? If so, describe your prior relationship. In what ways were these prior relationships useful for the demonstration?

E.4. Partnership agreements. Did you establish common goals and objectives pertaining to the demonstration with partners? Was your agreement formalized? If so,how? [GET COPY OF CONTRACTS OR MOUS] If not, why? What are the sources of funding for partners’ activities?

E.5. Staffing structures. How many and what levels of staff are involved in the demonstration at each organization? What roles do they play? Were any new staff members hired as a result of the demonstration? If so, which ones? What new skills are required of existing staff to carry out demonstration services? Did these staff receive training to carry out their new responsibilities?

E.6. Continued collaboration between partners. What levels of staff communicate across these partner agencies/organizations? How often? For what reasons? Are these partners included in communications between the grantee and its subgrantees? Have there been challenges to maintaining collaboration? What forms of communication and collaboration have been most helpful?

E.7. Successes and challenges of the partnerships. What are the most effective elements of the partnership, and why? What have you learned about establishing and maintaining these partnerships? What do you view as the key logistical considerations that other state grantees and their partners would need to consider, if they were to implement a similar type of partnership? What has worked best in these partnerships? What partnership aspects could be improved? From which partners would you have liked more involvement?

Coordination with EBT Vendor

F.1. EBT system preparations for [SEBTC PROGRAM NAME]. Are changes to the vendor’s EBT system or third-party processor (TPP) systems needed to implement [SEBTC PROGRAM NAME]? If so, what changes are needed? What is the status of these changes? Who is involved with this process? What design, development, implementation, and testing steps have been completed? How long did this take? What activities took the most effort? What issues were encountered? What tasks remain to be done?

F.2. Point of sale (POS) system preparations for [SEBTC PROGRAM NAME]. Are changes to the POS system programming or data loads needed to implement [SEBTC PROGRAM NAME]? If so, what changes are needed? What is the status of these changes? Who is involved with this process? What design, development, implementation, and testing steps have been completed? How long did this take? What activities took the most effort? What issues were encountered? What tasks remain to be done?

F.3. Automated customer service system preparations for [SEBTC PROGRAM NAME]. Are changes to the interactive voice response (IVR) system, customer service scripts, or user website needed to implement [SEBTC PROGRAM NAME]? If so, what changes are needed? What is the status of these changes? Who is involved with this process? What design, development, implementation, and testing steps have been completed? How long did this take? What activities took the most effort? What issues were encountered? What tasks remain to be done?

OUTREACH AND Consent for the Demonstration
and Evaluation

H.1. Outreach and recruitment efforts. What outreach efforts have been conducted or are planned to inform parents/caregivers about the demonstration? What methods are being used (e.g., mail, flyers, radio, television)? What are the main messages in outreach materials? How widely are materials and information being distributed? In what ways are outreach efforts different for households already enrolled in SNAP/WIC versus those that are not? Did materials address potential concerns about immigration status and benefit receipt? What organizations are responsible for outreach? Are materials distributed in multiple languages and/or are media efforts targeted to subpopulations of non-English speaking households? If so, what languages? In what month did outreach efforts begin? Of these activities, what has been done so far? What is the planned schedule for remaining outreach efforts?

H.2. Consent collection process. Briefly, what is the planned approach to distributing and collecting consent forms from households? Are households required to “opt in” or provide active consent for the release of their contact information? What information are they asked to provide? If there is a participant agreement, what terms must participants accept? Are households told they must “opt out” or indicate if they do not wish to participate? In what calendar month are consent forms being distributed? How long do households have to respond?

H.3. Maximizing consent response, [IF ACTIVE CONSENT REQUIRED]. What efforts are being made to maximize response rates from households? What procedures are in place to contact nonresponsive households? How often are households contacted? In what ways? Are there procedures in place to encourage households that refuse to participate to reconsider that decision? If so, what are they? How long will they be administered before the list of consenting households is finalized? Are special outreach materials or information provided to encourage participation in the demonstration? Please describe them (media, languages, etc.). Who will be leading this requirement and contacting the nonrespondents (e.g., state/local agency, CBOs, school staff)? What challenges have you faced in encouraging household response? Did you overcome them and how? What kind of technical assistance or support would be most useful?

H.4. Tracking and managing consent distribution and receipt. What approach is being used to track consent distributions and receipts? How are returned letters handled, if addresses are incorrect? How are re-mailing or follow-up needs assessed, if returned forms are incomplete or missing information? How are the data stored and cleaned? Who has access to the data? What aspects of consent management and tracking have been successful?

H.5. Staff responsibilities related to consent. How are consent management responsibilities divided among organizations and staff? Who has primary responsibility for managing the production, distribution, and collection of consents? Who is responsible for tracking information about consent mailings and returns (such as the number of letters that have been mailed out, returned as undeliverable because of incorrect addresses, positive responses [active consent only], refusals)?

H.6a. Consent return rates [IF ACTIVE CONSENT REQUIRED] for active sites. How many consent packages were sent in the initial mailing? What percentage of households that were sent mailings have returned consent forms with complete information? What percentage returned forms with incomplete or missing information? What percentage of letters were returned undeliverable? What percentage of households received multiple contacts? How successful were efforts to contact nonresponsive households?

H.6b. Consent return rates [IF PASSIVE CONSENT REQUIRED.] How many consent packages were sent in the initial mailing? What percentage of households that were sent mailings opted out or responded that they did not want to be part of the demonstration? Was there any effort to encourage these households to reconsider? What percentage of letters were returned undeliverable? Was an attempt made to determine the correct address and then mail the consent forms again?

H.7. Household reaction to the level of EBT benefits. What has been the reaction of households to the $60 per child monthly benefit level specified by FNS? Do they believe the level is too high, too low, or appropriate? Has this affected the consent rates for the demonstration in any way? If so,how?

H.8. Transfer of information. How will contact information on consenting households be transferred confidentially between leading agencies and the evaluation team? What information will be sent to whom? What challenges have you encountered with this transfer of information? What types of support would help overcome these challenges?

H.9. Lessons learned from consent process [ASK IF PROCESS COMPLETE]. What aspects of the consent process could have been improved? Why? What advice would you give others about tracking and managing consent information? What kind of technical assistance or support would be most useful?

Informing households about Random Assignment Results AND ENROLLING THEM IN [SEBTC PROGRAM NAME]

I.1. Timeline for informing households of random assignment results. Have you begun to inform households of the results? If yes, in what month did you begin to inform households? If no, when do you anticipate informing households? Do you feel the timeline for this process was/will be reasonable and early enough for parents to plan for the summer? If not, how much earlier would you like to complete the process in future summers?

[NOTE TO VISITORS: CHANGE NEXT QUESTIONS TO FUTURE TENSE IF PROCESS HAS NOT YET STARTED IN THE SITE]

I.2. Informing households that were not selected for [SEBTC PROGRAM NAME]. How were households that were eligible for [SEBTC PROGRAM NAME] but were not chosen for participation in the demonstration informed (e.g., letter)? What information was provided? Were they given materials about other summer feeding options, [SNAP/WIC], nutrition education, or other topics? If so, what was provided? [GET COPIES OF MATERIALS] What was parents’ response to these materials? Did you receive any complaints about households not being selected to receive benefits? If so, how many? Who responded to these households and how?

I.3. Informing households that were selected for [SEBTC PROGRAM NAME]. How were households that were selected for participation in the demonstration informed (e.g. letter)? We will ask more detailed questions about the materials later, but can you provide a brief description? What was parents’ response to these materials? Did you receive comments that households were pleased to be accepted? If so how many?

I.4. Enrolling households in the [SEBTC PROGRAM NAME] database. Are data on [SEBTC PROGRAM NAME] households being maintained in the existing [SNAP/WIC] databases? Or is a new system being developed for the demonstration? Why did you choose this approach? What is the status of the changes to the existing system or the new database? Who is involved with this process? What design, development, implementation, and testing steps have been completed? How long did this take? What activities took the most effort? What issues were encountered? What tasks remain to be done? How will the data flow between the grantee, the EBT vendor, and the evaluation contractor when the [SEBTC PROGRAM NAME] is operational?

distributing ebt cards⁴

J.1. EBT card distribution. How will the initial EBT cards be distributed to the [SEBTC PROGRAM NAME] households (by mail, in person, pick up, etc.)? Who will distribute the cards (e.g., eligibility staff, EBT vendors, school, CBOs, etc)? How will returned mail be handled (due to incorrect addresses, moves, etc.)?

J.2. Additional materials sent with EBT cards. Will letters or information materials accompany the EBT card? What type of information will be included (how to activate/access the card, where to use, contact information, etc.)? [GET COPIES] Will training be offered to households unfamiliar with the EBT process? [More questions about specifics of training below].

J.3. EBT card activation. What will be the process for households to activate the card (e.g., activated upon receipt or upon use, or call a number to activate.)? Is this the same process used for [SNAP/WIC]? Who will the household interact with to activate their card (e.g., eligibility worker, vendor, automated)? Will the household receive a Personal Identification Number (PIN) to be used with the card? How will PINs be distributed securely? May households change their PINs?

training for schools AND OTHER COMMUNITY PARTNERS

N.1. Types and extent of training. Will key staff at the participating schools and community partners receive training about the demonstration? What type of training will be provided (e.g., information materials distribution, one-on-one, group, computer)? What will be the focus of the training (e.g., related to developing lists of eligible children, the use of [SEBTC PROGRAM NAME] benefits and EBT cards, outreach and education to eligible households)? Who will conduct the training? Who will be invited to participate in the training? How long is the training? Has any of the training already happened? If so, describe what has been done and what is planned for the future? Is follow-up training planned after participating staff receive the initial training or as new staff members join participating organizations?

[NOTE TO VISITOR: ASK FOLLOWING QUESTIONS ONLY IF AT LEAST SOME TRAINING HAS ALREADY BEEN PROVIDED]

N.2. Experiences providing training. Was the level of training appropriate? How much follow-up training/support was needed? Overall, was more training needed/offered? Was the type of training appropriate? Would you suggest conducting more/less of certain types? How did the school staff respond to the training? Would you suggest making changes to the training (i.e., content, timing, length, location) for next summer?

N.3. Experiences of school staff and community partners with training. What type of training did you receive (e.g., information materials distribution, one-on-one, group, computer)? What was the focus of the training (e.g., related to developing lists of eligible children, the use of [SEBTC PROGRAM NAME] benefits and EBT cards, outreach and education to eligible households)? Who conducted the training? How long was the training? Was there any follow-up training provided? Was the level of training sufficient to understand the program and complete your role in the demonstration? Was the type of training appropriate? Would you suggest conducting more/less of certain types? What changes would you suggest the state makes to the training in the future (i.e., content, timing, length, location)?

training for parents on [SEBTC PROGRAM NAME] benefits

[NOTE TO VISITOR: ASK THIS SET OF QUESTIONS ONLY IF TRAINING FOR PARENTS HAS ALREADY STARTED AT TIME OF VISIT. OTHERWISE, QUESTIONS WILL BE ASKED DURING SUMMER SITE VISIT]

O.1. Training for parents. Beyond the information letter that will be sent to parents about participating in the [SEBTC PROGRAM NAME] program, will parents in the demonstration sites receive training or additional information packets? Will the materials or training differ for certain populations (e.g., [SNAP/WIC] recipients versus non-recipients)?

O.2. Content of materials and training. What will be the focus of the materials or training (e.g., information about the [SEBTC PROGRAM NAME], what the EBT card is and how to use it (including activation), what types of foods are allowed, locations for accessing benefits, who should receive the benefit (children))? Will the materials or training provide broader information about eligibility for other programs (e.g., SNAP, WIC, summer food programs in the community)? Will the materials or training address potential concerns about immigration status and benefit receipt? Will the training include information on nutrition education (if so, we will talk in more detail about this later)?

O.3. Format of training. How will materials or training be provided (e.g., information packets, one-on-one, classes, computer, web-based)? Who will prepare and distribute materials? Will households need to attend training to receive their EBT card? Where will trainings be conducted? If you are using training classes, how large will the classes be? Who will conduct training? How long will the training classes last? How often will the training be held? How is the training structured? Will materials and training be distributed in multiple languages? Which languages will be included? Is follow-up training planned?

O.4. Additional support for parents. Will households have access to an 800 telephone number to ask questions about the demonstration? Who will answer those calls? Is there a location where household members could go to get help (face-to-face)? Who will provide that help (e.g., local benefit office, CBO, school staff)? How convenient are these locations for parents/guardians? Will there be a website or webpage dedicated to [SEBTC PROGRAM NAME] information? How will access be controlled? What other types of media will be used to train/inform participants (e.g., training videos, media campaigns, radio ads)?

training and support for retailers

[NOTE TO VISITOR: ASK THIS SET OF QUESTIONS ONLY IF RETAILER TRAINING HAS ALREADY STARTED AT TIME OF VISIT. OTHERWISE, QUESTIONS WILL BE ASKED DURING SUMMER SITE VISIT]

P.1. Retailer training. From a retailer’s perspective, will the [SEBTC PROGRAM NAME] be very similar to [SNAP/WIC] or will there be a distinction? To what extent will retailers need informational materials or training? Will all [SNAP/WIC] retailers in the demonstration areas receive information/training? [For WIC sites] How will information on the WIC food package for [SEBTC PROGRAM NAME] households be communicated to retailers?

P.2. Type of materials or training. What will be the focus of the materials or training (e.g., information about the [SEBTC PROGRAM NAME], the look of the EBT card, use of the EBT card, the new WIC package)? What type of materials or training will be provided (e.g., information packets, one-on-one, group, computer, web-based)? Who will prepare and distribute materials? Will materials or training be offered in multiple languages? Which languages will be included? Will face-to-face training be conducted at the retailer’s location or with multiple retailers? Who will conduct the training? How long will training classes last? How often will the training be held? What retailer personnel will be invited to the training? What type of information will be discussed? What is the structure? Has any training been conducted so far? If so, please describe what has been done so far and what is planned for the future. Is follow-up training planned after retailers receive the initial training or as new staff members join the retailers?

P.3. Additional retailer support. Will retailers be expected to use the existing EBT retailer service number? Who answers this number (vendor personnel, automated, State, etc.)? What are the hours of operation? Will there be a special customer service/contact number for retailer questions about SEBTC? Who will answer this number (vendor personnel, automated, State, etc.)? What will be the hours of operation for the special SEBTC retailer service number? Are any other supports planned for retailers? Describe.

NUTRITION EDUCATION

[NOTE TO VISITOR: ASK THE FOLLOWING QUESTIONS ONLY IF NUTRITION EDUCATION WAS ALREADY BEGUN OR WAS CONDUCTED AS PART OF OUTREACH]

Q.1. [SEBTC PROGRAM NAME] nutrition education. Can you describe the nutrition education that has been or will be offered to household members in the demonstration area? What are the key elements, in terms of the goals, planning, nutrition messages, tools, and format? Are comparable activities provided across the entire demonstration area? If there is variation across local areas or agencies, please describe that variation. What guidelines or requirements do you have in place for the demonstration as a whole and at the local level?

Q.2. Target groups for nutrition education. What individuals will be targeted for nutrition education? Are only households that receive the [SEBTC PROGRAM NAME] benefit participating or are households who do not receive the benefit also participating in nutrition education? Are subgroups of individuals targeted for particular education components (for example children only, parents/guardians but not children, etc.)? What criteria will you use to identify subgroups? How often will parents/guardians and children receive education and through what modes?

Q.3. Promoting nutrition education services. Will households have to participate in nutrition education to receive an EBT card? Why did you take this approach? What activities will be used to promote the awareness and use of nutrition education among targeted households and children (for example, have you distributed information, provided support services, or made referrals)? Why did you to select these promotion activities? Do you perceive these activities as helpful in encouraging targeted households to understand or take advantage of the education services?

Q.4. Adaptations to typical [SNAP/WIC] nutrition education. Does the nutrition education you are offering to parents, guardians, and children differ from what you offer as a part of typical [SNAP/WIC] services? What aspects are similar, and what aspects are different in terms of the delivery of education and materials used? Why are there differences between typical [SNAP/WIC] nutrition education and [SEBTC PROGRAM NAME] nutrition education? How will staff be advised of the change? Will staff be trained to use the adapted materials?

Q.5. Staff and resources to develop and provide [SEBTC PROGRAM NAME] nutrition education. Who are the key developers of the nutrition education approach and materials? Who are the staff members that will deliver the nutrition education? How many staff members will provide education? What percentage of their time will be allocated to educating participants? How will the messages delivered (e.g., written materials, group presentations, videos, etc.) Are materials geared toward adults or children? Are there any special resources or partnerships that have helped to plan for or provide this nutrition education? What are they? Are there referrals to sources of more in-depth nutrition counseling in the community?

Successes, Challenges, and Solutions

R.1. Successes. Thus far, what have been the greatest successes of the demonstration? What factors contributed significantly to this success? What, if anything, could be done differently to improve the planning process or the initiation of the program?

R.2. Challenges. Thus far, what have been the biggest challenges to meeting the goals of the demonstration? Have these challenges been resolved? If so, how? If not, why? What could [STATE] have done differently in the early phases of implementation to meet its goals more effectively?

CLOSING

Is there anything you think is important for the FNS to know about the [STATE]’s [SEBTC PROGRAM NAME] demonstration that we did not ask about?

Thank you for your time and helpful feedback. The information you have shared will be valuable to our team as we look across states and localities for themes and ideas that we can share with FNS.

Are there any questions you have for me before we finish?

END OF INTERVIEW

OMB Clearance Number: Expiration Date:

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is xxxxxx. The time required to complete this information collection is estimated to average 90 minutes per response. If you have any comments concerning the accuracy of time estimates or suggestions for improving this form, please contact: U. S. Department of Agriculture, Food and Nutrition Service, Office of Research & Analysis, Alexandria, VA 22302.

Evaluation of the SEBT for Children

Implementation Study Protocols

Private Entities

July Site Visits

February 3, 2011

INTRODUCTION

My name is [X], from [Abt Associates/Mathematica Policy Research].

As you may know, Abt and Mathematica are working together to evaluate the Summer EBT for Children demonstrations for the Food and Nutrition Service of USDA. My colleagues and I are visiting each of the five sites and collecting information from a wide range of stakeholders involved in the demonstration in order to learn about the process of implementation, the challenges you encountered, and lessons learned. I want to start by thanking you for taking time to speak with us today. Your perspective and insights on these issues are very helpful.

Our reports to FNS will describe the range of responses expressed by staff, and may list the names of agencies and partners who contributed information, but we will not quote you or anyone by name or title. However, because of the relatively small number of organizations participating in the study, there is a possibility that a response could be correctly attributed to you.

I expect our conversation will take approximately [XX] minutes.

[OPTIONAL IF INTERVIEWER CHOOSES TO RECORD:] I want to be sure I am keeping track of everything you are saying. May I record our discussion so that I can listen to it later when I write up my notes? No one outside of our research team will have access to the recording. [IF YES:] Thank you. It will be helpful if you speak up, speak clearly, and speak one at a time. [IF NO:] That’s no problem. I’ll take notes as you talk, but I may sometimes need to ask you to slow down or repeat so that I can get all the information.

First, do you have any questions for me about the project in general or what we will be discussing today?

[NOTE TO INTERVIEWER: FOR RESPONDENTS NOT INTERVIEWED IN APRIL, USE QUESTIONS FROM APRIL GUIDE]

NOTE TO INTERVIEWERS: THIS GUIDE IS ORGANIZED ACCORDING TO TOPIC AREAS, TO BE ASKED OF SPECIFIC RESPONDENTS ACCORDING TO THE MASTER TOPIC LIST. HOWEVER, IT IS EXPECTED THAT SOME OF THE RESPONDENTS WILL NOT BE KNOWLEDGABLE ABOUT TOPICS ASSIGNED TO THEM, DEPENDING UPON THE SPECIFIC DEMONSTRATION. IF A RESPONDENT DOES NOT HAVE KNOWLEDGE OF THE TOPIC, SKIP TO THE NEXT TOPIC ASSIGNED TO HIM/HER.

Demonstration Planning

A.1. Goals of the demonstration. Since the last site visit in [MONTH], have the goals of the demonstration changed? If so, how? For what reasons? What new goals do you hope to achieve? To what extent are the goals quantifiable? How will they be measured?

Local Context

B.1. Geographic area. Since the last site visit in [MONTH], has the local area targeted for the demonstration changed? If so, how? For what reasons?

B.2. Food environment. Since the last site visit, has the food environment in the demonstration area changed? If so, in what ways? For what reasons? Are more or fewer resources available to local households? What types?

B.3. Local schools. Since the last site visit, have there been any changes in the number or types of schools in the demonstration area? If so, please describe.

B.4. Summer Food Service Program (SFSP). Since summer 2010, has there been any change in the summer feeding programs available in the demonstration area? If so, what changed? For what reasons? Are more or fewer summer feeding sites operating? Are more or fewer children being served? Are the hours, days, or weeks of operation different?

B.5. Other existing summer feeding options. Since summer 2010, has there been any change in other summer feeding options (such as Seamless Summer Feeding Program, NSLP, kids' cafes, Child and Adult Care Food Programs) were available in the summer of 2010 in the target communities? If so, what changed? For what reasons? Are more or fewer summer feeding options for children? Are more or fewer children being served? Are the hours, days, or weeks of operation different?

B.6. Gaps in summer food services. Do gaps exist this summer in the availability of summer food services for children? How large is the unmet need? What are reasons for the gaps in services (e.g., limited funding, not enough sites, poor transportation to sites, not open the full summer, limited outreach, lack of awareness, stigma, limited hours or days of operation, parents concerns’ about neighborhood safety, lack of program activities)?

Project Organization and Management

D.1. Organizational structure to administer the demonstration. Has the overall organizational structure for administering the demonstration changed since the last site visit? If so, please describe how. Are any new organizations participating? If so, what are their responsibilities? How were they selected? Did any organizations drop out of the demonstration? If so, why? Which organizations have taken over their responsibilities? What are the main advantages to the current organizational structure? What aspect(s) would you recommend that other demonstration programs adopt? Why?

D.2. Agreements between organizations. Have agreements between [GRANTEE] and [EACH PARTICIPATING ORGANIZATION] changed since the last visit? If so, how and why? [GET COPIES OF CONTRACTS or MOUs]

D.3. Staffing structure. What is the current staffing structure for administering demonstration services, within and across each organization? How many staff members are involved at each organization at the state level? At the local level? What roles do key staff members play? What type of backgrounds and qualifications do they have? What roles do support staff play? What type of backgrounds and qualifications do they have? Are there shared responsibilities across organizations? Who leads or coordinates them? Were any new staff members hired as a result of the demonstration? If so, which ones? What new skills are required of existing staff to carry out demonstration services? Did these staff receive training to carry out their new responsibilities?

D.4. Changes in staffing and staff turnover. How has the staffing structure changed, if at all, since the last interview? How and why? Has there been staff turnover? In what positions and for what reasons? What were the effects of this turnover?

D.5. Communication between grantees and key partners. What levels of staff communicate across the lead agencies? How often? For what reasons? What topics are discussed? In what format are meetings held (large group meetings, conference calls, one-on-one with each partner)? What percentage of time do staff members dedicate to the activities of the partnership? Have there been challenges to maintaining communication? In what ways and for what areas could communications be improved? What forms of communication and collaboration have been most helpful? In what ways?

Involving Local Agencies and Community Organizations

E.1. Identification of partners. Has there been any change in the involvement of local agencies and community organizations that do not receive grant funds (such as, SFSP sites and other summer feeding programs, emergency food providers, hunger advocacy groups, other faith or community-based organizations, etc.)? If so, what changes were made? Why? At this stage, which partners play the most prominent roles in the demonstration?

E.2. Roles and responsibilities of prominent partners. Have there been any changes to the roles and responsibilities of partners involved at the time of the last interview? If so, please describe. If you have any new partners, what are the roles and responsibilities of each new partner for the demonstration? For example, does the partner play an advisory role in planning and decision making, volunteer resources and staff time for training and activities, provide complementary services to beneficiaries, provide outreach and referral, or something else? What do you believe are the key benefits of the partnership for the project, the participating households, and the partners?

E.3. Partnership agreements. Has there been any change to the agreements made with participating organizations? If so, what changes? Why? [GET COPIES.]

E.4. Staffing structures. How many and what levels of staff are involved in the demonstration at each organization? What roles do they play? Were any new staff members hired as a result of the demonstration? If so, which ones? What new skills are required of existing staff to carry out demonstration services? Did these staff receive training to carry out their new responsibilities?

E.5. Continued collaboration between partners. What levels of staff communicate across agencies/organizations? How often? For what reasons? In what way (large group meetings, conference calls, one-on-one with each partner)? Have there been challenges to maintaining collaboration? What forms of communication and collaboration have been most helpful?

E.6. Successes and challenges of the partnership. What are the most effective elements of the partnership, and why? What have you learned about establishing and maintaining these partnerships? What do you view as the key logistical considerations that other state grantees and their partners would need to consider, if they were to implement a similar type of partnership? What has worked best in these partnerships? What partnership aspects could be improved? From which partners would you have liked more involvement?

Coordination with EBT Vendor [CHRIS]

F.1. EBT system preparations. What steps in the implementation of changes to the vendor’s EBT system or third-party processor (TPP) systems for [SEBTC PROGRAM NAME] have taken place since the last interview? Who was involved with this process? How long did this take? What activities took the most effort? What issues were encountered in completing implementation? What operational issues have been encountered? What changes are you considering for next year?

F.2. Point of sale (POS) system preparations. What steps in the implementation of changes to the POS system programming or data loads for [SEBTC PROGRAM NAME] took place since the last interview? Who was involved with this process? How long did this take? What activities took the most effort? What issues were encountered in completing implementation? What operational issues have been encountered? What changes are you considering for next year?

F.3. Automated customer service system preparations. What steps in the implementation of changes to the interactive voice response (IVR) system, customer service scripts, or user website needed to implement for [SEBTC PROGRAM NAME] took place since the last interview? Who was involved with this process? How long did this take? What activities took the most effort? What issues were encountered in completing implementation? How well do the automated customer service systems (interactive voice response (IVR) system, customer service scripts, user website) meet customer needs? What operational issues have been encountered? What changes are you considering for next year?

Identifying Eligible Children and Households

[NOTE TO VISITOR: ASK FOLLOWING QUESTIONS IF PROCESS OF IDENTIFYING HOUSEHOLDS WAS NOT YET COMPLETE AT TIME OF LAST SITE VISIT]

G.1. Identifying children eligible for FRP meals. When did you complete the process for collecting data on children within the local SFAs who are eligible for free and reduced price meals? Who was responsible for compiling the data, the food service department, or the central administration or both? Were there any challenges in compiling the data (e.g., files from various school districts came in different formats, not all data items were available from each district, data on all eligible children were not provided by some districts, data split between nutrition department records and educational records)? If so, were they resolved and how? How many children were ultimately identified in the target area?

G.2. Aggregating data into household level files. When did you complete the process for consolidating the child-level data into a household-level file? Who was responsible for that task? Which SFAs (if any) already had a household or family identifier in their data? What identifiers were used? Were there any challenges in compiling the data (e.g. matching children without [SNAP/WIC] identifier, difficulty matching guardian names, addresses, and telephone numbers across databases)? If so, were they resolved and how? How many households were ultimately identified in the target area?

OUTREACH AND Consent for the Demonstration
and Evaluation

H.1. Outreach and recruitment efforts. Since the last interview, what outreach efforts were conducted to inform parents/caregivers about the demonstration? What methods were used (e.g., mail, flyers, radio, television)? What were the main messages in outreach materials? How widely were materials and information distributed? In what ways are outreach efforts different for households already enrolled in SNAP/WIC versus those that are not? If so, what were they? What organizations were responsible for outreach? Were materials distributed in multiple languages and/or were media efforts targeted to subpopulations of non-English speaking households? If so, what languages? [GET COPIES OF MATERIALS]

[NOTE TO VISITOR, ASK FOLLOWING QUESTIONS IF CONSENT PROCESS WAS NOT YET COMPLETE AT THE TIME OF THE LAST SITE VISIT]

H.2. Consent collection process. When did process of obtaining consent from households end in the spring? Did the process go as expected? Did it take less or more time than expected?

H.3. Maximizing consent response, [IF ACTIVE CONSENT REQUIRED]. What efforts were made to maximize response rates from households? What procedures were in place to contact nonresponsive households? How often were households contacted? In what ways? Were there procedures in place for refusal conversions? If so, what were they? Were special outreach materials or information provided to encourage participation in the demonstration? Please describe them. Who led this outreach and contacted the nonrespondents (e.g., state/local agency, CBOs, school staff)? What challenges did you face in encouraging household response? Did you overcome them and how? What kind of technical assistance or support would have been helpful?

H.4. Tracking and managing consent distribution and receipt. What approach was used to track consent distributions and receipts? How were returned letters handled, if addresses were incorrect? How were re-mailing or follow-up needs assessed, if returned forms were incomplete or missing information? How were the data stored and cleaned? Who had access to the data? What aspects of consent management and tracking were successful? What aspects needed improvement? Why? What advice would you give others about tracking and managing consent information? What kind of technical assistance or support would have been useful?

H.5. Staff responsibilities related to consent. How were consent management responsibilities divided among organizations and staff? Who had primarily responsibility for managing the production, distribution, and collection of consents? Who was responsible for tracking information about consent mailings and returns (such as the number of letters that have been mailed out, returned as undeliverable because of incorrect addresses, positive responses [active consent only], refusals)?

H.6a. Consent return rates [IF ACTIVE CONSENT REQUIRED] for active sites. How many households were sent in the initial mailing? What percentage of households that were sent mailings returned consent forms with complete information? What percentage returned forms with incomplete or missing information? What percentage of letters were returned undeliverable? What percentage of households received multiple contacts? How successful were efforts to contact nonresponsive households?

H.6b. Consent return rates [IF PASSIVE CONSENT REQUIRED]. How many consent packages were sent in the initial mailing? What percentage of households that were sent mailings opted out or responded that they did not want to be part of the demonstration? What percentage of letters were returned undeliverable? How successful were efforts to encourage these households to reconsider?

H.7. Transfer of information. How was contact information on consenting households transferred confidentially between leading agencies and the evaluation team? What information was sent to whom? What challenges did you encountered to this transfer of information? What types of support would have helped overcome these challenges?

Informing households about Random Assignment Results AND ENROLLING THEM IN [SEBTC PROGRAM NAME]

[NOTE TO VISITOR, ASK FOLLOWING QUESTIONS IF NOTIFICATION PROCESS WAS NOT YET COMPLETE AT THE TIME OF THE LAST SITE VISIT]

I.1. Informing households that were not selected for [SEBTC PROGRAM NAME]. How were households that were eligible for [SEBTC PROGRAM NAME] but were not chosen for participation in the demonstration informed (e.g., letter)? What information was provided? Were they given materials about other summer feeding options, [SNAP/WIC], nutrition education, or other topics? If so, what was provided? [GET COPIES OF MATERIALS] What was parents’ response to these materials? Did you receive any complaints about households not being selected to receive benefits? If so, how many? Who responded to these households and how?

I.2. Informing households that were selected for [SEBTC PROGRAM NAME]. How were households that were selected for participation in the demonstration informed (e.g. letter)? We will ask more detailed questions about the materials later, but can you provide a brief description? What was parents’ response to these materials? Did you receive comments that households were pleased to be accepted? If so how many?

4. Enrolling households in the [SEBTC PROGRAM NAME] database. What steps in the process of implementing the household database for [SEBTC PROGRAM NAME] took place after the last interview? Who was involved with this process? How long did this take? What activities took the most effort? What issues were encountered in completing implementation? How do data flow between the grantee, the EBT vendor, and the evaluation contractor? Who manages database operations? What operational issues have been encountered? What changes are you considering for next year?

5. Technical support to site operations. Who provides technical support for site-level operations, such as issuing cards and resolving problems requiring in-person contact? What are the main technical support requirements? What issues were encountered? What changes are you considering for next year?

distributing ebt cards⁵

J.1. EBT card distribution. Were new EBT cards distributed according to plan (method, time frame)? If not, what deviation was there and why? If cards were available for individual children, how often were they issued this way vs. for the household?

J.2. Additional materials sent with EBT cards. Did the distribution of materials with EBT cards proceed according to plan? If not, what deviation was there in content or media, and why? [GET COPIES] What was the participant response to the materials (questions, concerns, kudos)?

J.3. Challenges with EBT card distribution. Were there any problems with the method for distributing the EBT cards to households? Were there problems with the cards not reaching the families (due to lack of contact information, families moved, incorrect addresses, etc.)? How often did this occur? How was returned mail handled (due to incorrect addresses, moves, etc.)? How many households with returned mail subsequently got cards? If cards had to be picked up, what was the no-show rate? Were any changes in the process needed to address the distribution issues? Did the changes resolve the problems? Would you suggest changes to the process for next summer?

J.4. EBT card activation. What was the process for households to activate the card (e.g., activated upon receipt or upon use, or call a number to activate.)? Was this the same process used for [SNAP/WIC]? Who did the household interact with to activate their card (e.g., eligibility worker, vendor, automated)? Were Personal Identification Numbers (PIN) assigned, or did households select their initial PIN? How were PINs distributed securely? Could households change their assigned PIN?

J.5. Challenges with EBT card activation. Were there any problems with households’ ability to activate their cards? Did the overall process work as expected? Would you suggest changes to the process for the next summer?

J.6. Household responses to EBT card activation. Did households generally understand what the new cards were, how to activate them, and how to use them? Was the branding on the card clear (for households, retailers)? Were there any issues that arose related to using one card versus multiple for the food programs (by households, retailers, vendor, etc.)? Would you suggest making any changes to the EBT cards for the next summer?

replacing ebt cards and other participant support

K.1. Process for replacing cards. If the household’s EBT card is lost, stolen or damaged, what is the process for replacement? Who does the household contact? What information is needed to receive a replacement card? Who sends the new card? How long does it take for the household to receive a replacement card? Is the activation process the same as described earlier for the initial card? If different, how and why? What, if any, delay is there in the benefits being loaded onto the new card? Will the card have the same ID number as the previous card? Is there any fee for replacement? Will the households retain and use the same card each summer or will new cards be issued each year? Is this process similar to the process used for [SNAP/WIC] replacement cards? If different, how and why?

K.2. Extent of card replacements. How often have you had to replace EBT cards?⁶ What are the most common reasons? Are there any issues with the process for replacing the cards? Do households seem to understand the process? Would you make any changes to the process for the next summer?

K.3. Participant support. Do households have access to online or telephonic account information for the [SEBTC PROGRAM NAME] card, as they would for [SNAP/WIC]? Is account information provided on transaction receipts when purchases are made at participating retailers? What type of information can households access (e.g., balance inquiry, PIN selection, report problems)? Is there a contact number for households to call if they have questions about the program, benefits, or card? Who are they contacting (e.g., eligibility worker, CBO, vendor, automated)? What are the hours of operation for the customer service number? Is there any face-to-face support offered? Who provides the assistance (e.g., eligibility worker, CBO)?

K.4. Extent of participant support provided. How often do households use the additional supports (account information and customer service)? Have there been any issues with the process? Do households seem to understand the process? Would you make any changes to the process for the next summer?

preventing and detecting abuse

L.1. Process for preventing and detecting abuse. What is the process for detecting and preventing abuse of [SEBTC PROGRAM NAME] funds? What type of monitoring system does the state have in place to detect abuses (e.g., computer programs that monitor EBT transactions at retailers)? Is there a system for fraud reporting and claims? How are claims and suspected fraud investigated?

L.2. Extent of reported fraud and abuse. Has the process worked as expected? Have you had to make any changes to the process to better detect and prevent abuse? Have you received reports of fraud or abuses? Have your monitoring systems detected abuses by retailers or households? What kinds of fraud and abuse have been identified? How prevalent has fraud and abuse been among the demonstration population?

Expunging funds

M.1. Timing and process for expunging funds – SNAP approach. When will unused funds be expunged from accounts for [SEBTC PROGRAM NAME]? What process is being used to expunge funds from the cards? Is this similar to the process used for SNAP when the cards are inactive? What happens to unused funds? What notification or reminders will households receive before their benefits are expunged?

M.2. Expiration of benefits – WIC approach. What are the rules for expiration of [SEBTC PROGRAM NAME] benefits? Are the rules the same as for WIC? How does the State account for expired benefits? Do households receive any notification or reminders before their benefits expire?

M.3. Challenges with expunging funds. Has the process for expunging funds/benefit expiration worked as anticipated? Have there been any problems with expunging funds for the [SEBTC PROGRAM NAME] benefits? Did you have to change the process after implementation? Is there anything you would do differently for the next summer?

M.4. Deactivating SEBTC cards. When will [SEBTC PROGRAM NAME] cards be deactivated?

SECTION N: training for schools AND OTHER COMMUNITY PARTNERS

N.1. Types and extent of training. Did plans for providing training to schools and community partners change since the last site visit? If so, how (e.g., format, organizations involved, number or types of staff included, timing)? Why were these changes made? What training was provided and to whom?

N.2. Experiences providing training. Was the level of training appropriate? How much follow-up training/support was needed? Overall, was more training needed/offered? Was the type of training appropriate? Would you suggest conducting more/less of certain types? How did the school staff respond to the training? Would you suggest making changes to the training (i.e., content, timing, length, location) for next summer?

N.3. Experiences of school staff and community partners with training. What type of training did you receive (e.g., information materials distribution, one-on-one, group, computer)? What was the focus of the training (e.g., related to developing lists of eligible children, the use of [SEBTC PROGRAM NAME] benefits and EBT cards, outreach and education to eligible households)? Who conducted the training? How long was the training? Was there any follow-up training provided? Was the level of training sufficient to understand the program and complete your role in the demonstration? Was the type of training appropriate? Would you suggest conducting more/less of certain types? What changes would you suggest the state makes to the training in the future (i.e., content, timing, length, location)?

SECTION O: training for parents on [SEBTC PROGRAM NAME] benefits

O.1. Training for parents. Beyond the information letter that will be sent to parents about participating in the [SEBTC PROGRAM NAME] program, did parents in the demonstration sites receive training or additional information packets? Did materials or training differ for certain populations (e.g., [SNAP/WIC] recipients versus non-recipients)?

O.2. Content of materials and training. What was the focus of the materials or training (e.g., information about the [SEBTC PROGRAM NAME], what the EBT card is and how to use it (including activation), what types of foods are allowed, locations for accessing benefits, who should receive the benefit (children))? Did the materials or training provide broader information about eligibility for other programs (e.g., SNAP, WIC, summer food programs in the community)? Did the materials or training address potential concerns about immigration status and benefit receipt? Did the training include information on nutrition education (if so, we will talk in more detail about this later)?

O.3. Format of training. How was materials or training be provided (e.g., information packets, one-on-one, classes, computer, web-based)? Who will prepare and distribute materials? Did households need to attend training to receive their EBT card? Where were trainings conducted? If you used training classes, how large were the classes? Who conducted training? How long did the training classes last? How often was the training held? How was the training structured? Were materials and training distributed in multiple languages? Which languages were included? Was follow-up training planned?

O.4. Partner agencies’ experiences with training. Was the level of training appropriate? How much follow-up training/support was needed? Overall, was more training needed/offered? Was the type of training appropriate? Would you suggest conducting more/less of certain types? For classes, was the size and turnout about what you thought it would be? Where were the classes located? Were those locations convenient for parents/guardians? When were the classes held? Were the times convenient for parents/guardians? How did the parents respond to the training? Would you suggest making changes to the training (i.e., content, timing, length, location) for the next summer?

O.5. Additional support for parents. Do households have access to an 800 telephone number to answer questions about the demonstration? Who answers those calls? Is there a location where household members can go to get help (face-to-face)? Who provides that help (e.g., local benefit office, CBO, school staff)? How convenient are these locations for parents/guardians? Is there a website or webpage dedicated to [SEBTC PROGRAM NAME] information? How is access controlled? What other types of media were used to train/inform participants (e.g., training videos, media campaigns, radio ads)?

O.6. Extent of additional support provided to parents. How often did parents use the customer service number? How long were wait times (can you track)? Did questions arise from these calls that you would consider including in the overall parent training? How many parents used the website (can you track)? Did you receive any feedback on the effectiveness of the various types of media training? Were there any issues with the overall customer support process? Would you make any changes to the process for the next summer?

SECTION P: training and support for retailers

P.1. Retailer training. Since the last interview, has information/training been provided to retailers? Have all [SNAP/WIC] retailers in the demonstration areas received information/training? [For WIC sites] Have the changes to the WIC food package for [SEBTC PROGRAM NAME] households required additional retailer training?

P.2. Type of material or training. What has been the focus of the materials or training (e.g., information about the [SEBTC PROGRAM NAME], the look of the EBT card, use of the EBT card, the new WIC package)? What type of materials or training has been provided (e.g., information packets, one-on-one, group, computer, web-based)? Who prepared and distributed materials? Were materials or training offered in multiple languages? Which languages were included? Have face-to-face trainings been conducted at the retailer’s location or with multiple retailers? Who conducts the training? How long did the training classes last? How often are the trainings held? What retailer personnel were invited to the training? What type of information is discussed? What is the structure? Is follow-up training provided or planned?

P.3. Extent of retailer training provided. Was the level of training appropriate? How much follow-up training/support was needed? Overall, was more training needed/offered? Was the type of training appropriate? How did the retailers respond to the training? Did any issues arise that were not anticipated or addressed through training (stocking issues due to high demand, concerns/reactions from retailers about the new program, problems with cashiers recognizing and accepting cards)? Would you suggest making changes to the training (i.e., content, timing, length, location) for the next summer?

P.4. Additional retailer support. Are retailers expected to use the existing EBT retailer service number? Who answers this number (vendor personnel, automated, State, etc)? What are the hours of operation? Is there a special customer service/contact number for retailers to call if they have questions about the [SEBTC PROGRAM NAME]? Who answers this number (vendor personnel, automated, State, etc.)? What are the hours of operation for the special [SEBTC PROGRAM NAME] retailer service number? Are any other supports provided for retailers? Describe.

P.5. Extent of additional retailer support provided. How often did retailers use the customer service number? Did questions arise from these calls that you would consider including in the overall retailer training? Were there any issues with the customer service process? Would you make any changes to the process for the next summer?

P.6. Retailer response to structure of training. What type of training did the stores that you represent receive (e.g., information materials distribution, one-on-one, group, computer)? What was the focus of the training (e.g., information about the [SEBTC PROGRAM NAME], the EBT look and use, the new WIC package)? How long was the training? How accessible and effective was it? Was there any follow-up training provided? If so, what and how?

P.7. Retailer response to usefulness of training. Overall, how well did training prepare stores for the demonstration, both inside the demonstration site and beyond? Did the training/materials cover all that retailers needed to know? Was the type of training appropriate? Would you suggest conducting more/less of certain types? What changes would you suggest the state makes to the training in the future (i.e., content, timing, length, location)? Did store managers provide additional training to your staff/cashiers? What type of training did they conduct? What information did they share? Were there any problems with staff understanding the program or allowing households to use their new cards? [For WIC sites] Did the special [SEBTC PROGRAM NAME] version of the WIC food package cause any problems for you or your systems? Was follow-up training needed during the demonstration period?

P.8. Retailer response to additional support. How often did stores need to contact the retailer customer service line with questions or problems related to [SEBTC PROGRAM NAME]? Was the level of service better than, the same as, or worse than the regular [SNAP/WIC] calls, in terms of wait times and quality of service? What recommendations do you have, if any, for improving retailer customer service for [SEBTC PROGRAM NAME]?

NUTRITION EDUCATION

Q.1. [SEBTC PROGRAM NAME] nutrition education description. Can you describe the nutrition education being offered to household members in the demonstration area? What are the key elements, in terms of the goals, planning, nutrition messages, tools, and format? Are comparable activities provided across the entire demonstration area? If there is variation across local areas or agencies, please describe that variation. What guidelines or requirements do you have in place for the demonstration as a whole and at the local level? Did plans for this education changes since the last site visits?

Q.2. Target groups for nutrition education. What individuals have been targeted for nutrition education? Is there nutrition education offered only to households that receive the [SEBTC PROGRAM NAME] benefit? If so, what are the methods and messages, and how many households have received this training? Were subgroups of individuals targeted for particular education components (for example children only, parents/guardians but not children, etc.)? What criteria do you use to identify subgroups? How often do parents/guardians and children receive education and through what modes?

Q.3. Promoting nutrition education services. What activities were used to promote the awareness and use of nutrition education among households and children (for example, have you distributed information, provided support services, or made referrals)? Why did you to select these promotion activities? Do you perceive these activities as helpful in encouraging targeted households to understand or take advantage of the education services?

Q.4. Adaptations to typical [SNAP/WIC] nutrition education. Does the nutrition education you are offering to parents, guardians, and children differ from what you offer as a part of typical [SNAP/WIC] services? What aspects are similar, and what aspects are different in terms of the delivery of education and materials used? Why are there differences between typical [SNAP/WIC] nutrition education and [SEBTC PROGRAM NAME] nutrition education? How were staff advised of the change? Were staff trained to use the adapted materials?

Q.5. Staff and resources to develop and provide [SEBTC PROGRAM NAME] nutrition education. Who are the staff members that deliver the nutrition education? How many staff members provide education? How are the messages delivered (e.g., written materials, group presentations, videos, etc.)? Are materials geared toward adults or children? Are there any special resources or partnerships that have helped to provide this nutrition education? What are they? Are there referrals to sources of more in-depth nutrition counseling in the community?

Q.6. Attitudes towards [SEBTC PROGRAM NAME] nutrition education. What has been the reaction of the staff towards the nutrition education component (e.g., receptive, enthusiastic, indifferent, etc.)? What have participants’ reactions been? Are there particular components or modes of education that they prefer? If so, for what reasons?

Q.7. Successes and challenges of [SEBTC PROGRAM NAME] nutrition education. How successful has this nutrition education component been in meeting your goals? What has been successful about the planning process and delivery of education? What factors have led to these successes? What challenges have you faced? What are you and local agencies doing to overcome these challenges? What kinds of improvements could be made?

Successes, Challenges, and Solutions

R.1. Goals. Did [STATE] have to adjust its goals or expectations during the course of demonstration implementation? If so, what specifically changed? Why was this necessary? How was this accomplished?

R.2. Perceived impacts of the demonstration. To what extent do you believe the demonstration has affected children’s food security during the summer? To what extent has it affected child nutrition during the summer? To what extent has it affected household food security? To what extent has it affected new applications for [SNAP or WIC]? To what extent has it affected other summer feeding programs such as the SFSP, or emergency food services such as food banks and kitchens? Did they experience an increase or decrease in demand for food? Are effects more apparent for certain groups of populations? If the [STATE] implemented services beyond the EBT benefit, to what extent does it appear that these services affected children’s outcomes? What events during the summer outside of the demonstration may have affected its impacts?

R.3. Successes. Thus far, what have been the greatest successes of the demonstration? What factors contributed significantly to this success? What, if anything, could have been done differently to achieve greater improvements in children’s food security?

R.4. Challenges. Thus far, what have been the biggest challenges to meeting the goals of the demonstration? Have these challenges been resolved? If so, how? If not, why? What could the [GRANTEE] or other agencies done different to meet demonstration goals more effectively? How have these challenges affected the effectiveness of the demonstration?

CLOSING

Is there anything you think is important for the FNS to know about the [STATE]’s [SEBTC PROGRAM NAME] demonstration that we did not ask about?

Thank you for your time and helpful feedback. The information you have shared will be valuable to our team as we look across states and localities for themes and ideas that we can share with FNS.

Are your any questions you have for me before we finish?

END OF INTERVIEW

OMB Clearance Number: Expiration Date:

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is xxxxxx. The time required to complete this information collection is estimated to average 90 minutes per response. If you have any comments concerning the accuracy of time estimates or suggestions for improving this form, please contact: U. S. Department of Agriculture, Food and Nutrition Service, Office of Research & Analysis, Alexandria, VA 22302.

Evaluation of the SEBT for Children

Implementation Study Protocols

September Telephone Calls

February 3, 2011

INTRODUCTION

My name is [X], from [Abt Associates/Mathematica Policy Research].

As you may know, Abt and Mathematica are working together to evaluate the Summer EBT for Children demonstrations for the Food and Nutrition Service of USDA. My colleagues and I are conducting telephone interviews with a wide range of stakeholders involved in the demonstration in order to learn about the process of implementation, the challenges you encountered, and lessons learned. I want to start by thanking you for taking time to speak with us today. Your perspective and insights on these issues are very helpful.

Our reports to FNS will describe the range of responses expressed by staff, and may list the names of agencies and partners who contributed information, but we will not quote you or anyone by name or title. However, because of the relatively small number of organizations participating in the study, there is a possibility that a response could be correctly attributed to you.

I expect our conversation will take approximately [XX] minutes.

[OPTIONAL IF INTERVIEWER CHOOSES TO RECORD:] I want to be sure I am keeping track of everything you are saying. May I record our discussion so that I can listen to it later when I write up my notes? No one outside of our research team will have access to the recording. [IF YES:] Thank you. It will be helpful if you speak up, speak clearly, and speak one at a time. [IF NO:] That’s no problem. I’ll take notes as you talk, but I may sometimes need to ask you to slow down or repeat so that I can get all the information.

First, do you have any questions for me about the project in general or what we will be discussing today?

NOTE TO INTERVIEWERS: THIS GUIDE IS ORGANIZED ACCORDING TO TOPIC AREAS, TO BE ASKED OF SPECIFIC RESPONDENTS ACCORDING TO THE MASTER TOPIC LIST. HOWEVER, IT IS EXPECTED THAT SOME OF THE RESPONDENTS WILL NOT BE KNOWLEDGABLE ABOUT TOPICS ASSIGNED TO THEM, DEPENDING UPON THE SPECIFIC DEMONSTRATION. IF A RESPONDENT DOES NOT HAVE KNOWLEDGE OF THE TOPIC, SKIP TO THE NEXT TOPIC ASSIGNED TO HIM/HER.

Project Organization and Management

D.1. Organizational structure to administer the demonstration. Did the mix of organizations involved in the demonstration or the organizational structure change since the last interview? If so, how and why? Does this include both organizations that are receiving grant funds and those that are not? What were the main advantages to the organizational structure used in [STATE]? What aspect(s) would you recommend that other demonstration programs adopt? Why?

D.2. Communication between grantees and key partners. Thinking back over the course of the spring and summer, have there been any challenges to maintaining communication with key partner agencies or organizations? In what ways and for what areas could communications be improved? What forms of communication and collaboration have been most helpful? In what ways?

D.3. Changes in staffing and staff turnover. How has the staffing structure changed, if at all, since the last interview? How and why? If there were temporary staff hired for the demonstration, when were/will they be terminated? Has there been other staff turnover? In what positions and for what reasons? What were the effects of this turnover? Overall, where were the strengths and challenges of staffing for the demonstration?

Involving Local Agencies and Community Organizations

E.1. Successes and challenges of the partnership. Thinking about local agencies and community organizations that served as partners but did not receive grant funds, what were the most effective elements of the partnership, and why? What have you learned about establishing and maintaining these partnerships? What do you view as the key logistical considerations that other state grantees and their partners would need to consider, if they were to implement a similar type of partnership? What has worked best in these partnerships? What partnership aspects could be improved? From which partners would you have liked more involvement?

E.2. Relationship of Summer EBT for Children with SFSP. Overall, how would you describe the relationship between the agencies operating [SEBTC PROGRAM NAME] and the agencies operating the SFSP, at the State and local levels? In what ways did [SEBTC PROGRAM NAME] affect attitudes of parents and children toward the SFSP? Was the SFSP more or less successful in the project site than in previous years, and why?

Coordination with EBT Vendor

1. EBT system preparations. Since the last interview, have there been any changes to the vendor’s EBT system or third-party processor (TPP) systems for [SEBTC PROGRAM NAME]? Who was involved with this process? How long did this take? What activities took the most effort? What issues were encountered in completing implementation? What operational issues have been encountered? What changes are you considering for next year? What recommendations about vendor EBT system or TPP system design or operations do you have for other sites operating Summer EBT for Children?

2. Point of sale (POS) system preparations. Since the last interview, have there been any changes to the POS system programming or data loads for [SEBTC PROGRAM NAME]? Who was involved with this process? How long did this take? What activities took the most effort? What issues were encountered in completing implementation? What operational issues have been encountered? What changes are you considering for next year? What recommendations about POS system design or operations do you have for other sites operating Summer EBT for Children?

3. Automated customer service system preparations. Since the last interview, have there been any changes to the interactive voice response (IVR) system, customer service scripts, or user website for [SEBTC PROGRAM NAME]? Who was involved with this process? How long did this take? What activities took the most effort? What issues were encountered in completing implementation? How well do the automated customer service systems (interactive voice response (IVR) system, customer service scripts, user website) meet customer needs? What operational issues have been encountered? What changes are you considering for next year? What recommendations about automated customer service systems do you have for other sites operating Summer EBT for Children?

Informing households about Random Assignment Results AND ENROLLING THEM IN [SEBTC PROGRAM NAME]

I.1. Enrolling households in the [SEBTC PROGRAM NAME] database. Since the last interview, have there been any changes to the structure and processing of the household database for [SEBTC PROGRAM NAME]? Why were changes needed? Who was involved with this process? How long did this take? What activities took the most effort? What issues were encountered? What operational issues have been encountered? What changes are you considering for next year? What recommendations about tracking and managing household data do you have for other sites operating Summer EBT for Children?

I.2. Technical support to site operations. Have there been any changes in technical support for site-level operations, such as issuing cards and resolving problems requiring in-person contact? Since the last interview, what issues were encountered? What changes are you considering for next year? What recommendations about technical support for site operations do you have for other sites operating Summer EBT for Children?

replacing ebt cards and other participant support

I’d like to turn now to some operational questions about customer service for [SEBTC PROGRAM NAME] households, focusing on the time since our last interview.

K.1. Extent of card replacements. During the late summer months, how often did you have to replace EBT cards that were lost, stolen, or damaged? What were the most common reasons? Were there any issues with the process for replacing the cards? Did households seem to understand the process? How long did it take for households to receive replacement cards? Would you make any changes to the process for the next summer?

K.2. Other participant support provided. Were there any changes to participant supports? If so, how and why?

preventing and detecting abuse

L.1. Process for preventing and detecting abuse. Did the process work as expected? Were there any changes in late summer to the process for detecting and preventing abuse of [SEBTC PROGRAM NAME] funds? If so, how and why?

L.2. Extent of reported fraud and abuse. How many reports of fraud or abuse were received? What were the main types of reports or allegations? How many were substantiated? How many claims were established, and in what amount? How much was recovered? What other enforcement actions were taken against households, retailers, or others? What lessons did you learn from this process about assuring the integrity of Summer EBT for Children?

DEMONSTRATION PHASE-OUT

M.1. Process for expunging funds—SNAP approach. What process was used to expunge unused benefits cards? Was this similar to the process used for SNAP when the cards are inactive? When did this happen? Did the timeline change? If so, why? Is the process complete? If so, when did you complete it? If not, when do you anticipate finishing? What happened to unused funds? Did households receive any notification or reminders near the end of the summer that the benefits would be forfeited if not used? If so, how and when?

M.2. Expiration of benefits—WIC approach. Were there any changes to the rules for expiration of benefits? If so, how and why?

M.3. Deactivation of cards. Were EBT cards deactivated, and if so how was this done? Did households receive any notification or reminders near the end of the summer that the demonstration was ending and cards would become inactive? If so, how and when? Will households that participated in the summer keep the same card for use in future summers or will they be issued a new card?

M.4. Challenges with closing out SEBT for Children accounts. Did the process for closing out accounts work as anticipated? Were there any problems with the process? Did you have to change the process from what you planned? What were the responses of households? Is there anything you would do differently for the next summer?

training for parents on [SEBTC PROGRAM NAME] benefits

[NOTE TO VISITOR: ASK FOLLOWING QUESTIONS IF SITE HAD HOTLINE OR OTHER SUPPORTS FOR HOUSEHOLDS]

O.1. Extent of additional support provided to parents. How often did parents use the customer service number? How long were wait times (can you track)? Did questions arise from these calls that you would consider including in the overall parent training? How many parents used the website (can you track)? Have there been any issues with the process? What feedback did you get about participant support or the effectiveness of the various types of media/training? Were there any issues with the overall customer support process? Would you make any changes to the process for the next summer?

training and support for retailers

[NOTE TO VISITOR: ASK FOLLOWING QUESTIONS IF RETAILER TRAINING AND OTHER SUPPORT PROVIDED]

P.1. Extent of retailer training provided. Was the level of retailer training for the demonstration appropriate? How much follow-up training/support was needed? Overall, was more training needed/offered? Was the type of training appropriate? How did the retailers respond to the training? Did any issues arise that were not anticipated or addressed through training (stocking issues due to high demand, concerns/reactions from retailers about the new program, problems with cashiers recognizing and accepting cards)? Would you suggest making changes to the training (i.e., content, timing, length, location) for the next summer?

P.2. Extent of additional retailer support provided. How often did retailers use the customer service number for questions related to [SEBTC PROGRAM NAME]? Did questions arise from these calls that you would consider including in the overall retailer training? Were there any issues with the customer service process for retailers? Would you make any changes to the process for the next summer?

P.3. Retailer response to structure of training. Since the last interview, was any additional training for [SEBTC PROGRAM NAME] provided to the stores that you represent? If so, who provided it, and what type of training occurred (e.g., information materials distribution, one-on-one, group, computer)? What was the focus of the training (e.g., information about the [SEBTC PROGRAM NAME], the EBT look and use, the new WIC package)? How long was the training? How accessible and effective was it? Was there any follow-up to the training? If so, what and how?

P.4. Retailer response to usefulness of training. Overall, how well did training prepare stores for the demonstration, both inside the demonstration site and beyond? Did the training/materials cover all that retailers needed to know? Was the type of training appropriate? What changes would you suggest the state makes to the training in the future (i.e., content, timing, length, location)? Did store managers provide additional training to staff/cashiers? What type of training did they conduct? What information did they share? Were there any problems with staff understanding the program or allowing households to use their new cards? [For WIC sites] Did the special [SEBTC PROGRAM NAME] version of the WIC food package cause any problems for you or your systems? Was follow-up training needed during the demonstration period?

P.5. Retailer response to additional support. How often did stores need to contact the retailer customer service line with questions or problems related to [SEBTC PROGRAM NAME]? Was the level of service better then, the same as, or worse than for regular [SNAP/WIC] calls, in terms of wait times and quality of service? What recommendations do you have, if any, for improving retailer customer service for [SEBTC PROGRAM NAME]?

Successes, Challenges, and Solutions

R.1. Goals. Did [STATE] have to adjust its goals or expectations during the course of demonstration implementation? If so, what specifically changed? Why was this necessary? How was this accomplished?

R.2. Perceived impacts of the demonstration. To what extent do you believe the demonstration affected children’s food security during the summer? To what extent did it affect child nutrition during the summer? To what extent did it affect household food security? To what extent did it affect new applications for [SNAP or WIC]? To what extent did it affect applications for NSLP or SBP? To what extent did it affect other summer feeding programs such as the SFSP, or emergency food services such as food banks and kitchens? Did they experience an increase or decrease in demand for food? Are effects more apparent for certain groups of populations? If the [STATE] implemented services beyond the EBT benefit, to what extent does it appear that these services affected children’s outcomes?

R.3. Successes. What have been the greatest successes of the demonstration? What factors contributed significantly to this success? What, if anything, could have been done differently to achieve greater improvements in children’s food security during the summer?

R.4. Challenges. What have been the biggest challenges to meeting the goals of the demonstration? Have these challenges been resolved? If so, how? If not, why? What could [STATE] done different to meet its goals more effectively?

CLOSING

Is there anything you think is important for the FNS to know about the [STATE]’s [SEBTC PROGRAM NAME] demonstration that we did not ask about?

Thank you for your time and helpful feedback. The information you have shared will be valuable to our team as we look across states and localities for themes and ideas that we can share with FNS.

Are your any questions you have for me before we finish?

END OF INTERVIEW

1 Questions about WIC retailers will be addressed using State WIC Agency reports to the extent possible.

2 Questions about SNAP retailers will be addressed using FNS reports to the extent possible.

3 If reports from the state on SFSP (data on sponsors, sites) for 2010 are available, discussion will confirm accuracy of reports and focus on questions not addressed by reports.

4 Note to visitor: if process for distributing EBT cards, replacing cards, or other EBT operations is specified in proposal, confirm and clarify as needed.

5 Note to visitor: if process for distributing EBT cards, replacing cards, or other EBT operations is specified in proposal, confirm and clarify as needed.

6 If possible, this information will be obtained from EBT customer service reports as part of the administrative data collection.