Business- Profit & Not -for- Profit

SEBTC Evaluation

July 2011 Site Visits

Interview with EBT Processors

[Note to site visitors:] Unless otherwise noted, all questions will be asked of a single respondent only. Site visitors will identify the roles of respondents during the site visit planning process and determine which respondent should address each major topic area. The site visit guide also indicates questions within each topic area that should be asked of more than one respondent.

INTRODUCTION

My name is [X], from [Abt Associates/Mathematica Policy Research].

As you may know, Abt and Mathematica are working together to evaluate the Summer EBT for Children demonstrations for the Food and Nutrition Service of USDA. My colleagues and I are visiting each of the five sites and collecting information from a wide range of stakeholders involved in the demonstration in order to learn about the process of implementation, the challenges you encountered, and lessons learned. I want to start by thanking you for taking time to speak with us today. Your perspective and insights on these issues are very helpful.

For stakeholders not receiving grant funds: Your participation is voluntary, and your responses will be kept confidential to the extent provided by law. You may refuse to answer any question and may stop the interview at any time.

Our reports to FNS will describe the range of responses expressed by staff, and may list the names of agencies and partners who contributed information, but we will not quote you or anyone by name or title. However, because of the relatively small number of organizations participating in the study, there is a possibility that a response could be correctly attributed to you.

I expect our conversation will take approximately 120 minutes.

[OPTIONAL IF INTERVIEWER CHOOSES TO RECORD:] I want to be sure I am keeping track of everything you are saying. May I record our discussion so that I can listen to it later when I write up my notes? No one outside of our research team will have access to the recording. [IF YES:] Thank you. It will be helpful if you speak up, speak clearly, and speak one at a time. [IF NO:] That’s no problem. I’ll take notes as you talk, but I may sometimes need to ask you to slow down or repeat so that I can get all the information.

First, do you have any questions for me about the project in general or what we will be discussing today?

A. BACKGROUND ON RESPONDENT

[Ask this section of all new respondents]

A.1 What organization do you work for?

A.2 What services does [ORGANIZATION] provide?

A.3 What is your position? What are your day-to-day responsibilities?

A.4 How long have you worked for [ORGANIZATION]?

A.5 When did you begin working on [SEBTC PROGRAM NAME]?

A.6 What are your specific responsibilities?

B. Demonstration Planning

Goals of the demonstration.

B.1 Since the last site visit in [MONTH], have the goals of the demonstration changed?

[If so, probe:]

- How?

- For what reasons?

- What new goals do you hope to achieve?

- To what extent are the goals quantifiable?

- How will they be measured?

C. Local Context

Food environment.

C.1 Since the last site visit, has the food environment in the demonstration area changed (such as the number of [SNAP/WIC] retailers, the types of retailers, or local access to those retailers)?

[If so, probe:]

- In what ways?

- For what reasons?

- Are more or fewer resources available to local households?

- What types?

D. Basic parameters of the summer EBT MODEL

EBT cards.

C.1 Since the last site visit, have there been changes to the approach for the distribution of EBT cards? If so, please explain.

Timeframe of EBT benefits.

C.2 Since the last site visit, have there been any changes to the schedule for issuing EBT benefits?

[If so, probe:]

- Can you explain the changes?

- Why were the changes made?

- What has been the reaction of households?

D. Project Organization and Management

Organizational structure to administer the demonstration.

D.1 Has your organization’s structure for administering the demonstration changed since the last site visit?

[If so, probe:]

- Can you please describe how?

D.2 Are any new units or subcontractors participating?

[If so, probe:]

- What are their responsibilities?

- How were they selected?

D.3 Did any units or subcontractors stop working on the demonstration?

[If so, probe:]

- Why?

- Which organizations have taken over their responsibilities?

D.4 What are the main advantages to the current organizational structure?

[Probe:]

- What aspect(s) would you recommend that other demonstration programs adopt?

- Why?

Agreements between organizations.

D.5 Has your firm’s contract with [GRANTEE] changed since the last visit?

[If so, probe:]

- How

- Why?

- Can we have copies of the new agreements?

Staffing structure

D.6 What is your current staffing structure for administering demonstration services?

[Probe:]

- How many staff members are involved?

- What roles do key staff members play? What type of backgrounds and qualifications do they have?

- What roles do support staff play? What type of backgrounds and qualifications do they have?

- Are there shared responsibilities across organizations? Who leads or coordinates them?

D.7 Has there been turnover among key staff?

[If so, probe:]

- In what positions and for what reasons?

- What were the effects of this turnover?

D.8 Were any new key staff members hired since the last visit for the demonstration? Which ones?

Communication with grantee and key partners.

D.9 What levels of staff communicate with the grantee and other lead agencies?

D.10 How often? For what reasons? What topics are discussed?

D.11 In what format are meetings held (large group meetings, conference calls, one-on-one with each partner)?

D.12 Have there been challenges to maintaining communication? In what ways and for what areas could communications be improved?

D.13 What forms of communication and collaboration have been most helpful? In what ways?

E. Informing households about Random Assignment Results

AND ENROLLING THEM IN [SEBTC PROGRAM NAME]

Enrolling households in the [SEBTC PROGRAM NAME] database.

E.1 Since the last interview, were additional steps needed to officially enrolled households assigned to the demonstration in the program?

[Probe:]

- Did you have to make any modifications to the database for this demonstration?

- What child or household information had to be entered?

- Who was responsible for entering this data?

- How long did it take?

E.2 Since the last visit, did you encounter any new challenges while enrolling households?

[If so, probe:]

- Can you describe them?

- Were you able to resolve these challenges? How?

- What would you do differently in the future if the program were continued?

F. EBT SYSTEMS MODIFICATIONS AND OPERATIONS

EBT system modifications.

F.1 Since the last interview, what modifications were made to the EBT system or third-party processor (TPP) systems for [SEBTC PROGRAM NAME]?

[Probe:]

- Who was involved with this process?

- How long did this take?

- What activities took the most effort?

- What issues were encountered in completing implementation?

EBT system interfaces.

F.2 Since the last interview, what modifications were made to system interfaces with the [SEBTC PROGRAM NAME] eligibility or certification system?

[Probe:]

- Who was involved in this process?

- How long did this take?

- What activities took the most effort?

- What issues were encountered?

EBT system settlement and reconciliation.

F.3 Since the last interview, what changes were made to the settlement and reconciliation of the [SEBTC PROGRAM NAME] EBT system?

[Probe:]

- Who was involved in this process?

- How long did it take?

- What activities took the most effort?

- What issues were encountered?

EBT retailer coordination.

F.4 Since the last interview, what changes were made the way that EBT retailers and third party processors participate in [SEBTC PROGRAM NAME]?

[Probe:]

- Why were the changes needed?

- Who was involved in this process?

- How long did it take?

- What activities took the most effort?

- What issues were encountered?

Point of sale (POS) system.

F.5 Since the last interview, what changes were made to the POS system programming or data loads for [SEBTC PROGRAM NAME]?

[Probe:]

- If programming or data loads were required, were you able to do this remotely?

- Who was involved with this process?

- How long did this take?

- What activities took the most effort?

- What issues were encountered in completing implementation/changes?

Automated customer service system.

F.6 Since the last interview, what changes were made to the interactive voice response (IVR) system, customer service scripts, or user website for [SEBTC PROGRAM NAME]?

[Probe:]

- Who was involved with this process?

- How long did this take?

- What activities took the most effort?

- What issues were encountered in completing implementation?

- How well do the automated customer service systems (interactive voice response (IVR) system, customer service scripts, user website) meet customer needs?

Operational issues.

F.7 Since the start of operations for [SEBTC PROGRAM NAME], what operational issues have been encountered?

[Probe:]

- What parts of system operations were affected? [host, TPP, reconciliation/settlement, POS, IVR, customer service]

- How were service levels affected?

- How were the problems addressed?

- Are these problems resolved or ongoing?

- What changes are you considering for next year?

G. distributing ebt cards¹

EBT card distribution.

G.1 Were new EBT cards distributed according to plan (method, time frame)?

[If not, probe:] What deviations were there and why?

Additional materials sent with EBT cards.

G.2 Did the distribution of materials with EBT cards proceed according to plan? [If not, probe:] What deviation was there in content or media, and why? [GET COPIES]

G.3 What was the participant response to the materials (questions, concerns, kudos)?

Challenges with EBT card distribution.

G.4 Were there any problems with the method for distributing the EBT cards to households?

G.5 Were there problems with the cards not reaching the families (due to lack of contact information, families moved, incorrect addresses, etc.)?

[If so, probe:]

- How often did this occur?

- How was returned mail handled (due to incorrect addresses, moves, etc.)?

- How many households with returned mail subsequently got cards?

- If cards had to be picked up, what was the no-show rate?

G.6 Were any changes in the process needed to address the distribution issues?

G.7 Did the changes resolve the problems?

G.8 Would you suggest changes to the process for next summer?

Challenges with EBT card activation.²

G.9 Were there any problems with households’ ability to activate new EBT cards?

[If so, probe:]

- What types of problems?

- How common were they?

G.10 Were there any problems with issuing additional benefits to existing EBT cards?

[If so, probe:]

- What types of problems?

- How common were they?

G.11 Did the overall process work as expected?

G.12 Would you suggest changes to the process for the next summer?

Household responses to EBT card activation.

G.13 Did households generally understand the program and their benefits?

[Probe:]

- What the new benefits were?

- How much their benefit was?

- When the benefits would be available?

- How to activate new cards?

- How to use their cards, if they had not used EBT before?

G.14 [For sites with new cards] Were there any issues that arise as a result of distributing new cards?

[Probe:]

- Was the branding on the card clear (for households, retailers)?

- Did retailers have any confusion about how to process the cards?

G.15 [For sites that issued benefits to existing cards] Were there any issues that arose as a result of having benefits from multiple programs on the same cards?

[Probe:]

- Did households understand what portion of their benefits were from each program?

- Did households understand what benefits were being used first?

- [WIC sites] Did households understand when benefits from each program would expire?

- Did retailers have any confusion about how to process the cards?

G.16 Would you suggest making any changes to the EBT cards for the next summer?

Replacing EBT cards.

G.17 If the household’s EBT card is lost, stolen or damaged, what is the process for replacement?

[Probe:]

- Is the process the same as [SNAP/WIC]? If not, what is different?

- Who does the household contact?

- How long does the process take?

G.18 Is the rate of card replacements for households that just get [SEBTC PROGRAM NAME] benefits different from households that get [SNAP/WIC]?

[Probe:]

- Is the rate higher or lower for households that just get [SEBTC PROGRAM NAME] benefits?

- What are the reasons for this difference?

- [If there is a problem] What actions have been taken to reduce the rate of replacements?

- How has the strategy for issuing cards [one card/multiple cards per household] affected the rate of replacements?

H. training of households and other participant support

Participant support.

H.1 Do households call your firm’s customer service hotline if they have questions about the [SEBTC PROGRAM NAME] program, benefits, or card?

[Probe:]

- Is this contact number the same as [SNAP/WIC]?

- What are the hours of operation for the customer service number?

- What are the qualifications of customer service representatives?

- Have customer service representatives been trained on how to respond to questions specific to [SEBTC PROGRAM NAME]?

H.2 What happens if a customer service representative cannot answer a question?

[Probe:]

- Is there a local contact number? Who answers that line?

H.3 Is there a website or webpage dedicated to [SEBTC PROGRAM NAME] information? [If so:] How is access controlled?

Extent of participant support provided.

H.4 [Ask if respondent indicated that processor provided supports for households if they had questions about the program] How often do households use the customer supports (account information, customer service number, website)? Were you able to track this?

H.5 What are common questions from households?

[Probe:]

- Account balances

- Eligible foods

- Expiration of benefits

- Other issues?

H.6 Have there been any issues with the participant support process?

H.7 Do you have any knowledge of the effectiveness of the various types of training? Do households seem to understand the process?

H.8 Would you make any changes to the participant support process for the next summer?

I. training and support for retailers

[NOTE TO VISITOR: ASK FOLLOWING QUESTIONS IF RETAILER TRAINING OR OTHER SUPPORT PROVIDED]

Retailer training.

I.1 Since the last interview, has your organization provided information/training about [SEBTC PROGRAM NAME] to retailers?

I.2 Have all [SNAP/WIC] retailers in the demonstration areas received information/training?

Type of material or training.

I.3 What was the focus of the materials or training (e.g., information about the [SEBTC PROGRAM NAME], the look of the EBT card, use of the EBT card, the new WIC package)? What type of materials or training will be provided (e.g., information packets, one-on-one, group, computer, web-based)?

I.4 Who prepared and distributed materials?

I.5 Are materials or training offered in multiple languages? Which languages are included?

I.6 Were face-to-face trainings conducted at the retailer’s location or with multiple retailers?

I.7 Who conducted the training?

I.8 How long did training classes last?

I.9 How many trainings were there?

I.10 What retailer personnel were invited to the training?

I.11 What type of information was discussed?

I.12 How much training has been conducted so far? Please describe what has been done so far and what is planned for the future.

I.13 Is follow-up training planned after retailers receive the initial training or as new staff members join the retailers?

Assessment of retailer training.

I.14 What were the strengths and weaknesses of the retailer training?

I.15 How much follow-up training/support was needed?

I.16 How did the retailers respond to the training?

I.17 Did any issues arise that were not anticipated or addressed through training?

[Probe:]

- Stocking issues due to high demand

- Concerns/reactions from retailers about the new program

- Problems with cashiers recognizing and accepting cards

- Other issues?

I.18 Would you suggest making changes to the training (i.e., content, timing, length, location) for the next summer?

Additional retailer support.

I.19 Does your firm provide support to retailers through the existing EBT retailer service number?

[Probe:]

- Do retailers use this number when they need support for SEBTC?

- What are the hours of operation?

- What are the qualifications of staff who answer this number?

- What training did staff receive about SEBTC?

I.20 Does your firm operate a special customer service/contact number for retailer questions about SEBTC? [IF VENDOR DOES NOT OPERATE RETAILER SUPPORT NUMBER, SKIP TO NEXT SECTION]

[Probe:]

- What are the hours of operation?

- What are the qualifications of staff who answer this number?

- What training did staff receive about SEBTC?

- Why is this number separate from regular EBT retailer support?

I.21 Are any other supports available to retailers? [e.g., website, IVR] If so, please describe them.

Extent of retailer support provided.

I.22 How often do retailers call the customer service number about SEBTC? Were you able to track this?

I.23 What are the common issues they raised?

I.24 Did questions arise from these calls that you would consider including in future retailer training?

I.25 Were there any issues with the customer service process?

I.26 Would you make any changes to the process for the next summer?

J. Successes, Challenges, and Solutions

Successes.

J.1 Thus far, what have been the greatest successes of the demonstration?

J.2 What factors contributed significantly to this success?

J.3 What, if anything, could have been done differently to make the demonstration work better?

Challenges.

J.4 Thus far, what have been the biggest challenges to meeting the goals of the demonstration?

J.5 Have these challenges been resolved?

- If so, how?

- If not, why?

J.6 What could your organization have done differently to meet demonstration goals more effectively?

J.7 How have these challenges affected the effectiveness of the demonstration?

K. Feasibility of Continuing and Replicating DemonstrationS

Conditions needed to continue the demonstration.

K.1 What conditions would need to be maintained for the demonstration to continue successfully next year?

K.2 What program components or implementation practices would need to be maintained?

K.3 What changes, if any, would need to be made in order to operate successfully next year?

Conditions needed for replication in other sites.

K.4 What state and local conditions are critical to replicate the demonstration in other areas of the country?

K.5 What program components or implementation practices are essential for replication?

K.6 Are there practices you would advise other states and local areas to avoid?

[If so, probe:]

- What are they?

- Why would you suggest avoiding them?

K.7 What advice would you give to other states and local areas interested in replication of the demonstration?

L. Implementation and Operational Costs

L.1 Now I would like to turn to some questions related to the cost of implementing the demonstration. I brought copies of the grant expenditure data that you provided before the site visit and would like to review them with you in detail.

Review of grant expenditures. [Review cost forms]

L.2 Can you please briefly describe the responsibilities of each staff member listed?

L.3 What staff title applies to each person (for example, senior executive, project manager, IT/MIS manager, IT/MIS developer, customer service, administrative support)?

L.4 Can you please describe what is included in the other direct costs by category?

Start-up versus on-going costs.

L.5 What types of activities are covered by your start-up expenditures?

[Probe:]

- Over what period do they pertain?

- How did you determine what activities would be considered start-up expenditures?

- If the demonstration were to continue for another year, would any of these costs be repeated next spring? If so, which ones?

L.6 What types of activities were covered by on-going expenditures?

[Probe:]

- Over what period do they pertain?

- If the demonstration were to continue for another year, would all of these costs be repeated next spring and summer? If not, which will not be needed?

Sufficiency of funds.

L.7 Do you believe that your organization had sufficient funds through your grant to implement the [SEBTC PROGRAM NAME] benefits successfully?

[If not, probe:]

- Why was there a gap between funds and needed resources?

- How large was this gap?

- How did you deal with this gap?

- What would you do differently if you didn’t have this gap?

Accounting systems.

L.8 Did you need to adapt your accounting systems to meet the grant requirements or evaluation cost data collection needs?

[If so, probe:]

- Please describe how.

- Did these changes influence other aspects of your agency’s work?

- Will these revised procedures be used within the organization beyond the grant?

Tracking expenditures.

L.9 Did you experience any challenges in tracking grant expenditures over the course of the start-up period? If so, please describe.

L.10 Over the course of the summer? If so, please describe.

CLOSING

Q.1. Is there anything you think is important for the FNS to know about the [STATE]’s [SEBTC PROGRAM NAME] demonstration that we did not ask about?

Q.2. Are your any questions you have for me before we finish?

Thank you for your time and helpful feedback. The information you have shared will be valuable to our team as we look across states and localities for themes and ideas that we can share with FNS.

END OF INTERVIEW

1 Note to visitor: if process for distributing EBT cards, replacing cards, or other EBT operations is specified in proposal, confirm and clarify as needed.

2 Note to visitor: review design to determine whether activation and/or loading benefits requires a separate step.

13