Business

Appendix H1. Staff Semi-structured Interview Guide

This page has been left blank for double-sided copying

Introduction

Good [afternoon/morning]. My name is __________________, and I am from Mathematica. [Introduce others from the team.] I want to start by thanking you for taking time to speak with us today.

As you might know, we are evaluating the Supplemental Nutrition Assistance Program (SNAP) Employment and Training (E&T) Rapid Cycle Evaluation (RCE) intervention for the Food and Nutrition Service (FNS) of the U.S. Department of Agriculture. The intervention in [site] included [describe change] that you implemented over the last few months. For the purposes of this study, we are interested in understanding how the intervention was implemented and the implementation lessons learned from this experience. Your perspective and insights on these issues will be very helpful to the study.

Your participation in this interview is voluntary, and your responses will be kept private to the extent provided by law. We will not share the information you provide with any other people, including your supervisor and State-level staff. All confidential information will be stored safely and destroyed at the end of the study. If you do not feel comfortable answering a question, please let us know, and you may stop the interview at any time.

We will take notes over the course of the interview so that we can remember the information we collect. We will use this information in our reports to FNS, describing the range of responses expressed by staff. The reports might list the names of organizations that contributed information, but we will not quote you or anyone by name or title. However, because of the relatively small number of organizations participating in the study, there is a possibility that a response could be correctly attributed to you.

I expect our conversation will take about [60-90 minutes]. First, do you have any questions for me about the project in general or what we will be discussing today? If you have any questions in the future, you can contact Mathematica at [NUMBER] or by e-mail at [EMAIL].

Before we begin, I’d like to have your consent documented for our records. Do you agree to participate in this interview?

Public Burden Statement

This information is being collected to assist the Food and Nutrition Service in evaluating operational improvements in Supplemental Nutrition Assistance Program (SNAP) Employment and Training (E&T) programs that aim to improve delivery of services and program outcomes. This is a voluntary collection and FNS will use the information to assess the effectiveness of changes made to the SNAP E&T program. This collection does request any personally identifiable information under the Privacy Act of 1974. According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-[xxxx]. The time required to complete this information collection is estimated to average 90 minutes (1.5 hours) per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, 1320 Braddock Place, 5th Floor, Alexandria, VA 22306 ATTN: PRA (0584-xxxx). Do not return the completed form to this address.

Privacy Act Statement

Authority: This information is being collected under the authority of Section 9 of the Food and Nutrition Act of 2008, as amended, (7 U.S.C. 2018). Disclosure of the information is voluntary.

Purpose: The information is being collected to evaluate operational improvements in Supplemental Nutrition Assistance Program (SNAP) Employment and Training (E&T) programs using rapid cycle evaluation.

Routine Use: The information may be shared with SNAP contract researchers and United States Department of Agriculture (USDA) SNAP research and administrative staff.

Disclosure: If all or any part of the information is not provided, interviews may not be admissible in data sets.

[Guidance to interviewers: Site visit teams will tailor the interview guide for the site given the staff who are involved in the intervention and information about the site and the intervention known from technical assistance discussions.

Respondents to be interviewed will include: Leadership (SNAP administrators (State or local office) or provider staff leading the intervention), supervisors (service provision supervisors at the State, local, or provider-level), and frontline staff (case managers or other staff working directly with participants). Each site visit team will identify the specific people to be interviewed based on key staff who were involved in the intervention.]

1. Background

Respondents: All staff (unless information is known)

1. What is your official job title or position?

2. What are your responsibilities related to the intervention?

2. Intervention environment and context

Respondents: Leadership and supervisors

1. Where was the intervention operated? Are the locations urban/rural?

2. What is the availability and accessibility of providers and resources?

Probes:

• Number and types of E&T providers

• Types of services offered

• Availability of funding for SNAP E&T

• Types of resources available in the community and lack of key resources (for example, transportation, food banks, child care, clothing closets)

3. What were the labor market conditions at the time of the intervention in the area where it operated?

Probes:

• Key industries hiring E&T participants

• Types of available jobs for E&T participants

• Unemployment rates

• Unique employment or economic factors that affect E&T participants

4. What are the common barriers to employment among E&T participants?

5. Are there any unique State or local factors that might affect the intervention?

Probes:

• How might those factors lead to success?

• How could those factors create challenges?

3. Intervention implementation

Respondents: All staff

6. What challenge was the intervention designed to address?

7. Describe the key features of the intervention.

8. Were any changes made to the design after piloting (road testing) the intervention? If so, describe.

9. What are the demographics of the population targeted by the intervention?

Probe:

• How does the intervention’s target population differ from all E&T-eligible participants or typical E&T participants served?

10. What was the process for implementing the intervention?

[Note to interviewer: much of this information is discussed in the process mapping exercise, so only ask questions needed to fill in the gaps in our understanding. Depending on the number of questions you need to ask, you may want to let the respondent know these questions are related to the process mapping exercise they completed earlier, and you have a few more questions to better understand the process.]

Probes:

• Describe the steps in the process

• Staff involved at each step

• Timeline for aspects of implementation

• Point(s) of random assignment

11. Did the actual implementation deviate from the planned implementation? If so, describe.

Probes:

• Describe specific aspects that deviated

• Changes in timelines

• Reason for the deviation

• Challenges that emerged because of the deviation

• Effect on fidelity of the overall model

12. Were there any challenges in initially implementing the intervention? If so, describe.

Probe:

• How were they resolved

4. Staffing

Respondents: Supervisors and frontline staff

13. What additional staff and resources were needed to design and implement the intervention?

14. What staff training was needed to implement the intervention?

Probes:

• Type and method of training

• Was training sufficient

• Any follow-up training provided

15. Did staff understand how clients were intended to flow through the intervention?

Probe:

• What kind of challenges did staff face?

16. Did staff understand their role in the intervention?

Probe:

• What kind of challenges did they face?

17. How did the intervention affect staff workloads/caseloads compared with the pre-intervention period?

18. Overall, how would you rate staff satisfaction with the intervention?

Probe:

• What aspects had a positive or negative impact on job satisfaction?

5. Outcomes

Respondents: Leadership and supervisors

19. What were the key outcomes that each part of the intervention was targeting?

20. What processes and systems were in place to measure data and outcomes for the intervention?

Probe:

• Who tracked, collected, reported, and monitored data

21. To what extent do you think the intervention met the goals established during the design phase?

22. What aspects of the intervention do you believe are most responsible for any observed improvement in outcomes?

Probes:

• Based on what evidence

• Any difference based on subgroups (for example, women vs men, those in rural vs urban areas)

6. Costs

Respondents: Leadership and supervisors

23. What were the total costs of the changes for all entities involved in the implementation?

Probes:

• Staff time

• Training time and materials

• Technology upgrades or expansion

• Fees for service (for example for texting)

24. Were there any cost savings from the changes once implemented? If so, describe.

Probes:

• To Staff

• To E&T participants

• To organization

25. Could these changes continue with the current level of SNAP E&T funding that is allocated to your organization?

Probe:

• If no, could you leverage other funds to continue?

7. Lessons learned

Respondents: All staff

Challenges

26. What aspect of the intervention posed the biggest challenge for you in your role?

Probes:

• How were challenges resolved

• How could challenges have been avoided

27. What were the biggest challenges for implementing the intervention overall?

Probes:

• How were challenges resolved

• How could challenges have been avoided

Successes

28. What aspects of the intervention worked best and were most successful?

Assessments

29. What is your assessment of the intervention?

Probes:

• Did staff have the training and resources needed?

• Aspects that had a positive effect on service delivery

• Aspects that had a negative effect on service delivery

• How did participants respond?

30. What modifications (e.g. add, remove, change) could help ensure better [recruitment/engagement/service delivery]?

31. [For staff who participated in the design of the intervention.] Do you think the RCE process was well suited for the intervention? Describe.

Probes:

• Suggested changes to the design process or technical assistance

• Aspects that should not have been tested

8. Scalability and replicability

Respondents: All staff

32. What, if anything, would you do differently if you could implement the intervention again?

33. If applicable, how easy or difficult would it be to implement these changes in the rest of the State/with all participants?

Probes:

• Conditions needed to be maintained

• Changes required

• Funding to offset costs

• Additional technical assistance needed

34. What advice would you give other States/providers that want to implement these changes?

I. Wrap up and Closing

Respondents: All staff

1. Is there anything else that you would like to add about your program experiences, or are there other thoughts you had during our discussion that you would like to mention before we finish?

Thank you for taking the time to share your thoughts. This discussion has been very useful in helping us learn more about [SNAP E&T Program Name] in [LOCATION].

Process mapping discussion guide

INTERVIEWER: READ IF NEW RESPONDENT

Introduction

Good [afternoon/morning]. My name is __________________, and I am from Mathematica. [Introduce others from the team.]

FNS has contracted with Mathematica to help States develop and implement small scale interventions that can improve the operation of or participation in the SNAP E&T program. We also are evaluating if these changes have produced the expected outcomes.

As part of the evaluation, we will be learning more about the process for implementing the intervention and how services were provided. This activity should take about 90 minutes and will visually chart out the process flow of the intervention. The goal is to make sure we understand each step in the process.

Your participation in this discussion is voluntary, and your responses will be kept private to the extent provided by law. We will not share the information you provide with anyone outside of this room and the research team, [including your supervisor and State or Federal-level staff]. You may refuse to answer any question or may stop the interview at any time. There will not be any penalties if you refuse to participate in part or full.

We will take notes during the discussion so that we can remember the information we collect. We will use this information in our reports, describing the range of responses expressed by staff. The reports might list the names of organizations that contributed information, but we will not quote you or anyone by name or title. However, because of the relatively small number of organizations participating in the study, there is a possibility that a response could be correctly attributed to you.

I would like to record our discussion so I can listen to it later when I write up my notes. No one outside the immediate research team at Mathematica will listen to the recording. We will destroy the recording after the study is complete. If you want to say something that you do not want recorded, please let me know and I will be glad to pause the recorder. Do you have any objections to being part of this interview or to my recording our discussion?

Do you have any questions before we begin?

Public Burden Statement

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-xxxx. The time required to complete this information collection is estimated to average 90 minutes (1.5 hours) including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, 1320 Braddock Road, 5^th Floor, Alexandria, VA 22314, ATTN: PRA (0584-XXXX). Do not return the completed form to this address.

Privacy Act Statement

Authority: This information is being collected under the authority of Section 9 of the Food and Nutrition Act of 2008, as amended, (7 U.S.C. 2018). Disclosure of the information is voluntary.

Purpose: The information is being collected to evaluate operational improvements in Supplemental Nutrition Assistance Program (SNAP) Employment and Training (E&T) programs using rapid cycle evaluation.

Routine Use: The information may be shared with SNAP contract researchers and United States Department of Agriculture (USDA) SNAP research and administrative staff.

Disclosure: If all or any part of the information is not provided, interviews may not be admissible in data sets.

The process mapping activity will be used to chart the flow of the [RCE intervention name] and its services, as well as the timeline. We will use a virtual white board (called Mural) to record information during the conversation and as a visual aid to identify missing information.

We’ll start with the first action that staff took to begin the intervention and then ask a series of questions about that step. We’ll repeat a similar process for each of the subsequent steps of the intervention. We’ll focus first on steps taken to implement the intervention for the treatment group(s), then ask similar questions for the control group(s), if there was one.

1. Walk us through what happened when the intervention started. What was the first activity that staff did?

Probes:

• What staff were involved?

• How time consuming was this step for staff?

• What data did you collect?

• What went well, and what challenges did you face?

• Would any technical assistance have been helpful?

• How sustainable or unsustainable was the level of staff effort, in terms of time or cost, for this step if the intervention were to continue long-term?

• (For subsequent steps) How long after the previous step did this occur?

2. Repeat the questions above for as many steps as necessary until you reach the end of the intervention.

3. Now I will ask you a few questions as a reflection on the whole process.

1. How and when did the intervention deviate from the planned implementation (if at all)?

1. What steps did you take to remedy the situation?

2. What additional resources (if any) would have been helpful to prevent this from happening?

2. Describe any events, policy changes, or local economic shifts outside of your control that happened during implementation and could have affected the intervention.

Now we will move onto the actions that staff took during the intervention period for the control group.

4. At which step in the process map described previously did staff activities start to differ for the control group? (If not at step one, or if not already clear from the discussion above)

5. Starting at this step, walk us through what happened during the intervention period for control group members.

3. Was this process similar or different to what you would have done for participants without the intervention taking place?

Probe:

• Besides added data collection for evaluation purposes for the control group, was there any difference in the amount or type of outreach or contact that the control group received during the intervention, compared to what they would have received without the intervention?

1. What staff were involved?

2. How time consuming was this step for staff (relative to the treatment group)?

3. What data did you collect?

4. What went well, and what challenges did you face (relative to challenges faced during implementation of treatment group processes)?

5. (For subsequent steps) How long after the previous step did this occur?

6. Repeat the questions above for as many steps as necessary until you reach the end of the intervention.

Thank you very much for participating in this activity and sharing your insight on the [RCE intervention name] processes. This discussion has been very helpful in learning more about making improvements to the [SNAP E&T Program Name].