Attachment A MATHEMATICA
Note: if we have already conducted information gathering interviews with this respondent, we will skip introduction script.
My name is [NAME] and I am from Mathematica Policy Research. On behalf of the Office of Planning, Research, and Evaluation in the federal Administration for Children and Families, we are conducting a study of innovative, promising employment interventions for low-income individuals facing complex challenges to employment. “Complex challenges to employment” is a broad concept that could encompass, for example, physical and mental health conditions, substance misuse, a criminal history, or limited work skills and experience. The study is called the Next Generation of Enhanced Employment Strategies Project (NextGen Project). It will help us learn more about how employment programs can help low-income people move toward economic independence.
If asked: The Office of Planning, Research, and Evaluation is also conducting a companion study, Building Evidence on Employment Strategies for Low-Income Families Project (BEES). The projects are both designed to test innovative employment interventions that are rooted in the best available research and aim to help low-income people move toward economic independence. The projects are coordinating closely. Depending on the types of interventions we find that warrant evaluation, the two projects may focus on different interventions or different populations.
If a program that might qualify for Section 1110 funds: Additionally, our project [and the BEES project if above noted] is working closely with the Social Security Administration (SSA) to incorporate a focus on employment-related early interventions for individuals with current or foreseeable disabilities who have limited work history and are potential applicants for Supplemental Security Income (SSI).
We plan to conduct a rigorous, random assignment study of about nine interventions. The study will examine the implementation of the intervention and estimate its impacts on employment and other outcomes. We will also assess the intervention’s costs. We have identified [PROGRAM NAME] as operating an intervention to consider for inclusion in the evaluation. Right now, we are collecting information on interventions that might be a good fit for the study; we will select programs to invite to participate in the study later. We would like to ask you some questions about your program and intervention to determine whether they are well suited to be included in the evaluation. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. This information collection has been approved under OMB information collection request 0970-0356, which expires on June 30, 2021.
Participating in an evaluation has benefits to you, other programs like yours, and the clients you all serve. Funds are available for us to work with programs to refine their operations to get the interventions working as well as possible before evaluation. All programs participating in the evaluation will receive funding to support their participation in a rigorous evaluation.
If a program might qualify for Section 1110 funds: For interventions focused on people with current or foreseeable disabilities who are potential applicants for SSI, there may be additional funds that can be used to bolster the intervention—for example, by adding an employment service component to an existing intervention, by increasing the services already being provided, or by increasing the number of people who are recruited for or served by the program.
Programs that participate in random assignment studies can benefit by learning about the effectiveness of their interventions and how to improve them. They will build their capacity to collect and use data to inform their operations. They also gain an objective assessment of their intervention that policymakers and funders will trust. If the study shows the intervention is effective, it might be easier to pursue additional funding.
Your participation in this information-gathering interview is voluntary and, should you wish to participate, the information you provide will be kept private within the project team and the Office of Planning, Research, and Evaluation. This interview should take about an hour [for staff and supervisors] OR an hour and a half [for administrators/managers].
Before we begin, do you have any questions about the project?
Note: Probe for any changes as a result of COVID-19.
What is your job title?
How long have you been working at [organization name]?
What is your main role at [organization name]?
What is your role in implementing [intervention name]? What are your main responsibilities?
Note: in some cases we will have already collected this information through stage 2 information-gathering phone calls or through direct outreach to program administrators. If that is the case, we will use the information already collected under that effort, but not ask respondents to repeat information. As applicable, probe for whether the intervention changed as a result of COVID-19.
This section is for the program administrator/manager.
Tell me about [intervention name].
What are the objectives of the intervention?
What are the main components of the intervention?
How is the intervention situated within the larger organization? That is, is it the main service provided, or a small part of the overall organization’s services?
For programs that operate in multiple locations: Where is this intervention available? How similar is the intervention and target population as implemented in these multiple locations? Who makes most decisions about how the program is implemented in these locations (local staff or centralized staff)?
How long has the organization been implementing the intervention?
What are the characteristics of participants? Do you have any data or reports we could see? Probe for age, education level, income, disability, health or substance use challenges, work history, and other employment challenges. Do you know about what proportion might be receiving Supplemental Security Income?
How is the intervention funded?
What written materials about the intervention or program could you share with me?
Note: in some cases we will have already collected this information through stage 2 information-gathering phone calls or through direct outreach to program administrators. If that is the case, we will use the information already collected under that effort, but not ask respondents to repeat information. As applicable, probe for whether the intervention made changes as a result of COVID-19. Changes may include offering virtual services, implementing social distancing or other safety measures, adding or discontinuing services, and furloughing staff or changing roles.
How do participants usually find out about the intervention? Probe for referral from human service agency, court, health treatment center, etc.
Are any referrals for mandatory participation (i.e., courts)?
Walk me through what happens when someone first expresses interest in participating.
Where are they determined eligible, and by whom? Are partners involved in intake?
What are the eligibility criteria? To what extent do intake staff have discretion over the criteria?
What do prospective participants need to do to be enrolled? For example, what documentation do they need to provide?
How long does it take from first contact to enrollment? About what fraction of potential participants who begin enrollment end up completing enrollment?
If an individual does not seem right for the intervention, what happens? Do staff refer them to [organization] or other providers for services?
If referred to [organization], would they receive any services received by [intervention] participants?
What are the main services the intervention offers to participants? Probe for:
Assessments
Education
Occupational skills training
Life or soft skills training
Job search assistance
Paid or unpaid work experiences other than employment
Subsidized or competitive employment
Case management
Coaching
Benefits counseling
Service coordination
Support services
Referrals
Financial support or incentives (probe for how much)
Substance abuse treatment
Mental, emotional, or physical health treatment
Expungement or legal assistance
Do all participants receive the same suite of services? If not, how do you decide which services are provided to each participant, and in what order?
If all do not receive the same services, what services do most participants receive?
What are some of the main drop-off points—that is, points at which participants tend to disengage? About what fraction drops off at these points?
Of the services you just described, are these only for [intervention] participants, or can people enrolled in other programs access them? Please explain.
Are any of the services provided by employers?
How many employers are involved? What are their industry, size, etc.?
What do they offer to participants?
Are participants paid? Who pays them (that is, the program, employers, some combination)?
Are any of the services provided by partners other than employers?
How many partners are involved? What types of organizations are they (e.g., health treatment providers, educational institutions, other non-profits)?
What services do they offer to participants?
Do program staff encourage participants to apply for public assistance benefits, such as TANF, SNAP, or SSI? What kind of supports do they provide to help them?
[For administrators/managers] How many staff work directly with participants?
What are their backgrounds and qualifications?
How many participants does each staff member work with? Do staff carry a caseload, and if so what is the average?
How many staff would the typical participant interact with?
How many supervisors are there? How many staff do they supervise?
What is staff turnover like? About what fraction leave each year?
[For administrators/managers] Do you track intervention costs? If so, how much does your program cost per participant? Do participants pay any out-of-pocket costs?
Note: this section only asked of administrators/program developers.
How was the intervention developed? Who developed it? Were employers involved in its development?
Has [intervention] ever been evaluated?
If so, who conducted the evaluation? Is there a report available that describes the evaluation’s methods and findings?
If not, did it involve an evaluation of its effectiveness? What outcomes were measured and when? What were the findings?
Note: As applicable, probe for whether changes happened as a result of COVID-19, for example, changes to participant engagement, program completion expectations, and staff training.
[For administrators/managers] Does the intervention have a well-defined, standardized model? If so: Has fidelity to this program’s implementation of the intervention been assessed by a third party? What was the result of that assessment?
[For administrators/managers] About what percentage of the staff would you estimate is implementing the program as designed? How do you know? What strategies do you use to address any issues?
[For administrators/managers] Are specific program components less likely than others to be implemented as designed? Which ones and why?
On average, how frequently do participants receive intervention services? How does that compare to your expectations for how often they should be receiving services?
How do you define program completion? About what fraction of participants complete the program? On average, how long (in calendar time) do participants participate in the program? What follow-up services are provided?
For those who do not complete the program, about how often do they return in the same year (or other program-relevant period)? How does the program handle this?
How engaged do participants seem to be? How well do they think it is working?
How much training on the intervention did you/your staff receive, both upfront and ongoing?
How is it delivered (for example, in person, webinar, on-the-job)? By whom?
Is there a training manual? If so, could we have a copy?
Is any follow up or ongoing training provided?
Was the training you/your staff received sufficient to implement the program as designed? If not, what additional training would be helpful?
How comfortable are you/your staff in implementing the intervention?
Does the intervention have standardized tools and resources that you/your staff are supposed to use with participants?
[If so]: Describe the tools and what each is used for.
Do the tools have a cost? If so, what is it?
Do you/your staff regularly use these tools with participants? If not, why not?
Do you/your staff use these tools as intended? If not, how do you/they use them?
Could you share copies of the tools with us?
[If not]: Are staff using other tools and resources that are not standard to the program? If so, please describe.
How often do supervisors meet with staff to discuss program implementation? Probes: this could include case consultations, observations of staff-client interactions, staff meetings, etc.
What happens if a supervisor determines a staff person is not implementing the intervention as intended? Probe for whether additional training is provided, staff person is given more supervision, and so on.
[For supervisors:] About what percentage of staff need ongoing support to implement the intervention?
[For managers/supervisors:] Do you have processes in place to monitor whether the program is being implemented as intended? If so, describe.
How enthusiastic are you about the intervention? How well do you think it is working?
How enthusiastic are your staff/your leadership about the intervention? How well do they think it is working?
Note: this section only to be asked of managers/administrators
About how many new participants does your program serve in an average month? How many total participants does your program serve each month? How many participants did you serve in the last year? Probe for whether this changed during COVID-19.
Do you maintain a waitlist? If you had the funds to serve more people, do you think you could find people eligible for the intervention who would want to participate? How would you find these people?
What changes to staffing or other resource allocation would be needed to serve additional participants? What factors might limit growth of the program? Probe for: facilities, costs, staff, eligible population, competing programs.
Are there other places in your community where participants could receive similar services as those offered by the intervention?
If so, what are some of those places? How are they similar to or different from your program?
Would participants need to pay for those services out-of-pocket?
If we were to conduct an evaluation, we would need to randomly assign clients to either receive the intervention services or not. [Suggest to the respondent how we would see this working. Note that if they recruit more eligible participants, then they would serve the same number of people.]
Do you think the proposed process would work?
Can I address any questions you have about that?
Can staff commit to abiding by group assignment (i.e., not offering the intervention to someone who was randomly assigned to the control group)?
Are leadership and staff interested in participating in research about the intervention? What would be the next steps for getting approval to participate in research about the intervention?
How likely is it that the intervention will continue to be implemented without notable changes (e.g., staffing, budget, reorganization) in the near future?
4. Partnerships
Note: As applicable, probe for whether changes happened as a result of COVID-19.
How do you coordinate with partner service providers or employers? Do you share a data system?
For those partners that refer participants to you, what information about participants do they share?
Do you think partners would agree to participate in a random assignment study? Would they have any particular concerns? Please explain.
For which partners would you/we need buy-in to implement an evaluation?
Note: As applicable, probe for any changes made as a result of COVID-19 and how the program operated during the pandemic.
Do you use data to inform program services or improve program operations? Explain.
What is working well in how [intervention] is implemented? What are some examples of successes?
What have been the main challenges? How were those challenges addressed?
Is there anything you’re still working on addressing? What supports might you need?
How has your work been affected by COVID-19? Probes: working from home, changes to performance expectations, personal challenges faced, support received and still needed
[For leadership only:] Describe how leadership responded to the pandemic. Probes: the planning process, implementation, communication, staff and participant considerations
[For leadership only:] What, if any, changes made during the pandemic do you anticipate continuing after the pandemic?
What lessons have you learned from this experience that can help inform future crisis response?
Thank you very much for your time today. I know I asked a lot of questions and probably raised some questions for you. What questions do you have? What concerns?
We are currently conducting similar visits to other programs to assess their suitability for the evaluation.
We may have some further questions for you before deciding on the set of interventions for the evaluation. Would that be ok?
If we reached mutual agreement to partner with you for this study, then it sounds like the next steps are [fill in based on responses to questions 44 and 48].
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Kristen Joyce |
| File Modified | 0000-00-00 |
| File Created | 2021-01-12 |