Program Managers, Staff, and Partners Interview Guide (Attachment L)

Attachment L – Program Managers, Staff, and Partners Interview Guide

OMB Control No: ____-____

Expiration Date: __/__/____

Program Managers, Staff and Partners Interview Guide

This document covers the key topics to be covered in individual interviews with program managers, staff, and partners. Each topic indicates the appropriate staff (i.e., program managers, program staff, partners) that will respond. The questions will be tailored to the specific intervention and program being evaluated. The interviews will be conducted at two points in time: 1-2 months after random assignment starts and toward the end of the random assignment period to document changes over time, overall implementation challenges and successes and program sustainability (if applicable). A table at the end of the document summarizes which staff (e.g. managers, staff, and partners) will be included in each topic of the interviews. Items marked with an asterisk (*) indicate that the topic will be covered in both the first and second round of interviews. As shown, some topics will only need to be collected in the first round of site visits, and other topics will focus on changes or new activities between the first and second round.

Introductory statement for staff: The Building Evidence on Employment Strategies for Low-Income Families (BEES) study is being conducted by MDRC/Abt/MEF Associates, under contract to the U.S. Department of Health and Human Services. The study aims to build evidence on the effectiveness of innovative programs designed to boost employment and earnings among low-income Americans. As part of this evaluation, evaluation staff members will visit each of the study sites. A major aim of the evaluation is to identify lessons learned from your experiences in implementing the projects evaluated under this initiative. In conducting site visits to each of the programs, we are talking to project directors and staff, as well as partner organizations. We are here to learn about the service delivery of your program model and understand how it operates. Our aim is to learn from your experiences, not audit or judge your programs. This interview will take between 60 and 90 minutes.

Privacy Statement: Before beginning the interview, I (we) want to thank you for agreeing to participate in the BEES study. I (we) know that you are busy and we will try to be as focused as possible. We have many questions and are going to talk to many different people, so please do not feel as though we expect you to be able to answer every question. And, we understand that your participation in this discussion is voluntary and you may choose to not answer some questions.

Information collected will be kept private to the extent permitted by law. The views you express will be kept private, and nothing we publish in this evaluation will identify you by name. Though we take notes during these interviews, information is never repeated with the name of the respondent. When we write our reports and discuss our findings, information from all interviews is compiled and presented so that no one person can be identified. We also ask that you refrain from sharing anything we discuss today with others to help us ensure your privacy and the privacy of others we are interviewing. Do you have any questions before we begin?

The Paperwork Reduction Act Statement: This collection of information is voluntary and will be used to understand programs that aim to improve employment outcomes for low-income adults. Public reporting burden for this collection of information is estimated to average 90 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: XXXX-XXXX, Exp: XX/XX/XXXX. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to Dan Bloom (MDRC); 200 Vesey Street, 23^rd Floor, New York, NY 10281-2103.

1. Respondent Information*

• Name. title, organization/affiliation, length of involvement with the organization and with the program

• Educational background and prior work experience

• Overall role/job responsibilities

2. Program Context and Environment—Program Managers and Partners

1. Lead organization background—Program Managers

• History of organization/agency

• Other programs and services offered by organization/agency

• Organization’ budget for most recently completed program year

• Organization’s major sources of funding (e.g., funding from federal/state/city agencies, foundations, private contributions, fee for service, etc.)

• Organization’s total # of paid staff: _______ # of people, _______ # of FTEs

• Types of individuals served or targeted

• Annual number of participants served overall

2. Geographic area served^*—Program Managers and Partners

• Program’s service area, possibilities include: portion of a city/county, single county, multiple counties, other (regional effort)

• Size of service area (in terms of geography and population)?

• Characteristics of this area (e.g., is the service area geographically large; urban, rural, mixed)

3. Area demographics—Program Managers and Partners

• Demographics of the service area, including education levels, race and ethnicity, income, poverty, immigrants)

• Changes in demographics over time

• Opioid influence

4. Local economic conditions*—Program Managers and Partners

• Major industries and employers

• Recent economic trends

• Unemployment levels, job availability, wage levels

• Other factors that affect economy (e.g., natural disaster, companies coming or going, policing practices/criminal justice involvement of community members)

5. Community and government supports—Program Managers and Partners

• Community and government supports commonly used by program participants (e.g., TANF, SNAP, housing assistance)

  • Services provided

  • Formal and informal coordination with community and government support program

• Other organizations that provide services to program’s target population

3. Program Goals and Structure—Program Managers and Partners

1. Program goals*—Program Managers

• Goals of the program, including changes over time

• Reasons for implementing/developing the program

• When developed and changes since initial implementation

• Hypothesized impacts of the program on target population (immediate, short-term, long-term)

• Source of the program model (i.e., is it based on existing program that has been operated elsewhere)

2. Resources and capacity*—Program Managers

• Resources required to operate the program

  • Funding sources and amount

  • Sources of curriculum/materials

  • Facilities (classrooms, labs, program offices)

  • Technology (laptops)

  • Other

• Annual budget and funding sources

• Future funding, stability of funding

3. Management and staffing*—Program Managers

• Number of staff and positions

  • Titles

  • Required experience and qualifications

  • Primary responsibilities

  • Location of staff members

  • Full-time employees, part-time employees

• Staffing/organizational structure (e.g., program departments, how the program is situated/managed within the larger organization or institution, who has authority over the program).

• Staff development and trainings (e.g. required trainings, how often, and provided by whom)

• Staffing shortages and staff turnover

• Performance measures for staff

4. Partnerships and organizational linkages* -- Program Managers and Partners

• Type of organizations

• Length of the partnership

• Historical relationship

• Nature of partnership

• Type of arrangement: Contracts, MOUs, type of contracts (performance-based, fixed price, etc.)

• Services provided by the organization to participants of BEES program

• Coordination mechanisms: staff responsible, regular meetings, data exchange

• Strengths and challenges with partnership

• Changes in partnership over time

5. Target population and program eligibility *—Program Managers and Staff

• Target population (socio-economic and demographic characteristics, including age, educational attainment, work history, income, marital status, children, US citizen or immigrant)*

• Program eligibility requirements (specific requirements to be met prior to enrollment)*

4. Program Service Components—Program Managers and Staff

1. Description of services provided*—Program Managers and Staff

• Occupational training (industry, credentials)

• Basic skills instruction

• Treatment services (treatment for substance use or opioid use disorder, mental health)

  • Medication-assisted treatment (medication provided; by whom)

  • Monitoring and support for adherence to treatment

  • Therapy

  • Recovery coaching or other recovery supports

• Employer-based training

  • Subsidized employment

  • Unpaid work experience

  • Paid internships

• Support services

  • Case management and counseling

  • Academic advising

  • Career advising

  • Child care

  • Work supports (tools, uniforms)

• Job readiness skill development (i.e. “soft” skills)

• Job search and placement assistance

• Referrals to other organizations and services provided through referral

2. Service delivery structure*—Program Managers and Staff

For each service listed above:

• Service sequencing

• Service provider and location

• Length and schedule of service component (e.g., total hours; length in weeks/months)

• Content and services provided

• Organizational partners in delivering service

3. Steps in Service Delivery—Program Managers and Staff

• Outreach/Recruitment*

  • Outreach and recruitment strategies (e.g., website, social media, word-of-mouth, fliers, referrals, community events, other agencies/programs)

  • Past strategies used

  • Effectiveness of strategies

• Application and Enrollment Process*

• How individuals apply to the program (i.e., online, on-site)

• Application process steps and information collected (in what order and by whom)

• Length of application process

• Use of waitlist

• Obtain copy of application

• Assessment and Participant Employment/Treatment Plans (including referrals)*

• How assessment is conducted

• Information collected from the applicant

• Academic and non-academic screening tools used

• Mental health diagnoses

• Substance abuse screening

• Development and content of employment/treatment plan

• Program Orientation*

• Orientation or information sessions

• Information provided to participants after enrollment

• Participant Monitoring *

• Use of employment/treatment plan to guide services

• Frequency and mode of monitoring

• Length of interactions and topics covered

• Actions taken if problems/issues

• Attendance or participation requirements

• Consequences for non-compliance

• Program Completion*

  • How completion defined; point at which a person completes the program

  • Completion rate: reasons for high or low completion

• Follow-up Services*

  • Services provided to participants once they’ve left the program and for how long

  • Job retention/advancement services

  • Follow-up data collected on participants

4. Data collection, performance management, and program outcomes*—Program Managers

• Data collected on participation in services and outcomes

• Data collected from program partners

• Performance measures in place to track the usage and quality of the services delivered

• Measures for tracking program performance

• Information collected from participants about their experiences in the program

• Recent trends in the performance of the program

5. Counterfactual Condition*—Program Managers, Staff and Partners

• Overview of services (if any) available to the control group from:

  • Organization

  • Partners

  • Other organizations in the community

• Other services you will refer control group members to

  • Provider, application process/waiting list, where provided,

  • Contacts at these organizations

  • Feasibility of /barriers to accessing these services

• Knowledge of control group members utilization on these services

6. Operational Challenges and Successes*—Program Managers, Staff and Partners

• Ways program has exceeded or fallen short of goals

• Program’s key successes

  • Factors that contributed to this success

• Challenges the program encountered and how they were overcome

• Areas for improvement

• Any plans for changes and modifications

• Any additional components program would like to consider

The following table provides a summary of interview topics and potential respondents for each.

Summary of Interview Topics and Respondent Type

Topic	Respondent Type
	Program Managers	Program Staff	Partners
Project Context and Environment Lead organization background Geographic area Served Area demographics Local economic conditions Community & Government Supports	X		X
Program Goals and Structure Program goals Resources and capacity Management and Staffing Target population and program eligibility	X		X
Program Service Components Description of services provided Service delivery structure Key steps in service delivery Data collection, performance management, and program outcomes	X	X
Counterfactual Condition	X	X	X
Operational Challenges and Successes	X	X

*^ Questions will be asked during first and second implementation visits to document change over time.

11