Attachment L – Program Managers, Staff, and Partners Interview Guide
OMB Control No: ____-____
Expiration Date: __/__/____
Program Managers, Staff and Partners Interview Guide
This document covers the key topics to be covered in individual interviews with program managers, staff, and partners. Each topic indicates the appropriate staff (i.e., program managers, program staff, partners) that will respond. The questions will be tailored to the specific intervention and program being evaluated. The interviews will be conducted at two points in time: approximately 6 months after random assignment starts and toward the end of the random assignment period to document changes over time, overall implementation challenges and successes and program sustainability (if applicable). A table at the end of the document summarizes which staff (e.g. managers, staff, and partners) will be included in each topic of the interviews. Items marked with an asterisk (*) indicate that the topic will be covered in both the first and second round of interviews. As shown, some topics will only need to be collected in the first round of site visits, and other topics will focus on changes or new activities between the first and second round.
Introductory statement for staff: The Building Evidence on Employment Strategies for Low-Income Families (BEES) study is being conducted by MDRC/Abt/MEF Associates, under contract to the U.S. Department of Health and Human Services. The study aims to build evidence on the effectiveness of innovative programs designed to boost employment and earnings among low-income Americans. As part of this evaluation, evaluation staff members will visit each of the study sites. A major aim of the evaluation is to identify lessons learned from your experiences in implementing the projects evaluated under this initiative. In conducting site visits to each of the programs, we are talking to project directors and staff, as well as partner organizations. We are here to learn about the service delivery of your program model and understand how it operates. Our aim is to learn from your experiences, not audit or judge your programs. This interview will take between 60 and 90 minutes.
Privacy Statement: Before beginning the interview, I (we) want to thank you for agreeing to participate in the BEES study. I (we) know that you are busy and we will try to be as focused as possible. We have many questions and are going to talk to many different people, so please do not feel as though we expect you to be able to answer every question. And, we understand that your participation in this discussion is voluntary and you may choose to not answer some questions.
Information collected will be kept private to the extent permitted by law. The views you express will be kept private, and nothing we publish in this evaluation will identify you by name. Though we take notes during these interviews, information is never repeated with the name of the respondent. When we write our reports and discuss our findings, information from all interviews is compiled and presented so that no one person can be identified. We also ask that you refrain from sharing anything we discuss today with others to help us ensure your privacy and the privacy of others we are interviewing. Do you have any questions before we begin?
The Paperwork Reduction Act Statement: This collection of information is voluntary and will be used to understand programs that aim to improve employment outcomes for low-income adults. Public reporting burden for this collection of information is estimated to average 90 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: XXXX-XXXX, Exp: XX/XX/XXXX. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to Dan Bloom (MDRC); 200 Vesey Street, 23rd Floor, New York, NY 10281-2103.
Respondent Information*
Name. title, organization/affiliation, length of involvement with the organization and with the program
Educational background and prior work experience
Overall role/job responsibilities
Program Context and Environment—Program Managers and Partners
Lead organization background—Program Managers
History of organization/agency
Other programs and services offered by organization/agency
Changes to organizational policies and approaches due to COVID-19
Organization’s budget for most recently completed program year
Organization’s major sources of funding (e.g., funding from federal/state/city agencies, foundations, private contributions, fee for service, etc.)
Organization’s total # of paid staff: _______ # of people, _______ # of FTEs
Types of individuals served or targeted
Annual number of participants served overall
Geographic area served*—Program Managers and Partners
Program’s service area, possibilities include: portion of a city/county, single county, multiple counties, other (regional effort)
Size of service area (in terms of geography and population)?
Characteristics of this area (e.g., is the service area geographically large; urban, rural, mixed)
Local response to the COVID-19 pandemic
Area demographics—Program Managers and Partners
Demographics of the service area, including education levels, race and ethnicity, income, poverty, immigrants)
Changes in demographics over time
Opioid influence
Changes in substance use trends and treatment needs during the COVID-19 pandemic
Local economic conditions*—Program Managers and Partners
Major industries and employers
Recent economic trends, including impacts of the COVID-19 pandemic
Unemployment levels, job availability, wage levels
Other factors that affect economy (e.g., natural disaster, companies coming or going, policing practices/criminal justice involvement of community members)
Community and government supports—Program Managers and Partners
Community and government supports commonly used by program participants (e.g., TANF, SNAP, SSI/SSDI, housing assistance)
Services provided
Formal and informal coordination with community and government support program
Other organizations that provide services to program’s target population
Program Goals and Structure—Program Managers and Partners
Program goals*—Program Managers
Goals of the program, including changes over time
Reasons for implementing/developing the program
When developed and changes since initial implementation
Changes to program operations in response to COVID-19
Hypothesized impacts of the program on target population (immediate, short-term, long-term)
Source of the program model (i.e., is it based on existing program that has been operated elsewhere)
Resources and capacity*—Program Managers
Resources required to operate the program
Funding sources and amount
Sources of curriculum/materials
Facilities (classrooms, labs, program offices)
Technology (laptops)
Other
Annual budget and funding sources
Future funding, stability of funding
Impact of COVID-19 on funding
Management and staffing*—Program Managers
Number of staff and positions
Titles
Required experience and qualifications
Primary responsibilities
Location of staff members
Full-time employees, part-time employees
Staffing/organizational structure (e.g., program departments, how the program is situated/managed within the larger organization or institution, who has authority over the program).
Staff development and trainings (e.g., required trainings, how often, and provided by whom)
Staffing shortages and staff turnover
Impact of COVID-19 on staffing (furloughs, layoffs, telecommuting)
Performance measures for staff
Partnerships and organizational linkages* — Program Managers and Partners
Type of organizations
Length of the partnership
Historical relationship
Nature of partnership
Type of arrangement: Contracts, MOUs, type of contracts (performance-based, fixed price, etc.)
Services provided by the organization to participants of BEES program
Coordination mechanisms: staff responsible, regular meetings, data exchange
Changes in partnerships due to COVID-19
Strengths and challenges with partnership
Changes in partnership over time
Target population and program eligibility *—Program Managers and Staff
Target population (socio-economic and demographic characteristics, including age, educational attainment, work history, income, marital status, children, US citizen or immigrant; other characteristics such as mental illness, justice involvement, housing instability, disabilities)*
Program eligibility requirements (specific requirements to be met prior to enrollment)*
COVID-19 screening requirements
Program Service Components—Program Managers and Staff
Description of services provided*—Program Managers and Staff
Occupational training (industry, credentials)
Basic skills instruction
Treatment services (treatment for substance use or opioid use disorder, mental health)
Medication-assisted treatment (medication provided; by whom)
Monitoring and support for adherence to treatment
Therapy
Recovery coaching or other recovery supports
Employer-based training
Subsidized employment
Unpaid work experience
Paid internships
Support services
Case management and counseling
Academic advising
Career advising
Child care
Work supports (tools, uniforms)
Peer support assistance
Housing assistance
Legal services
Job readiness skill development (i.e. “soft” skills)
Job search and placement assistance
Referrals to other organizations and services provided through referral
Service delivery structure*—Program Managers and Staff
For each service listed above:
Service sequencing
Service provider and location
Length and schedule of service component (e.g., total hours; length in weeks/months)
Content and services provided
Organizational partners in delivering service
Changes in service delivery due to COVID-19
Steps in Service Delivery—Program Managers and Staff
Outreach/Recruitment*
Outreach and recruitment strategies (e.g., website, social media, word-of-mouth, fliers, referrals, community events, other agencies/programs)
Past strategies used
Effectiveness of strategies
Changes in strategies due to COVID-19
Application and Enrollment Process*
How individuals apply to the program (i.e., online, on-site)
Application process steps and information collected (in what order and by whom)
Length of application process
Use of waitlist
Obtain copy of application
Assessment and Participant Employment/Treatment Plans (including referrals)*
How assessment is conducted
Information collected from the applicant
Academic and non-academic screening tools used
Mental health diagnoses
Substance use screening
Development and content of employment/treatment plan
Program Orientation*
Orientation or information sessions
Information provided to participants after enrollment
Participant Monitoring *
Use of employment/treatment plan to guide services
Frequency and mode of monitoring
Length of interactions and topics covered
Actions taken if problems/issues
Attendance or participation requirements
Consequences for non-compliance
Program Completion*
How completion defined; point at which a person completes the program
Completion rate: reasons for high or low completion
Follow-up Services*
Services provided to participants once they’ve left the program and for how long
Job retention/advancement services
Follow-up data collected on participants
Data collection, performance management, and program outcomes*—Program Managers
Data collected on participation in services and outcomes
Data collected from program partners
Performance measures in place to track the usage and quality of the services delivered
Measures for tracking program performance
Information collected from participants about their experiences in the program
Recent trends in the performance of the program
Impact of COVID-19 on performance of the program
Counterfactual Condition*—Program Managers, Staff and Partners
Overview of services (if any) available to the control group from:
Organization
Partners
Other organizations in the community
Effects of COVID-19 on the availability of these services
Other services you will refer control group members to
Provider, application process/waiting list, where provided,
Contacts at these organizations
Feasibility of /barriers to accessing these services
Knowledge of control group members utilization on these services
Operational Challenges and Successes*—Program Managers, Staff and Partners
Ways program has exceeded or fallen short of goals
Program’s key successes
Factors that contributed to this success
Challenges the program encountered and how they were overcome
Areas for improvement
Effect of COVID-19 changes on long-term operations
Any plans for changes and modifications
Any additional components program would like to consider
The
following table provides a summary of interview topics and potential
respondents for each.
Summary of Interview Topics and Respondent Type
Topic |
Respondent Type |
||
Program Managers |
Program Staff |
Partners |
|
Project Context and Environment
|
X |
|
X |
Program Goals and Structure
|
X
|
|
X |
Program Service Components
|
X |
X |
|
Counterfactual Condition |
X |
X |
X |
Operational Challenges and Successes |
X |
X |
|
* Questions will be asked during first and second implementation visits to document change over time.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Patrick Cremin |
| File Modified | 0000-00-00 |
| File Created | 2021-01-11 |