Ready to Work Evaluation Process Study Interview Protocol
Introductory statement for grantee/site staff: The Ready to Work (RTW) Evaluation is being conducted by Abt Associates, under contract to the U.S. Department of Labor. As part of this evaluation, evaluation staff members will visit each of the four RTW evaluation sites. A major aim of the evaluation is to identify lessons learned from your experiences in implementing the projects under this initiative. In conducting site visits to each of the grantees, we are talking to project directors and staff, as well as partner organizations. We are here to learn about your service delivery model and understand how it is being implemented under the RTW grant. Our aim is to learn from your experiences, not audit or judge your programs. The views you express will be kept confidential, and nothing we publish in this evaluation will identify you by name.
Confidentiality Statement: Before beginning the interview, I (we) want to thank you for agreeing to participate in the study. I (we) know that you are busy and we will try to be as focused as possible. We have many questions and are going to talk to many different people, so please do not feel as though we expect you to be able to answer every question. And, we understand that your participation in this discussion is voluntary and you may choose to not answer some questions.
In addition, before we start, I want to let you know that though we take notes during these interviews, information is never repeated with the name of the respondent. When we write our reports and discuss our findings, information from all interviews is compiled and presented so that no one person can be identified. We also ask that you refrain from sharing anything we discuss today with others to help us ensure your confidentiality and the confidentiality of others we are interviewing. Do you have any questions before we begin?
Introductory statement for partner organizations: The U.S. Department of Labor is funding a major evaluation of the Ready to Work (RTW) grant programs that provide job training, coaching, and job placement services in fields that rely on the H-1B visa program. Abt Associates is conducting the study, which is called the Ready to Work Evaluation. We are members of the RTW research team. [Local site or program name] is participating in this evaluation. We are speaking with employers and service providers in the area to document the other types of employment and training services available in the community. Today we’d like to spend some time learning about your organization and the services you provide. Neither your name nor the names of your colleagues we speak to will appear in any public document. Do you have any questions before we begin?
Information about Respondent
Obtain the following information from each respondent:
Name, title and organization/affiliation
Role in the RTW grant program
Length of involvement with the organization and with the program
Educational background and prior work experience (briefly)
Local Context
Audience: Program Managers, Partners
What is the program’s service area? Possibilities include:
Portion of a city/county
Single county
Multiple counties (within one state)
Other (regional effort)
How large is this area (in terms of geography and population)?
What are the demographics of the service area? (e.g., race/ethnicity, immigrants)
Have the demographics been consistent over time, or have they changed recently?
What has been the economic environment in which your RTW grant-funded project has operated?
What are the major industries and/or large employers in the area? What industries have had the most change since the start of the grant?
In particular, how would you characterize the IT, advanced manufacturing, and/or bioscience industries? Who are the employers in these sectors?
Unemployment rate for area served (start of grant and at time of visit; pull from BLS ahead of visit and confirm any trends)
How would you characterize the demand for jobs in the IT, advanced manufacturing, and/or bioscience industries?
Availability of job openings in area served (generally and for particular population being served). Has there been an increase or decrease in the number of jobs in the grant-related industries?
Wage rates in area served (generally and for particular population being served)
Other local economic conditions that may have affected the project’s ability to recruit and retain participants and training participants’ ability to find employment (e.g., in- or out-migration of major employers, major layoffs, and natural disasters)
What are the community and government supports available to low-income and unemployed individuals, and dislocated workers in the service area? (These can include support from your organization that is not part of this program; support from the government, such as cash assistance; or support from organizations active in your community, such as scholarships.)
What services do they provide?
Do you collect information on program participants’ enrollment in these programs?
If so, what information do you collect? Can you provide this information to us?
Program Design and Operations
Grantee Organization
Audience: Program Managers
Please provide background on your organization. (Obtain brochure/recent annual report on the organization)
Type of organization
Organization’ budget for most recently completed program year
Organization’s major sources of funding (e.g., WIA, funding from federal/state/city agencies, foundations, private contributions, fee for service, etc.)
Organization’s total # of paid staff: _______ # of people, _______ # of FTEs
Types of clients/customers served or targeted
Summary of other regular services provided by the organization to clients (e.g., through Career Center)
Please provide background on any major programs/initiatives by your organization (besides the initiative that is part of the RTW evaluation) which are designed to assist the grant’s target population in obtaining job training and employment:
Name of program/initiative
Number and types of clients/customers served for most recently completed program year (unduplicated count)
Service area for program
Brief description of program content and services provided
Whether the program is linked in any way to the RTW program
Whether program is being evaluated and type of evaluation
Did you establish a new program with your RTW grant, or expand an existing program?
Were there components to the program that you had in place before you received the grant?
Describe any other organizational features that have affected the RTW program implementation and operations.
Program Goals and Start-Up
Audience: Program Managers, Partners (if involved in program design)
What are the overall goals of the program?
Who originally established these goals?
Have these goals changed over time? If so, how and why?
Are the goals modeled on or informed by an existing program model that has been successful elsewhere?
What is the theory of change that informs the program’s goals (i.e. are there short-term, intermediate-term, and long term goals that feed into desired program outcomes/or impacts)? Please explain.
How does the design of the program link to employer or local workforce needs?
Which specific industries does the program target?
How did you determine what components or services to include in the design of the RTW program (e.g., discussions with employers, observations of client needs, labor shortage)?
Have you modified the SOW (in your contract with the U.S. Department of Labor) during the course of the project?
If so, how and why? (e.g., difficulties recruiting participants, unwillingness/inability of partner to be part of initiative, change to time period or scope of work, etc.)
How did your project start-up and early implementation go (e.g., on-time, slow, etc.)?
What factors facilitated project start-up?
What factors hindered project start-up?
Organizational and Staffing Structure
Audience: Program Managers, Partners (if applicable)
Please describe the RTW program’s position within the organizational structure:
How is the program situated/managed within the larger organization or institution?
Who has authority over the program?
Is there a role for the local American Job Center and/or other employment service network in the delivery or management of the program? If so, please describe the role.
Besides the RTW grant, are there other sources of funding for these training and services?
Please describe the program’s staffing structure (number of staff in each position):
Do staff work exclusively on program activities, or do they split their time with other duties/programs?
Do your program staff members ever work off-site?
Is the program fully staffed or short-staffed?
Have you had any staff turnover since the start of the grant?
Have there been any changes to the staffing structure since the start of the grant?
Target Group and Recruitment
Audience: Program Managers, Program Staff (those involved in recruitment)
Which types of individuals form the target population of the program?
How did you decide on this target population?
Has the target population changed over the course of the grant period?
What exactly are the eligibility requirements for the RTW-funded program? (Document any age, education, work experience, and criminal background requirements as well as any documentation that applicants are required to provide during intake.)
How have you recruited participants? What methods have you used? Possibilities include:
Distribution of flyers, posters or other educational/informational materials
Linkages with unemployment insurance (UI) system
Informational websites
Toll-free informational hotlines
Outreach campaigns using media (e.g., TV, social media, radio, newspaper, ads on buses/bus shelters, etc.)
Direct mail campaigns, including letters to unemployment insurance recipients
Door-to-door outreach campaigns
In-person outreach presentations in the community (e.g., K-12 schools, neighborhood centers, American Job Centers, libraries, etc.)
Word-of-mouth
Did you alter your recruitment strategy over the course of the grant or evaluation? If so, in what ways, and did the changes work?
Have you worked with any other organizations to assist with recruitment and/or provide referrals to your program? Possibilities include:
Partners
Educational institutions
American Job Centers
Community or faith-based organizations
Courts/correctional system
Employers and businesses
Have there been recruitment challenges? If so, what challenges have been encountered and how have each of these challenges been addressed? Some possible challenges include:
Difficulty finding eligible participants
Many who applied did not meet program eligibility requirements
Some of the outreach strategies did not generate many applicants
Partner organizations did not provide enough referrals
There were not enough resources for recruitment
Changing economic or other conditions in the recruitment areas
Other similar programs competing for the same pool of participants
Potential participants unwilling to consent to participation in the evaluation
What has been the response of the targeted population to the initiative?
Please describe any “best practices” that have emerged out of the recruitment strategies you have employed.
Institutional, Community, and Employer Partnerships
Audience: Program Managers, Partners
Under your RTW grant, does your institution/organization operate all grant-funded programs and activities or do you contract, have financial arrangements, or have memoranda of understanding with others for some programs, activities, or other components?
What percentage of your RTW grant activities do you contract out to other organizations?
How many and which organizations do you have a contract or other financial arrangements with to operate RTW grant activities?
Describe each partner’s role in the RTW grant activities.
Had you partnered with the organization on other initiatives prior to the RTW one? If yes, was the RTW set up in a similar way (such as regarding organizational responsibilities and contractual issues)? If deviations were made, how and why?
If you had not partnered before, how/why were the partners brought into the grant effort?
(Ask of each partner, whether contracted or not) What services do you provide to the grant-funded program?
Who coordinates your interactions with the RTW program?
Have you encountered any difficulties in this partnership? If so, please describe.
Has your role changed over time? Please explain.
For each contracted organization, what is total amount of the subcontract?
What kinds of linkages does the program have with employers?
Did employers have any role in developing the program?
Do employers assist with course/curricula development and oversight?
To what extent do employers offer work-based training options, such as internships or on-the-job training positions? Please describe how these opportunities. What are participants and employers expected to get out of them?
Does the training program have a hiring pipeline or hiring quotas with any employers?
What other services does the training program offer employers (screening, drug testing, etc.)?
What challenges and opportunities do you see in working together on initiatives such as that funded by the RTW grant? Have you been satisfied with the relationship?
Program & Service Delivery
Intake and Assessment
Audience: Program Managers, Program Staff
Walk us step by step through the intake process. (Document which staff are involved and what study information staff present at each contact with applicant. Distinguish between what occurs before and after random assignment, such as what decisions staff make about eligibility and suitability, what assessments are taken pre- and post-random assignment, etc.)
Discuss what goes on at each contact with an applicant, such as during an initial phone inquiry, at an information/orientation session, in an individual meeting with a staff member. What information is shared with the applicant about the program and the study? What information do learn from the individual?
What decisions do you make about eligibility and suitability before and random assignment?
Are certain assessments (formal or informal) required before random assignment? Which do people do after random assignment?
What documents do you require applicants to present to determine eligibility?
How long does this intake process take from initial inquiry through random assignment?
What are the next steps after someone is randomly assigned to the treatment group and admitted into the program? What instructions are they given at this point on what to do next? When do you consider someone to be enrolled in the RTW program?
Who determines eligibility of individuals to participate in your training programs, and what, if any, criteria are used to select among candidates recruited?
How do you determine what training and services people will receive?
What academic and non-academic assessment tools do you use under this grant (e.g., TABE, interest inventories, substance abuse screening, etc.)? (Request a copy of any assessment materials)
At what point are the academic and/or non-academic assessments administered (i.e., before study enrollment or after)?
How do you use the information collected on the assessments?
Do you meet with people, either individually or as a group, to determine training and service needs? Describe these meetings.
How many meetings do you have with each person? At what point do they occur (i.e., before or after study enrollment and typical timing)? How long does each meeting last?
What questions do you ask or what topics do you cover?
How do you use the information you learn in the meeting? How do you determine in what training a participant will enroll?
What are the key skills and personal qualities you assess for?
What tools and practices have been most helpful? Why?
Do participants develop an individual service strategy, educational plan, or employment development plan? (Request a blank copy of the form used.)
At what point do they develop the plan?
What does the individual contribute to the planning process? What role does staff play in developing the plan?
Describe the content that typically goes into a plan.
How do you and the participant use the plan during the program, if at all? Do you revisit or revise it?
Is this a useful tool for the program? Are there any changes you would like to make?
Training Strategy
Audience: Program Managers, Program Staff, Instructors
Please describe the specific training programs offered. (Ask the following questions of each program offered within the short- and/or long-term options. Be sure to understand what specific programs are offered. Obtain copies of any curricula or syllabi used in the training programs.)
What training programs are offered and in what industry areas? (Probe for short-term and longer-term training.)
Approximately what proportion of participants do you anticipate will participate in each?
Is there an option for participants to transition from short- to longer-term training? Or do they select one or the other?
Who provides the training (e.g., your organization, a partner, community college or university, private training provider)? How are participants referred to the training provider and enrolled in the program?
How many students participate in the program at once? What is the average class size?
How many instructors are there?
What is the sequence of the curriculum?
How long is each of the program components (e.g., total hours; hours per day; days per week)?
How long does it take a typical student to complete the curriculum/program?
Is there open program entry and exit (i.e., participants can stop and restart at various points)?
Does the program assist individuals in moving from one step on a career ladder to the next?
Are certain courses or units required? If so what is required?
What kinds of credits do participants get for each course they complete?
What credentials do the programs lead to (e.g., certificates, degrees)?
Is basic/remedial skills training available? Describe.
Does the program use contextualized instruction that integrates basic skills and training content?
Are there any work-based components (internships, etc.)? Please explain.
What happens once a participant completes this training program? What are his/her next steps (e.g., enroll in another training, meet with a staff member, participate in employment services)?
Do you offer a job readiness component, such as a “bootcamp” intended to help participants with their employment search and maintaining employment once hired? Please describe what this consists of. (Obtain a program schedule, curriculum, syllabus, worksheets/course materials, etc.)
At what point during your time working with a participant do they take the job readiness component (i.e., at the offset, after occupational training, at any point that works for them)?
Is it classroom-based in-person or online?
How many students participate in the job readiness program at once?
Is the component available only to participants in the RTW-funded program, or is it available to a broader audience?
What is the average class size?
How many instructors are there?
What is the length and schedule of the job readiness program?
How long does it take a typical student to complete it?
Describe the curriculum for the program. Is it based on any existing curricula or models?
What happens after an individual completes the job readiness component? What’s the next step?
Do you offer OJT? If so, please describe this service:
Is this a required component of any of the programs offered through RTW? How do you determine if a participant is a candidate for OJT?
Who provides the on-the-job training?
How are placements made?
What is the length and hours of the program?
What is the level of the wage subsidy?
What is the nature of the training – is it “hands-on” or classroom based or both?
Can individuals be hired in unsubsidized employment at the end of the OJT?
What is the next step after OJT?
Do you offer work experience or internships? If so, please describe:
Is this a required component of any of the programs offered through RTW? How do you determine if a participant is a candidate for a work experience or internship?
Who provides the work experience or internship?
How are placements made?
What is the length and hours of the work experience or internship?
Are participants paid for participating in the work experience or internship? How is their compensation determined? Can you give some examples of positions and compensations?
What is the nature of the work experience/internship – is it “hands-on” or classroom based or both?
Can individuals be hired in unsubsidized employment following the work experience/internship?
What happens after someone completes their work experience or internship?
What can participants expect upon completion of the program?
Do they earn any specific credentials?
If so, which ones?
Are they industry credentials (i.e., established and recognized by the industry) or were they developed by the grantee or other entity?
Are these recognized by employers or industry? (Get at what the credentials mean to employers.)
Is this an established credential or was it created for this program?
Does earning that credential require simply completing the course? Or is there some external exam or practicum that must be passed? Is the exam/practicum required or included as part of the program or do participants register for and take it separate from the program?
What connections to employers and/or career pathways are available to participants?
What changes to the training program have occurred over time? Please describe.
Employment, Counseling, Financial, and Support Services
Audience: Program Staff, Partners
What are the major support services available to program participants through your grant-funded program? Please explain in detail.
Probe specifically for: career coaching and job placement services; counseling (academic, personal, and financial); financial assistance or needs-related payments, child care and transportation, and life skills training.
Who provides these services?
How/where/when are they delivered?
How frequently do participants meet with support staff? Is there a requirement for how often or how many times they meet?
How do you determine what support services a participant will receive? Are any components required?
To what extent do participants use these services? What proportion of participants take up each of the services?
What is distinct about these services under the RTW grant compared with what your organization or partners offer otherwise?
Are these services available to other individuals not participating in the RTW program?
What types of job development activities does your program undertake (if any)? Describe any connections you make with employers to help participants find employment.
If they are not available in your grant-funded program, do students in your program access these services from other programs/places within your organization or in the community? Please describe.
Are there any requirements placed on program participants to utilize these services (e.g., meeting with a counselor or case manager every month)? What happens if they don’t comply with the requirement?
Does the program provide referrals to participants for other services?
If so, which organizations do you provide referrals to?
What services do these organizations provide?
Do you know if participants have been successful in obtaining services from these providers? If not, why (e.g. high demand)?
What services/supports are available upon program completion?
Probe for: connections to job openings, job search services, job retention, work supports, and internships/apprenticeships.
How long are these services generally provided to individuals after they leave the program?
What data are collected about/from individuals who utilize services after program completion?
To what extent do students use these services?
Service Utilization, Performance Management, and Data Collection
Participation Levels and Program Completion
Audience: Program Managers, Program Staff
How do you track student participation, progress and performance in each component of the training program? (Ask for each component if there are multiple phases of a program and probe for assessments, attendance, grades, etc.)
What assessments do you use?
How are these data collected, when (timing and frequency), and from whom?
How would you determine if a participant was having problems? What action would be taken?
How many individuals have completed the training to date? (Inquire about the RTW program as a whole, including veterans and people served before the study started, not just the research sample.)
What proportion of those who start the program complete the program?
Of those who complete, what proportion pass the certification or exam (if applicable)?
What is the total number of individuals you are expecting will complete the training throughout this grant program? Is training completion proceeding on target? Are certifications or degrees proceeding on target?
Do you track how often participants utilize employment, counseling, or financial services offered as part of the program?
How are these data collected, when (timing and frequency), and from whom?
Have you noticed any trends in the utilization of support services? If so, please describe.
What strategies do you employ to maintain participant engagement, attendance, and ultimately completion?
Are participants progressing in the program as intended by design (i.e., are they participating in the components you anticipated, in the order you expected, getting jobs)? What factors do you think contribute to or inhibit this progression?
Do you have issues with participants dropping out before completion? If so, about what percent of participants have dropped out in the course of this grant? Do you use any particular retention strategies?
Why do you think these participants were unable to complete?
What options are available to participants who aren’t able to complete the program?
What other pathways are they likely to pursue?
Do you provide referrals for other services in the community to participants who are unable to complete the program or otherwise meet program requirements? If so, please describe.
How many individuals have been placed in permanent jobs after completion to date (if there are any completions and this information is tracked)? Are these jobs in the same industry or occupation as the training? (Please do not include any incumbent workers in these numbers.)
Performance Management
Audience: Program Managers, Program Staff, Partners (if applicable)
What measures are in place to track the performance of the training program?
Are these measures mandated by your RTW grant?
Are the measures stored in an electronic system?
How do your track the performance of program components that are delivered by partner organizations?
What measures are in place to track the performance of the support services?
Are these measures mandated by your RTW grant?
Are the measures stored in an electronic system?
How do you track the performance of program components that are delivered by partner organizations?
Do you collect information from participants about their experiences in the program? About their employment status following program completion?
If so, please describe how you collect this information and what you do with it.
Can you describe the recent trends in the performance of the program overall?
Data Collection
Audience: Program Managers. Staff involved with data entry or data management
What data are collected by/about the RTW program and its participants? (Obtain a list of fields collected and/or a data dictionary)
Do you collect data related to both training and support services? Elaborate on the types of information you collect:
For example, for training, do you collect information on which program someone enrolls in, their attendance, completion, start and end dates of their enrollment, certificates attained?
For example, for support services, do you collect information on number of counseling/career coaching sessions attended by participant, length of each session, financial support received (type of financial assistance and dollar value), employment services received, whether someone participated in OJT or paid work experience (length, compensation), employment outcomes?
How are these data collected, when (timing and frequency), and from whom?
What tools or data systems do you use to enter and store the data (e.g., database such as Salesforce, Efforts to Outcomes, Excel, Access, multiple systems/tools)?
In what format is the data (e.g., web-based, exportable to Excel spreadsheet or .csv file)?
Is the database specific to the RTW grant or does the organization use the same system for other programs it offers? (Explore here whether a data export could include data on other activities or services a treatment group member participated in through the grantee. Also explore whether it is feasible to obtain data from the organization on activities/services pursued by control group members.)
How do you use or plan to use these data (i.e. what is the purpose of the data collection)?
Counterfactual Condition
Control Group Services
Audience: Program Managers, Partners
What are the other training programs individuals can access in the community?
What type of training do they provide?
Who operates these programs?
Where are these programs located?
Is there an application process/waiting period to access these services
When individuals are assigned to the control group what service referrals do you provide them? How is the referral made, for instance do you provide a list of other services or do you actively hand the person off to another staff member or partner?
To what extent are you aware that control group members are utilizing these services?
What barriers might they have to accessing these services?
Do you track what services control group members access?
Implementation Accomplishments and Challenges
Program Successes and Challenges
Audience: Program Managers, Program Staff, Partners
Are there ways in which the program has exceeded or fallen short of meeting its goals? Please describe. (Probe for whether grantee has been able to involve partners in the way desired/needed and whether grantee has been able to engage employers from the relevant industries.)
What do you believe have been the program’s greatest successes? What are the main factors that contributed to this success?
What challenges did the program encounter during start-up? How were these challenges overcome?
What challenges has the program encountered since then? How were these challenges overcome?
What are the most important areas for program improvement that your program or your organization/institution has identified? How do you plan to address these?
Are you planning on making any modifications and/or changes to the program?
Are there other approaches, strategies, or services that you believe would contribute to better outcomes for the target constituency?
OMB No. 1205-0507, Exp. 5/31/2016
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Jill Hamadyk |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |