OMB No. 1290-XXXX
Exp. Date XX/XX/2021
OMB Package
Evaluation of the American Apprenticeship Initiative (AAI)
Site Visit Interview Guide:
I am (we are) researchers from Abt Associates, a private research organization based in Cambridge, MA, which conducts policy-related research on social and economic issues. This project is being conducted by Abt Associates and Urban Institute under contract to the U.S. Department of Labor. The goal of our visit today is to help us understand the early implementation experiences of American Apprenticeship Initiative (AAI) grant-funded projects. The evaluation (DOL-ETA-16-F-00006) aims to identify lessons learned from your experiences in implementing the projects under AAI. As part of this evaluation, we are conducting visits to 10 grantees. In addition to our site visits, we are collecting and analyzing data on program participation, services, and outcomes provided by the U.S. Department of Labor as well as administering a survey to a subset of AAI participants.
During our site visits to each of the project sites, we are interviewing AAI grant project directors and staff, as well as partner organizations and employers (and those sponsoring the apprenticeship). We are here to learn about your service delivery model and understand how it was implemented under the AAI grant. Our aim is to learn from your experiences, not to audit or judge your programs.
Before beginning the interview, I (we) want to thank you for agreeing to participate in the study. I (we) know that you are busy, and we will try to be as focused as possible. We have many questions and are going to talk with many different people, so please do not feel as though we expect you to be able to answer every question. Your participation in this discussion is voluntary, and you may choose not to answer any question.
In addition, before we start, I want to let you know that though we take notes at these interviews, information is never repeated with the name of the respondent. When we write our reports and discuss our findings, we compile and present information from the full set of interviews without identifying any one person. We do present information at the organization level but in ways that do not identify particular individuals. Do you have any questions before we begin?
[Interviewer Instructions: If the interview is going to be recorded, inform the respondent and request permission to record the interview.]
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. Public reporting burden for this collection of information is estimated to average 1 hour per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. The obligation to respond to this collection is required to obtain or retain benefit (FOA-ETA-15-02). Send comments regarding the burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to [email protected] and reference the 0MB Control Number 1290-XXXX. Comments can also be mailed to: U.S. Department of Labor, Chief Evaluation Office, 200 Constitution Ave., NW, S-2312, Washington, DC 20210. Note: Please do not return the completed interview guide to the email or mailing address.
A. STAFF BACKGROUND AND PROGRAM RESPONSIBILITIES
What is your job title in the [name of AAI program]?
Do you work full-time or part-time for [name of AAI program]? If part-time about what percentage of your time is with [name of AAI program]? Is the grantee the sponsor for one or more apprenticeship program(s)? If so, please describe.
What are your job duties with respect to AAI?
For example, do you recruit and/or otherwise interact on a regular basis with apprentices or pre-apprentices? If so, in what ways for what percentage of your time?
Do you recruit and/or interact on a regular basis with employers? Are the employer(s) or other partners the sponsors for one or more apprenticeship program(s)? If so, in what ways and for what percentage of your time?
Do you have other duties that are not related to the AAI grant? If so, what are they and for what percentage of your time?
For how long have you had this job? For how long have you worked in apprenticeship? For how long have you worked in workforce development or occupational education? For how long have you been working on the AAI grant?
What training did you receive for your job in the [name of AAI program]
B. EMPLOYER RECRUITMENT AND ENGAGEMENT
(Note to interviewer: ask this module only if staff member is involved with employer (and/or sponsor) recruitment and engagement.)
Note: Prior to the site visit, collect the following information from the grant application:
Who were the collaborating employers listed on the proposal? Are any of these also sponsors?
[Provide a full list of employers.]
What methods did the proposal specify for reaching out to employers(and sponsors as applicable)?
What roles did the grantee propose to play in registering company apprenticeship programs?
In what ways did you as grantee expect to organize groups of employers? By sector, by location?
If available in the proposal, which employers listed on the proposal already had at least one registered apprentice program?
Note that although you should ask every question, the informant may not be able to answer all of them.
Is the grantee focusing on new employers, expanding apprenticeship slots at existing employer sites, or both? [Note: please consider any differences between new versus existing employers when responding to following questions. Consider if the employer is also a sponsor and if follow-up questions might be appropriate as related to this.]
If you are involved in recruiting new employers for the AAI grant, what methods have you used to identify potential employers (and sponsors) to participate in the AAI grant?
What methods have you used to reach out to and recruit employers (and sponsors)? Which methods have proved more and less successful? Some possible recruitment methods include:
Direct contact through telephone or emails
AAI program staff outreach presentations or orientations with employers or business organizations (estimate number provided please)
Suggestions or referrals from other organizations and employers (e.g., the Chamber of Commerce, other businesses organizations, employers sponsoring apprenticeships, workforce development programs, including WDB, AJCs, community colleges, etc.)
Outreach campaigns using media (e.g., PSAs, TV, radio, newspaper, etc.)
Websites/social media (e.g., LinkedIn, Facebook, etc.) – please specify, which types of social media are used and how
Word-of-mouth
Other, please specify
How many employers (and sponsors) have responded to the outreach and recruitment efforts?
Have you been able to recruit the number of employers (and sponsors as applicable) you expected? Will efforts continue? For how long? Any difference by industry/occupation?
Has the pool of potential employers (and sponsors as applicable for the program been sufficient given the number of planned apprenticeships? Any difference by industry/occupation?
Have there been more employers (and sponsors as applicable) interested in engaging with the AAI program than can be included (e.g., is there a waiting list for employers to join the program)? Any difference by industry/occupation?
Have you encountered barriers to employer (and sponsors as applicable) recruitment? If so, what have been some of the key barriers in persuading companies to start or expand apprenticeship programs? Possibilities include:
Lack of knowledge about apprenticeship
Difficulty in identifying apprentice occupations
Administrative hassle in registering the program
Costs of the program
Difficulty in finding related instruction provider that is a good fit
Difficulty on finding interested and qualified apprentices (if expanding slots)
Lack of trainers
What are the key factors that motivate employers (and sponsors as applicable) to start or expand apprenticeship programs? Possibilities include:
Existing or future shortage of qualified workers
Expected benefits during and soon after the apprenticeship
AAI grant support for
On-the-job learning (overhead costs, shadowing, mentoring)
Develop and deliver related technical instruction (RTI)
Development of skills standards
Apprentice recruitment
Support services
Ability to benchmark skills to high levels
Screening process effective at identifying workers likely to succeed
Help in reaching a diverse, qualified workforce
National credential linked to registered apprenticeship
Assistance in registering apprenticeships
Other
If grantee is recruiting employers from more than one industry/occupation, how do recruitment efforts vary, if at all?
For employers that agree to begin (and sponsors as applicable) an apprenticeship program, take us step-by-step through the process of planning, implementing and registering an apprenticeship program.
C. TARGET POPULATION AND PARTICIPANT RECRUITMENT
(Note to interviewer: ask this module only if staff member is involved with participant recruitment.)
Which organizations are involved in participant recruitment? Does AJC have a role? If they are, is this working well, or where might there be improvements? If not, should they?
How does the grant/partners market apprenticeships to prospective participants? Possibilities include:
Distribution of flyers, posters or other educational/informational materials
AAI program staff outreach presentations or orientations held in the area served (estimate number please)
Informational websites
Toll-free informational hotlines
Outreach campaigns using media (e.g., PSAs, TV, radio, newspaper, ads on buses/bus shelters)
Social media (e.g., LinkedIn, Facebook, Twitter, Instagram, etc.) – please specify, which types of social media are used and how
Word-of-mouth
For primary partners, tapping own customer base
What populations are targeted for special outreach? Does this vary by industry (if applicable)? Why were these populations selected? Since the start of the AAI initiative have there been any changes in populations targeted? If so, what are they? Some possible special target groups include:
Incumbent workers
New entrants to the labor force
Long-term unemployed
Low income
Populations that may traditionally be underrepresented in apprenticeship, including:
Women
Men and women of color
Individuals with disabilities
Low-skilled populations
Veterans, including transitioning service members
Out-of-school youth
Formerly incarcerated
What organizations provide referrals of prospective AAI participants? Which organizations have provided the most referrals? Have any organizations fallen short of or exceeded original expectation for providing referrals? Which organizations are sponsors and for which apprenticeships/how many per apprenticeship/industry/occupation? Possibilities include:
Employers
Unions
Workforce system – American Job Centers, Employment Service, and WIOA
Local Education Agencies (LEAs), such as the local public school system
Re-engagement centers (i.e., to link youth that have dropped out of school back to school or other types of training)
Community colleges and other educational institutions including proprietary schools
Courts/correctional system/juvenile justice/probation officers
Faith-based and community-based organizations
Other public and non-profit human service agencies
What has been the response of the targeted population(s) to the initiative, including underrepresented populations?
Is the AAI-funded project expected to meet its enrollment goal (total and by industry/occupation)? If not, why not?
Has the pool of potential applicants for the program been sufficient (by industry/occupation)?
Have there been recruitment challenges? If so, what challenges have been encountered and how have they been addressed? If not, what do you think are the reasons? Some possible challenges include:
Have difficulty finding eligible participants
Many who applied have insufficient basic skill levels
Some applicants are not interested or motivated to participate in apprenticeship
Apprenticeship period is too lengthy for some applicants
Some of the outreach strategies haven’t resulted in many applicants
Partners that were supposed to provided referrals have not provided them in sufficient numbers
Some applicants have difficulty getting to the program/employer location
Didn’t have enough resources for recruitment
Changing economic conditions in the areas has made recruitment more challenging
Other, please specify
Overall, what recruitment or referral strategies being used under the AAI grant have been most successful? Do you feel any of these strategies are new, unique and/or innovative? Replicable?
D. AAI PARTICIPANT ELIGIBILITY, INTAKE, AND ASSESSMENT: APPRENTICESHIP PROGRAMS
(Note to interviewer: ask this module only if staff member is involved with eligibility, intake or assessment; it is possible that the individual will be involved in only part of this process.)
Which organizations are responsible for eligibility assessment and intake of potential apprentices? Does AJC have a role? If they are, is this working well, or where might there be improvements? If not, should they?
What are the main steps involved in intake and enrollment? Please provide an overview of the flow of new recruits through the intake process. [Request copy of service delivery flow chart if one exists.]
What are the eligibility requirements to enroll in the apprenticeships funded under the AAI grant? [Note: Request copies of AAI participant eligibility or intake form(s).] Possibilities include:
Complete pre-apprenticeship program (i.e., for apprenticeship program component)
Be from a specific subpopulation
Reside within a specific geographic area
Meet income eligibility requirements (what are these requirements)
Meet education level requirements (e.g. high school diploma, GED/HSED) or basic skill levels
How are applicants screened (if at all) before being sent on interviews with employers? Please describe this process including (if applicable):
How does the AAI program (or other entity) determine whether an applicant is the right fit?
About what percentage of applicants are rejected (e.g., screened out prior to being sent for employer interviews)?
What are the main reasons that applicants are screened out or fail to move through the intake process to be considered by employers?
What role do employers (and/or sponsors) play in the screening and selection process? Please describe this process including (if applicable):
How do employers generally determine whether an applicant is the right fit for an apprenticeship program?
About what percentage of applicants are rejected (e.g., screened out during the employer interview process)? What are the major reasons employers reject applicants?
During the intake process, are any formal assessment tools/tests used by employers or the AAI grantee (including formal tests such as the TABE, interest inventories, substance abuse screening)? If yes, please identify these formal assessments. [Note: Request copies of assessment forms, if relevant.]
E. MAIN TRAINING PROGRAM COMPONENTS/SERVICES – EXAMPLE(S) OF APPRENTICESHIP TRAINING FUNDED UNDER AAI
[Note to interviewer: Ask this module only if staff member is involved in training program component or services; it is possible that the individual will be involved in only a portion of these services. Cover these topics to the extent that they are not covered during the earlier discussion of employer involvement. It may be necessary to focus in on a particular employer or apprenticeship program as an example of AAI-funded apprenticeship program. If a visit is conducted to a particular apprenticeship program or programs as part of the site visit, it may be appropriate to skip these questions and cover them at the sponsor, employer or training site where the apprenticeship training is being provided.]
Is the AAI-funded apprenticeship program time-based (e.g., completion of specific numbers of hours of training), competency-based or a combination of the two?
What is the usual duration of participation in the apprenticeship? Is there variation in how long it takes an AAI participant to complete the apprenticeship? If yes, what is the range of time (in weeks, months, or years) and why might it vary on a case-by-case basis?
If competency-based, how is it determined that a participant has achieved the necessary level of competence to complete the apprenticeship?
For each occupation, how many hours of RTI (i.e., classroom/web-based) are provided during the apprenticeship component? Over what period of time does RTI occur?
What is the typical schedule for an apprentice each week – e.g., 4 hours of formal classroom instruction and 36 hours of work)? Does this schedule change over the course of the apprenticeship?
Where and how is instruction provided (e.g., classroom, web-based, hybrid, laboratory/shop)?
Where is instruction provided (e.g., at a community college, at a union, at an employer site)?
Please provide a brief overview of the instruction provided (e.g., topics covered in classroom instruction). [Note: If applicable, request a syllabus for the coursework or a topic outline for the training.]
Are there innovative RTI methods being used as part of the apprenticeship initiative? (For each, please provide provider.) Please briefly discuss. Possibilities might include:
Accelerated learning modules
Contextualized learning
Team teaching
Self-paced learning
Competency-based learning
Prior learning assessments
Technology-enabled learning, such as use of simulators, on-line teaching/learning
Integrating basic skills instruction with technical instruction
Has AAI funding been used for any of the following with respect to RTI, and if so, how:
Development of courses at post-secondary education level that are integrated into apprenticeship
Tuition and other educational fees
Delivery of instructional requirements (e.g., virtual learning technology, classroom instructors)
Cost of training facilities
Books, computers, etc.
While involved in on-the-job training, how many hours of work at an employer site does an apprentice complete (i.e., each week, month, or year). Does this vary over time and, if so, how?
Where do apprentices work -- for which employers and in what types of jobs? For example, in what types of jobs do apprentices start and progress to during their involvement in the apprenticeship?
Who oversees and provide workplace/on-the-job instruction to the apprentice (i.e., while the apprentice is working)? Please briefly describe the nature of the on-the-job instruction.
What is the supervisor-to-apprentice ratio during the apprenticeship period (e.g., 1 supervisor to 3 apprentices)? Does it change over time?
With regard to on-the-job learning, have AAI funding been used to offset overhead costs associated with any of the following (and if so, how): providing training, job shadowing, mentoring, and additional supervision?
Is the apprenticeship training connected to higher education? If so, which institution(s) and how? Do AAI apprenticeship participants receive post-secondary (college) credits, credentials, or certificates? What are the credits, credentials, and/or certificates? What is the schedule for receipt of these credits, credentials, or certifications?
What is the schedule of wages and benefits for apprentices?
To what extent are non-training supports provided during the apprenticeship period (e.g., case management, career coaches/navigators counseling, and mentoring)? Who provides these services?
To what extent are supportive services provided such as transportation assistance, work clothes/equipment, childcare, and needs-based payments? If provided, how are these supportive services structured and paid for? Who provides these services?
What percentage of participants complete RTI (total and by industry/occupation)? Are there certain types of apprentices that are more/less likely to complete RTI? What are suspected reasons?
What services are provided to apprenticeship completers? Who provides them? Are incentives/incentive payments provided to completers?
LESSONS LEARNED TO DATE
To date, what do you consider to be the most important accomplishments under the AAI grant?
To date, what do you believe to be the main lessons learned from your AAI grant for pre-apprenticeship and/or apprenticeship program design and operations?
What innovations and lessons learned under the AAI grant can contribute to efforts to encourage employers or others to adopt pre- and/or apprenticeships?
What factors seem to explain success or lack of success for the apprenticeship program?
Are there recommendations for the WDB and/or AJCs to increase or enhance their engagement with grantees? Federal and state OA? Economic Development Agency/Agencies?
CHECKLIST OF ITEMS TO COLLECT FROM SITE (IF AVAILABLE)
Organizational chart
Copy of service delivery flow chart, if one exists
Copy of AAI participant eligibility or intake form(s)
Copies of assessment forms
Copy of syllabus for the coursework or a topic outline for the training
SUPPLEMENTAL DISCUSSION GUIDE:
PRE-APPRENTICESHIP TRAINING PROGRAM COMPONENT
[Note to interviewer: Omit this supplement if the AAI grant does not fund a pre-apprenticeship program component. If more than one pre-apprenticeship program component exists or training is provided for more than one type of occupation, where appropriate, indicate differences across programs/industry sectors/occupations on each of the questions. Ask this module only if staff member is involved in the pre-apprenticeship program component; it is possible that the individual will be involved in only a portion of these services.]
[If not already covered] What are your duties in the pre-apprenticeship program? About how much of your time is spent working with pre-apprentices?
If not already discussed, for which industries and occupations are pre-apprenticeship training provided? Why were these industries/occupations targeted?
Does the pre-apprenticeship program partner with your AAI apprenticeship program? If not, does it partner with another Registered Apprenticeship? If so, which one?
How do you market your program to prospective pre-apprenticeship participants? Possibilities include (collect samples if possible please):
Distribution of flyers, posters
AAI program staff outreach presentations or orientations held in the area served (estimate number provided please)
Informational websites
Toll-free informational hotlines
Outreach campaigns using media (e.g., PSAs, TV, radio, newspaper, ads on buses/bus shelters)
Social media (e.g., LinkedIn, Facebook, Twitter, Instagram, etc.) – please specify, which types of social media are used and how
Word-of-mouth
Primary partners
What populations and/or subpopulations has the project targeted for special outreach to the pre-apprenticeship program? Why have these populations/subpopulations been selected? As the AAI initiative has progressed, have there been any changes in special populations targeted? If so, what were they? Why? Some possible special target groups include:
Incumbent workers
New entrants to the labor force
Long-term unemployed
Low income
Populations that may traditionally be underrepresented in apprenticeship, including:
Women
Men and women of color
Individuals with disabilities
Low-skilled populations
Veterans, including transitioning service members
Out-of-School Youth
Formerly incarcerated
What organizations provide referrals to the AAI-funded pre-apprenticeship program? Possibilities include:
Employers/Sponsors
Unions/Sponsors
Workforce system – American Job Centers, Employment Service, and WIOA
Local Education Agencies (LEAs), such as the local public school system
Re-engagement centers (i.e., to link youth that have dropped out of school back to school or other types of training)
Community colleges and other educational institutions including proprietary schools
Courts/correctional system/juvenile justice/probation officers
Faith-based and community-based organizations
Other public and non-profit human service agencies
Which of those organizations have provided the most referrals? Are these sponsor organizations too? Have any organizations fallen short of original expectation for providing referrals?
What has been the response of the targeted population(s) to the initiative?
Are you on pace to recruit the intended number of pre-apprenticeship participants? If not, why?
Has the pool of potential applicants for the program been sufficient?
How successful has the AAI grant been in recruiting individuals from underrepresented groups?
Have there been recruitment challenges? If not, why do you think this? If so, what challenges have been encountered and how have they been addressed? Some possible challenges include:
Have difficulty finding eligible participants
Many who applied have insufficient basic skill levels
Some applicants are not interested or motivated to participate in apprenticeship
Apprenticeship period is too lengthy for some applicants
Some of the outreach strategies haven’t resulted in many applicants
Partners that were supposed to provided referrals have not provided them in sufficient numbers
Some applicants have difficulty getting to the program/employer location
Didn’t have enough resources for recruitment
Changing economic conditions in the areas has made recruitment more challenging
Other, please specify
Overall, what recruitment or referral strategies being used under the AAI pre-apprenticeship program have been most successful? Do you feel any of these strategies are new, unique and/or innovative? Replicable?
What are the main steps involved in the AAI pre-apprenticeship intake process? Please provide an overview of the flow of new recruits through the intake process, including:
What are the eligibility requirements to enroll in the pre-apprenticeship component funded under the AAI grant? [Note: Request copies of AAI participant eligibility or intake form(s).] Possibilities include:
Be from a specific subpopulation
Reside within a specific geographic area
Meet income eligibility requirements (what are these requirements)
Meet education level requirements (e.g. high school diploma, GED/HSED) or basic skill levels (what is level – 8th grad and above?)
Who determines eligibility to participate in the AAI grant? What role does staff from the associated Registered Apprenticeship play in recruitment and intake?
During the intake process, are any formal assessment tests used to screen pre-apprenticeship candidates (e.g., TABE, interest inventories, substance abuse screening)? [Note: Request copies of assessment forms, if relevant.]
14. Describe development of the curriculum for pre-apprentices? Were staff from the registered apprenticeship involved? What part did the AAI grant staff play? Employers? Educational/training providers? How is pre-apprenticeship training delivered, where and by whom?
15. Are there innovative instructional methods being used? Who is using them/each? Please briefly discuss. Possibilities might include:
Accelerated learning modules
Contextualized learning
Team teaching
Self-paced learning
Competency-based learning
Prior learning assessments
Technology-enabled learning, such as use of simulators, on-line teaching/learning
16. Per pre-apprenticeship, occupation…What is the nature of hands-on training? Is it volunteer work? Simulation lab? Who provides the training? What is the length of training?
17. What other services are provided to promote success in the pre-apprenticeship program and beyond? Who provides each/them? Examples include:
Job readiness classes
Career success workshops
Basic skills education (ESL or ABE)
Other
18. To what extent are non-training supports provided, such as case management, career coaches/navigators counseling, or mentoring? If provided, who provides them and how are these supports structured and paid for? If applicable, discuss differences across industry sectors/occupations for which AAI pre-apprenticeship training is being provided.19. To what extent are supportive services provided such as transportation assistance, childcare, work clothing/equipment, or needs-based payments? If provided, who provides them and how are these supportive services structured and paid for? If applicable, discuss differences across industry sectors/occupations for which AAI pre-apprenticeship training is being provided.
19. Is the pre-apprenticeship training connected to higher education (articulation agreement/transfer of credits/courses)? If so, which institution(s) and how? Do AAI pre-apprenticeship participants receive post-secondary (college) credits, credentials, and/or certificates? Please specify types of credits/credentials/certificates received. If applicable, discuss differences across industry sectors/occupations for which AAI pre-apprenticeship training is being provided.
How is the pre-apprenticeship connected to registered apprenticeship program(s)? Are there apprenticeship slots available for all pre-apprentices who successfully complete?
What other post-completion services are provided to participants? If applicable, who provides them/each and discuss differences across industry sectors/occupations for which AAI pre-apprenticeship training is being provided.
What percentage of AAI participants complete(d) pre-apprenticeship training? [Note: If available, collect this data from administrator sources prior to the visit or request data on attrition from the grantee.]
Is attrition greater than expected? If so when and why does attrition typically occur?
Are there certain types of participants that are more/less likely to complete pre-apprenticeship training?
If applicable, discuss differences across industry sectors/occupations for which AAI pre-apprenticeship training is being provided.
What percentage of AAI pre-apprenticeship completers enter apprenticeship programs? [Note: If available, collect this data from administrator sources prior to the visit or request data on attrition from the grantee.]
Are there factors that help explain the success or lack of success?
If applicable, discuss differences across industry sectors/occupations for which AAI pre-apprenticeship training is being provided.
What (if any) training participation and performance benchmarks are used for the AAI-funded pre-apprenticeship program? Are incentives/incentive payments used to reach performance levels?
Were any pre-apprenticeship training program components or service elements of the original program design not implemented or abandoned early on? If yes, which ones and why?
What factors seem to explain success or lack of success for the pre-apprenticeship program?
Have you faced any challenges in establishing the pre-apprenticeship training component? How have these challenges been addressed?
What aspects, if any, of the pre-apprenticeship program component do you feel are most innovative? Effective? Replicable?
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | ATTACHMENT 7 |
| Author | admined |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |