Attachment G - YARG Interview Protocol for Program Staff

^{Attachment G}

Apprenticeship Evidence-Building Portfolio

Youth Apprenticeship Readiness Grant Semi-Structured Interview Protocol for Program Staff

Apprenticeship Evidence-Building Portfolio

Youth Apprenticeship Readiness Grants (YARG)

YARG Administrator and Staff Discussion Guide

INTRODUCTION 

I am/we are researchers with The Urban Institute/Mathematica/Capital Research Corporation, private research organizations based in Washington, DC/Arlington VA which conduct policy-related research on a variety of social welfare and economic issues. This project is being conducted under contract to the U.S. Department of Labor.

Our visit here today is part of the Apprenticeship Evidence-Building Portfolio project, a national study sponsored by the U.S. Department of Labor. The study will learn how apprenticeship programs can help improve the skills and employment outcomes of American workers. A major aim of the study is to learn more about the different models of apprenticeship being implemented. In each grantee we visit, we will be speaking with grant administrators and staff, program partners (including employers), and apprentices. For this discussion, we are focused on activities conducted under the Youth Apprenticeship Readiness Grants (YARG).

Privacy Statement:  I/we know that you are busy and we will be as focused as possible and will only ask questions that are relevant to your experience. We have many questions and will be talking to many different people, so please do not feel as though we expect you to be able to answer every question. Your participation in this interview is voluntary and you may choose not to answer some or any of our questions. We will do everything we can to secure the privacy of the information you provide, but the technical limitations with Zoom and other internet platforms mean that we cannot guarantee the confidentiality of what is said. Also, we ask that you consider using headphones or finding a private space to talk so that other people cannot hear or see this conversation. We believe the risks of participating in this study are minimal.

We believe there are no direct benefits to you, but we hope that the findings from this study will benefit the Apprenticeship Evidence-Building Portfolio project by providing research insights on how apprenticeship programs can benefit workers.

My colleague and I will be taking notes in order to document what we hear during our discussion, and we may record this discussion. We do not share these notes with anyone outside of our research team, including Department of Labor, and we will destroy these notes after the end of our project. When we compile our reports, the names of individual respondents will not be included. If we choose to quote you, you will only be identified by your title. You will not be quoted directly by name in any of our reports. While it is possible that you might be identified by your title, we will do our best to minimize the chance of that occurring.

Finally, to help us accurately capture the information you share, we would like to record this interview. The recording is just a back-up for our notes and will be kept within our small research team. The interview recording will be deleted once we have developed a full set of notes from the interview.

OMB Burden Statement: According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays an Office of Management and Budget (OMB) control number. The valid OMB control number for this information collection is XXXX-XXXX. The time required to complete this collection of information is estimated to average 60 minutes, including the time to review instructions, search existing data resources, gather the data needed and complete and review the collection of information. Send comments regarding the burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to [email protected] and reference the OMB Control Number XXXX-0NEW.

Do you have any questions before we begin?

[If we decide to record the interview] Are you okay with us recording the interview to improve the accuracy of our notes?

Do I have your permission to begin the interview? [BEGIN INTERVIEW IF CONSENT GRANTED]

Note: Document the following information for each respondent:

• Name, title and organization/affiliation

• Role in the grant program and other organization roles

• Length of involvement with the organization and with the program

1. BASIC GRANT INFORMATION

1. Quickly verify with the respondent the following background information about the grant. [Note: Collect and pre-fill this information prior to the visit from the grant application or other available documentation and verify with the respondent.]

1. Original grant period: ___________ to ___________

2. Modified grant period (if applicable): ___________ to ___________

3. Does your grant include pre-apprenticeships?

4. Does you grant align high school Career and Technical Education (CTE) programs with apprenticeship/pre-apprenticeship programs?

5. Is a college or other training provider involved in providing related technical instruction (RI) to the apprentices?

6. For registered apprenticeship programs, please identify the following:

1. Industries targeted:

2. Occupations targeted:

7. For pre-apprenticeship program component, please identify the following:

3. Industries targeted:

4. Occupations targeted:

2. Have there been any modifications under your grant (e.g., scope of work, budget, performance period, etc.)? If yes, please briefly describe. [Note: Collect and pre-fill this information prior to the visit from the grant application or other available documentation and verify with the respondent.]

3. What is your organization’s previous experience with apprenticeships and pre-apprenticeship programs (prior to receipt of the grant)? [e.g., how long have you had an apprenticeship or pre-apprenticeship program?]

2. ENVIRONMENTAL CONTEXT

1. What geographic area does your program serve, including those operated by the grantee and various partners (e.g., identify specific states, counties or regions served])?

[Note: Grantees could choose Local/Regional, Statewide, or Nationwide service areas. Collect and pre-fill this information prior to the visit from the grant application or other available documentation and verify with the respondent.] ]

2. What is the economic environment and context for the area served, particularly as it relates to in- and out-of-school youth that may face barriers to employment? [Note: Collect and pre-fill this information prior to the visit from the grant application or other available documentation and verify with the respondent.]

[Probe for: growth industries, extent to which industries rely on apprenticeship; industries that are most likely to feature apprenticeships/pre-apprenticeships; economic conditions such as job openings in sectors/occupations that are focus of grant]

3. What was the apprenticeship “infrastructure” (especially for youth apprenticeships and pre-apprenticeships) like in the geographic area before your got this grant. Were there earlier programs or other programs you know of? Did you have apprenticeship linkages with your grantee/partners before this grant?

[Probe for: established relationships with apprenticeship programs, state apprenticeship agencies, apprenticeship intermediaries, industry associations, unions, schools, American Job Centers, and employers]

4. Is there any overlap/linkage between this grant-funded project and other apprenticeship initiatives in the geographic areas served?

[Probe other DOL apprenticeship grant initiatives (e.g., State Apprenticeship Capacity grants; other DOL youth training initiatives (e.g., the WIOA Youth program, YouthBuild, Job Corps, CTE); and other apprenticeship/pre-apprenticeship)]

5. Has the COVID-19 pandemic impacted the start-up, early implementation, ongoing operation and/or outcomes of under your grant? If so in what ways?

6. Are there any other factors since the start of your grant that have affected the programs’ ability to recruit apprentices or employers? Anything that has affected the ability of apprentices to complete training and find employment?

[Probe for: in- or out-migration of major employers, major layoffs, base closings, natural disasters, unwillingness of employers/partners to meet commitments they made during the proposal process]

3. GRANT ORGANIZATIONAL STRUCTURE AND KEY PARTNERS

1. Please give me/us an overall description of the organizational structure of the apprenticeship programs and partners involved. [Note: request organizational diagram if one is available.]

2. As the lead applicant under this grant, what is the key role of your organization under the grant? In particular, what are the roles and activities your organization performs, as it relates to (a) administering the grant; (a) overseeing/coordinating work conducted by required/optional partners and/or (c) delivering services or setting up or expanding registered apprenticeship programs (e.g., to establish apprenticeship or pre-apprenticeship programs, expanding existing programs, sponsoring apprentices, recruit and enroll youths, develop curriculum, etc.).

[Note: The lead applicant may be an Education and Training Provider, Workforce Development System Entity, an Organization Functioning as a Workforce/Industry Intermediary, or a State Agency. See FOA-ETA 20-06, p. 13-14 for details on these 4 types of organizations, as well as the expected role of the lead applicant]. Additionally, program activities/allowable activities under the grant are listed on pages 7-8 of the FOA.

3. For the grantee, who are the key staff and their roles under the grant?

1. Number and position/title of staff funded under grant (including percent of effort for each grantee staff person)

2. Staff role performed under the grant

3. Since the grant’s inception, to what extent has there been turnover in staff? Overall, to what extent and how has staff turnover affect the grant and participant outcomes?

4. Under the grant, there are four types of required partners (depending on the type of organization the lead partner is): (a) two employer partners or an industry/trade association that represents at least 2 employers; (b) an education or training providers; (c) the state apprenticeship agency (for states with a federally-recognized SSA), and (d) an entity that carries out activities under WIOA (such as a local workforce board). Additionally, lead grantees are encouraged to partner with a broad range of other partners that can support and advance the work on fostering youth apprenticeship under their grants [Note 1: Partners may include workforce intermediaries, labor-management organizations, community-based organizations, training providers, and service providers; other organizations to support outreach and training activities, such as: industry-led training organizations, industry intermediaries, unions, or educational organizations; Small Business Development Centers; American Job Centers; community organizations that provide social support and/or wrap-around services; YouthBuild programs; Job Corps centers; WIOA Youth programs; foundations and philanthropic organizations; and federally-funded youth-serving programs.] [Note 2: See the FOA-ETA-20-06, for description p. 12-15 for additional details about types of required/optional partners and their potential role under the grant.] Some roles partners could play include the following:

• Outreach, recruiting, or referring individuals for placement in apprenticeships

• Conducting employer outreach/engagement

• Helping with apprenticeship program or curriculum design

• Serving as an apprenticeship sponsor

• Serving as a pre-apprenticeship sponsor

• Serving as a related training instruction provider

• Providing support services for apprentices]

1. Under your grant, how many employers are you partnering with? What are the company names? What industry sector are they in? We’d like to hear about a few of the most important of these employers:

1. Whether there was a previous relationship with the employer

2. The employer’s role and key services delivered under the grant

3. Goal (if any) with the employer for youth to be placed in apprenticeships/pre-apprenticeships with the employer

4. Numbers of youth (if any) placed to date with the employer in apprenticeships/pre-apprenticeships

5. How did the partnership with the employer change or evolve over the course of the grant?

6. Challenges and benefits of the partnership

2. Under the grant, have you partnered with industry/trade associations? If yes, what are the association(s) names, what industries or occupational groups do they represent? How many employer members does each association have? For up to 3 of the most important industry/trade associations grant partners, please discuss the following:

1. Whether there was a previous relationship

2. The association’s role and key services delivered under the grant

3. Goal (if any) with the association for youth to be placed in apprenticeships/pre-apprenticeships

4. Numbers of youth (if any) placed to date in an apprenticeship or pre-apprenticeship program sponsored by the associations

5. How did the partnership with the association change or evolve over the course of the grant? Challenges and benefits of the partnership

3. Under this grant, have you partnered with any unions? If yes, what are the union(s), what industries or occupational groups do they represent? How many employer members does the union have? For up to 3 of the most important union grant partners please discuss the following:

1. Whether there was a previous relationship

2. The union’s role in key services delivered under the grant

3. Goal (if any) with the union for youth to be place in apprenticeships/pre-apprenticeships.

4. Number of youth (if any) placed in an apprenticeship or pre-apprenticeship program sponsored by the union.

5. How did the partnership with the union change or evolve over the course of the grant?

6. Challenges and benefits of the partnership.

4. Under the grant, have you partnered with education and training providers? [Note: Education and training providers include the following types of organizations: local and state education agencies (including high schools and school districts); institutions of higher education; state college coordinating entities, such as a community college system office or a single state educational board; and community-based organizations that offer job training.] If yes, what are the names of the providers and what types of institutions are they? For up to 3 of the most important of these education and training providers partners in your grant activities, please discuss the following:

1. Whether there was a previous relationship with the education and training provider

2. The provider’s role and key services delivered under the grant

3. Goal (if any) with the provider for youth to be trained (and/or placed in apprenticeships/pre-apprenticeships)

4. Numbers of youth (if any) trained by the provider to date (and/or placed to date with the provider in apprenticeships/pre-apprenticeships)

5. How did the partnership with the provider change or evolve over the course of the grant? Challenges and benefits of the partnership

5. Under the grant, to what extent have you partnered with the State Apprenticeship Agency (SAA) or OA representative?

1. Whether there was a previous relationship with the SAA or OA representative

2. The SAA or OA role and key services delivered under the grant

3. How did the partnership with the SAA or OA change or evolve over the course of the grant?

4. Challenges and benefits of the partnership

6. Under the grant, have you partnered with an entity carrying out activities under WIOA (such as a state or local workforce development board)? If yes, with which WIOA organization(s) have you partnered under the grant? For up to 3 of the most important of these WIOA entities partners in your grant activities, please discuss the following:

1. Whether there was a previous relationship with the entity

2. The entity’s role and key services delivered under the grant (including referrals of youth to the program, whether the entity has served as an apprenticeship sponsor, whether the entity has acted as an intermediary between the education and workforce systems in the local areas served by the grant)

3. Goal (if any) with the entity for youth to be recruited/referred and/or placed in apprenticeships/pre-apprenticeships

4. Numbers of youth (if any) recruited/referred by the entity and/or placed to date in apprenticeships/pre-apprenticeships

5. How did the partnership with the entity change or evolve over the course of the grant?

6. Challenges and benefits of the partnership

5. Who are the other key partnering organizations (e.g. community-based organizations, AJCs)? For each key partner, please discuss: name and type of the organization, role and key services being provided under the grant, whether partner is meeting expectations and any challenges in engaging the partner, and benefits of the partnership. [Note: See the list of additional optional partners and probes in C4, above.]

6. How do these partners collaborate during the grant? What are some examples of collaborative activities between partners?

7. Did the grantee or partners experience start-up or early implementation issues (e.g., difficulties hiring staff, problems engaging partners, lack of employer interest/engagement, lack of potential apprentice interest, external (environmental) factors)? Please discuss each challenge, including how it affected grant implementation, if and how the challenges were overcome.

8. Did you offer technical assistance to your partners and youth apprenticeship/pre-apprenticeship sponsors? What did this entail? How did they respond to the technical assistance provided? Where they aspects they used more than others? Were there some aspects of the technical assistance provided that they didn’t seem to find helpful?

4. POLICY CHANGES, PROGRAM INFRASTRUCTURE AND DATA COLLECTION/USE

1. Have you influenced, developed, and/or implemented any policy or procedural changes to support apprenticeship expansion as a part of or as a result the grant? What are they intended to do? Have they worked as intended? How so or not? [Probe for:

• State or local legislation

• State or local policy guidance

• Increased state or local government funding

• Enrollment policy changes at a community college

• Credit or articulation policy changes at a community college (e.g., credit for prior learning)

• Policies/procedures on apprentice safety

• Policies/procedures on apprentice supervision

• Policies/procedures on equal employment opportunity for apprentices

• Policies on other aspects of apprenticeship (e.g., wage progression or employer incentives)]

2. What systems and processes were implemented or enhanced to help simplify program development and reduce program start-up times and costs for registered youth apprenticeship/pre-apprenticeship programs? Have these systems or processes made it easier for partners to implement the program(s) at scale? Have the changes improved the program’s quality? If yes, how so?

3. Have you developed new or enhanced apprenticeship standards for the registered youth apprenticeship/pre-apprenticeship programs implemented under your grant? If yes, for which programs/industries? Please describe how they were developed and deployed.

4. Have you adopted new or enhanced industry-approved/recognized credentials for the youth apprenticeship/pre-apprenticeship programs implemented under your grant? If yes, which ones? Please describe how they were developed and implemented.

5. Have you developed new or enhanced existing curriculum for the youth apprenticeship/pre-apprenticeship programs implemented under your grant? If yes, which ones? Please describe how they were developed and deployed.

6. Have you implemented strategies to facilitate industry-wide acceptance of the youth apprenticeship/pre-apprenticeship model(s) being deployed? How has your organization and partners been promoting youth apprenticeship/pre-apprenticeship? If so, please describe these activities. What have the successes and challenges been?

7. What quality assurance systems did you design and implement as a part of the grant? Did you build off of existing structures and systems? If yes, how so? Have they improved the programs?

8. What data management systems do use you for this grant and the programs? What metrics and data do you, the partners, and other stakeholders use to measure success of the youth apprenticeship programs? [Probe if they used others than the required performance measures for the grant.] Do you and your partners use the metrics and data to track improvement?

9. Were you able to collect all individual-level data for apprentices? Were there any difficulties in collecting data on grant apprentices? What types of data do you collect on apprentice outcomes? How do you use that data to assess progress towards your goals? [Note: ask for a copy of database documentation and fields if available].

5. TARGET POPULATIONS AND APPRENTICE RECRUITMENT

1. What populations and/or subpopulations have been targeted for recruitment (for youth apprenticeships/pre-apprenticeships) (i.e., by both the grantee and key partners)? Why have these populations/subpopulations been targeted? [Note: Collect and pre-fill this information prior to the visit from the grant application or other available documentation and verify with the respondent.] [Probe for specific age ranges and:

• In-school youth (16-18)

  • CTE students

  • Non-CTE students

  • Students at risk of dropping out

• Out-of-school youth

• Employed youth

• Unemployed youth

• Youth ex-offenders

• Foster youth

• Parenting youth

• Homeless youth

• College students

• First generation

• Youth populations that may traditionally be underrepresented in apprenticeship, including:

  • Women

  • People of color

  • Persons with disabilities

2. As the program has progressed, have there been any changes in the populations targeted? If so, what are they? Why?

3. Have partners provided input on the target population or admission criteria? Please describe nature of input and specific criteria or requirements.

4. What strategies for marketing youth apprenticeship have seemed to work best in attracting prospective apprentices?

5. What organizations refer prospective apprentices? Which organizations have provided the most referrals? Have there been any issues or concerns about referral numbers or procedures? [Probe from:

• Employers

• Professional and industry organizations

• Unions

• Workforce system – American Job Centers, Employment Service, and WIOA

• Local Education Agencies (LEAs), such as the local public school system

• Re-engagement centers (i.e., to link youth that have dropped out of school back to school or other types of training)

• Community colleges and other educational institutions including proprietary schools

• Courts/correctional system/juvenile justice/probation officers

• Faith-based and community-based organizations

• Other public and non-profit human service agencies]

6. How have K-12 schools been involved in marketing of apprenticeships and the recruitment of apprentices?

7. What other programs or services do your referral sources offer? If your program wasn’t here and there were no youth apprenticeship or pre-apprenticeship programs, what else might these people have access to?

8. Where do youth apply for your apprenticeship programs? [Probe for:

• Employers

• Community college

• Your organization

• Union/labor organization

• Another intermediary

• Pre-apprenticeship program

• American Job Center]

9. What strategies have you and your partners used to recruit individuals from groups who are underserved by apprenticeship programs, such as women, Black/African American and Latinx/Hispanic youth, out-of-school youth, veterans, people with disabilities, etc.? Which groups have you been successful with and with which groups have you been unsuccessful? Which strategies seem successful or not?

10. Have there been recruitment challenges? If so, what challenges have been encountered and how have they been addressed? [Probe for:

• COVID pandemic

• Have difficulty finding eligible apprentices

• Struggling with finding/engaging out-of-school youth

• Many who applied have insufficient basic skill levels

• Some applicants are not interested or motivated to participate in apprenticeship

• Apprenticeship period is too lengthy for some applicants

• Some of the outreach strategies haven’t resulted in many applicants

• Partners that were supposed to provided referrals have not provided them in sufficient numbers

• Some applicants have difficulty getting to the program/employer location

• Didn’t have enough resources for recruitment

• Changing economic conditions in the areas has made recruitment more challenging]

11. Overall, what recruitment or referral strategies being used under the grant have been most successful? Do you feel any of these strategies are new, unique, and/or innovative?

6. APPRENTICE ELIGIBILITY/INTAKE AND PARTICIPATION LEVELS UNDER THE GRANT

1. What are the eligibility requirements for enrollment? [Note: Request copies of apprentice eligibility or intake form(s).] [Probe for:

• Complete pre-apprenticeship program (i.e., for apprenticeship program component)

• Be from a specific subpopulation

• Reside within a specific geographic area

• Meet income eligibility requirements

• Meet an age requirement

• Meet education level requirements (e.g. high school diploma, GED/HSED) or basic skill levels]

2. Describe the intake process for the apprenticeship/pre-apprenticeship programs operating as a part of your grant? How do apprentices apply to the program and what steps do they take to advance to being an apprentice?

   1. At what point is someone considered an apprentice/pre-apprentice?

   2. At what point does an apprentice you are serving become an apprentice/pre-apprentice?

   3. Does this process vary across apprenticeship/pre-apprenticeship programs or is it consistent?

3. What role, if any, do employers play in the screening and selection process for apprenticeships/pre-apprenticeships, particularly in the case of incumbent workers served under the grant? Please describe this process including (if applicable):

   1. When in the recruitment process does the employer decide to hire the apprentice?

   2. How do employers generally determine whether an applicant is the right fit for an apprenticeship/pre-apprenticeship program under the grant?

   3. About what percentage of applicants are rejected (e.g., screened out during the employer interview process)? What are the major reasons employers reject applicants?

   4. What are the main reasons that applicants are screened out or fail to move through the intake process to be considered by employers?

4. Beside employers, are there other organizations that assist in the intake, screening, and enrollment process for apprentices? [Probe for:

• K-12 school system

• Education and/or training providers (for apprenticeship/pre-apprenticeship programs under the grant)

• Community or technical college

• Labor organization (e.g. union, labor association/labor federation)

• American Job Center

• Nonprofit (e.g., community or faith-based) service/training provider

• Industry association

• Local government agency

• State government agency]

7. KEY FEATURES OF AN APPRENTICESHIP PROGRAM FORMED/EXPANDED UNDER THE GRANT– AN EXAMPLE OF A REGISTERED APPRENTICESHIP PROGRAM FUNDED UNDER THE GRANT

[Note: If the grantee has more than one apprenticeship program, teams should focus on one to two programs and its associated employer(s) or sponsor. Use your discretion as to which programs (e.g., largest, newest, non-traditional industry programs). Ideally, the administrator should describe the programs associated with employers that the site visit team will visit. It may be appropriate to skip some of these questions with the grant manager and cover them at the employer (and/or sponsor) or training site where the apprenticeship training is being provided.]

1. What geographic area is the apprenticeship program serving?

2. What occupations or industry sectors is the focus of the apprenticeship program?

3. Who is the sponsor of the apprenticeship program (e.g., a group sponsor, employer, union, etc.)?

   1. If the apprenticeship program is a group program, how many employers have signed on to the group apprenticeship standards?

4. Is the recruitment and intake process for this program substantially different from what you described above? If so:

1. What is the typical recruitment source(s) for the apprenticeship program? What methods are used for recruitment? Are apprentices enrolled in cohorts, and if yes, how often are new cohorts enrolled (e.g., 4 times a year)?

2. Are there any other types of targeting of special populations? Please identify specific subgroup populations targeted. What strategies are used to recruit them?

3. What is the eligibility criteria used in selecting apprentices for the apprenticeship program? Please also briefly discuss the assessment and intake process.

5. Is the apprenticeship program time-based (e.g., completion of specific numbers of hours of training), competency-based or a combination of the two?

1. If time-based, what is the duration of the apprenticeship?

2. If competency-based, how is it determined that an apprentice has achieved the necessary level of competence to complete the apprenticeship? How long does it take an apprentice typically to reach the necessary level of competence and how much variation is there across apprentices?

6. How many hours of Related-Instruction (RI) (i.e., classroom/web-based) are provided in this program? Over what period of time does RI occur? [Note: The FOA refers to RI, which in the context of apprenticeship programs is sometimes referred to as related technical instruction (RTI).]

1. What is the typical schedule for an apprentice each week? Does this schedule change over the course of the apprenticeship?

2. Where and how is instruction provided (e.g., web-based, hybrid, laboratory/shop, in a classroom, or at a union or employer site)?

3. Who provides instruction?

4. Is RI and OJL/OJT provided concurrently, is it front-loaded, or does RI and OJL/OJT alternate during the apprenticeship?

5. Please provide a brief overview of the instruction provided (e.g., topics covered in classroom instruction). [Note: If applicable, request a syllabus for the coursework or a topic outline for the training.]

7. Are there innovative RI methods being used as part of the apprenticeship program? Please briefly discuss. [Probe for:

• Accelerated learning modules

• Contextualized learning

• Team teaching

• Self-paced learning

• Competency-based learning

• Prior learning assessments

• Technology-enabled learning, such as use of simulators, on-line teaching/learning

• Integrating basic skills instruction with technical instruction]

8. How many hours of OJL/OJT occurs at a job/employer site does an apprentice complete (i.e., each week, month, or year). Does this vary over time and, if so, how? How many total hours of OJL/OJT would an apprentice typically complete over the period of the apprenticeship?

   1. Where do apprentices work (i.e., the employer) and in what types of tasks do apprentices start and progress to during their involvement in the apprenticeship?

   2. Who oversees and provides workplace/on-the-job instruction to the apprentice (i.e., while the apprentice is working)? Please briefly describe the nature of the workplace instruction provided to apprentices.

   3. What is the supervisor-to-apprentice ratio during the apprenticeship period (e.g., 1 supervisor to 3 apprentices)? Does it change over time?

   4. How are mentors for OJT provided to apprentices?

9. What aspects, if any, of the apprenticeship RI or services do you think are most innovative? Effective?

10. How are grant funds expended to support RI efforts? Do expenditures include funding: (1) the development of courses at the secondary level that are integrated into the RAP and local and/or state educational standards; (2) apprentice tuition or other educational fees; and (3) the delivery of instruction requirements (e.g., virtual learning technology, classroom instructors)?

11. Is the apprenticeship training connected to higher education? If so, how? Do apprentices receive post-secondary (college) credits, credentials, or certificates? What are the credits, credentials, and/or certificates? What is the schedule for receipt of these credits, credentials, or certifications?

12. What is the schedule of wages and benefits for apprentices?

    1. What is the starting wage?

    2. What is the ending wage?

    3. On average, how large is the hourly wage increase in dollars: $_____

    4. What milestones trigger increases in hourly wages? (e.g., annual increases, achievement of certain set of competencies, completion of RI)

13. Do apprentices receive any of the following benefits other than their wages during their apprenticeship program?

1. Health benefits

2. Paid sick leave

3. Paid vacation time

4. Retirement benefits

5. Other (Please specify)

14. How much of the apprentice’s hourly wages are employers reimbursed for through grant funding?

15. To what extent are non-training supports provided during the apprenticeship period and what is the source of funding for these supports (e.g., case management, career coaches/navigators counseling, and mentoring)?

16. Are there additional supports or other strategies used to help apprentices from underserved populations successfully complete the program? If so, please describe.

17. To what extent are supportive services provided for apprentices, such as transportation assistance, work clothes/equipment, childcare, and needs-based payments? If provided, how are these supportive services structured and paid for? Who provides these services?

18. Do apprentices pay any costs of the RI (e.g., tuition, fees, equipment/books, etc.)?

19. What is the goal (if any) for enrollment in the apprenticeship program under the grant? To date, how many apprentices have been enrolled? To date, how many apprentices have completed their apprenticeships? Has the program experienced attrition – and if yes, why? If available, about what percentage of those enrolled in the program terminate before completion of the apprenticeship?

8. EMPLOYER RECRUITMENT, ENGAGEMENT, AND ROLES

Note: Prior to the site visit, collect the following information from the grant application on the following to the extent it is available:

1. Who are the collaborating employers listed on the grant application?

[Provide a full list of employers.]

2. What methods did the proposal specify for reaching out to and involving the employers?

3. What roles, if any, did the grantee propose to play in helping company to plan and/or implement apprenticeship programs?

1. Are grant funds used to market to, recruit, and/or engage employers to consider establishing or expanding with youth apprenticeship/pre-apprenticeship program(s)?

2. Does your grant fund staff at your organization to market to, recruit, and/or engage employers to consider establishing or expanding with youth apprenticeship/pre-apprenticeship program(s)? Do any of these staff work on employer engagement full-time?

3. Does your grant fund other partners to market to, recruit, and/or engage employers to consider establishing or expanding with apprenticeship/pre-apprenticeship program(s)?

4. At your organization and other partnering organizations, about how many full-time staff equivalents (FTEs) for the past year of your grant were funded under the grant to conduct employer outreach, recruitment, and engagement effort under the grant? __________FTEs

5. What is the goal under your grant for the total number of employers that adopt apprenticeship training as a result of your grant project? ____ employers adopting apprenticeship.

6. Under your grant, what number of employers have adopted apprenticeship training to date?

_____ employers (as of ___/___/___).

1. If known, about what percentage of those engaged have had fewer than 25 employees?

2. If known, what is the largest and smallest number of apprentices enrolled in apprenticeships/pre-apprenticeships at the employers that have been engaged?

7. Is your project on track to meet your goal for the total number of employers adopting apprenticeship programs under the grant? If not, why not?

8. If new employers have been recruited under the grant, what methods have been used to reach out to and recruit employers? Which methods have proved more and less successful? [Probe for:

• Direct contact through telephone or emails

• Distribution of flyers

• Program staff outreach presentations or orientations with employers or business organizations

• Referrals from other organizations and employers (e.g., the Chamber of Commerce, other businesses organizations, employers sponsoring apprenticeships, workforce development programs, etc.)

• Outreach campaigns using media (e.g., PSAs, TV, radio, newspaper, etc.)

• Websites/social media (e.g., LinkedIn, Facebook, etc.) – please specify, which types of social media are used and how

• Word-of-mouth]

9. Has any grant funding been used for financial supports to encourage employer engagement? Or funds from other sources? If so, how are the incentives structured and are they to defray costs for certain aspects of apprenticeship programs? How successful have these incentives been in encouraging employer engagement? What have the challenges been?

10. What message(s) have been used to engage employers? Do messages vary by size or other characteristics of the employer? Which messages have been most/least effective?

11. What are the key barriers to persuading employers to start or expand apprenticeship/pre-apprenticeship programs? [Probe for:

• Lack of knowledge about apprenticeship

• Administrative hassle and paperwork involved in registering the program

• Fear of involvement with the government

• Costs of the program

• Difficulty in finding related technical instruction provider that is a good fit

• Lack of trainers

• Concern about unionization

• Not enough of a pool of employers in the area served that are interested in apprenticeships

• Lack of employers hiring

• Lack of employer facing shortages of skilled workers

• COVID-19 pandemic]

12. What are the key factors that motivate employers to start or expand apprenticeship/pre-apprenticeship programs? [Probe for:

• Existing or future shortage of qualified workers

• Expected benefits during and soon after the apprenticeship

• Subsidies to firms or for related instruction

• Ability to benchmark skills to high levels

• Screening process effective at identifying workers likely to succeed

• Help in reaching a diverse, qualified workforce

• National credential linked to registered apprenticeship

• Assistance in developing skill standards and registration]

9. SUPPORTS FOR COMPLETION AND EMPLOYMENT SUCCESS

1. How do you have particular ways to support persistence and completion in apprenticeship programs? What features of the apprenticeship programs make it easier for apprentices to complete the program? [Probe for:

• Online related education and instruction (rather than going to campus or provider)

• Self-paced instruction

• Competency-based learning

• Contextualized learning

• Integrated instruction/team teaching

• Prior learning assessments/credit for prior learning

• Technology-enabled learning other than online learning (e.g., simulation labs)

• Paid time for related instruction

• Competency-based apprenticeship]

2. Which personal supports are available to apprentices? Through the grant or partners? [Probe for:

• Transportation

• Childcare

• Dependent care

• Housing assistance

• Needs-related payments that are necessary to enable an individual to participate in education and training activities funded through this grant]

3. Are these support services offered directly by the grantee or partners, or are apprentices provided with vouchers or stipends for the services? What percentage of apprentice costs are supportive services? (Note: May not exceed 20% of the funding level.)

4. Which of the following academic supports are routinely offered for apprentices participating under the grant, either through your organization or your partners? [Probe for:

• Academic advising and counseling

• Tutoring

• Tuition assistance

• Basic skills training

• Career counseling or coaching

5. Has there been a need for supportive services you have not been able to provide to or help apprentices access to support their success? If so, which ones and why?

6. Is there an individual, such as a case manager, who is assigned to work one-on-one with each apprentice throughout the program? If yes, what are the responsibilities of the individual assigned to work with each apprentice? [Probe for:

• Apprentice monitoring and case management

• Career counseling and coaching

• Financial counseling

• Monitoring satisfaction of apprentice with employer

• Personal counseling

• Referrals to services in the community]

7. What strategies do you have to support success for underserved populations that may differ from the generally available supports? How well do they appear to help these apprentices? Are there some services they need that you can’t offer or help them access?

8. How have supportive services helped apprentices persist in and complete their program? How have supportive services helped apprentices improve their employment opportunities and circumstances (e.g., find and retain job, get promoted, wages)?

10. SUSTAINABILITY AND LESSONS LEARNED

1. Have you developed sustainability plans for the apprenticeships/pre-apprenticeship programs implemented under the grant? If so, please describe these plans. What sources of funding will be used to sustain the apprenticeship activities implemented under the grant? Do you feel that your sustainability plan and leveraged resources are sufficient to sustain the activities of the grant after the completion of the grant?

2. To what extent do you think your program could and should be replicated in other localities? What features of your program are most amenable to replication? What features of project are least amenable to replication? How does location, the target population served, or other distinctive features of your program make it either non-transferable or limit transferability?

3. What partnerships have been most successful? To what extent will partnerships be sustained? With which partners and how?

4. To date, what do you consider to be your most successful strategies implemented under the grant?

5. Which apprenticeship/pre-apprenticeship programs or models have appeared most successful? Why so?

6. To date, what do you believe to be the main lessons learned from your grant for apprenticeship/pre-apprenticeship program design and operations?

7. What innovations and strategies under the grant can contribute to efforts to encourage more employers to adopt apprenticeships/pre-apprenticeships?

11. Pre-Apprenticeship Programs

[If the grantee has more than one pre-apprenticeship program, teams should focus on one program. Use your discretion as to which programs (e.g., largest, newest, non-traditional industry programs). Walk through the set of questions below for the program you are discussing in detail on the visit. It may be appropriate to skip some of these questions with the grant manager and cover them at the employer (and/or sponsor) or training site where the pre-apprenticeship training is being provided. For the pre-apprenticeship programs identified, cover generally the features across the pre-apprenticeship programs; alternatively, if the programs vary, the interview can focus on one specific program.]

1. Who is overseeing the pre-apprenticeship program(s) (e.g., the grantee, a partner, an employer, etc.)?

2. What are the recruitment sources for the pre-apprenticeship program? What methods are used for recruitment? Are pre-apprentices enrolled in cohorts, and if yes, how large are the cohorts and how frequently are new cohorts of pre-apprentices enrolled (e.g., four times a year)?

3. Are the pre-apprenticeship program(s) targeting any specific populations? Why have these populations/subpopulations been selected? [Probe for:

• In-school youth16-18)

  • CTE students

  • Non-CTE students

  • Students at risk of dropping out

• Out-of-school youth

• Employed youth

• Unemployed youth

• Youth ex-offenders

• Foster youth

• Parenting youth

• Homeless youth

• College students

• Youth populations that may traditionally be underrepresented in apprenticeship, including:

  • Women

  • People of color

  • Persons with disabilities

4. What is the eligibility criteria used in selecting apprentices for the pre-apprenticeship program? Please also briefly discuss the assessment and intake process.

5. What is the usual duration of participation in the pre-apprenticeship program?

1. How many hours of instruction (i.e., classroom/web-based) are provided during the pre-apprenticeship program? Over what period of time does instruction occur? What is the typical schedule for a pre-apprentice each week? Does this schedule change over the course of the pre-apprenticeship?

2. Where is instruction provided (e.g., at a community college)?

3. How is instruction provided (e.g., classroom, web-based, hybrid, laboratory/shop)?

4. Are there innovative methods being used as part of the pre-apprenticeship initiative? Please briefly discuss. [Probe for:

• Accelerated learning modules

• Contextualized learning

• Self-paced learning

• Competency-based learning

• Prior learning assessments/credit for prior learning

• Technology-enabled learning, such as use of simulators, on-line teaching/learning

• Integrating basic skills instruction with technical instruction]

6. Is there a work-based learning opportunity provided as part of the pre-apprenticeship program? If yes:

1. Where is hands-on training being provided (e.g., in workplace, simulated laboratory experience, work-based learning environment)?

2. How many hours of work-based learning do pre-apprentices complete?

3. Where do pre-apprentices work and in what types of jobs?

7. How are pre-apprenticeship programs aligned with CTE programs, and post-secondary education opportunities?

8. What credentials do pre-apprenticeship apprentices earn from this pre-apprenticeship program?

9. How often do pre-apprenticeship apprentices move on to apprenticeship programs?

10. Do pre-apprenticeship programs serve as on-ramps to RAPs? If so, how? If not, why not? (Note: pre-apprenticeship programs are supposed to facilitate apprentices moving on to RAPs leading to industry-recognized credentials.)

11. Do pre-apprentices receive credit in their apprenticeship programs for time in the pre-apprenticeship program? If so, is this time counted towards RI hour requirements?

12. To what extent are non-training supports provided during the pre-apprenticeship period (e.g., case management, career coaches/navigators counseling, and mentoring)?

13. To what extent are supportive services provided during the pre-apprenticeship period, such as transportation assistance, work clothes/equipment, childcare, rehabilitative services, and needs-based payments? If provided, how are these supportive services structured and paid for? Who provides these services?

14. What types of data are collected to track performance outcomes of enrolled pre-apprentices?

15. What, if any, post-completion services are provided to apprentices of the pre-apprenticeship program(s)? [Probe for:

• Referral to apprenticeship programs

• Referral to other education or training programs

• Help with paying for certification exams or obtaining credentialing

• Job search or placement assistance

• Job retention services

• Provision or referral to supportive services]

CHECKLIST OF ITEMS TO COLLECT FROM SITE (IF AVAILABLE). REQUESTED WHEN ORGANIZING THE SITE VISIT

• Background information about the grantee organization

• Additional documentation detailing key apprenticeship/pre-apprenticeship program components (e.g., RI/RTI, on-the-job training or mentoring)

• Evaluation reports that may have been prepared on the program

• Blank set of intake forms

• Diagram showing how apprentices flow through the program

• Organizational chart for the program

• Apprentice recruitment materials

• Employer recruitment material