Evaluation of the American Apprenticeship Initiative
ICR Reference Number 201802-1290-002
May 2018
The Department of Labor (DOL) Employment and Training Administration (ETA) contracted with Abt Associates and its partners to conduct an evaluation of the American Apprenticeship Initiative (hereafter, the AAI evaluation) and is seeking approval from OMB under the Paperwork Reduction Act for data collection instruments associated with the evaluation. With the goal of expanding registered apprenticeship in the United States, both at scale and into new sectors, the American Apprenticeship Initiative (AAI) provided $175 million in five-year grants to 46 grantees across the country. Grantees can use funds to support registered apprenticeships or pre-apprenticeships.
The evaluation, funded by the Employment and Training Administration (ETA) of the U.S. Department of Labor (DOL) will build evidence about the effectiveness of apprenticeship for participants and employers and provide lessons for developing and operating apprenticeships.
Part B of the Supporting Statement for the Evaluation of the AAI evaluation considers the issues pertaining to Collection of Information Employing Statistical Methods. The AAI evaluation includes the following four sub-studies:
The implementation study will describe how AAI programs develop, operate and mature. It will examine the context in which the apprenticeship and pre-apprenticeship programs operate, the target group and recruitment strategies, employer perceptions of apprenticeship, partnerships, and training strategies.
The outcomes study will examine in-program and post-program outcomes of apprentices, particularly around employment, earnings, wages, and employment retention, as well as pre-intervention and post-intervention certification and credential attainment. Particular attention will be given to outcomes for underrepresented populations in apprenticeship.
The return on investment study will estimate the benefits and costs of apprenticeship to employers.
An impact study of employer outreach activities in a subset of grantees will examine which recruitment methods and marketing strategies most successfully encourage employers to offer apprenticeships.
The three data collection instruments described in this submission pertain to the implementation study and the impact study of employer outreach activities of the AAI evaluation. This submission includes the following instruments: implementation study interview guides, grantee survey, and the management information system (MIS) for up to four impact study sites.
DOL awarded 46 AAI grants to state agencies, community-based organizations, post-secondary educational institutions, unions, and industry associations in September 2015. The AAI evaluation involves four sub-studies as described above. The evaluation as a whole includes all 46 grantees, although not all grantees will participate in each sub-study. This section outlines the respondent universe and sampling methods for each of three instruments that are involved in the implementation and the impact sub-studies.
To support the implementation study, an online survey will be administered to all 46 AAI grantees for the purpose of systematically documenting program operations and the type of services provided across the study sites. The respondent universe for the online grantee survey comprises all 46 AAI grantees that were awarded funds through AAI. No statistical sampling will be used. As noted below, the expected response rate is 90 percent. The list of primary (and secondary, if available) respondent names and emails from each grantee will be developed in collaboration with DOL staff.
Implementation Study Interview Guides
The respondent universe for the interview guides is a subset of approximately 10 of the 46 grantees. The implementation study includes two rounds of site visits to the 10 grantees to gather information on grantee planning for implementation, implementation activities and timelines, changes made to plans and reasons for those changes, partnerships and partner contributions, and the role of employers in designing and implementing the programs.
Selection of grantees. The approximately 10 grantees will be selected purposively because DOL wants to ensure that sites with certain features are included in the implementation study for the purpose of learning about features and conditions of interest to the Department. In addition, resources do not allow all of the sites to be visited as part of the implementation study. The evaluation team will select six of the 10 grantees using the following criteria:
Size of grantee program. The grantees selected will include a mix of “large” (1,000 or more target apprentices) and “small” (less than 1,000).
Type of grantee. The grantees selected will include a mix of grants operated by state agencies, colleges, non-profit organizations and unions.
Prior experience. Selected grantees will vary by whether they have prior experience with apprenticeships or are new to operating a program.
Promising employer recruitment approaches. To complement the impact study of employer outreach activities, the visited sites will include those that have established procedures for recruiting employers.
The other four grantees included in the implementation study will be those selected to participate in the impact study and will be selected using different criteria. The selection process for these grantees involves the evaluation team reviewing information gathered during a knowledge development phase of the evaluation, as well as grantee quarterly reports. The selection criteria that will be used to select the impact study sites, includes:
Interest in employer recruitment and progress in recruiting employers to sponsor apprenticeships. Grantees that have expressed interest in recruiting a large number of employers to sponsor apprenticeships and have not made much progress in recruiting employers.
Number of apprenticeships. Grantees that are committed to enrolling a fairly large number of apprentices (a minimum of 250), as these grantees will likely need to recruit more employers.
Employer recruitment strategies. Grantees that plan to utilize recruiters or employer points of contact as part of their employer recruitment approach or grantees with strong linkages to trade associations or other groups through which many employers could be randomly assigned to treatment and control groups (or the grantee could be expected to generate such a large group of prospective employers on its own).
Geography. Grantees with a limited footprint (i.e., focus on larger counties or cities rather than grantees dispersed widely over a state or multiple states).
The evaluation team will then conduct one-day site visits to up to six of the grantees identified as candidates for the impact study of employer outreach activities to further narrow the pool of candidates and obtain additional information required for planning the treatment and control group interventions for the impact study. DOL expects response rates for the site visits to be 100 percent.
Selection of interviewees. For each of the 10 grantees in the implementation study, the evaluation team will hold semi-structured interviews with staff in roles funded by the AAI grant, as well as grant partners, including employers. The evaluation team will interview up to 12 grantee staff and staff employed by partnering organizations and employers. Staff interviewed may be affiliated with the grantee itself, a workforce board, a union, a provider of related technical instruction, and/or an employer of AAI apprentices depending on grant staffing and partners. The exact number of interview respondents in each of these categories will be unique to the grantee depending on the entities collaborating with them on the grant and the structure of their grant activities. The evaluation team has estimated interviewing a total of 10 grant directors, 20 grantee staff, 30 related technical instruction provider that include 10 community college (or other training provider) leaders and 20 community college (or other training provider) instructors, 20 workforce development staff or staff from partnering non-profit organizations or social service providers, 10 union representatives, and 20 employers. For grantees participating in the impact study of employer outreach activities, the evaluation team will also interview staff associated with the impact study activities.
The respondent universe for interviewees is not known because the evaluation team has not gathered detailed information on the staffing for the AAI grants, including staffing within each partnering entity and among employers. Once the evaluation team and DOL have selected the grantees to participate in the impact study, the evaluation team will contact each of them to discuss the structure of their grant (i.e., within their organization, and with partners and employers) and staffing at each organization involved. This information will provide the respondent universe from which the interviewees will be identified. The evaluation team will work closely with grantee staff to identify appropriate respondents in each category.
MIS for the Impact Study of Employer Outreach Activities
The respondent universe for the impact study MIS is the four grantees selected for the impact study. Grantees will be identified for participation in the impact study based on the criteria outlined above. The MIS serves two purposes. First, it will be used by the evaluation team to randomly assign employers drawn from a list generated from a database to either a control group that receives nominal or no contact from the AAI grantees involved in the impact study or to a treatment group that receives intensive outreach and marketing from the grantee. The evaluation will provide the four grantees with these lists of employers and they will be instructed to focus their outreach and marketing efforts on the employers assigned to the treatment group. Second, the MIS will be used by the four grantees to track their outreach efforts and contacts made with employers in the treatment group to encourage them to sponsor apprentices. Grantees will enter into the MIS information regarding the type of marketing and frequency of contacts, as well as services the grantee provides to the employer to help establish AAI apprenticeships and the outcome of these efforts. A response rate is not applicable for the MIS.
Exhibit B1 presents the sampling methods and target response rates for each of the data collection instruments.
Exhibit B1. AAI Evaluation Respondents
Respondent Universe |
Respondent Subgroup |
Sampling Methods and Target Response Rate |
Data Collection Instrument |
Grantees, partners, and employers |
Grantees |
The 10 grantees in the implementation study will be selected purposively. Four of these grantees will also be participating in the impact study of employer outreach activities.
Evaluation team members review the topics of interest with grantees to identify appropriate respondent(s) based on who is most knowledgeable about the topics of interest.
Grantees have agreed to participate in the evaluation as a condition of receiving AAI grant funding. Therefore, the team expects a 100 percent response rate. |
Semi-structured telephone interviews |
|
Grant director and grantee staff |
A very high response rate (100 percent) is expected among grant directors and staff. |
Semi-structured interview guide for two rounds of in-person site visits |
|
Workforce board staff |
A very high response rate (100 percent) is expected among workforce board staff, if involved in grant operations |
Semi-structured interview guide for two rounds of in-person site visits |
|
Union representatives |
A very high response rate (100 percent) is expected among union representatives, if involved in grant operations |
Semi-structured interview guide for two rounds of in-person site visits |
|
Related technical instruction (RTI) staff |
A very high response rate (100 percent) is expected among RTI staff, if involved in grant operations |
Semi-structured interview guide for two rounds of in-person site visits |
|
Employer staff |
A very high response rate (100 percent) is expected among employer staff, if involved in grant operations |
Semi-structured interview guide for two rounds of in-person site visits |
Grantees |
Grant director or other staff identified by grantee as appropriate respondent |
All 46 AAI grantees will be asked to respond to the grantee survey. Grantees have agreed to participate in the evaluation as a condition of receiving AAI grant funding. Therefore, the team expects a 90 percent response rate. |
Grantee survey |
Grantees |
Grantee staff |
Four grantees in the impact study of employer outreach activities will be selected based on interest, number of apprenticeships to be offered, employer recruitment strategies, and geography.
A response rate is not applicable to the impact study MIS |
MIS for the impact study |
Though this will be the first set of data collection activities for the AAI evaluation, the evaluation team has had extensive experience administering similar data collection instruments and has been able to achieve high response rates for them. Also, the Department of Labor Funding Opportunity Announcement (FOA-ETA-15-02) for AAI established that the department is committed to producing strong evidence on effectiveness of grantee programs and that all grantees must fully participate in any evaluation initiated by DOL as a condition of grant award.
This section describes the data collection procedures for the AAI evaluation’s implementation study and impact study of employer outreach activities. The implementation study includes two rounds of site visits to 10 grantees (four of which are also involved with the impact study) to collect in-depth qualitative information on grantee operations. The evaluators will hold semi-structured interviews with a variety of stakeholders involved in the AAI grants that will allow them to document grant planning and implementation, changes to operations over time, participant outcomes, and post-grant sustainability. The implementation study also relies on the grantee survey, designed to collect consistent information from the 46 AAI grantees. The survey will be administered using a web-based system. No statistical methods will be used for stratification and sample selection for the implementation study.
The impact study of employer outreach activities will rigorously estimate the impact of intensive marketing and outreach on the number of apprenticeships that employers offer. Prior to launching the impact study, the four grantees selected to participate will be trained on data collection procedures, which primarily involves entering data into the impact study MIS. Each grantee in the impact study will use the MIS to track contacts with employers, including the type of marketing and frequency of contacts. The MIS will also track services the grantee provides to the employer to help establish AAI apprenticeships as well as the outcomes of these efforts.
Procedures with special subpopulations: With the AAI grants, DOL is interested in increasing apprenticeship opportunities for all American workers, particularly underrepresented populations in apprenticeship (including women and minorities); low-skilled populations; and veterans, including transitioning service members. The proposed data collection efforts do not collect data from AAI participants, but rather from grantees and their partners which do not constitute a special population.
Use of Periodic Data Collection Cycles to Reduce Burden: The implementation study site visits will occur twice. The first round is planned for summer 2018, and the second round will occur in summer 2019. The grantee survey is a one-time data collection effort. The impact study MIS will be used on an ongoing basis by four grantees to enter real-time data on their outreach efforts.
Grantee Survey
The evaluation team will administer the survey to the universe of 46 AAI grantees. Thus, no statistical methods will be used to select grantees. The grantee survey will be used to develop an inventory of grantee goals, activities, project context, and future project plans. Estimation procedures will be, for the most part, very simple. Much of the data collection will be on a census basis, removing the need for survey weights. There will be one grantee survey per grantee. No statistical techniques will be used to ensure accuracy. The study team does not foresee any unusual problems that would require specialized sampling procedures.
Implementation Study Interview Guides
The data will be collected through semi-structured interviews held on-site with 10 grantees selected for the implementation study. As discussed in section B.1, the 10 grantees for the implementation study will be selected purposively. Four of the 10 will be selected using the criteria outlined for the impact study of employer outreach activities in B.1. No statistical methods will be used to select grantees. The sample is intended to be neither random nor representative.
The site visits are designed to provide in-depth qualitative information about grantees; no estimation procedures will be used. The data analysis will be descriptive. No statistical techniques will be used to ensure accuracy. The study team does not foresee any unusual problems that would require specialized sampling procedures.
MIS for the Impact Study of Employer Outreach Activities
No statistical methods will be used in selection of the four grantees. The four grantees will be chosen by DOL for the study based on a number of substantive factors, including interest in the participating, the number of apprenticeships, employer recruitment strategies, and geography. Once the four impact study sites have been chosen purposefully, the MIS will be used to randomly assign a list of employers drawn from a database sampling frame to either a control group that receives nominal contact from the AAI grantees involved in the impact study or to a treatment group that receives intensive outreach and marketing from the grantee. The random assignment ratio is planned to be 1:1. Employers will be selected for the list based on location in the grantee service area and industrial sector that aligns with the grantee apprenticeships.
The MIS will be used to gather information from participating grantees on marketing efforts to each employer randomly assigned to the treatment group, such as the number and types of marketing attempts made to the employer, results of recruitment contacts, reasons for non-participation or participation, and number of apprentices participating employers serve or expect to serve. No statistical techniques will be used to ensure accuracy. The study team does not foresee any unusual problems that would require specialized sampling procedures.
This section describes the methods to maximize response rates for the AAI Evaluation. The AAI Evaluation data collection efforts are heavily dependent on gaining grantees’ and their partners’ cooperation, buy-in, and collaboration. The evaluation team believes that the respondent groups—grantees, partners, and the employers they engage—are interested in supporting DOL efforts to expand apprenticeship and thus are willing to contribute to participating in an evaluation to build the evidence base around this workforce model. Grantees are aware of the federal evaluation and have agreed to participate as a condition of their DOL grant. And the evaluation team is committed to providing the support and guidance to ensure minimal burden and high response rates. The evaluators expect high response rates on each of the data collection efforts described in this package, ranging from a 90 percent response rate on the grantee survey to a 100 percent response rate for the implementation study.
Implementation Study Interview Guides
For the site visits, it is expected that all of the grantee organizations approached will agree to participate.1 Once selected sites have been confirmed, site visitors will work closely with the primary DOL contact for each grantee to help in scheduling the site visit. One member of the two-person site visit team will take responsibility for working with the primary contact person to handle the scheduling and logistics, e.g., identifying appropriate interview respondents. Dates for site visits will be set at least one month in advance to allow ample time to schedule interviews. Interview appointments will then be confirmed via e-mail the week prior to the visit. Should a potential respondent not be available during the visit, the research team will follow up with a time to interview the person by phone.
Grantee Survey
To achieve a high response rate on the grantee survey, the evaluation team will take the steps outlined below:
DOL will send advance letters or emails to all grant directors one month before the survey. The communication will specify the date on which the survey is scheduled to be sent, the formats in which it will be available (online or in a Microsoft Word version, if needed), the time expected to complete the survey, and the survey’s originator.
On the scheduled date, the evaluator will email all primary grantee contacts the link to the online survey and instruction for completion. The respondents will be provided with a contact should they encounter any problems or questions as they complete the survey.
The evaluator will track who has started the survey and monitor their progress and follow up with any grantees that have not started or completed the survey. Follow-up with the grantee respondents will occur through periodic email reminders and telephone calls.
As each survey is reviewed, follow-up emails and telephone calls will be made to those respondents whose surveys contain errors, unclear responses, or missing information.
The evaluator will remind grantees in survey-related communications that participation in evaluation activities is a requirement of their grant.
MIS for the Impact Study of Employer Outreach Activities
A response rate is not applicable to the MIS for the impact study of employer outreach activities; however, the evaluation team expects that the four grantees that agree to participate in the study will consistently enter data in real-time. The evaluation team will identify grantees that are strong candidates for the impact study and also ensure that the grantees want to be in the impact study. Since those that participate will have expressed a desire to be in the study, we expect them to comply with data entry requirements. The evaluation team will review the MIS periodically to determine if data is entered consistently and correctly.
This section describes any tests of the data collection instruments included in this submission.
Grantee Survey
The grantee survey was developed and reviewed by DOL staff and evaluation team members. The evaluation team pre-tested the grantee survey with two AAI grantees and two recipients of DOL Trade Adjustment Assistance Community College Career Training (TAACCCT) grants that operate apprenticeship programs. Grantees provided feedback on the experience of completing the survey, both in written comments and in telephone conversations with an evaluation team member. Respondents commented on their perceptions of the clarity and flow of survey items, ease of completion, and time requirements. After pretesting, the evaluation team revised the instruments based on the feedback. Changes made to the instruments are included in this revised clearance request for OMB to review. AAI grantees that completed the survey during the pre-test will be given their completed surveys to review and update when the full survey is fielded to reduce burden while ensuring all responses are accurate and up-to-date.
Implementation Study Interview Guides
Experienced implementation researchers developed the implementation study interview guides, deriving the types of questions from protocols used for other federal evaluations of workforce programs. The estimated completion time for the implementation study interviews is based on extensive experiences with previous site visits to DOL grantees for other projects.
MIS for the Impact Study of Employer Outreach Activities
The time estimates for data entry by grantee staff into the impact study MIS are based on the expected number of staff involved in the impact study and the number of grantees that will receive marketing. Most fields are drop-down menus.
With DOL oversight, Abt and its partners are responsible for conducting the AAI Evaluation. The individuals listed in Exhibit B2 below made a contribution to the design of the implementation study and impact study of employer outreach activities. The data collected for the impact study will be analyzed under the direction of John Trutko, Capital Research Corporation. Both the conduct and analysis of data for the Implementation Study will be under the direction of Alan Werner, Abt Associates. Karen Gardiner, the project director for the AAI evaluation, will have oversight of all sub-studies and data collection efforts.
Exhibit B2: Individuals Consulted on Data Collection
Name |
Telephone Number |
Role in Study |
Karen Gardiner |
(301) 347-5547 |
Project Director |
Robert Lerman |
(202) 261-5676 |
Principal Investigator |
Karin Martinson |
(301) 347-5726 |
Co-Principal Investigator |
Jacob Klerman |
(617) 520-2613 |
Project Quality Advisor |
Alan Werner |
(617) 349-2832 |
Implementation Study Lead |
John Trutko |
(703) 522-0885 |
Impact Study Lead |
Inquiries regarding the statistical aspects of the study’s planned analysis should be directed to:
Karen Gardiner |
Project Director |
301-347-5116 |
Michelle Ennis |
Contracting Officer’s Representative, Employment and Training Administration |
202-693-3636 |
1 The expected response rate by the grantees is 100 percent. Participation in evaluation activities is required as a condition of the grant award.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-21 |