Evaluation of the American Apprenticeship Initiative
ICR Reference Number 201802-1290-002
August 2018
The Employment and Training Administration (ETA) in collaboration with the Chief Evaluation Office (CEO) of the U.S. Department of Labor (DOL) is undertaking the Evaluation of the American Apprenticeship Initiative (hereafter, the AAI evaluation) and is seeking approval from OMB under the Paperwork Reduction Act for data collection instruments associated with the evaluation. With the goal of expanding registered apprenticeship in the United States, both at scale and into new sectors, the American Apprenticeship Initiative (AAI) provided $175 million in five-year grants to 46 grantees across the country. Grantees can use funds to support registered apprenticeships or pre-apprenticeships. DOL contracted with Abt Associates and its partners to conduct an AAI evaluation to build evidence about the effectiveness of apprenticeship for participants and employers and provide lessons for developing and operating apprenticeships. The evaluation will use a multi-pronged approach including 1) an implementation study, 2) an outcomes study, 3) a return on investment study, and 4) an impact study of employer outreach activities in a subset of AAI grantees. Data collection instruments described in this submission pertain to the AAI evaluation implementation study and the impact study of employer outreach practices of a subset of grantees.
Subsequent OMB submissions will seek clearance for additional data collection instruments for the outcomes study and return on investment study.
The AAI is authorized by Section 414(c) of the American Competitiveness and Workforce Improvement Act of 1998 (ACWIA), as amended, codified at 29 USC 3224a(7). Through the provision of $175 million in five-year grants to 46 grantees, the AAI aims to support and encourage growth of an American Apprenticeship system that meets the country’s economic, industry and workforce needs. This includes not only increasing the number of registered apprenticeships available in the U.S, but also encouraging apprenticeships in non-traditional industries such as IT and healthcare. The initiative also aims to engage non-traditional populations, such as women and people of color, in apprenticeships.1
The Department of Labor Funding Opportunity Announcement for the AAI grants (FOA-ETA-15-02) established that DOL is committed to producing strong evidence on the effectiveness of grantee programs and that all grantees must fully participate in any evaluation initiated by the department as a condition of grant award.2 Further, the June 15, 2017 Presidential Executive Order Expanding Apprenticeships in America promoted rigorous third-party evaluations of apprenticeship programs, including those that use a multi-site randomized control trial.3
The AAI evaluation will answer six broad research questions:
What are AAI grantees doing to generate apprenticeship slots?
What strategies do AAI grantees and employers use to identify strong candidates for apprenticeships?
What are the characteristics of AAI apprenticeships?
What are the in-program experiences and post-program outcomes for apprentices?
What innovations and lessons form the basis for broader change and sustainability that encourages employers to adopt apprenticeships?
Do benefits accrue to employers from apprenticeship?
The implementation study and impact study of employer outreach activities—subjects of this data collection request—will address questions 1, 2, 3, and 5. Specifically, this submission is to request clearance for the first three data collections below that are needed to address the implementation study and impact study-related questions: a grantee survey, site visit interview guides for two rounds of implementation study site visits, and a management information system (MIS). These, along with other data, are necessary to ensure that the AAI evaluation can adequately document and assess grantee programs. DOL anticipates submitting an additional OMB package to request permission to conduct a participant survey and an employer survey, which will support the outcomes study and the return-on-investment study, respectively. Following is a description of each data collection instrument that requires OMB clearance.
Site visit interview guides to be used in the first and second rounds of site visits to approximately 10 grantees for the implementation study and employer engagement impact study (clearance requested in this package)
A one-time online grantee survey administered to all 46 AAI grantees for the implementation study (clearance requested in this package)
A management information system (MIS) to collect information on outreach to employers by the up to four AAI grantees participating in the employer engagement impact study (clearance requested in this package)
A one-time apprenticeship/pre-apprenticeship participant survey for the outcomes study (clearance will be requested in a future package)
A one-time survey of employers participating in apprenticeships for the implementation study and return on investment study (clearance will be requested in a future package)
Site Visit Interview Guides. Two rounds of site visits to approximately 10 grantees will collect in-depth qualitative information on grantee operations in support of the implementation study. The interviews will be with the following:
grant director,
grantee staff,
related technical instruction provider staff,
employer partners,
workforce development partner staff, and
union representatives.
These site visit interview guides will cover a wide range of topics that align with the AAI Logic Model. While the grantee survey will capture consistent information about grantee practices at a point in time, the site visits will document grantee practices at two points in time to better understand programs were implemented, what changed over time, challenges grantees faced in implementing their programs, and how they overcame barriers to implementation.
Grantee Survey. The grantee survey will collect consistent information about apprenticeship programs from all 46 grantees and will support the implementation study. The survey covers grantee program context; program development; partners and their involvement; program implementation, including participant recruitment and services provided; employer engagement; and grantee perspectives on participant experiences.
Management Information System (MIS). The MIS will be used by up to four grantees that DOL will select to be a part of an impact study that explores whether intensive marketing and outreach efforts to a randomly selected group of employers lead to an increase in employers sponsoring apprenticeship programs. Grantees will use the MIS to track contacts with employers, including the type of marketing and frequency of contacts, as well as services the grantee provides to the employer to help establish AAI apprenticeships. This data will be used to document the implementation of employer outreach activities. Specifically, it will provide information on the intensity of grantee efforts to engage employers in the treatment group. The key outcome for the employer outreach impact study is apprenticeship registration, and data on this will be collected from the AAI grants’ Quarterly Performance Results database for both treatment and control group members.
The AAI is a significant DOL investment to expand apprenticeship in the U.S., and the implementation study and employer outreach impact study will provide a wealth of information on creating and implementing apprenticeships to meet worker and employer needs. The immediate users of data collected will be the Abt Associates evaluation team, who will use the data to describe how the grants are being implemented and derive lessons for developing, operating, and expanding apprenticeships. The data will also build evidence as to how effective different marketing and recruitment approaches are in persuading more employers to offer apprenticeships.
Indirect users of the data collected include:
DOL and other federal agency staff
State and local workforce agencies or apprenticeship systems and organizations such as State Apprenticeship Agencies
Other sponsors of apprenticeships such as nonprofits, unions, and community colleges
Researchers
Policymakers at the local, state and federal levels of government looking to increase and expand apprenticeships
Others interested in understanding the experiences and lessons from expanding registered apprenticeships
Finally, a Public Use File will be prepared and made available to future researchers.
The grantee survey and the MIS will each build in multiple technological elements to minimize burden.
The grantee survey will be administered online. Burden-reducing features include:
Secure personalized access. Each grantee will receive a link and password to the web-based grantee survey. The survey will be created using commercial software that will allow grantees to answer questions at a time that suits them.
Automated skip patterns. Skip logic embedded in the survey place less of a burden on the respondent than the customary “if-then go to” instructions of a paper and pencil questionnaire. Including efficient skip patterns minimizes respondent burden by not asking inappropriate or non-applicable questions
Modularity. The survey will be in a modular format that allows the primary respondent to pass sections or questions on to other staff members who may be better equipped to address particular topics.
Automated edit checks. The computerized instrument will check for allowable ranges for quantity and range value questions, minimizing out of range or unallowable values. These eliminate entry errors that can require follow-up via phone calls to gather correct details.
The evaluation team will use a web-based MIS system to collect uniform data elements from grantees involved in the employer outreach impact study. Up to four grantees will use the electronic system to document contacts with employers and the type of activities they engage in with employers. The evaluation team will provide comprehensive training on how to upload and input all reporting information. The MIS will use drop-down menus and response categories to minimize the data entry burden on grantee staff.
This will be the first evaluation of the AAI grants. Qualitative grantee-level data that we request in this PRA package has not been collected before and is not collected in other formats or through other means. In regards to the ICR for the implementation study site visit interview guides, the protocols are designed to complement and build on information provided by grantees in their annual reports. In regards to the ICR for the grantee survey, grantees currently submit narrative quarterly reports to DOL, but they do not provide sufficient detail across implementation study topics. Grantees do provide information about their program activities in the Quality Performance Results system, and the survey is designed to complement this data by exploring study topics in further detail.
Similarly, for the MIS, there is no other source to collect data on grantee interactions with employers. It is needed in order to collect data consistently across the four grantees involved in the employer outreach impact study. In order to reduce burden, the MIS will not collect information provided elsewhere. This includes information from the Quality Performance Results about which employers sponsor apprenticeships and from the workforce agency or similar database about employer size, industry and contact information.
The involvement of small businesses or other small entities that are not grantees or sub-grantees is expected to be limited. For the employer outreach impact study, grantees may market apprenticeships to small businesses, and these employers can choose whether or not to engage with the grantees. Additionally, for the implementation study, interviews with grantee partners or employers may involve small businesses or other small organizations. For the interviews, no information will be collected without their voluntary consent and the discussions would be limited to 60 minutes. Only the minimal amount of data needed for this study will be collected.
The grantee survey is a one-time data collection activity. The survey is critical to the implementation study. It will systematically document how all AAI-funded grants operate. Without this information, the evaluators will not have details on the implementation of programs across all grantees.
The two rounds of site visits to approximately 10 grantees will also be critical to the implementation study. The two visits are timed to capture information about initial grant implementation, changes made over time and reasons for them, and plans to sustain activities following the end of the grant. The first round of visits will document services implemented as compared to services outlined in the grant application. The second round of visits will occur when grantee operations are mature and planning is underway for sustainability. The in-depth information gathered during the site visits will complement the information gathered from the survey. The additional detail captured by the visits will help to explain findings for the sub-studies. Without these visits, the evaluation team would not be able to capture the evolving nature of program services, as well as document challenges and their solutions.
The MIS will be used by staff of the up to four participating grantees to document contact with employers and activities to help interested employers plan, register and implement apprenticeships. Grantee staff will enter information for the duration of the study, which is expected to last about 24 months. The information that is collected from grantees through the MIS is essential to the study because the data on number of contacts, nature of employer outreach, and services provided to employers to help them offer apprenticeships is not available from any other source. Employer engagement is a key factor that limits the number of apprenticeships available, and the information captured in the MIS will help DOL learn the effectiveness of different marketing and outreach approaches to employers.
There are no special circumstances for the proposed data collection.
The evaluation team consulted with members of the project Technical Work Group on the evaluation design and data collection tools:
Carolyn Heinrich, Professor of Public Policy, Education and Economics, Vanderbilt University; Research Professor at the Lyndon B. Johnson School of Public Affairs at University of Texas Austin;
Harry Holzer, Professor, McCourt School of Public Policy, Georgetown University; Nonresident Senior Fellow of Economic Studies, Brookings Institution;
Chris King, Senior Research Scientist, Ray Marshall Center and Lecturer at the Lyndon B. Johnson School of Public Affairs, University of Texas Austin; and
Stefan Wolter, Managing Director, Swiss Coordination Centre for Research in Education; Professor of Economics at the University of Bern.
Respondents will not receive payments.
Staff with Abt Associates, the prime contractor, and its subcontractors signed contractor confidentiality/non-disclosure agreements (NDAs) with DOL.
To protect site visit respondents’ privacy, all hard copies of site visitor notes will be stored in a locked file cabinet at Abt Associates when not in use. Electronic versions of site visitor notes will be stored on a password protected drive set up by the IT departments at Abt and each of the subcontractors involved with this evaluation. Access to this drive will be limited to research staff members who are working on the project and have signed the non-disclosure agreement.
There are no sensitive questions on the implementation study interview guides, the grantee survey, or in the MIS.
Exhibit A2 presents the estimated respondent burden on grantee staff and their partners to complete the data collection instruments included in this package. It shows the average time, in hours, to participate in interviews for the implementation study, to complete the grantee survey, and to enter data into the MIS.
The AAI evaluation team will interview up to 12 staff and partners at approximately 10 grantees. They will interview the Grant Director at each grantee and one or two additional respondents from the grantee organization. They will also interview three respondents representing employers or unions, two or three respondents representing related technical instruction providers, and two or three respondents representing the workforce development board or other community partners. With the exception of the Grant Director (two hours), each interview will take approximately one hour.
The evaluation team estimates the grantee survey will take an average of 4 hours to complete. The evaluation team pre-tested the grantee survey with two AAI grantees and two colleges associated with DOL Trade Adjustment Assistance Community College Career Training (TAACCCT) grants that operate apprenticeship programs. The response time is an average based on the amount of time it took the pre-test organizations to complete it. The respondent will most likely be the Grant Director or an administrator at the grantee. The estimated response rate is 90 percent. Although participation in evaluation activities is required as a condition of the grant award, it is likely that due to changes in staffing, about 10 percent of grantees will not respond to the survey.
Grantee staff will be asked to enter data into the MIS for a limited number of fields, primarily focused on their attempts to contact employers regarding take-up of apprenticeships. It is estimated that each of the four grantees will contact up to 200 employers and data entry is expected to take 15 minutes per employer. The annual estimated burden is 400 hours.
Exhibit A2: Estimated Annualized Respondent Hour and Cost Burden
Instrument/ data collection activity |
Number of Respondents |
Number of Responses per Respondent |
Total Number of Responses |
Average Burden per Response (In hours) |
Estimated Burden Hours |
Average Hourly Wage Rate |
Total Annual Burden Cost |
Site Visit Interview Guide – Grant Director a |
3 |
2 |
6 |
2 |
12 |
$30.84 |
$370 |
Site Visit Interview Guide –Grantee staff b |
7 |
2 |
14 |
1 |
14 |
$18.30 |
$256 |
Site Visit Interview Guide –Related Technical Instruction Provider c |
10 |
2 |
20 |
1 |
20 |
$38.73 |
$775 |
Site Visit Interview Guide -Employer Partners d |
7 |
2 |
14 |
1 |
14 |
$55.09 |
$771 |
Site Visit Interview Guide- Workforce Development partners e |
7 |
2 |
14 |
1 |
14 |
$41.86 |
$586 |
Site Visit Interview Guide-Union Representatives f |
3 |
2 |
6 |
1 |
6 |
$29.71 |
$178 |
Grantee Survey g |
14 |
1 |
14 |
4 |
56 |
$30.84 |
$1,727 |
MIS h |
8 |
200 |
1600 |
0.25 |
400 |
$18.30 |
$7,320 |
Total |
31 |
NA |
1,688 |
NA |
536 |
NA |
$11,983 |
a The median hourly wage for Grant Directors was calculated based on information from the Bureau of Labor Statistics; May 2016 National Industry-Specific Occupational Employment and Wage Estimates found at https://www.bls.gov/oes/current/naics3_624000.htm#11-0000: Social and Community Service Managers (NAICS 62400, SOC code 11-9151 wage rate of $30.84.
b The median hourly wage for Grantee Staff was calculated based on information from the Bureau of Labor Statistics; May 2016 National Industry-Specific Occupational Employment and Wage Estimates found at https://www.bls.gov/oes/current/oes211093.htm Community and Social Service Occupations. (NAICS 62400 SOC code-210000 $18.30.
c The median hourly wage for Related Technical Instruction staff was calculated based on information from the Bureau of Labor Statistics; May 2016 National Industry-Specific Occupational Employment and Wage Estimates found at https://www.bls.gov/oes/current/oes251194.htm: Education, Training, and Library Occupations (NAICS 611300,SOC code 25-000) wage rate of $38.73.
d The median hourly wage for Employer Partners was calculated based on information from the Bureau of Labor Statistics; May 2016 National Industry-Specific Occupational Employment and Wage Estimates found at http://www.bls.gov/oes/current/naics2_31-33.htm#11-0000: Human Resources Managers (NAICS Manufacturing 31, 32, 33, SOC code 11-3121,) wage rate of $55.09.
e The median hourly wage for Workforce Development Agency Partners was calculated based on information from the Bureau of Labor Statistics; May 2016 National Industry-Specific Occupational Employment and Wage Estimates found at http://www.bls.gov/oes/current/naics4_999300.htm: Local government managers, excluding schools and hospitals (NAICS 999300, SOC code 11-0000) wage rate of $41.86.
f The median hourly wage for Union Representative was calculated based on information from the Bureau of Labor Statistics; May 2016 National Industry-Specific Occupational Employment and Wage Estimates found at https://www.bls.gov/oes/current/naics4_813900.htm: Labor Relations Specialists (NAICS 813900 , SOC Code 13-1075) wage of $29.71.
g There are 46 grantees with an expected response rate of 90 percent. The most likely respondent in this case will be the Grant Director. The median hourly wage for Grant Directors was calculated based on information from the Bureau of Labor Statistics; May 2016 National Industry-Specific Occupational Employment and Wage Estimates found at https://www.bls.gov/oes/current/naics3_624000.htm#11-0000: Social and Community Service Managers (NAICS 62400, SOC code 11-9151) wage rate of $30.84.
h The most likely respondent in this case will be Grantee Staff. The median hourly wage for Grantee Staff was calculated based on information from the Bureau of Labor Statistics; May 2016 National Industry-Specific Occupational Employment and Wage Estimates found at https://www.bls.gov/oes/current/oes211093.htm Community and Social Service Occupations. (NAICS 62400 SOC code-210000 $18.30.
There are no additional costs to respondents other than their time.
The total annualized cost to the federal government is $893,174. Costs result from the following two categories:
The annualized cost to the federal government for the contractor to carry out this study is $881,094. This is calculated by the following: the estimated cost to the federal government for the contractor to carry out this study is $654,014 for the implementation study; $411,539 for the grantee survey; and $1,577,730 for the impact study for a total of $2,643,283 for three years. Therefore, the annualized cost is $2,643,283 / 3 years = $881,094.
The annualized cost for federal technical staff to oversee the study is $12,080. This is calculated by the following: an annual level of effort of 200 hours for one Washington, DC–based Federal GS-14 step 4 employee earning $60.40 per hour. (See Office of Personnel Management 2018 Hourly Salary Table at https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2018/DCB_h.pdf). Therefore, the annualized cost is 200 hours x $60.40 = $12,080.
The total annualized cost to the federal government is $893,174 ($881,094 + $12,080 = $893,174).
This is a new data collection effort (OMB No. 1290-XXX).
For the evaluation of the American Apprenticeship Initiative grants, ETA is seeking OMB approval beginning in 2018 and ending in 2020.
First round implementation study site visits will occur in 2018.
Second round implementation site visits will occur in 2019.
Site visits to train the employer outreach impact study sites will occur in 2018.
The employer outreach impact study and associated data collection begins after sites have been trained and continues for up to two years.
Grantee survey will be administered in 2018.
Deliverables for the AAI Evaluation will be completed on the following schedule:
Analysis plan, which will include a data security plan, will be submitted in 2018.
Interim report, summarizing findings from the first round of site visits, preliminary QPR information, and the grantee survey will be submitted in 2019.
Final report, including findings from the second round of site visits; the outcomes study using data from the QPR, NDNH, and the participant survey, and the ROI study and employer outreach impact study will be submitted in 2021
Public use or restricted use data file containing all data collected for the study with personal identifiers removed will be submitted in 2021.
All instruments created for the AAI evaluation will display the OMB approval number and the expiration date for OMB approval.
No exceptions are necessary for this information collection.
1 Department of Labor Funding Opportunity Announcement for the American Apprenticeship Initiative Grants (FOA-ETA-15-02), https://doleta.gov/grants/pdf/FOA-ETA-15-02.pdf
2 Ibid
3 Presidential Executive Order Expanding Apprenticeships in America, June 15, 2017, https://www.whitehouse.gov/the-press-office/2017/06/15/presidential-executive-order-expanding-apprenticeships-america
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Abt Single-Sided Body Template |
| Author | Katheleen Linton |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |