OMB Supporting Statement Part A: Justification
A.1. Circumstances Necessitating the Information Collection
The Employment and Training Administration (ETA) in collaboration with the Chief Evaluation Office (CEO) of the U.S. Department of Labor (DOL) is undertaking the Evaluation of the American Apprenticeship Initiative (hereafter, the AAI evaluation) and is seeking approval from OMB under the Paperwork Reduction Act for new data collection instruments associated with the evaluation. The new data collection instruments included in this request are the AAI evaluation’s Employer and Participant Surveys.
AAI is authorized by Section 414(c) of the American Competitiveness and Workforce Improvement Act of 1998 (ACWIA), as amended (codified at 29 USC 2916a). Through the provision of $175 million in five-year grants to 46 grantees, AAI aims to support and encourage growth of an American Apprenticeship system that meet the country’s economic, industry and workforce needs. This includes not only increasing the number of registered apprenticeships available in the U.S., but also encouraging apprenticeships in non-traditional industries and engaging groups who have not traditionally enrolled in apprenticeship programs.1
The DOL Funding Opportunity Announcement for the AAI grants (FOA-ETA-15-02) established that DOL is committed to producing strong evidence on the effectiveness of grantee programs and that all grantees must fully participate in any evaluation initiated by the department as a condition of grant award.2 Further, the June 15, 2017 Presidential Executive Order Expanding Apprenticeships in America promoted rigorous third-party evaluations of apprenticeship programs.3
A.2. Purpose and Use of the Data
Purpose
The AAI evaluation will address the following six research questions:
What are AAI grantees doing to generate apprenticeship slots?
What strategies do AAI grantees and employers use to identify strong candidates for apprenticeships?
What are the characteristics of AAI apprenticeships?
What are the in-program experiences and post-program outcomes for apprentices?
What innovations and lessons form the basis for broader change and sustainability that encourages employers to adopt apprenticeships?
Do benefits accrue to employers from apprenticeship?
To answer these questions, the evaluation is implementing four sub-studies: (1) an implementation study that describes how AAI programs develop, operate and mature; (2) an outcomes study that examines in-program and post-program outcomes of apprentices; (3) a return on investment (ROI) study that seeks to determine average ROI estimates for employers that hire apprentices; and, (4) an impact study that assesses employer outreach activities for a subset of AAI grantees. OMB approved of the data collections instruments (i.e., grantee survey, site visit discussion guides, and management information system fields) that support the implementation and impact studies (OMB Control Number:1290-0017; expires 11/30/2021). This new submission is to request clearance for two data collection instruments—the Employer Survey and the Participant Survey—that will support the ROI study and outcomes study, respectively.
Employer Survey. The Employer Survey is the primary data source for the ROI study. Much of the information gathered by this survey is private to employers, thus is not available through other sources. The survey asks employers to provide information about a particular apprentice’s wages, productivity, and progression. The survey also includes questions about the employer’s background, recruitment of employees, and characteristics of the apprenticeship. There are two categories of respondents for the Employer Survey. The first set of respondents is employers who hire apprentices associated with AAI grantees. The survey is voluntary for employers. The second set of respondents is all 46 AAI grantees who will be asked to provide background information for a subset of employers (two per grantee). Grantees are required to participate in evaluation activities per terms of their grants. The survey results will be used to estimate the ROI for the AAI overall, and also generate ROI estimates across industries and regions.
Participant Survey. The Participant Survey will support the outcomes study by gathering information about apprentices’ experiences in AAI apprenticeships. The information collected through this survey is not available through any other source. It includes questions on an apprentice’s background prior to apprenticeship (e.g., past experiences in labor market, demographics), apprenticeship experiences, skills and knowledge gained, and outcomes. The Participant Survey respondents are a sample of apprentices in AAI-affiliated programs who have completed their program or are still enrolled. This survey is also voluntary. The data collected complements information grantees report about apprentices in the DOL Quarterly Performance Results (QPR) system.
Users of Data
AAI is a significant DOL investment to expand apprenticeship in the U.S., and the evaluation will provide a wealth of information on apprenticeship outcomes and ROI for businesses that invest in apprenticeship. The immediate users of data collected will be the evaluation team. A Public Use File will be prepared and made available to future researchers. In addition to the users of data, stakeholders who will benefit from the information generated by the data collection include:
DOL and other federal agency staff
State and local workforce agencies or apprenticeship systems such as State Apprenticeship Agencies
Other sponsors of apprenticeships such as nonprofits, unions, and community colleges
Researchers
Policymakers at the local, state and federal levels of government looking to increase and expand apprenticeships
Others interested in understanding the experiences and lessons from expanding registered apprenticeships
A.3. Use of Improved Information Technology
The Employer Survey will be administered by phone to a sample of approximately 100 employers.4 The evaluator will first facilitate completion of the Employer Survey by speaking with each AAI grantee about the selected employers. As they walk through the survey by phone with the grantee, the evaluator will fill in relevant sections of the survey. The evaluator will then interview the employers by phone, reading the remaining questions and recording the information on the survey.
The Participant Survey will be fielded using dual-mode web and telephone administration. Both modes will use the same Computer-Assisted Interviewing (CAI) technology. CAI technology reduces respondent burden, as respondents and interviewers can proceed more quickly and accurately through the survey instruments, minimizing the interview length. Computerized questionnaires ensure that the skip patterns work properly, minimizing respondent burden by not asking non-applicable questions. Computer-assisted interviewing can build in checkpoints, which allow the interviewer or respondent to confirm responses, thereby minimizing data entry errors. Finally, automated survey administration can incorporate hard edits to check for allowable ranges for quantity and range value questions, minimizing out of range or unallowable values.
A.4. Efforts to Identify Duplication
While background information such as the employer size and industry may be available through third party sources for a limited number of employers, much of the information gathered by the Employer Survey is private and therefore is not available elsewhere. Where possible, the study team will prefill response answers using QPR data, which may include information such as the employer’s industry and occupation. However, many QPR fields are not required and therefore not complete. Additionally, the evaluator will call AAI-grantees in order to pre-fill early survey questions related to apprentice occupation and program size, for two employers. Additionally, many of the survey questions ask for apprentice-level data, such as wage progression. This apprentice-level data is not available through any other source than the employer’s own records.
The majority of information collected through the Participant Survey is not available elsewhere. The exception is demographic information, which is reported by grantees in the QPR. However, as previously noted, many fields in the QPR are not required, and analysis of the data indicates a high rate of missing demographic data. Because DOL, through AAI, is interested in increasing apprenticeship opportunities for non-traditional populations, the study team needs to obtain demographic information in order to answer evaluation questions.
A.5. Collection of Data from Small Businesses
The Participant Survey does not involve data collection from small businesses or other small entities. The Participant Survey will collect information directly from apprenticeship participants. The Employer Survey respondents are employers that have hired one or more apprentices, and some of them may be small businesses.
The study team will take steps to reduce the burden on small businesses. It will collect the minimal amount of data needed. As mentioned in A.2, the survey is voluntary; any employer, regardless of size, can opt not to participate in it. The evaluator will first ask AAI grantees to answer some questions, reducing the time small businesses spend gathering data and answering survey questions.
A.6. Consequences of Less Frequent Data Collection
Both the Employer Survey and the Participant Survey will be administered once. The evaluator would have great difficulty addressing the outcomes study and ROI questions in absence of this data collection. The Employer Survey is the key source of information for the ROI study and the information in the survey could not be obtained from any other sources. Similarly, the Participant Survey is the only source of information about apprentice motivations, in apprenticeship experiences, and outcomes.
A.7. Special Circumstances of Data Collection
There are no special circumstances for the proposed data collection.
A.8. Federal Register announcement and Consultations Outside the Agency
A 60-day notice to solicit public comments was published in the Federal Register, Volume 83, No. 166, page 43708, on August 27, 2018. DOL received one request for electronic versions of the Employer Survey and Participant Survey.
The following people were consulted on the evaluation design and data collection tools for this study:
Carolyn Heinrich, Professor of Public Policy, Education and Economics, Vanderbilt University; Research Professor at the Lyndon B. Johnson School of Public Affairs at University of Texas Austin;
Harry Holzer, Professor, McCourt School of Public Policy, Georgetown University; Nonresident Senior Fellow of Economic Studies, Brookings Institution;
Chris King, Senior Research Scientist, Ray Marshall Center and Lecturer at the Lyndon B. Johnson School of Public Affairs, University of Texas Austin;
Stefan Wolter, Managing Director, Swiss Coordination Centre for Research in Education; Professor of Economics at the University of Bern.
Additionally, Dr. Susan Helper, Frank Tracy Carlton Professor of Economics at the Weatherhead School of Management at Case Western Reserve University, reviewed the Employer Survey.
A.9. Payments to Respondents
Participant Survey respondents will receive a payment in appreciation of their time for completing the survey. The evaluator proposes an experiment to assess the effects of respondent payments on survey response rates. The evaluator proposes to randomly assign apprentices to one of three payment options:
Option 1: Constant $25 payment regardless of when the survey is completed and which mode used (online or phone)
Option 2: Constant $40 payment regardless of when the survey is completed and which mode used (online or phone)
Option 3: Two-tier payment: $40 payment if the survey is completed during the first four weeks of administration (the online mode); $20 if the survey is completed after the first four weeks (phone mode)
The survey will be released over many waves. In early waves, sample members will be divided evenly across the three payment options. In later survey waves, more sample members will be offered the payment option that is producing the highest response rate.
Employer Survey respondents will not receive any payments for completing the survey but will be offered an individualized return-on-investment estimate. More detail on this estimate can be found in Part B.
A.10. Assurance of Privacy
Information collected will be kept private to the extent permitted by law. All evaluation team members signed confidentiality/non-disclosure agreements (NDAs) with DOL. This includes Abt Associates, the prime contractor, and all subcontractors. Furthermore, the contractor complies with all DOL data security requirements. For both the Employer Survey and the Participant Survey, respondents will first be asked to consent to the survey. The consent states that the evaluators treat the information respondents provide as private. All publications based on the surveys will report findings at the aggregate level, not at the individual level, and the names of Participant respondents or companies that completed the survey will not be published in any form. Only the evaluation team will be able to identify individual responses. The evaluation team may create a public use file, but there will be no identifying information in the file.
To protect respondents’ privacy, all data will be stored on a password protected drive established at the contractor site. Access to this drive will be limited to research staff members who are working on the project and have signed the non-disclosure agreement.
A.11. Sensitive Questions
Both surveys include some questions about wage earnings that can be deemed sensitive. The Employer Survey asks employers for information about an apprentice’s wages that may be considered proprietary. These data are needed to estimate a ROI and are not available through other sources. DOL is also interested in learning about apprentices’ initial hourly wages, wage progression, and post apprenticeship wages. There is no available administrative dataset that has this information. As such, these two surveys are the only options for collecting data in order to answer the relevant research questions.
A.12. Estimated Response Burden
The table below describes the assumptions made about the annual number of responses expected, the average hours of burden per respondent, and the burden costs estimated for the both the Employer Survey and the Participant Survey.
Estimated Annualized Respondent Hour and Cost Burden
Type of Instrument (Form/Activity) |
Number of Respondents a |
Number of Responses per Respondent |
Total Number of Responses |
Average Burden Hour per Response (in Hours) |
Estimated Burden Hours |
Average Hourly Wage |
Annual Burden Costs |
Employer Survey - Employer and Grantee Respondents |
42 b |
1 |
42 |
.5 |
21 |
$45.10 c |
$947 |
Participant Survey |
667 d |
1 |
667 |
.5 |
334 |
$25.54 |
$8,530 |
Totals |
709 |
- |
709 |
- |
355 |
- |
$9,477 |
a DOL seeks a clearance period of three years
b Includes both employer and grantee respondents. Assumes a sample of 100 employers with an 80 percent response rate and 46 grantees with a 100 percent response rate.
c This is calculated by taking the average of the combined hourly wages for an employer respondent ($59.38) and grantee respondent ($30.82).
d Assumes a sample of 2,500 participants with an 80 percent response rate.
A.13. Cost to Respondents
There are no additional costs to respondents other than their time.
A.14. Cost to the Federal Government
The total annualized cost to the federal government is $627,995. Costs result from the following two categories:
The annualized cost to the federal government for the contractor to carry out this study is $615,915. This is calculated by the following: the estimated cost to the federal government for the contractor to carry out data collection for the Employer Survey is $725,910 and for the Participant Survey is $1,121,835, for a total of $1,847,745 for three years. Therefore, the annualized cost is $1,847,745 / 3 years = $615,915.
The annualized cost for federal technical staff to oversee the study is $12,080. This is calculated by the following: an annual level of effort of 200 hours for one Washington, DC–based Federal GS-14 step 4 employee earning $60.40 per hour. (See Office of Personnel Management 2018 Hourly Salary Table at https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2018/DCB_h.pdf). Therefore, the annualized cost is 200 hours x $60.40 = $12,080.
The total annualized cost to the federal government is $627,995 ($615,915 + $12,080= 627,995).
A.15. Reasons for Program Changes
This is a new data collection.
A.16. Publication Plans and Project Schedule
For the AAI evaluation, DOL is seeking OMB approval beginning in 2019 and ending in 2020. The Employer Survey and Participant Survey will be administered late 2019 through early 2020.
Publications resulting from data collection instruments outlined in this PRA request will be completed on the following schedule:
Final report, including findings from the Participant Survey and Employer Survey will be submitted in 2021.
Public use or restricted use data file containing survey and QPR data collected for the study with personal identifiers removed will be submitted in 2021.
A.17. Approval to Not Display the Expiration Date for OMB Approval
All instruments created for the AAI evaluation will display the OMB approval number and the expiration date for OMB approval.
A.18. Exceptions to the Certification Statement
No exceptions are necessary for this information collection.
1 Department of Labor Funding Opportunity Announcement for the American Apprenticeship Initiative Grants (FOA-ETA-15-02), https://doleta.gov/grants/pdf/FOA-ETA-15-02.pdf
2 Ibid
3 Presidential Executive Order Expanding Apprenticeships in America, June 15, 2017, https://www.whitehouse.gov/the-press-office/2017/06/15/presidential-executive-order-expanding-apprenticeships-america
4 The evaluators will ask AAI grantees to provide the names and contact information for the employers to be surveyed.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |