State Apprenticeship Expansion (SAE) Grant Research Study
OMB Control No. 1205-0NEW
March 2019
SUPPORTING STATEMENT
STATE APPRENTICESHIP EXPANSION (SAE) GRANT RESEARCH STUDY
OMB Control Number 1205-0NEW
PRA Supporting Statement Part B
1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.
This section outlines the respondent universe and expected response rates for each data collection activity of the proposed research. The four activities include:
State survey. The respondent universe for the survey is all 50 states, the District of Columbia, and 3 territories with federal apprenticeship liaisons. All respondents in the universe will be invited to complete the survey, so no statistical methods will be used to select a sample. Each state has a unique context and approach to apprenticeship, and the state survey is designed to capture the breadth of variation across states. A 90 percent response rate is expected for the survey.
State Apprenticeship Expansion grantee telephone interviews. All 37 State Apprenticeship Expansion grantees are the respondent universe for this activity. The study will conduct telephone interviews with administrators in each of the grantees that received a State Apprenticeship Expansion grant. All respondents in the universe will be invited to an interview, so no statistical methods will be used to select a sample. Each grantee has a unique context and approach to implementing the grant, and the study is designed to capture the breadth of variation across grantees. We expect a 100 percent response rate.
National industry intermediary telephone interviews. All eight national industry intermediaries whose contracts were renewed for a second year are the respondent universe for these telephone interviews. The study will conduct telephone interviews with each of the eight contractors; thus, no statistical methods will be used to select a sample. Each contractor had a different goal and purpose for their contract and the study will capture the implementation and achievements across all contractors. We expect a 100 percent response rate.
National equity partner telephone interviews. The two national equity partners whose contracts were renewed for a second year are the respondent universe for these telephone interviews. The study will conduct telephone interviews with both contractors; thus, no statistical methods will be used to select a sample. Each contractor had a different goal and purpose for their contract and the study will capture the implementation and achievements across all contractors. We expect a 100 percent response rate.
2. Describe the procedures for the collection of information including:
* Statistical methodology for stratification and sample selection,
* Estimation procedure,
* Degree of accuracy needed for the purpose described in the justification,
* Unusual problems requiring specialized sampling procedures, and
* Any use of periodic (less frequent than annual) data collection cycles to reduce burden.
Understanding the effectiveness of DOL’s efforts to expand and diversify registered apprenticeship requires data collection from multiple sources. All findings are to be considered descriptive and are based on the triangulation of multiple perspectives across the various key stakeholders. To collect these data, the study team will administer a state survey in spring 2019, as well as conduct semistructured telephone interviews with grantees and contractors in summer 2019. The data covered by this clearance include those from the state survey and telephone interviews. The population frame and sample sizes are presented in Table B.1.
Table B.1. Population frame and estimated sample sizes
Type of Instrument (Form/Activity) |
Population frame |
Sample size |
State survey |
54 |
54 |
Semi-structured telephone interviews (state grantee administrator) |
37 |
37 |
Semi-structured telephone interviews (national industry intermediaries) |
8 |
8 |
Semi-structured telephone interviews (national equity partners) |
2 |
2 |
State survey. A 30-minute web survey will be administered to workforce system administrators in all 54 “states” to systematically capture information about state-specific registered apprenticeship activities. Login information for the survey will be sent via email to the lead contact at each state. During the fielding window, periodic reminders to complete the survey will be sent to lead contacts who have not yet returned the survey.
State Apprenticeship Expansion grantee telephone interviews. The study team will conduct a two-hour in-depth telephone interview with a leader from each of the 37 State Apprenticeship Expansion grantees. An email invitation to schedule a call will be sent to lead contacts for each grantee, and follow-up emails will be sent as needed. These in-depth interviews will discuss their efforts to build partnerships and programs, efforts to expand interest and use of apprenticeships by individuals and employers, the perceived challenges encountered in this work, and the strategies used to overcome them.
National industry and equity contractor telephone interviews. The study team will conduct a two-hour telephone interview with leaders from each National Industry Intermediary and National Equity Partner contractor. An email invitation to schedule a call will be sent to lead contacts for each grantee, and follow-up emails will be sent as needed. These in-depth interviews will discuss their progress in implementing strategies, the perceived challenges they have faced, their perceived successes, and the lessons they have learned in expanding apprenticeship for their industries and for under-represented populations.
The data gathered through the state survey will be tabulated using descriptive methods to provide contextual information about the characteristics of the grant programs. Analysis will follow a common set of steps: data cleaning, variable construction, and computing descriptive statistics. Key variables that will be constructed for the analysis include state apprenticeship program characteristics, key partners and their roles, the types of industries targeted, activities conducted, training provided, and challenges faced. To prepare data for analysis, a series of data checks will be run, allowing for an examination of frequencies and means, and an assessment of the extent of missing data. As the states and organizations in the population are not representative of one another, weighting or imputations to account for missing data will not be possible. We anticipate in most cases that missing data will be deleted and the sample size noted for each question where responses are missing.
Data collected from the interviews will be qualitative information about respondents’ experiences and insights implementing the grant and contracts. Thus, no statistical methodology (such as sample stratification) or estimation will be needed in the analysis of the interview data. Data analysis will involve tabulating coded data to identify themes and comparing perspectives across respondents. Detailed notes from telephone interviews will be translated into a structured format. To code the qualitative data for key themes and topics, the study team will first develop a coding scheme organized according to key research questions and topics and guided by the conceptual framework as well as constructs from the Consolidated Framework for Implementation Research (CFIR; www.cfirguide.org) on factors that affect implementation. A positive or negative flag will be applied to each segment of coded data to identify perceived facilitators of or barriers to implementation. This process will reduce the data into a manageable number of topics and themes for analysis (Ritchie and Spencer 2012). A small team will be trained to code the data using qualitative analysis software, such as NVivo. To ensure reliability across staff, all coders will code an initial set of documents and compare codes to identify and resolve discrepancies. As coding proceeds, the lead coders will conduct periodic reliability checks. Lastly, the study team will search the coded text to gauge consistency and identify themes.
Each type of data collection involves the universe, so we do not anticipate any specialized sampling procedures. Data collected from states and contractors need to be collected at the same point in time because the information requested is sensitive to policy changes that might occur at the federal level. As a result, the data cannot be collected in periodic cycles to reduce burden.
3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.
State survey
Response rates. The state survey will be administered via web to 54 “states” in spring 2019 to collect information on their apprenticeship programs. The study team will send the link to the web survey via email to the lead contact at each state and monitor progress throughout the fielding period. Reminders to complete the survey will be sent on a weekly basis to states that have not yet fully completed the survey instrument. These reminders will be sent first by email, then by mail, and eventually by phone until the desired response rate is met. Sample members who have not responded to the survey online after four weeks will receive a hardcopy mailing with a letter describing the effort, how to complete the survey through the web and a hardcopy version of the survey, should they prefer to complete it in that format. In addition, DOL will support the research team’s efforts to engage state contacts. This support, the dual completion options, and the development of a succinct survey instrument will facilitate a response rate of 90 percent. Similar approaches have been successful in previous evaluations conducted by Mathematica. On the Implementation of Title I/II Program Initiatives evaluation, for the U.S. Department of Education, a similar survey of state education agency administrators was fielded using the same methods and achieved a 100 percent response rate. On the evaluation of the Race to the Top and School Improvement Grants, also for the U.S. Department of Education, a survey of state education administrators achieved a response rate of 98 percent. Lastly, on the Study of the WIA Allocation Formula project for DOL, Mathematica fielded a one-hour survey of Title I-B administrators and received completed responses from 89 percent of 52 possible respondents.
Data reliability. Data from completed web surveys will be reviewed throughout the fielding period for accuracy and consistency. The use of the web mode allows for sophisticated skip logic and fills within the instrument, further improving the overall reliability of the data collected.
Semi-structured telephone interviews
Response rates. As described in section B.1, the study team plans to conduct telephone interviews with State Apprenticeship Expansions grant administrators and national industry intermediary and equity partner contractors. To ensure full cooperation, the study team will be flexible in accommodating respondents’ schedules. In addition, the protocols will be developed to collect only information that cannot be obtained from other sources, streamlining the topic areas that will need to be covered. With these approaches and DOL support, the study team anticipates a 100 percent response rate for administrator interviews. We received a 100 percent response rate on the interviews conducted with national industry and equity partner contractors as part of the pilot of those instruments.
Data reliability. Several well-proven strategies will be employed to ensure the reliability of the interview data collected during these telephone calls. First, the experienced group of study team members leading the calls, all of whom already have extensive experience with this data collection method, will be thoroughly trained in aspects particular to this study, including how to probe for additional details to help interpret responses to interview questions. Second, this training and the use of the protocols will ensure that the data are collected in a standardized way across respondents. Finally, all interview respondents will be assured that their responses will remain private; reports will never identify respondents by name, and any quotes will be devoid of identifying information, including organization name. This may encourage otherwise reluctant respondents to participate in interviews.
4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.
All procedures, instruments, and protocols to be used in the State Apprenticeship Expansion Grant Research Study were reviewed by content and methodological experts to ensure clarity and optimal ordering of the questions.
The state survey was pretested among a sample of four individuals to assess the clarity of the questions and identify possible modifications to either the wording of the questions or question order to optimize the quality of the data. After pretest participants completed the survey, members of the study team used a standard debriefing protocol to determine whether any words or questions were difficult to understand or answer. The results of these pretests and the protocols used during the debriefings are discussed in the pretest memo included with this submission.
To ensure that the State Apprenticeship Expansion grant interview protocol is used effectively as a guide that yields comprehensive and comparable data across the study sites, following the initial interviews, the study team will debrief on the effectiveness of the instruments and ensure that they are practical, given the amount of data to be collected and the amount of time allotted for each data collection activity. Adjustments to the protocol will be made as needed to ensure that the information necessary for the analysis is collected within the available time.
The national industry and equity contractor protocols were pretested in fall 2017 and revised based on those responses. The two protocols were piloted separately with nine national industry and four equity partner contractors.
5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
Consultations on the statistical, data collection, and analytic methods used in this study were part of the study design phase to ensure technical soundness. The individuals listed below were consulted on the statistical methods discussed in this submission to the Office of Management and Budget:
Mathematica Policy Research
Linda Rosenberg
Human Services Senior Researcher
(609) 936-2762
Samina Sattar
Human Services Senior Researcher
(609) 945-3358
Annalisa Mastri
Human Services Senior Researcher
(609) 275-2390
Alicia Harrington
Survey Researcher
(609) 945-3350
Urban Institute
Robert Lerman
Institute Fellow
(202) 885-3761
Social Policy Research Associates
Kristin Wolff
Senior Analyst
(510) 763-1499
Staff responsible for overseeing the collection and analysis of data are listed below.
Mathematica Policy Research
Linda Rosenberg
Human Services Senior Researcher
(609) 936-2762
Samina Sattar
Human Services Senior Researcher
(609) 945-3358
Alicia Harrington
Survey Researcher
(609) 945-3350
| File Type | application/msword |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |