Workforce innovation and opportunity act (WIOA) Implementation Study,
ICR Reference Number 181812-1290-001
April 2019
In this document, the Department of Labor (DOL) requests clearance from the Office of Management and Budget (OMB) under the Paperwork Reduction Act (PRA) for a new collection: a survey of state-level workforce staff associated with the Workforce Innovation and Opportunity Act (WIOA) Implementation Evaluation.
WIOA (Pub. L. 113-128), signed into law on July 22, 2014, aims to transform and modernize the public workforce system. Building on reforms under its predecessor, the Workforce Investment Act (WIA), WIOA’s overarching goals include improving the quality and outcomes of workforce services; ensuring that workforce services are evidence-based and guided by the needs of business and workers; and increasing access to the public workforce system, especially among individuals with significant barriers to employment. Many of the changes to the workforce system under WIOA went into effect on July 1, 2015, and additional changes will be implemented through June 30, 2018.
The WIOA implementation evaluation was funded by the DOL Chief Evaluation Office (CEO) in partnership with the Employment and Training Administration (ETA) with the goal of better understanding the implementation of WIOA, the variations in implementation across states and localities, and the need for further administrative guidance on regulations, technical assistance (TA), and policy. DOL contracted with Mathematica Policy Research in partnership with Social Policy Research to conduct the evaluation. The implementation evaluation and other evaluations of WIOA are authorized by Section 169 of WIOA which states that, “…the Secretary, through grants, contracts, or cooperative agreements, shall provide for the continuing evaluation of the programs and activities under this title, including those programs and activities carried out under this section” (Sec. 169(a) 2014). The study will include two data collection instruments:
1. In this package, DOL is seeking OMB approval for a survey of state-level workforce administrators in 50 states and the District of Columbia.
2. Additionally, a separate package received OMB approval (OMB control number 1290-0018; Information Collection Request number 201802-1290-001) for semi-structured interviews with state- and local-level workforce, adult education, and vocational rehabilitation staff, which will be administered during site visits to approximately 14 purposively-selected states and approximately 28 localities, over a six-month period.
In this request, we seek approval for the survey of state-level workforce administrators in 50 states and the District of Columbia. This survey will be conducted online with workforce staff from all 50 states and the District of Columbia. The survey of state-level workforce staff is necessary to enable a broad assessment of the progress of implementation across the country with various provisions of the law. A survey will allow the evaluation to collect systematic information on the breadth of experiences of diverse states, and create a national picture of implementation of WIOA. This information is not available through other sources in a comprehensive and systematic way.
While the U.S. economy has largely recovered from the Great Recession, many employers still struggle to find skilled workers, a problem that hampers the growth and productivity of the overall economy. Workers with low skills, disabilities, or other barriers to employment, the long-term unemployed, and others unattached to the workforce continue to face challenges finding and maintaining meaningful employment. WIOA seeks to address these challenges. To support implementation of the Act’s comprehensive reforms, DOL has provided extensive guidance and TA to state administrators, including draft and final rule documents, numerous Training and Employment Guidance Letters and Unemployment Insurance Program Letters, a “WIOA Implementation” TA website, and ongoing in-person and webinar trainings and sponsored peer-learning groups for state and local workforce stakeholders.
The WIOA implementation evaluation will examine the progress and variations in implementation across states and localities and assess their needs for further administrative guidance, regulations, technical assistance (TA), and policy changes. It will examine whether the implementation of WIOA is changing the public workforce system and core programs authorized under Title I (Adult, Dislocated Worker, and Youth programs) and the Employment Services program authorized under the Wagner-Peyser Act and amended by Title III. It also will assess whether implementation of the Act is contributing to greater integration with stakeholders in core programs authorized under Titles II (Adult Education and Literacy) and IV (Vocational Rehabilitation).
The evaluation will address the following research questions:
What changes under WIOA Titles I and III have been implemented, and what is still planned? How does implementation vary across states? What implementation challenges are states experiencing and how are they being addressed?
To what extent is WIOA’s vision for an integrated workforce system being achieved through state and local level synergies between Titles I and III and Titles II and IV stakeholders?
What additional TA, guidance, or policy changes would help states better administer core programs and provide guidance and oversight at the local level?
DOL will use the information produced by the evaluation to develop additional TA, guidance, and policies that can affect the workforce system and facilitate ongoing implementation of WIOA provisions.
Advanced technology will be used to conduct survey data collection and reduce burden on state-level workforce staff.
The survey will be conducted on the web to facilitate quick completion and submission. We will be using software from Confirmit (a leading platform for building and launching flexible, multi-mode surveys) to set up both the online and paper survey instruments. With Confirmit, we will be able to develop a single instrument and then deploy it across multiple modes. The web mode will be optimized for both computers and mobile/cellular access. If respondents complete and return a paper version, the system will allow data entry staff to record answers. Although respondents have the option to complete a paper version of the survey, we estimate that 80 percent of respondents will submit the survey electronically. Confirmit also provides instantaneous access to survey responses and data on who has completed the survey and when, which we will use to make real-time adjustments in how the survey is administered. Confirmit stores all “session” data so that survey respondents can visit and complete the questionnaire on their own schedule over as many sessions as they wish. Each state will be provided with a unique login username and password to access their survey.
This web survey approach will reduce the amount of interviewer time needed to assist with data collection and enable respondents to complete the survey on their own schedule and over multiple sessions. In addition, the burden will be minimized through web survey programming that includes skip pattern logic, response validity checks, respondent preload information, and consistency checks. As much information as possible will be preloaded into the web survey in order to reduce respondent burden. An example of this might be asking respondents to confirm the number of local workforce investment areas that the state has according to DOL records.
The web interface will be easy to navigate. The single survey instrument will be split into modules based on topic area to assist states who may want to have different staff complete different modules. To comply with Section 508 of the Rehabilitation Act, sample members who may have difficulty completing a web survey will be offered the option of completing the survey by telephone. The survey is expected to take three hours per state to complete.
The evaluation team will also use email to send survey invitations and reminders. Emails will contain a link to the survey website to reduce burden and error associated with manually typing a link into a web browser.
The WIOA Implementation Evaluation aims to collect survey data on a wide range of aspects of WIOA implementation in all states and the District of Columbia. Data on some specific topics related to implementation are, or will be, available from other sources, but the sample, mode of data collection, or research topic of those sources is too focused to be duplicative of this evaluation. Studies conducted by the Government Accountability Office have focused specifically on performance reporting, strategic planning, out-of-school youth, and Federal agency collaboration. The National Association of State Workforce Agencies has administered and plans to administer questionnaires with states on topics including workforce data systems, research capacity, staff capacity, accounting information technology systems, supplemental funding, and unemployment insurance tax. This evaluation’s survey is designed instead to provide a comprehensive picture of WIOA implementation through rich quantitative data from all 50 states and the District of Columbia, on a broad range of topics at the same point in time.
Some of the survey respondents may also participate in the site visit interviews (discussed in a separate clearance package that was previously submitted to and approved by OMB; see OMB No. 1290-0018) and may respond to questions on topics similar to topics on the survey. However, the survey questions will generally have closed-ended response options to facilitate a descriptive analysis of WIOA implementation across all states, while site visit questions are open-ended to prompt discussion and obtain rich and nuanced information on implementation experiences in specific contexts.
The survey will not be administered to any small businesses or entities. The survey respondents will all be staff employed by state workforce agencies.
The survey will be administered once to each respondent. Although the Federal investment of resources in WIOA implementation requires the systematic collection of comprehensive implementation data, the survey questions have been designed with a focus on prioritizing topics of primary interest to the evaluation and federal stakeholders, in order to minimize the burden on respondents. If these data are not collected, DOL and other stakeholders will not have the information necessary to understand how WIOA legislation is being implemented across the country and what additional TA, guidance, or policy changes will help states better administer WIOA’s provisions and provide guidance and oversight at the local level.
There are no special circumstances for the proposed data collection.
The 60-day notice to solicit public comments was published in the Federal Register on February 23, 2018 (83 FR 8110). A copy of this 60-day notice is included as supplemental information with this information collection request. No public comments were received.
Respondents will not receive payments.
The WIOA implementation study provides no express privacy assurance to respondents. The WIOA Implementation Evaluation will not collect or report any sensitive personally-identifiable information (PII). Nonetheless, the evaluation team will adhere to a set of strict policies that ensure information collected will be kept private to the extent permitted by law. All survey respondents will be notified on all survey forms, the survey website, and all survey reminders that the information that they provide will never be linked to their state or individual names, that their state or individual names will never be shared in any evaluation report, and that their state’s participation is voluntary. Any evaluation team staff conducting reminder calls or communicating with respondents will be trained in procedures to maintain privacy and will be able to describe the evaluation’s privacy policies in detail.
The WIOA Implementation Evaluation will not ask respondents to answer questions of a sensitive nature.
Table A.1 provides the annualized respondent hour and cost burden estimates for the survey data collection activities for which this package requests clearance. The evaluation is requesting clearance for a period of three years. Burden estimates are based on the evaluation team’s experience conducting similar data collections. The survey will be conducted with all 50 states and the District of Columbia. The table reflects total respondent numbers that have been annualized over the 3 years of the study. The survey is expected to take, on average, 3.0 hours per state.
Table A.1. Estimated Annualized Respondent Hour and Cost Burden
Type of Instrument |
Number of Respondents |
Number of Responses Per Respondent |
Total Number of Responses |
Average Burden Per Response (in hours) |
Estimated Burden Hours |
Average Hourly Wagea |
Annual Burden Costs |
Survey of Title I and Title III workforce programsb |
17 |
1 |
17 |
3 |
51 |
$34.07 |
$1,738 |
Total |
17 |
- |
17 |
- |
51 |
- |
$1,738 |
a Hourly wage reflects the May 2016 mean hourly wage estimate for “social and community service managers”, as reported by the U.S. Department of Labor, Bureau of Labor Statistics, Occupational Employment and Wage Estimates, 2016, Table 1 “National employment and wage data from the Occupational Employment Statistics survey by occupation, May 2016 (accessed from the following web site as of July 11, 2018: https://www.bls.gov/news.release/archives/ocwage_03312017.pdf.
b The survey will be conducted with all 50 states and the District of Columbia. The table reflects total respondent numbers that have been annualized over the 3 years of the study. The survey will take an average of 3 hours to complete per state.
There are no additional costs to respondents other than their time.
The total annualized cost to the federal government is $674,772. Costs result from the following two categories:
The annualized cost to the federal government for the evaluation contractor, Mathematica Policy Research (Contract Number: DOLQ129633249/DOL-OPS-16-U-00191), to carry out this evaluation is $662,418. The total cost of the implementation evaluation is $1,987,253 for three years. Therefore the annualized cost is $1,987,253 / 3 years = $662,418.
The annualized cost for federal technical staff to oversee the evaluation is $12,354. This is calculated by the following: an annual level of effort of 200 hours for one Washington, D.C.-based Federal GS-14 step 4 employee earning $61.77 per hour. (See Office of Personnel Management 2019 Hourly Salary Table at https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2019/DCB_h.pdfTherefore the annualized cost is 200 hours X $61.77 = $12,354.
The total annualized cost to the federal government is $662,418 + $12,354 = $674,772.
This is a new data collection.
The survey data will be analyzed using basic descriptive statistics to identify progress, challenges, trends, and variation in WIOA implementation across states. No complex analytical methods will be used. When the qualitative findings from the site visits are synthesized with the quantitative findings from the survey of 50 states and the District of Columbia, the evaluation team will be able to provide a national picture of WIOA implementation.
The evaluation team will develop several products to share the findings from the evaluation with key stakeholders at DOL (internal) as well as the workforce development community (external). Products that will include data collected from the survey include a final report, an issue brief, and a special topic paper, which will be made available on the DOL website. A final dataset of survey data will not be released to the public. An estimated project schedule is provided in Table A.2, with estimated dates of completion and publication of reports.
Table A.2 WIOA Implementation Evaluation project schedule
Action |
Estimated Date of Publication |
Issue briefs/special topic papers |
Beginning October 2019 |
Final report |
December 2020 |
The OMB approval number and expiration date will be displayed or cited on all forms completed as part of the data collection.
No exceptions are necessary for this information collection.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |