Evaluation of Employer Performance Measurement Approaches
ICR REF 201901-1290-001
Updated March 2019
Evaluation of Employer Performance Measurement Approaches
Part A: Supporting Statement for Request for OMB Approval Under the Paperwork Reduction Act
Overview
The U.S. Department of Labor (DOL) Chief Evaluation Office (CEO) is seeking Office of Management and Budget (OMB) approval to collect information from State and local public workforce system employees and partners, and to gather feedback from a group of U.S. employers, to inform the Analysis of Employer Performance Measurement Approaches study. The purpose of the study is to conduct a 36-month analysis of employer services measurement approaches and metrics, as well as their cross-State and cross-program applicability, with a goal of understanding and implementing a final indicator of performance. The study will explore and establish an understanding of employer services measurement and supplement the start-up of reporting by the States1 on the National Pilot measures. Key objectives of the study include: (1) developing and understanding how employer services are defined by the federal government, States, localities, and core Workforce Innovation and Opportunity Act (WIOA) programs and exploring options for developing a uniform definition of employer services; (2) identifying what measures exist for understanding employer services, key objectives of these measures, and possibilities for uniform implementation at the federal level; and (3) developing options for an evaluation design to assess the validity, reliability, and feasibility of proposed measures and alternative measures of effectiveness in serving employers. This ICR seeks clearance for: (1) a brief online survey of all State-level core WIOA program directors; (2) a brief qualitative employer information collection instrument (“information collection form”) to be offered to a group of employers identified in partnership with the National Association of State Workforce Agencies (NASWA) and DirectEmployers (DE); (3) protocol for interviews with State administrators of core WIOA programs in approximately 8 States; (4) protocol for interviews with local administrators of core WIOA programs in 8 local areas in a subset of 4 states; 5) protocol for interviews with State and local staff collecting performance data in a selection of approximately 8 States and 4 local areas within those states; (6) protocol for group interviews with American Job Center staff in 8 local areas in a subset of 4 states.
Circumstances Necessitating Collection of Information
When the Workforce Innovation and Opportunity Act (WIOA) was signed into law in July of 2014, several steps were taken within the law to advance the way the workforce system measures its programs through regular, integrated, and aligned performance accountability reporting. Within section 116 of WIOA, six primary indicators of performance were put into place: (1) employment in the second quarter after exit; (2) employment in the fourth quarter after exit; (3) median earnings in the second quarter after exit; (4) measurable skill gains; (5) credential attainment; and (6) measures of effectiveness in serving employers. The law established that the Secretaries of Labor and Education, through consultation with representatives outlined by the law, would establish one or more primary indicators of performance that indicate the effectiveness of the core WIOA programs (Titles I, II, III, and IV) in serving employers.2 Through town halls, workgroups, and questions posed through the notice of proposed rule-making, the Secretaries established three measures to be piloted by States: (1) an employee retention measure, (2) an employer penetration rate, and (3) a repeat business measure. States were required to implement two of the three pilot measures. Additionally, States were encouraged to implement and provide information on other, “alternative” measures being utilized to assess effectiveness in serving employers as part of the pilot.
It is necessary to conduct this new data collection to provide DOL and other stakeholders with key information to inform decision making about measures of effectiveness in serving employers. This data collection will help ensure that DOL meets the requirements of WIOA by supporting the implementation of appropriate, reliable, valid, and practical measures of the effectiveness of the core WIOA programs in serving employers through feedback from key stakeholders in the public workforce system as well as employers.
Purposes and Uses of the Data
Purpose
The overarching goal of this study is to assess the appropriateness, reliability, validity, and practicality of the national pilot measures, as well as alternative measures being used by States and localities, of effectiveness in serving employers to inform a set of options for measurement across the core programs of WIOA moving forward. Data collected as part of this study will enable us to conduct analyses of the landscape of employer services and measures and highlight variation across and within states. Key overarching research questions include the following:
How are “services to employers” operationalized within and across WIOA’s four core programs, and within and across states, regions, and local areas?
What are employer experiences with and perceptions of the public workforce system and performance measurement?
What pilot or additional measures are currently used, or have been used in the past or in other contexts, to understand engagement with employers by states, local areas, and federal agencies?
What are the options for identifying and defining measures of effectiveness in serving employers (including inputs, outputs, and outcomes)?
What are the options for an evaluation design to study the practicality, validity, and reliability of proposed measures to assess effectiveness in serving employers?
Three major data collection activities will be used to provide an understanding of employer services measurement and supplement the start-up of reporting by the States3 on the National Pilot measures:
State Program Director Survey. The brief survey of State-level WIOA program directors will collect information on which measures are being used by States, including national pilot measures and alternative measures, progress made in implementing those measures, and how those measures are being used beyond required federal reporting. The State Program Director Survey will allow for collection of similar information across all State agencies and offices that administer the core programs under WIOA, specifically, agencies administering the following: 1) Title I Adult Programs; 2) Title I dislocated worker programs; 3) Title I youth programs; 4) Title II adult education and literacy programs; 5) Title III Wagner Peyser Employment Service programs; 6) Title IV vocational rehabilitation programs for blind customers; and 7) Title IV vocational rehabilitation programs for other customers with disabilities.
Qualitative Employer Information Collection. The brief employer information collection form will be sent to a select group of employers and aims to surface potential issues in identifying appropriate measures by gaining the perspectives of a broad set of stakeholders that are knowledgeable about the workforce investment system. The employer information collection will gather perspectives and ideas from businesses that have experience working with the public workforce system, allowing the study team to explore key issues related to measuring employer services from the vantage point of businesses, key customers of the workforce system.
Perspectives will be gathered from two sources: DirectEmployers members and NASWA Business of the Year Award winners and nominees. DirectEmployers (DE) is a member-owned and managed association of 918 employers of Fortune 1000 companies that are mostly large federal contractors.4 Per Executive Order 112465 as amended, Section 503 of the Rehabilitation Act of 1973,6 and the Vietnam Era Veterans’ Readjustment Assistance Act of 1974 (VEVRAA),7federal contractors must demonstrate engagement with the public workforce system through mandatory job listing and positive recruitment. Additionally, information will be collected from the National Association of State Workforce Agencies (NASWA) Business of the Year Award Winners and Nominees, who are selected for this annual award based on demonstrated substantial engagement with the workforce system.
The purpose of this qualitative information collection is to get ideas from employers that have engaged with the workforce system to identify potential options for measurement and provide context on the employer perspective. Employer information collection results will be used to provide qualitative information that reflects a range of employer experiences and views, drawing from a convenience sample of employers more likely to have engaged with the workforce system. The results will not be used to make statistical conclusions or conduct inferences about how common specific employer perspectives or experiences are across all U.S. employers. Data collected from the employer information collection is not reflective of all employers in the United States. It is intentionally designed to conduct broad outreach and to solicit ideas for measurement. Most U.S. employers have no experience receiving services through the public workforce system; potential participants for this data collection are employers known to have an affiliation with the workforce system, who are more likely to access workforce system services.
We believe that even if our sample is not fully representative of all employers, our analyses will still support a greater understanding of WIOA program service delivery to employers that partner with the workforce system, including public and private (non-contractor) employers. Due to resource and time constraints, the number of interviews and/or group interviews we could conduct with employers is limited. Given this limitation, the population of employers we propose collecting feedback from is likely to be our best opportunity to collect a range of diverse perspectives from employers receiving workforce system services, which will be an important tool in informing our understanding of the accuracy, reliability, and practicality of the measures.
Survey and employer information collection responses and fieldwork data will be collected on a strictly voluntary basis, and individual responses will be kept private. Although the research team will encourage State and local employees and businesses to participate, said participation will not affect access to WIOA program funding or public workforce system services. Only the minimal amount of data needed for this study will be collected.
Key Informant Interviews and Group Interviews. The site visits to a selection of approximately 8 States are intended to allow a deeper understanding of why particular measures were selected, progress in implementing performance measures, and related challenges that will enable us to answer the study’s research questions about the validity, reliability, practicality and unintended consequences of implemented measures. This fieldwork will include semi-structured interviews and group interviews. The States will be selected based on the results of the survey and other study knowledge, to include a mix of locations in terms of geographic region, performance measures being used, WIOA program administration, and status of implementation.
Stakeholders to be interviewed include State workforce administrators (Titles I and III), adult education (Title II) administrators, vocational rehabilitation (Title IV) administrators, staff collecting performance data, and staff monitoring local performance. Roughly 8 local areas will be selected from 4 States selected for local-level fieldwork (two local areas each within a subset of four states). Stakeholders to be interviewed at the local level include staff overseeing Title I, II, and IV programs and individuals who coordinate with the State on administering Title III programs. We will also conduct group interviews with a mix of American Job Center (AJC) administrators and staff involved in providing employer services or collecting data related to employer services performance.
Uses of Data
Data collected from the survey, employer information collection, and key informant interviews and group interviews during site visits will be analyzed as part of an integrative, mixed methods design that will allow the research team to triangulate data across sources and build a multi-dimensional view of measures of public workforce system employer services.
Analysis based on survey, employer information collection, and key informant interviews and group interviews conducted during site visits will be used by DOL to understand 1) what services are being provided to employers through various service delivery models, 2) what performance measures are being used to track service delivery and gauge its effectiveness, and 3) the experiences of various stakeholders with the implementation of employer services measures. The comparative analysis of measures will yield a set of options for measuring the effectiveness of services to employers, including the implications of each approach and potential next steps for testing, evaluating, and improving approaches to measurement by federal agencies and State workforce systems.
A Federal Work Group (FWG) and Subject Matter Expert (SME) Panel will be convened to inform the study approach and our analysis of findings. Materials developed from the analysis of information collected are intended to reach multiple audiences, including:
DOL and other federal agencies
State and local workforce and education agencies and organizations
Researchers
Employers
Others interested in approaches to measuring the effectiveness of services to employers
Use of Improved Information Technology
Survey, qualitative employer information, and interview and group interview data collected during site visits will be collected efficiently and effectively.
Survey and Employer Information Collection. The State Survey and the Employer Information Collection will be fielded online using Qualtrics, a commercial software application for development and administration of online information collection. A major advantage of web-based information collection is the automatic tabulation of responses, which reduces the hours of staff time needed for response processing and eliminates the possibility of introducing human error into the data. Automated skip patterns embedded in online instruments also place less of a burden on respondents than manual skip patterns found in paper forms. Furthermore, the online instruments will allow respondents to pass sections or questions on to other staff members who may be better equipped to address particular topics, such as performance measures and data collection. Survey and employer information collection data will be stored on a private, secure drive that only research team members, who have signed privacy agreements, can access. We anticipate that one hundred percent of state survey and qualitative information collection responses will be submitted electronically using the online Qualtrics platform. However, reasonable accommodations will be made for respondents with disabilities who may wish to submit their responses by phone.
Key Informant and Group Interviews. All interviews conducted during site visits will be face-to-face; researchers may follow up with interviews by telephone if there is a need for further clarification after the visit or if a particular respondent wasn’t available for an interview during the time of the visit. While on site, interviews will be conducted by teams of two researchers, one who will guide the discussion and one who will primarily take notes on a laptop computer. Site visitors will also ask respondents for permission to record interviews with a digital recorder, which ensures full sets of notes can be developed at a later time if not all information discussed is captured via notes at the time. Electronic versions of key informant and group interview notes and digital recordings of interviews will be stored on a private, secure drive that can only be accessed by the research team. Respondents will not interface with any automated, electronic, mechanical, or other technological collection techniques or other forms of information technology during this portion of the proposed data collection.
Efforts to Identify Duplication
The information we propose to collect from State WIOA program administrators, a select group of U.S. employers, and other State and local workforce system entities is not otherwise available. No systematic, State-level survey or other qualitative assessment of employer services across WIOA programs and approaches used to measure the effectiveness of those services exists. The reporting data currently being collected from States as part of required WIOA reporting does not include information specific to what types of employer services are provided and how they are measured at the State or program level. Additionally, no information collection exists that asks a large number of employers about the types of information and services they have sought or received from the public workforce system while also getting employer feedback on potential measures of the effectiveness of those services. Finally, no site visits have been conducted to date that allowed for a deep dive into the administration and measurement of employer services at the State and local levels across WIOA programs in a variety of different States.
Collection of Data from Small Businesses
The State Program Director Survey and key informant interviews and group interviews conducted in a subset of States and local areas will not impact small businesses or other small entities. There may be small businesses among the group of employers that will be offered a chance to provide input via the employer information collection, although most will be large employers. However, employer information collection participation is voluntary, and will take 20 minutes or less to complete, with a 15-minute average. Only the minimal amount of data needed for this study will be collected.
Consequences of Less Frequent Data Collection
WIOA requires States to report key performance measure data collectively across Title I, II, III, and IV programs annually, including measures of effectiveness in serving employers. The law also requires the Secretaries of Labor and Education to establish one or more primary indicators of performance that indicate the effectiveness of the core WIOA programs in serving employers. No clear metric has emerged of success in providing services to employers. The proposed one-time (annual) data collection activities associated with this study will allow DOL to gain an understanding of the validity, reliability, practicality, and unintended consequences of national pilot and alternative measures of employer service effectiveness. Without that understanding, DOL will not be able to assess the appropriateness of measures of employer services and DOL’s ability to implement measures in accordance with statutory requirements under WIOA will be significantly constrained.
Special Circumstances of Data Collection
There are no special circumstances for the proposed data collection.
Federal Register Announcement and Consultation Outside the Agency
Federal Register Notice and Comments. Notification of this data collection was published in the Federal Register on March 6, 2018 (83 FR 9548). No public comments were received.
Consultation Outside the Agency. The research team consulted with the following:
members of the U.S. Department of Education (ED)
Subject Matter Expert (SME) Panel
State program directors of Title I, II, III, and IV programs
Employers who are DE members
Between August and December of 2018, the team engaged these stakeholders in reviewing the data collection instruments. The team tested the state survey (as described in Part B), employer information collection form, interview guides, and group interview guides to ensure that the data collection instruments (and each question) were clearly written and understandable to respondents, fully covered appropriate topics, asked questions appropriately, and offered respondents a complete listing of response categories for each closed-ended question. Fewer than 10 testers were involved in the testing of each instrument. Consultation with individuals outside DOL will ensure data collection instruments are accurate, clear, and capable of collecting the desired information.
Payments to Respondents
Respondents will not receive payments.
Assurance of Privacy
State survey, employer qualitative information collection, and key informant interview and group interview respondents will be assured that their specific, individual responses will be kept private and won’t be reported at the individual level with their names or positions attached. Steps will be taken, in accordance with the Urban Institute Institutional Review Board (IRB) guidelines, to offer respondents the assurance that the information they provide is considered private and will not be shared with anyone outside of the research team in a manner that would allow respondent identification, unless the research team is legally ordered otherwise. Findings from the survey will be presented at the aggregate or State level; and data from open-ended survey questions will be summarized across multiple respondents. Findings from the qualitative employer information collection will be reported without information that could potentially identify a specific respondent or firm. Findings from the key informant interviews and group interviews conducted during site visits will be presented at the State level, in order to provide detail and illustrative examples, but no individual respondents will be identified or quoted in any publication. Prior to collecting data, each respondent will be given the pertinent privacy information, an explanation of the nature of the study, and a description of the time necessary to participate. However, no binding guarantees of privacy will be offered, as privacy laws do not apply to this data collection. Please see the first pages of each data collection instrument for the respondent privacy statements imbedded in the informed consent procedures.
Sensitive Questions
There are no questions of a sensitive, personal, or private nature included in the survey, qualitative employer information collection form or the key informant interview/group interview guides.
Estimated Response Burden
The burden estimates displayed in table 1 below are annualized across three years. We estimated the universe of State Program Director Survey respondents to be 215, based on information from program director contact lists shared by OCTAE, RSA, and DOL. We assume one potential respondent per program director that administers a core WIOA program. Some program directors oversee more than one program, such as a Title I program director that oversees Title I Adult, Dislocated Worker, and Youth programming in their state. Individuals that oversee multiple programs will only complete one survey; therefore, they are only counted once in the respondent universe. We assume an 80 percent response rate for the State Program Director Survey, and a total of 172 state program director responses. The total number of responses is annualized across three years in Table 1.
For the qualitative employer information collection, we calculated the estimated number of responses based on DE’s current roster of individual contacts (10,214), plus 25 NASWA Business of the Year Award winners and nominees, for a total of 10,239 contacts. We expect 1,536 employer responses, based on the level of response on other DE surveys or similar surveys fielded to employers that partner with the workforce system. The total number of responses is annualized across three years in Table 1.
For the site visits to approximately 8 states and approximately 8 local areas, we assume a 100 percent participation rate for key informant interviews and group interviews, as we will seek confirmation of willingness to participate prior to conducting the fieldwork. We assume 32 State administrator respondents, 16 local administrator respondents, interviews with 16 State and local staff collecting performance data, and 40 American Job Center group interview participants. The total number of responses is annualized across three years in Table 1. Average hourly wage estimates were selected based on information found in the May 2017 National Industry-Specific Occupational Employment and Wage Estimates from DOL’s Bureau of Labor Statistics.8
Table 1: Estimated Annualized Respondent Hour and Cost Burden
Type of Instrument (Form/Activity) |
Number of Respondents |
Number of Responses Per Respondent |
Number of Responses |
Average Burden Per Response (in Hours) |
Estimated Burden Hours |
Average Hourly Wage |
Annual Burden Costs |
Survey-- State Program Director |
57 |
1 |
57 |
45/60 |
42.75 |
$43.76 |
$1,870.74 |
—Qualitative Employer Information Collection Form |
512 |
1 |
512 |
15/60 |
128 |
$31.84 |
$4,076.52 |
Key Informant Interview Protocol-- State Administrator |
11 |
1 |
11 |
1.00 |
11 |
$43.76 |
$481.36 |
Key Informant Interview Protocol-- Local Administrator |
5 |
1 |
5 |
1.00 |
4 |
$49.12 |
$245.60 |
Key Informant Interview Protocol-- State and Local Staff Collecting Performance Data |
5 |
1 |
5 |
45/60 |
4 |
$44.92 |
$168.45 |
Group Interview Protocol-- American Job Center Staff |
13 |
1 |
13 |
1.00 |
13 |
$28.18 |
|
Total |
603 |
-- |
-- |
-- |
203.5 |
-- |
$7,208.01
|
Cost to Respondents
There are no additional costs to respondents other than their time.
Cost to the Federal Government
The total annualized cost to the federal government is $87,272. Costs result from the following two categories:
The annualized cost to the federal government for the contractor to carry out this study is $75,192. The estimated annual cost includes research team staff hours for survey and employer information collection administration and fieldwork, as well as costs associated with the site visits.
The annualized cost for federal technical staff to oversee the study is $12,080. This is calculated by the following: an annual level of effort of 200 hours for one Washington, DC–based Federal GS-14 step 4 employee earning $60.40 per hour. (See Office of Personnel Management 2018 Hourly Salary Table at https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2018/DCB_h.pdf). Therefore, the annualized cost is 200 hours x $60.40 = $12,080.
The total annualized cost to the federal government is $87,272 ($75,192 + $12,080 = $87,272).
Reasons for Program Changes
This is a new data collection effort (OMB No. 1290-0NEW).
Publication Plans and Project Schedule
The aim of the study is to build a greater understanding among stakeholders about employer services offered and used, employer services measurement indicators previously or currently implemented, approaches to measurement across the core programs of WIOA, and options for assessing or analyzing employer measures and indicators. Target audiences for this information include DOL, ED, and other federal agencies working with employers; State and local agencies and organizations providing employer services; researchers studying employers’ interactions with the public workforce system; and employers themselves. The study will disseminate findings from the surveys and key informant interviews and group interviews conducted during site visits as follows:
2020: A research brief describing interim findings, including services offered to employers, implementation of measures and measurement, key challenges, and insights to inform future measurement
2020: A Final Report summarizing key findings from the study, including a comparative assessment of each of the national pilot measures and any identified emergent measures from the synthesis or scan, options for measuring employer services, measures that should be tested, steps that can be taken to improve performance measurement and policy, and potential statistical approaches to assessing performance
2020: A final Design Report, summarizing the design and implementation of the study and making recommendations for a potential evaluation of employer services performance measures
2020: A Briefing to DOL or the Federal Work Group (FWG) on the Design Report and the Final Report
2020: A Webinar co-hosted by NASWA summarizing key findings from the study, either for their members only or open to public
Approval to Not Display the Expiration Date for OMB Approval
The OMB approval number and expiration date will be displayed or cited on all forms completed as part of the data collection.
Exceptions to the Certification Statement
No exceptions are necessary for this information collection.
1 For purposes of this ICR supporting Statement, “State” and “States” refer to the fifty States, the District of Columbia, Puerto Rico, and U.S. territories and outlying areas.
2 WIOA sec. 116(b)(2)(A)(i)(VI) requires the Departments to establish a primary indicator of performance for effectiveness in serving employers.
3 For purposes of this ICR supporting Statement, “State” and “States” refer to the fifty States, the District of Columbia, Puerto Rico, and U.S. territories and outlying areas.
4 As of February 2019, over 95% of 918 DirectEmployers members are federal contractors
5 https://www.dol.gov/ofccp/regs/statutes/eo11246.htm
6 https://www.dol.gov/ofccp/regs/compliance/section503.htm
7 https://www.dol.gov/ofccp/regs/compliance/vevraa.htm
8 https://www.bls.gov/oes/current/oes_nat.htm
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |