Key Informant Interview Protocol-- State and Local Staff Collecting Performance Data

OMB Control No.: 1290-XXXX

Expiration Date: XX/XX/2021

Analysis of Employer Performance Measurement Approaches

State and Local Performance Data Staff [DRAFT]

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. Public reporting burden for this collection of information is estimated to average 45 minutes per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. The obligation to respond to this collection is voluntary. Send comments regarding the burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to [email protected] and reference the OMB Control Number 1290-XXXX. Comments can also be mailed to: U.S. Department of Labor, Chief Evaluation Office, 200 Constitution Ave., NW, S-2312, Washington, DC 20210. Note: Please do not return the completed interview guide to the email or mailing address.

Introduction/Purpose of the Study

Thank you for agreeing to participate in this interview today.

My name is _______________, and I’m a researcher from [The Urban Institute, a non- profit research organization located in Washington, DC; George Washington University, located in Washington, DC; Capital Research Corporation, located in Arlington, VA]. With me today is [name and affiliation]. We’re here today because the Urban Institute and its partners are under contract with the U.S. Department of Labor (DOL), Chief Evaluation Office (CEO) to conduct an analysis of new performance measurement approaches for measuring the effectiveness of employer services in the public workforce system. As part of the study, we are conducting site visit interviews with state and local WIOA administrators and other workforce development staff in several different states, including staff that work closely with workforce system data. The information collected during these interviews will inform our reporting to DOL on the use, validity, reliability, and practicality of measuring the effectiveness of services provided to employers in the public workforce system.

We are interested in learning about your experiences related to providing services to employers in your [state/area] and measurement approaches, including the national pilot measures and other measures of employer services. This interview should last about 45 minutes.

Privacy Statement

Before we begin the interview, we want to let you know that, although we will take notes during this interview, when we write our reports and discuss our findings, information from all the people we speak with will be compiled and summarized without identifying individual respondents. We also want to make it clear that your participation is strictly voluntary, and you may choose to not answer questions at any time.

Finally, to help us accurately capture the information you share, we would like to record this interview. The recording is just a back-up for our notes and will be kept within our small research team. The interview recording will be deleted once we have developed a full set of notes from the interview. Is it okay with you if we record the interview? [GET CONSENT] Do you have any questions before we begin?

[BEGIN RECORDING IF CONSENT GRANTED]

1. BACKGROUND INFORMATION

1. Obtain the following information for each respondent [Fill in as much information in advance of the interview as possible, and request a business card from each interviewee]:

   1. First and last name

   2. Name of agency

   3. Job title

   4. Number of years with the agency _____; Number of years in the current position _____

   5. Cities/counties included in local workforce area (if applicable)

   6. Description of job duties [Probe for job duties related to the data collection, management, and analysis of workforce program data across core programs of WIOA including, Title I Adult Program, Title I Dislocated Worker Program, Title I Youth Program, Title II Adult Education Program, Title III Wagner-Peyser Employment Service, Title IV Vocational Rehabilitation (for the blind), and Title IV Vocational Rehabilitation (general). Or, if previous interviews have indicated certain parameters of service, probe based on those interviews.]

   7. Are you responsible for data collection and management for any other state or federal programs? Please describe.

[Note to interviewer: Based on their answer regarding the scope of their job responsibilities, state again the purpose of this interview to focus on employer services measures for core WIOA programs.]

2. BACKGROUND ON DATA COLLECTION/ANALYSIS STRUCTURE IN THE STATE

[For this question, we are trying to get an overview of the systems, actors, and processes that govern your collection of data and analysis methods for the effectiveness in serving employers indicators (national pilot measures) across each core WIOA program. We will get into more detail later in the interview.]

2. Please describe to the best of your knowledge the overall data collection and data analysis approach for each of these WIOA programs and the combination across those programs:

   1. WIOA Title I Adult Program,

   2. Title I Dislocated Worker Program,

   3. Title I Youth Program,

   4. Title II Adult Education Program,

   5. Title III Wagner-Peyser Employment Service,

   6. Title IV Vocational Rehabilitation (general), and

   7. Title IV Vocational Rehabilitation (for the blind).

3. THOUGHTS ABOUT MEASURING EMPLOYER SERVICES

3. From your perspective, what are the most important things to try to capture when measuring business’s interactions with the public workforce system in your state/area? What suggestions do you have for how to best measure the amount and effectiveness of the services?

4. From your perspective, what is the best way to handle performance measurement when reporting shared outcomes across the core WIOA programs

5. Do you feel that measuring the services provided to employers is beneficial to your agency? Why or why not?

4. DETAILS OF EMPLOYER SERVICES NATIONAL PILOT MEASURES DATA COLLECTION AND MANAGEMENT

Now we’d like to ask you some questions about the national pilot measures states are currently required to implement. As you probably know, states are required to collect data for at least two of three national pilot measures: 1) retention with the same employer, 2) employer penetration rate, and 3) repeat business customers. States are also encouraged to implement alternate measures to represent the level and quality of services to employers. If not familiar, provide a brief overview of the pilot study, the three measures, and option for alternate measures.

6. [If interviewer knows which measures were selected (e.g., from other interviews or from state reporting data), then ask about each measure. If not, then ask.] Which national pilot measures or alternate measures has your state/area implemented?

7. We’ll ask you a set of questions for each pilot measure and alternative measure being implemented.

   1. What data do you use for measurement of [indicator]?

[Probe on whether this varies by Title I Adult Program, Title I Dislocated Worker Program, Title I Youth Program, Title II Adult Education Program, Title III Wagner-Peyser Employment Service, Title IV Vocational Rehabilitation (general), and Title IV Vocational Rehabilitation ( for the blind). Or, if previous interviews have indicated certain parameters of knowledge about cross-program approaches, probe based on those parameters.]

2. How frequently are those data reported to you/collected?

[Probe on whether this varies

- by WIOA program

-by particular measure]

3. How are the necessary data transmitted to you?

[Probe on whether this varies

- by WIOA program

-by particular measure]

4. What are the limitations of the data, if any?

[Probe on whether this varies

- by WIOA program

-by particular measure]

8. What types of management information systems does your state use to maintain the national pilot measures data? Are these the same systems the state uses to maintain data for the alternative measures? Are data collected from front line staff working with businesses transferred to a separate statistical database or system for analysis? How old are your data systems? How long does it take to add new fields or configure fields differently? Probe on whether this varies by WIOA program or by particular measure]

9. What staff resources are needed to collect, maintain, and analyze the pilot performance measures? The alternative measures? [Probe for the following as needed, by WIOA program and measure]

   1. Who collects the data?

   2. Who stores/maintains the data? [Probe for across departments]

   3. Who analyzes the data?

   4. Is your agency experiencing, currently or in the past, any resource limitations that make collection, maintenance, or analysis of the national pilot or alternative measures data difficult?

[Probe on whether there are any particular challenges combining data across the WIOA programs, if data is being combined and reported in a singular way.]

10. What steps/procedures are taken to ensure data quality and consistency? Probe for the following as needed:]

    1. How do you assure that the correct data are provided for each measure?

11. How have you been instructed to handle counts when agencies or AJCs provide services to the same employer under multiple WIOA programs, how do you handle counting those services? Please describe, including if you have been instructed to handle counts in a specific way.

    1. How do you screen for duplicate entries?

[Probe for methods to prevent duplication across services reported and multiple establishments affiliated with the same business.]

12. What is the time lag between data collection on the measures and data reporting?

13. Do you handle reporting on federally-mandated measures differently from how you would handle a state or locally mandated measure? Please explain.

Closing

Is there anything you would like to mention that you haven’t had a chance to say before we end the interview?

Thank you so much for your participation. Please feel free to reach out to us should you have any questions or comments in the future.

4

Analysis of Employer Performance Measurement Approaches

State and Local Performance Data Staff Interview Guide