Key Informant Interview Protocol-- Local Administrator

OMB Control No.: 1290-XXXX

Expiration Date: XX/XX/2021

Analysis of Employer Performance Measurement Approaches

Local WIOA Administrator Interview Guide [DRAFT]

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. Public reporting burden for this collection of information is estimated to average 60 minutes per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. The obligation to respond to this collection is voluntary. Send comments regarding the burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to [email protected] and reference the OMB Control Number 1290-XXXX. Comments can also be mailed to: U.S. Department of Labor, Chief Evaluation Office, 200 Constitution Ave., NW, S-2312, Washington, DC 20210. Note: Please do not return the completed interview guide to the email or mailing address.

Introduction/Purpose of the Study

Thank you for agreeing to participate in this interview today.

My name is _______________, and I’m a researcher from [The Urban Institute, a non- profit research organization located in Washington, DC; George Washington University, located in Washington, DC; Capital Research Corporation, located in Arlington, VA]. With me today is [name and affiliation]. We’re here today because the Urban Institute and its partners are under contract with the U.S. Department of Labor (DOL), Chief Evaluation Office (CEO) to conduct an analysis of new performance measurement approaches for measuring the effectiveness of employer services in the public workforce system. As part of the study, we are conducting site visit interviews with state and local WIOA administrators and other workforce development staff in several different states. The information collected during these interviews will inform our reporting to DOL on the use, validity, reliability, and practicality of measuring the effectiveness of services provided to employers in the public workforce system.

We are interested in learning about your experiences related to providing services to employers in your state and measurement approaches, including the National Pilot Measures and other measures of employer services. This interview should last about 60 minutes.

Privacy Statement

Before we begin the interview, we want to let you know that, although we will take notes during this interview, when we write our reports and discuss our findings, information from all the people we speak with will be compiled and summarized without identifying individual respondents. We also want to make it clear that your participation is strictly voluntary, and you may choose to not answer questions at any time.

Finally, to help us accurately capture the information you share, we would like to record this interview. The recording is just a back-up for our notes and will be kept within our small research team. The interview recording will be deleted once we have developed a full set of notes from the interview. Is it okay with you if we record the interview? [GET CONSENT] Do you have any questions before we begin?

[BEGIN RECORDING IF CONSENT GRANTED]

1. BACKGROUND INFORMATION

1. Obtain the following information for each respondent [Fill in as much information in advance of the interview as possible, and request a business card from each interviewee]:

   • First and last name

   • Official job title

   • Name of department/agency

   • WIOA programs you oversee in your role:

     1. [If respondent oversees multiple programs] Do the services provided to employers and the approach to data collection and measurement of employer services vary across programs? _______

   • Number of years with the agency ____; Number of years in position ______

   • Cities/counties included in local workforce area

   • Please describe in a few sentences the scope of your job responsibilities related to WIOA programs and any other programs you may be responsible for.

In this interview, we will be focused primarily on understanding your goals associated with providing services to employers under WIOA. [Note to interviewer: Hand to the administrator the fact sheet with the National Pilot Measures and definitions, If the state survey responses are available to the study team in advance of the site visit, also provide the list of National Pilot and alternative measures we understand them to be using based on the state survey responses .] Here is information for your reference during the interview on the National Pilot Measures. We will be referencing these measures periodically during the interview.

2. EMPLOYER SERVICES PROVIDED

2. Describe your department/program’s overall strategy or goals in providing services for employers in your area. Do you focus outreach activities and services on specific industry sectors? Why or why not? Please describe your approach across each WIOA program you oversee.

3. What types of outreach does your program conduct to let employers know about the information and services your program can provide? Please describe for each program you oversee.

4. About how many different employers did your program serve last year? Please provide an estimate for each program you oversee. If you would prefer to share that information with us following the interview by email or phone, please let us know. ____

   • What types of services did you provide to them?

   • What information and services do employers most frequently seek out?

   • To what extent do employers in your area participate on advisory boards, volunteer their time, or donate materials for training or other services?

5. Are there employer services you would like to provide or provide more of but can’t due to resource constraints? If so, please explain. Please describe for each program you oversee.

6. Are there services being requested that you cannot provide due to policy, regulatory, financial or other constraints? If so, please explain. Please describe for each program you oversee.

7. To what extent do you think your program’s strategies and experiences serving employers is

   • Similar to or different than other workforce programs/agencies in your state?

Please describe for each program you oversee.

3. NATIONAL PILOT

8. Are you familiar with the National Pilot Measures of employer services being implemented by all WIOA programs as required by DOL and Ed? [If so, proceed to question 10. If not familiar, provide brief overview of the National Pilot Measures, and optional alternative measures, and move to question 9]

9. [For respondents not familiar with the measures, provide reference to TEGL 10-16, PM 17-2, and TAC 17-01, and explain that for the purposes of this discussion, we are not able to answer detailed questions about how the measures are constructed or calculated due to time constraints.]

   • Retention with the same employer: Is it important that job candidates that are placed with an employer are still employed with the same firm 12 months after they were hired? Why or why not.

   • Repeat business customers: Is it important to have repeat business customers, where the same firm is requesting services from the workforce system multiple times? Why or why not.

   • Employer penetration rate: Is it important to understand the percentage of all employers in your locality or state using workforce program services? Why or why not.

[SKIP to question 17]

10. Which National Pilot Measures or alternative measures has your program implemented? Please describe for each program you oversee, if measures vary across programs. [Probe for if they have used the same measure(s) or something similar previously]

11. How/when did your agency learn about these new measures?

12. What information, guidance, or training was offered or provided to the staff in your program to help you implement the new measures? Please describe for each program you oversee. [Probe on how guidance/ information was shared. If they received written memos or guides about how to collect data or enter data, probe for how these measures might be different than what they have done before, whether they were offered any on-line or in-person training.]

13. When did your program begin collecting data for these measures? What have been the challenges in collecting the necessary data for the pilot measures? Please describe for each program you oversee. [Probe for any confusion about the definitions or counts; follow up service vs. new service, etc.] Where can your agency get assistance on clarifications for collecting data for the measures or reporting the data to the state?]

14. Does your agency provide services to the same employer under multiple WIOA titles? If so, how have you been instructed (or what guidance was given) on handling counts when your agency provides services to the same employer under multiple WIOA titles? To your knowledge, how are data combined?

15. Do the measures seem like a fair representation of the amount and quality of the services your local workforce system provides to employers? Why or why not?

16. Have the pilot or alternative measures contributed to your program’s strategic planning or policies in any way? If so, please explain for each program you oversee.

17. What challenges have you experienced, if any, related to data collection requirements under WIOA?

4. ADDITIONAL THOUGHTS ABOUT MEASURING EMPLOYER SERVICES

18. Does your program have any measures of employer services that are locally driven or different from the state measures of employer services? If yes, why and what are they? Please describe for each program you oversee.

19. From your perspective, what are the most important things to try to capture when measuring business’s interactions with the public workforce system in your area? What suggestions do you have for how to best measure the services provided and effectiveness of those services?

20. Do you feel that measuring services to employers is beneficial to your program? Why or why not? Please describe for each program you oversee.

Closing

Is there anything you would like to mention that you haven’t had a chance to say before we end the interview?

Thank you so much for your participation. Please feel free to reach out to us should you have any questions or comments in the future.

4

Analysis of Employer Performance Measurement Approaches

Local WIOA Administrator Interview Guide