Survey-- State Program Director

DOL Employer Performance Measurement State Survey

Start of Block: Intro

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number.  Public reporting burden for this collection of information is estimated to average 45 minutes per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information.  The obligation to respond to this collection is voluntary. Send comments regarding the burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to [email protected] and reference the OMB Control Number 1290-XXXX.  Comments can also be mailed to:  U.S. Department of Labor, Chief Evaluation Office, 200 Constitution Ave., NW, S-2312, Washington, DC  20210. 

Note: Please do not return the completed survey to the email or mailing address. All responses should be submitted via the online survey platform.

Thank you for taking the time to participate in this survey of state Program Directors of core WIOA Titles: Title I (Adult, Dislocated Worker, Youth Programs), Title II (Adult Education Program), Title III (Wagner-Peyser Employment Services), and Title IV (Vocational Rehabilitation General Program, Vocational Rehabilitation Program for the Blind).  

The Urban Institute, a nonprofit, nonpartisan research organization, is under contract with the U.S. Department of Labor (DOL), Chief Evaluation Office (CEO) to study national pilot WIOA performance measures and alternative measures of the effectiveness of employer services that states have been testing as part of the implementation of WIOA sec. 116(b)(2)(A)(i)(VI), which required the Departments of Education and Labor to establish a primary indicator of performance for the effectiveness in serving employers (described in TEGL 10-16 Change 1, PM 17-2, and TAC 17-01). The effectiveness in serving employers indicator, “Indicator 6”, is a new approach for measuring performance, which serves as a shared outcome measure across all WIOA core programs. The Departments are piloting three approaches (“National Pilot Measures”) for Indicator 6. During the pilot period, States must select two of the three National Pilot Measures to report data that the Departments will use to assess a permanent indicator of performance. States may also voluntarily develop and report on additional “alternative measures”, or State-specific approaches to employer services measurement. The analysis will answer questions about the accuracy, reliability, and practicality of the National Pilot and alternative measures; provide insights about potential unintended consequences of the measures; and provide considerations for finalizing measures in the future.  

The survey will collect information from core WIOA Program Directors in the 50 States, the District of Columbia, Puerto Rico, Guam, and the U.S. Virgin Islands. Please note that, to minimize burden on survey respondents, we will refer to the 50 States, DC, territories, and outlying areas simply as “state” or “states” throughout.  For the purposes of this survey, WIOA Program Directors include the individuals at the state level who oversee core WIOA programs. We will be surveying all individuals that administer WIOA-funded programs in your state.  

We are sending this to you because you have been identified as a program director for one of WIOA’s core programs. Feel free to coordinate with appropriate members of your staff to complete the survey. These could include individuals who oversee the implementation of business services to employers (e.g., administrators and directors who oversee business relations) and staff who are involved with reporting on Indicator 6 as part of the annual WIOA data reporting process. You may choose after reviewing the PDF of the questions asked in this survey to refer this to another primary contact in your department who is better suited to coordinate a response across staff. The survey link you were sent can be shared within your department, but multiple individuals should not submit responses at the same time.

Privacy Statement

When you complete your survey, your answers will be visible only to staff at the Urban Institute who are working on the evaluation. Individual responses will be shared only within the study team, not with DOL. Because you may engage others from your department or program division in completing this survey, we are asking for contact information for the primary individual completing the survey in case we need to follow up with questions. Responses will be aggregated at the program level when reported and not attributed to individual states unless we first obtain permission.  

Responses will be used to select a small number of states for voluntary site visits. The purpose of the site visits is to provide the study team an opportunity to learn more about states’ experiences to inform the field and surface best practices. Responses will also be used to conduct analyses and provide reports to DOL. The reports are likely to be published to the DOL website and Urban Institute website and may be presented at conferences. The information collected through the survey and the site visits will not be used for site monitoring purposes.  

Your participation in the survey is voluntary. Higher response rates will improve the quality of the study and strengthen its recommendations. Although survey respondents will receive no direct benefits for participating, the workforce system as a whole will benefit from the survey analysis. The questions asked in this survey are not of a sensitive nature and are aimed at gathering information that can inform DOL’s decisions about performance measures moving forward.  

If you have any questions, please email [email protected] or call our survey help line at (202) 261-XXXX.  

Please complete the survey by XXXX.

Page Break

Q1.1 Please choose if you would like to continue to the survey.

• Continue to the survey

• Decline to take the survey

Skip To: End of Block If Continue = Decline to take the survey

Page Break

Instructions:

The survey asks questions about the administration and performance measurement of the WIOA Title I Adult Program, Title I Dislocated Worker Program, Title I Youth Program, Title II Adult Education Program, Title III Wagner-Peyser Employment Service, Title IV Vocational Rehabilitation Program (general), and Title IV Vocational Rehabilitation Program (services for the blind). Recognizing that each state may configure responsibilities for each of these titles differently, we have designed the survey into modules. At the beginning of the survey, you will indicate which Title(s)/Program(s) you oversee, and then you will only receive questions related to those Titles/Programs. The second half of the survey will ask questions about measuring the effectiveness of employer services. 

This web-based survey can be saved and completed over several sessions. Simply click the "Next Question" button at the bottom of the page to save your work. Upon reentering the survey, you will be routed to the last completed question. You may also navigate to previous questions to review or change your answers if necessary. You can consult with others in your agency to complete the survey responses that pertain to the program(s) you administer. If you oversee a program in part but not in its entirety, you can also share your user name and password with colleagues involved in program administration to complete the survey. 

The survey is divided into the following sections:  

A.  Scope and Background 

B.  Services for or Engagement of Employers 

C.  National Pilot and Alternative Measures 

D.  Past Measures

End of Block: Intro

Start of Block: A. Scope and Background

Q2.1 Section A. Scope and Background

Q2.2 Please fill in the following information.

• Your name (first and last): ________________________________________________

• Your agency: ________________________________________________

• Your agency department or division: ________________________________________________

• Your state government position title: ________________________________________________

• Your email address: ________________________________________________

• Total years in current position (indicate combination of months and years as a decimal): ________________________________________________

• Total years in the field: _________________________________________________________

Page Break

Q2.3 For each program you oversee, please indicate which program(s) are responsible for providing services to employers. “Services to employers” in this survey refers to recruitment services, education and training services, strategic planning and partnership services, and other employer services.

Please respond “yes” only if services are provided to employers through the program.

If you are responding on behalf of the program director in your state, please indicate which programs the director has responsibility for administering.

	Yes	No
Title I Adult Program
Title I Dislocated Worker Program
Title I Youth Program
Title II Adult Education Program
Title III Wagner-Peyser Employment Services
Title IV Vocational Rehabilitation (general)
Title IV Vocational Rehabilitation Program (for the blind) - only select "yes" if this is a separate division

If respondent selects no for all, route to the end of the survey. Please email Research Assistant Ian Hecker, [email protected], on the Urban Institute study team at [email protected], with your name and contact information so that we can identify why you received this survey in error.

We have questions for you about the employer services offered, the configuration of those services, and ways your state is measuring the effectiveness of those employer services (performance measures) for each program. As a reminder, we will refer to the 50 States, DC, territories, and outlying areas simply as “state” or “states” throughout.   

Please click on the section that you wish to start with on the next screen.

End of Block: A. Scope and Background

Start of Block: B. Services for or Engagement of Employers - Title X Program (e.g. Title I Adult Program)

Q3.1 Section B: Services for or Engagement of Employers - Title X Program (e.g. Title I Adult Program)  

In the following few questions, please indicate which types of services are currently provided for employers through the Title X Program (e.g. Title I Adult Program).  

Note: The definitions for each service type are available by hovering your mouse over the purple text. The full list of services and definitions can be found in the printable PDF file here.

Page Break

Display This Question if respondent has responsibility for overseeing more than one program

Q3.2 Are all services provided to employers through the Title X Program (e.g. Title I Adult Program) the same as another program you oversee? Select "no" if you have not yet responded to this section of the survey, or if the services to employers differ across programs you administer.

• Yes

• No

Page Break

Display This Question:

If EmpSvcs Same – as another program name in 2.3, it will appear in the list below

Q3.3 For which program(s) are the Title X Program (e.g. Title I Adult Program)'s services to employers the same? Select all that apply.

TitleI_DW = 1 in Q 2.3

• Title I Dislocated Worker Program

TitleI_Youth = 1 in Q 2.3

• Title I Youth Program

TitleII_Adult Education = 1 in Q 2.3

• Title II Adult Education Program

TitleIII_Wagner-Peyser = 1 in Q 2.3

• Title III Wagner-Peyser Program

TitleIV_Vocational Rehab (general) = 1 in Q 2.3

• Title IV Vocational Rehab Program (general)

TitleIV_Vocational Rehab program (for the blind) = 1 in Q 2.3

• Title IV Vocational Rehab Program (for the blind)

[If respondent selects that the services offered to employers for a program they have already completed responses for are the same, SKIP to end of module.]

Page Break

Q3.4 Which of the following recruitment services are currently provided for employers through the Title X Program (e.g. Title I Adult Program)? Please select one response for each row.

	Service not offered	Service offered statewide	Service offered, but varies by region/locality	Don’t know	Prefer not to answer
Workforce recruitment assistance
Accessing untapped labor pools
Working with businesses to offer competitive and integrated employment for people with disabilities

Page Break

Q3.5 Which of the following education and training services are currently provided for employers through the Title X Program (e.g. Title I Adult Program)? Please select one response for each row.

	Service not offered	Service offered statewide	Service offered, but varies by region/locality	Don’t know	Prefer not to answer
Training services
Incumbent worker training services for current employees
Sector-based or industry-wide training
Assistance with registered apprenticeship
Providing training services to improve the skills of jobseekers via adult basic education and skills training (reading, math, writing, employability, or technology skills at the high school level or less) for workers, including English language instruction
Providing resources for employers that hire individuals with disabilities for work-based learning, recruitment/onboarding, training, and retention

Page Break

Q3.6 Which of the following strategic planning and partnership services are currently provided for employers through the Title X Program (e.g. Title I Adult Program)? Please select one response for each row.

	Service not offered	Service offered statewide	Service offered, but varies by region/locality	Don’t know	Prefer not to answer
Engagement in strategic planning/economic development
Building relationships with workforce intermediaries, such as nonprofit intermediaries and service providers
Supporting the development of working relationships between State vocational rehabilitation agencies, their community partners, and employers

Page Break

Q3.7 Which of the following other business services are currently provided for employers through the Title X Program (e.g. Title I Adult Program)? Please select one response for each row.

	Service not offered	Service offered statewide	Service offered, but varies by region/locality	Don’t know	Prefer not to answer
Employer information and support services
Rapid response/business downsizing assistance
Planning layoff response
Providing support to employers on workplace accommodations, assistive technology, and facilities and workplace access
Assisting employers with utilizing available financial support for accommodating individuals with disabilities
Assistance with fostering a culture of workforce diversity
Providing information and resources on compliance with ADA laws and regulations
Other service (please describe or select "Prefer not to answer"):
Other service (please describe or select "Prefer not to answer"):
Other service (please describe or select "Prefer not to answer"):

Page Break

Q3.8 Does the Title X Program (e.g. Title I Adult Program) target particular industries or sectors to provide employer services?

• No

• Yes

• Don't know

• Prefer not to answer

Page Break

Display This Question:

If Target Sects - T-I Adult = Yes

Q3.9 Which industries or sectors has your program targeted for Title X Program (e.g. Title I Adult Program) services over the last 3 years? The North American Industry Classification System (NAICS) sector code is provided for reference, though all states/programs may not utilize the NAICS system for this purpose. 

	No	Yes	Don't know
Agriculture, Forestry, Fishing and Hunting (NAICS sector 11)
Mining, Quarrying, and Oil and Gas Extraction (NAICS sector 21)
Utilities (NAICS sector 22)
Construction (NAICS sector 23)
Manufacturing (NAICS sectors 31-33)
Wholesale Trade (NAICS sector 42)
Retail Trade (NAICS sectors 44-45)
Transportation and Warehousing (NAICS sectors 48-49)
Information (NAICS sector 51)
Finance and Insurance (NAICS sector 52)
Real Estate and Rental and Leasing (NAICS sector 53)
Professional, Scientific, and Technical Services (NAICS sector 54)
Management of Companies and Enterprises (NAICS sector 55)
Administrative and Support and Waste Management and Remediation Services (NAICS sector 56)
Educational Services (NAICS sector 61)
Health Care and Social Assistance (NAICS sector 62)
Arts, Entertainment, and Recreation (NAICS sector 71)
Accommodation and Food Services (NAICS sector 72)
Other Services (except Public Administration) (NAICS sector 81)
Public Administration (NAICS sector 92)

Q3.10 To what extent is the Title X Program (e.g. Title I Adult Program) utilizing each of the following strategies to improve services to and engagement of employers? Please make one selection for each row.

	Not a strategy	Planned strategy	Newer strategy (implemented in the last 3 years)	Long-term strategy (in place for 3 or more years)	Don’t know	Prefer not to answer
Build communications/collaborations among all agencies that offer business and education services
Establish point of contact for employer services across WIOA programs
Develop a consistent communication/marketing plan for all services to employers
Train staff to meet multiple employer needs and offer multiple services
Build systems that help track and maintain repeat employer customers and/or employer engagement (e.g. Customer Relationship Management systems)
Increase outreach to larger number of businesses/employers
Obtain monetary support or donated equipment for training programs
Create long-term partnerships that result in ongoing internship, apprenticeship, or other on-the-job training (OJT) opportunities
Create long-term partnerships that result in ongoing unsubsidized employment opportunities
Hold programs and staff accountable for specific performance goals associated with serving employers
Other (please describe or select "Prefer not to answer"):

Page Break

Q3.11 Are employer services for the Title X program (e.g. Title I Adult Program) provided in coordination with other core WIOA programs, as part of a state strategy for integrated employer service delivery? “Coordinating with another program” could involve administering employer services through a blended funding stream or through a separate cross-agency unit that provides employer services.

• Yes

• No

Q3.12 With which of the following program(s) are you coordinating to provide services to employers through the Title X program (e.g. Title I Adult Program)? For each program you oversee in the left column, select the name of the other program(s) you are coordinating with to carry out employer service delivery in the right column.

WIOA programs you oversee	Coordinating with the following programs to provide services to employers
Title X program (e.g. Title I Adult Program)	[Select from other Title I, II, III, and IV programs]

You have completed "Section B. Services for or Engagement of Employers - Title X Program (e.g. Title I Adult Program)."    Please click "Next Question" to begin the next section or navigate to the Table of Contents using the button below.

Table of contents

End of Block: B. Services for or Engagement of Employers

Q4.1 C. National Pilot and Alternative Measures

The Departments of Education and Labor are piloting three measures (“national pilot measures”) to assess how effective states are in providing services to employers.

The National Pilot Measures are:

• Approach 1 - Retention with the same employer - Percentage of participants with wage records who exit and were employed by the same employer in the second and fourth quarters after exit.

• Approach 2 - Repeat Business Customers - Percentage of employers who have used WIOA core program services more than once during the last three reporting periods; and

• Approach 3 - Employer Penetration Rate - Percentage of employers using WIOA core program services out of all employers in the State.

The Governor of each state may also establish alternative measure(s) relating to services for employers. Data on measures for the effectiveness in serving employers indicator (the national pilot and alternative measures) selected during the pilot phase are reported annually to DOL/Ed.  We want to understand more about the measures being used by your state across programs.

Q4.2 To the best of your knowledge, which of the following national pilot measures is your state currently collecting data for?

National Pilot Measure Name	No	Yes	Don't know	Prefer not to answer
Repeat business customers
Retention with the same employer
Employer penetration rate

Page Break

Display This Question:

If National Pilot Measures = Repeat business customers [ Yes ]

Q4.3 Please indicate the reasons your state is piloting [MEASURE X, e.g. repeat business customers].   Select one response per row.

	N/A	Yes	Don't know	Prefer not to answer
Local/regional agencies have the capacity to collect the data for this measure.
This measure is aligned with the goals of our employer services efforts.
The measure is easy to implement.
The data needed to implement this measure is available in our state on a consistent, statewide basis.
We use the information to provide technical assistance and training to our local workforce agencies for continuous quality improvement.
This is a measure that was in use prior to WIOA, and we use the information to track progress over time.
We are only using the measure to meet DOL reporting requirements.
Other (please describe or select "prefer not to answer"):

Page Break

Display This Question:

If National Pilot Measures = Repeat business customers [No]

Q4.4 Please indicate the reasons your state is not piloting the measure [MEASURE X, e.g. repeat business customers].

	N/A	Yes	Don't know	Prefer not to answer
We tried something like this before, and it didn’t work well for our state.
We were only required to pilot two of the measures.
We identified alternative measure(s) that we thought better reflected the goals of our efforts to serve businesses.
We were concerned about the capacity of local/regional agencies to collect the data needed for this measure.
We were concerned implementing this measure would result in a negative view of our employer services efforts: please explain.
This measure seems contradictory to the goals of our employer services efforts: please explain.
We were concerned implementing this measure would create harmful incentives/disincentives.
The data needed to implement this measure are not currently available in our state on a consistent, statewide basis.
We were concerned about the costs of implementing this measure.
Other (please describe or select "Prefer not to answer"):

Q4.5 In addition to selecting and reporting on two of three national pilot measures to measure WIOA core programs’ effectiveness in serving employers, states also have the option to develop up to two additional state-specific approaches, at the Governor’s discretion, which are reported annually along with the other effectiveness in serving employers indicators. Is your state currently using one or more alternative measures to assess its effectiveness in serving employers?

• No

• Yes, one alternative measure

• Yes, two alternative measures: ________________________________________________

• Don’t know

• Prefer not to answer

Page Break

Display This Question:

If Alternative Measures = Yes, one alternative measure

Or Alternative Measures = Yes, two alternative measures

Q4.6 Please provide the specific definition (or your best approximation of the wording) for each alternative measure. In addition, please describe in 1-3 sentences the data sources used to track this measure, defining any acronyms used in your description.

	Name of Measure	Definition	Data Source
Alternative Measure 1
If applicable Alternative Measure 2

Page Break

Display This Question:

If Alternative Measure Definitions Alternative Measure 1 - Name of Measure Is Not Empty

Q4.7 Please indicate the reasons your state is piloting [Alternative Measure 1].

	N/A	Yes	Don't know	Prefer not to answer
Local/regional agencies have the capacity to collect the data for this measure.
This measure is aligned with the goals of our employer services efforts
The measure is easy to implement.
The data needed to implement this measure is available in our state on a consistent, statewide basis.
We use the information to provide technical assistance and training to our local workforce agencies for continuous quality improvement
This is an existing a measure that was in use prior to WIOA, and we use the information to track progress over time.
We are only using the measure to meet DOL reporting requirements
Other (please describe or select "Prefer not to answer"):

Page Break

Display This Question:

If Alternative Measure Definitions Alternative Measure 2 - Name of Measure Is Not Empty, repeat scale for each question

Page Break

Display This Question:

If National Pilot Measures [Yes] (Count) >= 1

Or Alternative Measures = Yes, one alternative measure

Or Alternative Measures = Yes, more than one alternative measure (please indicate number of alternative measures, up to 2):

Q4.8 Please indicate whether you agree or disagree that the measures your state is piloting accurately capture the quality of employer services provided.

	Strongly disagree 1	Disagree 2	Neutral 3	Agree 4	Strongly agree 5	Don’t Know	Prefer Not to Answer
National Pilot Measures = Repeat business customers [Yes ] The measure repeat business customers accurately captures the quality of employer services provided.
National Pilot Measures = Retention with the same employer [ Yes ] The measure retention with the same employer accurately captures the quality of employer services provided.
National Pilot Measures = Employer penetration rate [ Yes ] The measure employer penetration rate accurately captures the quality of employer services provided.
If Alternative Measure Definitions Alternative Measure 1 - Name of Measure Is Not Empty The measure [Alternative Measure 1] accurately captures the quality of employer services provided.
If Alternative Measure Definitions Alternative Measure 2 - Name of Measure Is Not Empty The measure [Alternative Measure 2] accurately captures the quality of employer services provided.

Page Break

Q4.9 Please indicate if you agree or disagree that the measures your state is piloting accurately capture whether employer services provided by the workforce system are achieving desired outcomes.

	Strongly disagree 1	Disagree 2	Neutral 3	Agree 4	Strongly agree 5		Don’t Know		Prefer not to answer
National Pilot Measures = Repeat business customers [ Yes ] The measure repeat business customers accurately captures whether the services provided to employers by the state workforce system are achieving desired outcomes.
National Pilot Measures = Retention with the same employer [ Yes ] The measure retention with the same employer accurately captures whether the services provided to employers by the state workforce system are achieving desired outcomes.
National Pilot Measures = Employer penetration rate [ Yes ] The measure employer penetration rate accurately captures whether the services provided to employers by the state workforce system are achieving desired outcomes.
If Alternative Measure Definitions Alternative Measure 1 - Name of Measure Is Not Empty The measure [Alternative Measure 1] accurately captures whether the services provided to employers by the state workforce system are achieving desired outcomes.
If Alternative Measure Definitions Alternative Measure 2 - Name of Measure Is Not Empty The measure [Alternative Measure 2] accurately captures whether the services provided to employers by the state workforce system are achieving desired outcomes.

Now we want to ask some questions about how the measures your state is piloting reflect your program(s) ability to assess the impact of employer services provided, and key limitations of the measures (if applicable).

Q4.10 What (if anything) does the measure [MEASURE X, e.g. repeat business customers] tell your program(s) about its effectiveness in serving employers? How does your team use that information? Please type your response in the textbox below. 

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

Q4.11 What (if anything) are the key limitations of the measure [MEASURE X, e.g. repeat business customers]? Please type your response in the textbox below. 

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

[Repeat 4.10 and 4.11 for each additional National Pilot Measure selected]

Display This Question:

If Alternative Measures = Yes, one or more alternative measures

Q4.12 What (if anything) does the measure [Alternative Measure 1] tell your program(s) about its effectiveness in serving employers? How does your team use that information? Please type your response in the textbox below. 

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

Display This Question:

If Alternative Measure Definitions Alternative Measure 1 - Name of Measure Is Not Empty

Q4.13 What (if anything) are the key limitations of the measure [Alternative Measure 1]? Please type your response in the textbox below. 

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

[Repeat 4.12 and 4.13 for each alternative measure selected in Q.4.5]

Q4.14 The Departments of Labor and Education have issued guidance on the implementation of the common core measures that recommends that states centralize the coordination of collection and reporting of shared data into a single agency to report on the national pilot measures and any alternative measures. (See TEGL 10-16, Change 1, PM 17-2,and TAC 17-01).

Do your program(s) sit within an agency responsible for annually reporting data directly to DOL or Ed on the effectiveness in serving employers indicators?

• Yes

• No

• Don’t know

• Prefer not to answer

[If no, SKIP to Q4.23]

Display “Scenario A” Question Block

If Respondent answers “yes” to Q4.14

[Scenario A Question Block—for respondents answering “yes” to Q4.14]

Q4.15 Is your agency involved in collecting and compiling data across core programs and submitting a coordinated response to DOL or Ed on the employer services performance measures?

• Yes

• No, my program sits within an agency that reports data directly to DOL or Ed on the employer service measures, but that agency does not submit a coordinated response across core programs

• Don’t know

• Prefer not to answer

If yes or no, continue to next question. If don’t know or prefer not to answer, SKIP to Q4.27.

Q4.16 Which core WIOA programs are contributing data for [Measure X, e.g. repeat business customers]? Repeat for each measure.

	Yes	No
Title I Adult Program
Title I Dislocated Worker Program
Title I Youth Program
Title II Adult Education Program
Title III Wagner-Peyser Employment Services
Title IV Vocational Rehabilitation (general)
Title IV Vocational Rehabilitation Program (for the blind)

If yes in Q4.15, continue to Q4.17. If no in Q4.15, SKIP to Q4.19

Q4.17 On a scale of 1 to 5, how would you classify the ability of your agency to aggregate information on the measure [MEASURE X, e.g. repeat business customers] across programs into a statewide figure?

• Very Poor 1

• Poor 2

• Adequate 3

• Strong 4

• Very Strong 5

• Don't know

• Prefer not to answer

Q4.18 What are one or two key challenges related to capturing and reporting data across programs on the measure [MEASURE X, e.g. repeat business customers]? Challenges might include duplicative data, availability of data, etc. Please type your response in the text box below.

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

Q4.19 For each measure your state is currently piloting, are local areas or regions required to collect data for the measure?

[Will carry forward only names of measures currently piloted by the state]	No	Yes	Not Applicable-Single Area State	Don't Know	Prefer not to answer
Repeat business customers
Retention with the same employer
Employer penetration rate
[Alternative Measure 1]
[Alternative Measure 2]

If yes for any in Q4.19 continue to Q4.20 for each measure for which respondent indicated “yes”. If no for any, proceed to question 4.19B. If “not applicable/don’t know/prefer not to answer”, SKIP to Q 4.27 at end of module.

Q4.19B If no for any, please explain why your local areas are not providing data on the measure(s).

[TEXTBOX]

If no for all, SKIP to Q 4.27

Q4.20 Which core WIOA programs are involved in collecting local or regional data and reporting it to your agency for the purposes of annual reporting on Measure X?

Repeat for each measure.

	Yes	No
Title I Adult Program
Title I Dislocated Worker Program
Title I Youth Program
Title II Adult Education Program
Title III Wagner-Peyser Employment Services
Title IV Vocational Rehabilitation (general)
Title IV Vocational Rehabilitation Program (for the blind)

Display These Questions:

If yes for any in Q 4.19

Q4.21 On a scale of 1 to 5, how would you classify the ability of your agency to aggregate the information on Measure X across regions/localities into a statewide figure? Repeat for each measure.

• Very Poor 1

• Poor 2

• Adequate 3

• Strong 4

• Very Strong 5

• Don't know

• Prefer not to answer

Q4.22 What are one or two key challenges related to capturing and reporting data across regions/localities on Measure X? Please type your response in the text box below. Challenges might include duplicative data, availability of data, etc. Repeat for each measure.

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

[End of scenario A. Respondents move to question 4.27]

Display “Scenario B Question Block”

If Respondent answered “no” to Q4.14

[Scenario B Question Block—For respondents answering “no” to Q4.14]

Q4.23 Which of the following describe your state's data collection and reporting strategy for the national pilot and alternative measures (Indicator 6)? Choose the one best response.

• My program sits within an agency that reports data on employer services measures to another state agency. That agency reports these data to DOL or Ed.

• My program does not report data on the measures (to another state agency, DOL or Ed) [SKIP to Q4.27]

• Other (please specify): ________________________________________________ [SKIP to Q4.27]

Display This Question:

If “My program sits within an agency that reports data on employer services measures to another state agency. That agency reports these data to DOL or Ed.”

Q4.24 Provide the name of the agency with annual reporting responsibility to DOL/Ed on the effectiveness in serving employers indicators.

[TEXTBOX] (open response)

Q4.25 Are any of the programs you oversee involved in collecting and reporting data to send to another state agency on the employer services performance measures outcomes?

• Yes

• No [SKIP to Q4.27]

• Don’t know [SKIP to Q4.27]

• Prefer Not to Answer [SKIP to Q4.27]

Display This Question:

If Yes to Question 4.25

Q4.26 Which core WIOA programs you oversee are contributing data for [Measure X]? Select yes if data is being collected for the measure.

Repeat for each measure.

	Yes	No
[Pipe in data for programs respondent oversees]
[Pipe in data for programs respondent oversees]
[Pipe in data for programs respondent oversees]

Q4.27 Do you have any suggestions for measures outside of the pilot measures that should be used in addition to or in the place of current measures your state is using for the purposes of annual reporting to the Departments on the effectiveness of employer services? Please type your response in the text box below. 

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

You have completed "Section C. National Pilot and Alternative Measures."    Please click "Next Question" to begin the next section or navigate to the Table of Contents using the button below.

End of Block: C. National Pilot and Alternative Measures - Submit Response for This Section

You have completed "Section C. National Pilot and Alternative Measures."    Please click "Next Question" to begin the next section or navigate to the Table of Contents using the button below.

End of Block: C. National Pilot and Alternative Measures - Submit 1 Response for This Section

Start of Block: D. Past Measures - Title X Program (e.g. Title I Adult Program)

Q5.1 D. Past Measures - Title X Program (e.g. Title I Adult Program) The next block of questions asks about whether the Title X Program (e.g. Title I Adult Program) has a history of using other measures to understand their effectiveness in serving employers, outside the scope of the national pilot measures and alternative measures.

Display This Question if respondent is responsible for overseeing multiple programs

Q5.2 Are all measures used in the past for the Title X Program (e.g. Title I Adult Program) the same as another Title I program for which you have already responded on this survey? Select "no" if you have not yet responded to Section D for another program or if the past measures differ between programs.

• Yes

• No

Page Break

Display This Question:

If Past Meas Same =Yes move to 5.3, if not, move to 5.4

Q5.3 For which program(s) are the Title X Program (e.g. Title I Adult Program)'s past measures the same? Select all that apply.

• [List of all Core WIOA programs here]

• [List of all Core WIOA programs here]

Q5.4 Has the Title X Program (e.g. Title I Adult Program) ever used the American Customer Satisfaction Index (ACSI) to measure employer satisfaction? 

• No

• Yes, we used the ACSI in the past

• Yes, we used the ACSI in the past and still use it today to measure employer satisfaction

• Don’t know

• Prefer not to answer

SKIP To: Q5.9 If ASCI Used Ever - T-I Adult = No

SKIP To: Q5.9 If ASCI Used Ever - T-I Adult = Don’t know

SKIP To: Q5.9 If ASCI Used Ever - T-I Adult = Prefer not to answer

Page Break

Q5.5 Did the survey using the American Customer Satisfaction Index (ACSI) give you the kind of employer satisfaction information you thought it would for the Title X Program (e.g. Title I Adult Program) ?

• No

• Yes

• Don't know

• Prefer not to answer

SKIP To: Q5.8 If ACSI Info - T-I Adult = No

SKIP To: Q5.8 If ACSI Info - T-I Adult = Don't know

SKIP To: Q5.8 If ACSI Info - T-I Adult = Prefer not to answer

Page Break

Q5.6 Please indicate whether you agree or disagree with the following statement: “The employer survey using the ACSI was a useful measure of the quality of employer services provided” for the Title X Program (e.g. Title I Adult Program)?  

• Strongly disagree 1

• Disagree 2

• Neutral 3

• Agree 4

• Strongly agree 5

• Don't know

• Prefer not to answer

Page Break

Q5.7 How did the Title X Program (e.g. Title I Adult Program) use the employer customer satisfaction information from the ACSI? Please type your response in the textbox below. 

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

Page Break

Q5.8 What were the key limitations of information from the ACSI for the Title X Program (e.g. Title I Adult Program)? Please type your response in the textbox below. 

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

Page Break

Q5.9 Has the Title X Program (e.g. Title I Adult Program) used any other indicators in the past to report on or measure business satisfaction to assess your program’s effectiveness in serving employers?

• No

• Yes, we have used other measure(s) in the past (please indicate how many, up to 5) ________________________________________________

• Don't know

• Prefer not to answer

Page Break

Display This Question:

If = Yes, we have used other measure(s) in the past (please indicate how many, up to 5)

Q5.10 Please provide the name of the measure(s) the Title X Program (e.g. Title I Adult Program) has used in the past and the specific definitions.  In addition, please describe in 1-3 sentences the data sources used to track this measure, defining any acronyms used in your description

	Name of past measures, excluding the ACSI	Definition (or your best approximation)	Data source
Past Measure 1
If applicable Past Measure 2
If applicable Past Measure 3
If applicable Past Measure 4
If applicable Past Measure 5

Page Break

Display This Question:

If Past Meas Defs - T-I Adult Alternative Measure 1 - Name of other measures, excluding the ACSI Is Not Empty

Q5.11 Please indicate whether you agree or disagree with the following statement: "[PAST MEASURE 1] was a useful measure of the quality of employer services provided” for the Title X Program (e.g. Title I Adult Program).

• Strongly disagree 1

• Disagree 2

• Neutral 3

• Agree 4

• Strongly agree 5

• Don't know

• Prefer not to answer

[Repeat for each past measure, up to 5]

Display This Question:

If Past Meas 1 - T-I Adult = Disagree 2

Or Past Meas 1 - T-I Adult = Strongly disagree 1

Or Past Meas 2 - T-I Adult = Disagree 2

Or Past Meas 2 - T-I Adult = Strongly disagree 1

Or Past Meas 3 - T-I Adult = Disagree 2

Or Past Meas 3 - T-I Adult = Strongly disagree 1

Or Past Meas 4 - T-I Adult = Disagree 2

Or Past Meas 4 - T-I Adult = Strongly disagree 1

Or Past Meas 5 - T-I Adult = Disagree 2

Or Past Meas 5 - T-I Adult = Strongly disagree 1

Q5.12 You disagreed that one or more measures accurately capture the quality of employer services provided for Title X Program (e.g. Title I Adult Program). Please describe why you think this. 

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

[Repeat for each past measure, up to 5]

You have completed "Section D. Past Measures - Title X Program (e.g. Title I Adult Program)."    Please click "Next Question" to begin the next section or navigate to the Table of Contents using the button below.

Table of contents

End of Block: D. Past Measures - Title X Program (e.g. Title I Adult Program)

Start of Block: Closing

[All respondents answer this question]

Q5.13 We welcome you to share any additional thoughts or comments about measuring services to employers provided by WIOA programs or anything else related to this survey. Please type your response in the text box below, then click "Next Question" to submit your survey.

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

Next Question

38