Att 2.5 State Title III Protocol

OMB No. 1290-XXXX
Exp. Date XX/XX/2021

2.5 PROTOCOL FOR STATE TITLE III STAFF
Instructions to site visitors: Bullets below each question represent probes for important details;
depending on how well you are doing on time, try to make sure respondents touch on each of
these issues. Text in brackets [ ] should be tailored based on the state or local area. Sections in
gray are the high priority sections for this respondent.

Introduction
Thank you for taking the time to meet with me today. My name is [NAME] and I work for
[Mathematica Policy Research/Social Policy Research Associates]. I am part of an independent
research team conducting a study of WIOA implementation on behalf of the U.S. Department of
Labor. Thank you very much for agreeing to participate in this conversation.
The purpose of this three-year study is to help DOL understand how states are implementing
changes to the core workforce programs authorized under Titles I and III of WIOA, as well as
how those changes are influenced by integration with other partners such as adult education,
vocational rehabilitation, and TANF. To help us better understand WIOA’s implementation, we
are conducting site visits to 14 states, including two local areas in each state, to learn about their
experiences. The information you share will help us understand the range of experiences that
states have had, including key successes and challenges.
We have about [duration] for our conversation. I want to let you know that all interview data will
be reported in the aggregate and your name will never be mentioned in any report that we write,
though we might use quotes from your interview to illustrate findings, without using your name.
I would also like to record our conversation so that I can listen to it later as I complete my notes,
and I will not share the recording with anyone outside of the research team. Being part of this
discussion is voluntary, and you may choose not to answer a question if you wish, or to pause the
recording at any time. Do you have any objections to being part of this interview or to my
recording our discussion?
Okay, I’m going to turn on the audio recorder now.

Respondent Background
1. [If not already known for each person participating in the interview]
a. What is your official title and role?
b. What are your main responsibilities related to implementation of WIOA?

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection
displays a valid OMB control number. Public reporting burden for this collection of information is estimated to average 1 hour and 20 minutes
per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and
completing and reviewing the collection of information. The obligation to respond to this collection is voluntary. Send comments regarding the
burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to
[email protected] and reference the 0MB Control Number 1290-XXXX. Comments can also be mailed to: U.S. Department of
Labor, Chief Evaluation Office, 200 Constitution Ave., NW, S-2312, Washington, DC 20210. Note: Please do not return the completed interview
guide to the email or mailing address.

WIOA IMPLEMENTATION EVALUATION

OMB SUPPORTING STATEMENT: PROTOCOL 2.5

A. Governance and Planning
1. State Plan Development
1. Describe your state’s general planning process for developing its WIOA plan. What
entity and staff members led the process?
a. How and to what extent were other entities involved?
2. Did this process differ from previous state plan development efforts?
a. Was the process longer than with previous plans?
b. Were certain partners more involved? Less involved?
c. How did any changes to the process from prior planning efforts improve the final
plan?
d. What were you able to achieve that had not been possible before?
e. What did you lose, if anything?
3. What, if any, significant challenges or promising practices were encountered?
a.

Did the plan timeline work with other required planning for core programs, state
planning, etc.?

4. How useful has your WIOA state planning process been for:
a. Developing and following through on your state’s workforce strategy?
b. Establishing, developing or improving your partnerships with other core WIOA
partners?
c. Establishing, developing or improving your partnership with other required AJC
partners?
5. Do you have any lessons to share related to the development of the combined/unified
plan on what has worked well or has been challenging?
6. Have there been changes in federal or state funding for the workforce system that have
affected your plan or how the plan will be implemented?
a. If there have been significant cuts to funding, how are you adjusting the system in
response to these cuts?
b.

What will be the impacts on infrastructure, programming, and service delivery?

2. State Workforce Development Board Role and Operations
1. [If Title I and III are in same agency] How is Title III/Employment Services represented
on the State Board? How does this impact policy for ES programs and staff?
2. Has the role of the state board changed under WIOA? If so, how and why?
3. What has been the board’s role in making WIOA’s changes to the AJC system?
a. Developing the AJC certification criteria?

2.70

WIOA IMPLEMENTATION EVALUATION

OMB SUPPORTING STATEMENT: PROTOCOL 2.5

b. Developing procedures?
c.

Developing local MOU/RSA/IFA policy?

d. Developing integrated services & customer-centered policies & procedures?
4. What has been the board’s role in developing stronger partnerships with core and
required partners?
a. Was the board involved in developing policies related to coordination of services?
b. How was the state partner MOU revised under WIOA?
c. How does the new MOU differ from the last one developed under WIOA? E.g.,
more focus on continuous improvement? On coordinated services?
d. What was the process used to develop the MOU?
5. What has been the board’s role in strengthening employer engagement across the system,
including sector strategies, career pathways, and work-based learning?
6. Is the board operating more strategically? Why or why not?
7. Is the board operating more efficiently? Why or why not?
8. Has the role of the chair changed?
9. Has the role of various members of the board changed?
a. Core partners?
b.

Employers?

c. TANF?
d. Other required partners on the board?
e. Other non-required partners on the board?
10. Is there more (or different training) provided for board members as compared to the
board under the last year of WIA? What kind of training?

3. Local Area Designations and Local Plan Development
1. What challenges did your local staff experience in developing local plans? What worked
well?
2. Are any changes planned for the next local planning process?

4. Regional Designations and Regional Plans
1. Do you think the regional designations are appropriate? Why or why not?
a. Do you think they are useful for improving service delivery & employer
engagement?

6. Local Board Employer Engagement; Sector Strategies
1. What policies and actions has the state taken to support employer engagement?

2.71

WIOA IMPLEMENTATION EVALUATION

OMB SUPPORTING STATEMENT: PROTOCOL 2.5

2. What changes in the workforce system (particularly at local and regional level) are you
seeing as a result of these efforts?
3. What policies and strategies has your state adopted to increase the use of sector strategies
in the workforce system?
a. Did your efforts in this area precede WIOA?
b. If yes, what new strategies or activities are being pursued under WIOA:
i. Involving new partners and collaborations?
ii. Expanding services and training features?
iii. Targeting different populations, such as youth, individuals with
disabilities, limited English and/or basic skills?
4. Has the focus on developing and expanding sector strategies been at the state, local, or
regional levels?
5. What funding sources are being used to develop and support your sector strategy
programs and initiatives (e.g., local formula funds, WIOA state set-aside funds, Title II,
III, or IV funding)?

B. AJC System
2. Infrastructure Funding and MOUs
1. In addition to required infrastructure costs, what were the most common additional costs
(career services and shared operating/service costs) your agency’s programs in local areas
included in their One-Stop operating budgets?
a. Why did they include those components?
b. Were any required by the state? Which ones?
2. What challenges came up for your agency’s programs in local areas in the process of
developing local One-Stop operating budgets?
3. Have you taken any steps to facilitate negotiation of One-Stop operating budgets for your
local agencies (e.g. state-level partner meetings to facilitate state participation)?
4. Were there any local areas in which consensus on IFAs was not achieved because your
local agency did not agree? What were their objections and how, if at all, were you
involved in addressing them?
5. Have there been any changes to partner participation in AJCs because of WIOA’s
resource sharing requirements? If so, describe.
6. Have there been any changes in your program’s local representation on boards because of
WIOA’s resource sharing requirements? If so, describe.
7. What TA has been provided by the state on resource sharing?
a. What successes or challenges have been experienced related to this TA?

2.72

WIOA IMPLEMENTATION EVALUATION

OMB SUPPORTING STATEMENT: PROTOCOL 2.5

8. What aspects of USDOL’s guidance on resource sharing could have been more helpful?
a. What additional guidance is needed from USDOL?
9. Have you made use of any of USDOL’s TA resources or activities related to local
partners developing MOUs or One-Stop operating budgets (including the sample
RSA/MOU developed by DOL)?
a. Did you use other TA?

3. One-Stop Operator
1. How did procurement of One-Stop operators change in your local areas after WIOA?
2. What challenges did local boards face in carrying out this requirement?
a. Did any local boards opt not to competitively procure their operator?
b. Were there any issues with boards unnecessarily restricting competition (e.g., too
little time for the bid; too little funding; restrictions on type/location of bidders).
3. How did you contribute, if at all, to any state-level guidance on the role of the operator?
a. How did this guidance differ from what was required under WIA?

4. Collocation at AJCs
1. Has your program’s collocation at AJCs changed since WIOA implementation? If so,
how and why?
3. What guidance has the state provided to local areas on collocation of partners at AJCs
under WIOA?
a. Did that guidance change from what was provided under WIA?
b. Any challenges/lessons learned related to your guidance?
4. What TA has the state provided on collocation?
5. Is any additional guidance needed from USDOL on collocation?

C. Overall AJC Streamlining & Accessible Services
1. Accessibility
1. Please describe any new state policies and actions that promote a more seamless,
accessible, and customer-focused workforce service delivery network.
a. Has the state encouraged or required any changes to AJC customer intake
processes? What changes did this involve at the state level?
b. Has the state encouraged or required a shared case management system? Which
programs or partners can access the case management system?
c. Has the state encouraged/required functional staffing arrangements at AJCs? How
common are such arrangements?

2.73

WIOA IMPLEMENTATION EVALUATION

OMB SUPPORTING STATEMENT: PROTOCOL 2.5

d. Can you describe any other new state policies and actions that promote a more
seamless, accessible, and customer-focused workforce service delivery network?
2. To what extent has the state emphasized customer or human-centered design as a key
method for making your state’s One-Stop career center system more customer-focused?
a. If you have emphasized this design approach, what primary problems or goals did
you want to address through this process?
b. Please describe some of the promising practices in human or customer-centered
design at the local level.
3. Please describe any state-wide efforts to improve the accessibility of One-Stops for
individuals with disabilities (e.g. policies, additional funding, or training).

2. Co-enrollment
1. How often are your program’s customers co-enrolled in other core partner programs?
Other required partners?
a. Do any new requirements particularly facilitate or impede co-enrollment?
2. Have any local areas experienced changes in co-enrollment rates since WIOA?
3. Has the state provided any TA or guidance on co-enrollment or referral processes?
a. Are you encouraging co-enrollment and if so, how?
b. How effective was this TA or guidance?
4. Does the state have any plans for additional guidance or TA?

3. Use of shared MIS
1. Which management information system (MIS) does your program use? For what
purposes?
i. Case management?
ii. Participant tracking?
iii. Performance reporting?
iv. Fiscal reporting?
2. Can other partners use this system too? If so:
a. Can all partners view data? Are there specific fields that are restricted?
b. Can all partners enter data into the system? Are there specific fields that only
some partners can enter?
c. Can all partners revise data? Are there specific fields than only some partners can
modify?
3. How do you ensure that the data being entered into these systems by different partners is
accurate?

2.74

WIOA IMPLEMENTATION EVALUATION

OMB SUPPORTING STATEMENT: PROTOCOL 2.5

4. How are you protecting the security of the data being entered and stored?
5. Do you have access to other MISs that are used by other partners in the One-Stop? How
do you use those systems, and what are the challenges with accessing partner systems?
6. How important is it for seamless services that partner programs use the same MIS?

D. Performance Accountability
2. Common Measures
1. Have you faced any challenges implementing the WIOA performance measures?
2. Which two approaches to measuring the effectiveness of employer services did you
choose to implement?
a. Why did you opt for those two approaches?
b. What metrics are you using?
c. What challenges have you faced in implementing these measures?
d. Do you have thoughts on other measures that might better capture the
effectiveness of serving employers?
3. Are you developing any state-specific measures that you will be reporting on? If so,
which ones and why?

3. Reporting and Evaluation
1. To what extent can your state track individual customers’ participation across more than
one partner via a common identifier, especially across the core partners?
a. Which partners use this identifier?
b. What efforts are currently underway to expand capacity to be able to do this—
especially across the core partner programs?
2. Have you faced any challenges in accessing/providing required partner data for
reporting?
a. If so, which types of data and from which partners?
b. How have you resolved or attempted to resolve these challenges?
3. What are your state’s plans and goals for evaluation?
a. How much capacity does your state have to carry out the required evaluation
component of WIOA?
b. If you do not have the capacity to conduct these activities internally, have you had
experience contracting with outside evaluators in the past? How did this
experience shape your evaluation plans under WIOA?

2.75

WIOA IMPLEMENTATION EVALUATION

OMB SUPPORTING STATEMENT: PROTOCOL 2.5

E. Services for Job Seekers and Employers
1. Has WIOA changed your state’s approach to providing services to businesses?
2. What strategies have you implemented to increase the use of work-based and job-driven
training (such as OJT, apprenticeships, transitional training, internships)?
a. What work-based and job-driven training models are most commonly used in
your state?
3. Please describe your state’s efforts to serve more customers with barriers, including
disabled customers, customers with low basic skills, or other focus populations.

G. Overall Questions
1. Partnerships
1. How would you assess the strength of your state’s relationships among core partners?
Required partners?
a. Has this changed because of WIOA?
b. If not, why do you think WIOA’s focus on enhanced partnerships has not affected
your partnerships?
2. Are there sufficient systems in place to ensure regular and effective communication
between partners?
3. What have been the key challenges and successes, if any, related to increasing
collaboration across partners?
a. Are there differences in the extent of collaboration with your core partners versus
other required partners, such as UI, SCSEP, Veterans programs, and TANF?
4. Overall, how integrated are workforce services now?
a. Has this changed because of WIOA?
b. What examples can you give that show this more integrated service delivery?
5. [If respondent feels workforce system is not integrated]: Why do you think WIOA’s
focus on enhanced partnerships has not affected the integration of services?

2. Systems Change
1. Overall, do you think the workforce system here has changed as a result of WIOA?
a.

If so, what are the major changes you can attribute to WIOA? E.g. employer
engagement, partnerships, services more streamlined, data sharing, others?

b. If not, do you feel systems had already begun to change prior to WIOA, or that
efforts under WIOA have not yet been effective in changing the system?
2. [If respondent doesn’t think WIOA has changed the system]: Do you feel systems had
already begun to change prior to WIOA, or that efforts under WIOA have not yet been
effective in changing the system?

2.76

WIOA IMPLEMENTATION EVALUATION

OMB SUPPORTING STATEMENT: PROTOCOL 2.5

3. What additional changes are planned or needed to transform your workforce system to
meet the goals of WIOA?
a. What changes would you like to see to make the workforce system better overall?

3. Guidance and Technical Assistance
1. Of the TA provided by DOL national and regional offices on WIOA implementation,
which assistance has been most helpful?
a. Which TA formats and types of materials have been most useful for you?
2. Are you satisfied with the level of TA DOL has provided on WIOA implementation?
3. Have you accessed the ION site to obtain TA materials or guidance on specific issues?
a. Which topics have you sought out guidance on?
4. In what areas would you like to receive additional TA?

Wrap-up
1. How would you describe your overall progress to date with WIOA implementation?
a. Where do you still have plans to make significant changes?
2. Beyond what we’ve discussed today, are there other areas you would like to highlight?
3. Are there any other areas of WIOA implementation our study should explore?

2.77

This page has been left blank for double-sided copying.