Download:
pdf |
pdfOMB No. 1290-XXXX
Exp. Date XX/XX/2021
3.1 PROTOCOL FOR LOCAL WORKFORCE DEVELOPMENT
BOARD STAFF
Instructions to site visitors: Bullets below each question represent probes for important details;
depending on how well you are doing on time, try to make sure respondents touch on each of
these issues. Text in brackets [ ] should be tailored based on the state or local area. Sections in
gray are the high priority sections for this respondent.
Introduction
Thank you for taking the time to meet with me today. My name is [NAME] and I work for
[Mathematica Policy Research/Social Policy Research Associates]. I am part of an independent
research team conducting a study of WIOA implementation on behalf of the U.S. Department of
Labor. Thank you very much for agreeing to participate in this conversation.
The purpose of this three-year study is to help DOL understand how states are implementing
changes to the core workforce programs authorized under Titles I and III of WIOA, as well as
how those changes are influenced by integration with other partners such as adult education,
vocational rehabilitation, and TANF. To help us better understand WIOA’s implementation, we
are conducting site visits to 14 states, including two local areas in each state, to learn about their
experiences. The information you share will help us understand the range of experiences that
states have had, including key successes and challenges.
We have about [duration] for our conversation. I want to let you know that all interview data will
be reported in the aggregate and your name will never be mentioned in any report that we write,
though we might use quotes from your interview to illustrate findings, without using your name.
I would also like to record our conversation so that I can listen to it later as I complete my notes,
and I will not share the recording with anyone outside of the research team. Being part of this
discussion is voluntary, and you may choose not to answer a question if you wish, or to pause the
recording at any time. Do you have any objections to being part of this interview or to my
recording our discussion?
Okay, I’m going to turn on the audio recorder now.
Respondent Background
1. [If not already known for each person participating in the interview]
a. What is your official title and role?
b. What are your main responsibilities related to implementation of WIOA?
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection
displays a valid OMB control number. Public reporting burden for this collection of information is estimated to average 1 hour and 10 minutes
per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and
completing and reviewing the collection of information. The obligation to respond to this collection is voluntary. Send comments regarding the
burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to
[email protected] and reference the 0MB Control Number 1290-XXXX. Comments can also be mailed to: U.S. Department of
Labor, Chief Evaluation Office, 200 Constitution Ave., NW, S-2312, Washington, DC 20210. Note: Please do not return the completed interview
guide to the email or mailing address.
WIOA IMPLEMENTATION EVALUATION
OMB SUPPORTING STATEMENT: PROTOCOL 3.1
A. Governance and Planning
1. Local Plan Development
First, I’d like to talk about your local and regional planning process and your local board’s
structure and role.
1.
What input did you have in the state plan, and how did the state plan influence your
local plan?
2.
Please describe the general planning process your local board undertook to produce
your WIOA local plan.
a. Who led the process?
b. Who else was involved?
3.
Describe your general planning process for developing your local plan. What entity
and staff members led the process?
a. How and to what extent were other entities involved?
4.
Did this process differ from previous local plan development efforts?
a. Was the process longer than with previous plans?
b. Were certain partners more involved? Less involved?
c. How did any changes to the process from prior planning efforts improve the final
plan?
d. What were you able to achieve that had not been possible before? What did you
lose, if anything?
5.
Do you have any lessons to share on developing your local plan around what worked
well or has been challenging?
6.
Did you need to revise your local plan after submission? If so, what revisions were
requested and why?
7.
What aspects of your state’s guidance on the local plan were helpful?
a. What could have been more helpful?
8.
What additional guidance is needed from the state or USDOL for you to be able to
carry out the required revision to your local plan?
9.
Did the state provide any TA resources or activities related to developing your local
plan that your area participated in?
a. How useful was it, and what additional TA on the topic could the state provide
that would be helpful for revising your local plan?
10.
Have there been changes in federal or state funding for the workforce system that
have affected your plan or how the plan will be implemented?
a. If there have been significant cuts to funding, how are you adjusting the system in
response to these cuts?
3.6
WIOA IMPLEMENTATION EVALUATION
OMB SUPPORTING STATEMENT: PROTOCOL 3.1
b. What will be the impacts of these cuts on infrastructure, programming, and
service delivery?
2. Regional Designations and Regional Plans
1.
How satisfied are you with your state-designated region under WIOA?
a. Do you think it accurately reflects your regional labor market?
2.
Did you challenge your regional designation?
a. What happened as a result of your challenge?
3.
Has being part of a region helped you to better engage with regional employers? With
regional economic development entities?
4.
How did the board develop the regional plan?
a. What role did the local board members play?
b. Who else was involved? Was there anyone who wasn’t involved that you think
should have been at the table?
5.
What challenges did you experience in developing your regional plans? What worked
well?
a.
Is there anything you plan to change related to the regional planning process for
the next time?
6.
Did your regional plan have to be revised after submission? If so, why?
7.
What guidance did you receive from the state or USDOL on how to develop your
regional plan?
a. How clear was this guidance?
b. Did you need any additional guidance?
3. Local Workforce Development Board Role
1.
How many staff does the board have?
a. What are their roles?
b. Is this different from the number in place the year (or so) prior to WIOA? If so,
how? Why did you make these changes?
2.
What, if any, committees does the board have? How and why, if at all, has this
changed due to WIOA?
3.
If the board has a Youth committee, did the board choose to keep its previous Youth
Council?
4.
How has membership of the local board changed under WIOA?
a. What AJC partners are no longer on the board?
b. What other entities are now represented?
3.7
WIOA IMPLEMENTATION EVALUATION
OMB SUPPORTING STATEMENT: PROTOCOL 3.1
c. How have membership changes impacted these organizations’ involvement with
the AJC system? In particular, how have changes in the involvement of employers
on the board affected their engagement in the system?
5.
Have the new membership requirements under WIOA streamlined board operations?
6.
What challenges, if any, have you faced in implementing these membership changes?
7.
How, if at all, have the role and approaches of the board changed under WIOA?
a. Is it more strategic, more innovative, more focused on continuous improvement,
more focused on improving service integration?
b. Why has the role of the board changed?
i. New board mandates under WIOA?
ii. Required changes in membership?
iii. Some other reason?
8.
Has the state or USDOL provided guidance on local board membership or
operations? If so, how helpful was this guidance? Is any additional guidance needed?
a)
Employer Engagement
1.
What policies and strategies has your state adopted to support LWDBs in increasing
employer engagement in the workforce system?
2.
What policies and strategies has your local board adopted to support employer
engagement? How successful have these been?
3.
How has employer representation on the Board changed, and how have these changes
affected employer engagement in the workforce system?
4.
How have employer engagement efforts and activities changed the workforce system
in your local area?
b)
Sector Strategies
1.
What policies and strategies has your state adopted to increase the use of sector
strategies in the workforce system?
2.
What policies and strategies has your local board adopted to increase the use of sector
strategies in the workforce system? Involving new partners, expanding services and
training options, targeting new populations, etc.?
3.
Has the focus on developing and expanding sector strategies been at the state, local,
or regional levels?
4.
What funding sources are being used to develop and support your sector strategy
programs and initiatives (e.g., local formula funds, WIOA state set-aside funds, Title
II or IV funding)?
3.8
WIOA IMPLEMENTATION EVALUATION
OMB SUPPORTING STATEMENT: PROTOCOL 3.1
B. AJC System
1. AJC Certification
1.
How many AJCs does your local area have?
a. Has this changed since the end of WIA?
b. How many comprehensive and how many affiliate centers are there, and how
have those numbers changed?
c. What are the reasons for changes (e.g. resource sharing requirements or changes
in requirements for each type of AJC)?
2.
Have you developed any additional criteria or processes beyond the state criteria for
AJC certification?
a. Why did you choose to do that and how did you develop them?
3.
Where are you in the process of certifying your AJCs?
a. What is your expected schedule for certifying all of your AJCs (comprehensive
and affiliate)?
4.
Did your AJCs that have been through the process complete it successfully?
a. For those that did not, what certification criteria did they not meet and how are
they remedying these issues?
5.
Have you faced any challenges in implementing the certification process? If so, what
were they and how did you address them?
6.
What worked well in your implementation of the certification process?
7.
What is your schedule for re-certifying?
a. Why did you develop this certification process?
8.
How useful do you think the certification process is in ensuring AJC effectiveness,
accessibility, and continuous improvement? How could it be improved?
2. Infrastructure Funding and MOUs
1.
How does your MOU (excluding resource sharing components) differ from the last
one you had under WIA?
2.
What did the LWDB include in local One-Stop operating budgets?
a. Other than IFAs, what did you include?
b. Why did you include those components? [Make sure to get a copy or copies.]
3.
What process did the LWDB follow to negotiate the required MOU and One-Stop
operating budget(s)?
a. Did you negotiate umbrella budgets or separate budgets for each AJC?
4.
What was the role of board staff in negotiating MOUs and budgets? How did this
compare with the role played by the operator?
3.9
WIOA IMPLEMENTATION EVALUATION
5.
OMB SUPPORTING STATEMENT: PROTOCOL 3.1
What methodologies did the LWDB choose to use for determining their various types
of costs?
a. Why did the LWDB choose those methodologies?
b. What are the advantages and disadvantages of these methodologies?
6.
How did the LWDB determine proportionate use and relative benefit for each
partner?
a. Why did the LWDB choose these methods?
b. How well do these methods work?
c. Are there differences in their appropriateness for on-site vs. off-site partners?
7.
What non-cash contributions do partners make?
a. Why were they not able to contribute cash?
b. How did the LWDB determine how to value non-cash contributions? How well
did that work?
8.
How has the WIOA RSA requirement affected partnerships (either positively or
negatively)?
9.
How often is the LWDB reviewing your agreement (for issues like value of non-cash
contributions, proportionate use and relative benefit)?
a. Are there any challenges related to this review process?
10.
What challenges did you face in the process of developing your MOU and One-Stop
operating budgets?
11.
Has your state has taken any steps to facilitate negotiation of One-Stop operating
budgets (e.g. meetings at the state level, with state partners, to facilitate their
participation)?
12.
Did you achieve consensus on your IFA?
a. [If so]: Why do you think your area succeeded?
b. [If not]: Which required partner(s) did not agree? What were the sticking points?
i. Where are you in the process of working with the state on the steps of the
SFM?
ii. How effective has the state been in carrying out the SFM? What could be
improved?
13.
How helpful was the guidance provided by your state and USDOL on resource
sharing and MOUs? What could have been more helpful
14.
What TA have you participated in on MOUs and resource sharing?
a. How helpful was this TA?
b. What could have been more helpful?
3.10
WIOA IMPLEMENTATION EVALUATION
15.
OMB SUPPORTING STATEMENT: PROTOCOL 3.1
What additional guidance or technical assistance do you need from your state or
USDOL on MOUs and resource sharing?
3. One-Stop Operator
1.
Do you have an operator for your entire local area or for separate AJCs?
a. What entity/entities serve as operator?
2.
Operators have different responsibilities in different areas; for your local area, what is
the operator’s role as it relates to:
a. Partner meetings?
b. Cross-training sessions?
c. Coordinating AJC operations & policies?
d. Managing AJC facilities?
e. Implementation of MOUs/RSAs?
3.
Does the operator have any other significant roles in your local area?
4.
How and why, if at all, has the operator’s role changed with the implementation of
WIOA?
5.
What is the role of the AJC manager?
6.
How and why has the AJC manager’s role changed, if at all, with the implementation
of WIOA?
7.
Prior to WIOA, what entity/entities served as operator?
a. How did you select that operator?
8.
Under WIOA, did you select your operator through competitive procurement (as part
of Title IB provider procurement or otherwise) or as a sole source procurement?
a. How does this differ from your operator selection process under WIA?
9.
How much funding have you allocated for the operator function for this program
year?
a. How did you determine that amount?
b. Is that amount sufficient?
10.
What was your process of developing and sharing your RFP/RFQ? [Make sure you
get a copy of their RFP and other procurement documentation]
a. Who developed the RFP/RFQ?
b. How long did bidders have to respond?
11.
What type of agreement did you develop with your operator—contract, MOU or other
agreement?
a. Why did you choose that vehicle? [Make sure to get a copy of the agreement.]
12.
How did you determine performance metrics for your operator?
3.11
WIOA IMPLEMENTATION EVALUATION
OMB SUPPORTING STATEMENT: PROTOCOL 3.1
a. (If the operator is a for-profit entity) How did you determine allowable profit?
13.
How are you monitoring your operator?
a. If the local board is the operator, how is it being monitored for the operator role?
14.
What challenges did you face in selecting an operator per WIOA’s requirements?
15.
Were you able to competitively procure your operator?
a. How many bidders responded to the RFP/RFQ?
b. Could there have been more bidders?
c. What could have been done to increase the number of bidders?
16.
Did you have an outside entity review your bidding process? Why?
17.
If you were not able to competitively procure, did you bring the function under the
board?
a. If so, how challenging was it to include the required firewalls between the
operator and board functions?
18.
How, if at all, do you think these changes in what entity serves as the operator will
affect AJC operations?
19.
What aspects of your state’s or USDOL’s guidance regarding operator selection or
role were helpful?
a. What could have been more helpful?
20.
What additional guidance is needed from your state or USDOL regarding operator
selection and roles?
4. Collocation at AJCs
1.
Across all the AJCs in the local area, what partners are always or almost always
collocated at comprehensive centers? At affiliate sites?
a. What do you think are the primary reasons for this?
2.
Across all the AJCs in the local area, what partners are never or almost never
collocated at comprehensive centers? At affiliate sites?
a. What do you think are the primary reasons for this?
3.
What partners are collocated at this AJC?
4.
If any required partners that are active in the local area are not collocated at
comprehensive AJC(s), are they providing access to services via training an onsite
staff member or via a “direct linkage”?
5.
Why did they opt for that approach and how is it being operationalized?
a. What challenges and benefits have you experienced with each approach?
6.
Has partner collocation at AJCs changed since WIOA implementation? If so, how
and why?
3.12
WIOA IMPLEMENTATION EVALUATION
7.
OMB SUPPORTING STATEMENT: PROTOCOL 3.1
What guidance have you received from the state on collocation of partners at AJCs
under WIOA?
a. Did that guidance change from what was provided under WIA? If so how? [make
sure to get copies of this guidance]
b. Is any additional guidance needed?
C. Overall AJC Streamlining & Accessible Services
1. Accessibility
1.
How integrated and streamlined is service delivery in the local area?
2.
Please describe any new state or local policies and actions that promote a more
seamless, accessible and customer-focused workforce service delivery network?
3.
Has the state or LWDB encouraged or required any changes to your AJC customer
intake process? What changes did this involve?
4.
To what extent has the state or LWDB emphasized customer or human-centered
design as a key method for making your state’s One-Stop career center system more
customer-focused?
a. What problems or goals do you plan to address through this approach?
b. What challenges and benefits have you encountered in operationalizing it?
5.
Please describe any state or local efforts to improve the accessibility of One-Stops for
individuals with disabilities (e.g. policies, additional funding, or training).
1. Referral Process
1.
How frequently are referrals are made between partner programs?
a. Which partners are most likely to initiate referrals?
b. Which partners are customers most frequently referred to?
2.
How are referrals made to each agency?
a. Do staff provide contact information for the program to a customer?
b. Do staff connect directly with the program they are referring a customer to?
c. Do staff submit a referral via shared MIS?
d. Or, is there some other referral process?
3.
Are referrals tracked? Is information shared about referred customers with the
program the customer is being referred to?
4.
What do you think is the most successful aspect of your referral process?
5.
Is there anything about the referral process you would like to improve?
3.13
WIOA IMPLEMENTATION EVALUATION
OMB SUPPORTING STATEMENT: PROTOCOL 3.1
2. Co-enrollment
1.
About how frequently are local customers are co-enrolled in partner programs?
Which programs and for what reasons?
1.
How, if at all, do co-enrollment processes differ from referral processes?
2.
How, if at all, is information about co-enrolled customers shared with partner
programs?
3.
What is the most successful aspect(s) of local co-enrollment processes?
a. What requirements particularly facilitate co-enrollment?
b. Are there factors that impede co-enrollment?
4.
Is there anything about local co-enrollment processes you would like to improve?
3. AJC Partner Meetings
1.
Do you have regular AJC partner meetings? (If no, skip to cross-training)
2.
Are these meetings for individual AJCs or only across the local area?
3.
What partners attend regularly?
4.
What level of staff attend these meetings? Program managers? Line staff? Both?
5.
Who organizes and facilitates these meetings?
6.
How often do they occur? For how long?
7.
What topics are covered in these meetings? [Try to get a copy of the agenda for the
most recent 1-2 mtgs.]
4. Cross-training
1.
Have you organized or participated in any cross-training sessions across different
programs? (If not, skip to 9)
2.
How often have these sessions occurred? At what locations?
3.
Are these in-person or virtual?
4.
How long did/do they last?
5.
Who attends regularly? Are these local or state-wide trainings?
6.
Who organizes/facilitates?
7.
What is covered in these sessions? [try to get an agenda for the last 1-2 cross-training
sessions.]
8.
How helpful have these sessions been?
a. Is there any way they could have been improved?
9.
In what other ways—if any—do you share information on partner programs?
a. How well do these other methods work?
3.14
WIOA IMPLEMENTATION EVALUATION
OMB SUPPORTING STATEMENT: PROTOCOL 3.1
5. Use of shared MIS
1.
Do any core or required partners use the same management information system
(MIS)? If so, what systems? (If not, skip to #5)
a. Which partners use them?
b. For what purposes?
i. Case management?
ii. Participant tracking?
iii. Performance reporting?
iv. Fiscal reporting?
2.
Are there differences in how partners can use any of these common systems?
a. Can all partners view data? Are there specific fields that are restricted?
b. Can all partners enter data into the system? Are there specific fields that only
some partners can enter?
c. Can all partners revise data? Are there specific fields than only some partners can
modify?
3.
How do you ensure that the data being entered into these systems by different
partners is accurate?
4.
How are you protecting the security of the data being entered and stored?
5.
If not all programs use the same MIS, why not?
a. Are there any plans to add more partners to shared MIS?
b. Which ones?
c.
With what data entry or viewing privileges?
d. If there are no plans to add partners, why not?
e. Have you developed any workarounds to allow partners to share data in the
absence of a shared MIS?
6.
Has the state or LWDB encouraged, required, or led the development of a shared case
management system?
7.
How important is it for seamless services that partner programs use the same MIS?
D. Performance Accountability
1. ETPL Changes
1.
What challenges did you face in implementing the new ETPL requirements?
3.15
WIOA IMPLEMENTATION EVALUATION
2.
OMB SUPPORTING STATEMENT: PROTOCOL 3.1
How and why, if at all, have these ETPL changes impacted the number of providers
available for your local area?
2. Common Measures
1.
Have you faced any challenges implementing the WIOA common performance
measures?
2.
Which challenges have you faced in measuring the effectiveness of employer
services?
a. Do you have thoughts on measures other than the ones being used that might
better capture the effectiveness of serving employers?
3.
Have you faced any challenges in implementing state-specific performance
measures?
4.
Are you developing any local measures that you and your local partners will be
reporting on?
5.
Have you faced any challenges in implementing local performance measures?
E. Services for job seekers and employers
1.
Has WIOA changed your approach to providing services to businesses? In what way?
2.
What strategies have you implemented to increase the use of work-based and jobdriven training (such as OJT, apprenticeships, transitional training, internships)?
a. What work-based training models are most commonly used in your local area?
b. How is your local area working to improve the use of registered apprenticeship?
c. Are there regulations around apprenticeship that could or should be relaxed to
improve the use of registered apprenticeship in your state/region?
3.
Please describe your efforts to increase the use of registered apprenticeship programs.
a. Have there been any challenges related to including apprenticeships on the ETPL?
4.
Please describe your efforts to improve access and services to more barriered
customers, including disabled customers and those with low basic skills.
5.
Have you received any guidance or TA from the state on improving services to
customers with barriers? How helpful was it?
6.
What, if any, additional guidance should USDOL or your state should provide on
serving customers with barriers?
F. Youth
1.
How successful have you been in meeting the 75% OSY requirement?
a. What strategies or policies have you implemented to meet this requirement?
3.16
WIOA IMPLEMENTATION EVALUATION
OMB SUPPORTING STATEMENT: PROTOCOL 3.1
2.
Please describe any key challenges you have had in meeting this requirement. If
you’ve met the requirement, has anything in particular worked well for accomplishing
that?
3.
What guidance has your state provided regarding the 75% youth OSY expenditure
requirement?
4.
What, if any, additional guidance should USDOL or your state should provide related
to the 75% OSY requirement?
G. Overall Questions
1. Partnerships
1.
How would you assess relationships among core partners? Required partners? Other
partners?
a. Has this changed because of WIOA?
b. If not, why do you think WIOA’s focus on enhanced partnerships has not affected
your partnerships?
2.
Are there sufficient systems in place to ensure regular and effective communication
between partners?
3.
What have been the key challenges and successes related to increasing collaboration
across partners?
a. Are there differences in the extent of collaboration with your core partners versus
other required partners, such as UI, SCSEP, Veterans programs, and TANF?
4.
Overall, how integrated are workforce services now?
a. Has this changed because of WIOA?
b. What changes reflect this more integrated service delivery? For example, are case
managers of different programs meeting more often to discuss co-enrolled clients?
5.
[If respondent feels workforce system is not integrated]: Why do you think WIOA’s
focus on enhanced partnerships has not affected the integration of services?
2. Systems Change
1.
Overall, do you think the workforce system here has changed as a result of WIOA?
a. If so, what are the major changes you can attribute to WIOA? E.g. employer
engagement, partnerships, services more streamlined, data sharing, others?
b. If not, do you feel systems had already begun to change prior to WIOA, or that
efforts under WIOA have not yet been effective in changing the system?
2.
Have there been any negative, unintended consequences of implementing the WIOA
regulations?
3.17
WIOA IMPLEMENTATION EVALUATION
OMB SUPPORTING STATEMENT: PROTOCOL 3.1
a. Are there ways in which you feel the new regulations actually make it more
difficult for you to achieve your goals?
3.
[If respondent doesn’t think WIOA has changed the system]: Do you feel systems had
already begun to change prior to WIOA, or that efforts under WIOA have not yet
been effective in changing the system?
4.
What additional changes are planned or needed to transform your workforce system
to meet the goals of WIOA?
b. What changes would you like to see to make the workforce system better overall?
3. Guidance and Technical Assistance
1. Of the TA provided by DOL national and regional offices on WIOA implementation,
which assistance has been most helpful?
a. Which TA formats and types of materials have been most useful for you?
2. Are you satisfied with the level of TA DOL has provided on WIOA implementation?
3. Have you accessed the ION site to obtain TA materials or guidance on specific issues? If
so, which issues?
4. In what areas would you like to receive additional TA?
Wrap-up
1. How would you describe your overall progress to date with WIOA implementation?
a. Where do you still have plans to make significant changes?
2. Beyond what we’ve discussed today, are there other areas you would like to highlight?
3. Are there any other areas of WIOA implementation our study should explore?
3.18
| File Type | application/pdf |
| File Title | Report |
| Author | Mathematica Staff |
| File Modified | 2018-07-16 |
| File Created | 2018-07-16 |