Site Recruitment Process and Materials

APPENDIX G
SITE RECRUITMENT PROCESS AND MATERIALS

Site recruitment began in February 2010. At that time, general information was provided
about the study in briefings to the DOL Regional Administrators. In addition, in a series of
meetings, staff from the Department’s Employment and Training Administration (ETA) and its
evaluation contractors briefed stakeholders in the workforce investment system about the study.
These stakeholders included: the National Association of Workforce Boards, National
Association of State Workforce Agencies, National Association of Counties, United States
Conference of Mayors, National Governors’ Association, and the National League of Cities. The
names of the selected sites were not provided. The study was generally well received.
In April 2010, ETA issued a Training and Employment Notice (TEN 37-09) notifying all
workforce investment areas in the country about the study (attached). The TEN did not name the
30 selected sites.
In May 2010, Jane Oates, the Assistant Secretary for ETA, called each Local Workforce
Investment Area (LWIA) selected as a study site. On this call, Ms. Oates briefly described the
study, its importance, and DOL’s commitment to the study. The site was typically represented
on the call by the local Workforce Investment Board’s executive director. The director of the
state workforce investment board or his/her representative and the regional administrator also
participated in the call. While several site directors expressed concerns about the study, only
one—Ohio Area 7—refused to participate in the study at that time. After the call, a letter from
Ms. Oates describing the study and thanking the site for their participation was mailed to the site
(attached).
Soon after the letter was sent to the site, a senior member of the evaluation team together
with an ETA representative conducted an in-person meeting at the site. During this visit, the
study was presented to senior site staff (the powerpoint is attached) and their questions
addressed. State representatives were informed about the meeting and some chose to attend.
While some sites agreed to participate in the study during the first meeting, others required
further discussions. Many sites required us to present a discussion of the study to the site’s
Workforce Investment Board. Further visits were made to each site to discuss study details such
as the point of random assignment and any site-specific exemptions from random assignment.
A meeting of representatives of all sites that had agreed to participate was held in February
2011.
The table below shows the burden to senior site staff during recruitment. In total, the burden
is 3,567 hours. If the staff time is valued at $44 per hour (the median wage of a manager), the
cost is $156,948 (=$44 x 3,567 hours).

1

Burden on Site Staff During Recruitment
Hours per
Respondent

Activity
Call with Ms. Oates
First visit from contractors
Second visit from contractors
Presentation to Workforce Investment
Board
Phone calls/emails
Conference
Total
a.
b.
c.

Hours per
Site

Total Hours

0.3
2
1

Number of
Respondents per
Site
3
3
12

1
6
12

33a
186b
372b

0.25
12
12

12
2
2

48
24
24
115

1,488b
744b
744c
3,567

Thirty-three sites were asked to participate—the thirty originally chosen sites plus three replacement sites.
Visits and phone calls were made to 31 sites.
Representatives from 28 sites attended the conference.

2

TRAINING AND EMPLOYMENT
NOTICE

NO.
37-09

DATE
April 6, 2010

TO:

ALL STATE WORKFORCE AGENCIES
ALL STATE WORKFORCE LIAISONS
ALL STATE ONE-STOP CAREER CENTER SYSTEM LEADS
ALL STATE AND LOCAL WORKFORCE INVESTMENT BOARDS
ALL STATE UNEMPLOYMENT INSURANCE DIRECTORS

FROM:

JANE OATES /s/
Assistant Secretary
Employment and Training Administration

SUBJECT:

The Workforce Investment Act Gold Standard Evaluation

1. Purpose. Over the past two years, the workforce investment system has had to confront
daily difficult recession challenges by working tirelessly to address the needs of the unemployed,
dislocated workers, and many others. The Employment and Training Administration (ETA)
commends all stakeholders for the remarkable work you are doing and we know that the
programs you are managing will continue to make positive change that benefits workers and
employers.
To assess the critical work that has been done and yet to be done, ETA is announcing the “kickoff” of a rigorous random assignment evaluation of the Workforce Investment Act of 1998
(WIA) Adult and Dislocated Worker programs. This evaluation presents a tremendous
opportunity for the workforce investment system to document the importance of WIA-funded
services to the participants we serve, our oversight bodies including Congress, the Office of
Management and Budget, and the general public. This evaluation, through its rigorous
examination of the operations and effectiveness, is consistent with the Obama Administration’s
Memorandum on Transparency and Open Government by promoting transparency and
accountability.
The evaluation will specifically examine the Adult and Dislocated Worker formula programs by
1) analyzing program impacts on participants’ post-program employment and earnings and their
cost effectiveness; and 2) comparing outcomes of WIA participants to the outcomes of similar
individuals who do not receive WIA services. Throughout the evaluation effort, state and local
workforce agencies, project partners, and workforce investment system stakeholders will receive
updates on the study’s progress.
2. Evaluation Goals. Mathematica Policy Research, Inc. (MPR) will conduct the evaluation
working with subcontractors Social Policy Research Associates, MDRC, and the Corporation for
a Skilled Workforce. The evaluators will examine net impacts by subgroups including age,
gender, race/ethnicity, and veteran status, among others with an additional emphasis placed on

EMPLOYMENT AND TRAINING ADMINISTRATION
U.S. DEPARTMENT OF LABOR

WASHINGTON, D.C. 20210

the impact effectiveness of specific intensive services and the combination of intensive
services on workforce investment system client’s employment and earnings.
3. Evaluation Tasks. This evaluation will compare the outcomes of WIA participants to the
outcomes of similar individuals randomly assigned to a control group. MPR and its
subcontractors will design and implement random assignment procedures, monitor the evaluation
throughout the client observation period, collect project and state administrative data, analyze the
data, and produce interim and final reports. Please note that the control group in this study will
be greatly minimized to reduce the impact on One-Stop Career Center operations; however, a
small number of individuals will only receive core services rather than the full menu of One-Stop
Career Center services. In this case, the limited services group is absolutely necessary to ensure
that the results of the evaluation are rigorous. The evaluation will also include a comprehensive
process evaluation of program activities in the thirty sites that will chronicle the best practices
and lessons learned from field operations and will provide a rich context for interpretation of the
impact results.
Data sources for this evaluation include: 1) client data collected through a project-specific
management information system; 2) interview and focus group data collected through visits to
participating local workforce investment boards (LWIBs) and One-Stop Career Centers; 3) state
unemployment insurance wage and benefit records; 4) follow-up client surveys conducted 15 and
30 months after random assignment; and 5) other administrative data related to participation in
Federal and state programs. The strictest confidentiality procedures will be maintained for all
client data. Information collected during the evaluation process on individual LWIBs will be
aggregated for research purposes to protect site confidentiality and will not be shared with other
LWIBs or used for performance or management purposes by ETA. LWIBs will also receive
individualized analysis of the data collected from the evaluation that could be used for their own
program assessment and management improvement.
4.
Evaluation Site Selection. This evaluation requires participation of thirty randomly
selected LWIBs across the country to enable evaluators to construct a nationally representative
sample of the workforce investment system. Without a nationally representative sample of
LWIBs, the findings would not accurately portray the national workforce investment system.
Through participation in this evaluation, selected LWIBs will have the opportunity to
demonstrate the effectiveness of the workforce investment system. Therefore, participation of
those randomly selected LWIBs is critical to the overall success of the evaluation. ETA will
soon be contacting the randomly-selected LWIBs directly to notify them they have been selected
to participate in the evaluation. The selected LWIBs will be compensated for the additional costs
incurred for participating in the evaluation. LWIBs also will be part of a national learning
network that will include a discussion with ETA leadership and a forum to discuss WIA
reauthorization. Selected LWIBs will be contacted within 30 days after publication of this TEN.
ETA will also notify the appropriate states of the LWIBs selected.
5.
Action Requested. State Workforce Agencies are requested to: 1) review and convey the
information contained in this issuance to local workforce partners, as appropriate; and 2) work
2

with MPR and their subcontractors to facilitate the acquisition of all necessary state
administrative and unemployment insurance wage and benefit record data for the evaluation.
State Unemployment Insurance Directors are requested to fully cooperate with MPR and their
subcontractors in providing all necessary wage and benefit record data for the evaluation.
LWIBs are requested to: 1) review and convey the information contained in this issuance to the
One-Stop Career Centers and appropriate workforce partners in their area; and 2) actively
participate in the evaluation, if selected, to ensure there is a nationally representative sample of
LWIBs and evaluation results can be generalized across the nation; and 3) work with MPR and
their subcontractors to facilitate the acquisition of all necessary administrative and participantlevel data for the evaluation.
6. Evaluation Time Frame. The period of performance for this evaluation effort will be for
five years beginning in July of 2010. An interim report is expected in 2012 and a final report is
expected in July 2015.
7. References. P.L. 105-220, as amended: Workforce Investment Act of 1998, Section 172
8. Inquiries. For more information about this study, please contact Jonathan A. Simonetta,
Federal Project Officer, Office of Policy Development and Research, ETA at (202) 693-3911, or
[email protected].

3

LETTER FROM JANE OATES TO SELECTED LWIAS

[LWIA contact information]
Dear Mr. xxx:
As a follow-up to our conversation on May 27, 2010, over the past two years the public
workforce system has had to confront difficult economic challenges by working tirelessly
to address the needs of the unemployed, dislocated workers, and many others. The
Employment and Training Administration (ETA) commends you and stakeholders across
the county for the remarkable work being done.
As announced on April 6, 2010 in Training and Employment Notice 37-09, ETA is
beginning a rigorous random assignment evaluation to examine the effectiveness of
intensive and training services offered through the Workforce Investment Act (WIA)
Adult and Dislocated Worker programs. This evaluation presents a tremendous
opportunity for the workforce system to document the importance of WIA-funded
services to the participants we serve and share our findings with Congress, the Office of
Management and Budget, and the general public.
[LWIA name] is one of the 30 local workforce investment boards (LWIBs) randomly
selected for this national evaluation. It is critical that you participate in the evaluation so
that the results of the evaluation accurately portray the public workforce system and that
the evaluation yield reliable and valid results and findings.
Mathematica Policy Research will be conducting the national evaluation on behalf of
ETA. Mathematica and its partners, Social Policy Research Associates, MDRC, and the
Corporation for a Skilled Workforce, are committed to conducting this evaluation so that
it minimizes disruptions to your operations and is rewarding to those local areas
participating. [LWIA name] along with the other 29 LWIBs that participate, will 1)
belong to a learning community that will have opportunities to discuss policy
recommendations with senior ETA staff; 2) receive site-specific information that will not
be shared with other sites or with ETA; and 3) receive modest financial compensation for
participating in the evaluation.
In addition, all data collected from customers and staff will be held strictly confidential
and evaluation results will be aggregated when they are presented in reports so no
individual LWIB is singled out. However, practices that are deemed promising and/or
successful will be identified so the workforce investment system may learn about them
and replicate them.
A representative from the evaluation team will contact you shortly to discuss next steps.
If you have questions or comments about the evaluation, or if you would like more
information, feel free to contact Jonathan Simonetta, Project Officer at the U.S.
Department of Labor at (202) 693-3911 or [email protected]. You may also

contact Sheena McConnell, the Mathematica Project Director at (202) 484-4518 or
[email protected].
Your participation and support in this valuable effort will be critical to shaping future
WIA policy and research. I look forward to working with you.
Sincerely,

Jane Oates
cc:

[state agency]
[board chair]
[regional administrator]

The Workforce Investment Act Adult and
Dislocated Worker Programs Gold Standard
Evaluation
[DATE]

[names of study team members attending]

Meeting Purposes



Discuss the study and the importance of
your participation

 Answer your questions about the study


Start discussing point of random assignment

2

What We Will Learn from the Study

 The value of WIA services for participants
– Focus on Title I – adult and dislocated worker formula
funding programs

– WIA-funded intensive services and training



For whom particular WIA-funded services are
most beneficial



Whether the benefits vary by how services are
provided

3

Why Conduct this Study?



Congress requires evidence of effectiveness;
OMB and GAO have been critical of the lack of
rigorous evidence



DOL is firmly committed to rigorously evaluating
WIA-funded services



Important for policymakers and administrators at
all levels to know what is working and for whom



May lead to important policy and programmatic
improvements

4

Main Features of the Study



30 LWIAs randomly selected to participate



Eligible customers seeking WIA-funded services
beyond core are randomly assigned to 3 groups:
– Full-WIA group: access to all WIA-funded services
– Core-and-intensive group: access to all WIA-funded
services except training
– Core-only group: access only to core services

 [Most] eligible customers may access all WIA-funded
services (as funding allows)



Extensive supports to minimize disruptions to your
operations
5

How Will Random Assignment Work?

 All WIA-eligible customers seeking intensive
services (at comprehensive and satellite centers)
are informed of the study

 Customers must be eligible for further WIA
services and provide their consent to be randomly
assigned



Random assignment is conducted by a secure
web-based program accessed by local staff

 Random assignment occurs within minutes
6

What Happens to Customers in the Study?



[Most] eligible customers are in the full-WIA group

 Only [small percentage] of customers are in the
core-only group; same percentage in core-andintensive services group

 Customers maintain their assignment for 15
months



All customers can continue to access core
services and non-WIA-funded services

7

How Will We Support You?



Development of study procedures that are tailored to
your existing operations



Training for all your staff

 Real-time technical assistance during the study (e.g.,
how best to inform customers about the study)



Financial compensation for costs incurred

8

What Will You Gain From Participating?



Contribution to what is known about the value of
WIA-funded intensive services and training



Findings about your own site (that are only shared
with you)



Knowledge-sharing opportunities with other study
LWIAs

 Participation in discussions with senior DOL staff

9

What Are You Being Asked to Do?



Work with us to prepare for and implement the
study



Inform and enroll eligible customers into the study

 Maintain customers’ group assignment


Provide us with information about customers and
how you deliver WIA services

10

Estimated Evaluation Timeframe


Training for LWIAs

May – Aug 2011



Customer intake period

July 2011 – Dec 2012



Feedback to sites

Begins 6 months after intake
period begins



Implementation report

Winter 2012/2013



Monitor assignments

May 2011 – Dec 2013



Short-term impact report

2014



Final report

2015

11

Questions?

12

Extra Slides

13

Difference Between Outcomes and Impacts

Percent of Full WIA Adult
Customers Employed
After One Year

Percent of Restricted
Group Employed After
One Year

Impact
(Difference)

A

60

60

0

B

45

40

5

C

50

40

10

LWIA

14
Mathematica® is a registered trademark of Mathematica Policy Research.

What Data Will the Study Use?

 Information from forms completed by customers
before random assignment



LWIA information (collected on site visits)

 Follow-up surveys of a subset of participants


State WIA management data (and local data as
needed)



State UI wage and benefit records



Other partner agency data

15

Research Groups for Adults and Dislocated Workers

Random Assignment: When a Customer is
about to Receive Intensive Services

Full-WIA

All WIA
services

16

Core and
Intensive
Core and
intensive
services only

Core
Core services
only