WIA Evaluation, OMB supporting statement, Part A, 9.22.2011

Part a: supporting statement for paperwork reduction act submission

The U.S. Department of Labor (DOL) Employment and Training Administration (ETA) is undertaking the Workforce Investment Act Adult and Dislocated Worker Programs Gold Standard Evaluation (WIA Evaluation). The overall aim of this evaluation is to determine whether adult and dislocated worker services funded by Title I of the Workforce Investment Act (WIA)—currently the largest source of Federal funding of employment and training services—are effective and whether their benefits exceed their costs. ETA has contracted with Mathematica Policy Research and its subcontractors—Social Policy Research Associates, MDRC, and the Corporation for a Skilled Workforce—to conduct this evaluation. This package requests clearance for three data collection efforts conducted as part of the evaluation:

1. A consent to participate in the study form (presented in Appendix A)

2. An eligibility checklist, a study registration form¹ (SRF), and a contact information form (CIF) (all presented in Appendix B)

3. Site visit guides (presented in Appendix C)

At a later date, ETA will request revision to this information collection clearance package to obtain approval for the remaining data collection forms for the evaluation, including two follow-up surveys of study participants and protocols for the collection of information about costs of WIA services. The full package for this study is being submitted in two parts because data collected through the evaluation’s initial stages will inform development of the cost and follow-up data collection instruments. As a result, the study schedule requires that the development of systems to collect the baseline data begin before all data collection instruments are developed and tested. We understand that the approval of the current package does not constitute approval for the cost protocols or the follow-up surveys.

During the review process, discussions with Office of Information and Regulatory Affairs (OIRA) staff, and upon careful review of Paperwork Reduction Act (PRA) requirements, a potential and certainly unintentional PRA violation has come to the attention of DOL. The Department reached out to sites for recruitment before obtaining OMB clearance for this collection. The Department is committed to PRA compliance and is exploring avenues to ensure timely clearance of PRA-covered information collections in a manner that does not compromise obtaining information needed to conduct meaningful evaluations of important programs. Appendix G discusses the recruitment process and provides the instruments used in that process.

A. Justification

1. Circumstances Necessitating the Data Collection

Passage of the Workforce Investment Act of 1998 led to a major redesign of the country’s public workforce system. WIA programs serve more than two million people annually at a cost of $3 billion (U.S. Department of Labor, Employment and Training Administration 2007). Among its goals, WIA aims to bring formerly fragmented public and private employment services together in a single location within each community, make them accessible to a wider population than did prior employment and training services, empower customers with greater ability to choose from among services, and provide localities both greater local flexibility in using funds and greater accountability for customers’ employment outcomes. ETA is conducting the WIA Evaluation to provide rigorous, nationally representative estimates of the net impacts of WIA intensive and training services that are offered during the evaluation period. Generally speaking, intensive services are services that involve staff assistance and include assessments, counseling, and job placement. Training includes education and occupational skills building. This evaluation will offer policymakers, program administrators, and service providers information about the relative effectiveness of services, how the effectiveness varies by target population, and how the services are implemented. The study will also produce estimates of the benefits and costs of WIA intensive and training services offered during the evaluation period.

To provide context for the study, this section begins with a brief description of previous studies of employment and training programs and an overview of WIA, including its structure and the types of services offered. The section then provides an overview of the study. Finally, it provides an overview of the study’s data needs and collection plan, including data collection for which clearance will be requested in the future addendum to this package, as well as data collection for which we are requesting clearance in this package.

a. Previous Studies of Employment and Training Programs

Evaluations have been conducted of most of the major employment and training programs that preceded WIA, including the 1962 Manpower Development Training Act (MDTA), the 1973 Comprehensive Employment and Training Act (CETA) and the 1982 Job Training Partnership Act (JTPA). Although the methods and findings from individual studies vary, most reviews have drawn similar conclusions about these programs’ effects on the major target populations (D’Amico 2006; Friedlander et al. 1997; Greenberg et al. 2003; Heckman et al. 1999; LaLonde 1995). For women, the impacts of employment and training programs are generally positive, with significant and positive net impacts on earnings found in studies of MDTA, CETA, and JTPA (Ashenfelter 1978; Bassi 1983, 1984; Bloom et al. 1993). The estimated impacts for men are more modest and less consistent; whereas the MDTA study found a significant positive net impact on men’s earnings, the estimated impacts of CETA varied from large and negative (Dickinson et al. 1986, 1987) to large and positive (Bassi 1983), depending on the study. The JTPA study found positive impacts for men of about eight percent in earnings growth over a 30-month follow-up period (Orr et al. 1996).

Although there have been no experimental evaluations of WIA, several nonexperimental studies of the impact of WIA intensive services and training on adults and dislocated workers have been conducted (Heinrich et al. 2008; Hollenbeck et al. 2005; Hollenbeck 2009). The studies found that participation in the adult WIA program is associated with a several hundred dollar increase in quarterly earnings; participation in the dislocated worker WIA program is associated with an initial reduction in earnings relative to the comparison group, but later gains in earnings so that earnings ultimately match or overtake those of the comparison group. However, the impact for dislocated workers is lower than it is for the adult program. As with many nonexperimental studies, these results are open to the criticism that the observed differences between outcomes of program participants and outcomes of members of the comparison group are a result of preexisting differences between those groups, rather than a result of participation in the program.

b. Overview of WIA

Organizational structure. WIA funds originate at the Federal level and flow to states and local areas. Each state has one or more local workforce investment boards (WIBs) that govern nonoverlapping geographic areas within the state, known as local workforce investment areas (LWIAs). WIA requires that each LWIA establish One-Stop Career Centers at which customers can access WIA services. One-Stop Career Centers provide access, in one location, to local, state, and federal employment-related services and supports. Services from a range of other partner organizations and sources of funding, such as the Wagner-Peyser Act, are also provided at One-Stop Career Centers.

Currently, there are slightly fewer than 600 LWIAs. The size of the LWIAs varies dramatically, with some serving as few as a couple of dozen customers annually, whereas others serve tens of thousands. LWIAs can also vary in the geographic area they cover; for example, some states with large rural areas have one LWIA to cover the entire state except for its major cities. In these cases, the LWIA may be further divided into subareas for organizational and management purposes.

Target groups and services. Title I of WIA provides separate funding for two groups of customers who are 18 or older:

• Adults. An individual must be at least age 18 to be eligible for WIA adult funding. Priority for adult intensive and training services (described below) is given to low-income customers.

• Dislocated workers. Eligibility is limited to individuals who fall into one of the following categories: persons who have been terminated or laid off from a job without cause, whose employer has announced a facility closure within 180 days, or who were self-employed but are now unemployed.

Program Year 2009 funding amounts for serving adult customers and dislocated worker customers were $862 million and $1,467 million, respectively (U.S. Department of Labor, Training and Employment Guidance Letter 20-08). Programs funded with these large sums of taxpayer dollars have not been rigorously evaluated. Recognizing the importance of evidence-based policy development, this evaluation focuses on evaluating the impact of services provided to adults and dislocated workers served under Title I of WIA; accordingly, findings will be presented separately for adults and dislocated workers.

WIA provides for three levels (or tiers) of services for adults and dislocated workers: core, intensive, and training. The purpose of this tiered structure is to allow all customers to access some informational services (core), while concentrating the services that are the most expensive to provide (training) on a smaller number of customers with the greatest need. In general, customers receive services at one level before moving up to the next tier of services. Core services are the lowest tier, intensive services follow, and training is the highest tier of service offered (Federal Register, Notice of Proposed Rule Making, 2006). The following paragraphs describe the services that are statutorily considered to be part of each tier.

WIA-funded core services are the least resource intensive and can often be accessed by customers without staff assistance either in the resource room at the One-Stop Career Center or remotely via the Internet. Self-service offerings, which are to be provided universally to all customers who want them, regardless of income, include job listings and other information on the labor market (such as lists of high-demand occupations); information on services provided via WIA and other programs; information on WIA service providers; Internet access; computer software for assessments and resume writing; and access to telephones, fax machines, and copy machines. Other core services require some staff assistance. These include workshops on resume writing and interviewing; initial assessments of skills, aptitudes, and interests; determination of eligibility for programs; help in contacting an employer; and information about training services.

Intensive services are available to customers who are unable to obtain employment with the help of core services alone. The determination of the need for intensive services is made by One-Stop Career Center staff. The services include comprehensive and specialized assessments; development of an individual employment plan; group and individual counseling; placement in unpaid jobs to gain work experience; internships; assistance in finding employment; and short-term prevocational services, such as work skills development.

Authorized training services for building skills and increasing employability are delivered primarily through individual training accounts (ITAs), which are like vouchers that provide customers the flexibility to choose a training program that meets their needs. Training is available to customers who are eligible for intensive services but for whom intensive services are insufficient to enable them to obtain and retain employment. The training services may include occupational skills training, skills upgrading, and adult education and literacy activities (in concert with training activities) as well as counseling and support during training. To ensure that WIA-funded training programs meet certain standards for quality, ITAs may be used to pay only for training programs listed on a state’s Eligible Training Provider List. On-the-job training, customized training provided by an employer, and training designed for special populations facing multiple barriers to employment can be funded directly rather than through an ITA.

c. Overview of the Evaluation

Because WIA mandates universal access to services, all potential customers must have access to core services. Therefore, the evaluation cannot examine the impacts of core services alone. Rather, it will examine the impacts of intensive and training services on customers’ outcomes relative to a situation in which customers have access to core services only.

The evaluation will address the following research questions:

1. Does access to WIA intensive and training services—both individually and combined—lead adults and dislocated workers to achieve better educational, employment, earnings, and self-sufficiency outcomes than they would achieve in the absence of access to those services?

2. Does the effectiveness of WIA vary by population subgroup? Is there variation by sex, age, race/ethnicity, unemployment insurance (UI) receipt, education level, previous employment history, adult and dislocated worker status, and disability status?

3. How does the implementation of WIA vary by LWIA? Does the effectiveness of WIA vary by how it is implemented? To what extent do implementation differences explain variations in WIA’s effectiveness?

4. Do the benefits from WIA services exceed program costs? Do the benefits of intensive services exceed their costs? Do the benefits of training exceed its costs? Do the benefits exceed the costs for adults? Do they for dislocated workers?

To answer the research questions, the project has three main analysis components: (1) impact analysis, (2) implementation analysis, and (3) benefit-cost analysis.

Impact analysis. The major goal of the WIA Evaluation is to generate precise, unbiased estimates of the impacts of WIA intensive and training services for adults and dislocated workers and for those estimates to be broadly generalizable to the population of WIA customers who receive services during the study period. Toward this end, the evaluation will involve randomly selecting study sites and, within those sites, randomly assigning customers to research groups.

Random selection of sites. To obtain a nationally representative study sample, the design calls for first randomly selecting study sites. Since LWIAs typically administer local WIA funding and hence determine the services provided, an LWIA is considered a “site” in the evaluation. Thirty sites have been randomly selected from the set of all LWIAs on the U.S. mainland that serve 100 or more intensive services customers annually. This number of sites will allow for precise estimates and a low rate of assignment to the research groups that are not eligible to receive full WIA services (as described below). The random selection of sites was conducted using explicit and implicit stratification to take into account the enrollment levels at each site, the LWIA’s geographic location, and, as a proxy for the focus the site places on training, the proportion of LWIA intensive service customers who receive WIA-funded training.

We are currently negotiating with each of the randomly-selected sites about its participation in the evaluation. This negotiation involves describing the study to senior LWIB staff and addressing their questions. It also involves collecting information about intake procedures to determine the point of random assignment in each site. No customers were asked for information during this process.

Random assignment of customers within selected sites. The cornerstone of the WIA Evaluation impact analysis is random assignment of customers to research groups. A random assignment approach is generally viewed as the “gold standard” for evaluating social programs because, more than any other approach, it minimizes the chance that any observed differences in outcomes between research groups are due to unmeasured, preexisting differences between members of the groups.

During the study’s intake period, nearly all WIA adults and dislocated workers who would be offered intensive services in the absence of the study will be randomly assigned. Counselors in the sites can use the participant eligibility checklist (presented in Appendix B) to determine whether a customer is eligible for the study.

Customers who will be exempted from the study include:

• Customers under 18. While most customers under 18 are served by WIA’s youth programs, a few may apply for intensive or training services under the adult program. Including youth would require obtaining parental consent for participation in the study, placing additional burden on LWIA staff. Hence, these customers are excluded from the study.

• Customers who participate in the Trade Adjustment Assistance (TAA) program. Participants in TAA are entitled to intensive services. As restricting intensive services for these participants would affect the services offered by TAA, participants in TAA are exempted from the study.

• Veterans and “covered” spouses of veterans (as defined in 38 USC 4215). Veterans and certain categories of spouses of veterans are given “priority” of service in WIA programs. These customers are exempted from the study because many sites, as well as DOL’s Veterans’ Employment and Training Service, interpret restricting services as part of a random assignment study as inconsistent with their priority of service.

• Customers who are recruited by employers for on-the-job training. In some LWIAs, employers may recruit customers and hire them on condition they receive on-the-job training funds. These customers are excluded as random assignment would place a burden on employers and hurt relationships between the LWIA and employers.

WIA customers found eligible will be told about the study and asked to sign a form confirming they have been told about the study, that they understand its implications, and that they agree to participate. (The form they will be asked to sign is presented in Appendix A.) All customers who consent to participate will be asked to complete the SRF and the CIF. (These forms are presented in Appendix B.) LWIA staff will then enter the responses to questions 2, 3, 4, 5, D, E, and F from the SRF into a study-specific computer system, which is referred to as the Random Assignment System (RAS).² The system will randomly assign each customer into one of three research groups. The LWIA staff will notify the customer of his or her research group assignment.

The three research groups to which customers will be assigned are (1) full-WIA group—adults and dislocated workers in this group can receive any WIA services for which they are eligible, (2) core-and-intensive group—adults and dislocated workers in this group can receive any WIA services for which they are eligible other than training, and (3) core-only group—adults and dislocated workers in this group can receive only core services and no WIA intensive or training services (Figure A.1). Customers who do not consent to participate in the study will be allowed to receive core services only. The study enrollment period will be between 12 and 18 months in each site.

The rates at which individuals are assigned to the core-only and the core-and-intensive groups will be low in all sites for both adults and dislocated workers, ranging from 0.7 percent in each restricted-service group in the LWIAs serving the largest number of customers to eight percent in

Figure A.1. Research Groups

the LWIAs serving the smallest number. These rates were chosen to minimize the rates at which customers would be limited in access to services. These low rates of assignment to service-restricted groups will reduce the disruption to the LWIA. Across all sites, 2,000 customers will be assigned to each of the restricted-service research groups with the remainder of customers assigned to the full-WIA group. Based on recent data, if all 30 sites participate, we estimate that about 64,000 customers will be assigned to the full-WIA research group.³

Several key features of the random assignment process will be monitored to ensure that the integrity of random assignment is maintained in all sites. First, all customers who consent to participate in the study must go through random assignment once and only once. Second, customers cannot change their research group status during the study. Third, customers are to be offered those services, and only those services, available to the research group to which they were assigned for 15 months after random assignment. Fourth, sites should change as few of their routine procedures as possible during their involvement in the study. These features of random assignment will be monitored through the collection of data (described later in detail) on customers’ characteristics, the services they received, and sites’ implementation experiences.

Estimating program impacts. To answer the first research question of interest, about the impacts of access to WIA intensive services and training on customers’ outcomes, the outcomes of customers in the three research groups will be compared. Specifically, the outcomes of members of the full-WIA group and the core-only group will be compared; the outcomes of the full-WIA group and the core-and-intensive group will be compared; and the outcomes of the core-and-intensive group and the core-only group will be compared. To answer the second research question, about impacts on subgroups of customers, the same type of analysis will be conducted but on various subgroups that are defined by baseline characteristics of individuals, such as sex and previous employment history. Finally, the third research question, about how the effectiveness of WIA varies by how it is implemented, will be answered by using subgroups of sites with similar program implementation as determined by the implementation study (described next).

Our design will provide unbiased estimates of the effects of WIA intensive and training services that generalize nationally to the WIA program as it operates during the time that study sample members receive program services. The impact findings may not necessarily generalize to program impacts during other time periods, because program features, the economy, and customer characteristics and needs may change over time.

Implementation analysis. The implementation analysis will examine how implementation of WIA Adult and Dislocated Worker programs varies across the LWIAs participating in the evaluation. This analysis has five major objectives:

1. Interpreting the net-impact estimates. Information on the implementation of each component of WIA will aid in interpreting the net-impact findings. It will inform judgments about the mechanisms through which the program affects outcomes. For example, an understanding of observed differences in outcomes between the core-only group and the core-and-intensive group requires an understanding of the availability, content, and intensity of WIA intensive services as well as of non-WIA services that each research group might access.

2. Interpreting differences in net-impact estimates by site. The implementation analysis will compare service delivery and the context in which the services are delivered across sites. Sites will be placed in groups with similar service delivery models and impacts will be estimated separately for each group. This analysis will allow an investigation of how impacts vary among sites with different approaches to the implementation of WIA services.

3. Documenting the program. Although the overall goals of WIA and its early implementation are well documented (D’Amico et al. 2004), the program is likely to have changed in the years since its implementation was last studied in detail. It is important to provide stakeholders, program operators and policymakers with a description of WIA as currently implemented, along with findings from the net-impact and benefit-cost analyses.

4. Documenting fidelity to the evaluation procedures. The implementation analysis will document the extent to which the sites are faithful to the evaluation procedures.

5. Estimating program costs. The implementation analysis will include the collection of cost data that will describe the program and the relative intensity of its service components. These data will also be used in the benefit-cost analysis.

Benefit-cost analysis. The benefit-cost analysis will assess whether the benefits of intensive services and training for adults and for dislocated workers exceed the costs of the services and training that are offered to WIA customers during the period covered by the study. It will use an accounting framework that itemizes all the incremental benefits and costs of these WIA programs and places a dollar value on as many of them as possible. Potential benefits of these programs include increased earnings, increased taxes, reduced use of UI and public assistance, and reduced use of other employment services. The documented costs will include the direct and indirect costs of each service component.

d. Overview of Data Collection

Collection of detailed data from multiple sources will be required to address the research questions. Table A.1 summarizes the data needs together with the sources of those data. It shows all the data collection activities including those for which we will request clearance in a future addendum to this package.

Table A.1. Summary of Data Collection Activities for the WIA Evaluation

Type of Data Needed	Reason Data Needed	Source	Whom Covered
Baseline information	Describe study participants Ensure that random assignment created groups with similar baseline characteristics Subgroup analysis Enhance the impact analysisa	SRF State UI agencies 15-month follow-up survey	All study participants All study participants 6,000 study participants
Services received	Describe services received by members of three groups Describe impact of WIA on the receipt of any employment and training services	State’s and/or LWIA’s management information system 15- and 30-month follow-up surveys	All study participants 6,000 study participants
Outcomes	Estimate the impacts of intensive services and training services	State UI agencies 15- and 30-month follow-up surveys	All study participants 6,000 study participants
Implementation data	Document and describe the implementation of WIA services Monitor the implementation of the evaluation	Site visits: interviews with LWIA staff, group interviews with customers, review of program documents, site observations State’s and/or LWIA’s management information system Published sources	All LWIAs in the study
Costs	Estimate costs of services for the benefit-cost analysis	Cost collection forms completed by local WIA staff and supplemented by interviews at site visit Accounting data on ITA obligations and expenditures	All LWIAs in the study

^a Described in detail in response to question 2.

Specifically, the evaluation requires five types of data:

• Baseline information. Information collected upon study enrollment will be used to provide a description of the characteristics of study participants at the time they are randomly assigned to a research group, conduct random assignment, ensure that random assignment has taken place correctly, define subgroups for the impact analyses, increase the precision of net-impact analyses, and adjust for survey nonresponse.

• Services received. Information on receipt of WIA-funded services will be used to monitor the random assignment process and ensure that participating customers are receiving only the services intended for their assigned research group. In addition, information on employment and training services received by study participants from non-WIA funding sources is needed to estimate the impact that the availability of WIA services has on the receipt of employment and training services.

• Outcomes. Outcome data are needed to estimate net impacts. The main outcomes of interest for the evaluation are related to labor market success—employment, earnings, and quality of job as measured by wage, hours worked, and fringe benefits. Outcomes will also include measures of attainment of skills, such as degrees or certificates, and measures of self-sufficiency such as the receipt of UI, Supplemental Nutrition Assistance Program (SNAP) benefits, and Temporary Assistance for Needy Families (TANF).

• Implementation. Data are needed on the community and organizational context in which programs operate, the types of services offered, how the services are provided, and how the evaluation was implemented. These data will provide the basis of the implementation analysis and will come from in-person interviews with staff who administer the program and customers who participate, as well as from program documents, observations of WIA activities, and states’ and/or sites’ management information systems.

• Costs. Data on the costs will be used to describe the program and to assess whether benefits of the various levels of services received exceed the costs of those services. Forms specifically designed to gather this information will be completed by site staff. In addition, because information about the details of WIA-funded training often is not available through sites’ management information systems, accounting data about ITA obligations and expenditures will be collected as well.

These data will be collected from five sources:

The SRF and the CIF. The SRF will request some baseline information from all consenting adults and dislocated workers at the time they would (in the absence of the study) be offered intensive services in the evaluation sites. The CIF will ask for detailed contact information for friends and relatives of the study participant to be used to locate the participant for the follow-up surveys. The SRF and CIF are provided in Appendix B and discussed fully in the response to question 2.

Two follow-up surveys. Follow-up telephone surveys will be attempted with 6,000 study participants (all 2,000 members of each of the two restricted-service groups and 2,000 randomly selected study participants in the full-WIA group).

The two surveys will be conducted at about 15 and 30 months after random assignment. The decision about when to conduct the first follow-up survey required a balancing of the need for accurate recall of service receipt with the ability to estimate short-term employment impacts. We decided to conduct the first follow-up survey at 15 months after random assignment because at that time nearly all study participants will have completed training funded by WIA. A previous study of customers who receive training under WIA found that it took 14 weeks on average to begin training after the decision was made to participate in training and then training lasted on average 25 weeks (McConnell et al. 2006).⁴ Hence, a survey much earlier than 15 months after random assignment would be too early to capture any impacts on completion of training, acquisition of credentials, or employment outcomes. In fact, the results may be misleading as the impacts on employment at that time will likely be negative as members of the treatment group are still receiving training. Our experience on previous studies of this population (such as the ITA Experiment, McConnell et al. 2006) has been that we can receive accurate information and response rates of over 80 percent at 15 months after random assignment.

The first survey will collect some baseline data that will not have changed since random assignment, such as place of birth, for purposes of verifying identity of respondents. Both surveys will collect data on customers’ receipt of services and customer outcomes on attainment of education credentials, labor market success, and family self-sufficiency. The follow-up survey instruments will be included in the second part of this OMB submission for this study.

WIA service and cost data. To monitor that random assignment is being implemented correctly as well as to collect data on the receipt of WIA services, data extracts from the state and/or local management information systems (MIS) will be requested. If data on all services are not regularly collected in the MIS the evaluation team will work with the sites to determine the best way to provide basic service data—whether it is from another MIS or from modifications to their MIS. Data on LWIA expenditures on the part of the LWIA during the study period will be collected through quarterly reports that the LWIAs routinely submit to ETA. In addition, data on the costs of each service (for example, staff time and cost, cost of materials, overhead) will be collected through cost collection forms and interviews with program staff during the second site visit. These cost collection forms will be included in the second part of this package, which we will submit later for OMB clearance.

Administrative data from other agencies and programs. Both baseline (such as past earnings) and outcome data on quarterly earnings and UI benefits will be collected from records of state UI agencies. Data on the service and benefit receipt may also be collected from the Employment Service, Social Security Administration, TANF program, and/or SNAP.

We do not currently plan to collect data from SSA, TANF agencies, or SNAP agencies. Past experience has shown that in some cases the findings of a study suggest the desirability of additional data collection. Hence, to have the option of collecting these data at a later date, we state on the consent form that we may obtain data from these agencies.

We plan to begin the process of developing agreements to collect UI data from the participating states shortly. As developing these agreements is a time-consuming and hence costly process, we intend only to collect UI data from the states participating in the evaluation. However, we will ask for data on all sample members in our study from each study state. Hence, if a customer moves from one study state (such as New York) to another study state (such as New Jersey), we will capture their earnings in the state to which they move. However, if a customer moves to a non-study state (such as Connecticut), we will not capture their earnings in the state to which they move.

Site visits. Qualitative data on the context for the program and its implementation will be collected during two rounds of site visits to each of the evaluation sites. The site visits will involve interviews with key staff, case file reviews, group interviews with customers, and observations of program activities. The protocols for these activities are provided in Appendix C.

2. How, by Whom, and for What Purposes Will the Information Be Used

Clearance is currently being requested for data collection that will be used to perform and monitor random assignment, conduct the implementation analysis and contribute to the impact analysis. Each data collection instrument is described in detail below, along with information about how, by whom, and for what purposes the information will be used. A subsequent submission to OMB will include a request for clearance for three additional data collection instruments: the 15- and 30-month surveys of sample members and the cost collection form.

a. Consent to Participate in the Study

This form will be administered to all eligible WIA customers in the selected sites by a WIA counselor at the point the customer would normally be determined eligible for intensive services. The counselor will ask the customer to read the form and answer any questions that the customer has prior to signing the form. Completion of the consent form, which will be returned to the evaluator, ensures that the customer has been fully informed about the WIA study, including random assignment, all data collection, and confidentiality of the data. It ensures that customers are aware that they may choose to decline to participate in the study, but if they do they are ineligible for WIA-funded intensive and training services for 15 months. Agreement to participate ensures that they are aware that the services available to them will depend on the outcome of the random assignment process. The consent form will be given to all WIA customers in the evaluation sites who are eligible for and interested in receiving intensive services and who are not exempted from the study (exempted WIA customers include TAA customers, veterans and covered spouses, and customers referred by employers for on-the-job training slots). The consent form is presented in Appendix A.

b. The Eligibility Checklist

The checklist has two purposes: (1) to assist counselors in determining whether a customer is eligible for the study; and (2) for the researchers to determine the number of people who are found ineligible for each reason. Counselors will complete this form (presented in Appendix B) for each customer who is found eligible for and is interested in receiving intensive services. The counselor will check whether the customer falls into a category of customer that is exempt from the study as well as whether the customer has signed the consent form.

c. The SRF and the CIF

The SRF will collect basic demographic and socioeconomic characteristics on all consenting customers prior to random assignment. The CIF will collect the name, address, phone number, and email address of up to three individuals who are close friends or relatives of the study participant and, most likely, will have knowledge of his or her whereabouts at the time of follow-up data collection. The specific items collected on the SRF and each item’s uses are described in Table A.2. Both forms are presented in Appendix B.

Baseline and contact data are needed for the following purposes:

1. To conduct random assignment. Some basic identifying information (name, date of birth, sex) is needed to conduct random assignment.

2. To monitor random assignment. Baseline information will be used to ensure that individuals go through the random assignment process only once. Counselors at participating sites will be asked to enter identifying information on all new consented customers into the RAS. This system will alert the counselor if the customer has already been randomly assigned. This ensures that individuals always remain in the same research group and that the research sample does not include duplicate cases. Baseline data will also be used to detect differences in the characteristics of customers across research groups, which would suggest a problem with the random assignment process. (Service data collected from the LWIA’s management information system will also be used to detect whether customers have been randomly assigned and that study participants do not receive services for which they are ineligible during the study.)

Table A.2. SRF Items and Their Uses

Item (Question Number)	Conduct Random Assign-ment	Monitor Random Assign-ment	Track for Data Collection	Subgroup Definition	Covariate	Nonresponse Adjust- ments	Estimate Impact on Service Recipients	Enhance Survey Adminis- tration
Customer
Date form completed (1)	X
Name (2, 2a)	X	X	X
Address, telephone numbers, and email address (3, 7, 8, 9)			X
Date of birth (4)		X	X	X	X	X
Social security number (5)		X	X
Gender (6)		X		X	X	X
Race/ethnicity (10, 11)				X	X	X
Primary language (12)								X
Marital status (13)				X	X	X
Household size (14)				X	X	X
Education (15)				X	X	X
Work limitations (16)				X	X	X
Past and current employment (17, 18, 19, 22, 23)				X	X	X
Former employer and occupation (20, 21)								X
Participation in benefit programs (24)				X	X	X
Use of One-Stop Career Center Services (25)				X	X	X
Counselor Use Only
Counselors name, center, and LWIA (A, B, and C)	X
Customer’s qualification status (D)				X		X
Counselor’s predictions about service receipt (E and F)				X			X

3. To locate participants for surveys and collect administrative data on participants. Detailed identifying information for each customer and detailed information on people who may know the whereabouts of the participant will assist in the location of sample members for follow-up surveys and increase the response rates to those surveys. The participant’s social security number is essential for obtaining administrative data—agencies typically provide data by matching on social security number. The social security number is also helpful in locating participants for follow-up surveys.

4. To define subgroups for the impact analyses. Baseline data on the characteristics of sample members are essential to define subgroups for which WIA impacts can be estimated. These include whether the customer is an adult or dislocated worker as well as subgroups defined by characteristics such as sex, race/ethnicity, disability status, and employment history.

5. To increase the precision of impact estimates. By including information on the baseline characteristics of study participants in regression models, the precision of impact estimates can be enhanced.

6. To adjust for nonresponse. With random assignment, simple differences in the mean outcomes between the research groups provide unbiased estimates of the impacts. However, systematic differences between the characteristics of members of the research groups might occur because of differential rates of survey nonresponse across the groups. To the extent that these characteristics are correlated with the outcome variables, this may lead to biased impact estimates. Baseline characteristics can be used to adjust for potential bias that may arise from survey nonresponse. The approach to adjusting for nonresponse is discussed in Part B of this package.

7. To estimate impacts on service recipients. The main impact estimates will be estimates of the offer of a specific array of WIA services or intent-to-treat (ITT) estimates. To rigorously estimate the effectiveness of WIA services for those who actually received them or treatment-on-the-treated (TOT) estimates, counselors will be asked to predict, in a designated section of the SRF, how likely the customer is to receive intensive and training services, if offered. These data will be collected prior to random assignment and, thus, will be available for the full research sample. To the extent that these predictions are accurate, they will be used to estimate the effectiveness of the receipt of specific WIA services by estimating impacts using the sample of those predicted to receive such services. Provided that the predictions are fairly accurate, this approach for estimating TOT impacts provides better statistical precision than one based on an assumption that there are no treatment effects for program nonparticipants.

8. To enhance the follow-up survey administration. Collecting information on current employer and occupation at baseline facilitates the collection of a complete work history at the time of the follow-up interview by enhancing customers’ recall of employment over the past 15 months. In addition, knowing the primary spoken language of a customer can facilitate administration of a survey in that language when possible.

Both the SRF and the CIF will be completed by all WIA customers in the evaluation sites who have been found eligible for intensive services, been found eligible for the study, and have given their signed consent to participate in the study. As with the consent forms, WIA counselors will ask the customers to complete the forms and will be able to answer questions that customers have about them. Alternatively, if the customer has literacy problems, the counselor can administer the form to the customer. The form will be translated in Spanish and other languages as necessary. The information collected on these forms will be entered into the RAS, either by program staff or by evaluation team staff.

d. Site Visit Guides

These guides will be used by site visitors to collect data pertinent to the implementation analysis. This information will be used for two main purposes:

1. To describe program context, operations, and service flow. Understanding what services customers in each of the research groups receive and how service receipt varies across sites constitutes one of the most important objectives of the implementation study. Although the overall goals of WIA and its early implementation are well documented, the program is likely to have undergone changes in the years since its implementation was last studied in detail (D’Amico et al. 2004). It is therefore important to provide policymakers with a description of WIA as currently implemented. Further, documenting and analyzing the current program will help in both interpreting experimental results and assessing the policy implications of the study as a whole.

2. To assess the integrity of evaluation conditions. The implementation study will investigate sites’ compliance with the evaluation’s random assignment procedures manuals. Specifically, it will investigate whether (1) all customers are informed about the evaluation and asked to provide study consent; (2) all customers are randomly assigned before receiving intensive services, and they are assigned to a research group only once; (3) all study participants are informed about the services they are entitled to receive under their assigned research group and in fact receive only those services; (4) study participants are not offered or referred to services that they would not receive in the absence of the study; and (5) the evaluation has affected program implementation. In addition, these data will be used to assess whether any changes occurred in the program as a result of the evaluation.

Description of site visits. Members of the evaluation team will visit each of the study sites twice, once around the beginning of the intake period and again near the end of the study intake period. Although the length of each site visit will depend on the complexity of the LWIA’s organizational structure, the average visit by evaluation team members to a site will be about four days.

We plan two rounds of site visits for four reasons. First, over the 12-month period between the first and second visit, changes will occur in the economy, funding, and WIA policies and services. Even over the period of time that we have been recruiting the sites, we have observed changes in operators, service availability, and procedures. These changes could affect the magnitude of the impacts observed and so are important to document.

Second, two rounds of site visits will allow us the time to visit more One-Stop Career Centers and hence collect more information. As shown in Table A.3, the number of One-Stop Career Centers varies by site, from 1 in Essex County to 29 in the Gulf Coast LWIA. Visiting more One-Stop Career Centers is important because the centers within an LWIA vary in the type of operator, the services they provide, the context in which they are provided, and the types of customers they serve.

Third, two visits will allow us to interview different staff in each round. As an example, in the first visit we propose to ask questions about the accounting of costs at the site. In response to these questions, we will tailor the cost collection protocols to the site. We will use these protocols to collect the cost data in the second visit.

Fourth, two site visits will allow us to more closely monitor the sites implementation of the study. During the first visit, we will ensure that the site is implementing the intake procedures correctly. At the second site visit, we will check that the site understands that they need to maintain the integrity of the research groups for the entire embargo period (15 months) even if their intake period is over.

Table A. 3. Number of One-Stop Career Centers in Each Site

Site	Number of One-Stop Career Centers
Fresno Regional Workforce Investment Area	6
Sacramento City/County Consortium	11
First Coast Workforce Investment Area	7
Atlanta Regional (Area 7)	7
Chicago	30
EmployIndy	3
Greater Louisville Workforce Investment Area	4
Orleans Parish Workforce Investment Area	3
Michigan Works! Muskegon/Oceana	3
Southeast Michigan Community Alliance Michigan Works!	6
Central Region Workforce Investment Area	19
Twin Districts Workforce Investment Area	12
Essex County Workforce Investment Area	1
Capital Region Workforce Investment Board	3
Chautauqua Works	2
New York City Workforce Investment Area	10
Central Pennsylvania Workforce Investment Area	6
Northwest Pennsylvania Workforce Investment Area	5
Southwest Corner Workforce Investment Area	4
Lower Savannah Workforce Investment Area	9
Santee-Lynches Workforce Investment Area	4
South Dakota Consortium	12
Tennessee Local Workforce Investment Area 4	9
Gulf Coast Workforce Board-Workforce Solutions	29
South Plains Regional Workforce Development Board	9
Workforce Solutions for North Central Texas	15
Workforce Development Council of Seattle-King County	7
Waukesha, Ozaukee and Washington (WOW) Workforce Development	3
Total	209

Each round of site visits will involve interviews with staff, group interviews with customers, observations of program activities, case file reviews, and collection of program documentation. Evaluation team site visitors will be thoroughly trained on the purpose of each activity, the information needed, and best practices in collecting the data. Each activity is described below.

Interviews. Interviews will be conducted with local WIA administrators (executive directors, adult and dislocated worker program managers, and/or One-Stop Career Center operators); WIA frontline staff who provide services to adult and dislocated worker customers; LWIA financial officers; LWIA staff knowledgeable of the management information systems; and Employment Services managers and/or staff from other partner programs. Topics to be discussed include the context in which the LWIA operates, the local governance and administration of the LWIA, the operation of the evaluation, the nature and quality of services provided, and system and customer outcomes. A summary of the information collected from each type of respondent is presented in Table A.4.

Collecting information on the same topic from multiple respondents is important because it allows for the triangulation of perspectives. Different staff members often will give different responses. For example, an LWIA administrator will provide a description of the LWIA’s policy, while the frontline staff will provide a description of how the LWIA’s policy is actually implemented. Frontline staff members are also more aware than senior staff of customers’ experiences, the local labor market, other services available in the community, service provision, and problems in service provision. They are also more likely to provide an answer that is not “spun.” We plan to ask frontline staff about staffing, because they will be more likely to have noticed staff shortages than LWIA administrators, the nature of services as they are actually provided (rather than as administrators plan for them to be provided), and how customers are affected by services.

Table A.4. Topics by Interview Respondent

	LWIA Administrators
Research Topic	LWIA Executive Directors	Adult and Dislocated Worker Program Managers	One-Stop Career Center Operators	WIA Frontline Staff	LWIA Finance Officers	LWIA Staff with Knowledge about the Management Information System	Employment Services Managers and/or Staff at Other Programs	Customers
Context
Labor market and economic outlook	X	X	X	X			X	X
Workforce/customer composition	X	X		X			X
Education and training opportunities	X	X		X
Local Governance and Administration
One-Stop system organization	X	X	X
WIA partnerships	X	X	X		X
Local WIB	X
Staffing	X	X	X	X	X		X
Recruitment and enrollment	X	X	X
Nature of Services
Policy guidance/framework for service delivery	X	X	X	X			X
Enrollment/intake and orientation	X	X	X	X			X	X
Self-service core services/resources and tools	X	X	X	X			X	X
Staff-assisted core services	X	X	X	X			X	X
Intensive services	X	X	X	X				X
Training services	X	X		X				X
Case management, placement, and follow-up services	X	X		X				X
Supportive services	X			X
Assessment of services	X			X				X
Other services in the community	X	X	X	X			X	X
Customer Outcomes
Common measures/performance outcomes	X	X	X
Customer satisfaction	X	X	X				X	X
Workforce system changes	X	X		X

Table A.4. (continued)

	LWIA Administrators
Research Topic	LWIA Executive Directors	Adult and Dislocated Worker Program Managers	One-Stop Career Center Operators	WIA Frontline Staff	LWIA Finance Officers	LWIA Staff with Knowledge about the Management Information System	Employment Services Managers and/or Staff at Other Programs	Customers
Customer employment opportunities	X	X	X	X				X
Operation of the Study
Random assignment process	X	X	X	X			X	X
Maintaining integrity of research groups	X	X	X	X
Problems encountered		X		X
Service changes since the study began	X			X			X
Costs of the study					X
Financial Information and Management Information Systems
WIA allocation					X
Program expenditures					X
Program costs for different service groups					X
Data collection						X
Data use and reporting						X

The site visit guides list topics and suggested questions to be discussed by the evaluation team during in-person interviews with WIA staff. They are not meant to be administered word-for-word but used as a guide to topics to be covered during the interviews. They will ensure that all site visitors capture the necessary data consistently across sites, but allow for flexibility for site visitors to pursue interesting issues that arise and skip questions for which they already know the answer from previous contacts with the sites or from program documentations. Some topics will be discussed with multiple LWIA staff; doing so will provide information about topics from multiple perspectives. Guides for each key category of respondent are included in Appendix C.

Group interviews with customers. Informal group discussions with a sample of One-Stop Career Center customers who are in the study will be conducted to determine their perspectives on the accessibility and quality of services. A mix of customers who receive intensive services only and intensive services and training will be recruited for the group discussions. A protocol for these discussions is presented in Appendix C.

Observations of program activities. Activities, such as customers using the resource room and staff delivering group or one-on-one services, will be observed, with salient pieces of information recorded. Observing program activities will provide the evaluation team with detailed information about how an activity is actually implemented, which may differ from the description of its implementation given by staff. A protocol for recording information on an observation is provided in Appendix C.

Review of a small sample of individual case files. About four case files will be reviewed during each site visit. These files will be for four randomly selected customers in the study. Two will be for customers in the full-WIA group and two will be for customers in the core-and-intensive service group. (Customers in the core-only group are unlikely to have case files.) These reviews will provide concrete examples of the experiences of WIA intensive and training services customers, including services received and the way in which customers progressed through services. This will also be an opportunity to check whether participants in each research group have received only the WIA services allowed for that research group. Protocols used to conduct these reviews are provided in Appendix C.

Review of program documents. Program documents including annual plans, reports, staff manuals, and brochures and other materials provided to customers will be collected before and during the site visits. These documents may provide detailed information about program context and implementation.

3. Use of Improved Technology to Reduce Burden

Advanced technology will be used in the data collection efforts to reduce burden on program participants and on staff at participating agencies. All WIA counselors involved in the study will have access to the study-specific management information system, the RAS. The RAS is a web-based system that can be accessed from the counselor’s desktop with a username and password. It will be used to conduct random assignment, produce reports on the group assignment of study participants, produce letters that can be given to customers that describe their assignments, and allow counselors to search for whether a customer is already participating in the study. To conduct random assignment, counselors will need to enter the responses to questions 2, 3, 4, and 5 and the “counselor-only” questions D, E, and F into the RAS. The system will not conduct random assignment if the customer has previously been randomly assigned. To minimize data entry burden on the counselors, only this minimum set of data items on the SRF will be entered into the RAS. Hard copies of the forms will be sent to the evaluator. The evaluator will scan the forms to create electronic databases.

4. Efforts to Identify Duplication

To minimize duplicate data collection, the SRF has been reduced to items necessary to the evaluation. Only a limited amount of descriptive information is expected to be available from the data collected electronically by programs as part of their normal intake procedures. These existing data likely do not contain all the baseline characteristics of customers necessary nor will they be consistent across sites. If discussions with sites lead to a finding that all sites collect an item on the SRF in a consistent manner, and this information is not necessary to conduct random assignment, the item can be dropped from the SRF. To avoid duplication with administrative UI data, items related to earnings or history of earnings are not included in the SRF.

The detailed contact information for the participant and the data to be collected during the site visits are not available from any other source. Nor is the participant’s consent to participate in the evaluation available elsewhere.

5. Methods to Minimize Burden on Small Businesses or Entities

This data collection does not involve small businesses or other small entities.

6. Consequences of Not Collecting the Data

The data collection plan described in this submission is necessary for conducting this evaluation to the standards mandated by Section 172 (c) of the WIA. The plan will enable the WIA Evaluation to generate nationally representative, precise, unbiased estimates of the impacts of WIA intensive and training services. Results from this rigorous evaluation will inform policymakers about net impacts for WIA participants, the context within which the program operates, and whether the program is cost effective.

Without collecting baseline information on study participants, the study’s ability to implement random assignment correctly and ensure that random assignment was conducted appropriately would be severely limited. The lack of baseline information would limit the ability to describe the population of WIA customers and would limit the analysis of impacts of the program on subgroups, hence limiting the ability to determine the groups for which the program is most effective. Without baseline data, impact estimates would be less precise (so that small impacts would be less likely to be detected), and adjustments for nonresponse to the follow-up surveys would have to be based on less detailed administrative data.

Without collecting detailed contact information for study participants, the study’s ability to track participants over a 30-month follow-up period would be limited. This would likely lead to a higher nonresponse rate and thus a greater risk of compromise to the quality of survey data and hence the impact estimates.

Without collecting the information specified in site visit guides, an implementation analysis of the WIA program could not occur. This would prevent information being provided to policymakers about the context in which programs operate and any operational challenges faced by programs. Lack of implementation data would also prevent an examination of how the impacts of the services vary by how they are implemented and the context in which they are implemented. Without the site visits, there is also a greater chance that any deviations from study procedures would go undetected.

7. Special Data Collection Circumstances

In all respects, evaluation data will be collected in a manner consistent with Federal guidelines. There are no plans to require respondents to report information more often than quarterly, to submit more than one original and two copies of any document, to retain records, or to submit proprietary trade secrets.

8. Federal Register Notice

a. Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995, the public was given an opportunity to review and comment through the 60-day Federal Register Notice, published on November 10, 2010 (Federal Register, volume 75, no. 217, pp. 69126-69128). A copy of this notice is attached as Appendix D.

DOL received four sets of comments in response to this notice. These comments and the Department’s responses are presented in Appendix E. A summary of the comments and responses is presented in Table A.5.

As a result of these comments we made the following changes to the consent form:

• We changed the wording of the first sentence on the consent form from “a study of the nation’s employment and training programs” to “a study of some of its employment and training programs that serve adults and dislocated workers.”

• We changed the description of the core-only group to reflect that customers in the group will not have access to WIA services that require “substantial staff time.”

No other changes to the forms or protocols were made as a result of these comments.

In response to the comment about the burden of tracking ongoing customers, the burden estimate was increased. Tracking each customer when he/she comes back into the office is expected to add about 1 minute per customer. Assuming each customer comes back twice on average this adds 2,267 hours in total (=68,000 customers x 2 minutes / 60 minutes).

Table A.5. Summary of Public Comments and Agency Responses

Summary of Comment	Agency Response
Several commenters questioned the need for the evaluation to include random assignment.	Random assignment allows a comparison of a group of people who are offered services with a group who are similar on average in every characteristic, observable and unobservable, except that they are not offered services. This allows the researchers to isolate the effect of the services. If other approaches are used, the impact of the services may be confounded with differences in the characteristics of the group who received the services and the characteristics of the group who did not.
Several commenters questioned the ethics and legality of restricting services to customers for the purpose of a study.	The study is ethical because the workforce system cannot serve all people who are eligible for, and could benefit from, intensive and training services. During the study, the same amount of intensive and training services will be provided. The main difference is that during the study the people who receive services will be determined randomly rather than by circumstances such as the timing of when they come into the One-Stop Career Centers for services. The WIA legislation (PL 105-220, Sect 172c) required that “evaluations conducted under this section utilize appropriate methodology and research designs, including...random assignment” Moreover, receipt of intensive and training services are not entitlements.
Several commenters were concerned about the interaction of the WIA Evaluation with other studies such as DOL’s evaluation of the Reemployment and Eligibility Assistance (REA) program, DOL’s Regression Model Pilot, and other state-specific studies.	In each selected site, we have examined whether the WIA Evaluation will interact with existing studies. We have determined that there will be an interaction with the REA evaluation in a few sites. In these sites, to avoid contamination of the REA evaluation, we will exempt members of the REA treatment group from the WIA Evaluation. Any customers who are in the REA control group and the WIA Evaluation restricted-service groups will be removed from the REA control group. These actions will preserve the validity of both studies.
One commenter wanted clarification of the objective of the study and whether it was an attempt to understand the One-Stop system.	The purpose of the study is to evaluate intensive and training services for customers in the Adult or Dislocated Worker programs. The study will not evaluate the One-Stop system as a whole. The implementation study does try to understand the One-Stop system to provide context for the impacts found.
One commenter suggested DOL wait to conduct the study until after WIA reauthorization.	DOL chose to not wait for WIA reauthorization to conduct the study for three reasons: (1) WIA legislation required a study be conducted within by the end of fiscal year 2005—so the study is already overdue; (2) as the timing of reauthorization is unknown, planning for the study could be delayed for many years; and (3) if reauthorization takes place during the study, it may provide an opportunity to compare the impacts of the services prior to and after reauthorization.
One commenter suggested that DOL evaluate the role of the sectoral initiatives and services provided to businesses.	Studying sector initiatives and services provided to businesses is beyond the scope of this study.
One commenter noted that the study would increase burden by requiring the state or local agency to maintain hard copies or enter additional data into the site’s existing management information system.	All hard copy forms will be shipped to the contractor. Copies of forms do not need to be maintained by the local or state offices. Only eight items from the SRF are required to be entered into the random assignment system. Only information on the customer’s group assignment will be entered into the site’s existing management information system.
One commenter noted that the burden estimates did not take into account the burden of tracking ongoing customers.	The commenter is correct and we have increased the burden estimate accordingly. During the study, staff will need to check whether a customer who comes in for repeat services is already in the study, and if they are, identify the group to which they are assigned. We will work with LWIAs to determine the best approach for staff to make this determination given their current procedures. One or more of the following procedures could be used to determine whether the customer is in the study and the group to which they are assigned: (1) checking on the random assignment system; (2) including a note in the customers hard-copy file; and/or (3) creating new codes on the site’s existing management information system. We have increased our estimate of the burden on staff by 2,267 hours to account for this burden.

Table A.5 (continued)

Summary of Comment	Agency Response
One commenter asked how we propose to measure the benefits of the programs.	Benefits that accrue to participants, the government, and society as a whole as a result of increases in earnings (including fringe benefits and taxes) and reduced use of public assistance programs, including changes in the use of unemployment compensation, will be measured. The benefits will be estimated from the impacts on earning, use of public assistance programs, and UI receipt. We propose to measure the impacts on customers for 30 months after random assignment.
The commenter also asked how we propose to measure the costs of the programs.	Our cost estimates will be derived from aggregating the component costs of the services. The cost of intensive services will be estimated from information on (1) the average time spent by counselors providing case-management and assessments and (2) the cost of their time (including wage rate, fringe benefits, supervisor’s costs, and overhead). The information on time spent in each service will be collected by asking the staff to keep a timesheet during a specified week. The information on fringe benefits, supervisor costs, and overhead will be collected using the cost protocols that will be submitted in the addendum to this clearance package. The cost of training will include the full price charged by the training provider as well as the cost of any time taken by the counselors in monitoring the training. Data on the price of the training will be obtained from the local areas.
One commenter asked about the approach to estimating the impacts on the receipt of services rather than the offer of services.	To create unbiased estimates of impacts for program participants, we will adjust the impacts of offering the services using statistical methods that have been developed in the research literature over the last 25 years (Angrist, Imbens, and Rubin 1996; Bloom 1984). These methods yield unbiased impact estimates for those who actually receive services under minimal assumptions.
One commenter asked whether the evaluation would interfere with the requirement to administer the work test to UI customers.	The requirement to administer the work test to UI customers will be unaffected by the study.
Several commenters asked about exemptions from the study, including veterans and TAA participants.	Veterans and TAA participants will be exempted from the study.
One commenter asked about how economic factors will be taken into account in the analysis.	The study sample will contain about 30 randomly selected LWIAs that are geographically dispersed. Thus, there will be a considerable range of labor market conditions faced by customers in the sample. In the impact analysis, we will estimate impacts for subgroups defined by the customers’ local labor market conditions and examine the variation in these impacts. We will also examine how impacts change over time as the economy changes.
One commenter noted that the introduction to the consent form may give the impression that the study is evaluating all the employment and training programs and that the description of core services was inaccurate.	The sentence in the consent form describing the purpose of the study was changed.
One commenter was concerned about duplicate data entry and collection.	We acknowledge that some of the information collected on the and the CIF is currently collected in some site’s case management systems. It is important that we collect the same data across all sites. Hence, unless the case management system can provide responses to exactly the same questions, customers need to complete the SRF. (LWIAs that do collect information on the responses to the exact questions using their case management system can provide the evaluator these responses electronically.) The information collected on the SRF is minimal and is essential for random assignment, to locate the sample member for a telephone survey, and for the subgroup and survey nonresponse analysis.

Table A.5 (continued)

Summary of Comment	Agency Response
	The site staff will not enter all the information on the SRF or any of the CIF into a database. The only data that staff will enter into the random assignment system are the answers to questions 2, 3, 4, and 5, and the “counselor use only” questions D, E, and F on the SRF. The contractor will scan the responses to the remainder of the questions from the hard-copy form.
One commenter was concerned that the consent form was not comprehensive in its list of core services.	The consent form is designed to be administered across all LWIAs and One-Stop Career Centers in the study. While we acknowledge that some sites include a wide variety of services as core services, many LWIAs in the study do not include all these staff-assisted services as core. Based on recent visits to study LWIAs, use of the resource room and Internet searches are two examples of core services available in all the One-Stop Career Centers in our study. We have changed the description of core-only services to reflect that customers will not have access to WIA services “that require substantial staff time.” We also will work with each site to develop a list of services that are core, intensive, and training in their particular site, which will be given to customers.
One commenter asked for clarification of the purposes of the interviews of the WIA administrator, finance staff, and Employment Service (ES) manager.	The research team envisions interviewing several types of individuals under the broad category of WIA Administrator, including the Executive Director, One-Stop Career Center Managers, and the WIA Program Manager. These staff will provide information on the type and intensity of services provided and the programmatic and economic context in which they are provided. We want to estimate the costs of the services received by those in the three research groups—the core-only group, the core- and-intensive group, and the full-WIA group. An addendum to this package will provide details of how we propose to collect cost information. The finance staff protocol is designed to collect information on the financial information the LWIAs do collect and will help us tailor the cost protocols to each site. The evaluation team is interested in estimating the impact of WIA-funded intensive and training services in comparison to core services. To sensibly interpret these impacts, we need to have a full understanding of the nature of the services that customers receive in all three groups, including core services. Since, in many LWIAs, ES will provide much of the One-Stop Career Center core services, we will need to gather information about the nature and content of ES services so that we can properly understand the services offered to those in the core-only group.
One commenter questioned whether the study undermines local control.	During the study, workforce service areas can continue to identify who is eligible for and would benefit from services. The study will not affect this process. The only difference is that during the study a small proportion (between 1 and 16 percent) of these customers will be assigned to a group in which services are restricted.
One commenter was concerned about how the study would affect performance standards.	Because the percentage of all customers in an LWIA who are assigned to a restricted service group will be very small, LWIA’s performance measures are unlikely to be affected.
One commenter was concerned about the effects of staff sending a customer to another One-Stop Career Center.	All workforce centers in a LWIA will participate in the study. Hence, if customers move from one center to another, they will remain in the group to which they were assigned. Furthermore, we are asking states in the study to indicate in their management information system if a customer is in the study and to ask LWIAs neighboring our study LWIAs to maintain any service restrictions placed on the customers in the study.

b. Consultations Outside of the Agency

Consultations on the research design, sample design, and data needs are part of the study design phase of the WIA Evaluation. The purposes of these consultations are to ensure the technical soundness of the study and the relevance of its findings and to verify the importance, relevance, and accessibility of the information sought in the study.

The following individuals were consulted in developing the design, the data collection plan, and the three sets of forms for which clearance is requested—the eligibility checklist, the consent to participate form, the SRF and the CIF, and the site visit guides.

Mathematica Policy Research

Dr. Kenneth Fortson (510) 830-3711

Dr. Sheena McConnell (202) 484-4518

Ms. Julita Milliner-Waddell (609) 275-2206

Dr. Karen Needels (541) 753-0201

Ms. Patricia Nemeth (609) 275-2294

Ms. Linda Rosenberg (609) 936-2762

Dr. Peter Schochet (609) 936-2783

Social Policy Research Associates

Dr. Ronald D’Amico (510) 763-1499 (x628)

Ms. Kate Dunham (510) 763-1499 (x635)

Mr. Jeffrey Salzman (510) 763-1499 (x629)

Dr. Andrew Wiegand (510) 763-1499 (x636)

9. Respondent Payments

Customers will not be paid for completing the intake forms (the consent form, the SRF, and the CIF) required to participate in the evaluation. Staff will not be compensated for completing the eligibility checklist or interviews conducted during the site visits. LWIAs are compensated for the costs associated with participating in the study, with the amount of compensation depending on the expected number of customers who will be randomly assigned.

10. Confidentiality

Study researchers have a strong set of methods to ensure that the confidentiality of data is protected. They consist of policies related to confidentiality, physical and technical safeguards, and approaches to the treatment of personally identifying data.

a. Confidentiality Policy

All Mathematica and subcontractor study staff will comply with relevant policies related to secure data collection, data storage and access, and data dissemination and analysis. Mathematica’s security policy meets the legal requirements of The Privacy Act of 1974 (System of Records Notices DOL/ETA-15); the “Buckley Amendment,” Family Educational Rights and Privacy Act of 1974 (FERPA); the Freedom of Information Act; and related regulations to assure and maintain the confidentiality of program participants.

Prior to random assignment, all potential program participants will be told about the evaluation. Potential program participants will also be given information about confidentiality protection in the consent form that they will be asked to read and sign before being randomly assigned to a research group (see Appendix A). The information will introduce the evaluation team conducting the evaluation, explain random assignment and the research groups, explain that study participants will be asked to participate in voluntary telephone interviews, and inform participants that administrative records about their earnings and the services and government benefits they receive over the following ten years may be released to the evaluation team. Participants will be told that all information they provide will be treated confidentially and used for research purposes only. Further, they will be assured that they will not be identified by name or in any way that could identify them in reports or communications with DOL unless the law requires otherwise. The statutory authority for DOL to promise confidentiality is the Privacy Act of 1974 (5 USC 522a), Systems of Records Notices DOL/ETA-15. Researchers from Mathematica and its subcontractors who play a role in data collection and analysis will be trained in confidentiality procedures and will be prepared to describe these procedures in full detail, and to answer any related questions raised by WIA staff and participants. If asked how participation might affect their access to services and benefits, potential participants will be given information about the three study groups into which they could be randomly assigned and the services that are available to each group. Beyond that, they will be told that their participation in the study will not affect past or future eligibility for any government-sponsored programs.

Access to all data that identify respondents will be limited to staff at Mathematica and its subcontractors who have a data collection or analysis role in the project. Such data will be needed for assembling records and assuring data alignment. Any data sent to ETA will not contain personal identifiers nor any other identifier that would allow individual identification of study participants.

It is Mathematica’s policy to efficiently protect confidential information and data in whatever medium they exist, in accordance with applicable federal and state laws and contractual requirements. In conjunction with this policy, all Mathematica and subcontractor staff shall:

1. Sign and comply with the Mathematica Confidentiality Pledge and with the Mathematica Security Manual procedures to prevent the improper disclosure, use, or alteration of confidential information. Staff may be subjected to disciplinary and/or civil or criminal actions for knowingly and willfully allowing the improper disclosure or unauthorized use of confidential information.

2. Access confidential and proprietary information only in performance of assigned duties.

3. Notify their supervisor, the project director, and the Mathematica security officer if confidential information has been disclosed to an unauthorized individual, used in an improper manner, or altered in an improper manner. All attempts to contact project staff about any study or evaluation by individuals who are not authorized to access the confidential information will be reported immediately to both the cognizant Mathematica project director and the security officer.

As part of their contract with DOL, all regular status and on-call staff who have access to personally identifying information will adhere to all DOL security requirements, including fingerprinting and background checks.

b. Confidentiality Safeguards

Mathematica has established safeguards that provide for the confidentiality of data and the protection of the privacy of the sampled individuals on all of its studies. Safeguards to ensure the confidentiality of data include:

1. Facility. The doors to office space and the survey operations center (SOC) are always locked, and all SOC staff is required to display a current photo identification while on the premises. Visitors are required to sign in and out of company offices and are required to wear temporary identification badges while on the premises. Any network server containing confidential data is in a controlled-access area. All authorized external access is through a protected Internet network that is under strict password control.

2. Network. Data stored on network drives is protected using the security mechanisms available through the network operating system used on Mathematica’s primary network servers: Novell Netware 5–6.5. These versions of Novell Netware are compliant with the C2/E2 Red Book security specifications. Netware is certified at the National Computer Security Center’s Trusted Network Interpretation Class C2 level of security at the network level. The network is protected from unauthorized external access through the PIX Firewall from CISCO. This firewall resides between the network and the communications line over which the corporate Internet traffic flows. Access to all network features such as software, files, printers, Internet, email, and other peripherals is controlled by userid and password. Network passwords must be a minimum of eight characters in length and must be a combination of numbers and letters. All userids, passwords, and network privileges are revoked within one working day for departing staff and immediately for terminated staff. All staff members are required to log off the network before leaving for the day.

3. Printers. Printer access is granted to all staff with a valid userid and password. The physical hard disks on which the printer queues reside are subject to the same security/crash procedures that apply to the file servers. Print stations are monitored appropriately depending on the sensitivity of the printed output produced. No confidential or proprietary data or information may be directed to a printer outside of Mathematica’s offices.

4. Electronic communication. Ethernet is used for internal email communications over the network. As Ethernet communications use Novell Netware with built-in userid and password protections and Windows NT Challenge Handshake Authentication Protocols (CHAPs), sensitive information in both email text and attachments may be safely transmitted. Email transfer is also encrypted when sent to or from the Mathematica gateway facility, which allows staff to check and send emails from home. A dedicated private line supports cross-office communications between Mathematica offices.

c. Treatment of Data with Personal Identifying Information

All data containing personally identifying information, including social security number, name, home address, and home telephone number, are considered to be sensitive or confidential, project-specific WIA data. Specific details regarding the handling and processing of confidential, project-specific WIA information in this evaluation is provided next.

1. Access. Confidential electronic files will be stored in restricted access network directories. Access to restricted directories is limited on a need-to-know basis to staff who have been assigned to and are currently working on the project. When temporarily away from their work area, project staff is to close files and applications. Access to workstations will automatically lock within a set period of minutes, and staff must use a password to regain access through the protected screen saver.

2. Electronic communications. Although the protections offered by internal email are extensive, staff members are instructed not to transmit sensitive information as a regular file attachment to an internal email. Instead, staff members are instructed to use the insert shortcut feature in Outlook to include a shortcut to the file. This allows the receiver to go to the file directly but will not allow access to unauthorized individuals. Additionally, staff members are instructed not to include sample members’ names or other personal identifying information in internal emails so that there is no potential for these to be viewed by others. When information about a sample member is transmitted via email, a Mathematica identification number is used as a reference. To ensure the security of sensitive information sent outside of Mathematica through an email, the sender is obligated to ensure that the recipient is approved to receive such data. When files must be sent as attachments internally or outside of Mathematica, staff are instructed to use WinZip 9.0 (256-bit AES encryption) to password protect the file. When sending sample member name and contact information outside of the company, this information will be included in a secure attachment rather than in the text of the email.

3. Databases. The databases developed for this study containing confidential information will be password protected and accessible only to staff who are currently working on the project. To access the database, users will first log on to their workstations and then to the database using a separate log-in prompt. The database will be removed and securely archived at the end of the data-processing period.

11. Questions of a Sensitive Nature

The SRF will collect background information on WIA customers who have consented to participate in this evaluation. Information on date of birth, address, and telephone numbers is needed to identify and contact participants. The SRF also collects information on characteristics of participants, such as sex, race/ethnicity, marital status, and education level, which is used to ensure that random assignment was conducted correctly and/or to enhance the impact estimates. This type of information is routinely collected as part of enrollment in most programs and is, therefore, not considered sensitive.

The SRF contains four potentially sensitive items—the question requesting social security number (question 5), the question on work limitations (question 16), the question on recent pay rate (question 23), and the one on participation in government-sponsored programs (question 24). The social security number is important to be able to identify and track customers and collect administrative data—no other unique customer-identifying number is available. Customers already provide their social security number to the LWIA in order to receive services. Customers will be assured that this information will be treated confidentially and will be asked for only once. Although potentially sensitive, the questions on work limitations, pay, and participation in government programs have also been routinely used in surveys with no evidence of harm. This information will provide insights about who participates in the WIA program and whether WIA intensive and training services are more effective for some customers compared to others. Pilot test participants did not object to answering questions on the intake forms.

As described earlier, all participants will be provided with assurances of confidentiality prior to random assignment and the completion of study enrollment forms. Not all data items need to be completed. All data will be held in the strictest confidence and reported in aggregate, summary format, eliminating the possibility of individual identification. The requirements of The Privacy Act of 1974 (System of Records Notices DOL/ETA-15) will be complied with in collecting all information.

12. Hour Burden of the Collection of Information

The time burden for administering the consent form, the SRF, and the CIF is estimated at 13 minutes for each customer and approximately 13 minutes for each staff member. A pilot test of the three intake forms—the consent to participate form, the SRF, and the CIF—were conducted to determine burden estimates for customers. Seven customers at the Middlesex County One-Stop Career Center in New Brunswick, New Jersey, participated in the pilot test on March 29, 2010. These WIA customers took an average of 13.14 minutes to complete all three forms. Customers were asked to time each form separately. The consent form took an average of 3.57 minutes to complete, the SRF averaged 5.43 minutes, and customers completed the CIF in 4.14 minutes, on average. A memo detailing the pilot test experience is included as Appendix F.⁵

The staff time to process the intake forms will consist of two types of activities: (1) review of the information provided by the customer and completion of the eligibility checklist and counselor-only section of the SRF and (2) the data entry of the SRF into the study’s RAS. The first type of activity was not pilot tested because of its study-specific content and lack of relevance to Middlesex County One-Stop Career Center staff. However, the time for staff to review the forms completed by a customer, to complete the “counselor-only” section of the SRF, and to tell customers of their research assignment is estimated to be 10.5 minutes. The majority of the information that counselors are asked to complete on the SRF is known to them and does not require any research.

For the second type of activity, data entry of the SRF into the RAS, an estimate of the burden was gained through the data entry of the pretests conducted at the Middlesex County One-Stop Career Center in New Brunswick, New Jersey. Staff at Mathematica entered the SRF data collected from the seven pretest participants to provide estimates of how long the data entry would take for study sites. On average, staff took 7.86 minutes to enter data from the full SRF. Regular staff was used for this timing exercise rather than data entry staff to more closely approximate the data entry skill level of One-Stop Career Center staff. Also, the forms were entered using an MS Word document, not a data entry platform, which will be available through the RAS. Use of the RAS for data entry is expected to decrease data entry time. As a result of the pretest, it was decided that staff should enter only eight items on the SRF into the RAS. A contractor will scan in the remainder of the data from the SRF. Thus, we estimate that data entry of the eight items will take about 2.5 minutes.

The estimated total hour burden of collecting information at intake is 28,166 hours (Table A.5). About 65,000 customers (in the 28 sites that agreed to participate) will complete the three intake forms. Each customer will take an average of 13 minutes to complete the forms. Hence, the total time for customers to complete the forms is 65,000 x (13/60) hours, which is equal to 14,083 hours. About 270 staff will complete the SRF for about 241 customers each.⁶ An LWIA staff person will take about 13 minutes per customer (10.5 minutes for the forms and 2.5 minutes for data entry). The total staff time for intake is 65,000 x 13/60 = 14,083 hours.

During the study, staff will need to check whether a customer who comes in for repeat services is already in the study, and if they are, the group to which they are assigned. We expect that collecting this information will add about one minute for every repeat customer. Assuming each customer comes back twice on average this adds 2,167 hours in total (=65,000 x 2 / 60).

The burden cost for site visits is 1,418 hours. The site visits will involve discussions with an average of four customers during each of two visits in each of 28 sites. Hence, about 224 (= 4 x 2 x 28) customers will be involved in the discussions. Each discussion will last about one hour, so the total customer burden of the site visits is 224 hours. Our plan is to visit, across the first and second round of visits, all centers in the LWIA in sites with fewer than 6 centers, 9 centers in the two sites with more than 20 centers (Gulf Coast and Chicago), and 6 centers in all the other sites. As we will visit 143 of the 209 centers in the study, the number of center respondents will be 858 (=143 centers x 6 staff at each center) over the two visits. The number of respondents at the LWIA level is 168 (=6 staff x 28 sites), Each LWIA staff member will be interviewed twice, once at each visit.

Table A.5. Burden Estimates for WIA Customers and Staff

Respondents	Number of Respondents/ Instances of Collection	Frequency of Collection	Average Time Per Response	Burden (Hours) for Customers	Burden (Hours) for Staff	Total Burden (Hours)
Enrollment Forms
Customers	65,000	Once	13 minutes	14,083
Staff	270	Once	13 minutes per customer, with an average of 240.73 customers per staff respondent		14,083
Total for Enrollment Forms						28,166
WIA Customer Return Visits After Random Assignment
Staff	270	Twice	1 minute per customer, with an average of 240.74 customers per staff		2,167
Total for Return Visits						2,167
Site Visits
Customers	224	Once	60 minutes	224
LWIA Staff	168	Twice	60 minutes		336
Center Staff	858	Once	60 minutes		858
Total for Site Visits						1,418
Total		--	--	14,307	17,444	31,751

Note: Due to rounding, the numbers for the totals may differ from the sum of the component numbers.

The total burden for the site visits is hence 1,418 hours, which is the sum of the burden hours for customers and staff.

The estimated total number of responses is 67,228 and the total burden for the data collection included in this request for clearance is 31,751 hours, which equals the sum of the estimated burden for the intake forms, the burden for identifying whether repeat customers are already in the study, and the estimated burden for the site visits.

The total estimate of cost for the respondents is approximately $426,440. As noted above, the total estimate of burden on staff for this data collection effort totals 17,444 hours (14,083 + 2,167 + 336 + 858). At an average hourly wage of $18.50, the cost estimate for this staff burden is $322,714. (The hourly wage of $18.50 for staff is the average wage of staff found in the ITA Experiment (McConnell et al. 2006)). Also as noted above, the total estimate of burden on customers is 14,307 hours (14,083 + 224). At an average wage of $7.25 per hour—the Federal minimum wage—the cost estimate for this customer burden is $103,726. (The minimum wage was used as the opportunity cost of the customers—most of whom are unemployed.)

13. Estimated Total Annual Cost Burden to Respondents and Record Keepers

The proposed data collection will not require the respondents to purchase equipment or services or to establish new data retrieval mechanisms. Therefore, the cost to respondents solely involves answering the questions on the survey. Similarly, the contact data being requested from research group members will be known to those individuals and readily available for completing the forms. No capital or start-up costs are anticipated. Nor do we expect respondents to spend extensive time generating, maintaining, disclosing or providing the information.

14. Estimated Annualized Cost to the Federal Government

The total cost of the study to the Federal government is $23,311,040. Of this, $22,951,040 will be paid to contractors to conduct the study. Of the contractor amount, about $1.463 million is for design and planning, $2.498 million is for site recruitment, $4.433 million is for payments to sites and states as compensation for staff time spent on the study, $2.176 million is for training site staff and providing technical assistance throughout the study, $10.084 million is for data collection, and $2.297 million is for analysis and reporting. An estimated $360,000 (three staff-year equivalents) will be spent by DOL staff managing the study and overseeing the contractor. Since the study will last eight years, the annualized cost to the Federal government is $2,913,880.

15. Changes in Burden

This is the first submission for the WIA Evaluation. It is a one-time request and will count as 34,134 hours towards ETA’s Information Collection Burden. The second submission to OMB for this study, which will cover the collection of cost data and customer follow-up surveys, will increase the burden for staff and customers.

16. Publication Plans and Project Schedule

Three reports will present findings from the WIA Evaluation: (1) a report on the implementation analysis (available winter 2012/2013), (2) a report on the 15-month net impacts of WIA intensive and training services for adults and dislocated workers (available spring 2015), and
(3) a final report on the 30-month net impacts and cost-effectiveness of those services (available summer 2016).

Table A.7 shows the schedule for the evaluation.

17. Reasons for Not Displaying Expiration Date of OMB Approval

The expiration date for OMB approval will be displayed on all forms completed as part of the data collection.

18. Exception to the Certification Statement

Exception to the certification statement is not requested for the data collection.

Table A.7. Schedule for the Evaluation

Activity	Date
Intake period	November 2011 to April 2012
First site visits	Winter 2011/2012
Second site visits	Spring 2013
Implementation report	Fall 2013
15-month follow-up survey	February 2012 to July 2014
30-month follow-up survey	April 2014 to September 2015
First impact report	2015
Final report	2016

References

Ashenfelter, O. “Estimating the Effect of Training Programs on Earnings.” The Review of Economics and Statistics, vol. 60, no. 1, 1978, pp. 47–57.

Bassi, L. “Estimating the Effects of Training Programs with Non-Random Selection.” Review of Economics and Statistics, vol. 66, 1984, pp. 36–43.

Bassi, L. “The Effect of CETA on the Post-Program Earnings of Participants.” Journal of Human Resources, vol. 18, no. 4, fall 1983, pp. 539–556.

Bloom, H. S. “Accounting for No-Shows in Experimental Evaluation Designs.” Evaluation Review, vol. 8, no. 2, 1984, pp. 225–246.

D’Amico, R. “What’s Known About the Effects of Publicly Funded Employment and Training Programs.” Oakland, CA: Social Policy Research Associates, 2006.

D’Amico, R., K. Dunham, J. Henderson-Frakes, D. Kogan, V. Koller, M. Mack, M. Magnotta, J. Salzman, A. Wiegand, D. Weissbein, and G. Carrick. “The Workforce Investment Act After Five Years: Results from the National Evaluation of the Implementation of WIA: A Report Prepared as Part of the National Evaluation of the Implementation of WIA.” Oakland, CA: Social Policy Research Associates, 2004.

D’Amico, R., D. Kogan, S. Kreutzer, A. Wiegand, A. Baker, G. Carrick, and C. McCarthy (2001). “Interim Report: A Report on Early State and Local Progress Towards WIA Implementation.” Washington, DC: U.S. Department of Labor, Employment and Training Administration.

Dickinson, K P., T. R. Johnson, and R. W. West. “An Analysis of the Impact of CETA Programs on Participants’ Earnings.” Journal of Human Resources, vol. 21, no. 1, 1986, pp. 64–91.

Dickinson, K. P., T. R. Johnson, and R. W. West. “The Impact of CETA Programs on Components of Participants’ Earnings.” Industrial and Labor Relations Review, vol. 40, no. 3, 1987, pp. 430–441.

Federal Register. “Notice of Proposed Rule Making: Workforce Investment Act Amendments.” Federal Register, vol. 71, no. 244, 2006.

Friedlander, D., D. H. Greenberg, and P. K. Robins. “Evaluating Government Training Programs for the Economically Disadvantaged.” Journal of Economic Literature, vol. 35, no. 4, 1997, pp. 1832–1848.

Greenberg, D. H., C. Michalopoulos, and P. K. Robins. “A Meta-Analysis of Government-Sponsored Training Programs.” Industrial and Labor Relations, vol. 57, no. 1, October 2003.

Heckman, J. J., R. J. LaLonde, and J. A. Smith. “The Economics and Econometrics of Active Labor Market Programs.” In Handbook of Labor Economics, vol. 3, edited by O. Ashenfelter and D. Card. Amsterdam, Netherlands: Elsevier, 1999.

Heinrich, C., P. Mueser, and K. Troske. “Workforce Investment Act Non-Experimental Net Impact Evaluation”. Report prepared for the U.S. Department of Labor Employment and Training Administration (December 2008).

Hollenbeck, Kevin. “Workforce Investment Act (WIA) Net Impact Estimates and Rates of Return.” Kalamazoo, MI: WE Upjohn Institute for Employment Research. 2009.

Hollenbeck, K. D. Schroeder, CT King, and W.J. Huang. “Net Impact Estimates for Services Provided through the Workforce Investment Act” Employment and Training Administration Occasional Paper 2005-6. Washington, DC: U.S. Department of Labor, 2005.

LaLonde, R. J. “The Promise of Public Sector-Sponsored Training Programs.” Journal of Economic Perspectives, vol. 9, no. 2, spring 1995, pp. 149–168.

McConnell, Sheena, Elizabeth Stuart, Kenneth Fortson and others. “Managing Customers’ Training Choices: Findings from the Individual Training Account Experiment.” Report prepared for the U.S. Department of Labor, Employment and Training Administration (December 2006).

Orr, Larry L., Howard S. Bloom, Stephen H. Bell, Fred Doolittle, Winston Lin. Does Training for the Disadvantaged Work? Evidence from the National JTPA Study. Washington, DC: Urban Institute Press, 1996.

U.S. Department of Labor, Employment and Training Administration. “Workforce
System Results: January 1–March 31, 2007. Available at www.doleta.gov/Performance/Results/ Quarterly_report/quarterly_report_03-31-07.pdf#page=6.

1 In the draft package, this form was called the Baseline Information Form (BIF). We changed the name in response to comments made by staff at the study sites.

2 The name of this system was changed from the Participant Tracking System in response to comments made by staff at the study sites.

3 As discussed in Part B of this package, if all 30 sites do not participate these figures may overstate the number of customers in the study.

4 http://www.mathematica-mpr.com/publications/PDFs/managecust.pdf

5 In addition to the changes to the intake forms that are documented in the appendix, a few changes were made to the documents to improve their formatting and to update references to the study title.

6 The total number of customers (65,000) does not precisely equal 270 x 241 due to rounding.

2