WIA Gold Standard Evaluation, Part A of supporting statement, 10.2.2012

WIA Gold Standard Survey Follow Up and VETS

part a: Justification

The U.S. Department of Labor’s (DOL) Employment and Training Administration (ETA) is undertaking the Workforce Investment Act (WIA) Adult and Dislocated Worker Programs Gold Standard Evaluation (WIA Evaluation). The overall aim of this evaluation is to determine whether adult and dislocated worker services and training funded by Title I of WIA—currently the largest source of Federal funding of employment services and training—are effective and whether their benefits exceed their costs. ETA has contracted with Mathematica Policy Research and its subcontractors—Social Policy Research Associates, MDRC, and the Corporation for a Skilled Workforce—to conduct this evaluation. The evaluation was launched in 28 Local Workforce Investment Areas (LWIAs) starting in November 2011, and all sites began intake of customers into the study by August 2012.

An initial data collection package, approved by the Office of Management and Budget (OMB) in September 2011 (clearance number 1205-0482), requested clearance for a form to check the study eligibility of the customer, a customer study consent form (indicating the customer’s knowledge of the evaluation and willingness to participate) and the collection of data at baseline through a study registration form and contact information form, as well as site visit guides for the collection of qualitative information on WIA program processes, services, and training.

Subsequent to receiving OMB approval for qualitative data collection during site visits to the 28 LWIAs participating in the study and because veterans are excluded from the net-impact study, ETA, in consultation with DOL’s Veterans Employment and Training Service (VETS), decided to collect additional qualitative data in order to analyze veterans’ experiences in the 28 WIA Evaluation sites. The study of veterans in the WIA programs is called the Veterans’ Supplemental Study (VSS).

This second and final package requests clearance for the veterans’ data as well as two other remaining data collection efforts for the WIA Evaluation, specifically:

1. Two follow-up surveys conducted at 15 and 30 months after random assignment, with a sample of approximately 6,000 WIA customers included in the WIA Evaluation;

2. Cost data collected on three forms—a program costs questionnaire, a staff activity log, and a resource room sign-in sheet—for use in estimating the costs of WIA services received by sample members for the benefit-cost analysis; and

3. The aforementioned veterans’ data for the VSS, consisting of qualitative data on veterans served at the 28 LWIAs participating in the WIA Evaluation. For the VSS qualitative analysis, additional questions and several activities will be added to the WIA Evaluation’s second round of site visits to the 28 LWIAs. (A separate quantitative analysis will use two sets of administrative data that states already report to the DOL: the WIA Standardized Record Data [WIASRD] and Wagner-Peyser data. Because the data are already reported to the DOL, there is no additional burden associated with this quantitative data collection.)

This package includes:

1. Appendix A: 15-Month Follow-Up Survey Instrument

2. Appendix B: 30-Month Follow-Up Survey Instrument

3. Appendix C: Letters and Reminders to Survey Sample Members

4. Appendix D: Cost Data Collection Package

5. Appendix E: VSS Site Visit Protocols (and short form for focus groups members)

6. Appendix F: VSS Customer Consent Form

7. Appendix G: 60-Day Federal Register Notice

8. Appendix H: Public Comments

9. Appendix I: WIA Evaluation Telephone Survey Pretest Memo

1. Circumstances Necessitating the Data Collection

Passage of the Workforce Investment Act of 1998 led to a major redesign of the country’s public workforce system. WIA programs serve more than six million people annually at a cost of over $3 billion (U.S. Department of Labor, FY 2012 Budget in Brief). Among its goals, WIA aims to bring formerly fragmented public and private employment services together in a single location within each community, make them accessible to a wider population than did prior employment services and training, empower customers with greater ability to choose from among services and training, and provide localities both greater flexibility in using funds and greater accountability for customers’ employment outcomes. ETA is conducting the evaluation to provide rigorous, nationally representative estimates of the net impacts of WIA intensive services and training. Intensive services are services that involve substantial staff assistance and include assessments, counseling, and job placement. Training includes education and occupational skills building. The WIA Evaluation, including the VSS, will offer policymakers, program administrators, service providers, employers, and veteran service organizations with information about the relative effectiveness of WIA-funded services and training, how the effectiveness varies by target population, and how the services are implemented. The evaluation will also produce estimates of the benefits and costs of WIA intensive services and training. In addition, through the VSS it will provide information about available services and training for veterans, and describe their characteristics, the services they receive and their outcomes. Since the WIA Evaluation is excluding veterans from the net-impact study, this supplemental study provides the opportunity to analyze veterans’ experiences in the 28 representative LWIAs participating in the WIA Evaluation.

The remainder of Section 1 is organized as follows. To provide context for the study, the section begins with a brief description of previous studies of employment and training programs and an overview of WIA, including its structure and the types of services offered. It also provides an overview of the current services and training for veterans. The section then provides an overview of the WIA Evaluation, including the VSS. Finally, it describes the WIA Evaluation’s overall data needs and collection plan. It ends with a discussion of the need for the data collection efforts included in this package.

a. Previous Studies of Employment and Training Programs

Evaluations have been conducted of most of the major employment and training programs that preceded WIA, including the 1962 Manpower Development Training Act (MDTA), the 1973 Comprehensive Employment and Training Act (CETA) and the 1982 Job Training Partnership Act (JTPA). Although the methods and findings from individual studies vary, most reviews have drawn similar conclusions about these programs’ effects on the major target populations (D’Amico 2006; Friedlander et al. 1997; Greenberg et al. 2003; Heckman et al. 1999; LaLonde 1995). For women, the impacts of employment and training programs are generally positive, with significant and positive net impacts on earnings found in studies of MDTA, CETA, and JTPA (Ashenfelter 1978; Bassi 1983, 1984; Bloom et al. 1993). The estimated impacts for men are more modest and less consistent; whereas the MDTA study found a significant positive net impact on men’s earnings, the estimated impacts of CETA varied from large and negative (Dickinson et al. 1986, 1987) to large and positive (Bassi 1983), depending on the study. The JTPA study found positive impacts for men of about eight percent in earnings growth over a 30-month follow-up period (Bloom et al. 1993).

Although there are no experimental evaluations of WIA, several nonexperimental studies of the impact of WIA intensive services and training on adults and dislocated workers have been conducted (Heinrich et al. 2008; Hollenbeck et al. 2005; Hollenbeck 2009). The studies found that participation in the adult WIA program is associated with a several hundred dollar increase in quarterly earnings; participation in the dislocated worker WIA program is associated with an initial reduction in earnings relative to the comparison group, but later gains in earnings so that earnings ultimately match or overtake those of the comparison group. However, the impact for dislocated workers is lower than it is for the adult program. As with many nonexperimental studies, these results are open to the criticism that the observed differences between outcomes of program participants and outcomes of members of the comparison group may be a result of preexisting differences between those groups, rather than a result of participation in the program.

b. Overview of WIA

WIA funds originate at the Federal level and flow to states and local areas. Each state has one or more local workforce investment boards (WIBs) that govern non-overlapping geographic areas within the state, known as local workforce investment areas (LWIAs). WIA requires that each LWIA establish American Job Centers at which customers can access WIA services. American Job Centers provide access, in one location, to local, state, and Federal employment-related services and supports. Services from a range of other partner organizations and sources of funding, such as the Employment Services, are also provided at American Job Centers.

Currently, there are slightly fewer than 600 LWIAs. The size of the LWIAs varies dramatically, with some serving as few as a couple of dozen customers annually, whereas others serve tens of thousands. LWIAs can also vary in the geographic area they cover; for example, some states with large rural areas have one LWIA to cover the entire state except for its major cities. In these cases, the LWIA may be further divided into subareas for organizational and management purposes.

Title I of WIA authorizes separate funding for two groups of customers who are 18 or older:

1. Adults. An individual must be at least age 18 to be eligible for WIA adult services. Priority for adult intensive services and training (described below) is given to low-income customers.

2. Dislocated workers. Generally, individuals who are eligible for WIA dislocated worker services are individuals who fall into one of the following categories: persons who have been terminated or laid off from a job without cause, whose employer has announced a facility closure within 180 days, who were self-employed but are now unemployed, or who are displaced homemakers.

Program Year 2011 funding amounts for serving adult customers and dislocated worker customers were $771 million and $1,232 million, respectively (U.S. Department of Labor, FY2012 TEGL No. 19-11, March 16, 2012). This evaluation focuses on evaluating the impact of services separately for adults and dislocated workers.

WIA provides for three levels (or tiers) of assistance for adults and dislocated workers: core services, intensive services, and training. The purpose of this tiered structure is to allow all customers to access some informational services (core), while concentrating the assistance that is the most expensive to provide (such as training) on a smaller number of customers with the greatest need. Generally speaking, customers receive assistance at one level before moving to the next tier. Core services are the lowest tier, intensive services follow, and training is the highest tier offered (Federal Register, Workforce Investment Act Final Rule, 2000). The following paragraphs describe the assistance that is statutorily considered to be part of each tier.

WIA-funded core services are the least resource intensive and can often be accessed by customers without staff assistance either in the resource room at the American Job Center or remotely via the Internet. Self-serve offerings, which are to be provided universally to all customers who want them regardless of income, include job listings and other information on the labor market (such as lists of high-demand occupations); information on services provided via WIA and other programs; information on WIA providers; Internet access; computer software for assessments and resume writing; and access to telephones, fax machines, and copy machines. Other core services require some staff assistance. These include workshops on job-search related topics such as resume writing and interviewing; initial assessments of skills, aptitudes, and interests; determination of eligibility for programs; information about specific employers; and information about training.

Intensive services are available to customers who are unable to obtain employment with the help of core services alone. The determination of the need for intensive services is made by American Job Center staff. The services include comprehensive and specialized assessments; development of an individual employment plan; group and individual counseling; placement in unpaid jobs to gain work experience and internships; assistance in finding employment; and short-term prevocational services, such as development of learning and interviewing skills.

Authorized training for building skills and increasing employability is delivered primarily through individual training accounts (ITAs), which are like vouchers that provide customers with flexibility in choosing a training program that meets their needs. Training is available to customers who are eligible for intensive services but are unable to obtain or retain employment through these services. The training may include occupational skills training, skills upgrading, and adult education and literacy activities (provided in concert with other training activities) as well as counseling and support during training. To ensure that WIA-funded training programs meet certain standards for quality, ITAs may be used to pay only for training programs listed on a state’s Eligible Training Provider List. On-the-job training, customized training provided by an employer, and training designed for special populations facing multiple barriers to employment can be funded directly rather than through an ITA.

c. Overview of Veterans’ Services

The VSS will focus on two aspects of veterans’ services within the American Job Center system. First, VETS administers the Jobs for Veterans State Grants Program that provides funding for staff designated to focus on veteran customers within the workforce system. Second, all staff of American Job Centers are required by 38 U.S.C. 4215, as added by the Jobs for Veterans Act, to provide veterans with priority of service for DOL-funded services.

Jobs for Veterans State Grants. Through this formula grant program, states receive grants that are proportional in size to the number of veterans residing in the state who are seeking employment. In turn, states use the grant funds to hire staff across the state focused on providing outreach and support to veterans. Two types of positions are funded through the grants—Disabled Veterans’ Outreach Program (DVOP) specialists and Local Veterans’ Employment Representatives (LVERs).

DVOP specialists provide intensive services and facilitate placements for eligible veterans, with a priority of serving disabled veterans. Services are provided with maximum emphasis placed on assisting economic and educationally disadvantaged veterans. They provide direct services to veterans, promote community and employer support for internships and on-the-job training opportunities, provide case management for veterans with serious disadvantages in the job market and those who are enrolled in Federally-funded training programs, and follow up with those veterans and their employers to ensure necessary services are provided to promote job retention.

LVERs facilitate the provision of services provided by state employment delivery systems to veterans. They also monitor job listings by the Federal government and contractors, promote the participation of veterans in Federal employment and training programs, and develop contacts with community leaders, employers, unions, veteran service organizations, and other groups to promote hiring of veterans and to ensure veterans get the services to which they are entitled.

Priority of Service for Veterans. The Jobs for Veterans Act (JVA) of 2002 (Public Law 107-288) amended Title 38 of the United States Code to establish priority of service (POS) for veterans and eligible spouses in any workforce program or services directly funded, in whole or part, by DOL. “Priority of service” was defined to mean that a covered person should be given priority over nonveterans for the receipt of employment, training, and placement services, provided the person otherwise meets the eligibility requirements for participation in the program. State- and local-level service providers were required not only to provide POS and inform each POS-eligible person of the rights and benefits to which they are entitled, but also to provide information on benefits and services available from other providers.

Since passage of the JVA, VETS and ETA have provided guidance to state and local areas on the implementation of POS. Ten months after the legislation was enacted, ETA published Training and Employment Guidance Letter (TEGL) 5-03 with the purpose of informing states and other DOL workforce investment system partners of the POS provisions and providing general guidance on implementing them. However, a 2005 U.S. Government Accountability Office report (GAO 2005) found that administrators in some areas were unaware of, or confused by, DOL’s guidance on the JVA POS requirements, and suggested that DOL provide agencies with clear guidance and technical assistance to ensure implementation of POS.

Enacted on December 22, 2006, Section 605 of Public Law 109-461 required that the Secretary of Labor prescribe regulations to implement the POS within two years. The Secretary’s response was the Priority of Service for Covered Persons Final Rule, published in the Federal Register on December 19, 2008, as 20 CFR Part 1010. The final rule detailed the POS provisions of the JVA and gave clear definitions of who was eligible, the impacts on various programs, and what American Job Centers would have to do to comply. It also responded to questions and comments received during the public comment period, providing additional information on areas that had caused concern or that were not fully understood by readers of the initial notice.

On November 10, 2009, ETA and VETS jointly published TEGL 10-09 and Veterans’ Program Letter 07-09. These documents described the history of the POS provisions of the JVA up to that point, summarized much of the content of the Final Rule, and provided considerable clarification and detail on when veteran status is obtained, when priority should be applied, when veteran status should be verified, and the variety of documents and systems through which the verification could be made. Furthermore, the letters gave guidance for how POS for veterans interacts with the other service requirements in programs with statutory priorities and discretionary priorities. In addition, TEGL 10-09 stated that DOL would monitor the implementation of POS and that program operators were required to ensure that POS is applied throughout their systems.

One year after the issuance of joint guidance by ETA and VETS on the implementation of POS, Training and Employment Notice 15-10—a Protocol for Implementing Priority of Service for Veterans and Eligible Spouses—was issued. This publication was geared toward service providers and provided action steps to undertake in implementing POS. It was more concise and accessible than earlier documents and focused on the information a person delivering services would have to understand and follow.

d. Overview of the WIA Evaluation

This evaluation will examine the impacts of WIA intensive services and training on customers’ outcomes relative to a situation in which customers have access to WIA core services only. It will address the following research questions:

1. Does access to WIA intensive services, alone or in conjunction with WIA-funded training, lead adults and dislocated workers to achieve better educational, employment, earnings, and self-sufficiency outcomes than they would achieve in the absence of access to those services and training?

2. Does the effectiveness of WIA vary by population subgroup? Is there variation by sex, age, race/ethnicity, unemployment insurance (UI) receipt, prior education level, previous employment history, adult and dislocated worker status, and disability status?

3. How does the implementation of WIA vary by LWIA? Does the effectiveness of WIA vary by how it is implemented? To what extent do implementation differences explain variations in WIA’s effectiveness?

4. Do the benefits from WIA intensive services and training exceed program costs? Do the benefits of intensive services exceed their costs? Do the benefits of training exceed its costs? Do the benefits exceed the costs for adults? Do the benefits exceed the costs for dislocated workers?

In addition, the Department determined that veterans could be legally exempted from participating in the impact study of the WIA Evaluation. Thus, across the participating sites, veterans who seek WIA adult and dislocated worker program assistance are not enrolled in the study and are able to access services as they would in the absence of the study.

Veterans’ circumstances can pose particular challenges to employment and training staff. Veterans have usually spent several years out of the civilian labor market, have often worked in occupations different than those available in the civilian world, and may have been in combat or on other deployments. Because of these and other experiences particular to veterans, staff may need to provide different assistance to veterans than they provide to civilians in order for them to be successful. In order to understand the assistance provided to veterans since veterans are not participating in the random assignment part of the WIA Evaluation, the VSS will build on the knowledge gained about the participating LWIAs’ operations. This supplemental research will provide information about the assistance provided to veterans, the issues staff face in providing that assistance, how priority of service is operationalized, how veteran representatives and other staff interact, and the outcomes for veteran customers.

The purpose of the VSS is to learn how the 28 LWIAs participating in the WIA Evaluation provide assistance to veterans, as well as to analyze the characteristics of veterans who participate in American Job Center services, the assistance they receive, and their outcomes. The study has two components. First, during site visits to the participating sites, the qualitative study component will address the following seven research questions:

1. How do veterans learn about the assistance offered by the American Job Centers?

2. What are the procedures for orienting and enrolling veterans into WIA and/or Wagner-Peyser, and how do they differ from procedures for nonveteran customers?

3. How do American Job Center staff members operationalize the veterans’ priority of service for the WIA and Wagner-Peyser programs?

4. How do the DVOP specialists and LVERs coordinate to provide assistance to veterans, and how do these staff members interact with other American Job Center staff?

5. What services and training are provided to veterans through the American Job Centers?

6. What issues do staff face in providing services and training to veterans, and how do they differ from nonveteran customers?

7. What innovative or promising practices have states or local areas implemented to provide employment services and training to veterans?

Second, through administrative data reported by states, the quantitative study component will address five questions:

1. What are the characteristics of veterans receiving assistance through the American Job Center system?

2. How do the characteristics of veterans receiving assistance compare to those of nonveterans receiving assistance?

3. What assistance is provided to veterans through the American Job Centers?

4. What are the outcomes of participating veterans?

5. What associations, if any, do the data suggest between the assistance veterans receive and their outcomes?

The administrative data on which the quantitative analysis of assistance to veterans will be based is already submitted to DOL and, thus, is not a part of this clearance request.

While the overall WIA study was approved under OMB clearance number 1205-0482, it may be beneficial in understanding the current Information Collection Request to review some of the elements addressed in that earlier clearance. To answer the overarching WIA Evaluation research questions, the project has three main analysis components: impact, implementation, and benefit-cost. A description of each is presented next in items i, ii, and iii. A description of the qualitative and quantitative analysis associated solely with the VSS follows that discussion in item iv beginning on page 17.

i. Impact Analysis

The major goal of the evaluation is to generate precise, unbiased estimates of the impacts of WIA intensive services and training for adults and dislocated workers and for those estimates to be broadly generalizable to the population of WIA customers. Toward this end, the WIA Evaluation involves randomly-selected study sites and, within those sites, randomly-assigned customers to research groups.

Random selection of sites. To obtain a nationally representative study sample, the design calls for first randomly selecting study sites. Since LWIAs typically administer local WIA funding and hence determine the services and training provided, an LWIA is considered a “site” in the evaluation. Thirty sites were randomly selected from the set of all LWIAs on the U.S. mainland that serve 100 or more intensive services customers annually. This number of sites allows for precise estimates and a low rate of assignment to the research groups that are not eligible to receive full WIA services (as described below). The random selection was conducted using explicit and implicit stratification to take into account the enrollment levels at each site, the LWIA’s geographic location, and, as a proxy for the focus the site places on training, the proportion of LWIA intensive service customers who receive WIA-funded training. Each of the 30 randomly selected sites was asked to participate in the evaluation, and 26 of these initially selected sites agreed to participate. Four sites declined to participate, and replacement sites were identified for two of these sites (and agreed to participate); therefore, the total number of sites participating in the evaluation is 28. The other two initially selected sites declined to participate too late in the process for replacement sites to be recruited. (Details of the site selection and recruitment process, and its implications for the analysis are discussed in Part B, Section 1.)

Random assignment of customers within selected sites. The cornerstone of the impact analysis is random assignment of customers within these 28 randomly-selected sites to experimental groups. Experimental evaluations are generally viewed as the “gold standard” for evaluating social programs because, more than any other approach, they minimize the chance that any observed differences in outcomes between comparison groups are due to unmeasured, preexisting differences between members of the research groups.

During the study’s intake period, LWIA staff determines whether all WIA adults and dislocated workers who are eligible for WIA intensive services and would be offered intensive services in the absence of the study are eligible for the study (using the study eligibility checklist). Customers are eligible for the study if they are aged 18 or older, do not participate in the Trade Adjustment Assistance program, and are not a veteran or a covered spouse of a veteran. Once customers are found eligible for the study, they are told about the study and asked to sign a form (the consent form) confirming they have been told about the study, understand its implications, and agree to participate. All customers who consent to participate are asked to complete a study registration form and a contact information form. The eligibility checklist, consent form, study registration form, and contact information form have already been approved by OMB (clearance number 1205-0482).

Once these forms have been completed, LWIA staff enters customers’ data into a study-specific management information system. The system randomly assigns each customer into one of three research groups. The LWIA staff then notifies the customer of his or her research group assignment.

The three research groups to which customers (who consent to participate in the study) are assigned are: (1) full-WIA group—adults and dislocated workers in this group can receive any WIA services and training for which they are eligible; (2) core-and-intensive group—adults and dislocated workers in this group can receive any WIA services for which they are eligible but not training; and (3) core group—adults and dislocated workers in this group can receive only core services and no WIA intensive services or training (Figure A.1). Customers will remain in their study groups for 15 months after the date they are randomly assigned. Customers who do not consent to participate in the study will be allowed to receive core services only until intake for the study has ended.

Figure A.1. Research Groups

In most cases, the sample intake period will last between approximately 12 and 18 months in each site. The length of the intake period was determined in consultation with the Workforce Investment Board and/or LWIA administrators, as some preferred to minimize the intake period. Sample intake began in some sites in November 2011. The rest of the sites began on a rolling basis with most sites starting random assignment by April 2012.

The rates at which individuals are assigned to the core and the core-and-intensive groups will be low in all sites for both adults and dislocated workers; initially, based on an 18-month intake period, the rates ranged from 0.7 percent in each restricted-service group in the largest LWIAs to eight percent in the smallest ones. These rates were chosen to minimize the rates at which customers would be denied services and training. These low rates of assignment to restricted-service groups will reduce the disruption to the LWIAs. However, if the rate of enrollment is different than expected, then we will raise or lower the rate of assignment to each restricted-service group to meet the target number of customers in each group.

Across all sites, 2,000 customers will be assigned to each of the restricted-service research groups, with the remainder of customers assigned to the full-WIA group. Based on estimates using recent data, about 64,000 customers are expected to be assigned to the full-WIA group. All 4,000 members of the restricted-service groups and a random sample of 2,000 members of the full-WIA group will be asked to complete the follow-up surveys.

Several key features of the random assignment process ensure that the integrity of random assignment is maintained in all sites. First, all customers who consent to participate in the study must go through random assignment once and only once; if they return for services before the conclusion of their service embargo date (15 months after they are randomly assigned) they will have to adhere to their original group assignment. Second, customers cannot change their research group status during the study. Third, customers are to be offered those services, and only those services, available to the research group to which they were assigned, and only core services for those who decline to consent to the study. Fourth, sites should change as few of their routine procedures as possible during their involvement in the study. Finally, staff will not be informed of the precise date at which random assignment will conclude in order to prevent gaming of intake near the end of the intake period.

These features of the random assignment process will be monitored through the collection of data (described later in detail) on customers’ characteristics, the services or training they receive, the overall intake and nonconsent rates at the site, and sites’ implementation experiences.

Estimating program impacts. To answer the first research question of interest, about the impacts of access to WIA intensive services and training on customers’ outcomes, the outcomes of customers in the different research groups will be compared. Specifically, the outcomes of members of the full-WIA group and the core group will be compared; the outcomes of the full-WIA group and the core-and-intensive group will be compared; and the outcomes of the core-and-intensive group and the core group will be compared. To answer the second research question, about impacts on subgroups of customers, the same type of analysis will be conducted, but on various subgroups of interest, where those subgroups are defined by baseline characteristics of individuals such as gender, age, race/ethnicity, and previous employment history. Finally, the third research question, about how the effectiveness of WIA varies by how it is implemented, will be answered by using subgroups of sites with similar program implementation as determined by the implementation study (described next).

ii. Implementation Analysis

The implementation study will examine how implementation of WIA varies across the LWIAs and has five major objectives:

1. Interpreting the net-impact estimates. Information on the implementation of each component of WIA will aid in interpreting the net-impact findings. It will inform judgments about the mechanisms through which the program affects outcomes. For example, an explanation of observed differences in outcomes between the core group and the core-and-intensive group requires an understanding of the availability, content, and intensity of WIA intensive services as well as of non-WIA services that each group might access.

2. Interpreting differences in net-impact estimates by site. The implementation analysis will compare service delivery and the context in which the services and training are delivered across sites. Sites will be placed in groups with similar service delivery models and impacts will be estimated separately for each group. This analysis will allow an investigation of how impacts vary among sites with different approaches to the implementation of WIA services and training.

3. Documenting the program. Although the overall goals of WIA and its early implementation are well documented (D’Amico et al. 2004), the program is likely to have changed in the years since its implementation was last studied in detail. It is important to provide policymakers with a description of WIA as currently implemented, along with the findings from the net-impact and benefit-cost analyses.

4. Documenting fidelity to the evaluation procedures. The implementation study will also provide important information about the extent to which the sites are faithful to the evaluation procedures.

5. Estimating program costs. The implementation study will include the collection of cost data that will be used in the benefit-cost analysis but that are also important in describing the program and the relative intensity of its service components.

The implementation study protocols (absent the site visit data specific to veterans that are included in this request) have already been approved by OMB (clearance number 1205-0482).

iii. Benefit-Cost Analysis

The benefit-cost analysis will assess whether the benefits of intensive services and training for adults and for dislocated workers exceed the costs of the additional services, and training. It will use an accounting framework that itemizes all the benefits and costs of WIA and places a dollar value on as many of them as possible. The analyses will estimate the benefits and costs from three perspectives: WIA customers, nonparticipants (government and taxpayers), and society as a whole. Potential benefits of WIA include an increase in both earnings and taxes, and a reduction in the use of UI and public assistance as well as in employment services and training funded by sources other than WIA. The costs include the direct and indirect costs of additional services and training received by sample members.

iv. Veterans’ Supplemental Study

The supplemental study focused on veterans includes qualitative and quantitative analyses. The qualitative analysis builds on the qualitative research being conducted as part of the WIA Evaluation’s implementation study. However, the VSS analysis will focus on describing the processes through which veterans receive services and training at the 28 LWIAs as well as veterans’ experiences.

The VSS will also analyze two sets of administrative data that states report to DOL: WIASRD and Wagner-Peyser data. These data sets describe the veterans who have registered for services and the employment-related services they received. Since these data include customers’ employment and earnings data (through UI wage matching), we will not conduct special UI wage matching. The data will be stripped of any identifying information. The analysis of the data will include an exploration of veteran characteristics, the services they receive, and the outcomes they achieve. The analyses will also seek to identify any associations between specified “service menus” and outcomes experienced by different veterans subgroups.

e. Overview of WIA Evaluation Data Collection Needs and Plan

WIA Evaluation

To address adequately the four overarching research questions for the WIA Evaluation described earlier requires the collection of large amounts of detailed data from multiple sources (excluding the data needs specific to the VSS). Table A.1 summarizes the data needs together with the sources of those data and for which participants or sites they are collected. It shows all the data collection activities of the WIA Evaluation, including those for which we have already received clearance, as well as those for which we are requesting clearance through this package.

Table A.1. Summary of Data Collection Activities for the WIA Evaluation

Type of Data Needed	Reason Data Needed	Sources	For Whom Collected
Baseline information	Describe study participants	Study registration forms	All 68,000 study participants
	Check that random assignment created groups with similar baseline characteristics Define groups for subgroup analysis Enhance precision of the impact analysis	State UI agencies	All 68,000 study participants
Services received	Monitor random assignment Determine impact of WIA on the receipt of any employment services and training Assign a cost of WIA services and training per participant	State and/or LWIA management information systems	All 68,000 study participants
		15- and 30-month follow-up surveys	6,000 study participants in the survey sample (2,000 each in Core, Core-and-Intensive, and Full-WIA study groups)
Outcomes	Estimate the impacts of intensive services and training Estimate the benefits of intensive services and training	State UI agencies	All 68,000 study participants
		15- and 30-month follow-up surveys	6,000 study participants in the survey sample
Implementation data	Document and describe the implementation of WIA services and training Monitor the implementation of the evaluation	Site visits: interviews with LWIA staff, group interviews with customers, review of program documents, site observations	All 28 participating LWIAs
		State and/or LWIA management information systems	All 68,000 study participants
Costs	Estimate costs of services for the benefit-cost analysis	Cost data collection packages completed by local WIA staff: (1) program costs questionnaire, (2) front-line staff activity log, and (3) resource room sign in sheet	All 28 participating LWIAs
		Accounting data on ITA obligations and expenditures	All 68,000 study participants

Table A.1 (continued)

The evaluation requires data on five main topics:

1. Baseline information. Information collected upon study enrollment is used to provide a description of the characteristics of study participants at the time they are randomly assigned to a research group, conduct random assignment, ensure that random assignment has taken place correctly, track study participants over time, define subgroups for the impact analyses, increase the precision of net-impact analyses, and adjust for survey nonresponse. Baseline information should be available for all study participants.

2. Services received. Information on WIA-funded employment services and training received by all study participants will be used to monitor the random assignment process and ensure that participating customers are receiving only the services intended for their assigned research group. It will also be used to determine the extent to which access to WIA services affected use of those services, as we expect that some customers will not take up the offer of services. Finally, information on services received by the survey respondents will facilitate the benefit-cost analysis.

3. Outcomes. Outcome data are needed to estimate net impacts. The main outcomes of interest for the evaluation are related to labor market success—employment, earnings, and quality of job as measured by wage, hours worked, and fringe benefits. Outcomes will also include measures of attainment of skills, such as degrees or certificates, and measures of self-sufficiency such as the receipt of UI and public benefits through programs such as the Supplemental Nutrition Assistance Program (SNAP) (formerly the Food Stamp Program), the Temporary Assistance for Needy Families (TANF) program, and the Supplemental Security Income (SSI) program. Data on UI receipt will be collected from state UI agencies for all study participants. Data on other program receipt will be collected through the surveys for the sample of survey respondents.

4. Implementation. Data are needed on the context in which programs operate, the types of services and training offered, how the services and training are provided, and how the evaluation was implemented. These data—collected from all participating LWIAs—will provide the basis of the implementation analysis and will come from in-person interviews with staff who administer the program and customers who participate, as well as from program documents, observations of WIA activities, and management information systems from the participating states or sites.

5. Costs. Data on the costs of WIA services and training will be used to describe the program and assess whether benefits of the services and training received exceed the costs of those services and training. Three forms specifically designed to gather this information will be completed by each site. Because information about the details of WIA-funded education and training often is not available through sites’ management information systems, accounting data about ITA obligations and expenditures will be collected for each participant.

These data will be collected from five sources. The first three sources are:

• The study registration and contact information forms. The study registration form requests some information on demographic and socio-economic characteristics of all consenting adults and dislocated workers prior to random assignment. The contact information form asks for detailed contact information for friends and relatives of the study participant that will be used to locate the participant for the surveys. OMB clearance has been received for these forms (OMB clearance number 1205-0482, expiration date: 9/30/2014).

• Site visits. Data on the context for the program and its implementation will be collected during two rounds of site visits to each of the 28 participating sites. The site visits will involve interviews with key staff, case file reviews, group interviews with customers, and observations of program activities. OMB clearance has also been received for these site visit data collection protocols (OMB clearance number 1205-0482, expiration date: 9/30/2014).

• State and local administrative data. To determine that random assignment is being implemented correctly as well as to collect data on the receipt of WIA services, data extracts from the state and/or local management information systems and accounting data on ITA expenditures are being requested at the local level. In addition, administrative data on quarterly wage records and receipt of UI benefits from UI agencies in each state will be accessed.

The remaining two data collection sources for the overarching WIA Evaluation which we are requesting clearance in this package—two follow-up surveys and the cost data collection package—are discussed in the next section, following the discussion of the VSS data collection plans (below).

Veterans’ Supplemental Study

Qualitative study. The supplemental study’s qualitative component builds on the qualitative research being conducted in the 28 LWIAs as part of the WIA Evaluation’s implementation study. For the larger study, the evaluation team is conducting the first round of visits to sites from March 2012 through October 2012. The main purpose of the first-round visits is to document how the participating LWIAs provide WIA services to adult and dislocated worker customers and to use these data to inform results from the evaluation’s impact analysis. Information for the VSS is not being collected during these visits; however, the study will benefit from the information collected on the organization and delivery of American Job Center services in general.

In November 2012, the WIA Evaluation team will begin its second round of visits, which will include an examination of the partnerships formed through WIA. New data collection for the VSS proposed to be collected during the second round of visits, include (1) additions to existing interviews being conducted as part of the WIA Evaluation’s implementation study visits, (2) new interviews focused on veterans’ services in WIA Evaluation sites, and (3) in selected sites, new focus groups with veteran customers focused on their experiences.

Additions to existing interviews. At each of the 28 LWIAs, the supplement will add a veteran-related module to the protocols developed for the American Job Center staff members who are to be interviewed as part of the WIA Evaluation. These staff members include the WIA executive director, the WIA and Employment Services (ES) manager(s) of the American Job Centers, and American Job Center line staff, including intake and counseling staff. These additional questions will explore how the staff identify, prioritize, and reach out to veterans; how they identify service and training needs and provide that assistance; how they work and coordinate with the LWIA’s LVERs and DVOP specialists; and any challenges the staff encounter in serving veterans.

Interviews focused on veterans’ services only. Each site’s qualitative data collection also will include three additional interviewee types: DVOP specialist, LVER, and the state veterans’ coordinator. The interview with the state coordinator will focus on state guidance provided to LWIAs regarding the provision of services to veterans and POS, as well as the coordinator’s perspective on the coordination of services to veterans within the American Job Centers, innovative or unique services provided to veterans (particularly focused on the participating LWIA(s) in the state), and challenges to providing services to veterans. The DVOP and LVER interviews will focus on how and when veteran status and POS eligibility are identified and verified, what services and resources are available to veterans and how they differ from those available to nonveterans, the content and frequency of the interactions between LVERs/DVOP specialists and veterans and other staff, and innovative practices.

Focus group discussions to collect information on services to and experiences of veterans. In eight sites, the evaluation team will conduct focus groups with purposively-selected veterans to discuss their experiences in the American Job Center system. The designated American Job Center staff member will be provided text to email to veterans who have received services in the past six months. The email will briefly describe the nature of the study as well as when and where the focus group will be meeting, and will ask willing participants to contact the designated staff member. That staff member will compile a list of the volunteers and some characteristics needed to select a diverse focus group, along dimensions such as gender, presence of a service-connected disability, year of discharge, and job market prospects. The site visit team will then work with the designated staff member to select focus group members to recruit and the staff member will contact the selected individuals. If we are unable to convene a focus group in one of the sites, we will attempt to talk individually with up to three veterans from that site in person or by telephone, to gain some insight into veterans’ experiences.

We will select case files of three of the focus group participants in each LWIA for review. The review will illustrate how veteran customers receive services through the American Job Center system. For both these activities, we will request that the staff identify veterans so that they represent a range of backgrounds and experiences. Thus, we will request that veterans represented in both the case file reviews and focus groups be a mix of male and female veterans, pre- and post-9/11 veterans, and veterans with and without a service-connected disability.

We will obtain veterans’ consent to participate in this data collection. The consent form in Appendix F describes the purpose of the study and informs the veteran of the data that may be collected. In addition to consenting to participate in the focus group, which also includes completion of a short form, the veterans will provide consent to have researchers review their case files (also in Appendix F).

The eight LWIAs participating in these additional on-site activities will be purposively selected based on two factors—the size of the LWIAs’ veteran population and evidence of particularly promising or innovative practices. To assess the size of the LWIAs’ veteran population, we will use the most recent WIASRD and select those that have the largest number of veterans. However, as this study is also interested in learning about unique service delivery practices, we also will seek to identify sites among the 28 that may be implementing innovative practices. Our main source of information for this will be the WIA Evaluation’s liaisons, who will have in-depth knowledge about their sites as a result of their regular communications with the sites and their first-round visits. We also will solicit input from other sources, including VETS staff, the state coordinators, and available literature. Using these factors, the eight sites selected for additional activities will include those serving large numbers of veterans and those implementing innovative practices.

f. Data Collection Efforts for Which This Package Requests Clearance

This package requests clearance for two study participant follow-up surveys, forms to collect data on costs for the WIA Evaluation, and collection of qualitative data specific to services for veterans for the VSS.

Two follow-up surveys. To collect data on customers’ service and training use and outcomes—employment and self-sufficiency—we will conduct two follow-up surveys. The surveys will be administered by telephone once at 15 months after random assignment and again 30 months after random assignment. Follow-up telephone surveys will be conducted with 6,000 study participants (all 2,000 members of each of the core and core-and-intensive groups and 2,000 randomly-selected members of the full-WIA group). These data will be used to estimate the impacts of WIA intensive services and training.

The first and second follow-up instruments are very similar, differing only in the survey reference period. (The first survey is expected to take customers longer to complete than the second, because customers are expected to have received more employment services and training in the first 15 months after random assignment than in the second 15 months after random assignment.) The first survey asks about the first 15 months since the customer was randomly assigned and the second survey asks about the next 15 months (months 16-30). If a sample member does not respond to the first survey, the second survey will ask about all 30 months since random assignment. The follow-up survey instruments are presented in Appendices A and B, respectively.

Cost data collection package. The benefit-cost analysis of intensive services and training requires data on the costs of the services and training received by research group members. Data on the quantity of services received will be obtained primarily from the follow-up surveys. Data on the cost of each service will be obtained from the sites by asking the LWIA staff to complete a program costs questionnaire and a front-line staff activity log, and administer resource room sign-in sheets. The cost data collection package with all three forms and accompanying cover letter is presented in Appendix D.

VSS site visit protocols. This package requests clearance for collection of qualitative data for the VSS. This information will be collected through site visits, which will include interviews with American Job Center staff at the 28 sites in the WIA Evaluation, and, at the state level, the veterans coordinator. Information will also be collected through focus groups with veterans and POS-eligible spouses at eight selected sites. Each of the focus group participants will complete a background information form. In addition, the case files of three of the focus group participants at each site will be selected for review. The protocols for these interviews, focus groups, and case reviews and the consent form are in Appendices E and F, respectively.

2. How, by Whom, and for What Purposes Will the Information Be Used

Clearance is being requested for data collection instruments that will be used to conduct the 15- and 30-month surveys of research group members, three forms for the collection of information for the benefit-cost analysis, and site visit protocols and materials for the VSS. Each data collection instrument is described in detail, along with how, by whom, and for what purposes the information collected will be used, beginning with the survey instruments, then the cost collection instruments and the VSS site visit protocols and accompanying material.

a. WIA Evaluation Follow-up Surveys

The follow-up surveys will be the main way the evaluation team will collect data on the receipt of services and training (funded by WIA or not), as well as employment and self-sufficiency outcomes. As discussed in item 4 (below), administrative data on these topics falls short of providing the level of detail, coverage, and uniformity across sites in the data elements needed to conduct a comprehensive and fine-tuned analysis of the effectiveness of WIA’s intensive services and training. The data on service and training receipt will be used to determine the extent to which the offer of services and training actually led to an increase in services and training. Some customers will not take up the offer of services and training and some customers in the core and core-and-intensive groups will be able to access services similar to those restricted through the evaluation from sources other than WIA. Data from the surveys will also be used to compute the average cost of services and training received for each survey sample member. The data on outcomes will be used to estimate the impact of the services and training and their benefits.

i. Timing of Surveys

Collecting data at 15 and 30 months after random assignment best balances two goals:

• Maximizing the ability of sample members to recall information. The timing of follow-up with sample members requires a balance between allowing enough time to pass for impacts to occur and asking questions close enough to events to allow for accurate recall. Service receipt can be difficult to recall as time passes, yet it is important to allow enough time for sample members to have completed services and training, for the most part, at the time of the interview. Conducting the first follow-up at 15 months after random assignment allows time for service and training completion while still providing the opportunity for sufficient recall to capture the necessary detail about receipt of services and training.¹

• Maximizing the length of the follow-up period. Research has shown that the boost in earnings from receiving employment services and training often decays over time. (See Greenberg et al. [2003] for a summary of this literature.) Hence, to understand long-term impacts, it is important to follow sample members for several years after receipt of services. The proposed follow-up observation period of 30 months is the maximum possible within the constraint of the length of the evaluation period.

ii. Sampling for the Surveys

All adult and dislocated workers randomly assigned to the core-and-intensive (2,000 individuals) or core groups (another 2,000 individuals) will be included in the survey sample. However, only a random subset of 2,000 of the approximately 64,000 full-WIA group members will be included.² Sampling 2,000 of the full-WIA group members minimizes the cost and the burden on respondents while also yielding more precise impact estimates than would other allocations of the 6,000 customers. The random selection of full-WIA members for the survey sample will be stratified by site. Within each site, the survey sample size of full-WIA members will be the same as the sample sizes for the core-and-intensive and core groups. Stratification on other characteristics will be performed to ensure that the sample is balanced in terms of adult/dislocated worker status, sex, and race/ethnicity and is well matched to the core and core-and-intensive services groups on these dimensions.

iii. Survey Content

The follow-up surveys will include basic screening and tracking questions and detailed modules that obtain information on service receipt, participation in education and training programs, employment and earnings patterns, self-sufficiency, and some customer characteristics. An overview of the key items included in the survey and how they will be used is provided in Table A.2.

Table A.2. Data Items in the WIA Evaluation Follow-up Surveys

	Survey Items	Tracking Informa-tion	Descriptive Measure	Outcome Measure	Benefit-Cost Measure
Personal Identifying and Tracking Information
Verify name, date of birth, and last four digits of Social Security number	Section A	√
Address and telephone numbers of respondent and friend or relative	Section G	√
Service Receipt	Section B
Resource room
Number of times resource room visited in American Job Center	Items B3, B5		√	√	√
Number of times resource room visited at another provider	Items B8-B9, B11		√	√
Workshops
Attendance in specified staff-intensive workshops in LWIA	Item B15		√	√	√
Number of other workshops attended in American Job Center; average amount of time spent in workshop	Items B16, B18, B20		√	√	√
Number of workshops attended elsewhere; average amount of time spent in workshop; type of provider	Items B21-B22, B24, B26		√	√
Topics covered in workshops	Item B27		√
Assessments
Type of assessments taken	Item B28		√	√	√
Number of assessments taken at American Job Center	Items B29-B31		√	√	√
Number of assessments taken elsewhere; type of provider	Items B32-B33, B35		√	√
Peer support groups
Number of peer support groups attended at American Job Center	Items B36, B38		√	√	√
Number of peer support groups attended elsewhere; type of provider	Items B41-B42, B44		√	√
Individualized counseling services
Topic of counseling	Item B47b		√
Number of times met with a counselor at American Job Center; average length of meeting	Items B48-B50, B52b		√	√	√
Number of times met with a counselor at another provider; average length of meeting; type of provider	Items B53-B54, B56, B58		√	√

Table A.2 (continued)

	Survey Items	Tracking Informa-tion	Descriptive Measure	Outcome Measure	Benefit-Cost Measure
Support Services
Type of assistance received	Item B59b		√
Dollar value of assistance received from an American Job Center	Item B61		√	√	√
Dollar value of assistance received from another provider; type of provider	Items B63-B64		√	√
Education and Training	Section C
Complete history of participation in education and training programs in the past 15 months, including start and stop dates	Items C1-C9		√	√
Number of hours per week in program	Items C10-C11		√	√	√
Type of program (educational, occupational skills, English as a Second Language, on-the-job training)	Items C12-C14		√	√	√
Type of provider	Item C16		√	√
Total out-of-pocket costs and other sources of funding for programs	Items C17-C23		√	√	√
Whether program was completed, and if not, reasons for not completing	Items C25-C26		√	√
Whether a degree, diploma, license, or certification was received	Items C27-C31		√	√
Associated assessments or tests required, whether they were taken, and if so, their total cost and sources of payment	Items C32-C36		√		√
Type of occupation the program trained for, and whether the customer perceived that the training helped them get a job in that field	Items C15, C37		√	√
Employment Patterns, Job Characteristics, and Earnings	Section D
Complete history of employment in the past 15 months	Items D1, D5, D8-D12, D21-D29		√	√	√
Industry and main duties	Items D2-D3; D6-D7; D34-D35		√	√
Number of hours worked per week	Items D13-D15, D30-D32		√	√
Earnings in job	D4, D37		√	√	√
Employment status: regular, seasonal, contractor, temporary, casual, day laborer, on call	D17, D36		√	√

Table A.2 (continued)

	Survey Items	Tracking Informa-tion	Descriptive Measure	Outcome Measure	Benefit-Cost Measure
Fringe benefits	Items D18, D38		√	√	√
Whether job was unionized	Items D19, D39		√	√
Reason for job separations	Item D20		√
Income Sources and Household Characteristics	Section E
Number of months of receipt and average amount received per month of SNAP, TANF, SSI, or other cash assistance	Items E1-E3		√	√	√
Number of months received assistance from the Women, Infants, and Children Program (WIC)	Items E1 and E2		√	√	√
Total household income	Items E4-E7		√	√
Number of people in household, number of children in household	Items E8-E9		√
Demographic and Household Characteristics	Section F
Health problems limiting work	Item F1		√
Receipt of health insurance at baseline and during previous 15 months, type of insurance	Items F2-F5		√	√
Race, ethnicity, marital status	Items F7-F9
Educational attainment	Items F10-F11		√	√
Whether the respondent has been arrested or convicted of a felony	Items F12a-13b		√

Below, the types of information collected are discussed in approximately the order of their appearance in the survey instruments.

Personal identifying and tracking information. Tracking information to correctly identify the survey sample members and follow up with them at a later date bookends the start and end of the survey instrument. The survey starts with screening questions to ensure that the sample locating process has identified the correct individual. Respondents will be asked to confirm their name, date of birth, and last four digits of their social security number. At the end of the survey, respondents will also be asked to confirm or update the basic contact information gathered from the sample locating process so that incentive payments (discussed in Part A, Section 9) can be delivered and, in the case of the 15-month survey, to better ensure our ability to locate the customer for the 30-month survey.

Service receipt. Key to the interpretation of the impacts of WIA intensive services and training on customer employment and self-sufficiency outcomes is the impact of offering these services on actual service receipt across the study groups. While all members of the full-WIA and core-and-intensive groups are offered intensive services, and members of the full-WIA group will be offered training as well, some customers will not access all offered services. In addition, as WIA is not the only funder of employment services and training, sample members may access services funded by sources other than WIA. Hence, it is important to collect data on the amount and type of services and training received by members of all three study groups from all sources.

The surveys will collect data on the quantity of employment services, education, and training received since random assignment and whether these services were received at an American Job Center or elsewhere. How the quantity of an activity is measured depends on the type of service, education, or training received; measures may include the number of times accessed, length of time spent in service, and the dollar cost.

Services asked about will include:

• Use of resource rooms. American Job Centers usually have resource rooms that provide local labor market information such as specific job openings and employers and industries that are in need of workers. These resource rooms also provide technological assistance to support a job search such as computers, access to the internet, fax machines, and telephones. Other organizations also provide similar services. The surveys will ask about the number of times the customer visited a resource room, and the time spent there, in both American Job Centers and elsewhere.

• Attendance in workshops. American Job Centers offer workshops on a variety of topics aimed at helping the customer become employed. Most of these workshops require only one staff member and there is little individualized attention. However, we have identified some more intensive workshops in which staff provide one-on-one assistance to customers. As the costs of these workshops are much higher than average, we treat them separately. The surveys will ask about attendance in the identified intensive workshops specific to each study site. (Information about the length and intensity of these workshops will be collected during on-site interviews with American Job Center staff and, therefore, will not be collected in the survey.) The survey questions will then collect information on the number and average length of other workshops attended within the American Job Center (typically the core workshop series offered), and then the extent of attendance in workshops provided by other non-WIA funded agencies or organizations.

• Attendance at peer support groups. Sometimes referred to as job clubs or networking groups, peer support groups are offered by some American Job Centers and other organizations as a means through which participants can share experiences, resources, and leads throughout the job search process. The survey will ask about attendance in peer group meetings provided at the American Job Center as well as participation in such group meetings offered by another agency or entity.

• Completion of assessments. Assessments can be used to determine the level of an individual’s basic skills (such as math or reading), and/or to determine how the interests and abilities of an individual align with particular jobs. The surveys will collect information on the type and total number of assessments completed as well as on the agency or entity that provided the assessment(s).

• Receipt of individualized counseling. The receipt of individualized counseling to support an individual’s job search, career exploration, and training options is an important element of WIA services. The surveys will collect information about the receipt and content of counseling sessions. In addition, items on the surveys will collect details about the frequency and duration of counseling services by the type of counseling provider (whether within the American Job Center or other organization).

• Receipt of support services. Sample members may also be eligible for and receive an array of supportive services in the form of cash, voucher, gift card or reimbursement to help him/her with expenses to look for work or attend training or school. Questions on the surveys will collect information on the purpose of assistance (such as to purchase books or uniforms, or to support travel expenses) and the total value of such assistance received from the American Job Center or from other agencies or organizations.

Education and training. Sample members will be asked for information about each education and training program they attended from random assignment to the 15-month follow-up (in the 15-month follow-up survey), and between the 15- and 30-month follow-up (in the 30-month survey). First time responders to the 30-month survey will be asked about programs over the full 30-month period. The surveys will collect information to detail the duration of education or training pursued, type of education or training, the education or training provider, total costs and out-of-pocket costs, whether the course was completed, and resulting credentials as further detailed in Table A.2. This information will be collected on each program, regardless of whether it was funded by WIA, or whether the customer completed the program.

Employment and earnings. Because the goal of the WIA intensive services and training is to improve customers’ labor market outcomes, key outcomes for the evaluation are related to employment and earnings. Given the importance of these outcomes, we plan to collect a complete and detailed history of all jobs held by sample members for 30 months after they are randomly assigned. Items in each of the 15- and 30-month surveys will collect basic information about jobs for pay including: earnings (from each job), employment (current status, number of jobs, periods of unemployment); the characteristics of each job held (industry and occupation, hours worked, wage rates, and type of employment agreement); job retention (how long held each job, reasons for job separations); and measures of job quality (the availability of fringe benefits, presence of unions). Detailed information on earnings from each job is critical as earnings represent both a key outcome for the impact analysis as well as the main benefit that will contribute to the benefit-cost analysis.

Self-sufficiency. A goal of employment policy is self-sufficiency for the participant and his/her household. Thus, for the impact analysis, the two surveys will collect information on household composition and receipt of public assistance—whether Federal or state—by any member of the household in which the sample member lives. Specifically, the 15- and 30-month surveys will collect information on the receipt and amount of public assistance received, such as benefits through SNAP, TANF, or other cash assistance program, and the WIC program.³ In addition, we will ask about total household income in aggregate.

Demographic and individual characteristics. Respondents will be asked to confirm any items they did not complete on their study registration form such as their gender and race and ethnicity. In addition, we will ask about the extent and type of health insurance coverage for the sample member over the 30-month study period. Finally, the surveys will ask about limitations to work due to disabilities or health problems and any felony convictions. Limitations to work are important baseline measures because they can affect the impact of the intensive services and training. These items will be collected in both the 15- and 30-month surveys.

iv. Use of the Survey Data and by Whom

Mathematica will use the information collected from the 15-month and 30-month surveys to carry out the analysis needed to fully assess the effectiveness of WIA intensive services and training. This information will be used by Congress to determine future funding, by Congress and DOL to determine national workforce policy, and by state and local areas to decide on local policy. Specifically, the survey data will be used to conduct two analyses:

Impact analysis of services, training, and outcomes. Data collected through the two surveys will be used in the impact analysis to provide estimates of the offer of WIA intensive services and training on the receipt of services and training and on employment and self-sufficiency outcomes. The net impacts will be derived by comparing the average outcomes of individuals in each of the three research groups. Three comparisons of outcomes will be made: (1) those of the core group to those of the core-plus-intensive-services group, (2) those of the core-plus-intensive-services to those of the full-WIA group, and (3) those of the core group to those of the full-WIA group. Each respective comparison will provide information about the relative impact of intensive services over core, training over intensive services, and training (which by definition include intensive services) over core services. In addition to estimating overall impacts, impacts for different subgroups—by age, sex, race/ethnicity, and educational and employment background—will be estimated in order to determine who is or is not served well by the program as presently constituted. In addition to estimating the offer of intensive services and training the impact of the receipt of intensive services and training will be estimated.

Benefit-cost analysis. The impact estimates on employment and self-sufficiency outcomes derived from the survey data will be used to measure the benefits from increased employment, greater earnings, and reduced use of other public assistance. In addition, the survey data on service receipt—use of the resource room, workshops, peer-support groups, assessments, individual counseling and supportive services—will be used with cost data to assign a total cost of providing services for each customer. The surveys will ask about the cost of education and training programs. The benefit-cost analysis will place a dollar value on each benefit and cost of the program and then summarize in a single statistic all of the diverse impacts and costs associated with WIA service and training receipt.

b. Cost Data Collection Package

The surveys will collect the quantity of each service received, but for most services, they will not collect any data on the cost of each service because that information is generally not known to the respondents. For example, the survey will provide data on the number of hours of counseling received but not any information on the cost of this counseling. The cost data collection package will collect data on the costs of the services, such as an hour of counseling. The cost-per-unit for each service will be matched to customer service receipt data from the surveys to estimate a total cost of providing WIA services and training to each customer in the survey sample.

To facilitate the collection of data on the costs per unit of services received by study participants, the cost components that would be associated with the delivery of each service and training customers might receive was parsed out, as were described in the previous section. The cost components associated with each service and training are described below and listed in column 1 of Table A.3:

• Resource room. The resource room is predominantly self-serve; although a program employee typically staffs the resource room primarily to provide assistance logging on to computers, enforcing time limits, and directing customers to relevant materials. Therefore, the primary cost components of the resource room are overhead costs (for example, rent, utilities, and internet service provider fees) and the cost of employee time to staff the room, including both salary and fringe benefits. Since many people use the resource room at the same time, we must also have a measure of how many people visit in a typical week in order to calculate the average cost of such a visit.

• Workshops. The primary cost component of a workshop is staff time to prepare for, conduct, and follow up after the workshop. In addition, there might be costs of workshop-specific workbooks or other materials. Since more than one person typically attends a workshop, to calculate a per-participant cost of attending, data on the average number of customers typically at workshops will also be collected.

• Peer support group. A staff person typically moderates the peer support group and this person’s time is the primary cost of the group. Since more than one person typically attends a group, to calculate a per-participant cost of attending a group, data on the average number of customers typically attending these groups will also be collected.

• Assessments. The main cost components of assessments are the materials associated with paper-and-pencil assessments (for example, test booklets), licensing fees associated with online assessments, and any scoring fees associated with either type of assessment. The time spent by counselors discussing the findings from assessments with customers is accounted for in the individualized meetings with counselors.

• Individualized counseling. The primary cost component of a meeting with a counselor is the cost of that counselor’s time spent preparing for, conducting, and following up on the one-on-one meetings (including salary and fringe benefits).

• Supportive services. The cost of supportive services is simply the dollar value of those services, plus the cost of any counselor time to determine need for and process supportive services payments.

The cost of education and training programs includes the cost to enroll in the program. These costs can be paid for by an ITA, but also can be paid for out-of-pocket or by other programs.

In addition to the services customers can receive directly, American Job Center managers, supervisors, and staff, such as receptionists and administrative assistants, closely supports the work of the front line staff. Therefore, their salaries and fringe benefits contribute to the cost of delivering each service. Similarly, overhead costs are necessary for operating the programs but may not be associated with a particular activity. They include rent and utilities, computers, internet access, furniture, office supplies, and phone and fax lines. To incorporate these administrative and overhead costs, these costs per counselor hour will be calculated and added to the labor cost.

Data on costs will be obtained from five sources. Each source is described below. Table A.3 indicates which source will be used to collect each cost component described above.

Table A.3. Cost Components and Data Sources

Cost Component	Program Cost Questionnaire	Front Line Staff Activity Logs	Resource Room Sign-In Sheets	LWIA and Center Financial Records, ITA Data	15- and 30- Month Follow-up Surveys
Resource room
Resource room staff salary and fringe benefits				
Average number of customer visits to resource room			
Workshops
Workshop staff salary and fringe benefits				
Staff time preparing for, delivering, and following up after each workshop		
Average number of customers in each workshop	
Peer support groups
Group staff salary and fringe benefits				
Staff time preparing for, delivering, and following up after each group meeting		
Average number of customers in each group meeting	
Assessments
Licensing/scoring fees for assessments				
Individualized counseling
Time spent preparing for and following up after a meeting with a customer as a percent of time spent in meetings		
Counselor salary and fringe				
Supportive services
Cost of assistance					
Staff time spent prepping and processing supportive services				
Education and training
Tuition and cost of supplemental materials for training					
Administrative/supervision costs
Center manager salary and fringe				
Administrative staff (that is, greeter and administrative assistant) salary and fringe				
Overhead
Rent and utilities				
Furniture				
Computers				
Internet				
Phone/fax				
Photocopier				
Office supplies				

1. Program cost questionnaire. These detailed questionnaires will collect information from LWIA administrative staff on salaries and fringe benefits of all staff in the local area whose salary is paid at least in part by WIA. This includes administrative/executive staff, managers and supervisors, front line staff, and administrative assistants. In addition, the questionnaire collects information necessary to compute per-participant costs of attending workshops and taking assessments. For workshops, this includes the cost of materials and average number of attendees, while for assessments this includes per-assessment scoring fees. It also includes all the inputs necessary to compute overhead costs, such as rent/mortgage and equipment costs at each American Job Center, utilities payments, and expenditures for office supplies.

2. Front-line staff activity log. The front-line staff activity log will collect information on what activities counselors engage in on a typical day. Counselors will use a template (see Appendix D) to record their activities using a set of pre-specified codes for each activity for one week. This information will be used to estimate the average preparation and follow-up time associated with one-on-one meetings with customers, time spent in resource rooms, and time spent in workshops and peer support groups.

3. Resource room sign-in sheets. To calculate the average cost per visit to the resource room, the evaluation team needs to know about how many people use the resource room. Most American Job Centers already collect this information, in which case it will simply be requested from all the American Job Centers in the local areas. However, for centers that do not already collect this information, or do not collect it in a way that can be used by the study, a resource room sign-in sheet will be provided for each American Job Center.

4. Administrative data from the local workforce investment area. Data on the tuition, fees, and other costs associated with training programs that WIA customers take will be collected directly from the LWIA. ITAs are the most common way that participants cover these costs, as well as the costs of the textbooks, uniforms, or other school supplies necessary for participation in the training program.

5. Fifteen-and 30-month follow-up surveys. In addition to ITAs, customers may receive supplemental financial aid assistance, such as Pell Grants, or pay for portions of their training costs out-of-pocket. The LWIA administrative data will be used for the value of the ITA, but the customer will be asked on the surveys for the total cost of the program and other sources of funding. The surveys will also ask for the total dollar amount received in supportive services.

LWIA administrative staff likely will be most adept at gathering the information we are seeking on costs for the program cost questionnaire; other respondents might include American Job Center managers or the managers of service providers under contract with the LWIA. The job titles of respondents to the questionnaire will likely vary across sites because some sites manage all expenditures through the central administrative office, whereas others delegate this function to the service providers who operate the American Job Centers. Whoever the respondents are, they will use existing LWIA and American Job Center financial data to compile the information on costs. During the first implementation site visit, information about the best person(s) to receive the program cost questionnaire will be collected.

For completion of the activity logs, two American Job Centers in each local area will be randomly selected and five-to-eight counselors listed in the online Random Assignment System then will be randomly selected to complete the activity log. (The number of selected counselors will depend on the size of two selected American Job Centers.) Ideally, information on the number of users of the resource room will be collected from every American Job Center in the local area that has a resource room; therefore, sign-in sheets will be distributed to all American Job Centers in the local area that do not already collect such data.

The cost data collection package (Appendix D) includes the program cost questionnaire, front-line staff activity log, and resource room sign-in sheet. This package will have a cover letter (Appendix D) and will be sent to sites upon receiving OMB approval. A dedicated group of researchers will be responsible for ensuring that the data are collected in an appropriate and consistent manner across sites, and will provide technical assistance to all sites as necessary.

c. VSS Site Visits: Selection and Activities

Qualitative data collection activities for the VSS are planned to begin with the second round of WIA Evaluation site visits, which are scheduled to start in November 2012. While the first round of WIA Evaluation site visits (conducted from March 2012 through September 2012) will document the LWIAs’ delivery of services to their WIA adult and dislocated worker customers, the second round of visits will, as one of its objectives, examine partnerships developed in local areas’ provision of services and encouraged by WIA, providing a relevant framework for data collection for the VSS. As currently planned and approved (OMB clearance number 1205-0482), the second site visits for the WIA Evaluation, will involve interviews with WIA senior staff, American Job Center staff and partners, group interviews with customers, and observations of program activities. Across all 28 LWIAs, the VSS will add questions asked of WIA senior staff and American Job Center staff to the existing protocols for the WIA Evaluation and will also add new data collection activities, including interviews with the veterans coordinator at the state level and with DVOP specialists and LVERs at the local level. In a subset of eight LWIAs, chosen based on the size of the LWIA’s veteran population and information regarding the implementation of promising practices, the VSS will also conduct focus groups with, and case file reviews of, veterans participating in American Job Center services and training.

Evaluation team members will be thoroughly trained on the purpose of each activity, the information needed, and best practices in collecting reliable and high quality data. We anticipate that the WIA Evaluation site visits will be conducted over three-to-four days in each site. The VSS will add an additional half day to the visits to the 20 LWIAs in which staff are not conducting case file reviews and focus groups with veterans, and an additional day in the eight LWIAs where these activities are taking place.

i. Methods and Content of Qualitative Data Collection

Evaluation team members will use protocols to collect data through interviews, focus groups, and case file reviews on how veterans are identified, what services are provided, how priority of service is implemented, and other topics of interest. Understanding what services veteran customers receive and how service receipt varies across sites constitutes one of the most important objectives of the VSS. Protocols for each site visit activity are included in Appendix E while the consent form to participate in the focus group discussions and for case file review are included in Appendix F.

Staff interviews. As part of the VSS, site visitors will conduct interviews with three types of staff (each with its own set of interview questions):

1. State veterans coordinators. We will interview the state veterans coordinator in each of the 19 states with an LWIA participating in the WIA Evaluation. The interview will focus on state guidance provided to LWIAs regarding the provision of services to veterans and POS, as well the coordinator’s perspective on the coordination of services, the state’s and the participating site’s innovative or unique services provided to veterans, and challenges to, and promising practices in, providing services to veterans.

2. LVERs and DVOP specialists. We will interview an average of two Veterans State Grant-funded staff per LWIA to learn about their provision of services to veterans, how they interact with other staff in the American Job Center system, challenges working with veterans, and any promising practices in delivering services to veterans.

3. American Job Center staff. For interviews with up to six LWIA and American Job Center staff in each LWIA, site visitors will use a VSS module to ask questions related to the veteran customers. Topics include identification of veterans and eligible spouses, the services provided to veterans, and staff’s interactions with LVERs and DVOP specialists.

The protocols for each interview or activity list topics and suggested questions to be discussed by the evaluation team during in-person interviews with staff. They are not meant to be administered word-for-word, but rather used as a guide to topics to be covered during the interviews. They will ensure that all site visitors capture the necessary data consistently across sites, but allow for flexibility for site visitors to adapt to the particular service structure and context of each site, pursue interesting issues that arise, and skip questions for which they already know the answer from previous contacts with the sites or from program documentation. Some topics will be discussed with multiple staff members; doing so will provide information about topics from multiple perspectives.

Focus groups with veterans. In eight purposively selected sites, we will conduct focus group discussions with a convenience sample of American Job Center customers who are veterans to gather their perspectives on the accessibility and quality of services they received through the public workforce system. We will aim for variety within the group in terms of gender, enlisted or officer status, presence of service-connected disability, period of service, and job market prospects, as these will allow for a potentially more varied set of viewpoints. If we are unable to conduct a focus group in any of the eight selected sites, we will attempt to talk individually with up to three veterans who received services at that site.

Review of a small sample of individual case files. Case files of three focus group members will be reviewed during site visits to each of the eight purposively selected sites. The site visitor team will select cases that reflect a range of characteristics and experiences. Specifically, the team will aim for a mix of one post-9/11-era veteran and one pre-9/11 veteran, one woman, and one service-disabled veteran. These reviews will provide concrete examples of the experiences of veteran customers, including services received and the way in which customers progressed through services.

ii. How Data Will Be Used

The data collected will be used by the evaluation team to conduct analyses that can be used by DOL, policymakers, program administrators, and service providers to assess what services veterans are receiving in the American Job Center system; how services and POS are being implemented; the collaboration between WIA, ES, and Veterans State Grant staff; and challenges and promising practices in providing services. Decisions can then be made by policymakers, program operators and others to improve offerings, provide additional guidance, and offer continuing and effective services to the nation’s veterans.

3. Use of Improved Technology to Reduce Burden

The follow-up surveys will be administered by computer-assisted telephone interviewing (CATI). CATI provides many benefits for both the data collectors and the respondents. Using CATI allows greater flexibility in scheduling for survey respondents, making the survey less burdensome to them. Also, CATI programming ensures skip logic, restricts entries to valid responses and checks for logical consistency across questions. Interviewers are thus able to correct errors during the interview, eliminating the need for callbacks to respondents, further reducing the burden on respondents as well as keeping costs in check. In cases when field locators are needed (when sample members cannot be reached through multiple attempts by phone), locators will be equipped with cell phones and will encourage sample members to call into a centralized call center where a project-trained interviewer will administer the CATI interview. This is less costly and burdensome than paper-and-pencil interviewing which typically requires longer administration time; turning pages and following skip instructions using a hard copy questionnaire takes longer for the interviewer to administer, thus increasing respondent burden.

To further minimize burden for respondents, both surveys will be preloaded with key information to facilitate data collection. For example, the 15-month follow-up survey will include preloaded information from the study registration form completed by study participants at study intake (as discussed in the response to item 1). Data such as date of birth and the last four digits of the Social Security number will be used to confirm sample members’ identities. Employer names from the study registration form will frame questions about employment at time of (or just prior to) study intake. Similarly, data collected at the 15-month follow-up will be preloaded for surveys conducted at the 30-month follow-up (when applicable).⁴ Using previously collected data can aid respondent recall and ensure that only new information is collected, thereby reducing burden.

Finally, using CATI and a call scheduler translates into less time burdening the sample member’s household with calls at inappropriate times and/or in incorrect languages. An automated call scheduler will simplify scheduling and rescheduling of calls to respondents and can assign cases to specific interviewers, such as those who are trained in refusal conversion techniques or those who are fluent in Spanish. In addition, CATI almost completely eliminates dialing errors because calls are made through a preview dialer. The preview dialer allows interviewers to review case history notes and the history of dispositions. The interviewer then presses one button to dial the number after reviewing the case (this is akin to one-touch or speed dialing).

The cost data collection forms will be provided to the individuals previously identified by the WIA Evaluation liaisons with each site. To minimize burden placed on the LWIA staff, those responding will be asked to forward the data requested in the program costs questionnaire in their preferred form. For example, they may provide the study team with existing tables or access to existing databases. Alternatively, if they are more comfortable doing so, they may send us data electronically rather than completing the hard-copy questionnaire. We anticipate that most American Job Centers already collect information on the number of customers using the resource room, and many do so electronically.

The VSS’s qualitative component collects data on veterans and their services through on-site visits and thus does not have burden that could be reduced through use of technology.

4. Avoiding Duplication of Effort

There is no similar prior or ongoing data collection being conducted that duplicates the efforts of the proposed data collection for the evaluation with adult and dislocated workers or the data collected specifically for the study on veterans. Specific efforts have been made to reduce the overall burden on the sites and respondents by making efficient use of baseline data from the study registration form in the follow-up surveys, and supplementing administrative data with the rich and detailed data available only from direct customer surveys.

Some data items included in the follow-up surveys are available from administrative data sources, but not with the same level of detail and coverage as can be obtained from the direct customer surveys. For example, while UI quarterly earnings data will be collected for the entire evaluation sample, these administrative data tend to be less accurate than the survey data, for several reasons. The UI earnings data do not cover all workers (the data cover 90 percent of all workers); they exclude Federal workers, military staff, self-employed people, railroad employees, workers in service for relatives, most agricultural labor, some domestic service workers, part-time employees of non-profit organizations, insurance and real estate agents on commission, and workers performing what is referred to as casual labor (U.S. Department of Labor, 2004). They also exclude workers whose employers (illegally) fail to report their earnings to the UI agency. In addition, the administrative records will exclude earnings from UI-covered jobs held out-of-state or if the participant moved to a different state during either of the follow-up periods.

Similarly, administrative data on service and training receipt will be obtained from WIA files maintained by states with participating sites. The WIA state-maintained files provide information on services and training funded by WIA however these data do not include details on the types of services and training received. For example, they record that the customer received an intensive service, but not the type of service. Since the costs of such services differ depending on what specifically is received (for example, a one-on-one counseling session versus attending a workshop), it is important to distinguish service receipt at a finer level than is available in the administrative data. Most importantly, these data do not cover services and training that are not funded by WIA; the surveys are the only means of collecting information about non-WIA funded services and training received by members of the three research groups.

The data collected by the cost data collection package, including staff activity logs, are not currently available in the form needed to support the evaluation benefit-cost analysis. These data collection efforts make use of existing information on WIA costs, but do so in a manner that supports uniformity in the construction of cost items across the sites. For example, the allocation of staff time across the tasks relevant to the evaluation can only be collected through the stylized activity logs developed for this purpose; WIA staff members are not often required to break down the specific tasks they perform within WIA funding streams through existing methods for recording working hours. If LWIAs have the data needed in another form that still permits cross-site comparisons, the evaluation team will accept the data in whatever form is most convenient for the program staff. For example, some sites might already have a method for collecting information on how many customers access the resource room. If these approaches capture the number of customer visits in a consistent manner, sites will not be required to use the resource room sign-in sheet.

Veteran-related information collected from the 28 LWIAs will not duplicate any other effort. The VSS will be the only source of information for how the 28 LWIAs in the WIA Evaluation provide services to their veteran customers.

5. Methods to Minimize Burden on Small Businesses or Entities

Both follow-up surveys will be conducted with individual WIA program participants. The cost data collection will be compiled with the assistance of WIA administrative and front-line staff. The veterans’ information for the VSS will be collected directly from WIA administrative and front-line staff and veterans through in-person interviews. The evaluation team will not contact small businesses or entities.

6. Consequences of Not Collecting the Data

The data collection efforts in this data collection request are designed to provide information to answer questions of interest to policymakers and program operators. The consequences of not collecting these data are described in three subsections, one addressing each type of data collection—the follow-up surveys, the cost data collection forms, and the veterans’ data collection for the VSS.

a. Follow-up Surveys

The follow-up surveys will be a critical source of reliable and consistent data about sample members’ service use, employment, and self-sufficiency for all three study groups. Importantly, without data from the first follow-up survey, the study will not be able to capture information on the non-WIA employment services and training customers receive during their 15-month participation period. This information is critical in order to be able to assess the types of services the sample members in the core group received in the absence of the WIA Adult and Dislocated Worker programs. It is also critical to collect this information for the sample members of the other two study groups (core-plus-intensive and full-WIA) who may also access services and training from sources other than WIA. It is necessary to have information on the full set of services and training received by sample members in each of the three study groups to assess the impact of WIA intensive services and training on patterns of service and training receipt, as well as to contribute critical information to develop cost estimates for the benefit-cost analysis.

The two surveys will also be important for providing information on study participants’ earnings and other employment outcomes that will, again, contribute critical data elements for the impact analysis as well as the benefit-cost analysis. Although earnings and employment data will be collected from UI quarterly earnings records, these data are incomplete in ways that will affect the study’s ability to evaluate the impact of WIA intensive services and training (as discussed in section 4 above). For instance, UI earnings data do not contain the dollar value of any fringe benefits the employee might receive. The evaluation team will use the survey data on earnings—rather than the administrative data—to develop estimates of the benefits of each increasing level of WIA service/training receipt because of the greater completeness and accuracy of these data. Development of these estimates will not be possible without this data.

b. Cost Data Collection Package

The program costs questionnaire, which will be completed using existing LWIA financial data, will provide the only source of information about the cost of delivering WIA services to customers in each site for use in the evaluation. In addition, the front-line staff activity log will be the only source of information on how much preparation and follow-up time is associated with WIA counselor activities. It is important to incorporate this additional preparation and follow-up time into the total cost of a counselor’s time because otherwise the total time for services (such as workshops and counseling) would be underestimated. Finally, the resource room sign-in sheets are needed to obtain an estimate of the average number of customers accessing the resource room; without this information it would not be possible to calculate the average cost of a visit to the resource room. If these components of the cost data are not collected, then the evaluation will be unable to answer critical questions about the cost effectiveness of WIA intensive services and training.

c. Site Visit Data Collection for the VSS

The WIA Evaluation excludes veterans from the experimental study. However, the framework in place for the WIA Evaluation provides an important opportunity to collect information on how services are delivered to veterans within the American Job Center system. The VSS is necessary for generating focused information on what services are provided to veterans and how they are delivered, the roles and interactions of various staff in providing services to veterans, and special issues in serving veterans in the nationally representative set of 28 LWIAs. Without this information, we will lose understanding of how the WIA Evaluation’s LWIAs are providing services to an important set of their employment and training customers.

7. Special Data Collection Circumstances

No special circumstances apply to this data collection. In all respects, the data will be collected in a manner consistent with Federal guidelines. There are no plans to require respondents to report information more often than quarterly, to submit more than one original and two copies of any document, to retain records, or to submit proprietary trade secrets.

8. Federal Register Notice

a. Federal Register Notice and Comments

A Federal Register notice announcing plans to submit this data collection package to OMB was published on June 25, 2012 (FR, Vol. 77, No. 122, pp. 37923-37926) consistent with the requirements of 5 CFR 1320.8 (d). The Federal Register notice described the evaluation and provided the public an opportunity to review and comment on the data collection plans within 60 days of the publication, in accordance with the Paperwork Reduction Act of 1995. A copy of this 60-day notice is included as Appendix G.

DOL received two comments to the Federal Register notice published on June 25, 2012. In response to the second comment, ETA increased the estimate for the VSS burden. See below.

The private citizen, Jean Public, among other observations, questioned the value of the evaluation, especially during poor economic times.

Response: It is important that we conduct this study. In the 14 years since the WIA was enacted, its service components have never been rigorously evaluated. We need to learn if WIA-funded intensive services and training are as effective as they can be. This is especially important now when so many people are out of work and seeking services to help them transition to new employment opportunities.

A participating local site, while expressing full support for the research addressing the effectiveness of WIA services, had comments about the new data collection. The comments and ETA’s responses follow:

1. Comment: The commenter expressed concerns for the estimated length of the survey—40 and 30 minutes for the 15- and 30-month follow-up surveys, respectively—and the burden on customers. The commenter also indicated that this length could be longer when the survey is administered to customers who are Limited English Proficient (LEP).

Response: Mathematica has successfully administered surveys of this length with similar populations for studies conducted for the U.S. Department of Labor as well as for other agencies. However, we recognize that such surveys introduce burden and address this in several ways. First, customers who complete the survey will be paid $25 as a token of appreciation and acknowledgement of their time. Second, appointments to conduct interviews will be scheduled at the convenience of each customer. Furthermore, customers can complete the interview in more than one call if they prefer to do so. (We have found that most respondents complete surveys of this length in one call.) The survey has been comprehensively pretested with local WIA customers, and they were able to complete the survey without problem. Finally, most customers will not be required to answer every question in the survey. The questionnaire is designed to skip questions (and even entire sections) if it does not apply to a particular customer’s situation.

For customers who are limited English proficient (LEP), we do not expect the length of administration to be doubled. First, to address the needs of Spanish speakers, the survey can be administered in Spanish.. Thus, the administration of the survey to Spanish speakers should be similar to the administration for English speakers. Second, for people who speak neither English nor Spanish, the same option of completing the survey in multiple calls is available. Mathematica staff will be used as interpreters when possible, and we will employ interpreter services for languages for which staff are not available and the demand warrants it.

2. Comment: The commenter suggested that the study find avenues other than the surveys to collect data on the services participants receive. For service use data, the commenter notes that quarterly extract data is already planned from WIA databases.

Response: The survey is a critical source of information that we cannot obtain from administrative data. Most importantly, without the survey data, we would not obtain information about the employment and training services that customers receive outside of the WIA program. The study needs to capture this information for all customers in the evaluation in order to estimate the effects of WIA intensive and training services on customers.

As indicated by the commenter, the study team is collecting local areas’ WIA administrative data. We intend to use these data to confirm that customers have maintained their study group assignment and to describe WIA services received by customers. However, often, local areas’ administrative databases do not provide the detail about services required for the evaluation. For example, the administrative data might inform us that the customer has received intensive services, but not whether the service received was an intensive workshop or a counseling session. Through the survey, we will be able to collect this detailed information.

To be sensitive to the burden on customers, we do intend to rely on administrative data when available. For example, the study team will be collecting unemployment insurance information from state databases, so we will not ask customers about their receipt of unemployment insurance.

3. Comment: The commenter recommends that the team consider using a Web-based survey methodology for the follow-up surveys as well as electronic submission of responses in the other two proposed data collection efforts—the cost collection and Veterans’ Supplemental Study (VSS). For the survey, the commenter notes that electronic-methodologies may increase the efficiency of the survey data consolidation process.

Response: Based on our experience, we believe that computer-assisted telephone interviewing (CATI) is the best option for completing the follow-up surveys. Many customers will not have Internet access at home and will therefore need to use public facilities such as libraries or WIA offices to complete a web-based survey. Also, it is more likely that a respondent will start a web-based survey under these circumstances and not complete it than is the case with a telephone interviewer. In addition, Mathematica’s survey staff will offer customers flexibility when scheduling the CATI interview to meet the needs of each customer, and trained interviewers will be able to clarify questions if needed, improving the validity of responses and increased responses overall. The CATI administration provides the same benefits of consolidating the data as we would achieve with a web-based survey. CATI interviewers enter respondents’ information during the interview.

The cost data collection package provides a format for describing the types of data needed for the study’s analysis of the effectiveness of WIA services. We will encourage local areas to submit these data electronically if they are able to do so.

The VSS qualitative component does not lend itself to electronic submission of responses. In our experience, in-person meetings with staff and customers are invaluable to learning about how services are delivered. Both perspectives are important, and face-to-face discussions often elicit responses that one cannot obtain through other methods.

4. Comment: The commenter stated that the planning for the WIA Evaluation site visits takes a minimum of 40 hours and suggested that the burden estimate for the VSS on-site data collection include this staff time.

Response: We agree that planning the site visits requires staff time. Thus, we have added 4 hours per site for staff to coordinate the VSS portion of the second round visits. The full burden for the coordination of the WIA Evaluation site visits, which were cleared under a previous data collection package, is outside this clearance request.

These comment summaries and responses are also in Appendix H.

b. Consultations Outside of the Agency

DOL and the study team did not engage in any outside consultations for the follow-up survey, cost data collection instruments, or VSS.

9. Respondent Payments

This submission seeks approval to: (1) offer sample members selected for a follow-up survey a $25 incentive to complete the 15- and 30-month surveys and (2) offer $40 to sample members who are unresponsive to outreach attempts. At intake, participants are advised that they could be contacted to complete a survey and that they would receive an incentive payment for survey completion. The letter sent in advance of contacting the sample member for the telephone interview states the incentive amount (see advance letter in Appendix C). For the 15-month survey, all sample members will be offered $25 in the advance letter. For the 30-month survey, all sample members other than those who received $40 for the 15-month survey will be again offered a $25 incentive payment for survey completion in the advance letter; sample members who received $40 for completing the 15-month survey will be offered $40 in the advance letter. Any sample member who does not respond to repeated outreach attempts to complete either survey will be offered $40 in a reminder postcard (see nonrespondent reminder postcard in Appendix C).

A sample member will be deemed to be unresponsive to outreach attempts and hence eligible for the $40 incentive payment only if the sample member has not completed an interview after: (1) three months have passed since the first attempt to contact the sample member; and (2) within these three months, 15 attempts have been made to call the respondent and three letters or postcards have been sent. Sample members will be offered the $40 in a postcard sent to their home in the fourth month after the first attempt to contact the sample member (nonrespondent reminder postcard in Appendix C). The postcard will provide a telephone number for the sample member to call and complete the interview. The sample member will also be called with the $40 offer after the postcard is mailed. Five months after the first attempt to contact the sample member, if the sample member has not completed the interview, field locators will be sent to the last known address of the sample member and offer a $40 incentive for completing the interview.

Incentives can help support high data quality by ensuring high overall response rates and by increasing the response rates from subgroups that are less likely to cooperate with the survey request. Incentives can help achieve high response rates by increasing the sample members’ propensity to respond and can reduce the likelihood that we need to send a field locator to complete the interview (Singer et al. 2000). And, studies have shown that incentives may reduce differential response rates and hence the potential for nonresponse bias (Singer and Kulka 2002). For example, there is evidence that incentives are effective at increasing response rates for people with lower educational levels (Berlin et al. 1992) and low-income and nonwhite populations (James and Bolstein 1990). In addition, a recent study found that incentives increased the participation of sample members who were more likely to be unemployed (Jäckle and Lynn 2007). Further, studies have found that paying incentives does not distort responses and, thereby, impair the quality of the data obtained (as reflected in item nonresponse or the distribution of responses) from groups that would otherwise be underrepresented in the survey (Singer et al. 2000).

Evidence suggests that the incentive cannot be much lower than $25 for adults. An incentive experiment from the 1996 panel of the Survey of Income and Program Participation showed that a $20 incentive significantly increased response rates, while a $10 incentive had no effect relative to those who received no incentive. Burghardt and Homrighausen (2002) found response rates for the third follow-up survey of youth in the National Job Corps Study were low with only a $10 incentive. When OMB approval was received to increase the incentive to $25, the response rate increased and the cost per completed interview was nearly 20 percent lower than those interviews conducted with the $10 incentive.

Our request that a higher incentive be offered to WIA Evaluation sample members who are unresponsive to initial outreach efforts is based on our success with this incentive structure in DOL’s National Evaluation of the Trade Adjustment Program. In 2008, with OMB’s approval (OMB clearance number 1205-0460), Mathematica conducted an experiment that offered different levels of incentives to sample members who were not responding to outreach attempts. Nonrespondents were randomly assigned to three groups: (1) a group that was offered an incentive of $25, the same amount as paid to respondents; (2) a group that was offered an incentive of $50; and (3) a group that was offered an incentive of $75. The experiment found that the response rate was 9.4 percentage points higher with an incentive of $50 than an incentive of $25, a difference that was statistically significant; the response rate was 15.0 percentage points higher with an incentive of $75 than an incentive of $25.⁵ On the basis of this experience, we decided that a $50 incentive to sample members who do not respond to initial outreach efforts was cost effective but that paying an incentive of $75 was not cost effective.

Our estimated cost of providing incentives for completion of the two follow-up surveys is $260,760, assuming that 90 percent of completers (or 8,856 respondents) will receive an incentive of $25 and 10 percent (or 984 respondents) will receive an incentive of $40. We estimated this 10 percent based on Mathematica’s survey operational experience in completing surveys, specifically in the two rounds of interviews for the Evaluation of the Individual Training Account Demonstration. For that study, which had a similar field component for the nonresponders, field interviewers generated about 10 percent of all completes. By offering the higher incentive to this nonrespondent group, we hope to convert these cases to CATI completes from field interviewer completes, which are more expensive. In the absence of providing any incentive, we have estimated that to obtain the 82 percent response rate the cost of the follow-up surveys would increase by about $539,000 (15 percent).

Veterans will not be paid for participating in the focus group discussions or for the research team’s review of their case files. Staff will not be compensated for interviews conducted during the site visit discussions on services to veterans. Nor will staff be paid for completing the instruments in the cost data collection package.

10. Confidentiality

Evaluation researchers have a strong set of methods to ensure that the privacy of data is protected. Mathematica institutes, and researchers must follow, policies related to (1) privacy, (2) physical and technical safeguards, (3) approaches to the treatment of personally identifying information (PII), and (4) survey related procedures.

a. Policy

All Mathematica and subcontractor evaluation staff will comply with relevant policies related to secure data collection, data storage and access, and data dissemination and analysis. Mathematica’s security policy meets the legal requirements of The Privacy Act of 1974 (System of Records Notices DOL/ETA-15); the “Buckley Amendment,” Family Educational Rights and Privacy Act of 1974; the Freedom of Information Act; and related regulations to ensure and maintain the privacy of program participants.

It is the policy of Mathematica to efficiently protect this information and data in whatever medium it exists, in accordance with applicable Federal and state laws and contractual requirements. In conjunction with this policy, all Mathematica staff shall:

1. Comply with the Mathematica Confidentiality Pledge, which is signed by all Mathematica full-time, part-time, and hourly Mathematica staff, and with the Mathematica Security Manual procedures to prevent the improper disclosure, use, or alteration of personally identifiable information (PII) information. Staff may be subjected to disciplinary or civil or criminal actions or both for knowingly and willfully allowing the improper disclosure or unauthorized use of PII information.

2. Only access PII and proprietary information in performance of assigned duties.

3. Notify their supervisor, the project director, and the Mathematica security officer if PII information has been disclosed to an unauthorized individual, used in an improper manner, or altered in an improper manner. All attempts to contact Mathematica staff about any study or evaluation by individuals who are not authorized to access the PII information will be reported immediately to both the Mathematica project director and the Mathematica security officer.

4. As part of their contract with DOL, all regular status and on-call staff who have access to PII will adhere to all DOL security requirements, including fingerprinting and background checks.

b. Safeguards

Mathematica has established safeguards that provide for the security of PII data and the protection of the data provided by individuals on all of its studies. Safeguards to ensure the privacy of data include:

1. Facility. The doors to office space and the survey operations center (SOC) are always locked, and all SOC staff are required to display a current photo identification while on the premises. Visitors are required to sign in and out of company offices and are required to wear temporary identification badges while on the premises. Any network server containing PII data is in a controlled-access area. All authorized external access is through a protected internet network that is under strict password control.

2. Network. Data stored on network drives are protected using the security mechanisms available through the network operating system used on Mathematica’s primary network servers: Novell Netware 5–6.5. These versions of Novell Netware are compliant with the C2/E2 Red Book security specifications. Netware is certified at the National Computer Security Center’s Trusted Network Interpretation Class C2 level of security at the network level. The network is protected from unauthorized external access through the PIX Firewall from CISCO. This firewall resides between the network and the communications line over which the corporate internet traffic flows. Access to all network features such as software, files, printers, internet, email, and other peripherals is controlled by user ID and password. Network passwords must be a minimum of eight characters in length and must be a combination of numbers and letters. All user ID, passwords, and network privileges are revoked within one working day for departing staff and immediately for terminated staff. All staff members are required to log off the network before leaving for the day.

3. Printers. Printer access is granted to all staff with a valid user identification (ID) and password. The physical hard disks on which the printer queues reside are subject to the same security/crash procedures that apply to the file servers. Print stations are monitored appropriately depending on the sensitivity of the printed output produced. No PII or proprietary data or information may be directed to a printer outside of Mathematica’s offices.

4. Electronic communication. Ethernet is used for internal email communications over the network. As Ethernet communications use Novell Netware with built-in user ID and password protections and Windows NT Challenge Handshake Authentication Protocols, sensitive information in both email text and attachments may be safely transmitted. Email transfer is also encrypted when sent to or from the Mathematica gateway facility, which allows staff to check and send emails from home. A dedicated private line supports cross-office communications between Mathematica offices.

As part of the VSS, prior to discussions with staff, individual veterans, or focus groups of veterans, the research team conducting the evaluation will be introduced and participants will be told that all information they provide will be used for research purposes only. They will be assured that they will not be identified by name or in any way that could identify them in reports or communications with the U.S. Department of Labor.

Research team members who play a role in data collection and analysis will be trained in procedures for safeguarding PII and will be prepared to describe these procedures in full detail and to answer any related questions raised by participants.

c. Treatment of Data with Personal Identifying Information

All data containing PII, including social security number, name, home address, and home telephone number, are considered to be sensitive or private, project-specific WIA data. Specific details regarding the handling and processing of PII for the WIA Evaluation are provided next.

1. Access. Electronic files containing PII will be stored in restricted access network directories. Access to restricted directories is limited on a need-to-know basis to staff who have been assigned to and are currently working on the project. When temporarily away from their work area, project staff members close files and applications. Access to workstations will automatically lock within a set period of minutes, and staff must use a password to regain access through the protected screen saver.

2. Electronic communications. Although the protections offered by internal email are extensive, staff members are instructed not to transmit sensitive information as a regular file attachment to an internal email. Instead, staff members are instructed to use the insert shortcut feature in Outlook to include a shortcut to the file. This allows the receiver to go to the file directly but will not allow access to unauthorized individuals. Additionally, staff members are instructed not to include sample members’ names or other personal identifying information in internal emails so that there is no potential for these to be viewed by others. When information about a sample member is transmitted via email, a Mathematica identification number is used as a reference. To ensure the security of sensitive information sent outside of Mathematica through an email, the sender is obligated to ensure that the recipient is approved to receive such data. When files must be sent as attachments internally or outside of Mathematica, staff are instructed to use WinZip 9.0 (256-bit AES encryption) to password protect the file. When sending sample member name and contact information outside of the company, this information will be included in a secure attachment rather than in the text of the email.

3. Databases. The databases developed for this study containing PII information will be password protected and accessible only to staff who are currently working on the project. To access the database, users will first log on to their workstations and then to the database using a separate log-in prompt. The database will be removed and securely archived at the end of the data-processing period.

4. Interview recordings. We will ask respondents in interviews and focus groups for permission to record the sessions as a backup for thorough note taking. If a respondent does not consent, the session will not be recorded. If the respondent does consent, the session will be recorded on a digital recorder. We will assure respondents that all comments will remain anonymous, that only the project staff will have access to the audio files (and only for the purposes of note taking), and that all recordings will be destroyed when the final set of notes is completed. Site visitors will be instructed to download the recordings onto the secure network and then destroy the recordings on the recorders.

5. Public use data files. To allow external verification and replication of the study findings, as well as additional research, public use data files containing key analysis variables created for the evaluation will be produced at the end of the study and formatted to data.gov specifications. These public use files will follow the current OMB checklist on privacy to ensure that they can be distributed to the general public for analysis without restrictions. Steps will be taken to ensure that sample members cannot be identified in indirect ways. For example, categories of a variable will be combined to remove the possibility of identification due to a respondent being one of a small group of people with a specific attribute. Variables that will be carefully scrutinized include age, race and ethnicity, household composition and location, dates pertaining to employment, household income, household assets, and others as appropriate. Variables will also be combined in order to provide summary measures to mask what otherwise would be identifiable information. Although it cannot be predicted which variables will have too few respondents in a category, the study researchers plan not to report categories or responses that are based on cell sizes of less than five. If necessary, statistical methods will be used to add random variation within variables that would be otherwise impossible to mask. Finally, variables that could be linked to identifiers by secondary users will be removed or masked.

d. Follow-up Surveys: Privacy and Security

All respondent materials—letters and reminder postcards—will include assurances of privacy protection. In addition, as part of the interviewer’s introductory comments to the telephone interview, sample members will be told that their responses are anonymous and will have the opportunity to have any questions answered. Interviewers will be trained in these procedures and will be prepared to describe them in full detail, if needed, or to answer any related questions raised by participants. For example, the interviewer will explain that the individual’s answers will be combined with those of others and presented in summary form only.

All data items that identify sample members will be kept only by Mathematica, for use in assembling records data and in conducting the interviews. No data received by DOL will contain personal identifiers, thus precluding individual identification.

• Telephone interviewing. Telephone interviewers for the evaluation survey will be seated in a common, supervised area. As part of the process to verify that the correct sample members have been reached, interviewers will have access to respondents’ names and birthdates, as well as the last four digits of their Social Security Number (SSN). Birth date and the last four SSN digits will be displayed on the computer screen only temporarily, at the beginning of the survey, so that the interviewer can verify the sample member’s identity. Interviewing staff for this project will receive training that includes general security and privacy procedures, as well as project-specific training that includes explanation of the highly private nature of this information, instructions to not share it or any PII with anyone not on the project team, and warnings about the consequences of any violations. Telephone interviews are recorded for educational and training purposes only, to aid interview staff in improving their skills, and are then destroyed.

• Locating. Staff members who work on updating sample member contact information when the original contact information is no longer valid must have access to key identifying information for short periods. These staff members will receive training that includes general security and privacy procedures, as well as project-specific training that includes clear instructions on what data and databases can be accessed and what data are required and can be recorded in a database. In addition, locators may talk to a sample member’s family, relatives, or other references to obtain updated contact information. To protect the sample member, locators are given scripts on what they can and cannot say when using these sources to obtain information. For example, locators will indicate that Mathematica is trying to reach the sample member for an important study sponsored by the DOL, but will not reveal the nature of the study. Postcards will similarly describe Mathematica’s need to reach the sample member.

• Locating and calling contact sheets. Project team members keep only the minimum amount of printed PII information needed to perform assigned duties. Hard-copy materials (such as locating or calling contact sheets) containing data with any individual identifiers (for example, name, street address) are stored in a locked cabinet or desk when not being used. When in use, such materials are carefully monitored by a project supervisor and are never left unattended. At the conclusion of the project, a final disposition of all remaining sample members will be made, and contact sheets and other associated materials will be destroyed.

• Data files. Electronic files for everyday use are created without personal identifiers. Data and sample files that must contain sensitive data are stored and analyzed on one of Mathematica’s “Secure_Data” drives. Specifically, staff working on this project will be instructed to maintain all files with PII data in project-specific, encrypted folders on the Mathematica network. Access control lists restrict access on a need-to-know basis and only to project staff members who are specifically authorized to view the sample data (as designated by the project or survey director) to select and process the sample or to process the data files. Sensitive data that are no longer needed in the performance of the project will be magnetically erased or overwritten using Hard Disk Scrubber or equivalent software, or otherwise destroyed.

• Hard-copy printouts. Sensitive temporary work files, used to create hard-copy printouts and stored in temporary work files on local hard drives, are deleted on a periodic basis. PII hard-copy output is shredded or stored securely once no longer needed. Test printouts of data records carrying personal identifiers that are generated during file construction are shredded.

e. Cost Data Collection Packages: Privacy and Security

The program costs questionnaire and the Front-Line Staff Activity log will include statements protecting the privacy of customers. Customers will be asked to supply only their initials on the resource room sign-in sheet. The LWIA name and the names of counselors filling out activity logs will be taken off as soon as the forms arrive and replaced with codes. All electronic information will be kept on secured drives at all times, hard copies will be secured in locked filing cabinets, and no information will be linked to an individual LWIA or customer.

f. VSS: Privacy and Security

As described above, the research team has a strong set of methods to ensure that data are protected. They consist of policies related to privacy, physical and technical safeguards, and approaches to the treatment of PII.

11. Questions of a Sensitive Nature

a. Follow-up Surveys

The follow-up surveys contain some questions that may be considered sensitive by some sample members. Obtaining information about these potentially sensitive topics is integral to addressing the research questions posed by the study. The survey questions around these topics have been worded to show the highest level of objectivity and sensitivity. Interviewers will also be trained to show sensitivity to respondents while remaining impartial. All questions in the current survey, including those deemed potentially sensitive, have been thoroughly pretested and many have been used extensively in prior surveys with no evidence of harm.

Further, as described in item 10, all participants will be assured of privacy at the outset of the interview and reminded throughout the interview as needed. All survey responses will be held in strict confidence and reported in aggregate in any reports or publically available documents, eliminating the possibility of individual identification.

The potentially sensitive questions and justifications for their inclusion in the survey instruments are presented in Table A.4.

Table A.4. Justification for Sensitive Questions in the Follow-up Surveys

Question Topic	Justification
Receipt of financial assistance in support of work, training, or school (Section B, Items B59a-B64)	Information about the receipt of financial assistance received to support work or training is important in assessing the impact of the offer of WIA intensive and/or training services on service receipt patterns across the three study groups. In addition, it is necessary to collect information on the total amount of assistance received to estimate the costs of services for the benefit-cost analysis.
Type, location, costs, and completion of training or education program (Section C)	Specific information about each training or education program in which the sample member participates is essential in: (1) estimating the impacts of the offer of WIA intensive services and training on the participation in training across the three study groups, (2) estimating the impacts on the completion of training and receipt of associated degrees or credentials across the study groups, and (3) computing the costs of training (for the individual and for the government) for use in the benefit-cost analysis. These questions have been used frequently in other DOL surveys including the evaluations of the Individual Training Account Demonstration and the Trade Adjustment Assistance Program with few issues with nonresponse.
Employment history over study period; characteristics of jobs held; and earnings (Section D)	Employment and earnings patterns are key outcomes for this evaluation and are necessary for answering the research questions about the effectiveness of access to WIA intensive services and training in achieving better outcomes for individuals than would be experienced in the absence of the program. The impact estimates on earnings contribute an important element to the analysis addressing the question of whether the benefits of WIA services (in the form of increased earnings) exceed program costs. These questions have been used frequently in other DOL surveys including the Individual Training Account Demonstration and the Trade Adjustment Assistance program demonstration with few issues with nonresponse.
Household income and receipt of public assistance (Section E)	Total household income and the receipt of public assistance are used to measure self-sufficiency, another key outcome of this evaluation. Similar to employment and earnings, data on these topics is critical in estimating the impacts of the offer of WIA intensive services and training across the study groups, and using these estimates to contribute to both the benefit (household income) and cost (receipt of public assistance) side of the equation in the benefit-cost analysis. Household income and sources and amounts of public assistance have been collected on many national surveys, including the Survey of Income and Program Participation, and have been used frequently in other DOL surveys. The survey questions for this evaluation are quite brief on this topic and they are aggregated for the household as a whole to obtain total income and sources of public assistance. In this way, the sample member does not have to disclose which member of the household receives specific benefits.
Receipt of health insurance (Section F, Items F2-F5x)	Receipt of health insurance is an important indicator of self-sufficiency and hence an outcome measure.
Individual characteristics including age, race and ethnicity, marital status, and level of education (Section F, Items F7-F11)	Data on these topics are important to collect in order to conduct an analysis of the impacts of WIA intensive services and training by subgroups of survey respondents. Such an analysis addresses a key research question about whether the effectiveness of WIA varies by population subgroup. Nonresponse to these items is rare.
Limitations to work including health problems, arrests, and felony convictions (Section F, Items F1 and F12a-F13b)	Limitations to work are important baseline measures because they can affect the impact of the intensive and training services. Health problems that affect work and felony convictions are two important barriers to employment. Recognizing the sensitivity of collecting information about arrests and felony convictions, these questions are asked at the end of the survey.

Table A.4 (continued)

b. Cost Data Collection Package

Elements of the cost data collection package request information that may be considered sensitive in order to produce accurate estimates of the cost of services for the benefit-cost analysis. As part of the program costs questionnaire, sites will be asked to provide salary and benefits information for all employees to contribute to the computation of the direct labor costs in providing services as well as indirect labor costs associated with services (such as administrative and support staff). To decrease the potential sensitivity about providing the salary and benefits information, the directions for the questionnaire ask that respondents fill out only initials and job titles for each staff person. In addition, respondents to the program costs questionnaire are assured in a statement that all information provided will be kept private to the fullest extent allowed by law.

The front-line staff activity logs ask that staff members record their activities for a day, a needed element in computing costs for services requiring staff time that cannot be captured from the customer survey. Respondents are assured in a privacy statement that all information provided will be kept private to the fullest extent allowed by law. Finally, the instructions for the resource room sign-in sheet note that customers may simply record their initials to ensure privacy.

c. Veterans’ Supplemental Study

The data collected on services to veterans contain no questions of a sensitive nature. However, the veteran focus group protocol and short form do ask questions about the participants’ background. Participants will be informed that they do not have to respond to any questions that they feel uncomfortable answering.

12. Estimates of the Annualized Burden Hours

Table A.5 presents the number of respondents, the number of responses per respondent, the average burden hours per response, and the total annual burden hours for each data collection activity for which clearance is being sought in this package. Table A.6 presents annualized estimates of indirect costs to all respondents for each data collection instrument, which total $48,778. Details on the time and cost burdens are provided below for each of the separate data collection activities.

Table A.5. Annual Burden Estimates for WIA Evaluation Follow-up Surveys, Cost Data Collection, and Veterans’ Supplemental Study

Activity	Annualized Number of Respondentsa	Number of Responses per Respondent	Average Burden Hours per Response	Total Annual Burden Hours
WIA Evaluation Follow-up Surveys
15-month follow-up	2,460	1	40 minutes	1,640
30-month follow-up	2,460	1	30 minutes	1,230
Average annualized burden for follow-up surveysa	2,460	-	35 minutes	1,435
Cost Data Collection Package
Program costs questionnaire	28	1	12	336
Front-line staff activity log	336	1	1.25	420
Resource room sign-in sheet	10,000	1	30 seconds	83
annual burden for cost data collectiona	-	-	-	839
Veterans’ Supplemental Study (VSS)
VSS visits - staff preparations	28	1	4	112
Staff Interviews - AJC staff	168	1	20 minutes	56
Staff Interviews - DVOP/LVER staff	56	1	1	56
Staff Interviews - State Veteran coordinators	19	1	1	19
Focus groups - Staff preparation	8	1	1	8
Focus groups with veterans	56	1	1	56
Annual burden for VSSa	-	-	-	307
Total Annualized Burdena
	-	-	-	2,581

^aThe follow-up surveys each span two years, however, the cost data collection and VSS data collection happen within one year.

Table A.6. Monetized Burden Hours

Activity/Respondent	Annualized Number of Burden Hours	Type of Respondent	Average Hourly Cost	Annualized Indirect Cost Burden
WIA Evaluation Follow-up Surveys
15-month follow-up	1,640	WIA customer	$7.25	$11,890
30-month follow-up	1,230	WIA customer	$7.25	$8,918
Annualized cost burden for follow-up surveys	-	-	-	$10,404
Cost Data Collection Package
Program costs questionnaire	336	WIA admin staff	$35.18	$11,820
Front-line staff activity logs	420	WIA front-line staff	$22.20	$9,324
Resource room sign-in sheet	83	WIA customer	$7.25	$602
Annualized cost burden for cost data collectiona	-	-	-	$21,746
Veterans’ Supplemental Study (VSS)
VSS visit - staff preparation	112	Local staff	$22.20	$2,486
Staff Interviews - AJC staff	56	Local staff	22.20	$1,243
Staff Interviews - DVOP/LVER staff	56	Local staff	22.20	$1,243
Staff Interviews - State Veteran coordinators	19	State staff	35.18	$668
Focus group - Staff preparations	8	Local staff	22.20	$178
Focus groups with veterans	56	Customer	7.25	$406
Annualized cost burden for VSSa	-	-	-	$6,224
Grand Total—Annualized Cost Burden
Annual Total Costa	-	-	-	$38,3749

^aThe follow-up surveys each span two years, however, the cost data collection and VSS data collection happen within one year.

a. Follow-up Surveys

Attempts will be made to complete interviews with 6,000 sample members in each wave of the follow-up surveys (at 15 and 30 months). To achieve the targeted response rate of 82 percent, we expect to complete interviews with 4,920 sample members for each survey. The surveys will each be fielded over a two-year period, producing an annual number of respondents of 2,460. Each of the two evaluation follow-up surveys will be administered once to each respondent. The surveys were designed to take an average of 40 minutes to complete using CATI for the 15-month follow-up survey, and 30 minutes for the 30-month follow-up survey.⁶ Therefore, the total annual burden to conduct the 15-month follow-up survey is 1,640 hours (4,920 interviews ÷ two years × 2/3 hours per interview), and 1,230 hours to conduct the 30-month follow-up survey (4,920 interviews ÷ two years × 0.5 hours per interview), for an annual total of 2,870 hours. At an average wage of $7.25 per hour—the Federal minimum wage—the cost estimate for this customer burden is $20,808. (The minimum wage is used as the opportunity cost to the customers.)

b. Cost Data Collection Package

The cost data collection package will be mailed to the relevant administrators—identified in the first implementation site visit—in all 28 LWIAs. These individuals will then distribute the program costs questionnaire, front-line staff activity log, and resource room sign-in sheets to the appropriate staff/American Job Centers. It will take a total of approximately 12 staff hours per LWIA to gather the information requested and complete the program costs questionnaire. This might involve one person consulting with several others, or several individuals completing the form. Hence, the total estimated burden on respondents is 336 hours (12 hours × 28 LWIAs).

The front-line staff activity logs will be provided to an average of 12 front-line staff per each of the 28 sites, for a total of 336 staff. Staff will be asked to record their activities for the week each day using pre-specified codes for activities. It will take each respondent approximately 15 minutes per day to record this information, for a total of roughly 75 minutes per respondent
(15 minutes × 5 days). Therefore, the total estimated burden on front-line staff to complete the activity logs is 420 hours (28 sites × 12 staff × 75 minutes ÷ 60 minutes).

For the resource room sign-in sheets, most American Job Centers and satellite offices already collect this data, but approximately 10,000 customers (about 15 percent of the study sample) will need to use the sign-in sheet provided by the study team. It will take 30 seconds on average for each customer to sign in, for a total burden of 83 hours (10,000 customers × .5 minutes÷ 60 minutes). Thus, the total estimated burden for the cost data collection package is 839 hours
(336 + 420 + 83).

At an hourly wage of $35.18, the average hourly wage of social and community service managers taken from the U.S. Bureau of Labor Statistics, National Compensation Survey, 2010, the cost estimate for WIA staff to complete the program costs questionnaire is $11,820 (336 hours × $35.18). The total estimated burden on front-line staff to complete the activity logs is 420 hours. At an hourly wage of $22.20, the average hourly wage of miscellaneous community and social service specialists taken from the
U.S. Bureau of Labor Statistics, National Compensation Survey, 2010, the cost estimate for this staff burden is $9,324 (420 hours × $22.20). And, the resource room sign-in sheets will take a total of 83 hours of customer time; at an average wage of $7.25 per hour—the Federal minimum wage—the cost estimate for this customer burden is $602 (83 hours × $7.25). The cost estimate for this respondent burden is $21,746 ($11,820 + $9,324 + $602).

c. Veterans’ Supplemental Study

The total annual burden for the VSS, which includes staff preparation for the interviews and focus groups, staff interviews and veteran focus groups, is 307 hours. To begin with, staff time to prepare for the VSS on-site activities is estimated at four hours of burden per site, or a total of 112 hours (28 sites × 4 hours). The staff interviews themselves consists of interviews with American Job Center (AJC) staff, DVOP specialists and LVERs, and state veterans coordinators, as described below. In each LWIA, the VSS will ask additional questions related to veterans of approximately six AJC staff who will be interviewed as part of the WIA Evaluation. The additional module for the AJC staff is estimated to take about 20 minutes per interview. For an average of six staff in 28 sites, the estimated burden for the additional module is 56 hours (6 staff × 28 sites × 20 minutes ÷ 60 minutes). In addition, in each site, the research team will interview an average of two DVOP specialists and/or LVERs. Each of these 56 interviews is estimated to last one hour, for a total of 56 hours of burden (2 staff × 28 sites × 1 hour). We also will interview the state veterans coordinators in the 19 states funding the 28 participating LWIAs. These interviews will also last an hour for a total burden on state coordinators of 19 hours (1 staff × 19 states × 1 hour).

For the focus groups with veterans, we estimate that it will take about an hour of AJC staff time to help us coordinate each focus group, for a total of eight hours (1 staff × 8 sites with focus groups × 1 hour). The focus groups will involve discussions with an average of seven veterans at each of eight sites. Hence, about 56 veterans will be involved in the discussions. Each discussion will last about one hour for a total veteran respondent burden of 56 hours (7 veterans × 8 sites × 1 hour).

Combining the burden for site visit preparation, staff interviews, focus group preparation and focus group discussions, the total annual burden for the VSS is 307 hours (112 + 56 + 56 + 19 + 8 + 56).

The total estimated burden on local LWIA (AJC and DVOP/LVER) staff to prepare for and/or participate in veteran services-focused interviews is 224 hours (112 + 56 + 56). At the hourly wage of $22.20 (see discussion in 12.c, above), the cost estimate for this staff burden is $4,973 (224 hours × $22.20). The cost estimate for the state staff burden of 19 hours and at the hourly rate of $35.18 (see above discussion) is $668 (19 hours × $35.18). For staff preparation for the veterans focus groups, at an hourly wage of $22.20 (see above discussion), the cost estimate is $178 (8 hours × $22.20). We estimate total customer burden hours for the focus groups at 56 hours. At an average wage of $7.25 per hour—the Federal minimum wage—the cost estimate for this customer burden is $406 (56 hours × $7.25). Thus, the cost estimate for the VSS is $6,224.

The estimated total cost burden to respondents or record keepers for the value of their time for this entire package is $48,778 ($20,808 + $21,746 + $6,224).

13. Estimates of the Total Annual Cost Burden to Respondents or Record Keepers

There will be no direct costs incurred by WIA customers (survey sample members) or WIA staff associated with the follow-up surveys, the cost data collection package, or the veterans’ data for the VSS. The only indirect cost to respondents is the cost of their time (see Table A.6).

a. Follow-up Surveys

Evaluation participants who are selected as survey respondents will not incur any out-of-pocket costs. Telephone calls will be placed at the expense of the evaluation contractor (Mathematica), and respondents who wish to call the interviewers will be provided with a toll-free number billed to Mathematica.

b. Cost Data Collection Package

The proposed cost data collection package will not require the site respondents to purchase equipment or services or to establish new data retrieval mechanisms. No capital or start-up costs are anticipated for the collection of these cost data. In addition, the evaluation contractor will pay for any costs associated with mail delivery of the completed cost data collection package.

c. Veterans’ Supplemental Study

Participants in the VSS will not incur any out-of-pocket costs.

14. Estimates of the Annualized Cost to the Federal Government

The total cost of the WIA Evaluation, including the VSS, to the Federal government is $24,416,039. Of this $24,026,039 will be paid to the contractor and $390,000 will be spent by DOL staff managing the study and overseeing the contractor. Since the WIA Evaluation (including VSS) will last nine years, the annualized cost to the Federal government is $2,712,893 ($24,416,439 ÷
9 years).

Of the $24,026,039 paid to the contractor, $22,951,040 is for the WIA Evaluation and $1,075,000 for the VSS. Of the funds paid to the research contractors to conduct the studies, about $1.552 million is for design and planning, $2.498 million is for site recruitment, $4.433 million is for payments to sites and states as compensation for staff time spent on the study, $2.176 million is for training site staff and providing technical assistance throughout the study, $10.309 million is for data collection, and $3.058 million is for analysis and reporting.

DOL staff will spend an estimated $390,000 (3.25 staff-year equivalents) managing the study and overseeing the contractor. About 3 staff year equivalents will be spent on the WIA Evaluation and 0.25 staff years on the VSS.

15. Changes in Burden

This submission is for new data collection.

16. Publication Plans and Project Schedule

Four reports will present findings from the evaluation: (1) a report on the implementation analysis (available in the spring 2014), (2) a report on the VSS (available in spring 2014), (3) a report on the 15-month net impacts of WIA intensive services and training for adults and dislocated workers (available in spring 2015), and (4) a final report on the 30-month net impacts and cost-effectiveness of those services and training (available in summer 2016). Table A.7 shows the schedule for the evaluation.

Table A.7. Schedule for the Evaluation

Activity	Date
Participant Intake period	November 2011 through April 2013
First site visits conducted	Spring - Fall 2012
Second (and VSS) site visits	November 2012 – April 2013
Implementation report & VSS report	Spring 2014
Administration of 15-month follow-up survey	February 2013 to December 2014
Administration of 30-month follow-up survey	May 2014 to March 2016
First impact report submitted	Spring 2015
Final report submitted	Summer 2016

17. Reasons for Not Displaying Expiration Date of OMB Approval

The expiration date for approval issued by OMB for the survey data collections (15- and
30-month follow-up surveys) will be printed on all materials sent to sample members such as letters and reminder postcards. It will also be listed on all three forms contained within the cost data collection package. It will also be presented on the VSS data collection forms.

18. Exception to the Certification Statement

Exception to the certification statement is not requested.

1 On average, adults spend about 32 weeks (8 months) receiving WIA intensive services and training and dislocated workers spend 42 weeks (10.5 months) receiving WIA intensive services and training (PY 2010 WIA Standardized Record Data (WIASRD) Data Book, Tables II-20 and III-23, respectively).

2 Part B, Section 1 provides a detailed discussion of the respondent universe and sampling for the evaluation as a whole, including an explanation of assignment rates to each of the three research groups.

3 Only receipt of WIC will be collected, not amount, since the WIC package varies by family and the sample member is unlikely to know the benefit amount in dollar terms.

4 When information is missing from either the study registration form or the 15-month survey we will be unable to preload information. However, we will then attempt to collect any missing information from prior data collection efforts.

5 Results conveyed in a memorandum to DOL,”Short-Term Results of the New Survey Procedures for the TAA Evaluation,” November 20, 2008.

6 The 30-month follow-up survey is estimated to take less time primarily due to lower service receipt. Periods of WIA service receipt are typically shorter than 42 weeks (see footnote #1) and reenrollment rates in WIA are estimated at less than 10 percent.

6