Public Comments and Responses

Comments on Proposed ICR for the WIA Random Assignment Impact Evaluation of the Adult and Dislocated Worker Program

New York State Department of Labor

1. Evaluate whether the proposed ICR is necessary for the proper performance of the functions of the agency, including whether the information will have practical utility.

It is questionable whether the information gathered through this proposed ICR will have any practical utility in evaluating the impact of intensive and training services funded by WIA adult and dislocated worker funds. The study appears to be more of a broad-brush attempt to try and understand how One-Stop services are structured and being delivered across the nation, rather than an evaluation of the impact of intensive and training services in the WIA adult and dislocated worker programs. Questions are asked about governance, number of One-Stop centers, partners on site, training mix, sequence of service etc. The information being gathered through the surveys is about the One-Stop service delivery system rather than the nature and quality of the services being provided to participants. With changes to the Workforce Investment Act likely to be authorized within the next year, it might be more cost effective and efficient to wait until WIA is reauthorized and develop a study to evaluate the system (difficult to isolate program) under its new guidelines which will likely result in changes to governance, sequence of service, expenditure tracking, etc.

Given the fact that WIA title 1 adult and dislocated worker services are delivered through the One-Stop System, any implementation analysis has to be conceived in broad terms to fully define the landscape in which the workforce system operates and the value that it generates. Specific concerns include:

• Capturing the value created for business through sectoral initiatives that make businesses more

competitive; value that doesn’t necessarily accrue back to jobseekers in the evaluation.

• Capturing the value created by enabling privately funded workforce initiatives to leverage

fundamental workforce infrastructure provided by the public workforce system.

• Reflecting the objectives, policies and approach that New York State sets for all local areas.

It’s important to recognize that the philosophy and service strategies that some states shape can have a big influence on the value created by the workforce system, as can statewide initiatives such as a focus on key industry sectors and regional coordination.

• Reflecting the challenges that the workforce system faces such as restrictive and burdensome

federal policies and the fact that the public workforce development system is trying to fulfill a mandate with far fewer resources than originally envisioned when the mandate was created.

• Reflect the fact that the New York State system has been designed to drive outcomes through

staff assisted core services and that a result showing that access to intensive services produces a limited impact may partially be the product of high quality core services.

RESPONSE

The study is designed to estimate the impact of intensive and training services funded under the WIA adult and dislocated worker programs. The experimental design involves random assignment of customers eligible for, and interested in receiving, intensive services. These customers are assigned to three groups: (1) the full-WIA group; (2) the core-and-intensive group; and (3) the core-only group. Customers in the full-WIA group can receive core, intensive, and training services. Customers in the core-and-intensive group can receive core and intensive services but not training. Customers in the core-only group can receive core services but neither intensive services nor training. Impacts of training will be estimated by comparing outcomes of customers in the full-WIA and the core-and-intensive groups. Impacts of intensive services will be estimated by comparing outcomes of customers in the core-and-intensive group with the outcomes of customers in the core-only group. The outcomes will be measured using administrative data and two follow-up telephone surveys. The surveys will be included in a subsequent addendum to this clearance package.

The study will collect information on implementation of services and local context, but is not designed as a “broad-brush attempt to understand how One-Stop services are structured and are being delivered.” The site visit protocols included in this package are designed to: (1) describe the implementation of WIA intensive and training services; (2) describe the local context in which they are provided; and (3) monitor the implementation of the evaluation. To the extent that intensive and training services are managed within the One-Stop system, the protocols do include questions about the system. However, many of the questions are designed to describe the nature of the services (see, for example, sections E and F of the adult/dislocated worker line staff protocol).

New York State suggests that USDOL should wait until WIA is reauthorized to evaluate the services. USDOL decided to go ahead with the evaluation at this time rather than wait for reauthorization for three reasons. First, WIA legislation required a study be conducted within the first five years of its implementation—so the study is already overdue. Second, as the timing of reauthorization is unknown, planning for the study could be delayed for many years. Third, if reauthorization takes place during the study, it may provide an opportunity to compare the impacts of the services prior to and after reauthorization.

Sectoral initiatives and the services provided to businesses and the value created by using privately-funded workforce initiatives to leverage the public workforce system are important parts of the workforce system. However, this study focuses only on intensive and training services; it is not within the scope of the study to describe or evaluate sectoral initiatives and services to businesses. Moreover, the implementation study will focus on the value created by using privately-funded workforce initiatives only to the extent that it affects the nature of intensive and training services offered.

We recognize that New York State affects the service strategies adopted by the LWIAs in New York. In the report, we will be sure to document the service strategies that have been adopted because of statewide initiatives.

The study will describe the federal policy context in which services are delivered, but it is beyond the scope of this study to determine whether the services would be more effective with different federal policies.

The purpose of the study is to assess whether intensive and training services are effective. We will be sure to note in the final report that core services may be very effective and the effectiveness of core services is not reflected in our findings.

3. Evaluate the accuracy of the agency’s estimate of the burden of the proposed ICR, including the validity of the methodology and assumptions used.

1. The burden of the ICR is greatly underestimated. The Federal Register estimates the burden to be a cost of 31,867 hours and $421,083 broken down as follows:

		Total Hours	Total Dollars
Participant Cost
Intake - 13 minutes per participant	68,000 participants	14,733	$106,817
Site Visit – one hour per participant (4 part per location per 2 visits = 4 x 30 x 2)	240 participants	240	1,740
Total Participant burden		14,973	$108,557
Staff Costs
Intake - 13 minutes per staff per participant	270 Staff x 252 participants each	14,733	$272,567
Site Visit – 1 hour per 4 staff per visit – 2 visits (4 staff x 2 visits x 30 locations)		2,160	39,960
Total Staff burden		16,893	$312,527
Total Burden Estimate		31,869	$421,084

The above calculation of burden is limited to the estimated time to complete initial intake forms for each participant and to conduct the 2 site visits; it fails to take into consideration the broader impact that this evaluation will have on overall and ongoing One Stop Center operations over the 18-month evaluation period including the additional burdens/costs shown below:

• Both the Baseline Information Form and the Contact Information Form ask for additional information which is not now housed in the existing case management systems. There will be additional burden if the state or local area needs to either maintain hard copies of any of the forms or modify current systems to be able to data enter/record additional data specific to the evaluation.

• While the amount of time per participant may be minimal (13 minutes), there will be additional costs associated with modifications to the existing customer flow design within the Centers to allow for this additional data collection/entry. There are time and space constraints that may already be at maximum capacity within the One Stop Centers. Factoring in an additional 15 minutes per participant may well have trickle-over effects on all customers – equating to delays in services such as cases where not all customers walking in the door can be accommodated immediately and have to be asked to return for service at later time or date.

• The burden of tracking the status of individual customers on an ongoing basis has not been taken into consideration in the estimates. After Intake, each repeat contact with a customer will require staff to first determine whether the customer is a participant or not, to which one of the three groups he/she is assigned, and to insure that the participant is limited to the appropriate services available to that group assignment. The limitation of services by group assignment will require each One Stop Center to add process steps to each service point in the customer flow to insure proper tracking and limiting of services. These process steps will be needed for every One Stop Customer at each service contact in order to determine who is/is not a participant. This may require participant and group assignment data to be added to the case management system. This goes well beyond the 68,000 estimated participants – but will impact services to all customers. This will also impact staff throughout the Centers such as greeters, counselors, etc…. who will need to make sure that customers are correctly routed and/or limited in which services.

• The burden does not take into consideration the costs that will be incurred by managers, supervisors, and staff throughout the One Stop system to design and implement the operational changes that will be necessary to accommodate project design requirements imposed by this evaluation methodology.

RESPONSE

Staff will only enter the responses to eight questions (questions 2, 3, 4, 5, A, C, D, and E) on the study registration form (formerly the baseline information form) into the study’s random assignment system. They need not enter these data into the site’s case management system. No information from the consent or contact information forms will be entered into either the random assignment system or the site’s case management system. The LWIAs will ship completed eligibility checklists and consent, study registration, and contact information forms to Mathematica routinely at the expense of the study. Neither the state nor LWIAs need to maintain copies of the forms.

Currently, staff need to know what services each returning customer has received, and are eligible for, before they are served. Similarly, during the study, staff will need to check whether a customer who comes in for repeat services is already in the study, and if they are, the group to which they are assigned. We will work with LWIAs to determine the best approach for staff to make this determination given their current procedures. One or more of the following procedures could be used to determine whether the customer is in the study and the group to which they are assigned: (1) checking on the random assignment system; (2) including a note in the customers hard-copy file; and/or (3) creating new codes on the site’s existing management information system. If new codes are created, the staff will need to enter three pieces of information into the system: (1) whether the customer was randomly assigned; (2) the group to which they are assigned; and (3) the date at which the service restriction will be lifted.

None of these procedures are expected to add more than 1 minute for every repeat customer. Assuming each customer comes back twice on average this adds 2,267 hours in total (68,000 x 2 / 60) or $16,436 (2,267 x $7.25). We have added that to the total burden.

This will affect only staff who determine eligibility for or provide intensive or training services. Greeters should not be affected by the study because they do not determine eligibility for, or provide, intensive or training services.

The procedures have been designed to place minimal burden on staff and customers. We will work with the local areas to minimize both operational changes and the likelihood that the additional steps required for the study will affect customer flow.

2. How are benefits measured? One of the questions that the evaluation seeks to answer is whether benefits from WIA services exceed program costs; however it is not clear how benefits will be measured. Benefits from participation can accrue broadly to two groups – the government and the individual. An individual’s benefits are realized in terms of higher wages, and lower probability of unemployment. Benefits realized by the government include increased tax revenue, and lower payments for unemployment insurance and public assistance. The full benefits from participation should be measured over an individual’s lifetime. Although such time horizon is unrealistic, benefits should be measured over a long enough time span after exit to provide a sufficient approximation of the true value from program participation. Furthermore, it is not enough to compare various measures of benefits before and after participation – the true measure of participation is relative to what would have occurred in the absence of participation. If an individual’s wages increase by $5/hr. after participation it is incorrect to attribute this increase fully to participation. The question which needs to be asked is what would this person’s wage be without participation? Properly identifying this counterfactual scenario provides the basis upon which true program benefits should be measured.¹

RESPONSE

We intend to measure the benefits to the customers, the government, and to society as a whole. We will measure the benefits that accrue as a result of increases in earnings (including fringe benefits and taxes) and reduced use of public assistance programs including changes in the use of unemployment compensation.

We propose to measure the impacts on customers for 30 months after random assignment. Benefits may last longer than 30 months. In the comparisons of benefits and costs, the potential underestimate of benefits will be noted. Administrative data for the period after 30 months could be collected but such collection is not currently anticipated.

It is correct that identifying benefits requires knowing the counterfactual scenario of what the persons’ outcomes would have been in the absence of participation. This is precisely the motivation for the random assignment study design. Benefits will be measured by comparing the outcomes of customers in each of the three research groups and not by comparing the outcomes of customers before and after the receipt of services.

2. How will program costs be measured? How will this evaluation be able to determine “whether the adult and dislocated worker services funded by Title I of the WIA are effective,” when services are funded under multiple funding streams in an integrated system? For example, New York State has fully integrated its Wagner-Peyser and WIA Adult and Dislocated Worker (DW) programs. Individuals are automatically co-enrolled in both Wagner-Peyser and the WIA Adult or DW programs. Wagner-Peyser funded staff work side-by-side with WIA-funded staff to provide core and intensive services to customers, including the authorization of ITAs for training. In addition, services are being funded with TAA, TGAA, NEG, REA, and other state funds.

Beyond this issue of not being able to segregate out which funding stream may have been used to fund services for participants in the One-Stop system, WIA Title I fiscal tracking only requires reporting of administrative and program funds. There is no activity based costing to determine specific service costs. The ICR references there will be special fiscal reporting required, but the forms have not been shared at this time. A future ICR will be submitted for the cost collection forms.

The Fiscal Staff questionnaire asks questions for the local area as a whole, but it is likely that the fiscal person only sees budget information for the WIA title 1 programs. Also, questions about staffing do not differentiate between WIB staff versus front-line service delivery staff. It is very doubtful that any conclusions about costs for WIA services will be able to be derived.

RESPONSE

We are interested in estimating the economic costs of the services received by study participants—the resources in the economy that are used in providing the services. Hence, the total economic cost is independent of the funding source. We will estimate the total cost of any additional training, irrespective of whether it is funded by WIA, a Pell grant, or other sources.

Our cost estimates will be derived from aggregating the component costs of the services. The cost of intensive services will be estimated from information about (1) the average time spent by counselors providing case-management and assessments and (2) the cost of their time (including wage rate, fringe benefits, supervisor’s costs, and overhead). The information on time spent in each service will be collected by asking the staff to keep a timesheet during a week. The information on fringe benefits, supervisor costs, and overhead will be collected using the cost protocols that will be submitted in the addendum to this clearance package.

The cost of training will include the full price charged by the training provider as well as the cost of any time taken by the counselors in monitoring the training. Data on the price of the training will be obtained from the local areas.

Further details of how we propose to collect this cost information will be provided in addendum to this clearance package.

2. What does randomization achieve? Randomization in experimental design ensures that individuals do not differ systematically between the control and treatment group which might lead to bias in the estimated effects. If participants are randomly assigned to a control and a treatment group, then differences in outcomes can be attributed to program participation. In the absence of randomization, the decision to participate in WIA funded training may be a function of unobserved characteristics which are correlated with outcomes. This leads to the problem of self-selection which results in biased estimates of program participation. For example, suppose individuals who choose WIA services are more highly motivated than those who do not. Then comparing outcomes between these two groups will overstate the true impact from program participation since this difference includes the true impact from participation and the effect attributable to motivation.

Randomization as structured for the WIA Gold Standard Evaluation ensures that participants in each group will not differ in any systematic way which might bias the results. But within groups 1 and 2 participants have choice over what services they receive. At this point then participants may sort among services offered based on unobservable characteristics (such as ability or motivation). Estimates of the impact from program participation will suffer from self-selectivity bias and will not provide a true measure of the effect from program participation.² So randomization does not achieve the desired property of sorting individuals by services in such a way as to ensure no systematic differences exist. The way that randomization is currently structured self-selection still exists, which will lead to biased estimates. Given that bias will exist in the estimates, it is not clear what randomization achieves in this setting.³

In addition to the self-selectivity bias which exists when participants are allowed to choose their services after they have been randomly assigned into groups there is also self-selection into the study which needs to be addressed. In order for the results from this study to be generalizable to the full population, it must be the case that individuals who participate do not differ systematically from those who choose not to participate. To illustrate this point suppose that individuals who choose not to participate are those who would benefit most from core services only and those who choose to participate are those who would benefit most from training.⁴ A likely conclusion then is that training provides greater benefits than core services. But this is a result of individuals choosing to participate based on their need for training. Self-selection occurs at the point of entry into the program, which needs to be addressed when estimating the impacts from participation.

More importantly, there is a cost associated with random assignment. Unless an individual is randomly assigned into the Full-WIA group their choice of services will be limited. But randomly assigning participants into these three groups does not remove the self-selection bias. Participants do not differ systematically when they are randomly assigned to the different groups, but self-selection bias emerges when participants are allowed to choose among their available services. Since non-experimental methods will be needed to address self-selectivity, it is not clear that random assignment provides any benefit.

RESPONSE

Random assignment ensures that the observable and unobservable characteristics of customers in the three research groups will be balanced at the point of random assignment. Thus, pairwise comparisons of the average outcomes of WIA-eligible customers assigned to the three research groups will provide unbiased estimates of the impact of the offer of services. These estimates are known as is intention-to-treat (ITT) estimates in the statistical literature. For example, to assess the combined effects of the offer of WIA intensive and training services, the study will compare employment-related outcomes for those assigned to the full-WIA group and those assigned to the core-only group. Similarly, to assess the effects of the offer of training services beyond intensive services, the study will compare the outcomes of those assigned to the full-WIA group and those assigned to the core-and-intensive group.

As suggested by the question, there are likely to be unobservable differences between customers who actually receive specific services and those who do not. For example, there might be unobservable differences between those in the full-WIA group who receive training services and those in the full-WIA group who do not. For this reason, we will not directly compare the outcomes of customers who receive different program services.

This issue that not all treatment group members will avail themselves of offered services is present in almost any randomized controlled trial in any field, and is in no way unique to the WIA evaluation. To create unbiased estimates of impacts for program participants, we will adjust the ITT impacts using statistical methods that have been developed in the research literature over the last 25 years (Angrist, Imbens, and Rubin 1996; Bloom 1984). These methods yield unbiased impact estimates for those who actually receive services under minimal assumptions that are described below. These participant impacts are known in the literature as treatment-on-treated (TOT) impacts. Importantly, TOT impacts are obtained from adjusting the ITT impacts, not from comparisons of outcomes of customers who receive different program services.

To explain how to calculate the TOT impacts from the ITT impacts, consider ITT impacts obtained by comparing the outcomes of the full-WIA and core-only groups. These ITT impacts represent the impacts of the offer of intensive and training services. Suppose next that there are two groups of customers in each of these two research groups: (1) participants who would actually receive WIA services if they were assigned to the full-WIA group; and (2) nonparticipants who, for whatever reason, would not actually receive intensive and training services if they were assigned to the full-WIA group. Suppose further that p is the percentage of customers who are participants. The ITT impacts can be expressed as a weighted average of the impacts for participants (the difference in outcomes between participants in the full-WIA and core-only groups) and the impacts for nonparticipants (the difference in outcomes between nonparticipants in the full-WIA and core-only groups), where the weights are p and (1-p), respectively. If we make the reasonable identifying assumption that impacts of the offer of services on nonparticipants are zero, the TOT impacts can be calculated by dividing the ITT impacts by p. Stated differently, the TOT impacts are obtained from the ITT impacts by dividing the ITT impacts by the participation rate, under the assumption that all program effects are due to those who actually receive services. The TOT impacts are fully based on the random assignment design.

This method has been used extensively in medical drug trials (to account for those who do not take the tested medication) and in a wide range of social policy evaluations similar to this evaluation. We will use these methods extensively in the analysis to net out the effects of program nonparticipants from the ITT impact estimates.

The findings will be generalizable to all those customers who are eligible for, and interested in, receiving intensive and/or training services. They will not be generalizable to all customers who use the One-Stop Career Centers or all customers who receive core services. Generalizing the findings to customers who are eligible for and interested in receiving the services answers the policy relevant question: what are the impacts of offering the services to those who are eligible and want the services?

We do not expect there to be significant self-selection into the study because of refusal to consent to the study. As described in the consent form, customers who do not participate can receive core services only. Hence, the customers have a strong incentive to participate in the study. In previous studies using this consent process, such as the National Job Corps Study, very few people did not consent to participate in the study.

Angrist, Joshua D., Guido W. Imbens, and Donald B. Rubin. 1996. “Identification of Causal Effects Using Instrumental Variables.” Journal of the American Statistical Association 91 (434): 444–455.

Bloom, Howard S. 1984. “Accounting for No-Shows in Experimental Evaluation designs.” Evaluation Review 8 (2): 225-246.

2. Will this evaluation interfere with Wagner-Peyser requirements to administer the work test to UI customers? It is unclear how this study is to be implemented in light of Wagner-Peyser/WIA adult or dislocated worker co-enrollment. One of the key functions of the Wagner-Peyser program is to facilitate the reemployment of UI customers (which directly affects achievement of the UI facilitate reemployment performance measure). In New York State, all UI customers except those classified as work search exempt (union hiring hall or on temporary layoff) are required to report to the One-Stop system for an initial assessment. Additionally, all Wagner-Peyser customers are automatically co-enrolled WIA Adult or Dislocated Worker customers. It should be clarified whether Wagner-Peyser funded employees are prohibited from providing intensive services to Wagner-Peyser participants if they are also enrolled as WIA participants. If this study is presuming that Wagner-Peyser funded intensive services may not be provided due to the “WIA adult and DW” study guidelines, then the definition and scope of this project needs to be redefined and clearly articulate that Wagner-Peyser funded services are also being restricted and evaluated. The statutory authorization for the redefined study should also be provided.

RESPONSE

Random assignment to a restricted service group only limits customers’ access to services funded through WIA. It does not limit access to services provided via other funding streams. If a given service is funded through a source other than WIA (such as Wagner-Peyser), customers randomly assigned to a restricted service group will still be able to receive those non-WIA funded services. It is understood that in some cases, customers may have services available to them through another funding stream that are comparable to those provided through WIA training or intensive services. The study will not interfere with Wagner-Peyser requirements to administer the work test to UI customers.

2. How will this random assignment evaluation be coordinated with USDOL’s REA random assignment evaluation? New York State is concerned about the ability to operate REA and the WIA Evaluation in a compatible manner. NYS currently has the largest state Reemployment Eligibility Assessment (REA) grant in the nation representing 15.7% of the available national dollars awarded. REA is operating in 23 of the 33 local workforce areas of the state including the 3 local areas selected for the gold standard study. We are very concerned about how the proposed gold standard control group would impact the REA program which also requires a random sampling control group methodology and we need to insure that the gold standard design in no way negatively impacts to outcomes of the NYS REA program by altering the services provided to either the REA participants and/or the REA control group.

We note that the REA model in NYS is very different than our understanding of the REA design in many other states. We understand that in some states REA is limited to a one time REA interview session with the customer and that without the REA program there would otherwise be no one- to-one individual assessment interview with individual UI claimants in the One Stop.

In NYS, the REA program design is layered on top of a pre-existing state funded Reemployment Services program and general One Stop policies that already emphasize individual assessment of One Stop customers and delivery of subsequent services based on each individual’s assessed needs. REA program design was developed as a model “best practice” in the sense that it emphasizes an intensive, one on one level of continued service to UI customers – active, continued engagement with REA interviews scheduled every 2 weeks initially for at least the first 3 sessions but then continuing every 2 to 4 weeks for as long as the claimant remains unemployed. The first REA interview which is typically the initial service contact with the customer in the One Stop is considered an intensive service for all REA participants in NYS. The initial REA interview is scheduled to occur within 2 weeks of UI claim migration and is a one on one session lasting on average one hour per customer. So REA participants would all have to be excluded from the gold standard initiative.

While NYS has implemented a random sampling control group methodology for the REA program, we agreed to that only because the REA control group is not denied access to any services available through the One Stop system – individuals in the control group still have the same full access to any/all core, intensive, and training services offered through the One Stop system.

REA is funded from UI Administrative funds and as a result is limited in terms of the activities it can fund. REA funds are not intended to provide the types of employment and training services that the One Stops offer – in fact USDOL guidance prohibits using REA funds for that purpose. REA programs are required to be delivered within an existing One Stop location and REA is intended to “leverage” existing available One Stop Services. One Stops are required to agree to use their One Stop resources to provide services needed by REA participants. REA staff is limited to helping the customer develop an individual work search plan, to monitor compliance with the plan, to advise the customer on UI work search requirements and report potential issues to UI, to continually case manage and assess the customers’ needs and to refer the customer to available One Stop services. While REA control group claimants are not scheduled for and/or mandated to attend one on one REA interviews; they do still receive an individual initial assessment interview conducted by One Stop staff and have access to any services of the One Stop.

RESPONSE

We are aware of the implementation of REA in some of our sites. As REA is implemented differently in each site and is being evaluated by different approaches, we will treat REA participants on a site-by-site basis. In most sites, the provision of REA services will not affect the study as the services amount only to staff-assisted core services.

We recognize that in New York State, REA participants are required to have access to intensive services as part of the program. Because REA participants cannot be denied intensive services, we propose to exclude them from the study.

Customers who are eligible for REA services but are in the REA control group will be randomly assigned for the WIA Evaluation. However, those placed in the core-and-intensive or core-only groups should be removed from the control group for the REA study. We can provide a list of these customers to New York State. As the percentage of customers assigned to the restricted services groups is low, we expect few REA control group participants to be assigned to the core-only or core-and-intensive groups.

2. How will individuals eligible under special targeted programs be handled under this evaluation methodology? NYS is very concerned about the how the design of the gold standard evaluation insures that individuals eligible under other special targeted programs are not denied those services. Individuals eligible for services under such programs as TAA, TGAA, NEG, or REA (which is discussed above) should not be denied access to intensive/training services. We understand that the gold standard evaluation design will try to minimize the impact by keeping numbers of individuals in the control group as small as possible; however, NY does not believe that we could legally deny service to even one individual otherwise eligible for these targeted program services.

RESPONSE

We recognize the need to exempt certain categories of customers from the evaluation that are statutorily entitled to receive WIA services that are subject to restriction under the study design. Some customers, such as TAA and TGAA recipients, will be exempt across all sites. Because of state-specific laws or programs, certain other categories of individuals may be exempt in specific states. In their visits to sites, study staff members have been identifying groups of individuals cannot legally be denied services so that intake staff can be instructed to exempt all such customers from the study.

As NEGs are not locally-funded, customers in the study will not be denied access to NEGs.

As discussed in response to the previous question, members of the REA treatment group in New York State will be exempted from the experiment.

2. How will this evaluation maintain compliance with Veterans’ priority of service requirements? Veteran priority of service and service to other priority populations most in need are also significant concerns of our state. We are not convinced that the value of the proposed gold standard evaluation warrants denying service to veterans and/or other special populations.

RESPONSE

Veterans and “covered” spouses of veterans (as defined by WIA) will be exempt from the study.

2. Establishment of limited services groups is not ethical, particularly when economic conditions leave people with limited labor market opportunity. First, customers who opt out of the study can only receive core services. They will not be allowed intensive or training services. Further, customers who agree to participate will be randomly assigned to one of three research groups: (1) customers can receive any services for which they are eligible, (2) customers can receive core and intensive services, and (3) customers can receive core services only. Determination of a customer’s level of service based on the group to which they are randomly-assigned or based on not participating in the evaluation, on its face, seems to run contrary to WIA. If a customer is eligible for a level of service, they should be entitled to that level regardless of their participation in an evaluation.

Often customers do not recognize their need for intensive/training services at first visit. Especially in the case of training, customers at intake do not understand the labor market well enough to comprehend the skill gaps they have. New York is concerned that customers will initially agree to limit their services to core only, but will later (after they are unable to find a job and after long-term unemployment) return for intensive and training services.

RESPONSE

We are certainly aware that denying services to customers is difficult. To help minimize this difficulty, we will assign only a small number of customers to the restricted service groups. In New York City, only 136 customers will be assigned to the core-only group and 136 customers will be assigned to the core-and-intensive services group across all New York City workforce centers during the 18-month intake period. Furthermore, all customers in the sample will be eligible for core services and other employment-related services that are available in the community.

It is important to recognize that as currently funded, the WIA progam cannot provide intensive and training services to all customers who may benefit from them; thus, in nearly all local workforce investment areas around the country, WIA services are currently being denied to some eligible customers. For the evaluation, we are requesting that a very small percentage of customers be randomly denied WIA services rather than, for example, because training dollars run out. Importantly, all the sites can serve the same number of customers as they would without the study.

Finally, the purpose of the study is to provide rigorous evidence of the effects of WIA intensive and training services, which a large body of research has shown can only be accomplished using a random assignment design with restricted service groups. Positive study results will provide the strongest possible evidence of program effectiveness that could influence the WIA re-authorization process. This process could lead to many more customers receiving program services in the future. A less rigorous nonexperimental study without restricted service groups would be much less credible and would have much less influence than a well-executed random assignment study.

2. How will comparative economic factors be examined and controlled? The current labor market is

so weak that it may not be possible to detect a competitive advantage gained by program participants through an analysis of employment outcomes. In a labor market where there are ten jobseekers for every job opening, even candidates with a competitive advantage can’t find jobs because there aren’t any jobs.

RESPONSE

The research literature provides mixed evidence about the relationship between labor market conditions and program impacts. Some evidence suggests that impacts for reemployment programs are likely to smaller under a strong economy, because jobs are so readily available that reemployment services might not make a significant difference. However, other evidence suggests the opposite, namely that impacts may be smaller in economic downturns when jobs are scarce. Thus, there is considerable uncertainty about whether WIA impacts are likely to be smaller under a strong or weak labor market.

The study enrollment period will not start until after the start of the next program year in July 2011, and it is difficult to anticipate the state of the economy at that time and the ensuing 18-month sample intake period. Because of the mixed literature on the association between labor market conditions and program impacts and the uncertainty about economic conditions during the 18 month intake period, the expert advisory panel for the evaluation recommended that the study move forward.

The study sample will contain 30 randomly-selected LWIAs that are geographically dispersed. Thus, there will be a considerable range of labor market conditions faced by customers in the sample. In the impact analysis, we will estimate impacts for subgroups defined by the customers’ local labor market conditions and examine the variation in these impacts. We will also examine how impacts change over time as the economy changes.

4. Enhance the quality, utility, and clarity of the ICR.

The quality, utility and clarity of the ICR would be enhanced by aligning the stated purpose and scope of the study to be consistent with the data collection instruments that are proposed for use. For example, the consent form is headed, “Consent to Participate in the Workforce Information Act Evaluation of the Adult and Dislocated Worker Programs,” and the opening sentence of the paragraph following the heading states, “The U. S. Department of Labor is sponsoring a study about the nation’s employment and training programs to learn how well these programs are working and how they can be improved.” If the evaluators are being asked to evaluate the One-Stop system, then information needs to be collected across the system and reflect all the inputs and outputs of the system. The study is not structured to do this.

Consent Form: The description provided for Group 3 is inadequate and very misleading. Core services include much more than “services in the Resource Room such as job listings and access to the Internet.” Customers may receive all types of staff-assisted services including, but not limited to initial assessment, career guidance, job search planning, and workshops on topics such as interviewing skills, résumé development, networking, filling out online applications, using the Internet to find job opportunities, etc. Depending on the type of data collected from State agencies, the consent form may require revisions to come into compliance with 20 CFR 603.

WIA Baseline Information Form: This form duplicates intake information that is collected for all One-Stop customers and asks for information for which there is no “home” in New York’s case management system (OSOS). Examples of orphan information are: maiden name, under whose name is that phone listed, write in your race, what is your marital status, how many people live with you, education question structure/categories (don’t align), disability question not the same, job in past five years, how many hours work at main job, what was current or most recent rate of pay, and have you used services in O-S before. Having this number of questions differing from what is normally collected and data entered into the state’s case management system implies there will be duplicate data entry required (and possibly duplicate data collection). As noted earlier in our comments, the ICR burden projections do not account for duplicate data collection and duplicate data entry.

Contact Information Form: Although most local areas request a contact number and an alternative number, it does not align with this data collection form which requests multiple contacts. This too, implies duplicate data collection and data entry.

WIA Administrator: Who is to be defined as the WIA Administrator in a local area? There are chief-elected officials, WIB Chairs, WIB Executive staff, One-Stop Center Managers, supervisory staff overseeing WIA-funded staff – which entity is considered the “WIA Administrator” in a local area for the purpose of this study? Under C: Nature of Self-Service Core Services – what information is being sought with the question, “Describe any state/LWIB guidance on how partners should deliver self-service”?

Finance Staff Protocol: It isn’t clear whether the evaluators are asking about costs of the One-Stop system or how WIA title 1B adult and dislocated worker funds are being spent in the local workforce investment area (which is not going to be able to track back to the services provided in an integrated system since core and intensive services are provided by multiple partners, and other partners pay rent, provide supplies, etc.).

ES Manager Protocol: Why all the detailed questions about UI customers and ES processes? If this study is about WIA adult and DW and ES services, then the evaluation should clearly articulate this and the methodology should be modified accordingly. Individuals who access self-services count as participants in both WIA title 1 and Wagner-Peyser. As such, we don’t understand the questions “Which visitors to the One-Stop Center become registered as ES participants? When are they registered into ES? What services are they likely to receive before registering in ES (refer to TEGL 17-05)? Why are questions being asked about job matching tools?

RESPONSE

Consent Form. We have changed the wording of the first sentence on the consent form to “a study of some of its employment and training programs that serve adults and disclosed workers” rather than “a study of the nation’s employment and training programs.” This change is to avoid the perception that the study involves evaluating all the employment and training programs and the entire One-Stop system.

The consent form is designed to be administered across all LWIAs and One-Stop Career Centers in the study. While we acknowledge that New York State includes a wide variety of services as core services, many LWIAs in the study do not include all these staff-assisted services as core. Based on recent visits to study LWIAs, use of the resource room and Internet searches are two examples of core services available in all the One-Stop Career Centers in our study. We have changed the description of core-only services to reflect that customers will not have access to WIA services “that require substantial staff time.” We also will work with each site to develop a list of services that are core, intensive, and training in their particular site, which will be given to customers.

We believe that the consent form is in compliance with 20 CFR 603.

WIA Baseline Information Form. We acknowledge that some of the information collected on this form is currently collected in New York’s case management system (OSOS). It is important that we collect the same data across all sites. Hence, unless the case management system can provide responses to exactly the same questions, staff need to administer the study registration form (formerly the baseline information form). (LWIAs that do collect information on the responses to the exact questions using their case management system can provide the evaluator these responses electronically.) The information collected on the study registration form is minimal and is essential for random assignment, to locate the sample member for a telephone survey, and for the subgroup and survey nonresponse analysis.

The site staff will not enter all this information into a database. The only data that staff will enter into the random assignment system are the answers to questions 2, 3, 4, and 5, and the “counselor use only” questions A, C, D, and E. Those items will be entered into the study’s random assignment system, not the OSOS. The contractor will scan the responses to the remainder of the questions from the hard-copy form.

Contact Information Form. The customer will be asked for detailed contact information. This information is essential for locating sample members and achieving a high response rate to the follow-up surveys. However, One-Stop Career Center staff will not be asked to enter any of the information collected by the contact information form into a database. The contractor will scan the information recorded on the form.

WIA Administrator. The research team envisions interviewing several types of individuals under the broad category of WIA Administrator, including the Executive Director, One-Stop Career Center Managers, and the WIA Program Manager. The questions in this Field Protocol (and other protocols we have developed for this study) are not intended to be read verbatim. Instead, the protocols constitute loosely structured scripts, with the precise sequence of questioning and question wording contingent on exactly who the respondent is and the topics about which they are most knowledgeable. Moreover, we also intend to be flexible in using the protocols, so that we can follow unexpected threads of conversation to their logical conclusion. In keeping with these principles, the questions in the WIA Administrator Protocol may be tweaked by the field researcher while on site depending on the responsibilities of the individual who is being interviewed.

With regard to the question on state/LWIB guidance on self-services, we are interested in learning what formal guidance has been provided to the LWIA staff on how to orient customers to self-services, what level of staff assistance can be made available, how Resource Room staff should identify whether customers need additional (e.g., staff-assisted core or intensive) services, and how partners should work together in providing self-services.

Finance Staff Protocol. For the study’s benefit-cost analysis, we want to estimate the costs of the services received by those in the three research groups—the core-only group, the core- and-intensive group, and the full-WIA group—regardless of whether the services are provided by WIA, ES, or other One-Stop Career Center partners, and regardless of the category of costs (e.g., rent, the costs of staff time, direct payments to participants). An addendum to this package will provide details of how we propose to collect cost information. The finance staff protocol is designed to collect information on the financial information the LWIA do collect and will help us tailor the cost protocols to each site.

ES Manager Protocol. The evaluation team is interested in estimating the impact of WIA-funded intensive and training services in comparison to core services. To sensibly interpret these impacts, we need to have a full understanding of the nature of the services that customers receive in all three groups: the core-only group, the core-and-intensive group, and the full-WIA group. Since, in many LWIAs, ES will provide much of the One-Stop Career Center core services, we will need to gather information about the nature and content of ES services so that we can properly understand the services offered to those in the core-only group.

Under its functional alignment plan, we understand that New York LWIAs enroll all self-service customers in both WIA and Wagner-Peyser. However, based on our extensive data collection at LWIAs from prior projects, we know that not all LWIAs follow this practice. Moreover, we have found quite a lot of variability in the way LWIAs interpret the provisions of TEGL 17-05.

5. Minimize the burden of the ICR on those who are to respond, including the use of appropriate automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submissions of responses.

Access to Wage Record Data. Data will be requested on earnings and UI benefits; some sort of data-sharing agreement will need to be executed and there will be costs associated with the disclosure. At the Wage Record Interchange System (WRIS) Advisory Group meeting on December 9, 2010, it was announced that the workgroup conducting the evaluation is requesting access to wage record data through WRIS as part of the evaluation as outlined in TEN 37-09, issued April 6, 2010. They are hoping to be able to negotiate one Memorandum of Understanding (MOU) with WRIS instead of MOUs with each of the 20 participating states and 9 contiguous states. The MOU would be similar to the one used in the Trade Adjustment Assistance study conducted by Social Policy Research Associates. They are aware that some modifications may be required to be in compliance with state laws. The target date for having agreements in place was specified as “no later than September 2011”.

Some of the WRIS Advisory Group members were unaware of the WIA study and voiced their concerns that they could not present the issue to their management without more details. Some states reported political changes will be occurring within their State management in the next few weeks that could impact the review and approval process for an MOU.

Duplicate data collection and entry. As noted above, there are concerns that duplicate information will need to be collected from customers and duplicate data will be need to be entered by staff with the forms proposed for use under this ICR.

RESPONSE

The evaluation team is developing additional materials to share with the WRIS Advisory Group and team members are available to make presentations before this group at any time.

The study team is proposing that only limited information will need to be entered by LWIA staff into the study’s random assignment system (8 items per participant). To minimize burden on LWIA staff, other information collected from participants will be entered by the participants on hard-copy forms and the forms submitted to the study team for data entry.

Works Cited

Angrist, Joshua D, Guido W. Imbens, & Donald B. Rubin. “Identification of Causal Effects Using Instrumental Variables.” Journal of the American Statistical Association, vol 91, June 1996, pp. 444-445.

Bloom, Howard S. “Accounting for No-Shows in Experimental Evaluation Designs.” Evaluation Review, vol. 8, April 1984, pp. 225-246.

Bloom, Howard, Larry Orr, Stephen Bell, George Cave, Fred Doolittle, Winston Linn and Johannes Bos.

1997. “The Benefits and Costs of JTPA Title II-A Programs: Key Findings from the National Job Training Partnership Act Study.” The Journal of Human Resources, 32(3) pp. 594-576

Doolittle, Fred and Linda Traeger. Implementing the National JTPA Study. New York: Manpower

Demonstration Research Corporation, April 1990.

Minnesota Department of Employment and Economic Development

1. As we have stated in the past regarding similar, related projects (Project GATE II), we believe that random denial of services is unethical, especially on the heels of a recession.

RESPONSE

The workforce system cannot serve all people who are eligible for, and could benefit from, intensive and training services, Indeed, LWIAs frequently run out of training funds during the year. The recession only heightens the demand for these services without a concomitant increase in the supply of services. During the study, the same amount of intensive and training services will be provided. The main difference is that during the study the people who receive services will be determined randomly rather than by other circumstances such as the timing of when they come into the One-Stop Career Centers for services.

2. Workforce Services Areas should retain the ability to determine enrollment themselves, serving as the basis for local control, which underpins our entire system.

RESPONSE

During the study, workforce service areas can continue to identify who is eligible for and would benefit from services. The study will not affect this process. The only difference is that during the study a small proportion (between 1 and 16 percent) of these customers will be assigned to a group in which services are restricted.

3. Minnesota is one of nine states participating in DOL’s Regression Model Pilot Project. Minnesota has demonstrated commitment to evaluative research. This Random Assignment Evaluation could compromise the success of the Regression Model Pilot in Minnesota.

RESPONSE

We will be working with each state to ensure that this evaluation does not interfere with other important studies such as the Regression Model Pilot Project. In the case of Minnesota, although LWIAs in the state were included in the sample frame, no LWIA in Minnesota was randomly selected for this evaluation.

4. Participating in this study will skew Minnesota’s performance standards for those providers who participate in such a study. Regardless of any available performance waiver, local providers may opt to send customers to other service providers who were not randomly selected to participate in the study. These actions will provide the customers with the full array of services for which s/he is eligible, which also allows the service provider to maintain control over its own program performance.

RESPONSE

Because the percentage of all customers in a state who are assigned to a restricted service group will be very small, providers’ performance measures are unlikely to be affected.

5. If Minnesota’s service providers do intervene by sending individuals to other providers, the evaluation will be biased. Further these actions will have lasting impacts on the reputations of our providers throughout the state. From a customer’s viewpoint: Why could I receive services from that Workforce Center, but not the other?

RESPONSE

All workforce centers in a LWIA will participate in the study. Hence, if customers move from one center to another, they will remain in the group to which they were assigned. Furthermore, we are asking states in our study to indicate in their management information system if a customer is in the study and to ask LWIAs neighboring our study LWIAs to maintain any service restrictions placed on the customers in the study.

6. In light of the recent release of the Workforce Investment Act Non-Experimental Net Impact Evaluation completed by IMPAQ International LLC in December of 2008, and published by US DOL over this past program year, we understand that other non-experiment methods are achievable in evaluating such programs. Evaluators studied administrative data covering approximately 160,000 WIA customers receiving services from July 2003-June 2005 in 12 different states, comparing their depth of service (core only, core and intensive, and so on). While the group was “not a representative sample, there are several reasons why these results are likely to apply to the WIA program more generally” (Executive Summary).

RESPONSE

DOL’s nonexperimental study of WIA services has two inherent drawbacks. The first, and most important, is that it relies on comparing groups of customers who receive services with customers with similar observable characteristics who do not receive services. However, these groups of customers could have very different unobservable characteristics that could explain differences in outcomes. For example, the study compares the outcomes of people who receive training services to those who receive intensive services only. These groups of customers differ however, in that customers in one group chose to apply for training and customers in the other group either were ineligible for training or did not choose to apply for training. One reason they may have decided not to apply for training is lower motivation to develop new skills. If more motivated customers choose to participate in training, the estimated impact of training will be biased upwards. Random assignment ensures that both the observable and unobservable characteristics of customers in each group are the same on average except for the receipt of services.

The second drawback of the study is that the states in the study were not randomly selected and hence may not be generalizable to the entire nation. While the report suggests that they may be broadly representative, the states that chose to participate in the study may be different in the effectiveness of their programs than states that chose not to participate in the study.

7. Minnesota’s Governor’s Workforce Development Council has nearly completed a detailed Return on Investment model to evaluate publicly funded workforce development and training programs. Participating in this evaluation study will jeopardize this, among other, evaluation models.

RESPONSE

No LWIA in Minnesota was randomly selected for this evaluation.

Michael J. Williams, Executive Director, Rock River Training Corporation

1. Without giving participants the option of withdrawing at any time from the study group, I am very concerned that the design of this study could result in the denial of Core and Intensive and/or Training services to individuals who otherwise would lawfully be entitled, and who could/would benefit from the very services they have voluntarily and possibly unwittingly waived for the purpose of participating in this study. The study design seems to violate the basic tenet of human research—do no harm to the study participant.

• RESPONSE

At any time, study participants can opt to withdraw from the study. Customers who wish to withdraw from the study can write to Mathematica and ask to be withdrawn. The address will be provided to all customers. Staff will be trained on telling the customers about the study and asking for consent, as well as informing customers about how they can withdraw from the study.

Customers are unlikely to unwittingly participate in the study. The consent form lays out explicitly what the study involves. The program staff will be trained in explaining the study to the customer.

The workforce system cannot serve all people who are eligible for, and could benefit from, intensive and training services, Indeed, LWIAs frequently run out of training funds during the year. The recession only heightens the demand for these services without a concomitant increase in the supply of services. During the study, the same amount of intensive and training services will be provided. The main difference is that during the study the people who receive services will be determined randomly rather than by other circumstances such as the timing of when they come into the One-Stop Career Centers for services.

2. It would seem that the same information sought could be obtained from a study of existing data on the same target populations that are currently or have been enrolled for an 18-month period in WIA Adult and Dislocated Worker programs.

RESPONSE

Random assignment allows us to compare a group of people who are offered services with a group of people who are similar in every way except for the fact that one group is offered the services and the other is not. This allows us to isolate the effect of the services. With existing data, we would need to compare groups of customers who receive services with customers with similar observable characteristics who do not receive services. However, these groups of customers could have very different unobservable characteristics that could explain differences in outcomes. For example, the study compares the outcomes of people who receive training services to those who receive intensive services only. These groups of customers differ however, in that one group chose to apply for training and the other either was ineligible for training or did not choose to apply for training. One reason they may have decided not to apply for training is lower motivation to develop new skills. If more motivated customers choose to participate in training, the estimated impact of training will be biased upwards. Random assignment ensures that both the observable and unobservable characteristics of customers in each group are the same on average except for the receipt of services.

Comment by VETS on the Information Collection Request for the WIA Evaluation

VETS appreciates this opportunity to comment on the Information Collection Request (ICR) for the WIA Random Assignment Evaluation. VETS especially appreciates the in-person briefing provided by the Contracting Officer’s Technical Representative with ETA’s Office of Policy, Development and Research.

VETS believes that there are several key principles to be considered regarding the relationship between this evaluation and the delivery of services to veterans, on the one hand, and the opportunity to expand the body of knowledge about the effectiveness of services to veterans, on the other hand.

• It is VETS’ view, based on 38 USC 4215, that Congress did not intend for veterans to be denied services in a random assignment evaluation.

• It also is VETS’ view, based on 29 USC 2917(c), that Congress also did not intend for services to veterans to be excluded from the learning opportunity established by this statutory mandate.

• GAO expressed its support for including services to veterans in the WIA Evaluation in Report GAO-07-594.

• In light of these principles, VETS believes that the WIA Evaluation should be expanded to include supplementary surveys of those veterans who receive intensive services, as well as those veterans who receive training. VETS believes that: a) the experimental control group could serve as a comparison group for the two groups of served veterans; and, b) the impact estimates for veterans derived from this analysis could be compared with the experimental impact estimates for the two corresponding groups of non-veterans.

• VETS recognizes the cost implications of supplementing the existing experimental design and is willing to partner with ETA in seeking sources of funding for the additional effort required.

Please direct any further communication about this matter to Gordon Burke, Director of VETS’ Office of Operations, Grants and Transition Programs, Room S-1325, 3-4707, [email protected]

RESPONSE

As many sites in the study interpreted the veteran priority of service provisions (38 USC 4215) as implying that veterans could not be denied services, veterans and covered spouses (hereafter referred to as veterans) are excluded from the WIA Adult and Dislocated Worker Programs Gold Standard Evaluation.

Conducting a survey of veterans and comparing the outcomes of veterans with members of the core-only group and core-and-intensive group could be very misleading. This is because veterans are fundamentally different than nonveterans on many dimensions, some of which cannot be observed. For example, suppose veterans face more barriers than the average non-veteran. Then, the outcomes of a veteran who received intensive and training services may be worse than those of a nonveteran who received neither type of service. It would be wrong to interpret this finding as showing that intensive and training services had a negative impact; instead it reflects differences in the underlying characteristics of veterans and nonveterans. No rigorous estimates of the impacts of services on veterans can be obtained without randomly assigning veterans.

We do, however, look forward to discussing with VETS other possible additions to the study. Clearance for any additions will be requested in a separate OMB package.

1 For more information on measuring benefits from labor market training programs see Bloom et. al. (1997)

2 We do not know a priori whether the bias from self-selectivity will be positive or negative.

3 For an example of randomization in a similar study see Doolittle and Traeger (1990)

4 This hypothetical scenario is not only theoretically possible but a likely problem to be encountered in application. If participants know that they will only need core services then they have no incentive to enter into the study. Thus, it is likely that those who enter into the study are those who need more intensive services.

4