Responses to Public Comments

APPENDIX H
PUBLIC COMMENTS

Comment Request for Information Collection for the
Workforce Investment Act (WIA) Adult and Dislocated Worker Programs
Gold Standard Evaluation
Response to Comments

The U.S. Department of Labor, Employment and Training Administration (ETA) published
a notice of new data collection for the Workforce Investment Act (WIA) Adult and Dislocated
Worker Programs Gold Standard Evaluation in the Federal Register on June 25, 2012 (Federal
Register, Volume 77, Number 122, pages 37923-37926). This document responds to the two
comments received—one from a private citizen and another from a local site participating in the
WIA Evaluation—during the comment period, June 25, 2012 through August 24, 2012.

Comment: The private citizen questioned the value of the evaluation, especially during poor
economic times.
Response: It is important that we conduct this study. In the 14 years since the WIA was enacted,
its service components have never been rigorously evaluated. We need to learn if WIA-funded
intensive services and training are as effective as they can be. This is especially important now
when so many people are out of work and seeking services to help them transition to new
employment opportunities.

A participating local site expressed full support for the research addressing the effectiveness of
WIA services, but submitted comments about the new data collection included in the Federal
Register notice. The comments and the study team responses follow:
Comment: The commenter expressed concerns for the estimated length of the survey—40 and
30 minutes for the 15- and 30-month follow-up surveys, respectively—and the burden on
customers. The commenter also indicated that this length could be longer when the survey is
administered to customers who are Limited English Proficient (LEP).
Response: Mathematica has successfully administered surveys of this length with similar
populations for studies conducted for the U.S. Department of Labor as well as for other agencies.
However, we recognize that such surveys introduce burden and address this in several ways.
First, customers who complete the survey will be paid $25 as a token of appreciation and
acknowledgement of their time. Second, appointments to conduct interviews will be scheduled at
the convenience of each customer. Furthermore, customers can complete the interview in more
than one call if they prefer to do so. (We have found that most respondents complete surveys of
this length in one call.) The survey has been comprehensively pretested with local WIA
customers, and they were able to complete the survey without problem. Finally, most customers
will not be required to answer every question in the survey. The questionnaire is designed to skip
questions (and even entire sections) if it does not apply to a particular customer’s situation.
For customers who are limited English proficient (LEP), we do not expect the length of
administration to be doubled. First, to address the needs of Spanish speakers, the survey will be
developed and administered in Spanish as a standard practice. Thus, the administration of the

survey to Spanish speakers should be similar to the administration for English speakers. Second,
for people who speak neither English nor Spanish, the same option of completing the survey in
multiple calls is available. Mathematica staff will be used as interpreters when possible, and we
will employ interpreter services for languages for which staff are not available and the demand
warrants it.
Comment: The commenter suggested that the study find avenues other than the surveys to
collect data on the services participants receive. For service use data, the commenter notes that
quarterly extract data is already planned from WIA databases.
Response: The survey is a critical source of information that we cannot obtain from
administrative data. Most importantly, without the survey data, we would not obtain information
about the employment and training services that customers receive outside of the WIA program.
The study needs to capture this information for all customers in the evaluation in order to
estimate the effects of WIA intensive and training services on customers.
As indicated by the commenter, the study team is collecting local areas’ WIA administrative
data. We intend to use these data to confirm that customers have maintained their study group
assignment and to describe WIA services received by customers. However, often, local areas’
administrative databases do not provide the detail about services required for the evaluation. For
example, the administrative data might inform us that the customer has received intensive
services, but not whether the service received was an intensive workshop or a counseling session.
Through the survey, we will be able to collect this detailed information.
To be sensitive to the burden on customers, we do intend to rely on administrative data when
available. For example, the study team will be collecting unemployment insurance information
from state databases, so we will not ask customers about their receipt of unemployment
insurance.
Comment: The commenter recommends that the team consider using a Web-based survey
methodology for the follow-up surveys as well as electronic submission of responses in the other
two proposed data collection efforts—the cost collection and Veterans’ Supplemental Study
(VSS). For the survey, the commenter notes that electronic-methodologies may increase the
efficiency of the survey data consolidation process.
Response: Based on our experience, we believe that computer-assisted telephone interviewing
(CATI) is the best option for completing the follow-up surveys. Many customers will not have
Internet access at home and will therefore need to use public facilities such as libraries or WIA
offices to complete a web-based survey. Also, it is more likely that a respondent will start a webbased survey under these circumstances and not complete it than is the case with a telephone
interviewer. In addition, Mathematica’s survey staff will offer customers flexibility when
scheduling the CATI interview to meet the needs of each customer, and trained interviewers will
be able to clarify questions if needed, improving the validity of responses and increased
responses overall. The CATI administration provides the same benefits of consolidating the data
as we would achieve with a web-based survey. CATI interviewers enter respondents’
information during the interview.

4

The cost data collection package provides a format for describing the types of data needed for the
study’s analysis of the effectiveness of WIA services. We will encourage local areas to submit
these data electronically if they are able to do so.
The VSS qualitative component does not lend itself to electronic submission of responses. In our
experience, in-person meetings with staff and customers are invaluable to learning about how
services are delivered. Both perspectives are important, and face-to-face discussions often elicit
responses that one cannot obtain through other methods.
Comment: The commenter stated that the planning for the WIA Evaluation site visits takes a
minimum of 40 hours and suggested that the burden estimate for the VSS on-site data collection
include this staff time.
Response: We agree that planning the site visits requires staff time. Thus, we have added 4
hours per site for staff to coordinate the VSS portion of the second round visits. The full burden
for the coordination of the WIA Evaluation site visits, which were cleared under a previous data
collection package, is outside this clearance request.

5