Justification 2007 (2)

SUPPORTING STATEMENT FOR REQUEST FOR OMB APPROVAL
UNDER THE PAPERWORK REDUCTION ACT
PART A – JUSTIFICATION

This is a justification for the Department of Labor’s request for approval to extend and revise a
currently approved data validation requirement for six Employment and Training Administration
(ETA) programs. Data validation assesses the accuracy of data collected and reported to ETA on
program activities and outcomes. The accuracy and reliability of program reports submitted by
states and grantees using federal funds are fundamental elements of good public administration,
and are necessary tools for maintaining and demonstrating system integrity. The data validation
requirement for employment and training programs strengthens the workforce system by
ensuring that accurate and reliable information on program activities and outcomes is available.
The following programs are subject to the data validation requirement: Workforce Investment
Act (WIA) Title IB, Wagner-Peyser Act, Trade Adjustment Assistance (TAA), National
Farmworker Jobs Program (NFJP), Indian and Native American Program, and Senior
Community Service Employment (SCSEP).
1. Reasons for Data Collection
States and grantees receiving funding under WIA Title IB, Wagner-Peyser Act, TAA, and the
Older Americans Act are required to maintain and report accurate program and financial
information (WIA section 185 (29 U.S.C. 2935) and WIA Regulations 20 CFR 667.300(e)(2);
Wagner-Peyser Act section 10 (29 U.S.C. 49i), Older Americans Act section 503(f)(3) and (4)
(42 U.S.C. 3056a(f)(3) and (4)), and TAA Regulations 20 CFR 617.57). The text of these
citations can be found in Appendix A. Further, all states and grantees receiving funding from
ETA and the Veterans’ Employment and Training Service are required to submit reports or
participant records and attest to the accuracy of these reports and records.
In 2001, the President announced a Management Agenda to improve the management and
performance of the Federal government. One of the five government-wide goals – budget and
performance integration – emphasizes the importance of complete information for program
monitoring and improving program results.
The Department’s Office of Inspector General (OIG) conducted an audit of WIA performance
data oversight from July 2000 through October 2001. The audit, released in September 2002,
found that, “Because of insufficient local, state, and Federal oversight, the Employment and
Training Administration (ETA) has little assurance that the state-reported WIA performance
outcomes data are either accurate or verifiable.” The OIG recommended that states should
validate reported data using a statistically valid sampling method. To address the concerns
raised by the OIG and to meet the Agency’s goal for accurate and reliable data, ETA has
implemented a data validation requirement in order to ensure the accuracy of data collected and
reported on program activities and outcomes.

1

ETA has developed a process for validating data submitted by states and grantees. Data
validation consists of two parts:
1) Report validation evaluates the validity of aggregate reports submitted to ETA by checking
the accuracy of the reporting software used to calculate the reports. Report validation is
conducted by processing a complete file of participant records into validation counts and
comparing the validation counts to those reported by the state or grantee.
2) Data element validation assesses the accuracy of participant data records. Data element
validation is conducted by reviewing samples of participant records against source
documentation to ensure compliance with Federal definitions.
Since there are two basic sources of reporting error, both parts are necessary to ensure the
validity of information reported to ETA. First, if the data collected are incorrect or data entry
errors occur, then the outcome information will not be accurate. As described above, data
element validation addresses this issue by comparing performance-related data in each state’s
participant record file to the original data in the source files and determining an error rate that
indicates the degree of accuracy of each data element used in calculating the state’s performance
results. Second, even if the data collected are correct, if the state’s or grantee’s reporting system
does not meet Federal standards, it could calculate the performance outcomes incorrectly.
Report validation addresses this issue by independently calculating performance results for the
data submitted by each state in its participant record file and comparing those results to the actual
results reported by the state. Error rates are determined for each performance outcome reported
by the state.
ETA has developed a set of validation tools discussed below – software and instructional user
handbooks – that states and grantees can use to validate data.
WIA Title IB, Wagner-Peyser, and TAA program staff have been conducting data validation for
three years. The states received training prior to beginning validation and receive ongoing
training and technical assistance from ETA’s data validation contractor throughout the validation
process. NFJP grantees have been conducting data validation for two years and have received
ongoing training and technical assistance during this period from ETA’s data validation
contractor. SCSEP grantees will begin data validation by the end of Calendar Year (CY) 2007.
Indian and Native American program grantees will pilot validation by 2008.
In brief, the results of the past three years of data validation have indicated the following:
• States and grantees are able to conduct data validation with a reasonable, but sustained, level
of effort.
• The validation process allows states and grantees to identify and address reporting errors.1
1

In the first year of validation, fifteen states had core performance report validation items with an error rate of
greater than 4 percent. Twenty-three states had non-core report validation items with an error rate exceeding 4%.
Numerous states sought and received technical assistance to determine and correct these report discrepancies.
Results of data element validation have also identified significant discrepancies between some of the data submitted
in state participant record files and the data contained in original participant source files. Analysis of these
discrepancies is currently in process.

2

•
•

States and grantees make reporting errors which need detecting and fixing.
The average staff requirements for a state to complete validation for the WIA Title IB,
Wagner-Peyser, and TAA programs are about 792 hours per year (or less than 1/2 of a staff
year). There is no startup burden for these programs because this was incurred when data
validation was first implemented three years ago. The average annual time estimate for
NFJP, Indian and Native American program, and SCSEP grantees to complete validation is
approximately 103 hours (or about 1/20 of a staff year). Startup activities for the Indian and
Native American program and SCSEP will require an additional 75 hours on average per
grantee in the initial year of validation. There is no NFJP startup burden because this was
incurred when NFJP data validation was first implemented two years ago.

On the basis of three years of successful validation implementation, ETA wishes to extend and
revise the data validation requirement for employment and training programs. In order to ensure
the accuracy of reported information throughout the workforce investment system, states and
grantees are required to conduct data validation. Data validation is required annually as follows:
•

Report validation should be performed before submission of annual reports.

•

Data element validation must be completed within 120 days after required annual reports or
participant records are due to ETA. Exact deadlines for the completion of data validation
will vary by program.

•

States and grantees are required to send data element validation output reports to ETA within
120 days after the submission of required annual reports or participant records.

States and grantees operating ETA programs subject to the data validation requirement will
validate the reports and participant records shown in Table 1 below.
Table 1 – Reports and Participant Record Files Validated
Report/Records

OMB Approval
No.

Expiration Date

ETA 9091 (annual report)

1205-0420

2/09

ETA 9002, VETS 200

1205-0240

2/09

TAPR

1205-0392

1/09

WIASPR

1205-0425

12/09

Indian and Native American

ETA 9085 (annual report)
ETA 9084 (annual report)

1205-0422

12/07

Senior Community Service
Employment Program

ETA 5140 (annual report)

1205-0040

6/072

Program
Workforce Investment Act Title
IB
Wagner-Peyser
Trade Adjustment Assistance
National Farmworker Jobs
Program

2

DOL is currently seeking OMB approval for an extension of this reporting requirement.

3

The user handbooks for each program provide a more detailed overview of the validation
process. These are available on ETA’s validation tools web site at
. A copy of Training
and Employment Guidance Letter No. 3-03, which outlines ETA’s data validation policy, is
attached in Appendix B.
Copies of ETA’s validation reports for the WIA Title IB program can be found in Appendices C
and D. The Report Validation Summary and Summary and Analytical Reports for the other
programs are based on the WIA template.
2. Purpose of Information Collection
ETA uses data validation results to evaluate the accuracy of data collected and reported to ETA
on program activities and outcomes. This information collection enables ETA to assure its
customers, partners, and stakeholders of the validity of performance data which underlie the
workforce accountability system. Further, data validation ensures that performance information
used for WIA accountability purposes, and to meet Government Performance and Results Act
(GPRA) responsibilities, are accurate.
Data validation was also developed with the goal of assisting states and grantees in providing
more accurate data. Validation allows states and grantees to detect and identify specific
problems with their reporting processes, including software and data issues, and to enable them
to correct the problems. In addition, the tools developed by ETA help states and grantees
analyze the causes of performance successes and failures by displaying participant data
organized by performance outcomes. These tools are available at no cost to states and grantees.
For data validation to be effective and to allow for continuous improvement, ETA is establishing
acceptable levels for the accuracy of reports and data elements in phases. For report validation,

4

the first three validation years focused on detecting and resolving any issues with state and
grantee data and reporting systems. Error rates collected in these years will be analyzed and,
based on this information, standards for accuracy will be established for the Program Year (PY)
2007 data validation. The implementation of a set of common performance measures has
delayed the establishment of standards for data element validation until states have had at least
two years to validate the same data elements.
Once accuracy standards are established, states and grantees will be held accountable for meeting
those standards and will be required to address any issues concerning data accuracy. States and
grantees that fail to meet accuracy standards will receive technical assistance from ETA and will
develop and implement a corrective action plan. Data that do not meet accuracy standards will
not be acceptable for measuring performance, and may keep the state or grantee from being
eligible for incentives that are awarded based on performance results. Significant or unresolved
deviation from accuracy standards may be deemed a failure to report.
3. Technology and Obstacles Affecting Reporting Burden
ETA knows of no technical obstacles to implementing and continuing data validation. ETA has
developed standardized software and user handbooks that states and grantees can use to conduct
data validation:
•

Software developed by ETA generates samples, worksheets, and reports on data accuracy.
For report validation, the software validates the accuracy of aggregate reports that are
generated by the state's or grantee's reporting software and produces an error rate for each
reported count. For data element validation, the software generates a sample of the
participant records and data elements for the state or grantee to validate. The software
produces worksheets on which the validator records information after checking the source
documentation in the sampled case files. The software calculates error rates for each data
element, with confidence intervals of 3.5 percent for large states/grantees and 4 percent for
small states/grantees.

•

User handbooks provide detailed information on software installation, building and
importing a validation file, and completing report and data element validation. The
handbooks also explain the validation methodology, including sampling specifications and
data element validation instructions for each data element to be validated.

Currently, all states and grantees use the software provided by ETA to conduct validation for
WIA Title IB, Wagner-Peyser, and TAA programs and the NFJP. States and grantees can obtain
technical assistance on validation procedures and the use of the validation tools from ETA’s data
validation contractor.
As mentioned above, the ETA software can be used to generate the aggregate information
required in reports submitted to ETA. States and grantees that use the software provided by ETA
to generate this aggregate information are not required to conduct report validation. However,

5

states still must demonstrate that they used the validation software to calculate their aggregate
reports.
For both report validation and data element validation, the ETA software uses the validation data
provided by the states or grantees to produce validation summary reports which, in compliance
with the Government Paperwork Elimination Act, are submitted via the system now used for
electronic transmission of reports to ETA.
4. Duplication
The data validation requirement does not duplicate any existing ETA program.
5. Burden on Small Business or Other Small Entities
While data validation will mostly be conducted by state governments and large, private, nonprofit organizations, some small entities will be required to conduct validation. Some of the
grantees operating the NFJP, the Indian and Native American program, and the SCSEP are
small, private, non-profit organizations providing services to a low number of individuals.
However, because of the low burden estimates associated with data validation for these
programs, this information collection will not have a significant economic impact on a
substantial number of small entities as indicated in Item 5 on OMB 83-I.
The data element validation process allows states and grantees to select appropriate validation
samples necessary to compute statistically significant error rates, rather than requiring the
validation of every participant case file. To reduce the relative burden on smaller states and
grantees as much as possible, the sample size for smaller entities is set to yield a less precise
error rate than for larger grantees and states.
6. Consequences of Failure to Collect Data
As mentioned in Part A.1, ETA was criticized in the past for a lack of monitoring and a
consequent inability to assure the validity of performance outcomes reported by states and
grantees. ETA regional staff are conducting data quality reviews based on current data
validation efforts to determine if states are in compliance with data validation guidelines. The
proposed continuation of the data validation requirement will allow ETA to continue to address
these issues. If data validation is discontinued, ETA will not be able to ensure that critical data
used for performance reports and accountability purposes, to meet GPRA responsibilities, and
for other management purposes are reliable.
7. Special Circumstances Involved in Collection of Data Validation Information
This request is consistent with 5 CFR 1320.5.

6

8. Pre-Clearance Notice and Responses
A Pre-clearance Notice was published in the Federal Register on June 1, 2007 (Vol. 72, No. 105,
Pages 30639 thru 30641). Following the 60-day comment period, public comments will be
reviewed and summarized. A copy of the FRN is attached in Appendix E. A copy of the
comments and ETA’s response will be attached in a new Appendix F after the comment period.
The names of individuals who provided technical assistance on statistical aspects of the data
validation design are provided in Part B.5.
9. Payments to Respondents
This information collection does not involve direct payments to respondents. However, ETA
does provide administrative funding to the participating states and grantees, which are listed as
the respondents for purposes of the Paperwork Reduction Act. The requirement to perform data
validation derives from states’ and grantees’ responsibility to provide accurate information on
program activities and outcomes to ETA. States and grantees are expected to provide resources
from their administrative funds for the data validation effort. Validation of program performance
is a basic responsibility of grantees, which are required to report program performance, under
Department of Labor regulations (29 CFR 95.51 and 97.40).
10. Confidentiality
Participant record layouts used in data validation for the WIA Title IB, Wagner-Peyser, and TAA
programs, the NFJP and the SCSEP have been revised to replace social security number fields
with state-assigned individual identifiers. Data element validation involves accessing wage
records by social security number in order to verify the accuracy of wage information contained
in the participant records submitted to ETA. To protect the confidentiality of program
participants, the validation software includes user functionality that allows program
administrators to limit access to this information based on administrative clearance.
Confidentiality is not an issue with report validation, which simply involves verifying the
accuracy of aggregate reports submitted to ETA.
11. Questions of a Sensitive Nature
The data collection includes no questions of a sensitive nature.
12. Respondent Annual Burden
Data validation is estimated to require an annual burden of 69,332 hours and $1,883,326 for all
six programs subject to the validation requirement.
Burden estimates for state programs – WIA Title IB, Wagner-Peyser, and TAA – are outlined in
Part 12.A. Data validation is estimated to require a total annual burden of 41,970 and
$1,364,025 for all state programs. Burden estimates for grantee programs – NFJP, Indian and
Native American programs, and SCSEP – are outlined in Part 12.B. Data validation is estimated
to require a total annual burden of 27,361 hours and $519,301 for all grantee programs.

7

A. State Programs: WIA Title IB, Wagner-Peyser, and TAA
Table 2 provides an overview of the annual burden for the WIA Title IB, Wagner-Peyser, and
TAA programs, including average hours and costs across states in all three programs. The
estimated annual hours needed to conduct validation for these programs is 792 hours (rounded)
on average per state and 41,970 hours for all states. The estimated annual cost of performing
validation is $25,736 on average per state and $1,364,025 for all states.
Table 2 - Calculation of Combined Annual Burden for WIA Title IB,
Wagner-Peyser, and TAA Programs
No. of
States
18

Hours per
State
1,206

Medium State

18

Small State
All States Total
Average per
State

Large State

•
•
•

•

Total Hours

Rate in $/hr

Total Cost

21,708

$32.50

$705,510

746

13,428

$32.50

$436,410

17

402

6,834

$32.50

$222,105

53

--

41,970

$32.50

$1,364,025

--

792

--

$32.50

$25,736

The calculation of the hours required to conduct validation includes sample size, the time for
validators to review sampled case files (34 minutes per file), the travel time to local offices to review
the files, and 15% of a supervisor’s time.
States have been divided into three categories – large, medium, and small – based on the number of
participants that exit a state’s program in a year. As discussed in Part B, the size of the state impacts
the number of sampled case files that must be reviewed and the travel time to local offices.
The annual travel time per office is estimated as 8 hours for large states, 6 hours for medium states,
and 3 hours for small states. This estimate is based on the assumption that states will conduct data
element validation separately for the WIA Title IB and TAA programs. If states conduct data element
validation for both programs at the same time, the travel time required to perform validation will
decrease.
The hourly rate is the estimated average hourly earnings for employees in state Unemployment
Insurance (UI) agencies in FY 2003 (as used for FY 2003 UI budget formulation purposes).

8

B. Grantee Programs: NFJP, Indian and Native American, and SCSEP
Table 3 provides an overview of the annual burden for the NFJP, the Indian and Native
American program, and the SCSEP, including average hours and cost across grantees in all three
programs. The estimated annual hours needed to conduct validation for these programs is 103
hours (rounded) on average per grantee and 27,361 hours for all grantees. The estimated annual
cost of conducting validation is $1,960 on average per grantee and $519,301 for all grantees.
Table 3 - Calculation of Annual Burden for
NFJP, Indian and Native American, and SCSEP Grantees

NFJP
Indian and
Native
American
SCSEP
All
Grantees
Average per
Grantee
•

•
•

No. of
Grantees
50

Hours per
Grantee
158

Total
Hours
7,900

$11.76/$32.50

Average Cost
per Grantee
$2,055

141

53

7,473

$11.76

$623

$87,843

74

162

11,988

$11.76/$32.50

$4,442

$328,708

265

--

27,361

$11.76/$32.50

--

$519,301

--

103

--

--

$1,960

Rate in $/hr

Total Cost
$102,750

--

Total costs for each program in Table 3 were calculated by multiplying the number of grantees times
the average cost per grantee. Tables 4, 5, and 6 show how the average cost per grantee was derived
for each program. Please note that the total costs listed for each program in Tables 4, 5, and 6 deviate
slightly from the total costs shown in Table 3 because the latter were derived using the average cost
per grantee.
The calculation of the hours required to conduct validation includes the time for validators to review
sampled case files (40 minutes per file) and 15% of a supervisor’s time. (Travel is not required for
grantees to conduct validation).
The hourly rate used to calculate cost depends upon the type of organization receiving the grant. For
state, county, and U.S. territory government grantees, the hourly rate is the estimated average hourly
earnings for employees in state UI agencies in FY 2003 (as used for FY 2003 UI budget formulation
purposes). For private non-profit grantees and Federally-recognized tribes, the hourly rate is the
average hourly earnings in the social assistance industry (CY 2006, Current Employment Statistics
survey, U.S. Census Bureau, http://data.bls.gov/PDQ/outside.jsp?survey=ce).

9

Tables 4, 5, and 6 provide a more detailed account of the annual burden estimates for each
grantee program. Per grantee figures in the Cost column of each table are rounded. As a result,
the total cost for all grantees may deviate slightly from the product of total grantee hours and
hourly rate.
Table 4 - Calculation of Annual Burden for NFJP
No. of
Grantees

Hours

Rate in
$/hr

Cost

Private Non-Profit

47

158 (per grantee)

$11.76

$1,858 (per grantee)

State or County
Government

3

158 (per grantee)

$32.50

$5,135 (per grantee)

All Grantees

50

7,900

--

$102,731

Avg. per Grantee

--

158

--

$2,055

Type of grantee

Note: The hourly rate used to calculate cost depends upon the type of organization receiving the grant.
For state and county government grantees, the hourly rate is the estimated average hourly earnings for
employees in state UI agencies in FY 2003 (as used for FY 2003 UI budget formulation purposes). For
private non-profit grantees, the hourly rate is the average hourly earnings in the social assistance industry
(CY 2006, Current Employment Statistics survey, U.S. Census Bureau).

Table 5 - Calculation of Annual Burden for the Indian and Native American Program
No. of
Grantees

Hours

Rate in
$/hr

Cost

Private Non-Profit

67

53 (per grantee)

$11.76

$623 (per grantee)

Federally-Recognized
Tribe

74

53 (per grantee)

$11.76

$623 (per grantee)

All Grantees

141

7,473

--

$87,843

--

53

--

$623

Type of Grantee

Avg. per Grantee

Note: The hourly rate used to calculate cost is the average hourly wage in the social assistance industry
(CY 2006, Current Employment Statistics survey, U.S. Census Bureau).

10

Table 6 - Calculation of Annual Burden for SCSEP
No. of
Grantees

Hours

Rate
in $/hr

Cost

Private Non-Profit

18

162 (per grantee)

$11.76

$1,905 (per grantee)

State or U.S. Territory
Government

56

162 (per grantee)

$32.50

$5,258 (per grantee)

All Grantees

74

11,988

--

$328,738

Avg. per Grantee

--

162

--

$4,442

Type of Grantee

Note: The hourly rate used to calculate cost depends upon the type of organization receiving the grant.
For state and U.S. territory government grantees, the hourly rate is the estimated average hourly earnings
for employees in state UI agencies in FY 2003 (as used for FY 2003 UI budget formulation purposes).
For private non-profit grantees, the hourly rate is the average hourly earnings in the social assistance
industry (CY 2006, Current Employment Statistics survey, U.S. Census Bureau).

13. Estimated Cost to Respondents
There are no additional costs other than those reported in item 12 above.

11

14. Cost to Federal Government
Federal costs are the staff and contractor costs required to design, implement, and manage data
validation as outlined in Tables 10 and 11 below. As indicated, contractor costs to develop
software and other validation tools, provide training to grantees, and provide on-going technical
assistance to grantees for the Indian and Native American and SCSEP programs will total
$500,000. The annual cost of contractor support to provide continual technical support to
grantees and states and any needed updates to validation tools for WIA Title IB, Wagner-Peyser,
and TAA programs and the NFJP will total approximately $900,000 per year for all four
programs. Costs for ETA staff to manage the data validation program will be $58,453 during the
development and implementation phase and $268,866 per year for continuing operations.
Tables 10 and 11 - Cost of Data Validation to Federal Government

Development and
Implementation Phase
(Indian and Native American and SCSEP)

Continuing Operations
(WIA Title IB, Wagner-Peyser,
TAA, and NFJP)

Contractor Support

$500,000

Contractor Support

$900,000

ETA Staff Total

$58,453

ETA Staff Total

$268,866

1 GS-15 (1/10 time)

$10,702

1 GS-15 (1/8 time)

$13,378

1 GS-14 (1/4 time)

$22,746

1 GS-14 (1/4 time)

$22,746

1 GS-9 (1/8 time)

$5,581

1 GS-13 (1/2 time)

$38,498

3 GS-12 (1/10 time)

$19,424

6 GS-12 (1/2 time)

$194,244

Total Cost

$558,453

Total Cost

12

$1,168,866

Note: Staff costs are based on the average salaries for Department of Labor grade ranges as of January
2007.

15. Reasons for Program Change and Change in Burden
The increase in burden hours results from the new inclusion of Indian and Native Americans
and Senior Community Service Employment Program grantees into the Data Validation Program.
The reduction in hours captured as an adjustment results from the re-estimation of time for
previously included programs by removing the level-of-effort associated with start-up burden.
The previously reported cost burden has been eliminated because it duplicated costs captured in
the form of burden hours.
16. Publication Information
ETA intends to publish results of data validation in an annual validation report.
17. Reasons for Not Displaying Date OMB Approval Expires
ETA will display approval information on the validation reports.
18. Exceptions to Certification
There are no exceptions to the certification statement in OMB 83-I.

13