Download:
pdf |
pdfTAA
Data Validation
Handbook
for
TAADV Software
Release 2.0
December 2006
(REV--6/7/02)
Table of Contents
Chapter
Page
I.
OVERVIEW ................................................................................................... 2
II.
VALIDATION PROCEDURES................................................................... 11
III.
RECORD LAYOUT..................................................................................... 45
APPENDIX A: PERFORMANCE MEASURE SPECIFICATIONS.................... A.1
APPENDIX B: DEV SAMPLING AND ERROR RATE ESTIMATION............ B.1
APPENDIX C: DEV INSTRUCTIONS................................................................ C.1
APPENDIX D: SOFTWARE INSTALLATION .................................................. D.1
APPENDIX E: SOFTWARE REFERENCE GUIDE ............................................E.1
1
I. OVERVIEW
Trade Adjustment Assistance (TAA) programs assist individuals whose
employment has been impacted by increased imports from or shifts in production
to foreign countries. The goal of the Trade Act programs is to help trade-affected
workers obtain the skills needed in today’s economic environment. States submit
Trade Act Participant Report (TAPR) records to ETA on a quarterly basis, and
ETA uses these records to measure the extent to which the program is meeting its
performance goals. Participants who exit the TAA program after receiving
employment and training services are included in three performance measures:
1. The entered employment measure is based on whether the exiter entered
employment.
2. The employment retention measure is based on employment status during
the second and third quarters after exit for exiters who entered
employment.
3. The average earnings measure is based on the earnings of exiters who
entered and were retained in employment after exit.
In order to ensure that the data used to calculate the TAA performance measures
are accurate, states are required to validate the records of participants included in
these measures. According to TEGL 3-03, TEN 9-06, and recent ETA guidance,
states are required to validate their Fiscal Year (FY) 2006 exiter records by April
15, 2007. States conduct data validation after final TAPR records for the fiscal
year have been submitted to ETA. The data to conduct the FY 2006 validation
should be available as of November 15, 2006, when the final TAPR for FY 2006
was due to ETA.
This TAA data validation handbook describes the procedures for states to follow to
review the accuracy of their TAPR records.
The handbook also provides
instructions on using the TAA data validation (TAADV) software that ETA
provides to states.
2
I. OVERVIEW
A. TAPR REPORTING GUIDELINES
The TAA fiscal year runs from October 1st through September 30th. For example,
FY 2006 runs from October 1, 2005 to September 30, 2006. Each state is required
to make four quarterly submissions of TAPR records to ETA according to the
guidelines shown in Tables I.1 and I.2. These tables specify the report quarter, exit
quarter, and TAPR due date for each quarterly submission for FY 2006 and 2007.
The Report Quarter row lists the four reporting quarters in the fiscal year, which
are the time frames during which records of TAA program exiters should be
compiled and the TAPR assembled. The Exit Quarter row lists the corresponding
participant exit dates. For example, to assemble the FY 2006 Q1 TAPR, the state
should assemble records for all participants who exited the TAA program between
7/1/2004 and 9/30/2004. The TAPR Due Date row specifies when the quarterly
TAPR submission is due to ETA. Each submission is due five and a half quarters
after the quarter of exit, which provides sufficient time to obtain the wage data
needed for calculating the retention rate.
Table I.1
Fiscal Year 2006 – Report Periods
Q1
Q2
Q3
Q4
Report Quarter
10/1/2005 –
12/31/2005
1/1/2006 –
3/31/2006
4/1/2006 –
6/30/2006
7/1/2006 –
9/30/2006
Exit Quarter
7/1/2004 –
9/30/2004
10/1/2004 –
12/31/2004
1/1/2005 –
3/31/2005
4/1/2005 –
6/30/2005
TAPR Due Date
2/15/2006
5/15/2006
8/15/2006
11/15/2006
3
I. OVERVIEW
Table I.2
Fiscal Year 2007 – Report Periods
Q1
Q2
Q3
Q4
Report
Quarter
10/1/2006 –
12/31/2006
1/1/2007 –
3/31/2007
4/1/2007 –
6/30/2007
7/1/2007 – 9/30/2007
Exit
Quarter
7/1/2005 –
9/30/2005
10/1/2005 –
12/31/2005
1/1/2006 –
3/31/2006
4/1/2006 – 6/30/2006
TAPR
Due Date
2/15/2007
5/15/2007
8/15/2007
11/15/2007
To conduct data validation and calculate the annual performance measures, the
validation extract file must contain records for exiters for all four quarters in the
fiscal year. For example, the FY 2006 validation extract file must contain records
for all participants who exited between July 1, 2004 and June 30, 2005. The FY
2007 validation extract file must contain records for all participants who exited
between July 1, 2005 and June 30, 2006.
4
I. OVERVIEW
B. OVERVIEW OF DATA VALIDATION
Data validation is designed to accomplish the following goals:
1. Detect and identify data problems with a state’s TAPR to enable the state
to correct the problems.
2. Ensure that critical performance data used to direct incentives and
sanctions and to meet ETA’s Government Performance and Results Act
(GPRA) responsibilities are reasonably accurate by calculating an error
rate for each data element validated.
3. Provide tools that help states analyze the causes of their performance
successes and failures by displaying participant data organized by
performance outcomes.
4. Minimize the burden on states in conducting the validation by providing
standardized software that displays the individual participant records by
outcome and provides built in random samples, validation worksheets,
and automated validation reports.
5. Further minimize the burden on the states by selecting the smallest
possible validation samples necessary to compute valid error rates.
Data validation typically involves two distinct processes—report validation and
data element validation. Report validation assesses the accuracy of state reports
by comparing the values calculated by states to values calculated by the TAADV
software. Because ETA itself calculates performance measures for all TAA
participants, report validation is unnecessary. However, state staff can use the
TAADV software to view and analyze the performance of individual exiters. The
TAADV software also calculates the state’s overall performance and generates a
performance measure report.
5
I. OVERVIEW
Data element validation confirms the accuracy of key data elements in the TAPR
by examining a sample of participant records to assess whether the data in the
sampled records are correct. When a record is selected for validation, state staff
compare specified data elements in the TAPR record to source documentation for
that participant.
Data element validation results in an estimate of the error rate for each data
element selected for validation. Data elements are selected for validation based on
three factors:
• Feasibility—ETA can validate data elements only where it is practical and
efficient to locate and examine supporting evidence within the state case
files. Therefore, such items as race, ethnicity, and gender will not be
validated because these data elements are self-reported by participants and it
is not practical to locate the participant to verify these characteristics.
• Risk—The process for data validation is based partly on the risk that the data
element can be inaccurate. Data elements involving human judgment are
more prone to error than data elements that do not involve human judgment.
For example, determination of employment based on supplemental sources
is higher risk than determination of employment from wage records.
• Importance—Data elements are selected for validation based primarily on
their importance to the integrity of the individual participant records and
their significance for generating performance outcomes.
Error rates are computed by determining the degree to which the accuracy of
the selected data elements is supported by evidence in the case files or by other
sources (such as wage records).
6
I. OVERVIEW
The validation process is designed to compute a reliable error rate using the
smallest possible sample to minimize the state’s burden in performing the
validation. To accomplish this objective, three sampling techniques are used:
1) To lessen the relative burden on smaller states as much as possible, the
precision standards are reduced yielding a smaller sample than for larger
states.
2) Sampling of local offices is performed prior to the selection of participant
records to reduce the number of locations that state staff must visit to
access supporting documentation.
3) Participant records are over-sampled if they are determined to be
important for calculating performance outcomes and if they have a higher
risk of error. The software weights the results of the validation to correct
for over- and under-sampling so that the final results represent the overall
error rate of each data element in the state’s TAPR.
7
I. OVERVIEW
C. HANDBOOK OVERVIEW
The following chapters and appendices guide states through the process of
validating data from their TAPR submission using the TAADV Software.
• Chapter II—Validation Procedures
Chapter II provides a step-by-step description of how to conduct data
validation using the TAADV software. This chapter describes how to build
and import a validation extract file, validate the sample that is selected by
the software, and generate performance measure and summary and
analytical reports.
• Chapter III—Record Layout
Chapter III provides the record layout specification for creating the TAA
validation extract file.
• Appendix A—Performance Measure Specifications
Appendix A provides detailed performance measure specifications used by
ETA and by the TAADV software to generate the TAA performance
measures.
• Appendix B—DEV Sampling and Error Rate Estimation
Appendix B provides the specifications for sampling and error rate
estimation that are used by the TAADV software to select the data element
validation sample, and to calculate error rates on the summary and analytical
report.
• Appendix C—DEV Instructions
Appendix C provides the instructions used by the validator to validate each
data element on the data element validation worksheet.
8
I. OVERVIEW
• Appendix D—TAADV Software Installation
Appendix D provides a step-by-step description of how to install the
TAADV software on a PC.
• Appendix E—TAADV Software Reference Guide
Appendix E provides a condensed list of all the menus and functions of the
TAADV software for quick reference.
D. PREPARING FOR VALIDATION
As indicated, Chapter II of this handbook describes each task that states should
follow to complete validation. Responsibility for completing these tasks will be
divided among various staff.
Managers are responsible for assuring that programmers and validators have the
resources needed to complete the validation as required by ETA. They are also
responsible for keeping the data validation effort on schedule.
Programmers have the primary responsibility for building the validation extract
file in the format specified in the record layout. Programming staff may also be
responsible for loading the file into the TAADV software.
Validators conduct the validation once the extract file has been imported into the
software. Validators should also work closely with programming staff to
determine which state data elements best meet the requirements specified in the
record layout.
9
I. OVERVIEW
Table I.2 summarizes the preparation tasks that states should complete before
embarking on the validation effort, and the staff who will likely be responsible for
completing each preparation task. All of these tasks can begin immediately, so that
the state can maximize the time available to validate its sample of TAPR records.
Table I.2
Preparing for Data Validation
Preparation Activity
Responsible Staff
Handbook Reference
Assemble validation team.
Managers
Not applicable
Review handbook.
Validators, programmers
All chapters and appendices
Install TAADV software.
Programmers
Appendix D
Develop a data validation
schedule and make staff
assignments.
Managers, validators,
programmers
Chapter II
Build and test validation
extract file of all FY2006
TAA exiters. Final file should
be generated as soon as
possible after the final FY
submission in November,
2006, to maximize the time
available to conduct data
validation.
Programmers, validators
Chapter II—Sections A and B,
Chapter III
Review data validation
instructions and update state
version of sources column.
Validators, programmers
Appendix C
After reviewing this handbook, states should contact Mathematica Policy
Research, Inc. by emailing [email protected] with any questions on
validating their TAPR submission.
10
II. VALIDATION PROCEDURES
This chapter provides a step-by-step description of how to conduct data validation.
Following are brief summaries of each section of the chapter.
A. Accessing the TAADV Software
Section A describes how to access the TAADV software and how to identify the
reporting period to be validated. This section also describes how the user can
change the reporting options after the initial sign in.
B. Creating and Importing a Validation Extract File
Section B describes how to build and import a fiscal year file of TAA exiters. This
section also describes how to access and review duplicate detection and error
reports of records rejected by the validation software during the import process.
C. Report Validation
Section C describes how to review the performance measure report generated by
the software, and how to access and analyze records by performance outcome
groups.
D. Data Element Validation
Section D describes how to conduct data element validation for a sample of exiter
records, using the worksheets generated by the software. This section also
describes how to access the summary and analytical report and submit results to
ETA. Section D concludes with a summary table and flowchart of the DEV steps.
11
II. VALIDATION PROCEDURES
A. Accessing the Validation Software
Task 1—Install the Application
Users should refer to Appendix D for detailed instructions on installing the
TAADV software.
Task 2—Open the Validation Software
Go to the Start menu, click on Programs, and then scroll to find TAA. Click on
TAA to open the software.
Task 3—Create a New Database or Open an Existing Database
To create a new database, go to File and click on New. Then use the Create
Database window to find the appropriate location to save the database. Type a
name for the database in the file name cell.
To open an existing database, go to File and click on Open. Use the Create
Database window to find the appropriate database. Click on the database name
and then click on Open.
12
II. VALIDATION PROCEDURES
Task 4—Sign In Screen
On the Sign In screen, the user can either choose to validate records for an entire
fiscal year or for one of the four quarters within that fiscal year.
Enter the fiscal year to be validated in the cell labeled Fiscal Year. Move the
cursor to any other cell and the validation period start and end date range will
automatically fill based on the fiscal year entered. The start and end date range is
the period during which participants must have exited the TAA program to be
included in the TAPR for the fiscal year. As specified in Tables I.1 and I.2, the FY
2006 validation extract file must contain records for all participants who exited
between July 1, 2004 and June 30, 2005. The FY 2007 file must contain records
for all participants who exited between July 1, 2005 and June 30, 2006.
Alternatively, users can leave the Fiscal Year blank and manually enter a date
range, as long as it is within the range of exit dates in the validation file.
Click on the “Sign In” button after specifying the FY or date range to be validated.
13
II. VALIDATION PROCEDURES
Task 5—Change Reporting Options
To change the reporting parameters after signing in to the software, open the
Change Reporting Options menu and select Change Reporting Options.
In the Change Reporting Options screen, the user can select and change several
options: the fiscal year, period start and end date, office name, WIB name, and/or
case manager’s name. This function enables the state to produce substate level
performance measure reports based on the options selected.
While date ranges are auto filled based on the fiscal year selected, users may
overwrite these date ranges to analyze performance for specific time periods. If the
user decides to change the fiscal year after importing the data, the performance
measure report and the performance outcome groups will not accurately reflect the
new selection. For the software to calculate an accurate report, the user must reimport the data after changing the fiscal year.
The WIB, office, and case manager drop down menus include the unduplicated
values in each of these fields on the import file. After the user selects filters from
the drop down menus and clicks on Save, the source table, performance outcome
group table, and performance measure report will all be reconfigured to reflect the
data and results for the selected subgroup. The selected subgroup appears in the
footer of the performance measure report to distinguish it from the state level
report. Users should note that changing these filters does not impact the sample of
exiters selected for validation.
14
II. VALIDATION PROCEDURES
After selecting filters to create the desired report, click on Save to save the options.
15
II. VALIDATION PROCEDURES
B. Create and Import a Validation Extract File
States should consolidate the four TAPR submissions for the Fiscal Year being
validated, and import the file into the TAADV software.
Task 1—Create a File Based on the Record Layout
To view the record layout see Chapter III of this handbook, or open the Import
Data menu and select Source Table Record Layout. The record layout is in the
TAPR format with the addition of six fields. The additional fields are observation
number (field #1), sampling unit (field #60), WIB name (field #61), office name
(field #62), case manager (field #63), and user field (field #64).
The user field and the sampling unit are user-defined fields. The user field
appears on the validation worksheet and can be filled with any additional data
element that the state wishes to have available on the DEV worksheet. For
example, states can include the participant name or other identifying information
that should appear on the validation worksheet to help locate case folders for
sampled participants. It is not mandatory that you use this field.
The sampling unit field is a numeric designator for the physical location of the case
file, and is generally an office or WIB. This field is used by the TAADV software
to create a clustered sample. If the state does not currently have a numeric case file
location designator, it should create one for each case file location. This is not a
mandatory field, so the software will not reject a record if it is blank. However, the
software will treat all records with blanks in the sampling unit field as coming
from the same location and will not be able to create a clustered sample. No two
locations should have the same sampling unit number.
The exit date field (field #43) can represent either the date the participant exited
the TAA program, or the date the participant exited the WIA program if the
participant was coenrolled in WIA. However, for coenrolled participants states
should be sure to use the exit date that was used to obtain post-exit wages.
16
II. VALIDATION PROCEDURES
The extract file must be in ASCII comma-delimited format. The software will also
accept comma separated (CSV) files. Fields must be in the order and format listed
on the record layout. Mandatory fields are specified in the record layout. Blanks
are acceptable in optional fields. However, blank or null values are not valid for
mandatory fields and will result in the record being rejected. The software will
accept files with quotes around all elements or no elements, but not a mix of the
two.
Chapter I provides information about the date ranges for the exiters to be included
in the extract file. The extract file can be generated after the final quarterly file for
the fiscal year being validated has been submitted to DOL.
The validation extract file is equivalent to the four TAPR submissions for the fiscal
year, unduplicated using the most recent record for each participant. One way that
states can create the validation extract file is to aggregate all of their TAPR
submissions for the program year being validated. Before aggregating the files
states should mark each record to indicate which TAPR submission it came from
(e.g. 1st through 4th). States can then unduplicate the aggregated file by retaining
only the last record submitted for exiters with multiple records. The extract file
can be completed by adding observation numbers and sampling units to the
unduplicated records, and then extracting from the state data base the sampling
unit, WIB name, office name, case manager, and any user field applicable to each
record. The original marker for submission quarter may be retained in the user
field if states choose to do so.
Alternatively, states may develop an extract routine to pull the most recent data for
each fiscal year exiter directly from the state’s data base, in the format specified in
the record layout. States using this approach should ensure that they run the
extract routine as close as possible to their fourth quarterly submission, to
minimize any differences between the data in the TAPR submission and in the
validation extract file.
17
II. VALIDATION PROCEDURES
Task 2—Import Extract Data
Once the data are formatted according to the record layout, open the Import Data
menu and select Import from Extract File. Select the file to be imported using
the Select File box. Click Open to import the extract file.
Once this is done, the Import File box will display counters for both the numbers
of records imported successfully and the number of records with errors. The
Import File box also includes a cancel button that can be used to stop the loading
process. It may take several minutes to import the data, depending on the size of
the file. Click Exit when a message appears that the import is complete.
18
II. VALIDATION PROCEDURES
To view the imported file open the Import Data menu and select View Source
Table. This is a read only screen. However, the columns in the source table are
sortable to facilitate state review of the imported records. Users can sort on any
column in the source table.
The total number of records in the source table is provided on the lower left corner
of the screen. If the cursor is placed on a particular row of a column, the software
will show that row number in red at the bottom of the screen. This feature enables
the user to easily count the number of records with a particular characteristic in the
source table. For example, on the following source table the cursor is on row 7 as
shown on the lower right.
19
II. VALIDATION PROCEDURES
20
II. VALIDATION PROCEDURES
Task 3 – Review Error Report
During file import, the TAADV software reads each record to ensure that all fields
are valid based on the record layout and the TAPR edit check logic specified in
Chapter III. For example, any records with missing data in mandatory fields—
such as Observation Number or Date of Birth—are rejected. Duplicate records or
records with invalid values are also rejected. The TAADV will also generate an
error if the number of columns in the extract file does not match the number of
columns specified in the record layout, if text values are placed in a number field,
or if dates are not in the proper format.
The TAADV produces an error report listing the rejected records and records with
warning errors. To view the latest error report, click on the Import Data menu
and then select View Latest Error Report. If no errors are found, no error report
is produced. The error report is also automatically saved to the TAA folder
(C:\Program Files\TAA) and is titled importerror.txt. To save each error report,
the importerror.txt file should be renamed so that it is not overwritten by
subsequent error reports.
The error report indicates which observation has an error and the corresponding
error message from Chapter III. States should use the error report in conjunction
with the record layout and edit check document in Chapter III to determine why
records have been rejected or identified as having warning errors.
After reviewing any error reports generated by the software, staff should determine
if the validation file must be regenerated or reformatted and reloaded. If a very
small number of records are rejected, it may not be necessary to re-import the file
before beginning validation.
21
II. VALIDATION PROCEDURES
Task 4—View Duplicates
This task allows the user to view the duplicate records identified by the software.
No data entry is required, this option is for analytical purposes only.
Click on the Import Data menu and select View Duplicates.
This opens a report that displays the duplicate records, including their observation
number, SSN, and Exit Date. All records that have the same SSN and Exit Date
are considered to be duplicates and are rejected by the software. The user must
determine if it is necessary to fix the extract. If so, the user must determine which
observations to return to the extract file and then reimport the data before
beginning validation.
22
II. VALIDATION PROCEDURES
C. Report Validation
Because ETA calculates performance measures for all TAA exiters, report
validation is unnecessary. However, state staff can use the TAADV software to
view and analyze the performance of individual exiters. The TAADV also
calculates the state’s performance and generates a performance measure report.
Task 1—View Performance Outcome Groups
The TAADV software assigns each participant record to a performance outcome
group following the specifications in Appendix A of this handbook. Organizing
the records into outcome groups serves two purposes. First, it provides the
numerical counts used to calculate the performance measures. Second, it is a useful
tool for analyzing performance outcomes and factors that may influence
performance. Appendix A describes in detail how the software assigns exiters who
received employment and training services to 25 performance outcome groups for
the entered employment, retention, and average earnings measures.
Click on the Report Validation menu. Then click on View Performance
Outcome Groups. This opens a window that displays the 25 performance
outcome groups and the total number of records that have been assigned to each
group. A section of the performance outcome group summary screen is shown
below. Exhibit A.1 in Appendix A displays the full performance outcome group
table. No data entry is required; this function is for analytical purposes only.
23
II. VALIDATION PROCEDURES
Double click on the arrow in the far left column of each performance outcome
group to view the detailed records in that group.
24
II. VALIDATION PROCEDURES
Task 2—View Detailed Records
After double clicking the arrow in the far left corner of a performance outcome
group, the user can view the detailed records in that group.
Sort Detailed Records
To sort records in ascending or descending order, double-click the heading of the
column you wish to sort.
Widen Columns
To increase the width of the columns, go to the intersection of the relevant column
heading, click the mouse button and drag to the desired width.
25
II. VALIDATION PROCEDURES
Task 3—View Performance Measure Report
This function displays a report that calculates the values for the three TAA
performance measures, based on the specifications in Appendix A of this
handbook.
To view the performance measure report, click on the Report Validation menu
and select View Performance Report. No data entry is required; this is a readonly function.
Click on the printer icon to print the report using a live printer. To produce the
report in Adobe Acrobat (PDF) format, click on the export report button to the
right of the printer icon and follow the prompts.
If users have selected a WIB, office and/or case manager filter on the Change
Reporting Options screen, the performance measure report will be generated for
that subgroup. The footer of the report will display the selected subgroup.
26
II. VALIDATION PROCEDURES
D. Data Element Validation
Task 1—Access Sampled Records
The first tasks in DEV are to identify the records that must be validated and to
assemble the validation worksheets and source documentation corresponding to
those records.
The software selects a sample of the TAA exiter records that have been imported
into the software, and presents the data for each sampled record on a worksheet.
Appendix B of this handbook provides the validation sampling specifications.
Click on the Data Validation menu and select Edit Worksheets.
This will open the data validation worksheet summary of sampled records. This
summary screen is read only. Identifying information on this summary list (and on
the individual worksheet for each sampled record) helps state staff locate the
source documents for the selected records. The summary rows for sampled records
that have been completely validated will be shaded in purple.
27
II. VALIDATION PROCEDURES
Summary Worksheet
To access, update, and print individual sampled records, double click on the row of
the applicable record on the worksheet summary. This will open a validation
worksheet for the selected record.
28
II. VALIDATION PROCEDURES
Users can also search for worksheets that contain specific data by clicking the right
mouse button in the field in which they wish to search. A Find window will open.
Enter in the value that the TAADV should search for in the sampled records. If the
value needs to match exactly, check the Exact Match box. Then click the “Find
Next” button. The location of the record that contains the value in the “Find
What” box will be listed in blue.
Find Window
29
II. VALIDATION PROCEDURES
The online worksheet is on two tabs, one for validation fields 1 through 17 and one
for comments. Identifying information at the top of the worksheet helps validation
staff locate the case file for the selected record.
The validation worksheet lists all of the data elements to be validated. Validation
is required for each data element where check boxes are present and unshaded on
the worksheet. States do not need to validate data elements when check boxes are
shaded.
Sample Online Data Validation Worksheet
30
II. VALIDATION PROCEDURES
The validation worksheets can be completed online on a desktop or laptop
computer, or they can be printed and completed by hand. The TAADV may also
be loaded onto the state’s central server, allowing multiple validators to complete
the worksheets on workstations or computers connected to the server. If validators
use paper worksheets, the findings must later be data entered in the online
worksheet.
Task 4 of this section provides detailed instructions for completing the online
worksheet. “Not Completed” is displayed in red in the upper left corner of the
online worksheet until the validator completes every data element on the worksheet
that requires validation. At that point the message changes to “Completed”.
31
II. VALIDATION PROCEDURES
Task 2—Print Worksheets
The validator must assemble the printed worksheets for the sampled records. The
software sorts the worksheets by WIB and then by office within each WIB to
simplify the task of identifying which offices must be visited, and which cases
must be validated at each location.
To print all of the worksheets in batch mode, click on the Data Validation menu
and then click on Print Worksheets. All worksheets for the sampled records are
formatted for printing. Click the printer icon at the top left of the screen to begin
printing. The sampled records will print by WIB.
Sample Printed Worksheet
32
II. VALIDATION PROCEDURES
Task 3—Print Sampled Counts Report
To simplify the task of identifying which offices must be visited and which case
files must be reviewed at each location, the TAADV software sorts the worksheets
by WIB and then by office within each WIB. The TAADV produces a Sampled
Counts Report that lists the number of records in the sample by WIB name, office
name, and sampling unit.
States are also required to validate records on-site for the Workforce Investment
Act (WIA) program. Like the TAADV, the WIA Data Reporting and Validation
System (DRVS) software organizes worksheets by WIB and by office within WIB.
These identical approaches may reduce the burden on the state validator if the
TAADV and the WIA DRVS both select records from the same office. This
reduction in burden, however, may be limited. For the samples to be statistically
valid, each program must sample offices and records independently of each other.
Thus, it is likely that some TAA records will be selected from offices, and possibly
WIBs, that have no sampled records for WIA.
To obtain the Sampled Counts Report, click on the Data Validation menu and
then click on Print Sampled Counts Report. This report shows the distribution
of the sample by location. No data entry is required, this report is for analytical
purposes only.
33
II. VALIDATION PROCEDURES
34
II. VALIDATION PROCEDURES
Task 4—Assemble Supporting Documentation and Complete Worksheets
State staff must obtain the source documentation for each sampled record in order
to complete the validation worksheets. If some of the information is not available
at the state level, the validator must travel to the physical location where the case
files are stored to access the relevant source documentation for each sampled
record. (If the state wishes to follow a different procedure, it must obtain approval
from its regional officer.) The validator reviews the local office’s sampled case
files for each data element present on the worksheet, using the validation
instructions and criteria in Appendix C. Some elements (such as wage records)
may be validated against central computer files. Instructions for validating
elements against central computer files are also included in Appendix C.
Many of the data elements will be blank on the worksheet due to two factors. First,
not every data element applies to each participant. For example, if the participant
has not entered training, the data element for date entered training will be blank on
the worksheet for that participant. Similarly, if employment was established
through supplemental sources, the data element for wages will be blank on the
worksheet.
Second, except for elements that denote employment during the first and third
quarter after exit, negative values are generally not validated and therefore not
included on the worksheets. For example, if the participant did not receive a
relocation allowance (i.e. has a value of “2”), the data element will be blank on the
worksheet.
Blank data elements do not need to be validated unless the instructions specifically
direct the validator to do so. The validator should carefully read the instructions
for each data element because they indicate when blank data elements must be
validated. For example, all earnings fields are validated whether or not they are
blank.
Appendix C provides separate instructions for each data element; the instructions
are presented in the same order in which the data elements appear on the
worksheet. The data element reference numbers on the worksheet correspond to
35
II. VALIDATION PROCEDURES
the data element reference numbers in the validation instructions. Each instruction
specifies the recommended sources for validating the data element. If the validator
locates equivalent source documentation that is not specified in the instructions,
the validator should record the new source in the state version of sources column in
the validation instructions.
After reviewing the source documentation and following the validation
instructions, the validator records the result in the appropriate checkbox for each
element. The two possible validation outcomes for each data element are:
1. Check Pass if the element was supported by/matched the source
documentation
2. Check Fail if the source documentation showed that the data element was
incorrect or that no source documentation was available
The validation worksheets can be completed online by one or more validators
accessing the same database (see Task 8 of Appendix D for more information on
setting this up). However, if validators complete paper worksheets, the results must
later be data entered into the corresponding online worksheet.
The worksheet also includes a comments section. Any comments related to the
validation of a particular record can be entered in the comments text box on the
second tab of the online worksheet. These comments will then appear on the
printed version of the worksheet. Comments can include the reason a data element
failed validation, definitional problems encountered by the validator, the
validator’s name if the state chooses to record it, or other relevant information.
After completing the Pass or Fail checkbox for each data element to be validated in
a sampled record, and entering any relevant comments, the validator should click
on the “Save and Exit” button on the online worksheet (as shown in Task 1 of this
section).
Follow this same series of tasks for each record in the sample.
36
II. VALIDATION PROCEDURES
Task 5—Data Validation Summary/Analytical Report
The software generates a summary and analytical report after all the data validation
sample worksheets have been completed. The report shows the number of errors
and the error rates for each data element validated.
Two types of error rates—Overall Error Rate and Reported Data Error Rate—are
calculated using the information entered on the worksheets by the state validators.
For each data element, the Overall Error Rate is the number of records in which
that data element is in error divided by the total number of records sampled,
weighted to account for the over- and under-sampling of particular types of
records. As particular data elements may not be present in every record sampled,
the Reported Data Error Rate includes in the denominator only those records for
which the particular data element was validated. Please see Appendix B of the
validation handbook for more information about the two types of error rates.
To access the report, click on the Data Validation menu and select Print
Summary/Analytical Report. No data entry is required; this screen is for
analytical purposes only. To print the summary and analytical report, click on the
printer icon on the tool bar.
37
II. VALIDATION PROCEDURES
38
II. VALIDATION PROCEDURES
States may include comments on the summary and analytical report to describe
issues encountered during the validation. To do this, click on the Data Validation
menu and select Summary/Analytical Comments. In the text box, enter any
comments applicable to the data validation effort, up to 250 words. Click the
Update button for the comments to be transferred to the comments section on the
summary and analytical report. Click Cancel to return to previously entered
comments.
States should determine whether their error rates are acceptable or unacceptable for
each data element. Federal quality standards for determining unacceptable error
rates will be developed after sufficient data have been collected on the validation
results.
39
II. VALIDATION PROCEDURES
Task 6 – Submit Validation Results to ETA
After all validation worksheets have been completed, states must submit their
Summary and Analytical Report to ETA via the e-submit functionality of the
validation software. Data validation files, including the completed worksheets,
summary and analytical reports, and copies of supporting documentation, should
be retained on-site for three years for monitoring purposes.
Click on the Data Validation menu and select Export Summary/Analytical.
After a minute or two, the software will open a web browser. Enter the validation
password provided by Traci DiMartini at ETA ([email protected]) and
click on the Login button. The ETA-provided password must be the TAA
validation password, not the TAPR submission password.
Enter your name, e-mail address, and telephone number in the appropriate cells.
Then, click on Browse. Use the Choose File window to find the XML file created
by the software. The location and name of the file are listed in the white cell on
the top of the web browser. Click on the Open button. Then click the Submit
button in the web browser.
40
II. VALIDATION PROCEDURES
Path and file name of XML file
41
II. VALIDATION PROCEDURES
It may take up to a minute for the XML to upload. Once the file has been
successfully transmitted, the DRVS Confirmation of Submission screen will
appear. Users can choose to print a copy of the confirmation, have a copy e-mailed
to them, or logout.
42
II. VALIDATION PROCEDURES
Table II.1 and Figure II.1 summarize the TAA data element validation process.
Table II.1: Overview of Data Validation
Task
Task Description
1
The validator assembles validation worksheets for the sampled TAPR records and
retrieves the case files corresponding to those records.
2
After obtaining the case file for a sampled case, the validator uses the validation
worksheet to locate the first relevant data element that must be validated. Fields for
data that are not relevant to a particular record may be blank. For elements where
data are present or where the absence of data requires validation, the validator uses
the reference number to locate the validation instruction for that data element in
Appendix C. Each data element listed in Appendix A gives instructions and
acceptable source documentation for validating that element only. An edited
definition of the data element is also included in the instructions.
3
The validator obtains one or more of the source documents listed as an acceptable
source to validate the element.
4
Following the instructions in Appendix C, the validator determines whether the
data element meets the validation criteria based on the information in the source
document(s).
5
If the data element is supported by/matches the source document(s), the validator
enters a checkmark in the Pass box on the worksheet. If the source document(s)
does not support/match the data element or no source document(s) for the data
element is in the case file, the validator enters a checkmark in the Fail box on the
worksheet, indicating an error.
6
The validator proceeds through each data element for each sampled record in the
same manner. The “Comments” section of the worksheet can be used to record
notes or to document issues that may be helpful in future validations.
7
After each data element has been validated for every sampled record, the validator
reviews the Summary and Analytical Report generated by the software. The
validator then e-submits the report to ETA.
43
10
11
Date of exit
Employed in first full quarter
after exit
DV W orksheet is
printed or displayed
on a laptop
08
Ref. #
Trade Readjustment Allowance
Data Element
1
20031206
3
Reported
Value
Pass
1
Appendix C
List of Source
Docum ents
Fail
2
44
SUPPORTED
4
3
Enter a checkm ark
in the Pass box
E n t e r a c h e c k m a rk in t h e F a il b o x o n t h e
worksheet if the data elem ent in the validation
file is not supported by/does not m atch the
inform ation in the source docum ent(s), or if
there is no source docum entation in the case
file.
E n te r a c h e c k m a rk in th e P a s s b o x o n th e
worksheet if the data elem ent in the validation
file is supported by/m atches
the inform ation in
3
the source docum ent(s).
Instructions:
APPENDIX C
EXAMPLE
FIGURE II.I Flowchart of Data Validation Tasks
Subm it Sum m ary
and Analytical Report
to ETA
After com pleting all
validation worksheets,
review Sum m ary
and Analytical Report
6
W as data
elem ent
supported?
Determine if the data
element passes all
Determine
characteristic
validationif rules
using:
passes all validation rules
using:
1) W orksheet
2) Data Validation
1) W Instructions
orksheet
2)3)Data
ListElement
of Source
Validation
Instructions
Documents
3)4)Documents
Source Documents
7
Enter a checkm ark
in the Fail box
NOT
SUPPORTED
5
II. VALIDATION PROCEDURES
III. RECORD LAYOUT
Attached is the record layout that should be used to create the extract file of TAA
exiters as specified in Section B of Chapter II of this handbook. The file must be
in ASCII comma-delimited format or comma separated (CSV) format. Fields must
be in the specified order and in the format listed on the record layout. The
software will accept files with quotes around all elements or no elements, but not a
mix of the two.
The record layout provides the following information:
•
•
•
•
•
•
•
•
•
•
Field Number in TAADV validation file
TAPR field number
New TAPR data element name
Old TAPR field number
Old TAPR data element name
Valid values – the record will be rejected if the data element does not meet the
valid value criteria
Changes from old TAPR record layout to this new layout
Edit check logic – the record will be rejected or will receive a warning error if
the data element does not meet the edit check logic criteria
Error type – whether failure to meet the edit check logic results in a warning or
reject error
Error message – the text displayed on the import error report when the data
element fails to meet the edit check logic
45
46
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
New
TAPR
No.
2
1
Field
No.
Pell Grant Recipient
UC Eligible Status
Limited English
Language Proficiency
Employment Status
Eligible Veteran Status
White
Hawaiian Native or other
Pacific Islander
Black or African
American
Asian
American Indian or
Alaska Native
Ethnicity Hispanic /
Latino
Individual with a
Disability
Gender
Date of Birth
Individual Identifier
State FIPS Code
OBS
NEW DATA ELEMENT
NAME
1 = Male
2 = Female
Blank or 0 = did not self-identify
YYYYMMDD
XXXXXXXXX
Sequential number
XX
1 = Yes
2 = No
Blank or 0 = did not self-identify
15
14
13
12
11
10
1 = Employed
2 = Employed, but Received Notice of
Termination of Employment or Military
Separation
3 = Not Employed
1 = Yes, <= 180 days
2 = Yes, Eligible Veteran
3 = Yes, Other Eligible Person
4 = No
1 = Yes
Blank or 0 = did not self-identify
1 = Yes
Blank or 0 = did not self-identify
Pell Grant Recipient
Unemployment
compensation status
1 = Yes
2 = No
1 = Claimant Referred by WPRS
2 = Claimant Not Referred by WPRS
3 = Exhaustee
4 = Neither Claimant nor Exhaustee
Limited English Language 1 = Yes
Proficiency
2 = No
Veteran status
White
Hawaiian Native or other
Pacific Islander
Black or African American 1 = Yes
Blank or 0 = did not self-identify
9
1 = Yes
Blank or 0 = did not self-identify
Asian
American Indian or Alaska 1 = Yes
Native
Blank or 0 = did not self-identify
Ethnicity Hispanic or
Latino
CHANGES
EDIT CHECK LOGIC
No additional edit checks
No additional edit checks
No additional edit checks
A. Mandatory Field
A. Mandatory Field
A. Mandatory Field
A. Mandatory Field
Four (4) values in new TAPR replaces
three (3) values in old TAPR.
A. If TAPR Element #20 (Date of
Application) is less than 09/30/2006, then
accept "blank" on import.
New data element.
A. None. Accept
record as long as
condition is true.
A. Reject
A. Reject
A. Reject
A. Reject
A. Reject
ERROR TYPE
A. Mandatory Field
A. Mandatory Field
A. Mandatory Field
A. Reject
A. Reject
A. Reject
B. If TAPR Element #20 is greater than or B. Reject
equal to 10/01/2006, then reject records that
contain a "blank" on import. TAPR Element
#13 will essentially become a mandatory
field at that point.
A. Mandatory Field
Four (4) values in new TAPR replaces
three (3) values in old TAPR.
One (1) value in new TAPR replaces two No additional edit checks
(2) values in old TAPR.
One (1) value in new TAPR replaces two No additional edit checks
(2) values in old TAPR.
One (1) value in new TAPR replaces two No additional edit checks
(2) values in old TAPR.
One (1) value in new TAPR replaces two No additional edit checks
(2) values in old TAPR.
One (1) value in new TAPR replaces two No additional edit checks
(2) values in old TAPR.
Two (2) values in new TAPR replaces
three (3) values in old TAPR.
Two digit state code in new TAPR
replaces full state name in old TAPR.
Section I: Identification and Participant Characteristics
VALID VALUES
Individual with a Disability 1 = Yes
2 = No
Blank or 0 = did not self-identify
Gender
Date of Birth
Individual Identifier
OBS
State Name
OLD DATA ELEMENT
NAME
8
7
6
5
4
3
2
1
Old
TAPR
No.
TAA Edit Checks and Crosswalk
A. Pell Grant Recipient must be Yes (1) or No (2).
A. UC Eligible Status is missing or invalid.
A. Limited English Language Proficiency must be
Yes (1) or No (2).
B. Employment Status cannot be blank if Date of
Application (TAPR #20) is on or after 10/1/2006.
A. Eligible Veteran Status is missing or invalid.
A. Date of Birth is missing or invalid.
A. Individual Identifier is missing or invalid.
A. State code is missing or invalid.
A. OBS number is missing.
ERROR MESSAGE
47
New
TAPR
No.
17
18
19
20
21
22
23
Field
No.
18
19
20
21
22
23
24
Date of Program
Participation
Program Type
Petition Number
Date of Application
Tenure with Employer at
Most Recent Qualifying
Separation
Most Recent Date of
Qualifying Separation
NEW DATA ELEMENT
NAME
Highest School Grade
Completed
22
21
20
19
18
17
Date of Participation
Program of participation
Petition Number
Date of Application
Tenure with Employer at
Qualifying Separation
Most Recent Qualifying
Separation
Old
TAPR
OLD DATA ELEMENT
No.
NAME
16
Highest School Grade
Completed
YYYYMMDD
1 = TAA
2 = NAFTA-TAA
3 = Both TAA and NAFTA-TAA
00000000
YYYYMMDD
000
YYYYMMDD
VALID VALUES
00 = No school grades completed
01 - 12 = Number of
elementary/secondary school grades
completed
13 - 15 = Number of college, or full-time
technical or vocational school years
completed
16 = Bachelor's degree or equivalent
17 = Education beyond the Bachelor's
degree
87 = Attained High School Diploma
88 = Attained GED or Equivalent
89 = Attained Certificate of
Attendance/Completion
90 = Attained Other Post-Secondary
Degree or Certification
91 = Attained AS/AA Degree or Diploma
CHANGES
87 in new TAPR replaces 12 in old
TAPR.
89, 90, 91 are added in new TAPR.
A. Mandatory Field
A. Mandatory Field
Note: The TAPR on-line submission module
will accept values with and without the field
being padded with "0s".
A. Mandatory Field
A. Mandatory Field
Note: The TAPR on-line submission module
will accept values with and without the field
being padded with "0s". For example, the
system will accept values of "060" or "60"
for the number of months employed with the
employer.
A. Reject
A. Reject
A. Reject
A. Reject
A. Reject
B. Reject
B. Must be less than or equal to TAPR
Element #42 (Date of Exit)
A. Mandatory Field
A. Reject
ERROR TYPE
A. Reject
A. Mandatory Field
Note: The TAPR on-line submission module
will accept values with and without the field
being padded with "0s". For example, the
system will accept values of "01" or "1" for
the number of elementary/secondary school
grades completed.
EDIT CHECK LOGIC
A. Mandatory Field
TAA Edit Checks and Crosswalk
A. Date of Program Participation is missing or
invalid.
A. Program Type is missing or invalid.
A. Petition Number is missing or invalid.
A. Date of Application is missing or invalid.
A. Tenure with Employer at Most Recent
Qualifying Separation is missing or invalid.
B. Most Recent Date of Qualifying Separation is
after the Date of Exit (TAPR #42).
A. Most Recent Date of Qualifying Separation is
missing or invalid.
ERROR MESSAGE
A. Highest School Grade Completed is missing or
invalid.
48
New
TAPR
No.
24
25
26
27
28
29
30
31
32
33
34
35
36
Field
No.
25
26
27
28
29
30
31
32
33
34
35
36
37
Remedial Trade
Readjustment
Allowances (TRA)
Additional Trade
Readjustment
Allowances (TRA)
Basic Trade
Readjustment
Allowances (TRA)
Received Customized
Training
Received Remedial
Training
Received On-the-Job
Training
Received Occupational
Skills (classroom)
Training
Occupational Skill
Training Code
Subsistence While in
Training
Travel While in Training
Training Completed
Date Completed, or
Withdrew from, Training
Date Entered Training
NEW DATA ELEMENT
NAME
33
32
31
Trade Readjustment
Allowances (TRA)
Received Remedial
Training
Received On-the-Job
Training
Received Occupational
Skills (classroom)
Training
Occupational code system
29
30
Occupational Skill
Training Code
Subsistence While in
Training
Travel While in Training
Training Completed
Date Completed, or
Withdrew from, Training
Date Entered Training
OLD DATA ELEMENT
NAME
28
27
26
25
24
23
Old
TAPR
No.
YYYYMMDD
Blank = did not receive training services
1 = Yes
2 = No
1 = Yes
2 = No
1 = Yes
2 = No
1 = Yes
2 = No
Blank or 0 = did not receive training
services
1 = Yes
2 = No
Blank or 0 = did not receive training
services
1 = Yes
2 = No
Blank or 0 = did not receive training
services
1 = Yes
2 = No
Blank or 0 = did not receive training
services
00000000
Blank = code is not available or did not
receive training service
1 = Yes
2 = No
Blank or 0 = did not receive training
services
1 = Yes
2 = No
Blank or 0 = did not receive training
services
1 = Yes
2 = No
Blank or 0 = did not receive training
services
CHANGES
A. Must be 1 or 2 if TAPR Element #24
(Date Entered Training) has a valid date.
A. Must be 1 or 2 if TAPR Element #24
(Date Entered Training) has a valid date.
A. Must be 1 or 2 if TAPR Element #24
(Date Entered Training) has a valid date.
A. Must be 1 or 2 if TAPR Element #24
(Date Entered Training) has a valid date.
Note: The TAPR on-line submission module
will accept values with and without the field
being padded with "0s".
A. Mandatory Field
New data element.
A. Mandatory Field
(Replaces old TRA field values 3 and 4.)
New data element.
A. Reject
A. Reject
A. Reject
C. Reject
B. Reject
A. Reject
B. Reject
A. Reject
ERROR TYPE
A. Reject
A. Reject
A. Reject
A. Reject
A. Reject
A. Reject
A. Reject
A. Must be blank or 00000000 if TAPR #24 A. Reject
(Date Entered Training) is blank
A. Must be 1 or 2 if TAPR Element #24
(Date Entered Training) has a valid date.
A. Must be 1 or 2 if TAPR Element #24
(Date Entered Training) has a valid date.
A. Must be 1 or 2 if TAPR Element #24
(Date Entered Training) has a valid date.
C. Must be less than or equal to TAPR
Element #42 (Date of Exit) if a valid date
exists.
B. Must be greater than or equal to TAPR
Element #24 if TAPR Element #24 has a
valid date.
A. Must be blank if TAPR Element #24
(Date Entered Training) is blank.
B. Must have a valid date if either TAPR
Element #30 or #31 or #32 or #33 is value
1.
Old Trade readjustment allowances
A. Mandatory Field
(TRA) data element was split into two
data elements in the new TAPR.
(Replaces old TRA field values 1 and 2.)
New data element.
Deleted in new TAPR
Numeric code in new TAPR replaces
alphanumeric code in old TAPR.
EDIT CHECK LOGIC
A. Must be blank or less than or equal to
TAPR Element #42 (Date of Exit).
Section II: Activity and Service Record
YYYYMMDD
Blank = did not receive training services
VALID VALUES
TAA Edit Checks and Crosswalk
A. Remedial Trade Readjustment Allowances
must be Yes (1) or No (2).
A. Additional Trade Readjustment Allowances
must be Yes (1) or No (2).
A. Basic Trade Readjustment Allowances must be
Yes (1) or No (2).
A. Received Customized Training must be Yes (1)
or No (2) if a Date Entered Training (TAPR #24) is
specified.
A. Received Remedial Training must be Yes (1) or
No (2) if a Date Entered Training (TAPR #24) is
specified.
A. Received On-the-JobTraining must be Yes (1)
or No (2) if a Date Entered Training (TAPR #24) is
specified.
A. Received Occupational Skills Training must be
Yes (1) or No (2) if a Date Entered Training (TAPR
#24) is specified.
A. Occupational Skill Training Code is specified,
but no Date Entered Training (TAPR #24) is given.
A. Subsistence While inTraining must be Yes (1)
or No (2) if a Date Entered Training (TAPR #24) is
specified.
A. Travel While in Training must be Yes (1) or No
(2) if a Date Entered Training (TAPR #24) is
specified.
A. Training Completed must be Yes (1) or No (2) if
a Date Entered Training (TAPR #24) is specified.
C. Date Completed or Withdrew From Training is
after the Date of Exit (TAPR #42).
B. Date Completed or Withdrew From Training is
before the Date Entered Training (TAPR #24).
A. Date Completed or Withdrew From Training is
specified, but no Date Entered Training (TAPR
#24) is given.
B. On-the-Job (TAPR #31), Remedial (TAPR #32),
or Customized (TAPR #33) Training is indicated,
but no Date Entered Training is given.
A. Date Entered Training is after the Date of Exit
(TAPR #42).
ERROR MESSAGE
49
New
TAPR
No.
37
38
39
40
41
Field
No.
38
39
40
41
42
Other Federal CoEnrollment
Date of WIA Title IB
Participation
Relocation Allowance
Job Search Allowance
NEW DATA ELEMENT
NAME
Waiver from Training
Requirement
38
37
36
35
Other Federal CoEnrollment
Date of registration
Relocation Allowance
Job Search Allowance
Old
TAPR
OLD DATA ELEMENT
No.
NAME
34
Waiver from Training
Requirement
1 = WIA Dislocated Worker
2 = Nat'l Emergency Grant
3 = Both 1 and 2 above
4 = Other Federal Employment and Job
Training Program (in addition to either 1 or
2 or both)
5 = Other Federal Employment and Job
Training Program (not including either 1 or
2)
0 = not enrolled in any other Federal
program
YYYYMMDD
Blank = did not receive services funded by
a WIA Title 1B or National Emergency
Grant program
1 = Yes
2 = No
1 = Yes
2 = No
B. Must be 1, 2, 3, 4, or 5 if TAPR Element B. Reject
#40 (Date of WIA Title IB Participation) has
a valid date.
A. Reject
A. Reject
A. Must be less than or equal to TAPR
Element #42 (Date of Exit) if a valid date
exists
A. Mandatory Field
A. Reject
A. Reject
ERROR TYPE
A. Reject
A. Mandatory Field
A. Mandatory Field
VALID VALUES
CHANGES
EDIT CHECK LOGIC
1 = Recall
Seven (7) values in new TAPR replaces A. Mandatory Field
2 = Marketable Skills
two (2) values in old TAPR.
3 = Retirement
4 = Health
5 = Enrollment Unavailable
6 = Training Not Available
7 = Reason Unknown/Served Prior to 2002
Amendments
0 = did not receive training waiver
TAA Edit Checks and Crosswalk
B. Field value for Other Federal Co-Enrollment
must be 1-5 if a Date of WIA Title IB Participation
(TAPR #40) is specified.
A. Other Federal Co-Enrollment is missing or
invalid.
A. Date of WIA Title IB Participation is after the
Date of Exit (TAPR #42).
A. Relocation Allowance must be Yes (1) or No
(2).
A. Job Search Allowance must be Yes (1) or No
(2).
ERROR MESSAGE
A. Waiver from Training Requirement is missing
or invalid.
50
New
TAPR
No.
42
43
44
45
46
47
Field
No.
43
44
45
46
47
48
Employed in 2nd Quarter
After Exit Quarter
Occupational Code of
Employment (if
available)
Type of Employment
Match 1st Quarter After
Exit Quarter
Employed in 1st Quarter
After Exit Quarter
Other Reasons for Exit
(at time of exit or during
3-quarter measurement
period following the
quarter of exit)
Date of Exit
NEW DATA ELEMENT
NAME
Occupational Code (if
available)
Occupational code system
42
Employed in 1st full
quarter after exit
Date of Exit
OLD DATA ELEMENT
NAME
41
40
39
Old
TAPR
No.
1 = Yes
2 = No
00000000
1 = UI Wage Records (In-State & WRIS)
2 = Federal Employment Records (OPM,
USPS)
3 = Military Employment Records (DOD)
4 = Other Administrative Wage Records
5 = Supplemental through case
management, participant survey, and/or
verification with the employer
Blank or 0 = was not employed in the first
quarter after the exit quarter
1 = Yes
2 = No
Blank or 00 = exited for a reason other
than one of the conditions listed above
98 Retirement
99 Not a Valid SSN
01 Institutionalized
02 Health/Medical
03 Deceased
04 Family Care
05 Reserve Forces Called to Active Duty
YYYYMMDD
VALID VALUES
New data element.
Deleted in new TAPR.
Numeric code in new TAPR replaces
alphanumeric code in old TAPR.
New data element.
New data element.
Section III: Outcomes
CHANGES
C. Reject
C. Mandatory Field
B. Reject
B. Must be 2 if TAPR 48 is Blank or 0.
A. Reject
B. Reject
A. Must be 1 if TAPR 48 (Type of
Employment Match 2nd Quarter After Exit
Quarter) is 1-5.
Note: The TAPR on-line submission module
will accept values with and without the field
being padded with "0s".
No additional edits
B. Must be 0, 5, or blank if TAPR 54 is 0 or
blank.
A. Must be 1-4 if TAPR 54 (Wages 1st
Quarter After Exit Quarter) is >0 and
<999999.99.
A. Reject
C. Reject
B. Must be 2 if TAPR 45 is Blank or 0.
C. Mandatory Field
B. Reject
A. Must be 1 if TAPR 45 (Type of
Employment Match 1st Quarter After Exit
Quarter) is 1-5.
A. Reject
B. Reject
No additional edit checks
A. Reject
ERROR TYPE
B. Must be greater than or equal to TAPR
Element #23 (Date of Program
Participation)
EDIT CHECK LOGIC
A. Mandatory Field
TAA Edit Checks and Crosswalk
C. Employed in 2nd Quarter After Exit must be
Yes (1) or No (2).
B. Employed in 2nd Quarter After Exit must be No
(2) if Type of Employment Match is Not Employed
(blank or 0).
A. Employed in 2nd Quarter After Exit must be
Yes (1) if Type of Employment Match is Wage or
Employment Records (1-4) or Supplemental (5).
B. Type of Employment Match 1st Quarter After
Exit Quarter must be Supplemental (5) or Not
Employed (blank or 0) if Wages is blank or 0.
A. Type of Employment Match 1st Quarter After
Exit Quarter must be Wage or Employment
Records (1-4) if Wages > 0.
C. Employed in 1st Quarter After Exit must be Yes
(1) or No (2).
B. Employed in 1st Quarter After Exit must be No
(2) if Type of Employment Match is Not Employed
(blank or 0).
A. Employed in 1st Quarter After Exit must be Yes
(1) if Type of Employment Match is Wage or
Employment Records (1-4) or Supplemental (5).
B. Date of Exit is on or before the Date of
Program Participation (TAPR #23).
A. Date of Exit is missing or invalid.
ERROR MESSAGE
51
New
TAPR
No.
48
49
50
51
52
53
54
55
56
Field
No.
49
50
51
52
53
54
55
56
57
Wages 3rd Quarter After
Exit Quarter
Wages 2nd Quarter After
Exit Quarter
Wages 1st Quarter After
Exit Quarter
Wages 1st Quarter Prior
to Participation Quarter
Wages 2nd Quarter Prior
to Participation Quarter
Wages 3rd Quarter Prior
to Participation Quarter
Type of Employment
Match 3rd Quarter After
Exit Quarter
Employed in 3rd Quarter
After Exit Quarter
NEW DATA ELEMENT
NAME
Type of Employment
Match 2nd Quarter After
Exit Quarter
48
47
46
45
44
43
Old
TAPR
No.
000000.00
Blank = element does not apply
000000.00
Blank = element does not apply
Third quarter following
exit
000000.00
Blank = element does not apply
000000.00
Blank = element does not apply
First quarter following exit 000000.00
Blank = element does not apply
Second quarter following
exit
Two (2) values in new TAPR replaces
three (3) values in old TAPR.
CHANGES
New data element.
Eight (8) digit figure in new TAPR
replaces seven (7) digit figure in old
TAPR.
Eight (8) digit figure in new TAPR
replaces seven (7) digit figure in old
TAPR.
Eight (8) digit figure in new TAPR
replaces seven (7) digit figure in old
TAPR.
New data element.
Eight (8) digit figure in new TAPR
replaces seven (7) digit figure in old
TAPR.
Eight (8) digit figure in new TAPR
replaces seven (7) digit figure in old
TAPR.
1 = UI Wage Records (In-State & WRIS) New data element.
2 = Federal Employment Records (OPM,
USPS)
3 = Military Employment Records (DOD)
4 = Other Administrative Wage Records
5 = Supplemental through case
management, participant survey, and/or
verification with the employer
Blank or 0 = was not employed in the third
quarter after the exit quarter
1 = Yes
2 = No
VALID VALUES
1 = UI Wage Records (In-State & WRIS)
2 = Federal Employment Records (OPM,
USPS)
3 = Military Employment Records (DOD)
4 = Other Administrative Wage Records
5 = Supplemental through case
management, participant survey, and/or
verification with the employer
Blank or 0 = was not employed in the
second quarter after the exit quarter
Two quarters prior to most 000000.00
recent qualifying
Blank = element does not apply
separation
Three quarters prior to
most recent qualifying
separation
Employed in 3rd full
quarter after exit
OLD DATA ELEMENT
NAME
B. Reject
B. Warning
B. Warning
B. Warning
B. Must not be >50000.00 and <999999.99. B. Warning
A. Must be >=0 and <=999999.99 or blank. A. Reject
B. Must not be >50000.00 and <999999.99. B. Warning
A. Must be >=0 and <=999999.99 or blank. A. Reject
B. Must not be >50000.00 and <999999.99. B. Warning
A. Must be >=0 and <=999999.99 or blank. A. Reject
B. Must not be >100000.00 and
<999999.99.
A. Must be >=0 and <=999999.99 or blank. A. Reject
B. Must not be >100000.00 and
<999999.99.
A. Must be >=0 and <=999999.99 or blank. A. Reject
B. Must not be >100000.00 and
<999999.99.
A. Must be >=0 and <=999999.99 or blank. A. Reject
B. Must be 0, 5, or blank if TAPR 56 is 0 or
blank.
A. Must be 1-4 if TAPR 56 (Wages 3rd
Quarter After Exit Quarter) is >0 and
<999999.99.
A. Reject
C. Reject
B. Must be 2 if TAPR 50 is Blank or 0.
C. Mandatory Field
B. Reject
A. Reject
B. Reject
ERROR TYPE
A. Reject
A. Must be 1 if TAPR 50 (Type of
Employment Match 3rd Quarter After Exit
Quarter) is 1-5.
B. Must be 0, 5, or blank if TAPR 55 is 0 or
blank.
EDIT CHECK LOGIC
A. Must be 1-4 if TAPR 55 (Wages 2nd
Quarter After Exit Quarter) is >0 and
<999999.99.
TAA Edit Checks and Crosswalk
A. Wages 3rd Quarter After Exit Quarter are
invalid.
B. Wages 3rd Quarter After Exit Quarter cannot
be > 50000.
A. Wages 2nd Quarter After Exit Quarter are
invalid.
B. Wages 2nd Quarter After Exit Quarter cannot
be > 50000.
A. Wages 1st Quarter After Exit Quarter are
invalid.
B. Wages 1st Quarter After Exit Quarter cannot be
> 50000.
A. Wages 1st Quarter Prior to Participation are
invalid.
B. Wages 1st Quarter Prior to Participation cannot
be > 100000.
A. Wages 2nd Quarter Prior to Participation are
invalid.
B. Wages 2nd Quarter Prior to Participation
cannot be > 100000.
A. Wages 3rd Quarter Prior to Participation are
invalid.
B. Wages 3rd Quarter Prior to Participation cannot
be > 100000.
B. Type of Employment Match 3rd Quarter After
Exit Quarter must be Supplemental (5) or Not
Employed (blank or 0) if Wages is blank or 0.
A. Type of Employment Match 3rd Quarter After
Exit Quarter must be Wage or Employment
Records (1-4) if Wages > 0.
C. Employed in 3rd Quarter After Exit must be Yes
(1) or No (2).
B. Employed in 3rd Quarter After Exit must be No
(2) if Type of Employment Match is Not Employed
(blank or 0).
A. Employed in 3rd Quarter After Exit must be Yes
(1) if Type of Employment Match is Wage or
Employment Records (1-4) or Supplemental (5).
B. Type of Employment Match 2nd Quarter After
Exit Quarter must be Supplemental (5) or Not
Employed (blank or 0) if Wages is blank or 0.
ERROR MESSAGE
A. Type of Employment Match 2nd Quarter After
Exit Quarter must be Wage or Employment
Records (1-4) if Wages > 0.
52
WIB Name
Office Name
Case Manager
User Field
61
62
63
64
Participated in ATAA
Sampling Unit
58
59
NEW DATA ELEMENT
NAME
Recalled by Layoff
Employer
60
New
TAPR
No.
57
Field
No.
58
Old
TAPR
OLD DATA ELEMENT
No.
NAME
49
Recalled by Layoff
Employer
Text
Text
Text
Text
Number
1 = Yes
2 = No
1 = Yes
2 = No
VALID VALUES
New data element.
CHANGES
No additional edit checks
No additional edit checks
No additional edit checks
No additional edit checks
No additional edit checks
A. Mandatory Field
EDIT CHECK LOGIC
A. Mandatory Field
TAA Edit Checks and Crosswalk
A. Reject
ERROR TYPE
A. Reject
A. Participated in ATAA must be Yes (1) or No (2).
ERROR MESSAGE
A. Recalled by Layoff Employer must be Yes (1)
or No (2).
APPENDIX A
PERFORMANCE MEASURE SPECIFICATIONS
This appendix provides the specifications used by the software to assign the
TAA exiters to performance outcome groups and to calculate the entered
employment, retention, and average earnings measures. Performance is calculated
using the common performance measure calculation specifications used by the
WIA and Employment Service programs.
I. PERFORMANCE OUTCOME GROUP SPECIFICATIONS
As described in Chapter I, participants who exit the TAA program after
receiving employment and training services are included in three performance
measures:
• The Entered Employment measure calculates the percentage of exiters in the
fiscal year TAPR file who were employed in the first full quarter after the
exit quarter. This measure is calculated as follows:
Of those who are not employed at the date of participation:
# of participants who are employed in the first quarter (qtr) after the exit qtr
# of participants who exit during the qtr
• The Employment Retention measure calculates the percentage of those
exiters in the entered employment rate numerator who were still employed in
the second and third full quarters after exit. This measure is calculated as
follows:
# of participants who are employed in both the second and third qtrs after the exit qtr
# of participants who are employed in the first quarter after the exit qtr
A.2
• The Average Earnings measure is the total post-program earnings from
wage records in the second plus third quarters after exit divided by the
number of adults in the employment retention numerator for whom wage
records are available. This measure is calculated as follows:
[Total earnings in the second quarter plus total earnings in the third quarter after the
exit quarter]
# of participants who are employed in the first, second, and third quarters after the exit
qtr
The Performance Measure report generated by the TAADV software lists the
three TAA performance measures as well as the numerators and denominators used
to calculate each measure. The numerators and denominators are derived from the
number of participants assigned to each of the corresponding performance outcome
groups by the TAADV software.
Figure A.1 shows how the individual records in the TAPR file are assigned by
the software to one of 25 outcome groups based on the participants’ pre- and postprogram employment status, employment retention status, and the source used for
recording participant earnings. The TAADV determines the number of participants
who fit each outcome group and computes the three performance measures based
on the number of participants in each group.
Organizing the records into outcome groups serves two purposes. First, it
provides the numerical counts used in calculating the performance outcome
measures (see Appendix B for the detailed specifications). Second, it serves as a
useful tool for analyzing performance outcomes and factors that may influence
performance.
Columns 6 through 10 of Figure A.1 show how the count of participants in
each group, or the sum of their earnings for the average earnings measure, is used
in the calculation of each performance measure. For example, participants in group
1 are included in the numerators and denominators of the entered employment and
retention rates, and their earnings are included in numerator of the average
earnings measure.
A.3
A.4
Emp
Qtr+1
Emp
Status
UnEmp
UnEmp
UnEmp
UnEmp
UnEmp
UnEmp
UnEmp
UnEmp
UnEmp
UnEmp
UnEmp
UnEmp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
NA
Group
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
NA
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
UnEmp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
UnEmp
3
2
1
NA
Wage
Wage
Wage
Wage
Supp
Supp
Supp
Supp
NA
NA
NA
NA
NA
A.4
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
UnEmp
UnEmp
NA
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
UnEmp
UnEmp
NA
Emp
Qtr+2
Type
Match
Qtr+1
Wage
Wage
Wage
Wage
Supp
Supp
Supp
Supp
NA
NA
NA
NA
5
4
NA
Wage
Wage
Supp
Supp
Wage
Wage
Supp
Supp
NA
NA
NA
NA
Wage
Wage
Supp
Supp
Wage
Wage
Supp
Supp
NA
NA
NA
NA
Type
Match
Qtr+2
6
NA
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
UnEmp
Emp
UnEmp
NA
Emp
Emp
Emp
Emp
Emp
Emp
Emp
Emp
UnEmp
Emp
UnEmp
NA
Emp
Qtr+3
7
NA
Wage
Supp
Wage
Supp
Wage
Supp
Wage
Supp
NA
NA
NA
NA
Wage
Supp
Wage
Supp
Wage
Supp
Wage
Supp
NA
NA
NA
NA
Type
Match
Qtr+3
8
10
11
12
Yes
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
No
No
No
No
No
No
No
No
No
No
No
No
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
No
No
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
No
No
No
Other
Reasons
for Exit EER Num EER Den Ret Num
9
25 GROUPS OF TAA EXITERS USED TO CALCULATE
ENTERED EMPLOYMENT, RETENTION AND AVERAGE EARNINGS
EXHIBIT A.1
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
Ret Den
13
No
Yes
No
No
No
No
No
No
No
No
No
No
No
Yes
No
No
No
No
No
No
No
No
No
No
No
Earn
14
II. PERFORMANCE MEASURE CALCULATION SPECIFICATIONS
The following specifications detail how ETA calculates the three TAA
performance measures using the TAPR submission. These specifications are also
used by the TAADV software to calculate the three TAA measures.
Entered
Employment Rate
Numerator:
Count of unique RECORDS where (EMPLOYMENT STATUS AT
PARTICIPATION = 2 or EMPLOYMENT STATUS AT
PARTICIPATION = 3) and ((EMPLOYED 1ST QUARTER AFTER
EXIT QUARTER = 1 and TYPE OF EMPLOYMENT MATCH 1ST
QUARTER AFTER EXIT QUARTER = 5) or (WAGES 1ST
QUARTER AFTER THE EXIT QUARTER > 0 and WAGES 1ST
QUARTER AFTER THE EXIT QUARTER < 999999)) and DATE OF
EXIT is within the report period and (OTHER REASONS FOR EXIT =
00 or OTHER REASONS FOR EXIT = 98 or OTHER REASONS FOR
EXIT is null).
divided by
Denominator:
Count of unique RECORDS where EMPLOYMENT STATUS AT
PARTICIPATION is 2 or 3 and DATE OF EXIT is within the report
period and (OTHER REASONS FOR EXIT = 00 or OTHER REASONS
FOR EXIT = 98 or OTHER REASONS FOR EXIT is null).
A.5
Employment
Retention Rate
Numerator:
Count of unique RECORDS where ((EMPLOYED 1ST QUARTER
AFTER EXIT QUARTER = 1 and TYPE OF EMPLOYMENT MATCH
1ST QUARTER AFTER EXIT QUARTER = 5) or (WAGES 1ST
QUARTER AFTER THE EXIT QUARTER > 0 and WAGES 1ST
QUARTER AFTER THE EXIT QUARTER < 999999)) and
((EMPLOYED 2ND QUARTER AFTER EXIT QUARTER = 1 and
TYPE OF EMPLOYMENT MATCH 2ND QUARTER AFTER EXIT
QUARTER = 5) or (WAGES 2ND QUARTER AFTER THE EXIT
QUARTER > 0 and WAGES 2ND QUARTER AFTER THE EXIT
QUARTER < 999999)) and ((EMPLOYED 3RD QUARTER AFTER
EXIT QUARTER = 1 and TYPE OF EMPLOYMENT MATCH 3RD
QUARTER AFTER EXIT QUARTER = 5) or (WAGES 3RD
QUARTER AFTER EXIT QUARTER > 0 and WAGES 3RD
QUARTER AFTER EXIT QUARTER < 999999)) and DATE OF EXIT
is within the report period and (OTHER REASONS FOR EXIT = 00 or
OTHER REASONS FOR EXIT = 98 or OTHER REASONS FOR EXIT
is null).
divided by
Denominator:
Count of unique RECORDS where ((EMPLOYED 1ST QUARTER
AFTER EXIT QUARTER = 1 and TYPE OF EMPLOYMENT MATCH
1ST QUARTER AFTER EXIT QUARTER = 5) or (WAGES 1ST
QUARTER AFTER THE EXIT QUARTER > 0 and WAGES 1ST
QUARTER AFTER THE EXIT QUARTER < 999999)) and DATE OF
EXIT is within the report period and (OTHER REASONS FOR EXIT =
00 or OTHER REASONS FOR EXIT = 98 or OTHER REASONS FOR
EXIT is null).
A.6
Average Earnings
Numerator:
Sum of (WAGES 2nd QUARTER AFTER THE EXIT QUARTER and
WAGES 3rd QUARTER AFTER THE EXIT QUARTER) where
(WAGES 1ST QUARTER AFTER EXIT QUARTER > 0 and WAGES
1ST QUARTER AFTER EXIT QUARTER < 999999.99) and (WAGES
2nd QUARTER AFTER EXIT QUARTER > 0 and WAGES 2nd
QUARTER AFTER EXIT QUARTER < 999999.99) and (WAGES 3rd
QUARTER AFTER EXIT QUARTER > 0 and WAGES 3rd QUARTER
AFTER EXIT QUARTER < 999999.99) and DATE OF EXIT is within
the reporting period and (OTHER REASONS FOR EXIT is null or
OTHER REASONS FOR EXIT = 00 or OTHER REASONS FOR EXIT
= 98)
divided by
Denominator:
Count of unique RECORDS where (WAGES 1ST QUARTER AFTER
EXIT QUARTER > 0 and WAGES 1ST QUARTER AFTER EXIT
QUARTER < 999999.99) and (WAGES 2nd QUARTER AFTER EXIT
QUARTER > 0 and WAGES 2nd QUARTER AFTER EXIT
QUARTER < 999999.99) and (WAGES 3rd QUARTER AFTER EXIT
QUARTER > 0 and WAGES 3rd QUARTER AFTER EXIT QUARTER
< 999999.99) and DATE OF EXIT is within the reporting period and
(OTHER REASONS FOR EXIT is null or OTHER REASONS FOR
EXIT = 00 or OTHER REASONS FOR EXIT = 98).
A.7
APPENDIX B
DEV SAMPLING AND ERROR RATE ESTIMATION
The primary data validation objective is to identify the sources of error (that is,
which data elements or which kinds of cases are more prone to error) and to
provide error information for selected data elements irrespective of whether or not
they are used to measure performance. This appendix discusses the general
approach to error rate estimation (Section A) and detailed sampling specifications
for TAA (Section B).
A. GENERAL APPROACH
The sample design for TAA is a stratified sample. The software randomly
selects records with differential sampling rates by the type of record. Participant
subgroups are over-sampled if they are determined to be very important for
calculating performance outcomes, and if they have a higher risk of error. Records
are randomly sampled within the subgroups. Sample sizes vary by state, with
smaller states having smaller samples than larger states for two reasons. First, to
achieve a given level of precision, smaller states require smaller sample sizes than
larger states. Second, precision requirements are relaxed to reduce the relative
burden on smaller states of validating data. The sample design also clusters the
sample in a small number of offices.
This design yields estimates that have a variance that exceeds the variance of a
simple random sample. Currently, we assume the variance to be twice that of a
simple random sample (a “design effect” of 2). After data validation has been
implemented for a sufficient period of time, an analysis will be conducted to
estimate the true design effect and to determine how much it varies from current
estimates. The results of this analysis will be used to refine the sample design.
After the records are validated, the software uses the validation results to
generate error rate estimates using two methods. With the first method (Overall
Error Rate), the error rates equal the total number of records in error for a
particular data element divided by the number of records sampled, weighted to
account for the over- and under-sampling of particular types of records.
As particular data elements may not be present in every record, the second
method (Reported Data Error Rate) includes in the denominator only those records
for which that particular data element was validated. With this method of
calculation, the error rates equal the total number of records in error for a particular
data element divided by the number of records for which that data element was
B.2
validated, weighted to account for the over- and under-sampling of certain types of
records.
For example, if 100 records are sampled, only 50 have a particular data
element, and there are 5 errors in that data element, the first method results in an
error rate of 5 divided by 100, or 5 percent, while the second method results in an
error rate of 5 divided by 50, or 10 percent.1 The Data Validation Summary and
Analytical Report uses both methods to list the percent error for each data element
validated.
B.
SAMPLING SPECIFICATIONS
As described in Section A, the software selects a sample of exiters by
creating a clustered, stratified sample in order to minimize the burden that data
validation imposes on states and local offices and still provide sufficiently precise
results. The software first selects a sample of offices.2 From the sampled offices,
the software selects a sample of records of exiters. The size of the samples varies
by state with precision estimates for small states reduced to minimize the burden
validation imposes on them.
To decide which offices to sample, the software weights the records for each
office. The weights are based on the risk that the data may be incorrect and the
importance of the record for measuring performance. Data elements related to
employment and source of employment information are subject to the greatest risk
of error and are the most important for calculating performance. Supplemental
sources for wages are the riskiest data, followed by wage records. Data that
indicate an individual is unemployed present the least risk. Based upon these risk
assessments, records that use supplemental sources to demonstrate an individual’s
employment in the first and/or third quarter after exit receive a weight of 3, records
that use wage records to demonstrate an individual’s employment receive a weight
of 2, and records for individuals who are unemployed receive a weight of 1.
The software creates a weight for each office by summing the weights of each
of the office’s records. The software then selects specific offices based on the
1
Weighting for over- or under-sampling was not considered in this example.
2
Some states may receive data by Workforce Investment Boards (WIBs), not by offices. In this case, the
software selects WIBs as the primary sampling unit instead of offices.
B.3
offices’ weighted values. The number of offices (n) selected is a function of the
number of offices in the entire state (N). The more offices sampled, the greater the
accuracy of the estimates. As more offices are sampled, the burden that validation
imposes on the state increases.
Table B.1 below shows the sampling rule the software uses to determine the
number of offices to sample. Column A provides ranges for the number of offices
in a state. Column B provides the number of offices selected. Thus, for a large state
with 250 or more offices, the software selects 30 offices, as shown in Row 1 of the
table. For a medium size state with 80 offices, the software selects 15 offices, as
shown in Row 4 of the table.
TABLE B.1 Office Sampling
1
2
3
4
5
6
7
A
B
Number of Offices Number of Offices
in State (N)
Sampled (n)
250 or more
30
200-249
25
100-199
20
75-99
15
30-74
10
7-29
7
Fewer than 7
All
The software automatically selects any office(s) that accounts for 0.8 × 1/n or
more of the state’s total weight.3 The remaining offices are randomly selected
based on their weighted value.
Next, the software selects the individual records to validate, selecting a sample
of records from the sampled offices. The probabilities of selection are proportional
to the weights assigned to each record. Consequently, a greater proportion of
3
The initial weights and number of offices are estimates. The pilot validation and early implementation will
provide data to determine if the estimates are correct. If the information gained from these studies suggests that the
estimates are incorrect, they will be adjusted accordingly.
B.4
records with supplemental wage data will be selected than would be in a simple
random sample.
Table B.2 below illustrates how the software determines the sample size.
Column A gives ranges for the number of exiters. Depending upon the number of
exiters, the software selects a level of precision, shown in Column B.4 Column C
provides a range for the number of records to be sampled for validation. For
example, if a state has 480 exiters, the software selects a sample of between 100
and 150 records to validate, providing a precision level of 3.5 percent.
TABLE B.2
EXITER RECORD SAMPLING
A
B
C
# of Exiters
Confidence
Interval5
Range of Sample
1
300 or greater
3.5%
100-1506
2
0-299
4%
0-83
After the sampled records are validated, the software determines error rates for
each of the data elements specified. Because the software clusters and stratifies the
samples, it must weight the results yielding final weighted error rates for each data
element.
4
Precision is determined by the confidence interval. Confidence intervals measure the accuracy of the estimate.
For example, a data element with a 7 percent error rate might have a +/− 2 percent degree of accuracy. The +/− 2
percent is the confidence interval. In this example, the confidence interval means that while the error rate is given as
7 percent, the actual error rate is between 5 and 9 percent.
5
Several factors must be incorporated into the calculation of the confidence interval. Because of the small
sample size, a finite population correction (fpc) needs to be incorporated into the calculations. Assumptions about
the accuracy of the error estimate are also incorporated; the calculations assume a confidence interval at a 0.05 level
for a two-tail test.
6
The sample size will not exceed 150 participants per group.
B.5
APPENDIX C
DEV INSTRUCTIONS
Appendix C provides instructions for validating each data element listed on the
data validation worksheets. The instructions have the following components:
Data Element: The participant characteristic to be validated, as listed on the
record layout and validation worksheet.
Reference #: The number corresponding to each data element on the
validation worksheet.
Format: The format in which each data element value should be recorded in
the imported TAPR file.
Definition: Definition of the data element being validated.
Federal Sources: The definitive federal sources of information for validating
each data element.
State Version of Sources: The state version of the sources of information for
validating each data element. Some state sources may be directly available to
state staff, while other documentation may be stored at the local level. States
have flexibility to determine the most efficient source to use for validation
when more than one source is available.
Instructions: Rules that instruct the validator on how to validate each data
element listed on the worksheet.
The validator identifies each data element on the worksheet and then locates
the corresponding instructions for that data element using the reference number
listed on the validation worksheet. The validator then follows the instructions for
validating that particular data element.
Although the instructions for different data elements may be similar, the
validator should read each instruction carefully because in some cases a blank data
element is not validated, while in others (such as wage records) a blank data
element must be validated.
Please refer to Chapter II, Section D—Data Element Validation—for detailed
information on using these instructions to conduct data element validation.
C.2
Friday, December 01, 2006
Trade Adjustment Assistance Data Element Validation Instructions
Page 1 of 5
1. If a record is missing from the office in which it is supposed to be, the validator should check the box at the top of the worksheet that says "Missing
Record." No other elements should be validated.
2. If the validator cannot determine where a record is supposed to be located, the validator should check the box at the top of the worksheet that says
"Unable to Locate." No other elements should be validated.
3. If a record for a person not registered in TAA has wrongly been included in the extract file, the validator should check the box at the top of the
worksheet that says "Invalid Record." No other elements should be validated.
4. If a record has a wrong Social Security Number (SSN), the validator should check the "Wrong SSN" checkbox AND validate all data elements listed
on the worksheet.
There are 4 conditions used to describe higher level data problems:
MISSING AND INVALID RECORDS
2. If the validation instruction says SUPPORT: Enter a checkmark in the box in the pass column if the data on the worksheet are supported by the data
in the source documentation. Enter a checkmark in the box in the fail column if the data on the worksheet are not supported by the data in the source
documentation or if no source documentation is found.
To support, the data on the source documentation must provide evidence that the data on the worksheet are correct. This instruction is used
when information must be interpreted or processed before it can be used to assess the accuracy of the data on the participant records. For example, source
documentation can support waiver from training requirement.
The validator must validate all data elements that have checkboxes next to them.
1. If the validation instruction says MATCH: Enter a checkmark in the box in the pass column if the data on the worksheet match the data in the source
documentation. Enter a checkmark in the box in the fail column if the data on the worksheet do not match the data in the source documentation or if no
source documentation is found.
To match, the data on the worksheet must be the same as the data in the source documentation. For example, if the worksheet says a participant's date
of exit is July 1, 2002, then the source documentation must also have July 1, 2002 as the exit date.
Two types of validation rules exist:
VALIDATION RULES
Validation Instruction: These instructions present the data elements, reference numbers, formats, element definitions, federal sources, state/grantee
sources, and validation rules needed to perform data element validation. The federal sources are the generic, federally recommended source
documentation. The "State/Grantee Sources" column can be used to enter state-specific versions of the federally approved documentation.
Trade Adjustment Assistance Data Element Validation Instructions
18
Most recent qualifying
separation
Friday, December 01, 2006
12
2
1
Date:
YYYYMMDD
Numeric:
1 = Yes, <= 180
days
2 = Yes, eligible
veteran
3 = Yes, other
eligible person
4 = No
Number Ref # Format
Veteran status
Data Element
Layoff lists or rapid response list
Determination of eligibility form
Notice of termination
Letter from employer
DD-214
Cross-match with Veterans Data
Cross-match with Wagner-Peyser
State MIS
Federal Sources
Trade Adjustment Assistance Data Element Validation Instructions
Record the most recent date of
separation from trade-impacted
employment that qualifies the
individual to receive benefits
and/or services under the Trade
Act. Use the YYYYMMDD format.
Record the code that indicates
whether the individual served in
the active U.S. military, naval, or
air service for a period less than or
equal to 180 days, and who was
discharged or released from such
service under conditions other than
dishonorable, or whether the
individual met the conditions
above for more than 180 days.
Definition
State/Grantee
Page 2 of 5
Match
Support
Instructions
3. Case Notes: Case notes refer to either paper or online statements by the case manager that identify a participant's status for a specific data element, the date
on which the information was obtained, and the case manager who obtained the information.
2. State MIS: Unless otherwise noted, state MIS refers to specific, detailed information that is stored in the state information system that supports an
element. An indicator, such as a checkmark on a computer screen, alone is not acceptable source documentation. For example, state MIS is
acceptable source documentation for date entered training. To be an acceptable source to validate this date, in addition to the date of first training, the
state MIS should have detailed information about the type of training and the organization that provided the training. This detailed information makes
valid source documentation and makes it unnecessary for such states to validate this data element in local offices.
1. Cross-Match: A cross-match requires validators to find detailed supporting evidence for the data element. An indicator or presence of an SSN in a
non-TAA database is not sufficient evidence. For example, veteran status can be determined by a cross-match with the veteran's database. It is not
sufficient to find that the sampled SSN is present in this database; validators must also find dates of service, separation, and discharge information.
For the most part, definition of a particular source is clear. States have, however, had questions about three sources -- Cross-Match, State Management
Information System (MIS), and Case Notes. Definitions for these elements are:
For most data elements, the validation instructions provide multiple forms of acceptable source documentation. If the state collects multiple sources for
the same data element and the sources conflict, the most reliable source should be used to determine if the element passes or fails. For example, for
Employed in first full quarter after exit (Ref# 11) UI wage records are more reliable than supplemental data sources. Validators should use their best
judgment when deciding the most reliable source to validate an element.
TYPES OF SOURCE DOCUMENTATION
8
25
26
Date completed, or
withdrew from, training
Training completed
Basic Trade Readjustment 34
Allowance (TRA)
Friday, December 01, 2006
7
24
Date entered training
6
5
4
21
Petition number
3
20
Numeric:
1 = Yes
2 = No
Numeric:
1= Yes,
completed
approved training
course
2= No, did not
complete
(withdrew)
Date:
YYYYMMDD
Date:
YYYYMMDD
Numeric:
Date:
YYYYMMDD
Number Ref # Format
Date of application
Data Element
State UI records of TRA checks
issued
Request for allowance
State MIS
State MIS
Case Notes about
communications with program
Applicant Statement
UI Records
Vendor training documentation in
case file
WIA Status Form
Vendor training documentation in
case file
WIA Status Form
Training plan
Vendor training documentation in
case file
case notes
WIA status forms
State MIS
attendance lists
UI records
Worker group certification in case
file
State MIS
UI Records
DOLETA Website
Determination of Eligibility form
Application in case file
State MIS
WIA registration form
Designation of Eligibility form
Federal Sources
Trade Adjustment Assistance Data Element Validation Instructions
Record whether the participant
received Basic Trade Readjustment
Allowances (TRA).
Record the appropriate code:
Record the date (using
YYYYMMDD format) when the
participant completed training or
withdrew permanently from training.
Record the date (using
YYYYMMDD format) when the
participant's approved training
began. NOTE: Trade Act-approved
training which begins under the
WIA dislocated worker program
may have a date entered training
that is earlier than the participant's
application date.
Record the petition number of the
certification which applies to the
individual's worker group. If there
is more than one petition number
(for example, certifications under
both the TAA and NAFTA-TAA
programs), record the petition
number of the program from which
the training is paid, unless a
waiver was issued. Do NOT
include any alphabetic suffix;
record the petition number only.
Record the date, using
YYYYMMDD format, on which the
individual first applied for Trade
Act services/benefits under the
applicable certification.
Definition
State/Grantee
Support
Support
Match
Match
Match
Match
Instructions
Page 3 of 5
42
44
49
52
53
Date of exit
Employed in first full
quarter after exit
Employed in third full
quarter after exit
Three quarters prior to
participation quarter
Two quarters prior to
participation quarter
Friday, December 01, 2006
37
14
13
12
11
10
9
Definition
Numeric:
00000.00
Numeric:
00000.00
Numeric:
1 = Yes
2 = No
Numeric:
1=Yes
2= No
Date:
YYYYMMDD
Case file documentation that
includes initial approval and
renewals at 30 day intervals
State UI records of TRA checks
Form from employment counselor
Federal Sources
UI Wage Records
WRIS
Federal Wages Database
Other state wage records
UI Wage Records
WRIS
Federal Wages Database
Other state wage records
Supplemental Data Sources as
defined in TEGL 7-99
State MIS
UI Wage Records
WRIS
UI Wage Records
WRIS
State MIS
Supplemental Data Sources as
defined in TEGL 7-99
Trade Adjustment Assistance Data Element Validation Instructions
Earnings in the second full quarter
prior to the quarter of
participation.
Earnings in the third full quarter
prior to the quarter of
participation.
Record whether the participant was
employed in the third full quarter
after exit.
Record whether the participant was
employed in the first full quarter
after exit.
Record the last date on which
Documentation in case file
the participant received TAA or
State MIS
partner funded services after 90 days WIA exit form
without a service. See TEGL 17-05
for more information
Numeric:
Record the code that indicates
1= Recall
the reason the participant
2 = Marketable
received a waiver from the training
3 = Retirement
requirement.
4 = Health
issued
5 = Enrollment
unavailable
6 = Training Not
Available
7 = Reason Unknown/
Served Prior to 2002
Number Ref # Format
Waiver from training
requirement
Data Element
State/Grantee
Page 4 of 5
Match; Validate blank fields
Match; Validate blank fields
Support; Validate blank
Support; Validate blank
Match
Support
Instructions
55
56
Wages second quarter
after exit quarter
Wages third quarter
after exit quarter
Friday, December 01, 2006
52
17
16
15
Numeric:
00000.00
Numeric:
00000.00
Numeric:
00000.00
Number Ref # Format
Wages first quarter
after exit quarter
Data Element
UI Wage Records
WRIS
Other state wage records
Federal Wage Databases
UI Wage Records
WRIS
Other state wage records
Federal Wage Databases
UI Wage Records
WRIS
Other state wage records
Federal Wage Databases
Federal Sources
Trade Adjustment Assistance Data Element Validation Instructions
Earnings in the third full quarter
after the participant has exited.
Earnings in the second full quarter
after the participant has exited.
Earnings in the first full quarter
after the participant has exited.
Definition
State/Grantee
Page 5 of 5
Match; Validate blank fields
Match; Validate blank fields
Match; Validate blank fields
Instructions
APPENDIX D
TAADV SOFTWARE INSTALLATION
This appendix provides instructions on how to install the TAADV software.
Task 1—Uninstall the Prior Version of TAADV
Before you install a new version of the TAA software, it is recommended that you
uninstall the existing version of the software. Please follow the normal process for
uninstalling software on your computer.
To uninstall the TAA software using Windows 2000 or Windows XP, select Start,
then Settings, then Control Panel. In the Control Panel, double click on
Add/Remove Programs. Click on TAA. Then, click on Change/Remove. Follow
the InstallShield instructions to remove the software.
Users may need to contact their system administrator to uninstall the application.
Task 2—Download Software
Before starting installation of the software, quit any other programs that may be
running.
Download the software to your PC from the Internet at
http://www.doleta.gov/performance/reporting/tools_datavalidation.cfm. Do not
download the software to your C:\Program Files folder as this is the default
installation location.
Click on “setup.exe”. The InstallShield Wizard window will pop up. Click on the
“Next” button. Complete the Customer Information window as necessary, and
click on “Next.” Users may need to contact their system administrator in order to
install the application.
Minimum system requirements:
Operating System - Microsoft Windows 2000 or better
RAM - 256 megabytes
Disk Space - 500 megabytes
Processor - 100 mhz Pentium III (or equivalent processor)
Suggested screen area - 800x600
D.2
Task 3—Restart Computer
After completing installation of the TAADV, reboot your computer before
continuing any further.
Task 4—Fix Database/Data Source Errors During Installation
For some users of Windows 2000, XP and NT, the data source may not be linked
to the correct database or to any databases, which prevents the application from
functioning.
To determine if the correct database is selected, select the Data Sources (ODBC) in
the Control Panel—Administrative Tools, User DSN tab. Check that the TAADV
is linked to the right database. To do this, double click on the TAADV Data
Source (ODBC) and check the path under the “database” section. If the
application was saved to the C:\ drive then the database should be in the
C:\Program Files\TAA folder. If the software was saved to another location, check
to make sure that the data source points to this other folder and database.
Some users of Windows 2000, XP, and NT may also need to set up the ODBC in
the System DSN tab under Administrative Tools. Users should try this solution if
the application freezes on the initial splash screen. This setup procedure is identical
to the procedure used for the User DSN tab.
If you are still experiencing difficulties accessing the software after checking the
data source, you should consult your system administrator.
D.3
Task 5—Manually Register Files
While installing the TAA data validation software, users may get an error message
stating that a file is not registered or cannot be found. In certain cases, users may
get this error message after the software has been installed instead of during the
installation process.
These error messages can be resolved by manually registering the specific files that
failed to register. First, users should check for the unregistered file on their hard
drive. The file should be located in C:\program files\TAA, in C:\winnt\system32 or
in a different directory if the software was not installed in the default directory. If
the file is on the hard drive, users should follow the steps outlined below to
manually register the file. If the file cannot be found, users should contact
[email protected].
To manually register the files, go to the Start menu and select Run. In the Run
box, in the Open field, type in the regsvr32 command in the following format:
regsvr32 "PathName"
where “Pathname” is the full location of the file including the file name given in
the error message. Type in the entire pathname and filename with quotes around it.
For example, users with Windows 2000 or XP would type the following in
response to an error message noting that the file crviewer.dll is not registered:
regsvr32 "c:\program files\TAA\crviewer.dll"
Windows NT users would type in:
regsvr32 "C:\winnt\system32\crviewer.dll"
D.4
After typing in the regsvr32 command click “OK”. This manually registers the file
that did not register during the batch installation process.
Users should see a regsvr32 message that the manual registration succeeded.
Repeat this process for each unregistered file.
D.5
Task 6—Database Setup for Multi-User Access
The following instructions apply to states that want to provide multi-user access to
a pre-loaded database.
Definitions:
In the instructions, the term local is used to describe when a change is made on the
user’s machine, and
the term central is used to describe when a change is made to a database not
located on the user’s machine.
Instructions:
1. One user loads the extract file to be validated to user’s local machine.
2. Compact the database (under the Utilities Menu) and close out the software.
3. Find the local TAA software directory (note the default directory is C:\Program
Files\TAA).
4. Find the TAA database called “TAA.mdb” (or the database into which the
extract was loaded); note that this database contains the extract file that was
imported in addition to the DV worksheets that will be completed for
validation.
5. Save the local TAA database to a central location where other users can access
it in order for them to individually complete the validation worksheets.
Typically, this location will be a drive on a network server.
Users who are completing the validation worksheets using the database on the
server must know where the database is stored and the name of the database. They
must also have the TAA software loaded locally on their machines (note that all of
the users will need to have the same version of the software).
D.6
APPENDIX E
TAADV SOFTWARE REFERENCE GUIDE
This appendix provides a condensed list of all of the software menus and functions
for quick reference.
A. FILE MENU
The first menu on the toolbar is called “File.” Options in this menu include:
1. New
Users can create a new database in which to load their extract file.
2. Open
Users can open an existing database.
3. Save As
Users can save the database that they are currently using under a new name, and
continue working with this new database.
4. Show Tips at Start-Up
Users can choose whether or not the tips appear at start-up.
5. Exit
Select this to exit the program.
E.2
B. IMPORT DATA MENU
The second menu on the toolbar is called “Import Data.” Options in this menu
include:
1.
Import From Extract File
Select this tab to import data into the application. See the source table record
layout for the appropriate data record format.
2.
View Latest Error Report
This function displays a report that lists records that were rejected after importing
a file.
3.
View Duplicates
This function displays a report that lists duplicates that the software identifies and
rejects when importing the extract file.
4.
Source Table Record Layout
This function displays the record layout used to develop the validation file. The
record layout is in the TAPR format with the addition of six fields. A copy of the
record layout can be found in Chapter III of this handbook.
5.
View Source Table
This function displays all of the records that were imported into the software.
E.3
C. CHANGE REPORTING OPTIONS MENU
The third menu on the toolbar is called “Change Reporting Options.” The only
option in this menu is:
1.
Change Reporting Options
This function opens an expanded version of the Sign In pop-up window, where the
user can change the fiscal year and the period start and end dates. The user can also
select a WIB Name, Office Name, and/or Case Manager filter to view subsets of
records and to generate substate reports for management purposes.
E.4
D. REPORT VALIDATION MENU
The fourth menu on the toolbar is called “Report Validation.” Options in this
menu include:
1. View Performance Outcome Groups
This function provides a window where you can see a summary of each
performance outcome group and totals of records and dollar amounts, where
applicable, for each group. By clicking on the arrow in the far left column next to
a performance outcome group, the user can view the detailed records in that group.
2. View Performance Report
This function displays a report that calculates the values for the three performance
measures for the TAA program. The report may be printed by clicking the printer
icon on the tool bar.
3. Export Performance Outcome Groups
This function allows users to export the performance outcome groups into
individual text files. These files are saved are saved in the c:\Program
Files\TAA\Participant Extract directory.
E.5
E. DATA ELEMENT VALIDATION MENU
The fifth menu on the toolbar is called “Data Validation.” Options in this menu
include:
1.
Edit Worksheets
a.
Detailed Summary of TAA Validation Worksheets
This window displays a list of the records sampled. Double click on the relevant
record to view the online version of the worksheet. Any worksheets shaded in
purple have been completely validated. To search for a record with specific data in
a field, right click on the field name that you wish to search, and fill out the Find
box.
b.
Online Worksheet
The online validation worksheet allows users to enter pass or fail for the data
elements for the sampled record. The worksheet is tabbed to allow the user to
easily navigate though all of the data elements.
2.
Print Worksheets
This function enables the user to print the worksheets in batch mode.
3.
Print Summary/Analytical Report
This function displays a report that is generated after the worksheets are
completed. The report calculates two types of error rates for each applicable data
element.
E.6
4.
Print Sampled Counts Report
This function displays a report showing the distribution of sampled cases by WIB
Name, Office Name, and Sampling Unit.
5.
Summary/Analytical Comments
This opens a data entry screen where users can enter in comments related to their
data validation results. The comments then appear on the Summary and Analytical
Report.
6.
Export Summary/Analytical Report
This function exports the Summary/Analytical Report into an XML file and opens
a web browser so that users can e-submit their validation results to ETA.
E.7
F. UTILITIES MENU
The sixth menu on the toolbar is called “Utilities.” This menu includes:
1. Compact Database
Eliminates the temporary memory storage by compacting the database. This
feature removes unnecessary files remaining in temporary memory storage after an
import has been completed, without deleting files required for analysis and
validation. In order for the software and database to function properly and
efficiently during the import of large files, the user should compact the database
prior to every import. Otherwise, errors may result even if the underlying source
file is properly formatted. These overload errors may show up in both the error
counter on the import screen and as database “overload” errors in the import.txt
file. This function is only applicable to Access users. Users should exit the
application after compacting and then reopen the application for further use.
E.8
G. WINDOW MENU
The seventh menu on the toolbar is called “Window.” Options in this menu
include:
1. Cascade
Realigns open windows to appear in a cascade from the top left corner of the
screen.
2. Tile Horizontal
Realigns open windows horizontally.
3. Tile Vertical
Realigns open windows vertically.
This menu also indicates the names of the open windows, with a check next to the
dominant window.
E.9
H. HELP MENU
The eighth menu on the toolbar is called “Help.” Options in this menu include:
1.
Help Contents
This feature is under development.
2.
Contacting Mathematica Policy Research, Inc.
This feature provides contact information for users who would like additional
assistance with installing or using the software.
Users should e-mail
[email protected], and specify the software version being used, the
specific question, and the user’s contact information.
3.
About
This feature provides the software version number and product development
information for the application.
E.10
File Type | application/pdf |
File Title | Microsoft Word - 1.TAAhandbook120406.doc |
Author | GRoemer |
File Modified | 2006-12-05 |
File Created | 2006-12-05 |