Suppstmnt_1205-0359_final 9 27 07

SUPPORTING STATEMENT

FOR

OMB Clearance of the Benefits, Timeliness and Quality (BTQ)

Review System

A.JUSTIFICATION

1. Circumstances Making the Collection of BTQ Information Necessary.

The Secretary of Labor has a legal responsibility under the Social Security Act (SSA) Title III, Section 303(a)(1), for reimbursing to State Workforce Agencies (SWAs) the necessary costs of proper and efficient administration of state unemployment insurance (UI) laws (Appendix A). The Secretary must establish a means of measuring SWAs' "proper and efficient administration" to certify payments to states. Among other duties, the Secretary must also ensure that state laws conform to Federal law, and that states comply with them, in order for subject employers within the state to be allowed to receive offset credit under the Federal Unemployment Tax Act.

In support of these responsibilities, SSA Title III, Section 303(a)(6) authorizes the Secretary to require of SWAs the:

"...making of such reports in such form and containing such information, as the Secretary of Labor may from time to time require, and compliance with such provisions as the Secretary of Labor may from time to time find necessary to assure the correctness and verification of such reports."

The nature of the UI system, as administered under state laws in conformity with Federal laws and regulations, results in differences among state laws, policies, and operating methods. At the same time, however, the 53 SWAs perform similar functions and produce certain results in common for which minimum performance criteria may be applied and by which their relative effectiveness may be appraised and compared for purposes of their own internal management as well as Department of Labor (DOL) oversight. Not withstanding the state differences, performance measures based on program outcomes are even more critical for assuring that the DOL’s statutory oversight responsibilities are carried out. Program operating information is presently collected mainly in four ways: (1) through UI required reports (UIRR), (2) through the Benefit Accuracy Measurement (BAM) Program, (3) the Tax Performance System (TPS), and (4) the Benefits Timeliness and Quality (BTQ) Program. (See Appendix A-4 for OMB Approval Numbers) The BTQ performance measures are necessary to ensure that the UI program is properly administered. Good administration leads to public confidence in the UI program. Conversely, poor administration leads to a poor public image, fostering a lack of confidence in the UI program. SWA staffs also need to know the criteria against which the performance of their agency will be evaluated. The BTQ component provides these measures.

2. How, By Whom, How Frequently, and For What Purpose the BTQ Information is Used.

The information collected under the BTQ system has a number of uses. It is one of the primary means used by UI Regional and National Office staff to assess state performance levels and to ensure that the Secretary's legal oversight responsibilities for determining the proper and efficient administration of the UI program are carried out. SWAs also use the BTQ performance measures for their internal program assessment with the goal of continuous performance improvement. The information is collected monthly and quarterly as required for the UIRR system and for UI Performs, the performance management system for the UI program.

The BTQ results are used to facilitate state compliance with the terms of UI administrative grants. SWAs annually prepare State Quality Service Plans (SQSPs) (OMB approval No. 1205-0132, expiring 06/30/2008), which contain required budget worksheets, corrective action plans, and state plan narratives that detail how the SWAs intend to improve deficient performance in specific program areas.

3. Consideration of Information Technology Used to Reduce Burden.

Almost all the data used in constructing the BTQ measures is produced in the states in the course of taking claims and paying benefits. Most BTQ measures are computed directly from required reports that are submitted electronically and allow SWAs to load data directly from files created on their computer systems. With the various BTQ promptness measures, most of the record keeping used in BTQ is already highly automated at the state level. In order to comply with the Government Paperwork Elimination Act, under BTQ, the quality assessments data are entered into the UIRR for electronic transmission, eliminating the need for handling, shipping and review of paper reports.

4. Efforts to Identify Duplication.

There is no duplication between the BTQ and any other data collection.

5. Impact of Collection of BTQ Information on a Substantial Number of Small Businesses or Other Small Entities.

The collection of information for the BTQ measures does not extend to small businesses or other small entities.

6. Consequences if the Collection is not Conducted or is Conducted Less Frequently.

If the collection of BTQ information is not conducted, DOL would have difficulty discharging its oversight responsibilities effectively and efficiently. The monthly and quarterly reporting system is necessary for producing a continuous, consistent database offering comparison of data from state to state showing seasonal and cyclical economic factors. Monthly and quarterly collection of information will permit all the current oversight functions that coincide with the annual Federal budget cycle. This allows analysis and use of data for publication of an annual evaluation of state compliance with existing laws, reviewing SWA performance against BTQ measures, and monitoring SQSPs and performance improvements resulting from corrective actions undertaken.

7. Special Circumstances That Require the Collection to be Conducted in a Manner Inconsistent with General Information Collection Guidelines in 5 CFR 1320.5.

The BTQ information collection is consistent with the criteria outlined in section 1320.5, General Information Collection Guidelines, with the exception of the requirement of monthly collection of information as explained above. However, monthly collection of information is consistent with existing OMB approved UIRR reports such as ETA 5130 (OMB approval no. 1205-0172, expiring 10/31/2007), which contains required budget worksheets and corrective action plans that detail how the State Agency intends to improve Benefit Appeals, and ETA 5159, Claims and Payment Activities (OMB approval no. 1205-0010, expiring 07/31/2009).

8. Consultation on the Information Collection with Persons Outside the Agency.

In accordance with the Paperwork Reduction Act of 1995, the public was given an opportunity to review and comment through the Federal Register process. The Federal Register Notice was published 07/27/2007, allowing comments until 09/25/2007 (Vol 72, no.144, July 27, 2007, pages 41355- 41357); a copy is attached. The Bureau of Labor Statistics (BLS) was also given an opportunity to provide comments, and the package was accepted by BLS in its entirety. No other comments were received for this data collection.

9. Payment/Gifts to Respondents

There are no payments to the respondents.

10. Confidentiality Provided Respondents.

To ensure confidentiality, BTQ data does not include claimant identifiers or claimant demographic data such as gender, race, ethnicity, date of birth, citizenship, or geographic location below the state level. Aggregate data are reported as the percentage of nonmonetary adjudications and lower authority appeals that receive passing scores in the quality review. Data are not reported for population demographic subgroups that could lead to the identification of individuals.

11. Sensitive Information.

Information contained in this report is not sensitive.

12. Estimates of Annualized Cost and the Hour Burden of Collection of Information.

1. Ongoing Annualized Costs

The estimated annual cost of BTQ in the states is $1,391,311. Salary costs are calculated using an estimate of $37.07 per hour for state employees for 37,532 burden hours. SWA travel estimates of $341,003, paid by DOL, are for state participation in the quality review of nonmonetary determinations.

2. Estimates of Burden of Collection of Information

Monthly Universe Measures: State Staff Hours per Year

ETA Report	Measure	Number of Respondents	Reports Per Year	Total Responses	Hrs. Per Resp.	Total Hrs/ Year
9050	First Payment Time Lapse, Core Measure	53	12	636	.5	318
9050	First Payment Time Lapse, Partial/Part Total Claims, Management Information Measure	53	12	636	.5	318
9050	First Payment Time Lapse, Workshare Claims, Management Information Measure	53	12	636	.5	318
9051	Continued Weeks Compensated Time Lapse, Management Information Measure	53	12	636	.5	318
9051	Continued Weeks Compensated Time Lapse, Partial Part/Total, Management Information Measure	53	12	636	.5	318
9051	Continued Weeks Compensated Time Lapse, Workshare, Management Information Measure	53	12	636	.5	318
9052	Nonmonetary Determinations Time Lapse, Detection Date, Core Measure	53	12	636	1.0	636
9054	Lower Authority Appeals Time Lapse, Management Information Measure	53	12	636	.5	318
9055	Lower Authority Appeals Case Aging, Core Measure	53	12	636	1.0	636
9054	Higher Authority Appeals Time Lapse, Management Information Measure	53	12	636	.5	318
9055	Higher Authority Appeals Case Aging, Core Measure	53	12	636	1.0	636
	Subtotal					4452

Quarterly Sample Review Measures: State Staff Hours per Year
ETA Report	Measure	Number of Respondents	Sampled Cases Reviewed Per Year	Total Cases Reviewed Per Year	Hrs. Per Resp.	Total Hrs/ Year
9056	Nonmonetary Determination Quality, Core Measure	29 Small States	240	6,960	1	6,960
9056	Nonmonetary Determination Quality, Core Measure	24 Large States	400	9,600	1	9,600
9057	Lower Authority Appeals Quality, Core Measure	47 Small States	80	3,760	3.5	13,160
9057	Lower Authority Appeals Quality, Core Measure	6 Large States	160	960	3.5	3,360
	Subtotal					33,080

Total burden hours for SWA BTQ activity each calendar year will equal:

4452 (monthly responses) + 33,080 (reviews of sampled cases) = 37,532 Total Staff Hours.

Continuing report generation costs are negligible for all measures except those based on samples. Most SWAs have automated the process of transferring report data from their mainframe computers to the SWAs' Sun ADP systems, which provide the interface to the UI database in the National Office.

13. Provide an Estimate of the Total Annual Cost Burden to the Respondents.

There are no other costs involved.

13. Estimates of Annualized Cost to the Federal Government and Respondent.

In the DOL National and Regional Offices, approximately twelve professional and six support positions, part-time at the average grade of GS-12, are assigned to BTQ duties. Federal UI professional and support positions devote an estimated 25 percent of their time to BTQ activity. This 25 percent of their time for BTQ activities includes both information collection and subsequent year-round analysis and technical assistance activity. The estimates of annualized cost to the Federal government are as follows:

Federal Salary (base) $300,452

Federal Travel 22,250

Total: $322,702

13. Reason for Changes in Burden.

During a review of UI Performs, the performance management system for UI, the ETA 9053 was eliminated; therefore, the total number of responses was decreased by 636 hours.

13. Information to be Published.

The BTQ measures results that are published as part of the UI Performs annual report will be posted to the DOL website. The general public can access the reports on the Internet at the DOL website. An Executive Summary of the report is attached.

13. Waiver to not Display the Expiration Date.

ETA will display the OMB control number, expiration date and disclosure statement in an ETA directive.

13. Exceptions.

There are no exceptions.

B.COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling

The growth of automated claims processing in SWAs enables the use of data from the BTQ measures to evaluate performance primarily through analysis of universes of individual UI transactions. There are three areas, however, where it is necessary to select random samples to assess performance:

• adjudication quality reviews of separation issues

• adjudication quality reviews of nonseparation issues

• lower authority appeals quality reviews.

The BTQ collection of information is based solely on review of existing working SWA agency records and does not involve surveying individuals, administering questionnaires or any other form of survey research. Samples are selected quarterly from records reported on the ETA 9052 and ETA 9054 reports, which are censuses of nonmonetary determinations and lower authority appeals, respectively.

Quality Review Universes and Samples

Universe Sizes from CY 2006 by State

Items to Be Reviewed	Range of Annual Universe*	Quarterly/Annual Sample Size
Adjudications (Separation Issues)	5,248 -514,817	30/120 or 50/200
Adjudications (Non-Separation Issues)	3,091 - 489,286	30/120 or 50/200
Total Adjudications (30 Small States)	12,091 - 91,179	60/240
Total Adjudications (22 Large States)	101,449 - 1,004,103	100/400
Lower Authority Appeals (45 small states)	785 - 39,333	20/80
Lower Authority Appeals (7 large states)	41,754 - 227,427	40/160

* Excluding Virgin Islands. Source for separation and nonseparation adjudications is the ETA 9052 report. Source for lower authority appeals is the ETA 9054 report.

Both the adjudications and appeals quality measures have been designated as UI Performs Core Measures. Core Measures have National minimum performance criteria established, as opposed to individual SWA-negotiated criteria. Accurate estimates of quality are thus extremely important, but must be balanced with the costs of conducting the reviews. For this reason, BTQ has implemented varying sample sizes for both adjudications and appeals quality reviews based on the size of activity in each SWA.

Adjudication quality samples are stratified into separation and nonseparation issues to allow for separate estimates of quality. For SWAs where the total population of nonmonetary adjudications reported on the ETA 9052 report equaled or exceeded 100,000 in the prior CY, the quarterly sample size is 50 separation issues and 50 nonseparation issues. For SWAs where the population was less than 100,000, the quarterly sample size is 30 separation issues and 30 nonseparation issues.

Lower Authority Appeals quality samples are not stratified. For SWAs where the total population of lower authority appeals reported on the ETA 9054 report equaled or exceeded 40,000 in the prior year, the quarterly sample size is 40 appeals decisions. For SWAs where the population was less than 40,000, the quarterly sample size is 20 appeals decisions.

SWAs can draw more than the minimum required quality review samples for adjudications, in order to obtain more precise information about how operational decisions impact quality.

Because these samples are drawn from universes of records maintained by SWAs, the response rates are 100% except for cases of lost documents. In CY 2006, 28 SWAs reported no nonmonetary adjudications with missing documentation. For the 24 SWAs reporting missing documentation, the number of adjudications missing documentation ranged from 1 to 21.

2. Information Collection Procedures

All SWAs are expected to maintain computer files containing all adjudications. Stratifying the samples by separation and nonseparation issues is done on the basis of the adjudication issue code, which is stored with every adjudication record. It is thus straightforward for SWAs to extract two files containing all separation and nonseparation adjudications where the notice date falls within the quarter being sampled. SWAs then either 1) randomize the file using a computer program and select the first n records, with n equal to the quarterly sample of nonmonetary adjudications or lower authority appeals, as discussed in the previous section; or 2) draw a systematic sample by calculating a sampling (skip) interval (universe divided by desired sample size), selecting a random number between 1 and the interval number for the initial selection and then selecting every nth record to produce a systematic a random sample.

Not all SWAs have computerized records of Lower Authority Appeals decisions. SWAs that maintain automated records of appeals decisions will draw the quarterly sample using the procedures (random file or systematic sampling method) described above. SWAs that do not maintain automated records of appeals decisions must use a manual systematic sampling approach by counting the total number of appeals decisions for the quarter, calculating the sampling interval and manually selecting the appropriate appeals decisions for review.

Minimum performance criteria for UI Performs Core Measures were published in UIPL 14-05, issued February 18, 2005. The nonmonetary quality standards are set at 75% of cases scoring 95 or more points. The lower authority appeals quality Core Measure minimum performance criterion is 80% of the cases scoring at least 85% of the potential points. Given the sample sizes and the target scores, the confidence intervals for the estimates of annual performance are as follows:

95 Percent Confidence Intervals for Estimated Percentages

of Adjudications or Appeals Passing Quality Reviews

Measurement	Small States (Quarterly / Annual)	Large States (Quarterly / Annual)
Adjudications: Separation Issues Nonseparation Issues Universe of All Adjudications	75% + 15.8 / + 7.8 75% + 15.8 / + 7.8 75% + 11.1 / + 5.5	75% + 12.1 / + 6.0 75% + 12.1 / + 6.0 75% + 8.5 / + 4.3
Lower Authority Appeals	85% + 18.0 / + 8.8	85% + 12.6 / + 6.2

Note: The confidence intervals for nonmonetary adjudications assume simple random samples are selected and that 75 percent of the sample determinations receive passing scores (>= 95 points). The confidence intervals for lower authority appeals assume simple random samples are selected and that 80 percent of the sample determinations receive passing scores (>= 85 percent of the potential points). The 75 percent and 80 percent values are the minimum performance criteria established for these measures under UI Performs. Confidence intervals are expressed as + percentage points. See Appendix B-1 for the statistical formulas used to estimate annual population parameters and sampling variability.

Example: For a small state sample (30 cases) of separation adjudications, let:

n = the number of separation adjudications sampled for the quarter.

m = the number of completed sample separation adjudications for the quarter.

x = the number of separation adjudications in the quarter that meet the quality criterion of >= 95 points.

the proportion of separation adjudications in the quarter that meet quality criterion.

Assuming that the sampling fraction, f = n / N is negligible and that all sample cases were evaluated (that is, m = n -- no cases could not be evaluated due to missing documentation and no cases failed to meet the definition for inclusion in the population), then:

= (.75 * .25) / 29

= .1875 / 29

= .006466

The 95% confidence interval is:

= 1.96 * .0804

= .1576 or 15.8 percentage points.

Operational definitions of the populations of nonmonetary determinations are provided in ETA Handbook 301, 5th ed., chapter II, and sampling instructions are provided in Appendix A of the handbook. Operational definitions of the populations of lower authority appeals and sampling instructions are provided in ETA Handbook 382, 2nd ed., section II.

Sampling frames are validated as part of UI Data Validation to insure that they include all of the records meeting the nonmonetary determinations and lower authority appeals operational definitions. The BTQ validation procedures are documented in the UI Data Validation Benefits Handbook, ET Handbook 361, Module 4.

3. Methods to Maximize Response Rates

The Department is aware of missing administrative records as a potential issue in BTQ sampling. The data collection instruments record the incidence of missing records. If more than 10 percent of records are missing the results are not accepted as valid and are not included in published reports. The Department will work with SWAs to correct administrative record control problems.

4. Tests of Procedures or Methods

The BTQ appeals and adjudications quality instruments are variations of instruments that have been in use for over twenty years. The revisions were subjected to a field test, which occurred in six SWAs over five quarters in 1993 and 1994. The data collection instrument for the ETA 9056 Nonmonetary Determination Quality Review is provided in ET Handbook 401, 4th ed., section V, chapter 6, p. 2. The data collection instrument for the ETA 9057 Lower Authority Appeals Quality Review is provided in ET Handbook 401, 4th ed., section V, chapter 7, pp. 2-3.

5. Individuals Consulted on Statistical Aspects of the Design

Andrew Spisak

Mathematical Statistician

Division of Performance Management

Office of Workforce Security

202-693-3196

[email protected]

William Borden

Mathematica Policy Research (MPR)

Princeton, New Jersey

609-275-2321

Appendix B-1

Equations for Population Parameter Estimates

The following formulas are used to produce the annual estimates of the proportions of nonmonetary adjudications and lower authority appeals meeting quality standards, based on the quarterly samples.

The following notation will be used:

H = the number of calendar quarters for which the estimate is

being made.

N_h = the number of adjudications or appeals in quarter h.

X_h = the number of adjudications or appeals in quarter h which meet the quality criteria.

P_h = X_h/N_h = the proportion of adjudications or appeals in quarter h which meet the quality criteria.

N_ = = total number of adjudications or appeals in the period.

X_ = = total number of adjudications or appeals which meet the quality criteria in the period.

The parameter to be estimated, P, is the proportion of adjudications or appeals that meet the quality criteria during the period. We wish to estimate:

P = X_/N_ =

Now let:

m_h = the number of completed sample adjudications or appeals for quarter h.

m_ = = total number of completed sample adjudications or appeals in the period.

x_h = the number of adjudications or appeals in quarter h which meet the quality criteria.

= proportion of adjudications or appeals in quarter h which meet the quality criteria.

If it is assumed that non-response is "at random", then

.

It follows that is unbiased for P.

Furthermore, as sampling is independent within each quarter (stratum), it follows that:

where f_h = m_h/N_h. The usual estimator for is

.

If f_h is negligible then

can be used for variance estimation.

Proportions for Subgroups

Samples of nonmonetary determinations and lower authority appeals may contain elements that do not meet the definition for inclusion in the population. These “foreign” elements are not included in the estimates of the proportion of the population meeting the quality criteria. The sample elements that meet the operational definition of the population constitute the sample subgroup for which population parameters are estimated.

Building on the notation above, for the k^th subgroup and the h^th week let:

N_hk = the number of adjudications or appeals.

X_hk = the number of adjudications or appeals which meet the quality criteria.

P_hk = X_hk/N_hk = the proportion of adjudications or appeals which meet the quality criteria.

Then for the k^th subgroup we have

N_k = = total number of adjudications or appeals in the quarter.

X_k = = total number of adjudications or appeals which meet the quality criteria in the quarter.

The parameter to be estimated, P_k, is the proportion of adjudications or appeals in subgroup k that meet the quality criteria during the quarter. Analogous to previous work, we can write

P_k = X_k/N_k = .

Note that neither X_k nor N_k is known. For the k^th subgroup, h^th quarter, let

m_hk = the number of completed sample adjudications or appeals for quarter h.

x_hk = the number of adjudications or appeals in quarter h which meet the quality criteria.

Assuming nonresponse is "at random":

is unbiased for X_k and

is unbiased for N_k.

The ratio estimator is approximately unbiased for P_k, and

where f_hk = m_hk/N_hk and θ_hk = N_hk/N_h.

Assuming that f_hk is negligible, an estimate for the variance is given by

where:

and

^{  .

Confidence Intervals

The 95% confidence interval for any estimate (u) is:

Coefficient of Variation

The coefficient of variation (cv) of an estimate u is:

16