Tax Performance System

CHAPTER II

GENERAL PROCEDURES
COMPONENTS

INTRODUCTION
PLANNING FOR REVIEW
CONDUCTING SYSTEMS REVIEWS
SAMPLING
CONCLUDING REVIEW

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

INTRODUCTION

INTRODUCTION
The TPS review is designed as a cost effective means to evaluate the State's UI tax operations. This
chapter describes the general procedures to be followed in conducting the TPS review, and each
subsequent chapter contains the specific data collection instruments to be used for the review of each
tax function. The following major tax functions will be reviewed:

Tax Functions Examined
Status Determination
Cashiering
Report Delinquency
Collections
Field Audit
Account Maintenance
Two different methodologies are provided for evaluating the State tax operation. They can be used
in concert with each other in order to comprehensively assess the strengths and weaknesses of each
tax function. The methodologies are:

Methodologies
1

Computed Measures

2

Program Reviews
Systems Reviews
Acceptance or Estimation Sample

II - 1

R 04/03

Figure II-1
TPS ASSESSMENT METHODS OVERVIEW
Program Reviews
Computed Measures

STATUS
DETERMINATION

New determinations within 90 days
New determinations within 180 days
Successor determinations within 90 days
Successor determinations within 180 days

FIELD AUDIT

ACCOUNT
MAINTENANCE

Accuracy of new determinations
Accuracy of successor determ.
Accuracy of inactiv./terminations

New determinations
Successor determinations
Inactiv./terminations

Employer remittances
(*Estimation Sample 500)

Timely reports
Reports secured within 90 days
Reports secured or resolved within 180
days

Accurate identification of
delinquent employers
Take all reasonable actions to
secure/resolve rpt del

Accounts with delinquent
reports

Timely payments
Uncollectible
Accounts receivable

Take all reasonable actions to
manage accounts receivable

Change in total wages
Contributory employers audited
Total wages audited (annualized)

Ensure that audits meet ESM
requirements

Completed audits

Accuracy of contrib.rpt processing
Accuracy of billings
Accuracy of credits/refunds
Accuracy of benefit charging
Accuracy of experience rating

Active contributory accounts
Contrib./reimb. accounts receivable
Accounts payable credits/refunds
Accounts due benefit charges
Employer experience ratings

CASHIERING

COLLECTIONS

Samples (60)
*Cashiering sample 500

Accuracy of employer remittance
processing
Accuracy of posting

NA

REPORT
DELINQUENCY

Systems Reviews

NA

Accounts receivable

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

INTRODUCTION

Computed Measures
Specific indicators have been developed for the Status Determination, Report Delinquency,
Collections, and Field Audit functions. Data elements for these indicators will be gathered from the
current system used by states to electronically report quarterly data to the U.S. Department of Labor.
These measures will be automatically transformed into indicators of timeliness and completeness by
the TPS ADP system and provided to the TPS reviewer to aid in the final evaluation of state tax
operations.
Computed Measures

Status Determination
.
.
.
.

% New determinations made within 90 days
% New Determinations made within 180 days
% Successor Determinations made within 90 days
% Successor Determinations made within 180 days

Report Delinquency (contrib. & reimb.)
. % Timely Reports
. % Reports Secured within 90 days
. % Reports resolved within 180 days
Collections (contrib. & reimb.)
. % Timely Payments
. % Receivables declared uncollectible
. % Accounts Receivable
Field Audit
. % Total wage change
. % Contributory employers audited
. % Total wages audited (annualized)

II - 5

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

INTRODUCTION

The timeliness and completeness findings from these indicators should be presented along with the
findings about accuracy from program reviews when evaluating the quality of a particular tax
function in the Annual Report.

Program Reviews

The Program Review methodology is designed to alert the TPS reviewer to the
tax functions that are producing inaccurate, untimely, or incomplete outputs.
This is done as a two-fold process: Systems Review and Acceptance Sampling.

Systems Review

To ensure accurate and timely tax operations, each state has built in various
internal controls and quality assurance systems such as edits and checks,
reviews of completed work and audit trails. However, such systems are not
absolute. The systems themselves could be flawed, or stated policy may not
always be followed. Therefore, the TPS review begins with an extensive
examination of the state's controls, verifying their existence and use,
documenting areas of potential "risk" where controls are weak or nonexistent.

Acceptance
Sampling

To confirm that state controls are producing the desired outputs
(such as accurately determining the status of potential employers, properly
resolving delinquent reports, and following state procedures when collecting
accounts receivable), small "acceptance" or "discovery" samples are
examined for each tax function. Due to the small number of samples, this
test is intended simply to signal potential problem areas in those tax functions
where 3 or more cases are found to fail. It indicates - with a 90% confidence
level - that the particular tax function has an error rate of 8.8% or more. For
more details or to determine a specific level of error, the reviewer may draw
a larger sample (see Appendix A).

II - 6

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

INTRODUCTION

Review Steps
To conduct the TPS Review, the reviewer will need to complete four major steps: Plan for the
Reviews, Conduct Systems Reviews, Acceptance Sampling, and Conclude Reviews (see Figure II2). The following sections of this chapter provide directions for the review steps. Chapters three
through eight provide TPS data collection instruments for the review of each major tax function.

II - 7

R 04/03

Figure II-2

OVERVIEW OF REVIEW STEPS
I.
a.
b.
c.
d.
e.

PLANNING FOR REVIEWS

Review All TPS Materials
Ensure that Records are Retained
Schedule and Conduct Introductory Meetings
Lay Groundwork for Reviews and Computed Measures
Develop Workplans for Reviews

II.

CONDUCTING SYSTEMS REVIEWS
a.
b.
c.
d.
e.

III.

ACCEPTANCE SAMPLING
a.
b.
c.
d.
e.
f.
g.

IV.

Begin Reviews
Gather Information
Verify the Source of Information
Complete the Systems Review Forms
Complete the Program Review Chart (preliminary)

Select Samples
Replace Missing Cases (if needed)
Review Samples
Post Coding Sheets
Prepare Preliminary Findings
Select and Examine Expanded Samples (if needed)
Sampling by Exception

CONCLUDING REVIEWS,
a.
b.
c.
d.
e.

Assess Computed Measures' Findings
Complete Program Review Chart
Prepare Draft Report
Conduct Exit Interview
Prepare Annual Report

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

PLANNING FOR REVIEWS

I. PLANNING FOR REVIEWS

a. Review All TPS Materials
To complete the Program Review, the reviewer must be thoroughly familiar with the TPS review
procedures. Knowing the required review steps of Computed Measures, Systems Review, and
Acceptance Sampling will allow the reviewer to conduct all necessary activities without undue
delays. Understanding the review questions is also essential to building reviewer credibility. If the
reviewer is not knowledgeable, respondents may think their time is being wasted and be less
cooperative.
The reviewer should thoroughly examine the materials in this handbook and other TPS background
materials before contacting the state UI tax staff regarding the review. If there are any questions,
please call the Regional TPS staff for clarification.
b. Ensure that Records are Retained
Upon familiarization with the TPS design, the reviewer must ensure that the state is retaining the
necessary records from the specific tax functions to such a degree as to allow all Acceptance Sample
questions to be answered (e.g., paper copies, imaging, microfiche, back up discs, etc). If employers
are registering and reporting electronically, or entering data directly into the state’s automated
system, the exact information entered must be recorded by the state, along with an electronic audit
trail including the entry’s date and identity of the employer or representative entering the data.
For example, in the Status Determination chapter an Acceptance Sample question asks if, at the time
of the TPS review, evidence exists the account had been correctly set up. In this instance to
properly answer the question the state must have retained sufficient documentation for the reviewer
to make that determination, whether examining the registration form or a printout of an electronic
data screen which captured the original information entered by the employer. Another question
asks if the employer provided all the information that is material to the Inactivated/Terminated
Determination. For the reviewer to properly answer this question, documentation authorizing the
termination must exist. Such documentation may be found in the form of a letter for the termination
from the employer, or a report from state field staff indicating the business is closed, or on a
computer listing of accounts terminated after the required quarters have lapsed with no reported
wages.

II - 9

R 11/07

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

PLANNING FOR REVIEWS

It is the responsibility of the TPS reviewer to become knowledgeable about the information
requested in the Acceptance Samples and ensure if the State is not currently retaining the
information that arrangements are made to do so. In each chapter's Acceptance Sample instructions,
information is provided that will also assist the reviewer in clarifying what work product must be
retained by the State.
At a minimum, documentation must be recorded and maintained until the TPS Annual Report has
been reviewed and approved by the Regional Office. The records can be retained in their original
hard copy form, on microfiche or any other means that provides sufficient detail; or the State must
maintain an audit trail which permits the reviewer to follow the flow of the work in complete enough
detail so as to allow the Acceptance Sample to be answered in full.

c. Schedule and Conduct Introductory Meetings
The next step of the TPS review is to meet with the UI Director and the Tax Chief to ensure
management support for scheduling and conducting the review. In addition, the State administrator
who will deal with the reviewer in terms of TPS findings and recommendations must be identified.
A meeting should be scheduled with all appropriate personnel including unit supervisors and ADP
staff to discuss the review, to request necessary materials, procedures, charts, etc. and to ensure that
data processing staff are informed of their involvement in the creation of sampling universes and the
processing of computed measures.
At both administrative and tax staff meetings the TPS reviewer must explain the purpose and content
of the TPS review. It should be explained that the primary purpose of the review is to produce
information for the State to use to improve the quality of their revenue operation; and that the
secondary purpose is to make a formal judgment about whether reasonable assurance of quality
exists in each of the functional areas.

II - 10

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

PLANNING FOR REVIEWS

Other items to explain include:
the tax function subjected to review
the role of Computed Measures
the role of System Reviews, Acceptance Samples and
Expanded Samples
the steps of the review
the highly detailed nature of the review including the
need to identify information sources to verify findings

Other activities to be covered at the initial meeting with unit supervisors are:
Providing copies of all review documents to the units involved in the
review. This includes the individual tax functions (i.e., the Status
unit, Cashiering, Delinquency unit, Field operations, and
Accounting), as well as the Data Processing section.
Drafting a Flow Chart of the Tax Operation to serve as a road map to
the TPS Program Review. It should show units responsible for each
function and subfunction which will be reviewed and should identify
the individuals to contact about each. This will also guide the
reviewer in scheduling interviews, and clarify the operational flow of
the tax functions.
Arranging to obtain materials describing the State's revenue
procedures and rules. These materials should include UI Tax laws
and regulations, manuals, and other written descriptions of
organization and procedures.

II - 11

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

d.

CHAPTER TWO

PLANNING FOR REVIEWS

Lay the Groundwork for Reviews and Computed Measures

The first objective of the reviewer is to learn enough about the applicable laws, rules and regulations
of the State tax operation to effectively plan and administer the TPS review. The reviewer will
complete the following tasks as part of the preparation:

Examine the data elements for Computed Measures. The reviewer
must ensure that they are being gathered and that Federal definitions
are being interpreted and reported as intended.
Complete or update the Organizational Chart. The TPS review will
examine six tax functions in every State, but each State's tax agency
is organized differently. Some agencies may call these functions by
different names or have parts of a single function managed by many
different units. By understanding who does what, the reviewers will
become familiar with the specific structure of the State's Revenue
organization prior to conducting the review.
Listing the staff to contact for information on each TPS subfunction
will provide the reviewer with an understanding of the structure of
revenue operations.
Assemble and review the materials including State's organization
chart, laws, regulations, manuals and procedures. Many questions in
the Program Review materials ask the reviewer to determine whether
State procedures ensure that staff are correctly applying State UI tax
laws and regulations. Questions on Recorded Information &
Instructions ask whether such documentation reflect accurately and
completely the current laws and regulations.

II - 12

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

PLANNING FOR REVIEWS

To prepare to answer these questions the reviewer should examine
State UI tax laws, regulations and other written policies. These
materials may be lengthy. If the reviewer is very familiar with the
TPS review instruments it should be possible to identify those
sections of State law and regulations which are most relevant. The
reviewer might develop a reference sheet prior to conducting the
review which could note key features of the State law, citations, and
dates of enactment.
If an audit of the State has been recently performed by groups or
firms outside the State tax area, the reviewer should examine the
findings and the State's response to those findings. Information from
the findings can aid the reviewer in developing a comprehensive
understanding of the State's tax operation. Since audits are financial
in nature, their focus is likely to be different from that of the TPS
review. Therefore, the existence of a recent audit will not replace an
TPS review.
During the preparation for this review, the reviewer should determine if there might be any questions
in the Acceptance Sample Questionnaire which have no material effect on the current or future
payment of UI tax.

If such questions are identified in the Acceptance Samples, the State will need to coordinate with the
Regional representatives to obtain necessary approval to enter a code of "Not Applicable" one time
for the entire column deemed not material. Regions will produce a brief memorandum attesting that
due to State regulations, a particular question is Not Applicable. This memo will be attached as
part of the State's Annual Report.

II - 13

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

PLANNING FOR REVIEWS

e. Develop Workplan for Reviews and Schedule Interview
A central part of the preparation and workplan development will be to schedule all
interviews, Acceptance Sample reviews, and meetings with agency staff. The
reviewer should develop a workplan which lists dates of review activities. The
workplan should include starting dates as well as target completion dates for review
of Computed Measures findings, Systems Reviews, Acceptance Sampling and
completion of the Annual Report.

During the implementation of TPS, an integral part of the preparation and workplan is the
examination of the Acceptance Sample universe time frames and required identification dates to
determine whether each universe is to be identified by the "build as you go", or the "identify after
the fact" method (see Appendix A).
The workplan should ensure that all required review steps are anticipated, that data processing is in
simultaneous progression, that the State tax staff know when to expect review activities, and that all
parties know whether the review is on schedule.
The contact person for each tax function should be informed of progress and findings so that no
misunderstandings arise, and be made aware of any problem areas in the units as soon as possible.
In States where tax functions are spread out over many different locations, the reviewer should
factor this into planning. Ideally, all locations should be examined each year, but if this is not
possible, the reviewer should plan to visit a number of different locations each year, so that by the
end of the four year Systems Review cycle, each location has been visited and evaluated.
The reviewer will submit a workplan for each upcoming year to the RO. The RO will require
progress reports to ensure that projected work is being completed on schedule, to the extent
possible, and that the TPS Annual Report will be completed on time.

(Figure II-3 displays anticipated scheduling of Acceptance Sampling.)

II - 14

R 04/03

Figure II-3
Figure II-3, done on "Pagemaker," will be inserted on this page. It was originally recorded as
C:\document\PMDOCS\TPS\exhib2-5.PM5, but this filename has been changed and Bob Timms has
the correct filename.

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

SYSTEMS REVIEWS

II. CONDUCT SYSTEMS REVIEWS

a. Begin Systems Review
The Systems Review is an organized assessment of each State's internal controls or quality assurance
systems. The presence of these controls should ensure that the State's UI revenue transactions are
processed accurately.

A system of "internal controls" is typically based on methods and policies
designed to prevent fraud, minimize errors, promote operational efficiency,
and achieve compliance with established policies.

A "quality assurance review" system is a periodic review, generally
conducted at a point in the process where errors are likely to occur. The
review may be performed by an individual or team within or outside the unit,
to measure the performance of a function and make recommendations for
improvements where warranted.

The internal controls included in the TPS Systems Review are based on the standards set for auditing
which were published by the General Accounting Office (GAO/AIMD-00-21.3.1, Nov 1999). In
each chapter of this handbook, internal controls will be modified as they relate to each particular tax
function. Universal criteria are described below with added clarification.

II - 17

R 07/06

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

SYSTEMS REVIEWS

States should have the following types of internal controls:
Recorded Information and Instructions
Training Systems
Recording of Transactions and Events
Execution by Authorized Individuals
Systems to Assure Execution of Events
Review of Completed Work

Recorded
Instructions

The State establishes requirements, rules, and procedures to implement
laws and regulations. Information about systems and procedures which guide how
work is to be performed should be clearly documented and readily available for
examination. Relevant information should be available to the appropriate staff
both at the central and field offices. Recorded information and instructions should
be sufficient or complete enough to ensure that personnel can learn and
understand their jobs and perform their duties properly. Instructions should also
be up-to-date, reflecting current procedures or laws

Recorded information may include manuals, handbooks, desk aids, computer help
screens, training guides, organized collections of procedures or policies, or other
readily accessible instructions which can help staff do their work correctly.
Instructions will normally include both general information such as compilations of
relevant laws and regulations, as well as detailed instructions for carrying out individual jobs in the
agency. This means that reviewers may need to look in many places to examine all relevant
instructions.

II - 18

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES
Training Systems

CHAPTER TWO

SYSTEMS REVIEWS

Managers and employees responsible for key decisions should possess and
maintain a level of expertise which enables them to accomplish their
assigned duties. Training systems should be sufficient to ensure that
personnel understand and perform their duties properly. (New employees
should have some form of training, and experienced employees also
benefit from refresher courses.)

When reviewing training systems, reviewers must look for formal training
procedures (e.g., the training is conducted using an established schedule
and using set guidelines to make judgments about the quality of work being produced). There
should be procedures for identifying general and specific training needs and for delivering
training as needed.

Recording of
Transactions and
Events

Transactions and other significant events are to be recorded and
properly classified. The entire life cycle of a transaction or
event should be promptly recorded. This includes recording the
action which initiated the transaction(s), recording the transaction(s)
that took place and recording the end result.
Whether systems are manual or automated, audit trails are necessary
so that original information is recorded and retained. Hard copies,
microfiches or imaging of original information can provide audit
trails. For employers filing or reporting electronically, on disk or
tape, a copy of the reported information loaded into the employers’
account/file would be considered an “original” document and provide
an audit trail. Subsequent changes or deletions to the file must also
be recorded. If the reviewer comes across a system which simply
overlays new information over the original, without any apparent
provision for an audit trail, the DP unit should be contacted to locate
back up files such as tapes, disc, etc. of the computer program runs.
The DP or Internal Security units should be able to explain how
original information is recreated (short of running every back up
tape), and the method used to catalog information so that the reviewer
can be assured he or she is looking at original, versus adjusted,
information.

II - 19

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

SYSTEMS REVIEWS

Reviewers will determine whether State audit trail requirements
provide for the classification and prompt recording of all significant
events. The reviewer will also need to determine whether records are
readily available to be used as needed for agency operation and
management review.

Execution by
Authorized
Individuals

Transactions and other significant events are to be executed
only by persons acting within the scope of their authority. Only
authorized individuals have access to and accountability for resources
such as employer remittances.
Reviewers will determine whether State requirements provide for the
execution of significant events by only authorized individuals.

Systems to Assure
Execution of Events

Systems and controls such as monitoring procedures, exception
identifications, checks and balances, reconciliations and edits should
be routine in operational procedures. Automated or manual exercise
of these controls as an integral part of program operations assures
accurate and timely execution of transactions and events.
Reviewers should determine if such program management systems
have been built into the State revenue operation.

Review of
Completed Work

Qualified and continuous review is to be provided to ensure
that objectives which are material to the proper payments of tax are
achieved. The following tasks are involved:
Systematic review of assigned work on a regular, ongoing basis.
Systematic review to determine whether systems and procedures
are working as intended.

II - 20

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

SYSTEMS REVIEWS

Review of completed work can take many forms. It may be traditional
supervision where a manager reviews and approves the work outputs of
immediate subordinates. For some functions, such as cashiering, supervisory
review may take the form of checking to assure that reconciliations are being
performed. Other agencies may use peer review or quality assurance
techniques where representative samples of the work of a tax unit are
periodically reviewed, and based on the review, new procedures, training or
assignments are implemented to improve quality.

Not all tax functions are similarly affected by each internal control. Completing the TPS Systems
Review questions for some internal controls like Recorded Information and Instructions, and
Training may be repetitive in States where the same people (e.g., Technical Support Services or
Training Section) provide different training for various units; or where there is only one manual,
handbook, etc. for the entire tax operation.
Most of the questions in the Systems Review guide are evaluative. If the reviewer can not confirm
that a particular control exists and that no other compensating control exists, a weakness is presumed
to exist in the system and the State is judged to be at "risk". Some Systems Review questions are
non-evaluative and for informational purposes only. Lack of these controls does not put State at
risk. These informational questions, identified by an asterisk (*), are planned to be used to provide
technical assistance for States in their various tax functions.
Note that if a State elects to operate with a weakness/risk in its system and no major changes have
been put in place, the reviewer does not have to perform another Systems Review on that tax
function the following year. However, that tax function will be deemed at “Risk” until the problem
is resolved.
While it is important to identify areas of strengths and weaknesses for program improvement - the
final evaluation is based on the Acceptance Sample findings, not the Systems Review. The
design of Program Review focuses on the downstream effect -if there is no material effect on current
or future payment of UI tax (as evidenced by the sample results), then the State has reasonable
assurance of a quality tax operation.

II - 21

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

SYSTEMS REVIEWS

b. Gather Information
The Systems Review is directed and organized by a set of fact finding questions and
narratives. They are not intended to be "interview instruments" where the contents
of a dialogue between the reviewer and State staff are simply recorded. Rather, the
questions and narratives should serve as a vehicle for reviewers to record and verify
information from many sources about the systems and controls States have installed
to assure accuracy and timeliness in their tax operations. The reviewer will use and
probe as many sources as necessary to come to conclusions as to the proper answer
to each question.

A typical Systems Review will require the reviewer to gather information from three sources:
Examinations of manuals, handbooks, laws and other
documentation
Interviews with State staff
Direct observation

The reviewer will need to use all of these sources to complete the Systems Review, and in most
cases will need to go back and forth between the sources to fill in gaps. For example, a reviewer may
have an idea of the responsibilities of an individual staff member based upon a review of the desk
procedures for the person's job. Later, when interviewing the person the reviewer may hear a
different description of the responsibilities. By returning to the documentation the reviewer can
clarify the answers to the questions and identify additional questions which need to be asked to
avoid any confusion.

II - 22

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

SYSTEMS REVIEWS

Examination of Documentation. The least intrusive means of data collection
is review of documentation. By examining materials describing agency
practices the reviewer can obtain preliminary answers to Systems Review
questions without bothering agency staff or interrupting their work. More
importantly, when it does come time to interview staff, the reviewer will get
more accurate answers if he or she is credible.
The reviewer must be knowledgeable about both the TPS process and agency
procedures to assure the respondents that their time is not being wasted.
Review of documentation will help the reviewer become familiar with
agency procedures and terminology.

Interviewing Staff. A major source of information for the Systems Review
will be discussions with State staff who are familiar with the tax function
being reviewed.
While the questions in this handbook are a start, they should not be the end.
The reviewers will need to ask additional questions to be sure they fully
understand the responses. They also need, in most instances, to speak with
several staff persons to get full answers to all questions. Additionally, they
may need to come back to some questions later if they cannot verify the
response by examining documentation or by directly observing the system at
work.

Direct Observation. The review of documentation and interviews with staff
will leave the reviewer with an understanding of the way systems
"are supposed to" operate, or the way administrators think they are operating.
It is the reviewers' responsibility to assure themselves, through direct
observation, that the systems actually operate that way. Any number of
methods exist to observe a process, including: (1) reviewing reports,
calculator tapes and other evidence of a procedure, (2) observing a process
on-site, or (3) checking for the presence of a procedure while reviewing the
Acceptance Sample.

II - 23

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

SYSTEMS REVIEWS

All Systems Review guides begin with a form to list people and documents that have been reviewed
(Interview Sheet). This list is to be kept up to date as the review proceeds. The list will help other
reviewers in subsequent years in conducting their review by serving as a recorded register of
information sources used in support of the review effort. (It should be part of the TPS workpapers,
but does not have to be submitted with the Annual Report.)

c. Verify the Information Obtained
The reviewer is expected to document the evidence obtained to support
each "Yes" answer in the Systems Review checklist. Informational
questions (marked with an asterisk) do not require any verification since
they are not evaluative. This evidence will be recorded for each question
or clusters of related Systems Review questions as the "Verification
Source", often noted on the forms simply as VS. The Verification
Source should be the most direct evidence to support the answer recorded. Only when no
other source is available will the verification source be solely the response from an interview.
In completing the Systems Review, verification of some answers may only be possible after
examining a sample of outputs or after completion of the Acceptance Sample (e.g., to answer
whether certain actions taken by field audit staff are being documented in their audits). The Systems
Review will be in final form only after completion of Acceptance Sampling. Following are several
examples of appropriate review activities and verification sources for specific areas of inquiry:
Question: Does the State have recorded information and instructions to assist employees
performing Status functions in each category in accordance with State laws and written
policies?
Review Activities: Read the manual, handbook, or procedures that were assigned to
employee; interview staff to confirm use, etc.
Verification Source: Record the precise name and section or page number of
documents containing information and instructions for making status determinations.

II - 24

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

SYSTEMS REVIEWS

Question: Does the State use Employer account number edits (e.g., hash totals, check
digits) to assure accurate processing of employer contribution reports?
Review Activities: Read recorded instructions; interview staff to confirm or
modify knowledge of how system works; and observe computer edits while in
process for inputting reports or review system rejects.
Verification Source: Record the procedure or evidence observed (e.g., observing
actual inputting of contribution reports using hash totals or check digits, or
checking system edit rejects).

Question: Is there a systematic review of the accuracy of new employer status
determinations? If yes, what type of review? (e.g., supervisory, peer, quality review,
etc.)
Review Activities: Review procedural manual; interview supervisor and
employees; and use Acceptance Sample to look for initials on forms or other
evidence that supervisory review occurred. If Quality Review (QR) System is
said to exist, review the procedures, interview QR supervisor and employees, and
examine the evidence that review occurred or observe actual test run in process.

Verification Source: Record the procedure or evidence observed (e.g.,
supervisor's initials on forms, or observation of actual Quality Review in process).

II - 25

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

SYSTEMS REVIEWS

Question: Does the State have a means to assure prompt deposit of checks that must be
removed from normal processing (suspense account/exception file) due to problems (i.e.,
reports received from new employers whose liability has not yet been determined, or
reports with no employer account number, etc.)?
Verification Test: Review procedures State uses and select several employer
remittance items at random with unidentified employer account numbers from
their suspense account or exception file at beginning of the quarter, and at end of
quarter check to see what disposition has been made of items selected for the
sample.
Verification Source: Record suspense account procedures used by the State and
note the findings of the verification test conducted.

Question: Is there an automated system indicator to identify accounts that are
delinquent?
If yes, is a system check performed every time a program is changed?
Verification Test: Interview programming staff as to the frequency of program
changes. (These may be infrequent.) If programming changes are occurring,
observe system checks to verify accuracy. If no programming changes are
occurring, interview programming staff on the process that would be followed to
verify changes are made properly.
Verification Source: Record observations if programming changes have
occurred. Document interviews with programming staff if no changes have
occurred.

Verification Sources should be listed in detail for each question. If a particular control or quality
assurance measure can not be verified (and no compensating control can be identified), the
answer to that Systems Review question will be "No", and the reviewer will have identified a
"risk" in that tax function. It should be clearly noted on the Systems Review coding sheet and
referenced in the comments of the Annual Report.

II - 26

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

SYSTEMS REVIEWS

II - 27

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

SYSTEMS REVIEWS

d. Complete the Systems Review Forms
For each tax function, System Review forms are provided in this handbook. They consist of the
review form, coding sheets, and narratives. Several common features of the forms are described
below.

Review Form. Most questions on the form are answered either Yes
or No; however, some questions request a particular number or a
percentage. The reviewer should record the correct answer to each
question based upon review of all sources of information.
Some questions on the form provide for "A" (Not Applicable), which
should be used only when appropriate. An example of using an "A" could
be the answer to a Review question in Account Maintenance which states,
"Does the State apply a tolerance level for differences between amount
due and amount received?", when the State does not use a tolerance level
for such discrepancies.
Some questions ask the reviewer to explain or describe something. Space
is provided to write answers out in their entirety.
Questions marked with an asterisk (*) are for information only and are
non-evaluative. (NOTE: a portion of a question may be asterisked,
but other, sub components of the same question such as
"a.","b.","c.", etc. may not be asterisked. In such a circumstance,
only the asterisked portion is non evaluative.) Informational questions
in the Systems Review are not evaluative and do not require verification
or explanation of "No" responses.

Coding Sheet. Data entry procedures described in a separate handbook.

II - 28

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

SYSTEMS REVIEWS

Narratives. The System Review forms provide a narrative section
following each set of internal control questions to explain any "No",

"NA" or "Other" responses or exceptions that the reviewer thinks warrant
further details. Should there have been a Compensating Control that has
been identified, verified, and authorized by the Regional staff as an
appropriate control, it must also be described here.
Additional space at the end of each tax function's Systems Review is
provided to describe any exemplary procedures or practices used in a
particular tax function. If additional controls were identified beyond those
mentioned in the TPS review, they are to be described as well, and this
form is to be sent to the Regional Office who will transmit it to the
National Office at the end of the Program Review as technical assistance.

e. Complete the Program Review Chart (preliminary)
The Program Review Chart serves to compile the results of the Systems Reviews and
Acceptance Sampling. Upon completion of the Systems Review, enter the preliminary findings
on the Program Review Chart (Figure II-4). There are three possible entries for the Systems
Review findings - the State's system of internal controls could be: (C) Complete, (R) Risk
identified, or (O) Other Compensating Control identified. (Note: Later, after completion of the
Acceptance Sampling, if any case has been found to be unacceptable, yet the System review was
coded as "Complete", the reviewer may need to review the systems again to resolve the
inconsistency between the Systems Review and the Acceptance Sample findings. Such
additional review may yield some revisions to the Program Review Chart to identify the
particular systemic weakness which caused the case to fail.)

II - 29

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

SYSTEMS REVIEWS

Complete

For each tax function, record "C" if all internal controls and quality
assurance systems listed in the Systems Review were in place. This means that
all evaluative questions were verified (VS) and answered "Yes", except for
questions marked "Other"; and questions which are marked with an asterisk
(because they are non-evaluative).

Risk

If any "No" answers appear in the Systems Reviews, it identifies a potential risk
or weakness in that area of State controls. The reviewer must enter an "R" to
signal the area of risk. The actual significance of the problem may not be evident
until a sample of outputs is examined during the Acceptance Sampling phase of
review. If any sample cases are coded as having "failed", the reviewer can
quickly scan the Program Review Chart for the presence of an "R" to see what
may have caused the problem.

Other

If a "No" answer is entered for a particular control, but the State has an "Other"
control thought to compensate for this weakness or risk, the reviewer must
examine the control, verify its existence (VS), and describe it in the spaces
provided. Regional staff must agree that the control adequately substitutes for the
missing control. (Acceptance Sampling results should aid in this decision.)
An example for using "other" might be a State which does not reconcile total
benefits charged with benefits paid to ensure accuracy of their charges.
Normally, this would result in a "Risk" in their system of internal controls.
However, a legitimate compensating control could be the State's system of
sampling each potential charge allocation scenario on a random basis to confirm
accuracy. TPS's Acceptance Sampling would confirm the effectiveness of the
State's control and no "risk" would be assigned. An "O" is to be entered after the
final judgement is made by the Regional Office that this Compensating Control is
adequate.

II - 30

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

SYSTEMS REVIEWS

II - 31

R 04/03

State :
Review Period :

Figure II-4

PROGRAM REVIEW FINDINGS
Sampling

Systems Review Verified Controls for:

Tax Function
Recorded
Instructions

Training

Recording
of Events

Authorized
Individuals

Execution
of Events

Review of
Work

EXCEPTIONS

COMPLETE

Contr

Contr

Reimb

Reimb

*

STATUS DETERMINATION
New
Successor
Inact/Term.
CASHIERING
REPORT DELINQUENCY
COLLECTIONS
FIELD AUDIT
ACCOUNT MAINTENANCE
Contrib Rpt Processing
Employer Debits/Billings
Employer Credits/Refunds
Employer Charging
Employer Tax Rates

SYSTEMS
REVIEW:

C = All controls verified as present
SAMPLING EXCEPTIONS: S = Size of universe too small
SAMPLING COMPLETE: P = Passed
R = Risk, 1 or more controls missing
I = Invalid universe
F/# = Failed/# of Cases Failing
O = Other control compensated for
E = Exemption, tempo (Exp Rate only)
D = Discarded sample
missing control (requires Regional approval)
FM = Failed due to missing case information
N = Not marked complete
W = Waived, tempo (requires Regional approval)
U = Undeterminable (Cashiering only)
*
= Out of tax unit’s authority

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

ACCEPTANCE SAMPLES

III. ACCEPTANCE SAMPLING
a. Select Samples
Acceptance Samples are not meant to stand alone. They are to be the means of confirming the
performance of a system whose internal controls have already been assessed by a Systems
Review. If risks have been identified in the Systems Review, the samples will likely verify that
system outputs fail to meet minimum levels of accuracy or completeness. If a system is deemed
to be risk-free, samples of outputs should confirm this fact. For these purposes, it is not
necessary to draw large samples for estimating the defect or error rate. Large samples are costly
and time consuming. Since the purpose of the samples in the TPS Program Review is to verify a
level of performance, much smaller samples can be used.
In assembling files from which samples will be selected, the state must ensure that they are
randomized, or that a randomized algorithm is used before the sample is selected.
Once the System Review is complete, draw a sample of 60 cases from each tax function to
confirm that system outputs meet minimum levels of accuracy or completeness (e.g., all
appropriate actions are being taken to resolve delinquent reports, and field audits meet ESM
requirements).
The desired accuracy/completeness level is at 98% - that is, 98% of the tax function’s outputs or
work products should be accurate. For a tax unit operating with an underlying accuracy level of
98%, there is an 88% chance that two or fewer errors will be discovered in a sample of 60 cases.
As the tax function's underlying accuracy level increases, there is a proportional increase in the
chances of passing. Conversely, as the tax function's accuracy level decreases, the more cases in
the sample are likely to fail.
The failure of three or more cases is reason to conclude that the exception rate for that function is
at an unacceptable level. As the graph below shows, tax functions with accuracy levels below
98% still have a chance to pass sample review, but the probability of passing rapidly decreases
as the underlying accuracy level decreases. For instance, if the underlying error is 9%, there is
only an 8% chance of passing the sample.

II - 34

R 07/06

S

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

ACCEPTANCE SAMPLES

b. Case Non Use or Case Replacement.
There are three instances where cases may not be used, or case replacement may be necessary.
1. The particular case selected should not have been in the universe to be
sampled. Note that this should not occur if, when developing sampling systems,
the universe files were properly reviewed. There may be instances when the
universe was not assembled correctly. For example, when building the universe
of Collection cases, an account receivable which had less than $100.00 unpaid UI
tax due, could be inadvertently included. (The universe should consist of $100.
or more in unpaid tax) However, it is important to have some evaluation of each
tax function if at all possible. If cases that should not be in the universe are
selected, do not replace them, instead, continue extracting samples. As long as
the sample contains a minimum of 53 valid cases (i.e., cases that meet the

II - 35

R 04/03

S

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

ACCEPTANCE SAMPLES

universe definition), two cases can still fail and the results of the Acceptance
Sample will remain consistent with that of 60 case samples. In such instances, the
Region needs to notify the National Office, which will ensure that the SUN
system accepts such modification on a case-by-case basis.
If the sample size drops below 53, the tax function cannot be evaluated.
The DP section should be alerted in order to make modifications for future
sampling efforts. The reviewer must then advise the Regional Office to ensure
that future universes will be sound. Note must also be made in the Annual
Report of this problem as well as steps to rectify it.
2. In Collections, any cases selected that had already been subject to a TPS review
the previous year, for which no subsequent debt was created, should be replaced.
3. The particular case selected cannot be reviewed due to a missing case folder,
or other documentation is missing, such as the microfiche or image of source
documents.
Sometimes documents cannot be found in order to make a ruling on the case's
accuracy. Records may have been destroyed through circumstances beyond
control (e.g. fire, flood etc.). The documents may be misfiled or "checked out" to
some other staff member and cannot be found. IN INSTANCES OF THIS
TYPE REPLACEMENT IS ALLOWED FOR ONE AND ONLY ONE LOST
CASE.
Before replacement the reviewer must: Assure (as far as possible) that the work on the case was
actually done, and make every effort to find the information. Check all possible places/persons
where the information could be located. Inform the Regional Office of such instances.
If a SECOND case in the sample is missing then the reviewer CANNOT reach a conclusion
that there is reasonable assurance of accuracy, and the entire sample will fail. The failure
will count towards the total number of tax functions that have failed TPS quality review.
Mark the sample as a “Discard” and notify the National Office. Further details on the
finding can be explained in the Annual Report.
While the review may stop at this point, tax managers may wish to continue the review in order
to get an unofficial assessment of how the tax function is faring or to see the extent of missing
documentation.

II - 36

R 07/06

S

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

ACCEPTANCE SAMPLES

c. Review Samples

Once the sample is selected, each case in it must be reviewed, using the
appropriate sample questionnaire. Each questionnaire contains a series
of questions which must be answered for each case to determine
whether it is correct.
One of the more difficult aspects of rating the cases involves deciding
whether or not a given case is acceptable. For some transactions, the decision is relatively
simple. For example, if a given employer contribution was not credited to the proper account in
the right amount, then the transaction is not acceptable. However, Status Determinations,
Collections, and Field Audits have many more elements in them. Some of those elements may
be incomplete or inaccurate without materially affecting the accuracy of the outcome. In each
function, every effort has been made to identify the essential elements for analysis and to
exclude unnecessary elements.
Any question that the reviewer and State has about materiality of an element should be resolved
with Regional staff as per the earlier Planning for Reviews section.
Note that if a sample case uncovers that something was done in error, but that the error was
subsequently discovered and rectified due to the State's internal controls, the case would not fail.
If, however, the error is rectified due to the employer bringing it to the State's attention, the case
fails.
Appeals When a case appearing in the sample is under appeal, the review procedure should not
be affected when the reason or basis for an appeal or its outcome is not germane
to the purposes of the review. However, the appeal is a structured legal
proceeding. The reviewer should determine if the agency has followed
established procedure up to the point at which the case is being reviewed.
d. Complete the Acceptance Sample Coding Sheet
The answers to the Acceptance Sample Questionnaire questions should be recorded on the
coding sheet for the respective function. The coding sheet allows space to record an answer for
each question on the questionnaire. Most questions are coded Yes or No; however,
some questions on the form provide for "I" (Information Not Available) or "A" (Not Applicable)
which should be used only when appropriate.

II - 37

R 04/03

S

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

ACCEPTANCE SAMPLES

Information
Not
Available

There may be instances when information is not available. For example:
There is a question for the Collections sample which asks, "Were telephone
contacts made...for the most recent quarter of liability?". It cannot be answered
"yes" if no evidence can be located in the employer file to confirm a specific
telephone call. However, if a letter referencing a previous phone call is found the
TPS reviewer may draw the conclusion that such a phone call had probably been
made. Under such circumstances "I" would be the appropriate answer. If no
evidence exists that a phone call was made, yet State procedure requires one be
made, then the answer to this question would be "No". This is a judgment call on
the part of the reviewer, based on the information at hand and an understanding of
the State's particular policy regarding collections procedures.

Not
Applicable

Acceptance Sample questions are to be answered "A" only when appropriate.
This means that an "A" response is acceptable only if the State does not utilize
that process in their tax operations.. Regional Office approval would be sought
and this question would no longer be asked of the State.
The second instance when an "A" response would be acceptable is when the
process being reviewed was unnecessary in the case being examined. In the
Collection chapter a question is asked, "Did State procedures require enforcement
actions be taken to collect?" An example when "A" would be appropriate is if the
money was received after a couple of calls and further enforcement action was
unnecessary.

When all cases have been reviewed, the total number of acceptable cases should be entered at the
bottom of the coding sheet. At this time, the reviewer may wish to begin the process of entering
sample data into the SUN system.
An Acceptance Sample Explanation Sheet has been included in each chapter. Any sample case
that fails should be identified, and the potential responsibility for its failure should be noted.
This will assist reviewers later when they attempt to correlate systemic weaknesses with
resultant inaccuracies in various tax functions' output.

II - 38

R 04/03

S

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

ACCEPTANCE SAMPLES

e. Prepare Preliminary Findings and Meet with State Staff
At the conclusion of the Systems Reviews and Acceptance Sampling, the
reviewer should complete the Reasonable Assurance Chart and draw
conclusions as to whether the State has accuracy in all of the functions
examined. (Soon after each review is finished tax managers should also
be made aware of any problems that may have been found so they can
clarify any misunderstandings or begin to consider potential program
improvement strategies.)

Systems Reviews and Acceptance Sampling results are complementary. There are four possible
outcomes after both procedures have been conducted: If no risk was found in the review of State
controls, passing Acceptance Sampling is a consistent outcome. So is the situation of having
identified risks in the State's internal controls and having failed sampling. However, findings of
risk coupled with passing sampling; and findings no risk and failing sampling are inconsistent
and require analysis and explanation.

SYSTEMS REVIEW

ACCEPTANCE SAMPLE

No Risk Found

Pass (consistency)

Fail (inconsistency)

Risk Found

Pass (inconsistency)

Fail (consistency)

The reviewer must take whatever steps are necessary to make the findings from the Systems
Review and the Acceptance Sampling Review rational and consistent. For both inconsistent
outcomes, additional analysis will be necessary to resolve the findings or provide an explanation
for the inconsistency. Only then is the reviewer finished with preliminary findings.

II - 39

R 04/03

S

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

ACCEPTANCE SAMPLES

FRAMEWORK FOR PROGRAM REVIEW ANALYSIS
SYSTEMS REVIEW
FINDING

ACCEPTANCE SAMPLE
FINDING

No Risk Found

Acceptance Sample Passes
Finding: the tax function's controls are in place and producing high
quality outputs.

No Risk Found

Acceptance Sample Fails
Further Work: Re-examine the failed cases to confirm that they
should have been judged as having failed. Examine the cause of
sample failure, determine if it had simply been due to a rare case of
human error and whether another sample case should be pulled.
Re-examine Systems Review findings. Are there any controls that
should have been deemed at risk? If controls are proper, are they
being executed by staff - is there a defect in the control's design? Were the controls verified to be in place? Consider drawing second
sample to confirm or refute findings, or drawing an expanded
sample to produce a specific error level.
Finding: The tax function's controls are at risk and allowing
inaccurate or incomplete outputs.
OR
Expanded sampling indicates that the tax function's controls are in
place and producing quality outputs.

II - 40

R 04/03

S

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

Risk Found

CHAPTER TWO

ACCEPTANCE SAMPLES

Acceptance Sample Passes
Further Work: re-examine area at risk to determine whether 1)
there is significant risk; or 2) the existing controls are strong enough
to produce quality outputs; or 3) compensating controls have been
overlooked; or 4) there are off-setting factors (still considered a risk
by TPS) such as long-time competent employees in place.
Consider selecting another sample to confirm that the outputs are
truly accurate. The sample design is such at there is a change of
passing (2 or fewer errors in a sample of 60) even though there is an
underlying error rate of 8.6% - this may be such a situation.
Finding: The tax function's controls are in place/or have offsetting
factors/or are strong enough to produce accurate outputs. Produce
recommendations for improving any controls.
OR
Additional sampling has produced failed cases - confirming that the
tax function's controls are at risk and allowing inaccurate outputs.

Risk Found

Acceptance Sample Fails
Further Work: Analyze the cause and effect relationship of risk to
failure. Consider drawing an expanded sample to estimate a true
error rate. Develop recommendations for improvement.
Finding: The tax function's controls are at risk and allowing
inaccurate outputs.

As always, it is important that State staff be integrally involved in the review process.
Therefore, the reviewer should meet at this point with the appropriate State decision-maker (e.g.,
II - 41

R 04/03

S

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

ACCEPTANCE SAMPLES

UI Director and/or Tax Director) to discuss the preliminary findings. At the meeting the
reviewer should briefly remind the State staff of the objectives and sub-objectives being
analyzed. All findings should be discussed, the cause and impact of any problems should be
presented, and means of addressing the problems should be considered.
The reviewer should also have examined the findings from Computed Measures, before meeting
with State staff
If the State decision maker agrees with the findings the reviewer may proceed with data entry
into the SUN system and the preparation of a written "draft" Annual Report. If the State
decision maker does not agree with the findings, the reviewer will need to gather additional
information. The reviewer may choose at this point to return to parts of the Systems Review.
For example, the State staff might indicate that the reviewer has misunderstood something which
was reviewed, or that additional information is available from a source which the reviewer did
not use. The reviewer can repeat segments of the Systems Review to revise the findings or to
confirm that the findings are correct. The State may elect to draw a second Acceptance Sample
to confirm or deny the initial findings. Only two Acceptance Samples may be selected for each
function.
If the State does not agree with the findings from the second Acceptance Sample, the next
required step in the Program Review process becomes Expanded Sampling.

f. Select and Examine Expanded Samples
The purpose of expanded sampling in TPS is to support the findings of the review of internal
controls, quality assurance systems and the acceptance or procedure review samples. Expanded
sampling is not necessary when the State concurs with the findings. If, however, the State does
not agree, then an expanded sample is necessary to estimate the potential impact of the weakness
on program quality. Even when the State might agree with a finding, the reviewer has the option
to select an expanded sample when there is uncertainty about the nature of findings and more
detailed information is desired.
Expanded samples will be used to assess the extent of the problems identified by acceptance
sampling. They will need to be large enough to develop precise estimates of the error rate. The
State and TPS reviewer also may choose to use expanded sampling to identify the nature and
cause of errors found during acceptance sampling. When used for these purposes it may be
appropriate to draw the expanded sample only from a particular type of transaction.

II - 42

R 04/03

S

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

ACCEPTANCE SAMPLES

As the name implies, an expanded sample is an expansion of the acceptance sample. Therefore,
expanded sampling follows the same steps and the preparation conducted for the acceptance
sampling review can be used directly in conducting the expanded sampling.
The steps are:
1) determine the sample size
2) identify the transaction types and time periods
3) select the sample
4) review the sample, using the questionnaire for the function.
The steps are described in detail in Appendix A of this handbook.

II - 43

R 04/03

S

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

ACCEPTANCE SAMPLES

g. Sampling by Exception
The TPS Program Review examines the routine processing of work in each major tax function
by extracting samples of work products. By examining small samples of the majority of the
State's work, assessment is made on the quality of the overall tax operation. However, some
States may also be concerned about potential problems outside the routine flow of work. Areas
that are more error-prone may warrant special attention: an example could be employer charge
statements with credits which involve claimants with overpayments and/or monetary
redeterminations.
These areas have not been included in the basic TPS design even though they are much more
likely to result in error, because they only represent a small proportion of the tax operation's
output.
If the State wishes to examine such exceptions, they may design an Acceptance Sample for these
error-prone cases, or they may contact the Regional Office for assistance. Regional staff may
have knowledge of similar sampling processes used by other States, or the National Office may
have a design that would be applicable
The State may also want or need more information about a risk which the Systems Review
identifies or a problem which reoccurs in Acceptance Sample cases. In these instances States
may design a "special study" for informational purposes. Such a study may involve designing
special questions and selecting a sample where the questions may be used to gather information
or, in the case of an expanded sample, to determine the error rate. States should contact their
Regional Office for additional information regarding exception sampling.

II - 44

R 04/03

S

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

CONCLUDING REVIEWS

IV. CONCLUDING REVIEW

a. Assess Computed Measures' Findings
Program Reviews are intended mainly to assess reasonable assurance of accuracy in the tax
operation. Computed Measures provide indicators of end-product timeliness and completeness.
An assessment of a tax operation must weigh all these factors (i.e., timeliness and accuracy are
both important in the Status operation - however, timeliness of determinations should not be at
the cost of accuracy. An acceptable balance must be sought). Computed measures will be
generated based on data derived from routine State reports.
TPS has developed reports which display trends over time within a State and among States.
These reports are on the TPS section within the SUN system should be examined by the TPS
reviewer along with the findings from Program Reviews to develop a comprehensive evaluation
of the State's tax operation. Links to national tax data are also available at the web site
www://ows.doleta.gov, under “Unemployment Insurance”, “Performance Management”, “UI
Performs Performance Measures” and under “Tax Performance System”.
Refer to Appendix B for data processing instructions for Computed Measures.

b. Complete the Program Review Chart
Before completing the Annual Report, reviewers need to complete the Program Review Chart.
The purpose of the chart is to:

Summarize the results of the Acceptance Sampling relative to which
functions have reasonable assurance of accuracy,
Record how individual portions of the System Reviews contribute to
reasonable assurance of quality.

Reviewers should fill in every cell on the Chart which is not shaded. The allowable entries are
described. (Note that the electronic version of this Chart on the SUN system has slightly
different codes to reflect the automated nature of the SUN version.)

II - 45

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

CONCLUDING REVIEWS

COMPLETION OF THE PROGRAM REVIEW CHART

Systems Review
ENTRY

MEANING

C (Complete)

All internal controls were verified as being in place and
operating

R (Risk)

One or more controls were missing or not verified (One
or more "No" answers to evaluative questions)

O (Other)

Another, "compensating control" was verified as being in
place and operating (Regional approval required)

Acceptance Sampling
ENTRY

MEANING

Y (Pass)

Fewer than 3 cases failed sampling (internal controls are
effective in producing quality outputs)

N (Fail)

Three or more cases failed sampling review (internal
controls are not effective in producing quality outputs)

or
Two or more cases were missing (insufficient documentation
to rule on quality of outputs)
D (Discard)

Sample was discarded

At this point the data from the Systems Reviews and Acceptance Sampling should be consistent, in
which case judgments are reasonably straight forward. When/if there is inconsistency between
findings from the two sources and it's impossible to resolve them, the overall determination of the
tax function's quality is to be based on the findings of the Acceptance Sample (or Expanded Sample
if one was used). Inconsistencies should be noted in the Annual Report.
II - 46

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

CONCLUDING REVIEWS

The Program Review Chart should be attached to the Annual Report. Any Regional comments such
as those dealing with compensating controls or approval of "A" categories in Acceptance Sampling
should also be included.

c. Prepare Draft TPS Report
Upon completion of all portions of the review, the reviewer will prepare a
draft report of the findings. The report should follow the structure of the
TPS Report (example provided in Appendix D) with the exception of the
section on State response, which is not completed at this time.

The draft report is to include information from Systems Reviews,
Acceptance Sampling, Expanded Sampling (as necessary), and Computed Measures. It is meant to
convey in narrative form the areas of strengths and weaknesses in the State tax operation. The TPS
Report should simply and clearly inform State Administrators and Regional representatives of the
areas of concern, exemplary practices and program improvements.
The report should be organized under the following topics:

Purpose

A brief statement of the purpose of the report and the date the review was completed.

Summary of
Findings

The purpose of this part of the report is to provide a synopsis of
what was found, recommendations, and exemplary practices.
(The remainder of this report goes into greater depth on how the data was
gathered, sources, cause, effects, and more detailed recommendations.)

Objectives

Explain what was reviewed and the methodologies used.

II - 47

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES
Principal
Findings

CHAPTER TWO

CONCLUDING REVIEWS

Provide a detailed analysis of overall findings, identifying the tax
functions that failed Acceptance Sampling, and areas identified as needing
improvement.
For each tax function indicate:
If any cases failed, the number, and why they failed.
The risks identified during the Systems Review and where they were
found.
List any compensating controls found to exist. (Attach Regional approval
of the ability of such controls to substitute for TPS-listed controls)
List any controls that were said to be in place but could not be verified
(VS).
Correlate any Acceptance Sample failures with risk found in the Systems
Review, or explain any inconsistencies (e.g., risks but pass, no risks but
fail)
If computed measures data is provided for the function, examine the
trends they reveal, and if appropriate, correlate them with Program
Review findings.
Make recommendations - if appropriate, discuss possible solutions with
the individual responsible for the tax function being reviewed.

II - 48

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

CONCLUDING REVIEWS

Exemplary
Practices

Mention practices that are positive which could be used by other units
within the State or other States. Bring these to the Region's attention.

Global/Systemic
Trends

Mention any overall trends where risks are found or areas of concern
were noted. If systemic strengths are noted throughout the various reviews,
they should be elaborated on in this area of the report.

As the reviewer analyzes TPS findings, potential recommendations should be developed. Input
could be sought from those most directly involved with the tax functions at the State and Regional
levels.

d. Conduct Exit Interview
At the Exit Interview the entire review team should meet with the UI Director, Tax Director and
other staff designated by the State. The meeting will cover each section of the written report and is
to be shared with the State. If the preliminary meeting on findings was detailed, and if no expanded
sampling was done, this portion of the Exit Interview may be brief. It may simply confirm that the
written document incorporates previous understandings.
The second portion of the Exit Interview will be a discussion of State activities which could be
undertaken to correct problems identified or to expand the approaches which are producing high
quality products.

e. Prepare the TPS Annual Report
The Report should consist of a 1- or 2-page Executive Summary which briefly describes principal
findings and summarizes suggested improvements, and a more detailed section which evaluates
every tax function in terms of accuracy, timeliness and completeness. It must include information
from: Computed Measures, Program Reviews (i.e., Systems Reviews and Acceptance Sampling),
and Expanded Sampling (if performed).
Any comments the agency wishes to make should become a part of the report. Action(s) taken
and/or planned to be taken to correct any areas identified as needing improvement should also be
reported.
II - 49

R 04/03

ET HANDBOOK NO. 407
TAX PERFORMANCE SYSTEM
GENERAL PROCEDURES

CHAPTER TWO

CONCLUDING REVIEWS

The final evaluation of the State's tax operations is based on Computed Measures data and
Acceptance Sample findings. The Systems Review serves to identify areas of strengths and
weaknesses for program improvement. The TPS design focuses on the downstream effect - if there
is no material effect on current or future payment of UI tax (as evidenced by sample findings in
which all cases "pass" i.e., meet quality standards), then the State has reasonable assurance of a
quality tax operation. For instance, when risks are identified in the Status operation, but the unit is
still able to produce accurate and timely determinations, the TPS Annual Report will indicate that
there are specific areas of risk (and what the recommended solutions may be), but that the State
presently has confirmed acceptable levels of accuracy for its Status Determinations.

II - 50

R 04/03