CS-10-227 - Supporting Statement

OMB SUPPORTING STATEMENT

STUDY TO MEASURE CUSTOMER SATISFACTION

IRS APPEALS TAX CUSTOMERS

June 2010

A. Introduction

Background/Overview

IRS has replaced the traditional measures of accomplishment with a balanced measurement system consisting of business results, customer satisfaction, and employee satisfaction. As an important customer interface the Appeals Division needs feedback from customers to continuously improve its operations. This initiative is part of the service-wide effort to establish a system of balanced organizational performance measures mandated by the IRS Restructuring and Reform Act of 1998. This is also a result of Executive Order 12862, which requires all government agencies to survey their customers and incorporate customer preferences in their process improvement efforts.

Objectives of Data Collection

The primary goals of the study are to survey Appeals external customers on an ongoing basis regarding their expectations, track customer satisfaction progress over time nationwide, and identify operational improvements.

B. Methodology

Sample Design

Appeals will draw a census of closed case records from FY 2010 and provide a dataset to the survey contractor. The contractor will select from a random sample to survey 4,421 FY 2010 closed cases, stratified by category (e.g., CDP, OIC) and area. This yields an annual precision rate of better than 2.0 percent (at a 95 percent confidence level or better) at the national level and 10.0 percent by Category within each Area.

Data to be Collected

Appeals will draw a sample from approximately 133,616 closed cases (not including exclusions) from FY 2010 and send all cases to the contractor by October 2010. The contractor will select a random stratified sample to achieve 4,421 cases.

Sample by Area and Category
Area	Category	Population	Sample	95% +/-
1	CDP	7	*
	OICC	21	*
	INNSP	106	38	12.8%
	POSTPEN	348	75	10.0%
	IC	81	32	13.4%
	EXM/TEGE	3751	94	10.0%
	OTHER	406	78	10.0%
2	CDP	12504	95	10.0%
	OICC	3043	93	10.0%
	INNSP	31	17	16.0%
	POSTPEN	309	73	10.0%
	IC	2	*
	EXM/TEGE	15	*
	OTHER	3888	94	10.0%
3	CDP	2	*
	OICC	15	*
	INNSP	130	43	12.2%
	POSTPEN	461	80	10.0%
	IC	279	72	10.0%
	EXM/TEGE	5962	95	10.0%
	OTHER	235	68	10.0%
4	CDP	15	*
	OICC	10	*
	INNSP	69	24	16.0%
	POSTPEN	273	71	10.0%
	IC	156	55	10.1%
	EXM/TEGE	4461	94	10.0%
	OTHER	89	32	13.7%
5	CDP	6567	95	1000.0%
	OICC	3442	93	10.0%
	INNSP	0	*
	POSTPEN	1718	91	10.0%
	IC	0	*
	EXM/TEGE	4846	94	10.0%
	OTHER	7	*
6	CDP	11544	95	10.0%
	OICC	2174	92	10.0%
	INNSP	4387	94	10.0%
	POSTPEN	453	79	10.0%
	IC	0	*
	EXM/TEGE	2395	92	10.0%
	OTHER	3502	94	10.0%
7	CDP	3804	94	10.0%
	OICC	1023	88	10.0%
	INNSP	98	39	12.1%
	POSTPEN	475	80	10.0%
	IC	255	70	10.0%
	EXM/TEGE	5182	94	10.0%
	OTHER	1325	90	10.0%
8	CDP	2491	93	10.0%
	OICC	823	86	10.0%
	INNSP	75	29	14.3%
	POSTPEN	281	72	10.0%
	IC	87	32	13.8%
	EXM/TEGE	4447	94	10.0%
	OTHER	3780	94	10.0%
9	CDP	2292	92	10.0%
	OICC	622	83	10.0%
	INNSP	130	43	12.2%
	POSTPEN	185	63	10.0%
	IC	194	64	10.0%
	EXM/TEGE	5208	94	10.0%
	OTHER	888	87	10.0%
10	CDP	6787	95	10.0%
	OICC	249	70	10.0%
	INNSP	2	*
	POSTPEN	9118	95	10.0%
	IC	14	*
	EXM/TEGE	5254	94	10.0%
	OTHER	195	65	10.0%
AT/TG	CDP	2	*
	OICC	0	*
	INNSP	3	*
	POSTPEN	0	*
	IC	191	64	10.0%
	EXM/TEGE	121	42	12.2%
	OTHER	13	*
	CIC	298	73	10.0%
Total		133,616	4,421	***

How Data is Collected and Used

Once the Final Survey Instrument has been approved, the contractor’s designated survey programmer will transform the survey document into an instrument for Computer-Assisted Telephone Interviewing (CATI) administration. When the survey programming is complete, several rounds of testing are conducted by the contractor and Appeals staff. Once the testing is complete and the interviewers have been trained, the contractor will conduct a “slow start,” consisting of interviews with 10 potential respondents with intensive monitoring from project management staff, quality assurance staff, and supervisors. If changes are necessary based on the “slow start,” the contractor will make modifications until the survey is correct; if no changes are necessary these survey responses will be maintained as valid responses.

The contractor will summarize the quantitative ratings and produce one national report showing customer satisfaction scores on all Appeals survey items. The contractor will include any relevant database variables in the analysis and will weight the survey responses as necessary to accurately reflect the entire customer base.

For the analysis reports, the contractor will use basic and advanced statistical techniques including:

• Survey counts and overall response rates

• The overall level of customer satisfaction with services provided

• The frequencies and mean for all survey items

• The difference in satisfaction ratings and attitudes across customer segments

• Analysis of the relationship between survey responses

• Analysis of the open-ended questions for improvement suggestions

The contractor will hold the identities of respondents private. The contractor will not provide the IRS with data or status updates that are linked to individual respondents. Upon completion of fielding, the contractor will provide anonymous survey data to the IRS, without any individually identifying information such as name, address, or taxpayer identification number.

Dates Collection Begin and End

Assuming OMB approval, the survey is scheduled to begin by, approximately, October 13, 2010. The fielding period is estimated to take 9 weeks.

Who is Conducting the Research and Where

The contractor will be responsible for pulling the sample, administering the survey, and conducting data analysis. The survey data will be provided, in a secure manner, to the contractor’s staff for analysis and reporting.

Cost of Study

Approximate: $400,000.

There will be no stipends paid to participants.

Recruitment Efforts

Sample will be provided by the IRS.

Location-Region/City and Facility

Data extract: IRS

Sampling and data collection: Contractor (to be determined)

Expected Response Rate

The expected response rate is 30%-35% based on experience from prior years.

The IRS Appeals survey response rate is expected to be 30 percent to 35 percent, which is consistent with other IRS surveys that typically receive a 30 percent to 50 percent response rate, depending on the type of the survey. Although Appeals falls within the lower end of this range, the response rate and sample size will yield a margin of error of approximately 2 percent at the 95 percent confidence interval, which is a statistically strong outcome. When determining how to best leverage information garnered from the survey, Appeals will carefully consider the survey results. The consideration will included any limitations that may exist in the data, in conjunction with relevant information from other parts of the organization, before making strategic or tactical decisions about how best to manage our programs and serve our customers.

Methods to Maximize Response Rate

Standard procedures will be used in order to obtain the highest response rate possible for the telephone survey. These will include:

1) Incremental release of sample. The contractor will pull a small sample, begin administration, and then add sample batches.

2) Attempt protocol. The contractor will make up to 9 calls.

3) Call rotation. The CATI system automatically schedules calls to rotate among various times across weekdays and spacing calls at appropriate intervals.

4) Refusal conversion. The contractor will work with Appeals to determine appropriate levels of refusal conversion.

5) Flexibility. Respondents will have the option to call in to complete or continue a survey over multiple sittings. The contractor can also schedule appointments, so respondents can participate at a time convenient to them.

6) High quality interviewing. CATI instruments have clear instructions with conversational scripts. Interviewers are also highly trained and closely monitored.

Further, the contractor will take two concrete steps to help understand the extent of any non-response bias.

A) Analyze population and respondent demographics to determine face validity

B) Analyze late responders as a proxy for non-responders

Testing and Survey Structure/Design

The Appeals Customer Satisfaction Survey is attached and has been tested previously. The survey asks respondents to evaluate various aspects of their experience and to provide an overall summary evaluation. The results should facilitate more effective management of Appeals by providing insight from the customer’s perspective about possible improvements and providing useful input for program evaluation.

The survey includes several rating questions evaluating service delivery as well as one demographic question. In addition, there is one open-end question to allow for suggestions for improvement. Survey scoring for this contract will be based on the Customer Satisfaction Survey Score response average to the key question—“Regardless of the outcome of your appeal, please rate your overall satisfaction with the Appeals process.” Questions will utilize a 5-point rating scale, with 1 being very dissatisfied and 5 being very satisfied.

Efforts Not to Duplicate Research

This is the only study conducted by the IRS on this specific population. The contractor will make reasonable efforts to ensure that customers are only surveyed once within a 12-month period.

Participants Criteria

The population frame consists of all individuals with closed cases during FY 2010 using files provided by the IRS. All listed participants will be eligible except cases identified during the survey process as having an incorrect telephone number.

C. Privacy, Disclosure, Security Issues

All participants will be subject to the provisions of the Taxpayer Bill of Rights II during this study and the vendor will ensure that all participants are treated fairly and appropriately.

The security of the data used in this project and the privacy of taxpayers will be carefully safeguarded at all times.  Security requirements are based on the Computer Security Act of 1987 and Office of Management and Budget Circular A-130, Appendices A & B.  Physical security measures include a locked, secure office.  Notes are stored in locked cabinets or shredded.  Data security at the C-2 level is accomplished via the Windows NT operating system.  Systems are password protected, users profiled for authorized use, and individual audit trails generated and reviewed periodically. 

The IRS will apply and meet fair information and record-keeping practices to ensure privacy protection of all taxpayers.  This includes criteria for disclosure—laid out in the Privacy Act of 1974, the Freedom of Information Act, and Section 6103 of the Internal Revenue Code—all of which provide for the protection of taxpayer information as well as its release to authorized recipients.

The survey will not contain tax return or taxpayer information. Survey participants will not be identified in any of the documents or files used for this project.  We will limit and control the amount of information we collect to those items that are necessary to accomplish the research questions.  We will carefully safeguard the security of data utilized as well as the privacy of the survey respondents.  We will apply the fair information and record-keeping practices to ensure protection of all survey respondents. The criterion for disclosure laid out in the Privacy Act, the Freedom of Information Act, and section 6103 of the Internal Revenue Code provides for the protection of information as well as its releases to authorized recipients.

PRA Statement and OMB Control Number

The OMB Control Number and required information will be provided on the survey itself.

D. Burden Hours

The Appeals Survey has been designed to minimize burden on the taxpayer. The time that a respondent takes to complete the survey has been carefully considered and only the most important areas are being surveyed. The average time of survey completion is expected to be 9 minutes. This is based on the questionnaire consisting of several screener questions, 18 satisfaction questions, and one open-ended question. The questions are generally one sentence in structure and on an elementary concept level.

We expect to contact approximately 12,564 taxpayers to achieve a total of 4,421 completes. We anticipate the initial screening time to be minimal. The total estimated burden in hours is estimated to be:

8,143 Possible Non-conclusive Survey Calls x 1 minute = 135.71 hours

4,421 Completed Survey Calls x 9 minutes = 663.15 hours

12,564 Total possible survey attempts

E. Statistical Contact

For questions regarding the study or questionnaire design or statistical methodology, contact:

Until a contractor is selected:

Rocco D. DiLisio

Policy Analyst

Appeals Customer Satisfaction Survey Coordinatror

Internal Revenue Service

(412)-395-4994

Diana Palombo

Statistician

Appeals Customer Satisfaction Survey

Internal Revenue Service

(412)-395-5058

F. Attachments

Final Appeals Survey

9