CS-09-174 - Supporting Statement

OMB SUPPORTING STATEMENT

STUDY TO MEASURE CUSTOMER SATISFACTION

IRS APPEALS TAX CUSTOMERS

August 2009

A. Introduction

Background/Overview

IRS has replaced the traditional measures of accomplishment with a balanced measurement system consisting of business results, customer satisfaction, and employee satisfaction. As an important customer interface the Appeals Division needs feedback from customers to continuously improve its operations. This initiative is part of the service-wide effort to establish a system of balanced organizational performance measures mandated by the IRS Restructuring and Reform Act of 1998. This is also a result of Executive Order 12862, which requires all government agencies to survey their customers and incorporate customer preferences in their process improvement efforts.

Objectives of Data Collection

The primary goals of the study are to survey Appeals external customers on an ongoing basis regarding their expectations, track customer satisfaction progress over time nationwide, and identify operational improvements.

B. Methodology

Sample Design

Appeals will draw a census of closed case records from FY2009 and provide a dataset to the survey contractor, ICF Macro (Calverton, MD). ICF Macro will select a random sample of 11,140, stratified by category (e.g., CDP, OIC) and area. This yields an annual precision rate of better than 2.0% (at a 95% confidence level or better) at the national level and 10.0% by Category within each Area.

Data to be Collected

Appeals will draw a sample from approximately 103,000 closed cases (not including exclusions) from FY2009 and send all cases to ICF Macro by October 2009. ICF Macro will select a random stratified sample to achieve 3,345 cases.

National Sample
	Population	Number of Completed Surveys	Sampling Error (at 95% confidence)
Total	103,179	3345	2.2%

Sample by Area and Category
Area	Category	Population	Sample	95% +/-
1	CDP	42	20	15.9%
	OICC	1929	92	10.0%
	INNSP	107	38	12.8%
	POSTPEN	1016	88	10.0%
	IC	105	38	12.7%
	EXM/TEGE	7422	95	10.0%
	OTHER	237	69	10.0%
2	CDP	10096	95	10.0%
	OICC	3164	93	10.0%
	INNSP	11
	POSTPEN	267	71	10.0%
	IC	1
	EXM/TEGE	6
	OTHER	2674	93	10.0%
3	CDP	8554	95	10.0%
	OICC	2395	92	10.0%
	INNSP	2985	93	10.0%
	POSTPEN	136	43	12.4%
	IC	122	42	12.2%
	EXM/TEGE	4627	94	10.0%
	OTHER	2206	92	10.0%
4	CDP	21
	OICC	47	20	16.6%
	INNSP	107	38	12.8%
	POSTPEN	888	87	10.0%
	IC	262	70	10.0%
	EXM/TEGE	8080	95	10.0%
	OTHER	155	54	10.7%
7	CDP	3518	94	10.0%
	OICC	1088	88	10.0%
	INNSP	77	27	15.2%
	POSTPEN	414	78	10.0%
	IC	237	69	10.0%
	EXM/TEGE	5525	94	10.0%
	OTHER	2525	93	10.0%
8	CDP	7228	95	10.0%
	OICC	679	84	10.0%
	INNSP	42	20	15.9%
	POSTPEN	106	38	12.7%
	IC	61	25	15.1%
	EXM/TEGE	6995	95	10.0%
	OTHER	528	81	10.0%
9	CDP	2544	93	10.0%
	OICC	706	85	10.0%
	INNSP	87	32	13.8%
	POSTPEN	5510	94	10.0%
	IC	107	38	12.8%
	EXM/TEGE	6215	95	10.0%
	OTHER	704	85	10.0%
AT/TG	CDP	4
	OICC	0
	INNSP	1
	POSTPEN	3
	IC	107	38	12.8%
	EXM/TEGE	53	20	17.3%
	OTHER	0
	CIC	453	79	10.0%
Total		103,179

How Data is Collected and Used

Once the Final Survey Instrument has been approved, ICF Macro’s designated survey programmer will transform the survey document into an instrument for Computer-Assisted Telephone Interviewing (CATI) administration. When the survey programming is complete, several rounds of testing are conducted by ICF Macro and Appeals staff. Once the testing is complete and the interviewers have been trained, ICF Macro will conduct a “slow start,” consisting of interviews with 10 potential respondents with intensive monitoring from project management staff, quality assurance staff, and supervisors. If changes are necessary based on the “slow start,” ICF Macro will make modifications until the survey is correct; if no changes are necessary these survey responses will be maintained as valid responses.

ICF Macro will summarize the quantitative ratings and produce one national report showing customer satisfaction scores on all Appeals survey items. ICF Macro will include any relevant database variables in the analysis and will weight the survey responses as necessary to accurately reflect the entire customer base.

For the analysis reports, ICF Macro will use basic and advanced statistical techniques including:

• Survey counts and overall response rates

• The overall level of customer satisfaction with services provided

• The frequencies and mean for all survey items

• The difference in satisfaction ratings and attitudes across customer segments

• Analysis of the relationship between survey responses

• Analysis of the open-ended questions for improvement suggestions

ICF Macro will hold the identities of respondents private. ICF Macro will not provide the IRS with data or status updates that are linked to individual respondents. Upon completion of fielding, ICF Macro will provide anonymous survey data to the IRS, without any individually identifying information such as name, address, or taxpayer identification number.

Dates Collection Begin and End

Assuming OMB approval, survey is scheduled to begin by October 13, 2009. The fielding period is estimated to take 9 weeks.

Who is Conducting the Research and Where

ICF Macro will be responsible for pulling the sample, administering the survey, and conducting data analysis. Specifically, ICF Macro’s Burlington, VT and Plattsburgh, NY offices will be responsible for pulling the sample and administering the survey. The survey data will be provided in a secure manner to the staff at the ICF Macro Calverton, MD location for analysis and reporting.

Cost of Study

ICF Macro contract: $256,421.

There will be no stipends paid to participants.

Recruitment Efforts

Sample will be provided by the IRS.

Location-Region/City and Facility

Data extract: IRS

Sampling and data collection: ICF Macro, Burlington, VT, Plattsburgh, NY and Calverton, MD

Data analysis: ICF Macro, Calverton, MD

Expected Response Rate

The expected response rate is 30%-40% based on experience from prior years.

The IRS Appeals survey response rate is expected to be 30% to 35%, which is consistent with other IRS surveys that typically receive a 30% to 50% response rate depending on the type of the survey. Although Appeals falls within the lower end of this range, the response rate and sample size will yield a margin of error of approximately 2% at the 95% confidence interval, which is a statistically strong outcome. When determining how to best leverage information garnered from the survey Appeals will carefully consider the survey results, including any limitations that may exist in the data, in conjunction with relevant information from other parts of the organization before making strategic or tactical decisions about how best to manage our programs and serve our customers.

Methods to Maximize Response Rate

Standard procedures will be used in order to obtain the highest response rate possible for the telephone survey. These will include:

1) Incremental release of sample. ICF Macro will pull a small sample, begin administration, and then add sample batches.

2) Attempt protocol. ICF Macro will make up to 9 calls.

3) Call rotation. The CATI system automatically schedules calls to rotate among various times across weekdays and spacing calls at appropriate intervals.

4) Refusal conversion. ICF Macro will work with Appeals to determine appropriate levels of refusal conversion.

5) Flexibility. Respondents will have the option to call in to complete or continue a survey over multiple sittings. ICF Macro can also schedule appointments, so respondents can participate at a time convenient to them.

6) High quality interviewing. CATI instruments have clear instructions with conversational scripts. Interviewers are also highly trained and closely monitored.

Further, ICF Macro will take two concrete steps to help understand the extent of any non-response bias.

A) Analyze population and respondent demographics to determine face validity

B) Analyze late responders as a proxy for non-responders

Testing and Survey Structure/Design

The Appeals Customer Satisfaction Survey is attached and has been tested previously. The survey asks respondents to evaluate various aspects of their experience and to provide an overall summary evaluation. The results should facilitate more effective management of Appeals by providing insight from the customer’s perspective about possible improvements and providing useful input for program evaluation.

The survey includes several rating questions evaluating service delivery as well as one demographic question. In addition, there is one open-end question to allow for suggestions for improvement. Survey scoring for this contract will be based on the Customer Satisfaction Survey Score response average to the key question—“Regardless of the outcome of your appeal, please rate your overall satisfaction with the Appeals process.” Questions will utilize a 5-point rating scale, with 1 being very dissatisfied and 5 being very satisfied.

Efforts Not to Duplicate Research

This is the only study conducted by the IRS on this specific population. ICF Macro will make reasonable efforts to ensure that customers are only surveyed once within a 12-month period.

Participants Criteria

The population frame consists of all individuals with closed cases during FY2009 using files provided by the IRS. All listed participants will be eligible except cases identified during the survey process as having an incorrect telephone number.

C. Privacy, Disclosure, Security Issues

All participants will be subject to the provisions of the Taxpayer Bill of Rights II during this study and the vendor will ensure that all participants are treated fairly and appropriately.

The security of the data used in this project and the privacy of taxpayers will be carefully safeguarded at all times.  Security requirements are based on the Computer Security Act of 1987 and Office of Management and Budget Circular A-130, Appendices A & B.  Physical security measures include a locked, secure office.  Notes are stored in locked cabinets or shredded.  Data security at the C-2 level is accomplished via the Windows NT operating system.  Systems are password protected, users profiled for authorized use, and individual audit trails generated and reviewed periodically. 

The IRS will apply and meet fair information and record-keeping practices to ensure privacy protection of all taxpayers.  This includes criteria for disclosure—laid out in the Privacy Act of 1974, the Freedom of Information Act, and Section 6103 of the Internal Revenue Code—all of which provide for the protection of taxpayer information as well as its release to authorized recipients.

The survey will not contain tax return or taxpayer information. Survey participants will not be identified in any of the documents or files used for this project.  We will limit and control the amount of information we collect to those items that are necessary to accomplish the research questions.  We will carefully safeguard the security of data utilized as well as the privacy of the survey respondents.  We will apply the fair information and record-keeping practices to ensure protection of all survey respondents. The criterion for disclosure laid out in the Privacy Act, the Freedom of Information Act, and section 6103 of the Internal Revenue Code provides for the protection of information as well as its releases to authorized recipients.

PRA Statement and OMB Control Number

The OMB Control Number and required information will be provided on the survey itself.

D. Burden Hours

The Appeals Survey has been designed to minimize burden on the taxpayer. The time that a respondent takes to complete the survey has been carefully considered and only the most important areas are being surveyed. The average time of survey completion is expected to be 9 minutes. The initial screening time is anticipated to be minimal. This is based on the questionnaire consisting of several screener questions, 18 satisfaction questions, and one open-ended question. The questions are generally one sentence in structure and on an elementary concept level.

We expect to contact approximately 11,140 taxpayers to achieve a total of 3,345 completes. We anticipate the initial screening time to be minimal. The total estimated burden in hours is estimated to be:

3,345 surveys x 9 minutes = 501.75 hours

E. Statistical Contact

For questions regarding the study or questionnaire design or statistical methodology, contact:

Mr. Larry Luskin

Vice President

ICF Macro

11785 Beltsville Drive, Suite 300

Calverton, MD 20705

Telephone: 301-572-0334

E-Mail: [email protected]

F. Attachments

Final Appeals Survey

7