Supporting Statement for Clearance to Conduct an Appeals Customer Satisfaction Survey
(FY 2012)
JUSTIFICATION
Circumstances Making the Collection of Information Necessary
Background/Overview
IRS has replaced the traditional measures of accomplishment with a balanced measurement system consisting of business results, customer satisfaction, and employee satisfaction. As an important customer interface the Appeals Division needs feedback from customers to continuously improve its operations. This initiative is part of the service-wide effort to establish a system of balanced organizational performance measures mandated by the IRS Restructuring and Reform Act of 1998. This is also a result of Executive Order 12862, which requires all government agencies to survey their customers and incorporate customer preferences in their process improvement efforts.
Purpose and Use of the Information Collection
Objectives of Data Collection
The primary goals of the study are to survey Appeals external customers on an ongoing basis regarding their expectations, track customer satisfaction progress over time nationwide, and identify operational improvements.
Data to be Collected
Appeals will draw a sample from approximately 132,000 closed cases from FY 2012 and send all cases to the contractor by the 13th of each month following the close of a quarter (i.e., January, April, July, October). The contractor will select a random stratified sample to achieve approximately 4200 cases.
Sample Design
Appeals will draw a census of closed case records from FY 2012 and provide a dataset to the survey contractor. The contractor will select from a random sample to survey approximately 4200 FY 2012 closed cases, stratified by category (e.g., CDP, OIC) and area. This yields an annual precision rate of better than 2.0 percent (at a 95 percent confidence level or better) at the national level and 10.0 percent by Category within each Area.
How Data is Collected and Used
Once the Final Survey Instrument has been approved, the contractor’s designated survey programmer will transform the survey document into an instrument for web-based administration as well as Computer-Assisted Telephone Interviewing (CATI) administration. When the survey programming is complete, several rounds of testing are conducted by the contractor and Appeals staff. The contractor will send the quarterly sample to the government print office (GPO) contractor to mail an invitation containing the web address and log-in information unique to each respondent, inviting the respond to take the web survey. For all respondents who do not respond to the web survey, the contractor will make telephone calls to complete the surveys via CATI.
The contractor will summarize the quantitative ratings and produce quarterly national reports and 11 area reports showing customer satisfaction scores on all Appeals survey items. The contractor will include any relevant database variables in the analysis and will weight the survey responses as necessary to accurately reflect the entire customer base.
For the analysis reports, the contractor will use basic and advanced statistical techniques including:
Survey counts and overall response rates
The overall level of customer satisfaction with services provided
The frequencies and mean for all survey items
The difference in satisfaction ratings and attitudes across customer segments
Analysis of the relationship between survey responses
Analysis of the open-ended questions for improvement suggestions
The contractor will hold the identities of respondents private to the extent allowed by law. The contractor will not provide the IRS with data or status updates that are linked to individual respondents. Upon completion of fielding, the contractor will provide anonymous survey data to the IRS, without any individually identifying information such as name, address, or taxpayer identification number.
Dates Collection Begin and End
Assuming OMB approval, the survey is scheduled to begin by, approximately, January 24, 2012. The fielding period is estimated to end December 14, 2012.
Who is Conducting the Research and Where
The contractor will be responsible for pulling the sample, administering the survey, and conducting data analysis. The survey data will be provided, in a secure manner, to the contractor’s staff for analysis and reporting.
Recruitment Efforts
Sample will be provided by the IRS.
Location-Region/City and Facility
Data extract: IRS
Sampling and data collection: ICF International, Calverton, MD and Burlington, VT
Expected Response Rate
The expected response rate is approximately 35%, based on experience from prior years.
The IRS Appeals survey response rate is expected to be approximately 35 percent, which is consistent with other IRS surveys that typically receive a 30 percent to 50 percent response rate, depending on the type of the survey. Although Appeals falls within the lower end of this range, the response rate and sample size will yield a margin of error of approximately 2 percent at the 95 percent confidence interval, which is a statistically strong outcome. When determining how to best leverage information garnered from the survey, Appeals will carefully consider the survey results. The consideration will include any limitations that may exist in the data, in conjunction with relevant information from other parts of the organization, before making strategic or tactical decisions about how best to manage our programs and serve our customers.
Methods to Maximize Response Rate
Standard procedures will be used in order to obtain the highest response rate possible for the telephone survey. These will include:
1) Incremental release of sample. The contractor will pull a small sample, begin administration, and then add sample batches.
2) Multi-mode methodology. Respondents will have the option to complete the survey at their own convenience via web with the ability to start and suspend the survey.
3) Attempt protocol. The contractor will make up to 9 calls.
4) Call rotation. The CATI system automatically schedules calls to rotate among various times across weekdays and spacing calls at appropriate intervals.
5) Refusal conversion. The contractor will work with Appeals to determine appropriate levels of refusal conversion.
6) Flexibility. Respondents will have the option to call in to complete or continue a survey over multiple sittings. The contractor can also schedule appointments, so respondents can participate at a time convenient to them.
7) High quality interviewing. CATI instruments have clear instructions with conversational scripts. Interviewers are also highly trained and closely monitored.
Further, the contractor will take two concrete steps to help understand the extent of any non-response bias.
A) Analyze population and respondent demographics to determine face validity
B) Analyze late responders as a proxy for non-responders
Testing and Survey Structure/Design
The Appeals Customer Satisfaction Survey is attached and has been tested previously. The survey asks respondents to evaluate various aspects of their experience and to provide an overall summary evaluation. The results should facilitate more effective management of Appeals by providing insight from the customer’s perspective about possible improvements and providing useful input for program evaluation.
The survey includes several rating questions evaluating service delivery as well as one demographic question. In addition, there is one open-end question to allow for suggestions for improvement. Survey scoring for this contract will be based on the Customer Satisfaction Survey Score response average to the key question—“Regardless of the outcome of your appeal, please rate your overall satisfaction with the Appeals process.” Questions will utilize a 5-point rating scale, with 1 being very dissatisfied and 5 being very satisfied.
Efforts Not to Duplicate Research
This is the only study conducted by the IRS on this specific population. The contractor will make reasonable efforts to ensure that customers are only surveyed once within a 12-month period.
Participants Criteria
The population frame consists of all individuals with closed cases during FY 2012 using files provided by the IRS. All listed participants will be eligible except cases identified during the survey process as having an incorrect telephone number.
Consideration Given to Information Technology
A web survey with a follow-up telephone survey to non-respondents is being implemented in FY 2012.
Duplication of Information
No similar data are gathered or maintained by the Agency or are available from other sources known to the Agency.
Reducing the Burden on Small Entities
Small business or other small entities may be involved in these efforts but the Agency will minimize the burden on them of information collections approved under this clearance by sampling, asking for readily available information, and asking brief and poignant questions.
Consequences of Not Conducting Collection
Without these types of feedback, the Appeals will not have timely information to adjust its services to meet customer needs.
Special Circumstances
There are no special circumstances. The information collected will be voluntary and will not be used for statistical purposes.
Consultations with Persons Outside the Agency
No consultations are currently planned.
Payment or Gift
The Appeals will not provide payment or other forms of remuneration to respondents of the survey feedback.
Confidentiality
No PII information will be collected on the survey.
All participants will be subject to the provisions of the Taxpayer Bill of Rights II during this study and the vendor will ensure that all participants are treated fairly and appropriately.
The security of the data used in this project and the privacy of taxpayers will be carefully safeguarded at all times. Security requirements are based on the Computer Security Act of 1987 and Office of Management and Budget Circular A-130, Appendices A & B. Physical security measures include a locked, secure office. Notes are stored in locked cabinets or shredded. Data security at the C-2 level is accomplished via the Windows NT operating system. Systems are password protected, users profiled for authorized use, and individual audit trails generated and reviewed periodically.
The IRS will apply and meet fair information and record-keeping practices to ensure privacy protection of all taxpayers. This includes criteria for disclosure—laid out in the Privacy Act of 1974, the Freedom of Information Act, and Section 6103 of the Internal Revenue Code—all of which provide for the protection of taxpayer information as well as its release to authorized recipients.
The survey will not contain tax return or taxpayer information. Survey participants will not be identified in any of the documents or files used for this project. We will limit and control the amount of information we collect to those items that are necessary to accomplish the research questions. We will carefully safeguard the security of data utilized as well as the privacy of the survey respondents. We will apply the fair information and record-keeping practices to ensure protection of all survey respondents. The criterion for disclosure laid out in the Privacy Act, the Freedom of Information Act, and section 6103 of the Internal Revenue Code provides for the protection of information as well as its releases to authorized recipients.
PRA Statement and OMB Control Number
The OMB Control Number and required information will be provided on the survey itself.
Sensitive Nature
No questions will be asked that are of a personal or of sensitive nature.
Burden of Information Collection
The Appeals Survey has been designed to minimize burden on the taxpayer. The time that a respondent takes to complete the survey has been carefully considered and only the most important areas are being surveyed. The average time of survey completion is expected to be 9 minutes. This is based on the questionnaire consisting of several screener questions, 18 satisfaction questions, and one open-ended question. The questions are generally one sentence in structure and on an elementary concept level.
The annual burden hours requested (960 hours) are based on the number of collections we expect to conduct over the requested period for this clearance.
Estimated Annual Reporting Burden |
||||
Type of Collection |
No. of Respondents |
Annual Frequency per Response |
Hours per Response |
Total Hours |
Time to read mailed survey invitation |
12,000 |
1 |
1 minute |
200 |
|
7,800 |
1 |
1 minute |
130 |
[Household and Private Sector –Completed Surveys] |
[4,200] |
[1] |
[9 minutes] |
630 |
Total Burden 960
Costs to Respondents
No costs are anticipated.
Costs to Federal Government
The anticipated cost to the Federal Government is approximately $232,246 annually. These costs are comprised of contractor expenses.
Reason for Change
Not applicable.
Tabulation of Results, Schedule, Analysis Plans
Although Appeals does not intend to publish its findings, the we may receive requests to release the information (e.g., congressional inquiry, Freedom of Information Act requests). Appeals will disseminate the findings when appropriate, strictly following the Agency's "Guidelines for Ensuring the Quality of Information Disseminated to the Public.", and will include specific discussion of the limitation of the qualitative results discussed above.
Display of OMB Approval Date
We are requesting no exemption.
Exceptions to Certification for Paperwork Reduction Act Submissions
These activities comply with the requirements in 5 CFR 1320.9.
Dates collection will begin and end
January 24, 2012 through November 20, 2012
B. STATISTICAL METHODS
1. Universe and Respondent Selection
Appeals will provide the vendor with the Appeals closed case database for FY 2012. This data will not include taxpayer identification numbers. The vendor will use this database to select a random sample of taxpayers to be surveyed.
2. Procedures for Collecting Information
The activities under this clearance will involve a vendor-driven, randomly selected sample of taxpayers who have had cases closed within Appeals during FY 2012. These respondents were selected either to cover a broad range of customers or to include specific characteristics related to certain products or services. Results will not be used to make statements representative of the universe of study, to produce statistical descriptions (careful, repeatable measurements), or to generalize the data beyond the scope of the sample.
Qualitative surveys are tools used by program managers to change or improve programs, products, or services. The accuracy, reliability, and applicability of the results of these surveys are adequate for their purpose.
The samples associated with this collection are not subjected to the same scrutiny as scientifically drawn samples where estimates are published or otherwise released to the public.
3. Methods to Maximize Response
The contractor will make follow-up calls to anyone who does not respond to the web survey. The contractor will attempt to make up to 9 calls.
Information collected will not yield generalizable quantitative findings; it can provide useful customer input, but it does not yield data about customer opinions that can be generalized.
4. Testing of Procedures
Pretesting may be done with internal staffs, a limited number of external colleagues, and/or customers who are familiar with the programs and products. If the number of pretest respondents exceeds nine members of the public, Appeals will submit the pretest instruments for review.
5. Contacts for Statistical Aspects and Data Collection
Each program will obtain information from statisticians in the development, design, conduct, and analysis of customer/partner service surveys, when appropriate. This statistical expertise will be available from agency statisticians or from contractors and the Agency will include the names and contact information of persons consulted in the specific information collection requests.
Statistical Contact
For questions regarding the study or questionnaire design or statistical methodology, contact:
Until a contractor is selected:
Rocco D. DiLisio
Policy Analyst
Appeals Customer Satisfaction Survey Coordinatror
Internal Revenue Service
(412)-395-4994
Diana Palombo
Statistician
Appeals Customer Satisfaction Survey
Internal Revenue Service
(412)-395-5058
Attachments
| File Type | application/msword |
| File Title | Supporting Statement for Clearance to Conduct an Appeals Customer Satisfaction Survey |
| Author | mdsloa00 |
| Last Modified By | mdsloa00 |
| File Modified | 2012-04-10 |
| File Created | 2012-04-10 |