SS_for_1090-0007_Part_B_09282012

Supporting Statement B

“Customer Satisfaction Surveys”

OMB Control Number 1090-0007

Collections of Information Employing Statistical Methods

The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When the question “Does this ICR contain surveys, censuses, or employ statistical methods?” is checked "Yes," the following documentation should be included in Supporting Statement B to the extent that it applies to the methods proposed:

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The potential respondent universe is expected to vary based on participation by federal agencies as customer segments are added or deleted. The respondent selection method will take place as follows: Each agency will identify a customer segment. Most will be individuals who either (a) can be identified by screening of random-digit-dial selected representative samples of continental US households or (b) are on agency lists as recipients of agency services or benefits and provide telephone numbers and/or email addresses. For a few agencies, the respondents will be (c) particular types of managerial persons in state, local, or federal agencies or US private sector companies. Many potential respondents in households do not qualify as customers for particular agency services in Random Digit Dialing surveys. However, the historical response rate to the American Customer Satisfaction Index is 94% of those who pass screening and qualify as customers.

For agency-specific studies conducted under this generic clearance, the response rate for telephone surveys of those who answer the telephone and qualify is approximately 93%. The response rate for online surveys is approximately 80% of those who are invited that click on the survey link.

2. Describe the procedures for the collection of information including:

* Statistical methodology for stratification and sample selection,

* Estimation procedure,

* Degree of accuracy needed for the purpose described in the justification,

* Unusual problems requiring specialized sampling procedures, and

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

This project requires separate samples for each survey. Virtually all of these samples require individualized sample designs. Samples are increasingly selected from agency-provided lists, either telephone numbers or email addresses. In fact, almost 60% of surveys are conducted via email solicitation with a link to a secure web portal. Another 35% are conducted via telephone from agency-provided name list samples with telephone numbers. Names are randomly selected from the list samples. Random-Digit-Dialing (RDD) is increasingly rare and currently only comprises about 5% of surveys. This is because customer types often represent rare populations in low incidence in households, increasing the data collection costs. In these increasingly fewer cases, households will be selected by area codes, telephone exchanges, 100-number blocks with the last two digits random numbers.

Business, state, local and federal agency manager interviews will require multi-stage sampling, first selecting a sample of appropriate agencies or businesses, then screening to the desired manager by title or responsibility as names will usually not be known.

Projected estimates for fiscal years 2013-2015 are as follows:

Fiscal Year 2013 – 100 Surveys

The total sample is 100 X 350 = 35,000. This number is derived from the number of surveys being conducted (100) multiplied by the estimated number of respondents per customer sample (350).

Fiscal Year 2014 – 125 Surveys

The total sample is 125X 350 = 43,750. This number is derived from the number of surveys being conducted (150) multiplied by the estimated number of respondents per customer sample (350).

Fiscal Year 2015 – 150 Surveys

The total sample is 150 X 350 = 52,500. This number is derived from the number of surveys being conducted (150) multiplied by the estimated number of respondents per customer sample (350).

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

To maximize response rates, callers will use the CFI Group and/or the federal agency name in the introduction to indicate it is not a sales call. Experienced, trained, and monitored interview technicians will conduct the surveys in a highly professional manner. After an initial failed contact, three or more call backs will be made at different days and times. For email data collection, the introduction also identifies the federal agency name in the introduction. Moreover, multiple reminder emails encouraging customers to participate in the survey will be sent periodically. The completion of interviews/surveys is monitored based on the anticipated response rate. The actual number of responses is approximately 350 per study.

Because the index approach employs multiple questions to create the index and because the 1 – 10 rating scale used for the majority of the questions generates a mean (as compared to a proportion) which is then converted to a 0 – 100 scale and because we know empirically that the standard deviation tends to be approximately 20 for this survey data, a sample size of approximately 350 yields confidence intervals in the range of +/- 1.4 to +/- 3.2 at the 95% confidence level and confidence intervals of +/- 2.0 to +/- 2.5 at the 90% confidence level on the 0 – 100 scale.

The intended purpose of these data collections is to guide leaders and managers in making managerial decisions about ways to improve the quality of services provided by the government. The data collected in these surveys are not used to make policy decisions.

Using the American Association for Public Opinion Research (AAPOR) Call Disposition definitions, Cooperation Rate definitions and Response Rate calculations, the effectiveness of each survey conducted via telephone is monitored and reported. In addition, in a case where there was additional data available about the targeted respondent population, a comparison of respondents versus non-respondents was conducted. In comparing the geographic location, gender and race of the respondents to non-respondents, no differences were found for groups that contained at least 1% of the aggregate sample population. Measures to increase response rates and control for non-response have proven effective.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

Background

The CFI Group, which was founded in 1988, is a quality leader in customer satisfaction. CFI’s chairman, Dr. Claes Fornell, is also the Director of the National Quality Research Center at the University of Michigan Business School and creator of the ACSI. Along with other economic objectives – such as employment and growth – the quality of output (goods and services) is a part of living standards. Like other objectives, the quality of goods and services should be subjected to systematic and uniform measurement. This is the rationale for the ACSI. In the most general sense, the ultimate purpose of the ACSI is to help improve the quality of goods and services available to American citizens.

The ACSI is the only internationally recognized cross-industry, cross-agency methodology for obtaining comparable measures of customer satisfaction. It is a unique customer satisfaction measurement tool developed by the CFI Group and licensed to the American Customer Satisfaction Index. These organizations utilize the proprietary methodology behind the ACSI model to link the drivers and consequences of satisfaction. This patented econometric model enables agencies to obtain insights for valuable, high-return, customer-focused decisions. An important advantage, in contrast to methods that rely solely on survey questions, is that it produces results with statistical stability and low chance variation. This helps ensure uniform and consistent results that allow cross-agency, cross-company, and cross-industry comparisons.

In 1999, the Office of Management and Budget approved a request from the General Services Administration for an emergency collection clearance for the ACSI. That clearance was extended in 2000, when the Office of Management and Budget approved the General Services Administration’s request for a three-year generic clearance. In 2001, the General Services Administration agreed to transfer its generic clearance to the Federal Consulting Group, a franchise in the US Treasury. The Federal Consulting Group contracted with the CFI Group to offer the ACSI to Federal agencies. A request for a three-year extension of the generic clearance was granted in 2003, 2006, and 2009. The Federal Consulting Group transferred to the US Department of the Interior in November 2008 and continues as the Executive Agent for the ACSI in the Federal Government.

Since 1999, the ACSI has used the standard questionnaire to publish the annual national index of customer satisfaction with Federal government services and present these results and their analysis at the Annual Customer Satisfaction Forum sponsored and conducted by the Federal Consulting Group.

Testing

The ACSI methodology is patented and the standard questionnaire has undergone extensive, rigorous testing and study by the University of Michigan and the ACSI organization. The standard questionnaire has already been used by the University of Michigan and the ACSI for eighteen years with great success as a key quality metric for the U.S. economy and in over 20 foreign countries. More specifically, the questionnaire has been used in the Federal Government in numerous studies conducted over 13 years.

When the information needs of federal agencies require the design of a new type of question in the ACSI survey, various testing techniques are used depending on the data collection method to be used and the characteristics of the intended respondents. Typically, a new type of question is tested among those who are not familiar with the program or process being assessed; refinements are made, as needed, that enhance the comprehension of the question, the scale on which it is measured (as appropriate) and the ease with which the respondent can answer it.

5. Provide the names and telephone numbers of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Questions regarding any statistical aspects employed or data collection procedures used should be directed to:

American Customer Satisfaction Index (ACSI)

Professor Claes Fornell, Director

National Quality Research Center

Stephen M. Ross School of Business

University of Michigan

Ann Arbor, MI 48109-1234

Telephone: (734) 763-9767

Fax: (734) 763-9768

Email: [email protected]

Web: www.theacsi.org

Administrative questions regarding the use of this generic clearance by the US Department of the Interior’s Federal Consulting Group should be directed to:

Ron Oberbillig

Chief Operating Officer

Federal Consulting Group

1841 C St, NW

MS 314

Washington, DC 20240

Telephone: (202) 513-7677

Fax: (202) 513-7686

Email: [email protected]

Web: www.fcg.gov

5