1949.05 Part B

INFORMATION COLLECTION REQUEST

FOR EPA NATIONAL ENVIRONMENTAL PERFORMANCE TRACK PROGRAM

ICR 1949.05

Part B of the ICR Supporting Statement for the Performance Track Customer Satisfaction Questionnaire

1. Objectives, Key Variables, and Other Preliminaries

1. Objectives

EPA is interested in improving and increasing the value that Performance Track members receive from the Performance Track program. Members’ opinions about the program's effectiveness and level of service are used to shape the development of future member services and benefits. The Performance Track Customer Satisfaction Questionnaire is designed to assess the satisfaction level of current members as well as identify improvements to services that the program can implement to increase long-term member satisfaction. More specifically, the questionnaire’s objectives are to accurately ascertain the following information: program benefits and services that are important to members; member satisfaction with current services; potential improvements in communicating with members about the program; the level of promotion/publicity that members desire for their participation in the program; and any additional benefits and services that would increase member satisfaction.

2. Key Variables

Performance Track members reflect a broad array of industry categories, company size, facility size, geography, regulatory requirements, and length of time in the program. In addition, their reasons for joining the program, the program elements that they find most and least valuable, and the program’s effects on their operations can vary considerably from member to member.

3. Statistical Approach

The Performance Track program has decided that a census of all of its current members (417 facilities as of January 1, 2007) is the best way to meet the objectives described above. To better understand the variables that influence members’ satisfaction, the Performance Track program is interested in the questionnaire responses of numerous subpopulations within the membership. Many of these subpopulations are comprised of relatively few facilities. In addition, the questionnaire results may influence decisions to investigate smaller segments of the membership not previously considered. It may not be possible to determine ahead of time the size of some of these segments as they may be based upon members' responses to particular questions in the questionnaire. For example, if the questionnaire results show a significant number of members are dissatisfied with a particular service provided by the program, the Performance Track program may decide to look more closely at other characteristics of these facilities. A census of the entire population, therefore, will result in greater certainty in the overall results and will allow flexibility in analyzing various subpopulations identified after the questionnaire is administered.

4. Feasibility

The feasibility and effectiveness of the Customer Satisfaction Questionnaire were demonstrated in 2004 and 2006. In 2004, the questionnaire was successfully administered and the results were used to identify priority areas for program improvements and an action plan was developed and implemented to address these priority areas. The results from 2006 are currently being analyzed and EPA anticipates developing another action plan to further improve the program’s effectiveness. In both 2004 and 2006, the respondents provided valuable feedback through their answers to the specific questions included in the questionnaire, meeting the objectives of this information collection effort.

The content and method of administering the 2008 Customer Satisfaction Questionnaire will be similar to those of 2004 and 2006. The questionnaire will again be administered via the Internet and the results will be used to identify priority areas for improvement and to guide the development of an action plan.

2. Survey Design

1. Target Population and Coverage

To better understand and respond to the needs of Performance Track members, the questionnaire will be targeted at the current Performance Track membership. The questionnaire coverage will be across the entire population.

(b) Sample Design

As the questionnaire will be administered to the entire population of the Performance Track Membership, sample design does not apply to this study.

(c) Precision Requirements

The Performance Track program expects to achieve a 90 percent confidence interval of within five percent or less of the “true” value. Because the questionnaire will be administered as a census, sampling error will not be a factor in precision. The largest source of error will result from non-response bias. To minimize non-response bias, we will take a number of steps to obtain a high questionnaire response rate. These steps include: ensuring a complete and accurate list of member contact information, minimizing the burden associated with the questionnaire, assuring respondents that their answers will not be linked to their identity, notifying the membership of the importance of the questionnaire, and numerous reminders to members to complete the questionnaire. Survey response rates for 2004 and 2006, where 78 and 73 percent, respectively.

5. Questionnaire Design

The 2008 Performance Track Customer Satisfaction Questionnaire has not yet been finalized, but it’s content will likely be similar to the questionnaires administered in 2004 and 2006. The justification for the anticipated questions is discussed in detail in Part A of the Supporting Statement. The questionnaire will be a mix of multiple choice and open-ended questions that minimize burden on the respondents, while achieving the information collection objectives. Multiple-choice questions are used whenever possible, and open-ended questions are typically asked only when the list of possible responses is too long for a multiple-choice format, or the possible responses are yet unknown.

3. Pretests and Pilot Tests

A draft version of the on-line questionnaire will be tested on up to nine Performance Track members before finalizing the survey questions. The Performance Track program will solicit volunteer members deemed to be among the more active in the program and who are likely to take the time to provide a thoughtful assessment of the questionnaire. Pretesters will also be selected so that they represent a range of different industry sectors, facility sizes, and lengths of time in the program. Before completing the draft questionnaire, the pretesters will be instructed to note any technical problems with the survey instrument and to look for questions that are unclear or misleading. After the pretesters have completed the survey, the database of their responses will be examined to ensure that all responses are being properly recorded. The final version of the survey instrument will be revised to reflect any significant findings of the pretest.

4. Collection Methods and Follow-up

(a) Collection Methods

The Customer Satisfaction Questionnaire is administered every two years, via an online questionnaire in order to reduce the burden on respondents and encourage a high response rate. Notification is sent to all current members along with passwords to allow them to access the survey. The questionnaire is administered for one month. Reminder emails are sent weekly and in the fourth week telephone calls are made to each nonresponding member to further encourage their response.

(b) Survey Response and Follow-up

To reduce non-response bias, a response rate of over 70 percent of the Performance Track membership is desired. No follow-up to the survey is anticipated.

5. Analyzing and Reporting Survey Results

1. Data Preparation

Data from the questionnaire responses are automatically entered into a database as members complete them online. The instrument will be programmed to prompt respondents to complete the entire survey and will not allow them to skip individual questions. Data from any incomplete questionnaires will be automatically input into the database and will be presented and analyzed along with that of fully completed questionnaires. A contractor will be used to develop and program the questionnaire instrument and database and to process the data and develop a report of the findings.

2. Analysis

Responses will be summarized and analyzed at the conclusion of the one-month survey period. For each multiple-choice question, the number and percent of respondents answering will be presented. Open-ended questions will be coded and presented as number and percent responding within each coded category. Regression analyses are not anticipated. Depending upon the responses to the questions, segments of the population may be analyzed and presented separately.

3. Reporting Results

The results of the questionnaire will be presented in a report that will be available to the Agency, Performance Track members, and the general public via the Performance Track website. In accordance with the privacy statement, the full database with facility identifiers will not be made available to Agency personnel or to the public. Tailored numerical reports of the findings can be produced in response to inquiries.

4