Guidelines for Submission of a Survey

U.S. Department of the Interior

Office of Policy Analysis

Guidelines for Submission of a Survey under the DOI Programmatic Clearance for Customer

Satisfaction Surveys

Revised March 2012

Purpose

These guidelines are intended to help Department of the Interior (DOI) offices and bureaus successfully use the DOI Programmatic Clearance for Customer Satisfaction Surveys to collect customer satisfaction data.

Why Do We Need to Collect Customer Satisfaction Data?

DOI offices and bureaus must collect customer satisfaction data to comply with the Government

Performance and Results Act (GPRA) of 1993 (P.L. 103-62), Executive Order (E.O.) 12862, and Executive Order 13571. The DOI must measure customer satisfaction levels associated with its services, products, and information through periodic surveys, face-to-face meetings, and frequent communication with our customers. Data obtained from these customer interactions can be used to identify opportunities for improvement. Comment cards, questionnaires, and other survey methods are viable tools for collecting valuable customer information.

What’s Special about the Programmatic Clearance?

The Programmatic Clearance enables DOI offices and bureaus to collect customer information using a streamlined process for approving the research instruments used in the data collection. The Paperwork Reduction Act (PRA) of 1995 requires Federal agencies to obtain approval from the Office of Management and Budget (OMB) before they can collect information from the general public. The PRA defines “collection of information” quite broadly. It covers any identical questions posed to 10 or more members of the public, whether voluntary or mandatory, written, electronic, or oral. The term “public” does apply to State, local, and tribal governments. It does not apply to Federal agencies as long as representatives of Federal agencies are being surveyed as such and not as private citizens. The standard PRA compliance process used by agencies to obtain approval from OMB to collect customer satisfaction information typically takes more than 120 days. The Programmatic Clearance significantly reduces the time needed to obtain OMB approval for use of instruments covered by the Programmatic Clearance. Approvals can be obtained in as few as 45 days.

Approved Customer Research Tools

The DOI has received approval from OMB for its Programmatic Clearance under OMB control #1040-0001, so that DOI bureaus and offices can conduct customer research using the following tools:

In-person intercept surveys: In a face-to-face situation, the survey instrument is provided to a

respondent who completes it while on site and then returns it. This may include oral administration or the use of electronic technology and kiosks. The survey proctor is prepared to answer any questions the respondent may have about how to fill out the instrument but does not interfere or influence how the respondents answer the questions.

Telephone interviews or questionnaires: Using existing databases, an interviewer will contact customers who have had a specific experience with the agency. The interviewer will dial back until the customer has been reached. Once contacted, the survey respondent is given a brief introduction to the survey, including its importance and use. The interviewer will then expeditiously move through the survey questions.

Mail and e-mail surveys: Using existing lists of customer addresses, a three-contact approach based on Dillman's “Tailored Design Method” will be employed. The first contact is a cover letter explaining that a survey is coming to them and why it is important to the agency. The second contact will be the survey instrument itself along with a postage-paid addressed envelope to return the survey. The third contact will be a reminder postcard sent 10 days after the survey was sent. Finally, the respondents will receive a letter thanking them for the willingness to participate in the survey and reminding them to return it if they have not already done so. At each juncture, the respondents will be given multiple ways to contact someone with questions regarding the survey (including phone, FAX, web, and email). If the survey has been lost, the respondent can request that another be sent to them. Email is sometimes used instead of postal mail to communicate with customers. Although this is a cost-effective mode to survey a large group of people, it does not usually generate the best response rate. Telephone calls to non-respondents can be used to increase response rates.

Web-based: For products or services that are provided through electronic means, whether e-commerce or web-based information, a web or email survey may be most appropriate. During the course of their web interaction, users can volunteer to add their name to a list of future surveys. From this list, a respondent pool will be selected in accordance with the sampling procedures outlined above. An email will be sent to them explaining the need and importance of the survey with a web link to the survey. Within 5 days, a follow-up email will be sent to the respondents reminding them to complete the survey. Finally, the respondents will receive an email thanking them for the willingness to participate in the survey and reminding them to complete it if they have not already. The respondent will always have the option to submit the survey in paper form, should they elect to do so.

Focus groups: Some data and information are best collected through more subjective, conversational means. A focus group is an informal, small-group discussion designed to obtain in-depth qualitative information. Individuals are specifically invited to participate in the discussion, whether in person or through technologically enhanced means (e.g., video conferencing, on-line sessions). Participants are encouraged to talk with each other about their experiences, preferences, needs, observations, or perceptions. A moderator whose role is to foster interaction leads the conversation. The moderator makes sure that all participants are encouraged to contribute and that no individual dominates the conversation. Furthermore, the moderator manages the discussion to make sure it does not stray too far from the topic of interest. Focus groups are most useful in an exploratory stage or when the bureau/office wants to develop a deeper understanding of a program or service. Using the best in focus group research practices, groups will be constructed to include a cross-section of a given customer group. The questions and additional probes used during the focus groups will be consistent with the “guideline menu.”

Comment Cards: Comment cards, when provided to a customer at the time a product or service is provided, offer an excellent means to give the bureaus and offices feedback. A comment card should have a limited number of questions and an opportunity to comment. These comment cards provide managers and service providers with direct and specific information from their customers that could not be obtained through any other means. Electronic users may be offered the opportunity to complete a comment card via a “pop-up” window (or other web-enabled means that may be available). The “pop-up” window will not appear for every user; rather, the users will be selected randomly to receive the survey. This practice is widely used in private

industry. In other instances, the electronic user may be offered the option to self-select in answering the electronic comment card. Whether using paper or electronic comment cards, the intent is to provide a feedback mechanism. The data are not intended to be statistically significant. Although questions may include numeric scales, those data should be considered only in an anecdotal fashion and not reported as a significant measure.

OMB Guidance on Survey Design

All surveys must be designed and implemented in accordance with OMB "Guidance on Agency Survey and Statistical Information Collections (January 20, 2006)." The OMB guidance document, entitled "Questions And Answers When Designing Surveys For Information Collections," can be obtained from the OMB web site at the following link:

http://www.whitehouse.gov/omb/inforeg/statpolicy.html

Types of Questions Covered by the Programmatic Clearance

The DOI Programmatic Clearance was approved for customer satisfaction surveys. Therefore, survey instruments that will be approved under the authority of the Programmatic Clearance must focus on customer satisfaction data. It is important to note that no information collection instruments seeking to collect information beyond the scope of customer satisfaction data will be considered under the scope of this Programmatic Clearance. This includes information collections designed to obtain social science or visitor use information. To be considered under the scope of this Programmatic Clearance, information collection instruments are restricted to the following topic areas. Although no one survey will cover all the topic areas, these should be viewed as a “guideline menu” from which bureaus and offices will develop their survey questions.

1. Delivery, quality, and value of products, information, and services:

The range of questions envisioned for this topic area will focus on customer satisfaction with aspects of information, products, and related services offered by DOI. Information, products, and services include written reports, press releases, computer modules, workshops and seminars, or technical assistance. Respondents may be asked for feedback regarding the following attributes of the information, service, and products provided:

Timeliness (Was the information, service, product provided to you in a timely manner?)

(Was the information itself timely?)

Consistency (Was the quality of the service consistent with your expectations?)

Accuracy (Were the data provided accurate?)

Ease of Use and Usefulness (Was the product easy to use? Was the information useful to

you?)

Ease of Information Access (Were you able to find the information you needed easily?)

Helpfulness (Was the information helpful?)

Quality (Was the information of high quality?)

Value for fee paid for information/product/service (Was the cost of the product, information, or service appropriate for the value received?)

2. Management practices:

This area covers questions relating to how well customers are satisfied with DOI management

practices and processes, what improvements they might make to specific processes, and whether or not they feel specific issues were addressed and reconciled in a timely, courteous, responsive

manner. Questions within this area may involve feedback regarding how well DOI engaged

customers on a specific topic. They may also seek opinions from customers regarding how well

DOI programs are managing and administering specific processes (for example, the Bureau of

Land Management may ask customers how well it is administering its permitting processes.)

3. Mission management:

Questions will ask customers and partners to provide satisfaction data related to DOI’s ability to

protect, conserve, provide access to, provide scientific data about, and preserve natural, cultural, and recreational resources that we manage, and how well we are carrying out our trust responsibilities to American Indians, Native Alaskans, and Insular Areas. Questions will specifically ask customers and partners to provide satisfaction data related to each of its four mission areas as described in its GPRA Strategic Plan: Resource Use, Resource Protection, Serving Communities, and Recreation.

4. Rules, regulations, policies:

This area focuses on obtaining feedback from customers regarding fairness, adequacy, and

consistency in enforcing rules, regulations, and policies for which DOI is responsible. It will also help us understand public awareness of rules and regulations and whether or not they are articulated in a clear and understandable manner. It will not seek opinions from customers regarding the appropriateness of regulatory rulings themselves.

5. Interactions with DOI Personnel and Contractors:

Questions developed under this topic area will focus on obtaining customer feedback regarding

attributes of interactions with DOI office and bureau employees, as well as DOI contractors.

Attribute questions will range from timeliness and quality of interactions to skill level of staff

providing the assistance, as well as their courtesy and responsiveness during the interaction.

6. General demographics:

Some general demographics may be gathered to augment satisfaction questions in order to better

understand the customer so that we can improve how we serve that customer. Demographics data will range from asking customers how many times they have used a DOI service or visited a DOI facility within a specific timeframe, to their ethnic group and race. Sensitivity and prudence

will be used in developing and deploying questions under this topic area so that the customer does not perceive an intrusion upon his/her privacy. Additionally, these questions will only be asked as long as the data are critical to understanding customer satisfaction and the character of the customer base. Demographics may also be used as part of a non-response bias strategy to ensure responses are representative of the contact universe.

How to Use the Programmatic Clearance

To use the Programmatic Clearance, bureaus and offices must ensure that the proposed survey instrument is consistent with the intent of the Programmatic Clearance; i.e., that the instrument focuses on obtaining customer satisfaction data. For each proposed survey, a bureau or office must prepare a Justification Form and send that Form and the proposed survey instrument to a qualified statistician for review and approval. The name of the statistician who reviewed and approved the package, along with the date of his/her approval, must be added to the Certification portion of the Justification. Next the survey package must be reviewed by the bureau or office Information Collection Clearance Officer, who must also certify that the survey instrument complies with the terms of the Programmatic Clearance. The reviewed and approved package may then be submitted to the Office of Policy Analysis.

Each package must include the following:

A copy of the entire survey instrument;

A completed Justification Form, including the certifications of the statistician and bureau Information Collection Clearance Officer; and,

Other supporting materials, including cover letters, introductory scripts (primarily for focus groups), and follow-up letters (aimed at encouraging response).

Timelines for Requesting Approvals

The request for approval under the Programmatic Clearance and submission of a complete and accurate approval package should be made to the Office of Policy Analysis at least 45 calendar days prior to the first day the bureau/office plans to administer the survey instrument to the public. The Office of Policy Analysis will provide an administrative and technical review of the submitted materials and notify the requesting bureau/office of the results. The Office of Policy Analysis will work with the bureau/office to make any necessary revisions before submitting the package to OMB for approval. Once OMB approves the instrument, the bureau/office will be notified. No survey instruments may be administered until OMB approval is received.

Request and Approval Process Steps

The steps in the Programmatic Clearance approval process are, as follows:

Step 1

The requesting bureau/office completes the Justification Form.

Step 2

The bureau/office submits a copy of the entire approval package to a qualified statistician for review and approval of the instrument and survey methodology. The package must include:

(a) the completed Justification Form;

(b) any introductory script used in contacting the public;

(c) all cover letters, postcard reminders or follow-up letters to be sent to potential respondents;

(d) the entire survey instrument;

(e) necessary Paperwork Reduction Act (PRA) compliance language inserted into the survey instrument; and

(f) any other supporting materials.

For face-to-face information collections, such as interviews and focus groups, a short statement

describing how the bureau/office intends to communicate PRA compliance information to respondents is required in the description of the survey methodology.

Step 3

The bureau/office submits a copy of the entire approval package, including the certification by the statistician, to the bureau/office Information Collection Clearance Officer.

Step 4

Once both a qualified statistician and the bureau/office Information Collection Clearance Officer have reviewed and approved the package, it may be submitted to the Office of Policy Analysis (PPA). The entire package should be submitted electronically and be formatted in Word. .pdf files of some information collection instruments may be accepted in lieu of Word.

Step 5

PPA assigns a tracking number and confirms receipt of the package with an email to the bureau/office point of contact.

Step 6

PPA staff members conduct an administrative and technical review of the submission. The staff

recommends: (a) approval, (b) revision, (c) resubmission under the standard Paperwork Reduction Act approval process, or (d) rejection of the proposed survey. If the package is not approved, PPA will notify the originating office/bureau and coordinate revision and resubmission.

Step 7

Once approved, PPA staff transmits the package to OMB for final approval.

Step 8

OMB reviews the submission and notifies PPA of approval or necessary revisions.

Step 9

Once the submission is approved by OMB, PPA will notify the requesting bureau/office. If OMB attaches any special conditions to the approval, the requesting bureau/office will be informed and the conditions must be met in the conduct of the survey. Should OMB reject the submission or have specific questions about the survey instrument, PPA will immediately inform the requesting bureau/office. The bureau/office may submit an appeal of the OMB decision to PPA, in writing. PPA will submit the appeal to OMB and inform the bureau/office of the results.

Step 10

The bureau/office conducts the approved survey. Additions or changes to a survey instrument after it has been approved, even within the specific topic areas, are not allowed by OMB. An exception is that questions may be deleted after approval by OMB, if necessary.