BOHA Survey Justification

U.S. Department of the Interior

Office of Policy Analysis



Guidelines for Accessing the Department of the Interior’s Generic Clearance for Customer Satisfaction Surveys



Revised February 2009

ATTACHMENT 1:

Instructions for Completing Supporting Statement for DOI Generic Clearance Submission, OMB Approval Number 1040-0001



1. Survey Title/Date Submitted to the Office of Policy Analysis (PPA): Insert title for the proposed survey. Insert date that the expedited approval package will be submitted to PPA. Reminder: Please submit the package through your bureau/office Information Collection Clearance Officer.

2. Bureau/Office: Insert the name of the bureau/office conducting the survey.

3. Abstract: Summarize the proposed study with an abstract not to exceed 150 words.

4. Bureau/Office Point of Contact Information: Complete the bureau/office contact information. PPA will communicate with the point of contact listed here throughout the entire approval process.

5. Principal Investigator (PI) Conducting the Survey: Complete information about the PI who will be conducting the survey, if different than Point of Contact listed in #4. Otherwise note: Same as #4.

6. Name of Program Office Conducting Survey: Provide the name of the bureau program, office, or organizational unit conducting the survey.

7. Description of Customers/Services Provided: Provide a brief description of the customers who will be surveyed, the services provided by the program conducting the survey, and how these services are provided to customers.

8. Survey Dates: List the time period in which the survey will be conducted, including specific starting and ending dates. The starting date should be at least 45 days after the submission date. The request for expedited approval, and submission of a complete and accurate approval package, must be made at least 45 calendar days prior to the first day the PI wishes to administer the survey instrument to the public.

9. Type of Information Collection Instrument: Check the type(s) of information collection instrument(s) that will be used. If other, please explain.

10. Survey Development: Explain how the survey was developed. With whom did you consult during the development of the survey on content? On statistics? Did you pretest the survey? What actions did you take to improve the survey? What suggestions did you receive for improving the survey? Which of the six topic areas will be addressed? (Note: A description of any pre-testing and peer review of the methods and/or instrument is highly recommended.)

11. Survey Methodology: Explain how the survey will be conducted. Provide a description of the survey methodology including: (a) How will the customers be sampled? (if fewer than all customers will be surveyed); (b) What percentage of customers asked to take the survey will respond, and (c) What actions are planned to increase the response rate? If statistics are generated, this description must be specific and include each of the following:

- The respondent universe,

- The sampling plan and all sampling procedures, including how individual respondents will be selected;

- How the instrument will be administered;

- Expected response rate and confidence levels; and

- Strategies for dealing with potential non-response bias.

Note: Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by the web.

12. Total Number of Initial Contacts/Expected Number of Respondents: Provide an estimated total number of initial contacts and the total number of expected respondents.

13. Estimated Time to Complete Initial Contact/Instrument: Estimate the time to complete the initial contact and the survey instrument (in minutes).

14. Total Burden Hours: Provide the total number of burden hours. The total burden hours should account for the amount of time required to instruct the respondents in completing the survey, and the amount of time required for the respondent to complete the survey.

15. Reporting Plan: Provide a brief description of the reporting plan for the data being collected. A copy of all survey reports must be archived with PPA. Please note this in the reporting plan.

16. Justification, Purpose and Use: Provide a brief justification for the survey, its purpose, goals, and utility to managers. Specifically, describe how data will be tabulated and what statistical techniques will be used to generalize the results to the entire customer population. Describe how data from the survey will be used. Describe how you will acknowledge any limitations related to the data, particularly in cases where we obtain a lower than anticipated response rate. Note whether or not the survey is intended to measure a Government Performance and Results Act (GPRA) performance measure.

ATTACHMENT 2:

Approval Form for DOI Programmatic Clearance for Customer Satisfaction Surveys (OMB Control Number 1040-0001)

U.S. Department of the Interior Office of Policy Analysis (PPA)

PPA Tracking Number: (for PPA use only) CSS-9

Date Submitted to PPA:

01/19/2012

1.

Survey Title:

Visitor Services Project (VSP) Evaluation of Customer Service at Boston Harbor Islands National Park Area (BOHA)

2.

Bureau:

National Park Service

3.

Abstract: (not to exceed 150 words) Boston Harbor Islands National Park Area (BOHA) consists of 34 islands and peninsulas in Boston Harbor, managed by the Boston Harbor Islands Partnership. Currently, no comprehensive customer satisfaction survey data is available to evaluate customer experiences in the Boston Harbor Islands National Park Area. The VSP has been commissioned to conduct a comprehensive customer satisfaction survey of visitors to BOHA. The survey will evaluate: perception of current recreation opportunities evaluation of quality of information provided to visitors satisfaction with customer services The results will provide feedback to BOHA partnership management team to improve customer satisfaction with services and to provide guidelines for interpretive planning.

4.

Bureau/Office Point of Contact Information

First Name:

Sarah

Last Name:

Waterworth

Title:

Program Assistant

Bureau/Office:

Boston Harbor Islands National Recreation Area

Street Address:

408 Atlantic Ave

City:

Boston

State:

MA

Zip code:

02110

Phone:

617-223-8596

Fax:

617-223-8671

Email:

[email protected]

5.

Principal Investigator (PI) Information: Lena Le Phone: 208-885-2585 Fax:208-885-4261 email: [email protected]

6.

Name of Program or Office Conducting Survey:

Park Studies Unit, University of Idaho PO Box 441139 Moscow ID 83844-1139

7.

Description of Customers/ Services Provided:

The customers for this collection will include all recreation visitors to BOHA during summer 2012. Due to the variety of resources, a wide array of services can be grouped into 4 main categories: (1) Transportation services: ferries, water taxis, marinas, moorings; (2) Concession services: food service, campgrounds, bookstores/gift shops; (3) Support services: trails, picnic areas, restrooms; (4) Interpretation services: information services, exhibits, guided programs, special events/programs. The purpose of the survey is to evaluate customer satisfaction with current services provided by BOHA. Although different services are managed and operated by different entities in the partnership, the combination of the services provide a complete experience for visitors to BOHA. Satisfaction level will be segmented to address the differences (if any) in different user groups and services used.

8.

Survey Dates

6/22/2012

to

7/05/2012

9.

Type of Information Collection Instrument (Check ALL that Apply)

X Intercept

__Telephone

Mail

_ Web-based

Focus Groups

__Comment Cards

__Other

Explain:

10. Survey Development:

This questionnaire was developed by the VSP in consultation with the program specialist at BOHA. Some questions in the questionnaire have been previously used and tested in a Visitor Experience and Resource Protection (VERP) Framework study by Manning et al. (2003). The measures of satisfaction are based on a scale recommended by Sudman & Bradburn (1982) and Wittick & Bayer (1994). The questionnaire was revised based on several iterations of internal pretesting and comments from park partners. Questions were removed or rephrased that were deemed leading or biased. We simplified questions to provide clarity and avoid response interference.

The range of questions for this collection will focus on customer satisfaction related to customer services provided at BOHA. The respondents will be asked to provide feedback regarding how well they are satisfied with the different services provided on the islands and their perception of recreational opportunities in the park. Recommendations for improvements will also be asked to contribute to park planning.

Other information such as places visited and activities conducted will also be gathered to segment satisfaction by locations or user groups. Some general and very basic demographic information will be gathered to supplement the satisfaction questions for comparison purposes. We will ask age, zip code, group type, race/ethnicity, number of times visited, and gender to segment the satisfaction by demographic. Sources of information used to find out about activities in the area and visitor awareness of national park will also be asked to assist with the communication and interpretation efforts.

11. Survey Methodology:

(Use as much space as needed; if necessary include additional explanation on separate page).

Respondent Universe

The respondent universe for this collection will be recreational users 18 years old or older who visit BOHA during the survey period. Currently, visitor statistics for BOHA are not available. Current estimation based on ferry records is approximately 200,000 visitors. However, these numbers do not include visitors with private boats, canoes, or kayaks. An observational study will be conducted prior to this survey to better estimate the population size for the various user groups.

Sampling Plan/Procedure

Visitors to BOHA (at least 18 years old) will be intercepted at 16 different locations on 4 peninsulas and 12 islands. For the peninsulas, visitors will be contacted at the entrance point. On islands with docks or piers, visitors will be intercepted at the dock as they enter the islands. On islands with no docks or piers, visitors will be intercepted as they arrive on the beach. During the two-week survey period each location will be surveyed on at least one weekday and one weekend day to reflect different user types. Every 10^th visitor will be intercepted. If a group is encountered, the nearest birthday method will be used to randomize the survey participants. Approximately 1,400 questionnaires will be distributed during the survey period.

Instrument Administration

We will follow the drop-off/pick up method. The selected visitors will be asked if they are interested in participating in the survey. All selected individuals will be asked age, zip code, whether this is their first visit to BOHA, while gender will be observed and recorded. The information will be recorded in a survey log (see attached file) for non-response bias checking. A questionnaire will be distributed to those who agree to participate in the survey with instructions to return the instrument to surveyor before they leave the island. The survey will contain 19 questions in an easy-to-read format. Since the questionnaire will be completed onsite, follow up procedures will not be used.

Expected Response Rate Confidence Levels:

It is our experience that the drop-off/pick-up method generally has a higher response rate than the mail back method, we anticipate at least a 65% response rate for this collection. With the target sample size of 1,400, we anticipate 910 questionnaires will be completed

	Number of Contacts	Expected Response Rate	Expected Number of Responses
Sample size	1,400	65%	910

Strategies for dealing with potential non-response bias

We will test for non-response bias using the information obtained during the initial on-site contact with all selected participants. The surveyors will ask the following questions and record information on a survey log.

• Is this your first time visiting the island?

• What is your age?

• What is your zip code?

• Gender of the participant will be observed and recorded

We will compare survey respondents and non-respondents to determine non-response bias.

Description of any pre-testing and peer review of the methods and/or instrument (recommended)

A survey with similar questions was conducted by Manning et al. (2003). Based on the estimated time for this prior survey and reviews by BOHA staff, we agreed on the estimated burden time of 15 minutes for this collection. We incorporated suggestions, edits, and comments from BOHA staff and our internal review in the final survey.

12.

Total Number of Initial Contacts

Expected Number of Respondents

1,400

910

13.

Estimated Time to Complete Initial Contact (mins.):

Estimated Time to Complete Instrument (mins.):

2 minutes

15 minutes

14.

Total Burden Hours:

274 hours

15. Reporting Plan:

We plan to submit a technical report of the results to the Boston Harbor Islands Partnership and this information will be used in their business plan as well as any short-term improvements of customer services. The report will also be posted online on the Park Studies Unit website at: http:/psu.uidaho.edu/vsp.reports.htm.

16. Justification, Purpose, and Use:
Survey Justification and Purpose	The purpose of the survey is to evaluate customer satisfaction with services provided by BOHA. Although different services are managed and operated by different entities within the partnership, the combination of the services provide a complete experience for visitors to BOHA. Currently, there are no data describing customer satisfaction with any of the services offered at BOHA. Satisfaction will be segmented to address the differences (if any) in different user groups and services used.
Survey Goals	The survey goals are to learn about visitor use of the Boston Harbor Islands and their satisfaction with services and facilities currently provided.
Utility to Managers	The survey will identify use patterns establish the level of satisfaction with current services provided. The results will be used by site managers for business and interpretive planning.
How will the results of the survey be analyzed and used?	General descriptive statistics (i.e., frequency distributions) will be used to analyze the results of the survey. The results will be incorporated into the range of alternatives and use analysis needed for the business and interpretive and planning. Cross tabulations will also be conducted to compare the difference among user group types and services.
How will the data be tabulated? What Statistical Techniques will be used to generalize the results to the entire customer population? How will limitations on use of data be handled? If the survey results in a lower than anticipated response rate, how will you address this when reporting the results? (Use as much space as needed; if necessary include additional explanation on separate page). Based on the survey sample size (n=910), there will be 95% confidence with an estimation error of 3% or less. The data will only be generalizable to Boston Harbor Islands National Park Area visitors. We will not use the data to make comparisons of any users outside the scope of this collection. However, we will conduct comparisons between different user groups responding to the survey, to measure the differences in satisfaction level. The results will give an indication of the current level of use and satisfaction with customer services. A non-response bias check procedure will be conducted using 4 variables (age, zip code, gender, and number of visits). Any observable non-response bias will be noted and reported and the results will be weighted if necessary to counter-balance any bias. Write-ups of results will carefully describe any limitations of the data.
Is this survey intended to measure a Government Performance and Results Act (GPRA) performance measure? If so, please include an excerpt from the appropriate document. (Use as much space as needed; if necessary include additional explanation on separate page). Many of the services in BOHA are managed by non-NPS partner thus the GPRA measurement does not apply to those entities. However, the survey instrument includes two satisfactions question (Q10 and Q11) that can be used as GPRA measures. The results will be compared with the annual Visitor Survey Card results for this park.

References Cited

Dillman, D. A. (2007). Mail and Internet Surveys: The Tailored Design, Second Edition—2007 Update. Hoboken, NJ: John Wiley& Sons.

Manning R., Leung Y., and Budruk M. (2003) Research to Support Management of Visitor Carrying Capacity of Boston Harbor Islands. Boston Harbor Islands National Park Area: Natural Resources Overview, 12: 201-220.

Sudman, S. and Bradburn, N. M. (1982) Asking Questions: A Practical Guide to Questionnaire Design. San Francisco: Jossey-Bass.

Wittink, D. R. and Bayer L. R. (1994). The Measurement Imperative. Marketing Research , 6: 14-22.

ATTACHMENT 3

Checklist for Submitting a Request to Use DOI Programmatic Clearance for Customer Satisfaction Surveys

X All questions in the survey instrument are within the scope of one of the DOI Programmatic Clearance for Customer Satisfaction Surveys topic areas.



X The approval package is being submitted to the Office of Policy Analysis at least 45 days prior to the first day the PI wishes to administer the survey to the public.



X A qualified statistician has reviewed and approved your request.



X Your bureau/office Information Collection Clearance Officer has reviewed and approved the approval package.



The approval package includes:

X A completed Information Form

X A signed Certification Form

X A copy of the survey instrument

X Other supporting materials, such as:

• Cover letters to accompany the questionnaires

• Introductory scripts for initial contact of respondents

• Necessary Paperwork Reduction Act compliance language

• Follow-up letters/reminders sent to respondents



The survey methodology presented on the Information Form includes a specific description of:

X The respondent universe

X The sampling plan and all sampling procedures, including how respondents will be selected

X How the instrument will be administered

X Expected response rate and confidence levels

X Strategies for dealing with potential non-response bias

X A description of any pre-testing and peer review of the methods and/or the instrument is highly recommended.



X The burden hours reported on the Information Form include the number of burden hours associated with the initial contact of all individuals in the sample (i.e., including refusals), if applicable, and the number of burden hours associated with individuals expected to complete the survey instrument.

X The package is properly formatted (Word) and submitted to the Office of Policy Analysis electronically. 

ATTACHMENT 4

CERTIFICATION FORM FOR SUBMISSION UNDER OMB CONTROL NUMBER 1040-0001

This form should only be used if you are submitting a collection of information for approval under the DOI Programmatic Clearance for Customer Satisfaction Surveys.

If the collection does not satisfy the requirements of the Programmatic Clearance, you should follow the regular PRA clearance procedures described in 5 CFR 1320.

Bureau/Office Subgroup or Program: Park Studies Unit- University of Idaho
Title Visitor Services Project Evaluation of Customer Service at Boston Harbor Islands National Recreation Area
Burden Hour Initial contact Completing the survey		47 hours 227 hours		Estimated Number of Respondents Initial contact Completing the survey			1400 910
Total Burden Hours		274 hours		Hours/Min per Response Initial Contact Completing the survey			2 minutes 15 minutes
Bureau/Office Contact (who can best answer questions about content of the submission):
Name	Lena Le		Phone		208-885-2585
Certification: The collection of information requested by this submission meets the requirements of OMB control number 1040-0001
Bureau/Office Qualified Statistician Lena Le						DATE 1/19/2012
Bureau/Office Information Collection Clearance Officer Phadrea D. Ponds						DATE 1/19/2012
Office of Policy Analysis/DOI Information Collection Clearance Officer Donald J. Bieniewicz						DATE 2/09/2012
OMB, Office of Information and Regulatory Affairs (OIRA)						DATE

1