Justification for Palo Alto Field Trip Evaluation

U.S. Department of the Interior

Office of Policy Analysis



Guidelines for Accessing the Department of the Interior’s Generic Clearance for Customer Satisfaction Surveys



Revised February 2009

ATTACHMENT 1:

Instructions for Completing Supporting Statement for DOI Generic Clearance Submission, OMB Approval Number 1040-0001



1. Survey Title/Date Submitted to the Office of Policy Analysis (PPA): Insert title for the proposed survey. Insert date that the expedited approval package will be submitted to PPA. Reminder: Please submit the package through your bureau/office Information Collection Clearance Officer.

2. Bureau/Office: Insert the name of the bureau/office conducting the survey.

3. Abstract: Summarize the proposed study with an abstract not to exceed 150 words.

4. Bureau/Office Point of Contact Information: Complete the bureau/office contact information. PPA will communicate with the point of contact listed here throughout the entire approval process.

5. Principal Investigator (PI) Conducting the Survey: Complete information about the PI who will be conducting the survey, if different than Point of Contact listed in #4. Otherwise note: Same as #4.

6. Name of Program Office Conducting Survey: Provide the name of the bureau program, office, or organizational unit conducting the survey.

7. Description of Customers/Services Provided: Provide a brief description of the customers who will be surveyed, the services provided by the program conducting the survey, and how these services are provided to customers.

8. Survey Dates: List the time period in which the survey will be conducted, including specific starting and ending dates. The starting date should be at least 45 days after the submission date. The request for expedited approval, and submission of a complete and accurate approval package, must be made at least 45 calendar days prior to the first day the PI wishes to administer the survey instrument to the public.

9. Type of Information Collection Instrument: Check the type(s) of information collection instrument(s) that will be used. If other, please explain.

10. Survey Development: Explain how the survey was developed. With whom did you consult during the development of the survey on content? On statistics? Did you pretest the survey? What actions did you take to improve the survey? What suggestions did you receive for improving the survey? Which of the six topic areas will be addressed? (Note: A description of any pre-testing and peer review of the methods and/or instrument is highly recommended.)

11. Survey Methodology: Explain how the survey will be conducted. Provide a description of the survey methodology including: (a) How will the customers be sampled? (if fewer than all customers will be surveyed); (b) What percentage of customers asked to take the survey will respond, and (c) What actions are planned to increase the response rate? If statistics are generated, this description must be specific and include each of the following:

- The respondent universe,

- The sampling plan and all sampling procedures, including how individual respondents will be selected;

- How the instrument will be administered;

- Expected response rate and confidence levels; and

- Strategies for dealing with potential non-response bias.

Note: Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by the web.

12. Total Number of Initial Contacts/Expected Number of Respondents: Provide an estimated total number of initial contacts and the total number of expected respondents.

13. Estimated Time to Complete Initial Contact/Instrument: Estimate the time to complete the initial contact and the survey instrument (in minutes).

14. Total Burden Hours: Provide the total number of burden hours. The total burden hours should account for the amount of time required to instruct the respondents in completing the survey, and the amount of time required for the respondent to complete the survey.

15. Reporting Plan: Provide a brief description of the reporting plan for the data being collected. A copy of all survey reports must be archived with PPA. Please note this in the reporting plan.

16. Justification, Purpose and Use: Provide a brief justification for the survey, its purpose, goals, and utility to managers. Specifically, describe how data will be tabulated and what statistical techniques will be used to generalize the results to the entire customer population. Describe how data from the survey will be used. Describe how you will acknowledge any limitations related to the data, particularly in cases where we obtain a lower than anticipated response rate. Note whether or not the survey is intended to measure a Government Performance and Results Act (GPRA) performance measure.

ATTACHMENT 2:

Approval Form for DOI Programmatic Clearance for Customer Satisfaction Surveys (OMB Control Number 1040-0001)

U.S. Department of the Interior Office of Policy Analysis (PPA)

PPA Tracking Number: (for PPA use only) CSS-7

Date Submitted to PPA:

11/15/2011

1.

Survey Title:

Palo Alto Field Trip Evaluation

2.

Bureau:

National Park Service – Palo Alto Battlefield National Historical Park (PAAL)

3.

Abstract: (not to exceed 150 words) This information collection will be used to evaluate teachers and students who participate in school field trips held at Palo Alto Battlefield National Historical Park (PAAL). Two forms will be used to collect information. One will be completed by the lead teacher on the field trip and the other will be administered by the teacher in the classroom to the students following the completion of the field trip. Both evaluation forms will be completed off-site (in the classroom) and the teachers will return all the completed forms in a self-addressed stamped envelope provided by PAAL interpretative staff. The collection will evaluate the following: 1) Student learning and 2) The effectiveness of the activities encountered during the guided field trip. The results will allow PAAL interpretative staff and Park managers to assess the quality and impact of the ranger-led field trips based on the feedback from the students and teachers.

4.

Bureau/Office Point of Contact Information

First Name:

Karen

Last Name:

Weaver

Title:

Park Ranger

Bureau/Office:

NPS, Palo Alto Battlefield National Historical Park (PAAL)

Street Address:

1623 Central Blvd., Rm. 213

City:

Brownsville

State:

TX

Zip Code:

78520

Phone:

956-541-2785

FAX:

956-574-0062

Email:

[email protected]

5.

Principal Investigator (PI) Information (same as above)

6.

Name of Program or Office Conducting Survey:

Palo Alto Battlefield National Historical Park

7.

Description of Customers/ Services Provided:

Elementary school teachers and their students attending ranger-led field trips at Palo Alto Battlefield National Historical Park

8.

Survey Dates

11-01-2011

to

11-01-2013

9.

Type of Information Collection Instrument (Check ALL that Apply)

__Intercept

__Telephone

Mail

Web-based

Focus Groups

__Comment Cards

_X_ Other

Explain:

Evaluation Forms

10. Survey Development:

The evaluation forms were developed based upon the experience and need for information by PAAL interpretative staff. PAAL received in-kind assistance from a professional survey development company (Performance Dimensions International, LCC) to refine the questions we initially designed for the evaluation forms.

11. Survey Methodology:

(Use as much space as needed; if necessary include additional explanation on separate page).

Respondent Universe

The respondent universe for this collection will be Elementary Teachers (n = 72) and their students (n=1,800) who visit PAAL on scheduled ranger-led field trips.

Sampling Plan/Procedure

At the end of each field trip, teachers will be given a packet with evaluation forms, instructions, and a self-addressed envelope. The teachers will provide the forms to the students within 5 days of their visit to PAAL. All students will be asked to complete an evaluation form that will be easy to read and take less than 5 minutes to complete.

PAAL will provide evaluation forms for 100% of the teachers and students who attend and participate in the field trip.

Instrument Administration

Expected Response Rate Confidence Levels:

	Number of Contacts	Expected Response Rate	Expected Number of Responses
Teachers	72	80-100%	72
Students	1,800	80-100%	1,800

Strategies for dealing with potential non-response bias

PAAL anticipates that the teachers will administer the evaluation forms as an activity that culminates the students’ field trip experience. Because the evaluation is used as a classroom assignment, the students are more likely to complete the form. In the event that PAAL does not receive information within 10 days after the field trip, PAAL staff will send a reminder message to the teacher asking to return the evaluation forms.

Description of any pre-testing and peer review of the methods and/or instrument (recommended)

PAAL staff consulted with other National Park Service interpretative employees who conduct field trips as well as local educators to review the forms for accuracy, desired results, delivery, quality, and value. The reviewers also agreed on the average time to complete the teacher and student evaluation forms. We incorporated their suggested grammatical edits and comment concerning the formatting of the forms in the final versions.

12.

Total Number of Initial Contacts

Expected Number of Respondents

Teachers: 72

Students: 1,800

13.

Estimated Time to Complete Initial Contact Instrument (mins.):

Teachers: 2 minutes

Students: 5 minutes

14.

Total Burden Hours:

Teachers: 2 Hours

Students: 150 Hours

TOTAL: 152 Hours

15. Reporting Plan:

The data collected will first be tallied and entered into a database. Likert Scale information will be statistically analyzed and compared. Open-ended questions will be grouped into common themes/categories and will be described as qualitative feedback in the final report. A final report of this information will be produced and archived both in official park files and with the NPS Social Science Program.

16. Justification, Purpose, and Use:
Survey Justification and Purpose	The survey is needed to evaluate the: Effectiveness of PAAL’s current reservation system; Usefulness of pre-visit materials; Value of interpretative messages presented during field trips; Impact of PAAL staff as field guides; and Satisfaction with the overall field trip experience. PAAL interpretative staff will use the results to develop additional opportunities to enhance the park’s educational and public outreach efforts.
Survey Goals	PAAL interpretative staff needs clear and up-to-date information concerning the impact of the park’s outdoor learning opportunities have on elementary students and their teachers. Ultimately, PAAL interpretative staff would like to assess students’ understanding of the materials presented during ranger-led field trips and determine which activities are most effective as educational tools that students remember after their on-site field trip.
Utility to Managers	The evaluation will identify those activities that impact students after participating in ranger-led field trips. PAAL interpretive staff and park managers will use this information to better understand the needs of teachers and students during outdoor education learning experiences. The information collected will be used to re-design field trips (as needed) to ensure they are a quality learning experience for all students who participate.
How will the results of the survey be analyzed and used?	General descriptive statistics (i.e., frequency distributions) will be used to analyze survey results. Open-ended questions will be grouped into common themes/categories and will be analyzed. A final report of this information will be produced. Park Rangers will receive the raw data files and a report describing the analyzed data.
How will the data be tabulated? What Statistical Techniques will be used to generalize the results to the entire customer population? How will limitations on use of data be handled? If the survey results in a lower than anticipated response rate, how will you address this when reporting the results? (Use as much space as needed; if necessary include additional explanation on separate page). Because the entire population of potential school groups visiting PAAL each year cannot be determined with a high degree of accuracy, there will be some limitations in the results. The results will not be generalized to the entire population of PAAL visitors, but will represent the self-selected school groups participating in Ranger-led field trips. Results will indicate the current level of satisfaction with the entire field trip experience. Due to the nature of the respondents (engaged and active participants in the program), PAAL staff anticipates that they will be highly motivated to complete the survey and provide information that will be useful to PAAL managers and interpretative staff. As these respondents will not represent all student groups, results will be presented only in terms of active participants using either counts or percentages.
Is this survey intended to measure a Government Performance and Results Act (GPRA) performance measure? If so, please include an excerpt from the appropriate document. (Use as much space as needed; if necessary include additional explanation on separate page). This survey measures the following GPRA goals: IIB1 – Visitor Understanding and Appreciation IIB1X - Educational Programs – Student Understanding

ATTACHMENT 3

Checklist for Submitting a Request to Use DOI Programmatic Clearance for Customer Satisfaction Surveys

X All questions in the survey instrument are within the scope of one of the DOI Programmatic Clearance for Customer Satisfaction Surveys topic areas.



X The approval package is being submitted to the Office of Policy Analysis at least 45 days prior to the first day the PI wishes to administer the survey to the public.



X A qualified statistician has reviewed and approved your request.



X Your bureau/office Information Collection Clearance Officer has reviewed and approved the approval package.



The approval package includes:

X A completed Information Form

X A signed Certification Form

X A copy of the survey instrument

X Other supporting materials, such as:

• Cover letters to accompany mail-back questionnaires

• Introductory scripts for initial contact of respondents

• Necessary Paperwork Reduction Act compliance language

• Follow-up letters/reminders sent to respondents



The survey methodology presented on the Information Form includes a specific description of:

X The respondent universe

X The sampling plan and all sampling procedures, including how respondents will be selected

X How the instrument will be administered

X Expected response rate and confidence levels

X Strategies for dealing with potential non-response bias

X A description of any pre-testing and peer review of the methods and/or the instrument is highly recommended.



X The burden hours reported on the Information Form include the number of burden hours associated with the initial contact of all individuals in the sample (i.e., including refusals), if applicable, and the number of burden hours associated with individuals expected to complete the survey instrument.

X The package is properly formatted (Word) and submitted to the Office of Policy Analysis electronically. 

ATTACHMENT 4

CERTIFICATION FORM FOR SUBMISSION UNDER OMB CONTROL NUMBER 1040-0001

This form should only be used if you are submitting a collection of information for approval under the DOI Programmatic Clearance for Customer Satisfaction Surveys.

If the collection does not satisfy the requirements of the Programmatic Clearance, you should follow the regular PRA clearance procedures described in 5 CFR 1320.

Bureau/Office Subgroup or Program NPS, Palo Alto Battlefield National Historical Park (PAAL)
Title (Please be specific) Karen Weaver - Palo Alto Battlefield National Historical Park (PAAL) - Park Ranger
Burden Hour		Teachers: 2 Hours Students: 150 Hours		Estimate Number of Respondents			Teachers: 72 Students: 1,800
Total Burden Hours		152 hours		Hours/Min per Response			Teachers: 2 minutes Students: 5 minutes
Bureau/Office Contact (who can best answer questions about content of the submission):
Name	Karen Weaver		Phone		956-541-2785
Certification: The collection of information requested by this submission meets the requirements of OMB control number 1040-0001
Bureau/Office Qualified Statistician Dr. Lena Le						DATE 11/9/2011
Bureau/Office Information Collection Clearance Officer Phadrea D. Ponds						DATE 11/15/2011
Office of Policy Analysis Don Bieniewicz						DATE 11/22/2011
OMB, Office of Information and Regulatory Affairs (OIRA)						DATE

1