Justification for BOWA Questionnaires

U.S. Department of the Interior

Office of Policy Analysis



Guidelines for Accessing the Department of the Interior’s Generic Clearance for Customer Satisfaction Surveys



Revised February 2009

ATTACHMENT 1:

Instructions for Completing Supporting Statement for DOI Generic Clearance Submission, OMB Approval Number 1040-0001



1. Survey Title/Date Submitted to the Office of Policy Analysis (PPA): Insert title for the proposed survey. Insert date that the expedited approval package will be submitted to PPA. Reminder: Please submit the package through your bureau/office Information Collection Clearance Officer.

2. Bureau/Office: Insert the name of the bureau/office conducting the survey.

3. Abstract: Summarize the proposed study with an abstract not to exceed 150 words.

4. Bureau/Office Point of Contact Information: Complete the bureau/office contact information. PPA will communicate with the point of contact listed here throughout the entire approval process.

5. Principal Investigator (PI) Conducting the Survey: Complete information about the PI who will be conducting the survey, if different than Point of Contact listed in #4. Otherwise note: Same as #4.

6. Name of Program Office Conducting Survey: Provide the name of the bureau program, office, or organizational unit conducting the survey.

7. Description of Customers/Services Provided: Provide a brief description of the customers who will be surveyed, the services provided by the program conducting the survey, and how these services are provided to customers.

8. Survey Dates: List the time period in which the survey will be conducted, including specific starting and ending dates. The starting date should be at least 45 days after the submission date. The request for expedited approval, and submission of a complete and accurate approval package, must be made at least 45 calendar days prior to the first day the PI wishes to administer the survey instrument to the public.

9. Type of Information Collection Instrument: Check the type(s) of information collection instrument(s) that will be used. If other, please explain.

10. Survey Development: Explain how the survey was developed. With whom did you consult during the development of the survey on content? On statistics? Did you pretest the survey? What actions did you take to improve the survey? What suggestions did you receive for improving the survey? Which of the six topic areas will be addressed? (Note: A description of any pre-testing and peer review of the methods and/or instrument is highly recommended.)

11. Survey Methodology: Explain how the survey will be conducted. Provide a description of the survey methodology including: (a) How will the customers be sampled? (if fewer than all customers will be surveyed); (b) What percentage of customers asked to take the survey will respond, and (c) What actions are planned to increase the response rate? If statistics are generated, this description must be specific and include each of the following:

- The respondent universe,

- The sampling plan and all sampling procedures, including how individual respondents will be selected;

- How the instrument will be administered;

- Expected response rate and confidence levels; and

- Strategies for dealing with potential non-response bias.

Note: Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by the web.

12. Total Number of Initial Contacts/Expected Number of Respondents: Provide an estimated total number of initial contacts and the total number of expected respondents.

13. Estimated Time to Complete Initial Contact/Instrument: Estimate the time to complete the initial contact and the survey instrument (in minutes).

14. Total Burden Hours: Provide the total number of burden hours. The total burden hours should account for the amount of time required to instruct the respondents in completing the survey, and the amount of time required for the respondent to complete the survey.

15. Reporting Plan: Provide a brief description of the reporting plan for the data being collected. A copy of all survey reports must be archived with PPA. Please note this in the reporting plan.

16. Justification, Purpose and Use: Provide a brief justification for the survey, its purpose, goals, and utility to managers. Specifically, describe how data will be tabulated and what statistical techniques will be used to generalize the results to the entire customer population. Describe how data from the survey will be used. Describe how you will acknowledge any limitations related to the data, particularly in cases where we obtain a lower than anticipated response rate. Note whether or not the survey is intended to measure a Government Performance and Results Act (GPRA) performance measure.

ATTACHMENT 2:

Approval Form for DOI Programmatic Clearance for Customer Satisfaction Surveys (OMB Control Number 1040-0001)

U.S. Department of the Interior Office of Policy Analysis (PPA)

PPA Tracking Number: (for PPA use only) CSS-10

Date Submitted to PPA:

3/7/12

1.

Survey Title:

Booker T. Washington National Monument Formative Exhibition Evaluation Survey

2.

Bureau:

National Park Service

3.

Abstract: (not to exceed 150 words) The National Park Service (NPS) will conduct a survey to measure customer satisfaction related to new exhibit areas being developed for the Booker T. Washington National Monument (BOWA) by Mystic Scenic Studios and Wondercabinet Interpretive Design. The objective of the new exhibit is to strengthen the multifaceted approach to public education at BOWA with the thoughtful development, design, and production of new youth-friendly exhibits. To ensure that the final design, stories, and interactive elements meet the needs of the intended audiences, including school-age children, NPS managers need to understand how potential visitors use and learn from the exhibit. The survey will evaluate the following: 1. Did respondent groups understand the content, stories, interactive components, and main ideas of the exhibit? 2. Were respondent groups satisfied with the exhibit? The results of the information collected will inform the exhibit development process during the Design Development phase and subsequent phases of work.

4.

Bureau/Office Point of Contact Information

First Name:

Kenneth

Last Name:

Davis

Title:

Contracting Officer

Bureau/Office:

National Park Service

Street Address:

67 Mather Place

City:

Harpers Ferry

State:

WV

Zip code:

25425

Phone:

304-535-6415

Fax:

Email:

[email protected]

5.

Principal Investigator (PI) Information (same as above)

6.

Name of Program or Office Conducting Survey:

Nikki DeJesus Sertsu DeJesus & Associates, LLC for Wondercabinet Interpretive Design, Inc. 7 Muzzey St., Lexington, MA 02421

7.

Description of Customers/ Services Provided:

The Booker T. Washington National Monument (BOWA) located in Hardy, VA is the historic tobacco farm/plantation where Booker T. Washington was born in 1856 and spent the first nine years of his life enslaved. This important National Monument preserves and protects the birthplace and childhood home of Booker T. Washington, while interpreting his life experiences and significance in American history between the years of 1895 to 1915 (the year of his death). The current on-site tour begins with a 14-minute orientation film, used to present the difficult issues of slavery and Virginia tobacco plantation life related to the early life of Booker T. Washington. Following the film, visitors have an opportunity to view the exhibits in the Visitor’s Center. In addition to the indoor exhibits there are many outdoor resources, including interacting with farm animals, going inside reconstructed plantation buildings, and viewing plantation crops while walking along the park’s trails. During special occasions living history programs are presented. It is the mission of BOWA’s education program to satisfy Virginia Public Schools’ education needs by using the park as an additional classroom resource. The programs and activities at BOWA are designed to introduce students to the life of Booker T. Washington; however, the current exhibits are outdated, static and are not youth-friendly. To increase public awareness of Booker T. Washington and make the park more attractive to school groups and families, BOWA is currently developing new interpretive exhibits. In order to determine if the proposed designs are clear and the intended messages are communicated, the design team will: 1) collect observational data while respondent groups view text panels and manipulate interactive elements representative of the final exhibit design; and 2) administer a customer satisfaction questionnaire after the observation. The results of this collection will be used to inform the final exhibit. Three respondent groups will be a part of this collection: 1. Middle and Elementary School Principals 1. School-age children (fourth through sixth grades) 2. Adults (BOWA “VIP” Volunteers - 18 years and older) Middle and Elementary School Principals The principals at Benjamin Franklin Middle School (BFMS) and Burnt Chimney Elementary School (BCES) will be responsible for providing demographic information for the students participating in the collection. School-age children We have chosen to work with two schools that are within close proximity of the park and because nearly all kindergarten and first grade students in Franklin County visit BOWA each year on a school related field trip. Presently, the majority of school children that visit the park for education programs are K-1st grade. Many higher grade elementary and middle school teachers do not consider BOWA an education experience option because they assume that most students have already been to the park in 1st grade and have experienced all that the park has to offer. BOWA wants to change this perception by developing new exhibits that appeal to higher grade elementary and middle school teachers and students and encourage teachers to return with their students on future field trips. Burnt Chimney Elementary School (BCES) is within five miles of BOWA on Route 122. Respondents will be students from two 4th grade classes and two 5th grade classes. Benjamin Franklin Middle School (BFMS) is located in Rocky Mount, VA about 20 miles from the park. This school was selected because it is the only middle school within Franklin County. Respondents will be students from two 6th grade classes. Adults BOWA has a very active volunteer organization identified as “BOWA VIP” volunteers. This group will serve as the on-site focus group for this collection. The VIP volunteers are extremely dedicated and actively involved in increasing the public awareness and significance of the park. We feel that the VIP volunteers will provide critical review and feedback concerning elements of the narratives that may be revised in the final design. BOWA “VIP” volunteers represent a diverse group of people varying in ages and backgrounds (e.g. retirees, schoolteachers, administrators, business owners, trustees, farmers, former NPS employees, and young adults).

8.

Survey Dates

5/14/12

through

5/25/12

9.

Type of Information Collection Instrument (Check ALL that Apply)

__Intercept

__Telephone

__Mail

__Web-based

_X_Focus Groups

__Comment Cards

_X_Other

Explain:

Questionnaire

10. Survey Development: BOWA is interested in making the park more attractive to schools for field trip visits while at the same time increasing public awareness of the significance of Booker T. Washington’s life and legacy. One method is to improve the existing outdated interpretive exhibits. The management and staff agree that the current exhibits do not effectively communicate or offer young people an opportunity to fully engage; to use their imagination; and to explore, touch, feel, and interact in a meaningful way. The current interpretive program, including the exhibits was last evaluated in 1999 as part of BOWA’s overall Long Range Interpretive Plan. The BOWA is in the process of working with Mystic Scenic Studios and Wondercabinet Interpretive Design, to create new exhibits that are more youth- friendly and engaging to the general public. The current design does not encourage ongoing discussions about Booker T. Washington’s legacy and the context of race in American society. In 2011, the Mystic/Wondercabinet team was selected to design and install the new exhibits for BOWA. A part of the process is to evaluate the design and the functional elements before the final exhibits are installed on-site. This is a one-time opportunity to measure the effectiveness of the exhibit before production and final installation. Because there is a short window of time between the design development phase and production BOWA wants to measure visitor satisfaction by presenting representative mock-ups of the actual exhibit (content and interactive elements) to select groups of potential visitors. Based on the responses, BOWA will make necessary modifications prior to final installation. The purpose of the collection is to reveal whether the new exhibit content is clear. We would also like to understand what visitors do in the exhibit and the extent to which they understand and connect with main messages. We will develop four interpretive text panels and three interactive elements for this collection. The text panels and interactives will represent the design, language, use, and concepts that will be part of the final exhibit. Respondent groups will be observed as they engage with each panel and interactive. A questionnaire will be used to collect information from both respondent groups. The questions were developed in consultation with the design team and the BOWA management team. The responses will be used to measure customer satisfaction related to the content and interactive elements. The respondents will be asked to provide feedback regarding how well they understood and were satisfied with the stories (content) and interactive elements. A 3-question questionnaire will be distributed to the school-age respondent group; and a 6-question questionnaire will be distributed to the adult respondent group. The questions were vetted to determine if the responses would provide the information needed to understand the motivations and satisfaction of potential visitors to BOWA. The questionnaires were revised based on internal team feedback. Questions were removed or rephrased that were deemed leading or biased. Questions were simplified to provide clarity and avoid response interference. Age appropriate questions were developed for the school age group. General demographic information will be gathered to supplement the satisfaction questions. Personal demographic information is also needed to confirm target audience representation. School principals will complete the demographic information for their classes prior to the collection. The adult respondent group will complete the demographic information onsite.
11. Survey Methodology: (Use as much space as needed; if necessary include additional explanation on separate page).
Respondent Universe		The respondent universe for this collection will be: 1. Four classes at Burnt Chimney Elementary School in Franklin County, VA. – 2 fourth grade and 2 fifth grade classes. One hundred (100) students will participate. Each class has 25 students. 2. Two sixth grade classes at Benjamin Franklin Middle School in Franklin County, VA. Seventy (70) students will participate. Each class has 35 students. 3. The Principals (n=2) from Burnt Chimney Elementary School and Benjamin Franklin Middle School 4. Adults - BOWA “VIP” Volunteers. Volunteers are over the age of 18. Twenty-five (25) volunteers will participate.
Sampling Plan/Procedure		Principals: An e-mail message will be sent to the principals at each of the represented schools. The principals will be able to respond to the questions via e-mail. School-age children There are only two 4th grade classes and two 5th grade classes at Burnt Chimney Elementary School. All fourth and fifth grade classes will be included in the collection. There are five 6th grade classes at Benjamin Franklin Middle School. The school’s interim principal will randomly select two 6th grade classes to participate in this collection. Adults All BOWA VIP volunteers (n=25) will be asked to participate in the focus groups.
Instrument Administration		Principal Prior to arriving to the schools the principals at each of the participating schools will receive an e-mail five business days before the collection. The e-mail will provide the interviewers with information about the demographic make-up of the classrooms participating in the collection. Completing the e-mail survey and returning it to the interviewers should take about 5 minutes to complete. 2 Principals x 5 minutes = 10 minutes The e-mail will include the following three questions: 1. How many students that will participate in the evaluation: _____ are Hispanic or Latino _____ are not Hispanic or Latino How many students identify themselves as: (Note, some students may identify themselves in more than one category, so the numbers may add to more than the total number of students) _____ African American or Black _____ White _____ Native Hawaiian or other Pacific Islander _____ American Indian or Alaska Native _____ Asian How many students that will participate in the evaluation are: _____ Male _____ Female School-age Children We will travel to Franklin County for two working days for the collection. One day will be spent at BCES and one day at BFES. The team will set up four text panels and three interactive elements representative of the final exhibit in each school’s cafeteria. We will visit each classroom prior to the collection to explain the purpose (what & why) and process (when & how) of the collection. Teachers will dismiss students from the classroom in small groups (4-5 students per group for the elementary school; and 5-6 students per group for the middle school) to the school cafeteria. Each small group will have a maximum of 3 minutes to interact with each text panels and interactive (21 minutes maximum). 4 Text Panels x 3 minutes = 12 minutes 3 Interactives x 3 minutes = 9 minutes A Mystic/Wondercabinet team member will be available to answer questions and facilitate the interaction to make sure each student has an opportunity to read and see the interpretive panels and use the interactive elements. Once one group is finished with the first element, another group will enter, allowing for simultaneous collection. The primary methodology will include observing students while they interact with the text panels to see how they actually engage with the material and connect with the content. A log sheet will be used to document observations. We will collect observation data to include: How long did the student stay at each panel and interactive element? Did the student stay for the maximum time? Did the student move quickly or linger? Did the student understand how to manipulate the interactive elements? Was there discussion among the group? After experiencing the text panels and interactive elements, students will complete a brief 5-minute follow-up questionnaire. Chairs and writing utensils will be provided in a separate space in the cafeteria for this part of the collection. Upon completion of the questionnaire, we will thank the students for their participation and collect the questionnaire. A school faculty or staff member will escort the students back to their classroom. Adults We will travel to Franklin County for two working days for the collection. The team will meet with the BOWA “VIP” volunteers at the BOWA site for the collection. We will set up four text panels and three interactive elements representative of the final exhibits in BOWA’s Visitor’s Center. We will explain the purpose (what & why) and process (when & how) of the collection. The team will randomly select 5 small groups (5 volunteers per group). Each small group will have a maximum of 3 minutes to interact with each text panels and interactive element (21 minutes total). 4 Text Panels x 3 minutes = 12 minutes 3 Interactives x 3 minutes = 9 minutes A Mystic/Wondercabinet team member will be available to answer questions and facilitate the interaction to make sure each volunteer has an opportunity to read and see the text panels and use the interactive elements. Once a group is finished with the first element, another group will begin, allowing for simultaneous collection. The primary methodology will include observing volunteers while they interact with the text panels to see how they actually engage with the material and connect with the content. A log sheet will be used to document observations. How long did they stay at each panel and interactive element? Did they stay for the maximum time? Did they move on quickly or linger? Did they understand how to manipulate the interactive elements? Was there discussion among the group? After experiencing the text panels and interactive elements, volunteers will complete a brief 10-minute follow-up questionnaire. Chairs and writing utensils will be provided in a separate space in the Visitor’s Center for this part of the collection. Upon completion of the questionnaire, we will thank the volunteers for their participation and collect the questionnaires
Expected Response Rate Confidence Levels:		Principals We expect that both (n=2) principals will respond to our e-mail request for information. School-age children We expect 100% response rate for this collection – 170 students. Every student in each class selected will participate as part of their classroom’s daily schedule. Teachers will prepare students for the collection and ensure that every student in each class participates. Adults We expect an 88% response rate for this collection. BOWA managers will invite all 25 BOWA “VIP” volunteers to participate in the collection. The volunteers have a strong track record of dedicated service to BOWA. When asked to participate in BOWA activities, they respond enthusiastically, so we expect at least 22 volunteers for the collection. Given schedules and the one-time opportunity to view the interpretive panels and interactives, we expect that at most 3 volunteers may not be able to participate.
Strategies for dealing with potential non-response bias		We anticipate that the teachers will encourage participation as a classroom activity and because of this we believe that the students are more likely to complete the process. If a student respondent refuses to participate in the group activity we will offer the teacher an alternative method to allow the student to participate. The teachers will be given a visual representation of the text panels and interactives for the student to view on their own. The teacher will also be given the questionnaire to administer. For the non-respondents in the adult sample, we will ask if the respondent can be contacted by email. Visual representations of the text panels and interactives and questionnaire will be provided so that the non-respondent can complete the process on their own time. We will follow-up with the non-respondent by email to ensure the return of the questionnaire.
Description of any pre-testing and peer review of the methods and/or instrument (recommended)		The NPS/BOWA staff and exhibition development team members reviewed the survey. The design team incorporated their suggestions, edits, and comments in the final questionnaire. The reviewers also agreed with our estimated burden time of 26 minutes for the school-age children and 31 minutes for adults.
12.	Total Number of Respondents:		Principals: 2 School-age children: 170 Adults: 22
13.	Estimated Time to Complete Instrument:		Principals: 5 minutes School-age children: 26 minutes Adults: 31 minutes
14.	Total Burden Hours:		Principals: < 1hour School-age children: 74 hours Adults: 11 hours TOTAL HOURS: 86
15. Reporting Plan: We will submit a summary of the result to NPS/BOWA. Frequency distributions indicating how participants responded to the questionnaire will be presented. This information will be used to inform the design development and final phases of the exhibition process

16. Justification, Purpose, and Use:
Survey Justification and Purpose	This information collection is the only opportunity for BOWA to test the design features of the new exhibit. With this collection, BOWA will understand if the new exhibit successfully communicates the intended main messages and captures people’s attention, which could lead to repeat visits. BOWA would like to test the exhibit at this phase so that respondent feedback can be incorporated in the final exhibit design. To ensure that the final exhibit design, stories, and interactive elements meet the needs of the intended audiences, BOWA managers need a clear understanding of how potential visitors will use the exhibit material and interact with the content.
Survey Goals	The goal of the survey is to reveal whether the exhibit content and stories are clear, what visitors do, and the extent to which they understand the main ideas. Some of the questions we want to answer during the survey are: 1. Can respondents figure out what to do? 2. Does the exhibit make sense? Are there ways to improve the exhibit? 3. Do respondents find the exhibit interesting? Do they get the messages, key concepts, ideas presented in the displays? 4. To what extent is the exhibit enjoyable? Engaging? A learning experience? 5. To what extent are respondents satisfied? 6. Will respondents reach the conclusions we’ve intended by engaging with the exhibit? If not, how do we remedy this?
Utility to Managers	This information will directly inform the final phases of the exhibit development process.
How will the results of the survey be analyzed and used?	General descriptive statistics (i.e., frequency distributions) will be used to analyze the results of the survey. Key results will include: How much time respondents spent with text panels and interactives. Least and most enjoyable exhibition elements. Was there discussion among the group? The results will be presented to NPS/BOWA and the exhibition development team for use in the design development phase. The exhibit plans will be modified as necessary.
How will the data be tabulated? What Statistical Techniques will be used to generalize the results to the entire customer population? How will limitations on use of data be handled? If the survey results in a lower than anticipated response rate, how will you address this when reporting the results? (Use as much space as needed; if necessary include additional explanation on separate page). We acknowledge that there will be some limitations of the results. The respondent groups are used to provide feedback about the design and structural content of the interpretive exhibits. The results will not be used to generalize an entire population, but will only represent the sample of respondents. We will not use the data to make comparison of any users outside the scope of this collection. Results will only be used to provide feedback about the new exhibition treatments. Because of the nature of the respondents (attentive, engaged, and active participants) we expect that the participants will be highly motivated to complete the survey and provide information that will be useful to the planning process. We acknowledge, at this point, that these respondents will not represent all BOWA visitors however, we feel that the individuals participating in the survey represent a group of potential visitor attitudes and expectations that will provide meaningful information regarding the interests and user satisfaction with the BOWA.
Is this survey intended to measure a Government Performance and Results Act (GPRA) performance measure? If so, please include an excerpt from the appropriate document. (Use as much space as needed; if necessary include additional explanation on separate page). No. This information will only be used to inform the final phases of the exhibit development process.

ATTACHMENT 3

Checklist for Submitting a Request to Use DOI Programmatic Clearance for Customer Satisfaction Surveys

X All questions in the survey instrument are within the scope of one of the DOI Programmatic Clearance for Customer Satisfaction Surveys topic areas.



X The approval package is being submitted to the Office of Policy Analysis at least 45 days prior to the first day the PI wishes to administer the survey to the public.



X A qualified statistician has reviewed and approved your request.



X Your bureau/office Information Collection Clearance Officer has reviewed and approved the approval package.



The approval package includes:

X A completed Information Form

X A signed Certification Form

X A copy of the survey instrument

X Other supporting materials, such as:

• Cover letters to accompany mail-back questionnaires

• Introductory scripts for initial contact of respondents

• Necessary Paperwork Reduction Act compliance language

• Follow-up letters/reminders sent to respondents



The survey methodology presented on the Information Form includes a specific description of:

X The respondent universe

X The sampling plan and all sampling procedures, including how respondents will be selected

X How the instrument will be administered

X Expected response rate and confidence levels

X Strategies for dealing with potential non-response bias

X A description of any pre-testing and peer review of the methods and/or the instrument is highly recommended.



X The burden hours reported on the Information Form include the number of burden hours associated with the initial contact of all individuals in the sample (i.e., including refusals), if applicable, and the number of burden hours associated with individuals expected to complete the survey instrument.

X The package is properly formatted (Word) and submitted to the Office of Policy Analysis electronically. 

ATTACHMENT 4

CERTIFICATION FORM FOR SUBMISSION UNDER OMB CONTROL NUMBER 1040-0001

This form should only be used if you are submitting a collection of information for approval under the DOI Programmatic Clearance for Customer Satisfaction Surveys.

If the collection does not satisfy the requirements of the Programmatic Clearance, you should follow the regular PRA clearance procedures described in 5 CFR 1320.

Bureau/Office Subgroup or Program National Park Service - BOOKER T WASHINGTON NATIONAL MONUMENT
Title (Please be specific) Booker T. Washington National Monument Formative Exhibition Evaluation Survey
Burden Hours Principals: < 1 hour School-age children: 74 hours Adults: 11 hours			Estimated Number of Respondents Principal: 2 School-age children: 170 Adults: 22
Total Burden Hours 86 TOTAL			Minutes per Response Principal: 10 minutes School-age children: 26 minutes Adults: 31 minutes
Bureau/Office Contact (who can best answer questions about content of the submission):
Name	Kenneth Davis			Phone	304-535-6415
Certification: The collection of information requested by this submission meets the requirements of OMB control number 1040-0001
Bureau/Office Qualified Statistician Dr. Lena Le						DATE 3/7/12
Bureau/Office Information Collection Clearance Officer Phadrea D. Ponds						DATE 3/7/12
Office of Policy Analysis Donald J. Bieniewicz						DATE 3/9/12
OMB, Office of Information and Regulatory Affairs (OIRA)						DATE

1