Justification Document

Request for Approval under the “DOI Generic Clearance for the Collection of Qualitative Feedback on Agency Service Delivery”

OMB Control Number: 1090-0011

TITLE OF INFORMATION COLLECTION: Formative evaluation of proposed exhibits at the Valley Forge National Historical Park Visitor Center

PURPOSE:

Valley Forge National Historic Park (VAFO) is a unit of the National Park Service (NPS) seeking to understand how a general visitor and elementary school teachers will access and understand identified exhibit components and content in order to best deliver the exhibition message and experience goal. This evaluation is designed to inform specific components of an exhibit to be installed in the Visitor Center in King of Prussia, Pennsylvania. Formative evaluations are generally conducted during the design stage of the exhibition in order to detect problems and successes of interpretation before building the final product.

The NPS and its exhibit contractors, The Design Minds, Inc. (TDM) and Insight Evaluation Services (IES), have determined that formative evaluation to study how visitors experience six specifically identified exhibit components will provide the best information to move forward in the exhibit design phase. The evaluation will test if the directives and content are understood by the public as they were intended by the content specialist and designer. The evaluation will also help the NPS and its exhibit contractor understand the effectiveness of the design choices, and to determine if the information is presented in a manner that is understandable and relatable. The components and specific questions of this request were chosen to address areas identified by the exhibit team as areas of the design needing particular visitor feedback.

The components will include prototypes of images, text, and interactive features that will be displayed so that the respondents will have an understanding of the materials in context. General public visitors who enter the Visitor Center during the testing period will be randomly selected to look at and interact with the prototypes, which will be installed in an on-site classroom away from the currently installed exhibits. Once they have had a chance to look at each of the components they will answer questions about the components, during a brief individual interview.

Upon approval we will test prototypes of six selected elements developed by TDM including:

1. Introductory Area

2. Spinning Cubes Interactive Element

3. Doctor-Patient Group Facilitated Dialogue Interactive

4. Rifle Activity (Weapons/Tactics Section Interactive Element)

5. Perseverance/Legacy Area

6. Word Cloud Activity (“What Does Valley Forge Mean To You?”)

NPS and its exhibit contractor will use the results of the evaluation to provide timely feedback during the design development phase ensuring the opportunity to make any appropriate adjustments before the exhibit components go into the next phase of development.

DESCRIPTION OF RESPONDENTS: Members of the general public (adults 18 and over) who enter the Valley Forge Visitor Center during the sampling period and four fourth grade teachers from the Norristown Area School District and the Phoenixville Area School District. These teachers have been selected by the park staff because they are known to bring their students on annual field trips to Valley Forge.

At the end of the sampling period the prototypes will be moved into the main lobby of the visitor center. In addition to the study participants, all members of the general public visiting the site during the sampling period will be unobtrusively observed viewing the exhibits. The observations will be recorded. This will include all children, adults, staff, and volunteers.

TYPE OF COLLECTION: (Check one)

[ ] Customer Comment Card/Complaint Form [ ] Customer Satisfaction Survey [ ] Usability Testing (e.g., Website or Software) [x ] Small Discussion Group

[ ] Focus Group [ ] Other:

CERTIFICATION:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low-burden for respondents and low-cost for the Federal Government.

3. The collection is non-controversial and does not raise issues of concern to other federal agencies.

4. The results are not intended to be disseminated to the public.

5. Information gathered will not be used for the purpose of substantially informing influential policy decisions.

6. The collection is targeted to the solicitation of opinions from respondents who have experience with the program or may have experience with the program in the future.

Kirsten S. Buchner

Digitally signed by Kirsten S. Buchner

DN: cn=Kirsten S. Buchner, o=Insight Evaluation Services, ou, [email protected], c=US

To assist review, please provide answers to the following questions:

Personally Identifiable Information:

1. Is personally identifiable information (PII) collected? [ ] Yes [x] No

2. If Yes, is the information that will be collected included in records that are subject to the Privacy Act of 1974? [ ] Yes [ ] No

3. If Applicable, has a System or Records Notice been published? [ ] Yes [x] No Not applicable

Gifts or Payments:

Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to participants? [ ] Yes [ X] No (If yes, please explain.)

BURDEN HOURS

Category of Respondents	Number of Initial contacts	No. of Respondents	Participation Time (mins)	Burden (hrs)
Teachers	10	4	90 minutes	6
Visitors	25	10	45 minutes	8*
Totals	35	14		14

*Rounded

FEDERAL COST: The estimated one-time cost to the Federal government is $16,145. No additional federal salary is included in this cost

_The cost includes the following operational expenses:

Operational Expenses	Estimated Cost
Contract Support Survey materials preparation, coordination, oversight of survey implementation, data analysis and reporting	$15,825
Travel (this includes lodging) and any miscellaneous on-site administrative costs.	$ 320
Total	$16,145

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents

1. Do you have a customer list or something similar that defines the universe of potential respondents and do you have a sampling plan for selecting from this universe? [ X] Yes [ ] No

If the answer is yes, please provide a description of both below (or attach the sampling plan). If the answer is no, please provide a description of how you plan to identify your potential group of respondents and how you will select them.

Respondent Universe and Sampling Plan: The respondent universe will consist of all visitors who enter the Visitor Center during the testing period. Visitors will be randomly selected and invited to view the exhibits (see attached invitation language).

General Public: Visitors will be randomly recruited as they enter the visitor center. A trained interviewer will recruit visitors by approaching every nth adult visitor group or individual that enters the visitor center and crosses a predetermined location (a designated imaginary line in the visitor center). The interviewer will ask the adult who is physically closest to the line to participate in the study. Visitors will be told about the study when asked to participate. When the interview session is complete, the interviewer will be ready for the next interview, at which point the nth visitor to cross the imaginary line will select selected.

Invited Teachers: The fourth grade teachers from Norristown and Phoenixville Area School Districts will be invited to participate in this study; these teachers have be identified selected because they bring their students on annual field trips to Valley Forge.

All Visitors: After the visitor and teacher interviews are complete, the exhibits to be tested will be moved to the visitor center lobby. The evaluation team will unobtrusively observe and record all visitor interaction with the exhibit elements.

Administration of the Instrument

1. How will you collect the information? (Check all that apply) [ ] Web-based or other forms of Social Media

[ ] Telephone

[x] In-person [ ] Mail

[ ] Other, Explain

2. Will interviewers or facilitators be used? [x] Yes [ ] No

General Public

The sessions will consist of one-on-one interviews. Each session will only include one visitor at a time. The selected participant will be allowed to bring members of their group with them, however all questions will be directed to the specific person who was recruited through the random selection process. Once the visitor agrees to participate they will follow the interviewer to a classroom where the prototypes will be set up. The data collection method will be as follows:

• Before they view the prototypes, the visitor will be asked two background questions to determine the extent to which they may have prior knowledge about Valley Forge.

1. Before we get started, can you tell me if you have been here (Valley Forge National Historic Park) before?  Yes  No

[If yes:] When was the last time you were here?

2. Please tell me what you know about this place, Valley Forge. [Probe for what time period, war, and/or significance it might have.]

• Next, visitors will be told the prototypes are set up in the order they will be placed in the proposed exhibition and asked to spend up to 15 minutes looking at them and that afterwards they will be asked a series of questions about the prototypes.

• As visitors look at the prototypes, the data collector will observe them at specific elements and note their behaviors using the observation checklist the data collector will also note any comments overheard.

• At the end of 15 minutes or once visitors indicate they are finished looking at the prototypes, the data collector will begin the one-on-one interview. Questions are designed to gather detailed information about visitors’ reactions to and understanding of the prototyped exhibit elements. The data collector will write down visitors’ responses verbatim; TDM and Park staff may observe the interview and take notes as well.

• The interview is not intended to last more than 30 minutes. At the end of the interview, visitors will be asked to complete a card with four demographic questions. These data will be used to describe the sample of visitors who participated in the evaluation and to provide context to the responses.

Fourth Grade Teachers

The park staff will personally invite fourth grade teachers from the local area to participate in the study. The invitation will include the purpose and need of the study and will also provide the date and time to arrive at the visitor center. A trained interviewer will greet the teachers as they arrive and once in the visitor center the teachers will be guided to the classroom where the exhibits will be displayed. At that time they will receive an further information about the study and the extent of their participation.

The protocol will be as follows:

• The data collector will first provide instructions and direct teachers to spend up to 45 minutes looking at all of the prototypes, which will be in the order that they will be set up in the proposed exhibition.

• At the end of the 45 minutes (or when teachers are ready), the data collector will begin the group session. The session is designed to be more like a focus group versus one-on-one interviews. The intention is to prompt discussion among the teachers about the exhibits. The questions will be used to initiate the dialogues and designed to gather information about the teachers’ impressions of the exhibit. Our intention is to understand their perceptions of what fourth-grade audiences might understand and enjoy about the experience.

• The data collector’s role will be to facilitate the session and TDM and Park staff will observe, ask questions, and take notes during the session.

Please make sure that all instruments, instructions, and scripts are submitted with the request.

INSTRUCTIONS

TITLE OF INFORMATION COLLECTION: Provide the name of the collection that is the subject of the request. (e.g., Comment card for soliciting feedback on xxxx)

PURPOSE: Provide a brief description of the purpose of this collection and how it will be used. If this is part of a larger study or effort, please include this in your explanation.

DESCRIPTION OF RESPONDENTS: Provide a brief description of the targeted group or groups for this collection of information. These groups must have experience with the program.

TYPE OF COLLECTION: Check one box. If you are requesting approval of other instruments under the generic, you must complete a form for each instrument.

CERTIFICATION: Please read the certification carefully. If you incorrectly certify, the collection will be returned as improperly submitted or it will be disapproved.

Personally Identifiable Information: Provide answers to the questions.

Gifts or Payments: If you answer yes to the question, please describe the incentive and provide a justification for the amount.

BURDEN HOURS:

Category of Respondents: Identify who you expect the respondents to be in terms of the following categories: (1) Individuals or Households; (2) Private Sector; (3) State, local, or tribal governments; or (4) Federal Government. Only one type of respondent can be selected.

No. of Respondents: Provide an estimate of the Number of respondents.

Participation Time: Provide an estimate of the amount of time required for a respondent to participate (e.g., fill out a survey or participate in a focus group)

Burden: Provide the Annual burden hours: Multiply the Number of responses and the participation time in minutes and divide by 60.

FEDERAL COST: Provide an estimate of the annual cost to the Federal government.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents. Please provide a description of how you plan to identify your potential group of respondents and how you will select them. If the answer is yes, to the first question, you may provide the sampling plan in an attachment.

Administration of the Instrument: Identify how the information will be collected. More than one box may be checked. Indicate whether there will be interviewers (e.g., for surveys) or facilitators (e.g., for focus groups) used.

Please make sure that all instruments, instructions, and scripts are submitted with the request.

7