Wirthlin_Supporting_Statement_Final

Supporting Statement

Presidential Libraries Museum Visitor Survey

OMB Control No. 3095-0066

A. Justification

1. Circumstances making the Collection of Information Necessary. Presidential libraries are built by private, nonprofit charitable foundations that raise money from non-federal sources for that purpose. The Presidential Libraries Act (44 U.S.C. § 2112(a)(1)), authorizes the Archivist of the United States to accept these institutions on behalf of the Federal Government. In accepting Presidential Libraries, NARA becomes responsible for a major museum program and for providing the best possible experience to visitors. With the acceptance of the Richard Nixon Presidential Library this fiscal year, NARA now operates 12 libraries. Planning has also already begun on the new Bush Presidential Library.

With this broad mandate to serve the public, NARA needs visitor feedback to measure program effectiveness, to best utilize government resources, and to sustain and advance the mission of the NARA as the nation's records keeper, preserving and making available evidence of citizens' rights, responsibilities, and shared national experience. By measuring customer satisfaction, we can understand the ways in which we can make our museums better planned, more focused, and more dynamic.

From a resource perspective, the museums also provide revenue that helps offset their operations, saving NARA money both in terms of overhead and maintenance, staffing, and program development. With more information than currently available, we will be able to tailor our programs to maximize revenue generation.

The Office of Presidential Libraries has contracted with the audience evaluation firm, Harris Interactive, to brand and evaluate the public’s knowledge of the Presidential Libraries. Part of this contract is the development of a museum visitor survey.

2. Purpose and use of the information. The survey is comprised of a set of questions designed to allow for a statistical analysis that will ultimately provide actionable information to NARA for use internally. The survey includes questions that measure the visitor’s satisfaction in general and specific aspects of their visit. These questions serve as dependent variables for analytical purposes. Other questions provide attitudinal, behavioral, and demographic data that are used to help understand variation in the satisfaction variables. Using statistical analyses, Harris Interactive will determine the factors that drive the visitor’s perceptions of quality and satisfaction with the Library they visited. Additionally, natural groupings of visitors defined by similarity based on these attitudinal, behavioral, and demographic variables can be developed and targeted for outreach purposes.

The information collected through this effort will benefit Library and NARA staff and management in making critical decisions about museum operation and program direction. From museum curators, to Library directors, to NARA senior management in Washington, this information will inform program activity, operation, and oversight. Museum curators will learn insight into the ways in which the public interact with and relate to permanent and temporary exhibits. Library management will use the information internally to guide local museum operations. NARA senior management will use the information internally to assess program effectiveness and identify strategic directions when looking system-wide at program needs.

Information collected through this survey will only be used for internal programmatic purposes. Throughout the survey period, NARA with the contractor will analyze methodology and response rates. After the initial survey implementation, if response rates prove to be at acceptable OMB levels, NARA may seek approval for re-implementation of the survey for external reporting purposes to demonstrate Library performance and establish a baseline for measuring future program effectiveness. If, however, the response rate is as outlined in this document, then NARA will re-design the survey and methodology to satisfy OMB requirements for utilizing the data if the intent is to use data for external reporting purposes.

NARA was granted a one year approval for this new information collection on May 8, 2008. The expectation was that during this one year period, NARA would conduct the survey and have baseline data to proceed with a longer renewal period this year. However, due to contractual problems and staff reassignment to the presidential transition, NARA will begin conducting this survey on March 1, 2009. Since this expires May 31, 2009, NARA requests a one year extension to June 30, 2010 to collect necessary baseline data.

3. Use of Information Technology and Burden Reduction. An Internet method of data collection was chosen primarily to maximize response rate and allow respondents to complete the survey at their leisure. Respondents will also have the option to suspend the survey and resume it at a later, more convenient time. Understanding that not every visitor will have access to or may not prefer to use the internet to complete the survey, paper-and-pencil versions of the survey will be available.

Using a password protected internet methodology allows Harris Interactive to more carefully control who completes the survey and to ensure that the survey is completed only once per respondent. In addition, an internet methodology allows Harris to maximize response rate, resulting in an optimal sample size for analysis. Further, the programming used in internet surveys allows Harris to guard against respondent/interviewer error and incomplete/missing data.

4. Efforts to Identify Duplication and Use of Similar Information. Presidential Libraries have never before measured, at a system level, how visitors use resources provided in museums. Program managers have therefore lacked an important element that should be considered when allocating resources or creating new initiatives designed to meet the mission. In the past, visitor information consisted largely of anecdotal reports of visitor experience with a specific library. The current study uses a broader approach that will give both local information and also high-level, cross-Library system data.

5. Impact on Small Businesses or Other Small Entities. The collection of information does not have a significant impact on small businesses.

6. Consequences of Collecting the Information Less Frequently. It is estimated that the survey will be available for a full 9 month period. Conducting the survey during a shorter window of time will not provide the representative sampling we require. Survey invitations for the online version as well as the actual paper versions of the survey will not be passed out on a daily basis, but periodically through the 9-month period of time. Visitation volume at Presidential Libraries varies seasonally and also by library, and this approach will allow data to reflect the lifecycle of visitation over a year. We will work with the individual libraries to determine the distribution schedules that work best for them

7. Special Circumstances Relating to the Guidelines of 5 CFR 132.5. The information collection is conducted in a manner consistent with the guidelines in 5 CFR 1320.5.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency. No consultation has been made with persons covered by the information collection. A Federal Register Notice was published on March 3, 2009 (74 FR 9307 and 9308) inviting the public to comment. No comments were received.

9. Explanation of Any Payment or Gift to Respondents. No payment or gift is provided to respondents for this information.

10. Assurance of Confidentiality Provided to Respondents. Harris Interactive, as a member of the Council of American Survey Research Organizations (CASRO), certifies annually that it will uphold the high standards set forth by the Council. These standards require that survey research organizations protect from disclosure to third parties--including clients and members of the Public--the identity of individual respondents as well as the identifiable information from that individual, unless the individual expressly requests or permits such disclosure. Consequently, respondent-identifiable data will not be disclosed to the client (NARA), unless authorized by the respondent.

As set forth in the CASRO Code of Standards and Ethics for Survey Research, survey research organizations also have a responsibility to strike a proper balance between the needs for research and the privacy of individuals who become respondents in the research. To achieve this balance, respondents must be protected from unnecessary and unwanted intrusions and/or any form of harassment. Harris Interactive may analyze survey data in combination with other respondent-level data such as internal customer data, etc., but it is understood that the information will be used for model building, internal analysis, or the like, and not for individual outreach efforts, and that no action can be taken toward an individual respondent simply because of his or her participation in the survey.

The CASRO Code of Standards and Ethics for Survey Research is available for review at: http://www.casro.org/codeofstandards.cfm

11. Justification for Sensitive Questions. The survey asks visitors to specify gender and year of birth as well as ranges for education level and income. These questions are crucial to understanding which groups of the public visit Library museum exhibits and which groups are not accessing our holdings. In the analysis, demographic questions will be cross-tabulated with attitudinal and behavioral data which will allow us to assess any differences in perceptions and/or behaviors between visitor segments. Results will help to inform our strategies for broadening the public audience for Presidential Libraries and will provide valuable insights for tailoring communications and offerings to the various visitor groups.

‘Decline to answer’ options will be offered, allowing respondents not to answer sensitive questions. Data is reported in aggregate form so that survey responses cannot be linked back to individual respondents. No personally identifiable information (such as e-mail addresses) that is collected in the administration of the survey will be reported or linked to their survey response.

12. Estimates of Hour Burden Including Annualized Hourly Costs.

There will be an estimated 75,000 respondents over the course of this survey, with a total annual burden of 18,750 hours. The frequency is one response per participant with the survey administered periodically over a nine month period across the Presidential Library system. It takes 15 minutes for applicants to complete the survey. The burden estimate was determined through analysis based on the prior experience of the contractor.

The annualized hourly cost to each respondent is about $4.66 (.25 hour @ $18.62 per hour times 15 minutes). This estimate is abased on the average of $18.62 per hour found in the August 2006 report for private industry and State and local government workers in the United States by the Bureau of Labor Statistics, National Compensation Survey: Occupational Wages in the United States, June 2005 (http://www.bls.gov/ncs/ocs/sp/ncbl0832.pdf).

13. Estimate of Other Total Annual Cost Burden to Respondent or Recordkeepers. There are no costs to respondents other than their time to complete the survey. Respondents are not required to keep records to support their response.

14. Annualized Cost to the Federal Government. We plan to implement the survey, per section B, twice over three years. We estimate that the annual cost to the Government for this information collection in the first year to be $111,618 ($110,000 for the development, implementation, and analysis of the survey as part of the contract, which includes cost of supplies, and $1,618 for one percent of work time for 2 NARA FTE [1 manager at GS15 and 1 analyst at GS11] to supervise and facilitate completion of the first year’s survey). We intend to re-survey in the third year. We estimate that the annual cost to the Government for this information collection in the third year to be $51,618. $50,000 for contractor-led re-deployment and analysis of the survey as part of a future contract, which includes cost of supplies, and $1,618 for one percent of work time for 2 NARA FTE (1 manager at GS15 and 1 analyst at GS11) to supervise and facilitate completion of the third year’s survey.

15. Explanation for Program Changes or Adjustments. This survey is a new collection of information.

16. Plans for Tabulation and Publication and Project Time Schedule. Harris Interactive will provide two types of result reporting. The first type of reporting will stem directly from the online survey results in an online format via the contractor’s Web-based information system ViewPort. Specifically, they will provide NARA with a secure, password-protected reporting website. The website will provide summary information from each survey question that can be viewed in aggregate and at the library level. They will also provide a selection of results available for viewing by respondents. They expect to provide NARA with interim data to select questions which can be posted either on the NARA website, the website for individual libraries, or a site hosted by Harris. The link to the results will be published at the end of the online version of the survey informing respondents that they can view preliminary results after a certain date if they wish.

While the above online reporting will fulfill the interim reporting requirements, Harris Interactive also will complete a full analysis of the survey, which will include specific recommendations for re-deployment of the survey in the third year, improving the Library effectiveness, and outreach. One report will be completed for each of the 11 presidential libraries at the one-year conclusion of the survey period. The report will be provided in Microsoft PowerPoint format and will contain all pertinent methodology information, descriptive statistics for each library, and the aforementioned strategic analyses. If desired, a data file can be easily provided to NARA in a database format of our choice including SPSS, ASCII, and Excel.

The Office of Presidential Libraries will develop reports and presentations, as appropriate, to share survey findings with NARA offices and stakeholders. Information from the surveys will be used, for example to illustrate performance accomplishments and supplement the various performance measures NARA gathers and publishes in our annual performance and accountability report.

17. Reason(s) Display of OMB Expiration Date is Inappropriate. The expiration date for OMB approval of this information collection will be displayed on the form.

18. Exceptions to Certification for Paperwork Reduction Act Submissions. There are no exceptions to the certification statement identified in Item 19 of OMB Form 83-1.

B. Collection Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods.

The proposed tool surveys museum visitors. These are individuals who are exposed to a Presidential Library’s museum exhibits. In 2006, approximately 1.7 million members of the public had the opportunity to visit the Presidential Libraries’ museum exhibits nationwide. This represents the potential universe of respondents.

Per the recommendations of Harris Interactive and based on their extensive audience evaluation experience, 110,000 online survey invitations will be produced for distribution (approximately 10,000 postcards per library).

Respondents can access the survey online any day of the week. The survey remains “live” for the entire duration of the project, so respondents can answer at a time that is most convenient for them. Based on the experience of Harris Interactive, we anticipate that during the periods when the survey is implemented, approximately 90 percent of Library museum visitors will receive an invitation to participate in the survey.

For respondents who indicate they do not have access to the internet, we Harris Interactive will produce 6,000 paper-and-pencil forms. The paper-and-pencil forms will be bulk shipped to each Library for distribution as needed.

2. Procedures for the Collection of Information. Upon survey finalization and OMB approval, Harris Interactive will train library staff in the implementation of the surveys and begin data collection. It is estimated that the survey will be available for a full nine-month period. To ensure adequate representation of the museums’ visitor base and to avoid any bias due to seasonality, sample will be balanced over time, meaning that survey postcard invitations will be distributed to visitors on 2 or more days per week, including at least one weekend day and one weekday, over the 9-month field period of the study. It is recommended that all libraries follow the same pattern to provide comparability across libraries.

Library staff will distribute postcard-size invitations in English to visitors. The postcard will contain a note of thanks from NARA along with instructions for completing an online survey. Each postcard will be identical except for a unique control number preprinted at the top, which will serve as an electronic key to the appropriate survey.

For visitors who do not have access to the internet or prefer to take the survey on paper, printed versions of the questionnaire will be made available. These paper surveys will be handed out by library staff and will come with a pre-paid return envelope allowing respondents to fill out the surveys at a time most convenient for them and then mail them to back Harris Interactive.

Internet data collection offers several advantages. First, it allows for the data to be viewed in real time using the ViewPort tool. Second, internet data collection is the most efficient and cost effective method when compared to telephone or mail methodologies.

3. Methods to Maximize Response Rates and Deal with Non-response.

To achieve a representative sample and preserve a certain level of randomness to the survey collection, we anticipate distributing survey invitations and paper versions of the survey on certain days of the week (e.g., Tuesdays, Thursdays, and Saturdays). This scheme may have to be customized for each library depending on visitor traffic patterns.

Approximately 75 percent of the US population now has access to the internet. Based on this assumption, Harris Interactive will provide each Library’s museum with a certain percentage of paper versions or the survey versus online invitations. Demographics by library will be taken into consideration to assess percentage of online users among the group most in line with that particular library’s visitor demographics.

Data from the paper version of the survey will be entered by hand by Harris Interactive staff and merged with the data from the online survey. To minimize differences in results between the paper and online version, the online version will be presented in the same way on the screen as the paper version. Since both are self-administered and depend on visual means of data collection (vs. phone, for example), differences, if any, are expected to be minimal.

Based on the expertise of the contractor, we estimate that approximately 15-20 percent of those who receive a survey invitation will participate (n=56,250 - 75,000 total). No incentives will be used.

The goal is to have enough invitations to discern meaningful performance measures within each library. Not only are the performance measures relevant but the gaps in performance between libraries will be useful to understanding the strengths and weaknesses of each. In order to achieve ample base sizes and therefore precision in the survey results for each library and the entire presidential Library system, we expect to distribute 110,000 survey invitations with the number of postcard invitations proportional to the number of visitors per library. We estimate that approx 20% of those who receive a survey invitation will complete the survey (n=22,000). For some of the smaller libraries sample sizes may be in the hundreds, which is enough sample to obtain these meaningful indices. The larger ones will likely be in the thousands. The minimum threshold of completes to be targeted per library will be set at 300, which can be achieved even if the actual response rate is less than the anticipated rate of 20%.

The target of a minimum of 300 completed surveys per library is based on the desire to have a reasonably small margin of error. With a sample size of 300 per library, the estimated margin or error is +/- 5.7% at the 95% confidence level or +/- 4.7% at the 90% confidence level. These levels are generally accepted for this type of study.

It is possible that response rates and ultimately the number of completes will vary by library due to differences in visitor numbers, traffic patterns, and even visitor demographics. Still, we expect to achieve sufficient base sizes to allow for statistically valid analyses and reporting of results at the individual library level. If response rates vary greatly between libraries and result in completes that are not proportional to the number of visitors per library, the data may be weighted to make sure each library is proportionally represented in the total results. For example, if a library with relatively low visitor levels has a high response rate, their results would be weighted downward so that their responses do not have a disproportionate effect on the total results. This would only affect reporting across all libraries. When reporting at the individual library level, results would not be weighted.

The issue of non-response bias will be addressed by weighting on key demographic variables to the extent that any non-response is occurring at random and not in a systematic way. If somehow systematic bias were identified for a particular respondent group, Harris Interactive would work with NARA to learn the cause and suggest alternatives to remedy the process that is causing the bias.

4. Test of Procedures or Methods to be Undertaken. The program will go through numerous quality assurance steps. Harris will conduct testing to ensure survey clarity, survey flow, and programming accuracy. If the test uncovers any "hard to understand" questions or awkward transitions, we will make modifications to the survey to ensure quality. Once the survey is programmed and before it is posted “live” on the servers, a final quality check (Random Data Generation) is performed to ensure that the program is functioning correctly. In addition, respondents have the option to provide feedback or report any difficulties or concerns regarding the survey to Harris.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and or Analyzing Data

Sylvia Hauschildt

Research Director

135 Corporate Woods

Rochester, NY 14623

Harris Interactive Loyalty Phone: 585.214.7278

Email: [email protected]

Bill Salokar

VP Sr Consultant

4720 Jones Bridge Circle

Norcross, GA 30092

Harris Interactive Loyalty Phone: 770-662-3073

Email: [email protected]

9