National Museum Survey (NMS) Data Collection
Supporting Statement for PRA Submission
Part B: Collection of Information Employing Statistical Methods
The National Museum Survey (NMS) is a census of all museums in the United States, including zoos, aquariums, botanical gardens, and arboretums; nature and science centers; history museums and historic sites; art museums; children’s museums; natural history museums; and general and specialized museums. The purpose of the survey is to obtain up-to-date information directly from museums to measure and understand the scope, scale and nature of the role that the nation’s diverse museums play in American society. The survey includes questions covering a range of topics including organizational characteristics, facilities, finances, human resources, admissions and visitation, and digital presence.
The NMS will attempt to contact over 20,000 museums presently listed on the project’s population frame with expected responses from 7,000 to 8,000 museums. Participation by respondents is voluntary. This collection only asks respondents to report their answers; there are no record-keeping costs to the respondents. The data will be disseminated through a series of three reports (national, museums with animate collections, and museums with inanimate collections), and through a data portal hosted on the Institute of Museum and Library Services’ (IMLS) website.
Part B of the supporting statement includes details on the NMS population, methodology for administering the NMS, efforts to maximize response rate, and handling of nonresponse to the NMS. The following attachments are provided:
Attachment |
Contents |
Attachment A |
NMS Questionnaire |
Attachment B |
Prelaunch contacts (email, telephone, and mail) Launch contacts (email) Reminders (email and telephone) |
B.1. Respondent Universe
The NMS will be a census of all eligible U.S. museums representing a broad range of museum disciplines, including zoos, aquariums, botanical gardens, and arboretums; nature and science centers; history museums and historic sites; art museums; children’s museums; natural history museums; and general and specialized museums. No sampling will be performed for the survey.
Institutions must meet all the following criteria to be eligible for selection:
Be a unit of federal, state, local, or tribal government, or a nonprofit institution.
Serve the public in a physical location it owns or operates.
Provide exhibitions and programs.
Primarily function to house, display, and care for animate or inanimate objects that form the core of its exhibitions, programs, and research.
Under normal circumstances, be open to the public 90 days or more per year, either through specific hours of operation or by appointment.
Have at least one staff member, or the full-time equivalent, whether paid or unpaid.
IMLS originally developed the population frame for the NMS in preparation for the NMS pilot, which IMLS fielded in 2023. IMLS’ efforts on frame construction spanned several years and included the following steps, all of which are outlined in the agency’s report Construction of the Population Frame used for IMLS’ National Museum Survey (NMS) Pilot:1
Extensive private and public sector outreach to select a resource to serve as the basis for the frame
Selecting and contracting for use data from commercial vendor Yelp
Contracting with the Official Museum Directory to retrieve contact information that was then appended to the frame
Using data modeling, manual review, crowdsourced review, web scraping, and artificial intelligence to:
Remove ineligible units from the frame
Categorize all units from the frame into the project’s taxonomy of disciplines; and
Append additional contact information when missing from frame units.
The
methods engaged to develop the frame were evaluated for fitness by a
panel of technical subject matter experts who reacted positively to
the approach undertaken by IMLS. This panel was comprised of private
sector, public sector, and academic experts.
IMLS
has engaged in extensive additional work since the pilot in order to
ensure that the NMS population frame is valid and fit for purpose for
administering the full NMS. The numerous frame improvements engaged
for the 2025 NMS include: (1) conducting extensive outreach to all
included population frame units by mail and phone to ensure that data
are current and accurate;2
(2) launching an online data entry portal through which new and
previously-entered units can enter or update their museum’s
contact information;3
(3) engaging museum associations to send links to the frame’s
online data entry portal to their communities in order to boost
response to, and amplify awareness of, this effort; (4) updating
existing Yelp data with newly acquired Yelp data; (5) merging the
frame with updated information retrieved from respondents to the NMS
pilot survey; (6) determining the eligibility for both newly added
and existing records on the frame; (7) re-combing the data set to
identify and remove any remaining duplicates; and (8) merging contact
information retrieved from different approaches described above to
ensure that the most accurate and useful contact information is that
which is used for survey solicitations.
The frame currently includes over 21,000 records. Based on the comprehensive scan and outreach to museums, it is believed that the frame is more likely to suffer from over-coverage (ineligible units currently included on the frame) than under-coverage (eligible museums not on the frame). Since the NMS questionnaire includes a screener to identify ineligible units, IMLS prefers to err on the side of having ineligible records in the frame who can self-select out through its screening process rather than miss eligible museums who might be pre-emptively removed under a more aggressive approach; ineligible records will eventually be rooted out naturally through the data collection for an annual survey. The frame will be continually improved moving forward using similar methods of outreach and refinement.
The NMS pilot achieved a response rate of 16.5%. Based on the agency’s outreach to the field, groundwork research in preparation for the survey, revisions to the questionnaire, methodology, and population frame; and review of response rates from other efforts to survey the field, including those from its own pre-pilot respondent research, the anticipated response rate for the 2025 NMS is 35%. See Section B.3 for a discussion of the methods IMLS plans to use to maximize response to the 2025 survey.
B.2. Procedures for the Collection of Information
Preparation for and administration of the NMS will include annual reviews of the survey instrument; annual reviews of data elements, edit-checks, and post-collection processing; updates to the web survey for data collection; post-collection editing and weighting the data file; and release of the results via the data portal.
B.2.1. Data Collection
Programming the Web Survey
The NMS will be deployed as a web application on an online survey platform that meets all relevant federal security requirements.4 The survey will be accessible through any commonly used web browser, including Microsoft Edge, Firefox, Safari, and Google Chrome, using individualized links created by and administered through the survey platform. The programmed survey will be systematically tested to ensure that all included survey logic, such as skip patterns and screen out procedures, is functioning properly prior to deployment.
The web application is designed to minimize response burden and enable timely submissions of high-quality data. A user guide will be available to assist respondents in completing the survey.
Contacting Respondents
Official requests for survey response will be emailed to museum administrators (e.g., Executive Directors, Chief Executive Officers) and will stress the importance of their participation in the survey. These solicitations will be sent via the survey platform using the signatures of prominent IMLS staff to invoke familiarity and attention from respondents. Respondents without an email on file will receive a telephone call and/or postcard to obtain an email address so that they can receive the survey via email.
Techniques to Enhance Response
Nonrespondents will receive reminder emails throughout the data collection period to encourage survey completion. These reminders will occur each week in the form of emails and, for selected subgroups of hard-to-reach units, telephone calls.
See Attachment B for all survey communications.
B.2.2. Statistical Methods
The NMS is a census of all eligible museums in the United States and does not require statistical methods for sampling.
The NMS data will be weighted for nonresponse to reduce the risk of nonresponse bias. Nonresponse adjustments will be based on information known for the respondents and nonrespondents, including museum discipline and location. Information for respondents will be sourced from the population frame and survey data, while information for nonresponders will be sourced from frame research and pre-survey outreach and data available from public data sources such as the Census Bureau, Bureau of Labor Statistics (BLS), and the Internal Revenue Service (IRS).
The variables considered in the nonresponse adjustment will be identified as part of the nonresponse analysis (See Section B.3.2). If only a few variables are significant predictors of nonresponse, the adjustment will be a ratio adjustment based on classes such that the responding museums represent the nonresponding museums within each class. If there are several variables that are predictors of nonresponse, then nonresponse weight will be based on a propensity score adjustment using a logistic regression model with survey response as the outcome (1=respond, 0=no response). The inverse of the propensity score will be the nonresponse weight.
B.3. Methods to Maximize Response Rate and Deal with Nonresponse
B.3.1. Methods to Maximize Response Rates
The NMS methodology incorporates communications and reminders that promote increased response rates. As informed by lessons learned from the NMS pilot, this includes more email and telephone communications and fewer postal mail communications. It also includes the use of population frame outreach prior to the survey’s fielding period to raise survey awareness, gather missing or incorrect contact information, and obtain the name of the best person at the museum to complete the survey. These initial outreach contacts will also ensure that museums are prepared for the time needed to complete the survey.
IMLS is also engaging in extensive outreach with the museum field generally to publicize the survey and its importance. Prominently, these efforts include direct engagement with museum associations to encourage them to reach out to their constituencies with messages encouraging response. These associations have been engaged with and responsive to these efforts throughout the piloting period and into the project’s current phase.
IMLS will use several key components of the communications to demonstrate authenticity of the survey. The NMS prenotification communication will be signed by the IMLS Acting Director. IMLS will continue hosting a web page on its site that provides information about the survey (https://www.imls.gov/NMS). This page will provide a pdf-formatted copy of the questionnaire, information about the fielding period, the survey contractor’s name, and answers to frequently asked questions that may assist in responding to the NMS.
B.3.2. Statistical Approaches to Nonresponse
Nonresponse bias will occur if differential response occurs among population subgroups, where the units who respond to the survey differ from nonrespondents with respect to the survey’s subject matter (questions asked). Differential response occurs when one subgroup responds to the survey at a higher rate than another subgroup (e.g., based on museum size, if smaller museums respond at a higher rate than large museums).
Nonresponse analysis will inform the weighting adjustments made to correct for a respondent base that differs from the population. Nonresponse analysis will therefore focus on the distribution of responding museums as compared to the expected distribution based on the composition of characteristics within the population frame.
Nonresponse analysis will include three components:
Response rates: Response rates for key subgroups will be calculated and monitored throughout data collection. Subgroups will be defined based on the composition of characteristics within the frame, including geography, museum type and size. Differential response rates across key subgroups will guide data collection activities towards increasing response where it is low.
Early/Late Responders: The NMS data collection is designed to maximize response rates. In addition to monitoring response rates during data collection, the NMS team will compare key survey responses between early responders and those responding later in the survey (i.e., those more difficult to reach and/or reluctant responders) to identify any systematic differences in response.
Auxiliary Data: The NMS population frame includes limited information about each museum (e.g., type, size, location). However, information from other data sources may provide a broader understanding of responding and nonresponding museums. Information about the geographic location (e.g., urban/rural, socio-geographic variables from the Census Bureau) can be added to evaluate nonresponse. The survey team will conduct a logistic regression with survey response as the outcome (1=respond, 0=no response) and the auxiliary data as predictors of nonresponse using data appended from sources such as the Census Bureau, BLS, and IRS.
These nonresponse analyses will inform the weighting adjustments applied to correct for a responding sample that is disproportionate from the population. These weighting adjustments will mitigate the risk of non-response bias to the extent that the substantive survey data is correlated with the observed differences in respondents and non-respondents.
B.4. Tests to Minimize Burden and to Improve Utility
IMLS completed extensive groundwork research prior to and running through administering the NMS pilot in order to allow the field to guide what subject matter could be engaged, and which methods of administration could be employed, to minimize burden and improve the utility of the NMS.5 IMLS developed the web-based survey for the NMS pilot, which was thoroughly tested as part of the pilot. The survey team has revised the NMS based on lessons learned from the pilot and cognitive testing of updated questionnaire items. The revised web-based survey will be thoroughly tested by ICF and IMLS prior to survey launch.
B.5. Individuals Responsible for Study Design and Performance
The NMS will be conducted by IMLS. ICF is the contractor in charge of data collection. The following individuals are responsible for the study design and the collection and analysis of the data on NMS.
Personnel Involved with NMS
Person |
Contact Email / Phone |
Institute of Museum and Library Services (IMLS) |
202-653-4759 |
Matthew Birnbaum, Ph.D. Director of the Office of Research & Evaluation
|
202-653-4760
|
Jake
Soffronoff |
[email protected] |
ICF |
|
Sherri
Mamon
|
301-572-0342 |
Randy ZuWallack Principal Statistician |
[email protected] |
Jane Manweiler Principal Survey Methodologist |
[email protected] |
1 Planned for public release in early fall 2024, but is available sooner upon request.
2 Approved under OMB Control Number 3137-0081, to be completed in fall 2024.
3 Ibid.
4 In 2025 IMLS will use the platform Qualtrics for this purpose.
5 See Part A of this supporting statement for more information, specifically sections A.4, A.5, and A.12.
IMLS:
NMS Supporting Statement B |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Subject | Revised per IMLS |
| Author | Sherri Mamon |
| File Modified | 0000-00-00 |
| File Created | 2024-10-28 |