NMS Memo to OMB

To:	Nicholas Fraser, Senior Policy Analyst, Office of Management and Budget
cc:	Matthew Birnbaum, Director, Office of Research and Evaluation; Jacob Soffronoff, Survey Statistician
From:	Connie Bodner, Director, Office of Grants Policy and Management
Date:	November 2, 2022
Re:	Request for Approval under the “Generic Clearance To Conduct Pre-Testing of Surveys” (OMB Control Number 3137-0125) for IMLS National Museum Survey Cognitive Interviews

1. Submittal-Related Information

The Institute of Museum and Library Services (IMLS) is submitting this request for PRA clearance under its Generic Clearance to Conduct Pre-Testing of Surveys (OMB Control No. 3137-0125), through which IMLS is permitted to test various types of survey operations, questionnaire designs, and electronic data collection instruments (e.g., pilot tests, cognitive interviews, and usability studies). Specifically, we request OMB approval to conduct cognitive interviews with museum¹ administrators to test items for a new National Museum Survey (NMS) pilot. Included are two attachments: (1) an online interview sign-up form (Attachment A); and (2) a cognitive interview guide (Attachment B).

2. Background and Study Rationale

Museums contribute to the social, educational, cultural, and economic well-being of the communities they serve; yet, there have been no sector-wide data collection efforts to date that can demonstrate the collective contribution that museums provide to the American public.² Understanding these contributions through a regularly recurring, systematic, sector-wide survey of the nation’s museums will provide essential information to the public and policymakers, as well as to the museum sector and its funders.

In 2021, IMLS awarded a contract to the American Institutes for Research (AIR) to develop and pilot the NMS to gauge the collection’s long-term feasibility as a part of the Federal Statistical System. The vision for the NMS is to capture the scope and scale of museums’ presence and reach within the United States over time. To achieve this vision, the NMS will collect foundational, high-level data directly from museums to inform policymakers, the museum field, and the public about the social, cultural, educational, and economic roles that the nation’s diverse museums play in American society.

IMLS assembled a panel of subject matter experts (SMEs) in 2021, including museum association representatives, museum practitioners, and researchers, to inform the survey development process. This panel has been actively engaged in providing grounded guidance and feedback regarding field needs, existing research efforts, and technical feedback on survey design and implementation. In addition, AIR has undertaken several groundwork research efforts to inform survey development and ensure the collection of high-quality, relevant data. These efforts include conducting a literature review; a benchmarking assessment of relevant Federal Statistical System surveys; a review of state and association surveys of museums; and eight focus groups (under OMB Control Number 3137-0130) and a survey (under OMB Control Number 3137-0081) of museum administrators to better understand the types of data that museums track and can reasonably report.

Informed by this research and consultations with the project’s SME panel, the draft NMS questionnaire requests information about museums’ (a) institutional characteristics; (b) facilities; (c) finances; (d) human resources; (e) admissions and visitors; (f) digital presence; and (g) diversity, equity, and inclusion policies and practices.³

The next step in optimizing the NMS questionnaire is to conduct cognitive interviews with potential respondents to the NMS to test draft survey questions. Cognitive interviewing involves a set of techniques (e.g., think-aloud method, verbal probes) that enable researchers to analyze how participants understand survey questions and whether the questions measure what they were intended to measure.⁴ Conducting cognitive interviews to test draft NMS survey questions will ensure that the final survey questions are easier to understand, less burdensome, more relevant to respondents, and result in higher quality data overall.

3. Recruitment and Data Collection

   1. Design

AIR will conduct two rounds of cognitive interviews with 12 museum administrators per round, for a total of 24 interviews. The total burden will not change for respondents during the testing period, but the research will evolve iteratively as needed to address findings from the cognitive interviews. The first round of interviews will occur after the receipt of OMB approval (anticipated in November 2022), and the second round of interviews will occur after the first round’s results have been analyzed (December 2022/January 2023).

During the interviews, AIR will test draft survey questions for potential inclusion in the final NMS pilot questionnaire. The IMLS-AIR team selected this subset of questions for cognitive testing based on the question characteristics, insights from SMEs, and findings from the previously conducted groundwork research.

3.2 Sampling and Recruitment

AIR will collaborate with two SME panelists who lead professional museum associations (i.e., the American Alliance of Museums and the American Association for State and Local History) to recruit participants for the cognitive interviews. The SMEs will provide a convenience sample of up to 400 museum administrators from their membership lists for recruitment. The museums included on the list will vary in size, discipline, and location to help ensure diversity from across the museum field. The information included in this SME-supplied sampling frame will be maintained by AIR and destroyed following recruitment. AIR will not share the recruitment lists with IMLS.

AIR will use a four-stage process to recruit participants, based on the Dillman method.⁵

• Stage 1: SMEs will send an introductory email to potential participants from their membership lists, informing them of the cognitive interviews as well as a forthcoming email invitation from AIR.

• Stage 2: AIR will send an email invitation to museum administrators with information about the interviews within one or two days. The email will contain a link to an online sign-up form (Attachment A) for participants to complete. Answers to questions on the form will be used to determine potential respondents’ eligibility to t in the interviews and their availability.

• Stage 3: AIR will send an email to non-respondents reminding them to complete the online form.

• Stage 4: AIR will send a final reminder to non-respondents to complete the online form in an effort to fill all interview slots.

Museum administrators who access the online form will answer screening questions to determine their eligibility, after which eligible administrators will answer background questions about their museums and provide their availability for the interviews.

AIR will recruit the full list of museum administrators to achieve recruitment goals and to help ensure diversity in the final sample of participants. This strategy was successful when recruiting participants from this same population for the previously conducted focus groups. Based on that experience, we anticipate that 18% (n = 72) of the administrators on the list will complete the online sign-up form and that 70% (n = 50) of their museums will meet the eligibility criteria⁶ for inclusion in the interviews.

During recruitment, AIR will review responses to the online sign-up form on a rolling basis and purposefully select 24 eligible museum administrators who represent museums of varying sizes, disciplines, and locations for the interviews.

AIR will send an email confirmation to the selected administrators, as well as a calendar invitation and link to join their scheduled interview via Zoom. If a participant is unavailable at the selected time, cancels, or is a no-show, AIR will attempt to reschedule or replace that individual to meet recruitment goals. After recruitment is complete, AIR will send an email to the administrators who are not selected for the interviews, thanking them for their time and interest.

3.3 Data Collection Procedures and Format

AIR’s interviewers will conduct the cognitive interviews virtually via Zoom, recording each session with the permission of the participant. Each interview will last 60 minutes. Interviewers will use a semi-structured interview guide (Attachment B). Prior to beginning each session, the interviewer will explain the purpose and voluntary nature of the interview, as well as the procedures for maintaining participant confidentiality.

During each interview, the interviewer will show one survey item at a time on the screen and instruct the participant to read the item aloud and “think aloud,” or verbalize their thoughts, as they determine their answer or describe their process for answering the question. The interviewer will listen closely to the participant’s verbalizations and ask verbal probes to gather additional context and clarify points that were not evident from the think-aloud method. For example, the interviewer will use verbal probes to:

• verify the participant’s interpretations of the survey items;

• check the participant’s understanding of the meaning of specific terms, definitions, or phrases;

• identify concepts that the participant did not think were covered by the question, but that methodologists and SMEs consider relevant; and

• solicit recommendations for improving the items.

The research will be iterative, in that the survey item wording and format are expected to change during the testing period in response to problems identified during the interviews. However, it is not anticipated that the survey items or format will change substantially during this process, particularly in light of the groundwork already completed to develop the questionnaire.

3.4 Analysis Plans

A note taker will be present during each interview or will take notes from the recording of the interview. The notes will not contain participant identifiers. After each round of testing, AIR will conduct a notes-based analysis, reviewing the feedback provided on each item and summarizing themes or patterns identified across participants. AIR will use the feedback from each round of interviews to make improvements to the items. At the conclusion of the testing, AIR will prepare a final presentation summarizing the testing and results.

4. Consultations Outside the Agency

IMLS and AIR engaged an external expert panel of museum association representatives, museum practitioners, and researchers to provide input into the survey development process. IMLS and AIR also completed foundational research on potential survey subject matter with museum administrators through a quantitative survey and qualitative focus groups. The insights provided through these efforts informed the question development.

5. Justification for Sensitive Questions

This information collection does not request sensitive information.

6. Paying Respondents

The IMLS-AIR team will not provide remuneration to participants in the form of monetary incentives. The online form will instead ask eligible participants if they would like IMLS to recognize their museums (not individuals) publicly for their contributions to the NMS development process. After the testing is complete, AIR will provide a list of museums to IMLS to be publicly recognized in a blog post about the NMS development process. The list includes museums whose administrators both (a) participated in the interviews and (b) gave permission to IMLS to publicly recognize their museums. AIR will not provide IMLS with the names of, or any other information relating to, museums that do not provide this permission.

7. Assurance of Confidentiality

AIR’s institutional review board (IRB), which has federalwide assurance (FWA00003952, expiration 6/25/2024), has reviewed all study materials. AIR will collect personally identifiable information (PII) for recruitment and scheduling purposes only. AIR will not provide IMLS with any PII collected as a part of this study, and all PII will be destroyed by AIR following the project’s reporting. The SME-provided recruitment lists will be destroyed after recruitment is complete.

Each museum administrator who completes the online sign-up form will provide their name, email address, and institution name. AIR will use this information to (a) help ensure that administrators who complete the form do not receive subsequent reminder emails and (b) schedule participants for the interviews. AIR will maintain the recruitment lists and responses to the online form in a secure database separate from interview notes. Interview notes will not contain PII, and interviews will be recorded for research purposes only. Online form data, interview recordings, and notes will be stored on AIR’s secure data servers for the duration of the study, will not be shared with IMLS, and will be destroyed after the final report is submitted. All materials will be accessible only through password-protected laptops with disk encryption, and only AIR research team members working on the cognitive interviews will have access.

Prior to beginning each session, the interviewer will read a verbal script explaining the purpose of the study; the voluntary nature of the interview; and the procedures for confidentiality, including AIR’s assurance that it will not disclose participants’ names or attribute quotes to their museums in any reports or documents shared with IMLS.

8. Estimate of Hourly Burden

The estimated total burden for this data collection is 31.2 hours. We anticipate that 72 museum administrators will complete the 6-minute online sign-up form and that 24 museum administrators will participate in the 60-minute cognitive interviews. To determine the value of museum administrators’ time, we used the U.S. Bureau of Labor Statistics (BLS) wage rate for curators of $30.71 per hour.⁷

Estimated Hourly Burden for Cognitive Interviews

Form	Number of Respondents	Responses per Respondent	Average Response Burden (in hours)	Total Burden Hours	Value of Time
Online sign-up form	72	1	0.1	7.2	$221.11
Cognitive interview guide	24	1	1.0	24.0	$737.04
Total	96	-	-	31.2	$958.15

9. Federal Cost

The estimated cost to the Federal Government for this information collection is $12,320.20, which includes an estimated 176 hours of staff time for five IMLS employees of differing GS levels.

1 The term “museum” is used here to include the range of institution types identified in IMLS regulations, 2 C.F.R. 200 part 3187.

2 Gangopadhyay, P. (2021). Museum survey: Report with vision and recommendations. Institute of Museum and Library Services.

3 The NMS pilot questionnaire will also include subjective questions gauging users’ experience with the survey. These questions are not intended to recur in future administrations, however, and as such will not be included in the cognitive testing of the questionnaire proposed by this request for approval.

4 Tourangeau, R, Rips, L. J., and Rasinski, K. (2000). The psychology of survey response. Cambridge, England: Cambridge University Press.

5 Dillman, D., Smyth, J., and Christian, L. (2009). Internet, mail, and mixed-mode surveys: The tailored design method. New York: Wiley.

6 To be eligible, administrators’ museums must (a) be government or nonprofit entities, (b) serve the public from a fixed physical location, (c) be open at least 120 days per year, and (d) have staff and/or volunteers whose hours make up at least one full-time equivalent.

7 U.S. Bureau of Labor Statistics. (May 2021). Occupational Employment and Wage Statistics. National wage rate for occupation 25-4012, Curators. See https://www.bls.gov/oes/current/oes254012.htm.