The Institute of Museum and Library Services (IMLS) is working with partner OCLC (a global library cooperative previously known as the Ohio College Library Center and Online Computer Library Center), Battelle Memorial Institute (a global research and development organization), and leaders from the library, archives, and museum (LAM) fields on the Reopening Archives, Libraries, and Museums (REALM) Project to develop evidence-based information to help inform LAM institutions as they assess when and how to open public spaces in light of ongoing COVID-related health and safety concerns. The goal of the REALM Project is to develop science-based information and resources to help LAM institutions assess how to handle materials to mitigate exposure to COVID-19 and plan for reopening their premises to the public.
The novelty of the COVID-19 pandemic and its health, social, and economic implications indicated a need to include diverse voices with “on-the-ground” experience. As such, the REALM project formed different groups to take on different advisory roles to the project:
REALM Project Steering Committee: advises the overall scope, priorities, and needs of the project
REALM Science Working Group: advises on the scientific research for the project
REALM Operations Working Group: advises on how to apply the scientific research to the workflow of LAM institutions
REALM Communications Working Group: advises on strategy for external REALM project communications with the LAM community and the public at-large
The Steering Committee and Working Groups support IMLS, OCLC, and Battelle by communicating the concerns and needs of the LAM institutions and identifying the questions that are of greatest priority for empirical investigation. The research test results from Battelle and accompanying toolkit resources developed by OCLC are meant to provide information that LAM institutions can consider when developing policies and procedures for reopening.
As part of this overall effort, Partnership for the Public Good (PPG) is conducting an independent evaluation of the REALM Project. This data collection, comprising archive and museums focus groups and LAM organization audience surveys, supports the in-the-moment needs of OCLC and IMLS as they work to provide information to LAM community around the resilience of COVID-19 on common surfaces encountered in these institutions. The REALM Project evaluation is designed to inform both organizations as to the efficiency and effectiveness of the research process, as well as the processes of the Working Groups and Steering Committee. Much of this work is ongoing. It is also intended to provide information to OCLC and IMLS around the usefulness of the scientific information shared to the LAM community as well as the effectiveness of its distribution. The evaluation has not been designed to be a retrospective consideration of the Project, but rather to inform decision-makers at OCLC and IMLS, in near real-time, as to what is working and what isn’t; enabling them to modify the efforts of the Working Groups and Steering Committee as well as to adjust methods of distribution of information to the larger LAM community.
The REALM Project is unique in that it was created in response to an emergent public safety issue with various ramifications for the safe operations of LAM institutions. The design of the evaluation, and efforts already conducted to support it, reflect that emergent nature – working to get useful information around the process and effects of the Project to the key decision-makers (OCLC and IMLS), affording them the opportunity to adjust the project in as real-time as possible, and thus helping them effectively get information to the Project’s target audiences. We expect the results of this evaluation to not only serve by providing useful, near real-time information, but to also enable IMLS to consider the Project after the emergency has passed, enabling the Institute to better learn from the experience and have robust practices in place should a similar event occur in the future.
Given the range of project stakeholders and the project’s goals, this evaluation has two primary mandates:
Understanding how the distinct parties (i.e., OCLC, Battelle, IMLS, the Steering Committee, and the Working Groups) are working together and the implications of how these groups work together for the effectiveness of the project overall.
Understanding the extent to which LAM institutions have found that the research results and toolkit resources informative and useful.
While some elements of the REALM Project evaluation do not involve collecting information from the public, we present the complete evaluation design so that the evaluation’s focus groups and audience surveys involving public respondents can be considered in context of the larger plan. The activities for this entire evaluation include:
Steering Committee and Working Group Meeting Observations: PPG observed communication, decision-making, group dynamics, and the focus of discussions during Steering Committee and Working Group meetings to better understand how the groups were functioning within the overall project structure.
Steering Committee and Working Group Member Interviews: PPG conducted short interviews with a sample of representatives from the Steering Committee and Working Groups to gather their perceptions on the working relationships within and among the distinct groups, as well as their perceptions about the extent to which the project is addressing its stated goals.
Administrative Data: Comprising information created by OCLC over the course of the project, this data includes tracking of dates and number of Steering Committee and Working Group meetings, information presented and discussed at these meetings, recommendations shared by these groups, and eventual decisions and dissemination of information by OCLC through its various communication channels.
After Action Reviews: Following the dissemination of a research product (e.g., literature review, results of lab research), PPG will conduct an after-action review session (AAR) with the OCLC staff. The after-action review is a structured review process that allows groups to reflect on a recently completed activity and to think about what insights and learnings can be gleaned from their work for future opportunities for improvement.
Archive and Museum-Specific Focus Groups: Based on findings from PPG’s observations and interviews, a need emerged to gather additional information about museum and archive institutions, in particular, as distinctions in their re-opening concerns and priorities as compared to those of libraries emerged, and given that OCLC has more long-standing relationships with libraries as compared to museums and archives. Given that the guidance for the evaluation was to assess “the project’s capacity to reach and suitably communicate to a broad spectrum of archives, libraries, and museum[s]…,” PPG will conduct focus groups with archive and museum organizations in order to hear more intentionally from these institutions about how the REALM project is providing useful and relevant information. This information will help OCLC and IMLS adjust both outreach to the archive and museum communities as well as improve dissemination of valuable information.
Audience Survey: PPG will administer a survey to LAM institutions across the country to understand how the research and toolkit resources coming out of the REALM project have been used, if at all. The survey design seeks to test the following evaluation questions across the LAM community:
How has the LAM community responded to the laboratory research test results? To what extent have the laboratory research findings been valuable to the LAM community?
What was the quality of the LAM-specific research information? To what extent did perceived quality of the research information differ by sector (i.e., libraries, archives, museums)?
How did the results of the research and the availability of the toolkits shift LAM institutions’ practices?
How effective has the REALM Project been in disseminating the information? What channels were most or least effective?
To what extent do LAM institutions feel the toolkit provides them with more sector-relevant information than received from other sources?
We are planning to deploy this survey at two time points—May 2021 (pending OMB approval) and late summer/early fall 2021. Elements of the toolkit and research have already been rolled out so May is an ideal time to capture the sentiments of those who have already accessed the resources. This will provide OCLC and IMLS with information as to who is receiving the information from the toolkit and research as well as how it is being used.
A second survey was planned because OCLC anticipates 1) a heavier push of toolkit resources through the first half of 2021 and 2) changes will be made to the toolkit resources based on feedback from the first survey. A second survey will provide an opportunity to gauge community feedback after the changes are made – informing further modifications to the dissemination process used by OCLC.
The evaluation will provide rich information to OCLC and IMLS project leaders that assesses the process and effectiveness of the internal working group and steering committee efforts as well as provide insights as to who is receiving the resultant information and how it is being used. While intended to assist in near real-time decision-making, we expect that both OCLC and IMLS will use the results of the evaluation to inform future work should similar emergencies arise. Finally, while not considered primary audiences for the evaluation, other federal and non-federal entities could use the information derived from this evaluation to inform their ongoing efforts to address the current crisis as well as for future planning and implementation in similar emergencies.
All survey data will be collected online via SoGoSurvey. The survey instrument includes the instructions, the questionnaire, and the PPG contact information in the event a respondent requires assistance. Key terms are defined in the survey instructions, as well as throughout the survey, as needed. All data (e.g., survey responses) related to this evaluation are kept in a secure digital format, accessible to only the PPG evaluation team.
Additionally, the evaluators will use the Zoom virtual platform to connect participants in the focus groups. The discussion will be recorded for transcription purposes and will be destroyed once the transcription has been completed. Data collected in this manner will not be attributed to the individual participants in the focus groups, but rather will be reported only in aggregate as being done for the survey data collection. As such, no information will be shared in a manner that can be attributed to an individual. To further protect participant privacy, no data will be attributed to an individual focus group. As such, the themes that emerge will be represented as coming from the totality of the focus groups and not any individual or group.
This is an evaluation of the LAM community’s perceptions and use of the research test results and toolkit resources that were specifically developed from the REALM project. No other evaluation effort is underway or data sources available to assess community feedback (value, understanding, and additional needs of LAM community members) on the test results and the toolkit resources. Therefore, this evaluation is not duplicative of any other data collection.
LAM organizations will at most be asked to participate in the survey twice over the duration of the project. Access to the research test results and toolkit resources is not conditioned on survey participation and participants will be informed that participation in the survey is voluntary. There is expectation that no more than one person from any organization needs to participate (although more than one person per organization may participate). The average time to complete the survey is approximately 15 minutes. Surveys will be administered and submitted electronically. Respondents will not need to return anything via postage mail. The surveys will be open for a minimum of two weeks and that timeframe may be extended, should a representative sample of organizations’ data not be collected.
This evaluation is providing critical information to OCLC and the Project Steering Committees and Working Groups as well as to IMLS. By reducing the data collection from two surveys to one, the information would be helpful for initial modification of the project’s outputs but would not provide the insights necessary to identify whether those modifications were effective – providing IMLS with potential better practices should similar efforts be required in the future.
The proposed data collection activities are consistent with the guidelines set forth in 5 C.F.R. Part 1320 (Controlling Paperwork Burden on the Public). There are no special circumstances that require deviation from these guidelines.
IMLS published a 60-Day Notice of Proposed Information Collection Request for Comments in the Federal Register on October 16, 2020 (FR 85, 65878). Written comments were due to the Office of Grants Policy and Management, Institute of Museum and Library Services on or before December 11, 2020. One comment was received. A 30-Day Notice for OMB Review was published on March 17, 2021 (FR 86, 14770).
No payments or gifts will be given to respondents.
During each primary data collection, respondents will be notified of the following:
the purpose of the data collection and data collection methods being used;
that all data collected will be anonymized and only reported in de-identified, aggregate form; and
that participation in the data collection is voluntary.
Access to the raw data collected will only be granted to PPG team members actively involved in the project evaluation. PPG uses the cloud-based platform Dropbox to store its files. Dropbox is ISO, U.S. Privacy Shield, SOC 1,2, and 3 reports, CSA STAR and is HIPPA/HITECH compliant1. We also have access to a local secured server which can only be accessed through an encrypted VPN portal. These systems are accessible only to PPG employees.
PPG will collect no personally identifiable information (PPI). Data will be analyzed and reported in aggregate. Where individual quotes from qualitative data collected might be used, the quotes will be screened for identifying information and those portions redacted.
No sensitive questions will be asked of study participants as seen in Appendix B with the proposed instrument. All questions will be related to LAM community members’ satisfaction, perception of quality, perception of communication, and use of the research test results and toolkit resources, as well as how these can be improved to support the LAM community.
Instrument |
Number of Participants |
Estimated Response Time |
Total Burden Hours |
Archive & Museum-Specific Focus Groups |
36 |
60 minutes |
36 hours |
Audience Survey 1 |
4,000 |
30 minutes |
2,000 hours |
Audience Survey 2 |
4,000 |
30 minutes |
2,000 hours |
Research participants do not incur any costs other than their time responding. For participation in the surveys and focus groups, we estimate a total cost of $121,080, equaling a total of 4,036 hours at a rate of $30 per hour.
The total estimated cost to the federal government for the REALM Project evaluation is $92,056. This breaks out as follows:
IMLS staff time (two staff members for 62 hours at a rate $59 per hour): $3,658
OCLS staff time (three staff members for 83 hours at a rate of $90 per hour): $7,470
PPG contracted staff time (six staff members for 297 hours at average of $269 per hour): $79,893
REALM Project Steering Committee and Working Group members (23 individuals for 30-minute interviews at $90 per hour): $1,035
This is a new submission. There are no program changes or cost adjustments.
The evaluation starts July 2020 with document review and continues through October 2021.
Table A.16 Project Schedule
Introductory meetings with OCLC |
Jul – Sep 2020 |
Oct – Dec 2020 |
Jan – Mar 2021 |
Apr – Jun 2021 |
Jul – Sep 2021 |
Oct – Dec 2021 |
Document review |
Ongoing |
|||||
Meeting observations |
Aug |
|
|
|
|
|
After action review |
Aug |
|
TBD |
|
||
REALM Steering Committee and Working Group interviews |
|
Oct |
|
|
|
|
OMB approval |
|
|
|
May Est |
|
|
1st audience survey |
|
|
|
30 Days after OMB approval |
|
|
Archive & museum specific focus groups |
|
|
|
30 Days after OMB approval |
|
|
2nd audience survey |
|
|
|
|
Sep |
|
Final report |
|
|
|
|
|
Oct |
We are not requesting an exemption from the requirements to display the expiration date for OMB approval. All data collection materials and documentation will include the OMB approval number and expiration date.
No exceptions to the certification statement apply to the REALM Project evaluation.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Amanda Bakale |
| File Modified | 0000-00-00 |
| File Created | 2021-03-19 |