Part A - IMLS Response to OMB Comments 2019-04-22 Comments

Section A. Justification

A.1. Necessity of the Information Collection

The study is being instituted as part of IMLS’s statutory mission to conduct analyses, identify national needs, and identify trends for its services. https://www.imls.gov/sites/default/files/mlsa_2010_asamended.pdf

See 20 U.S.C. § 9108. The study is a new data collection request, and the data to be collected are not available elsewhere. The data collection activities are planned for April 2019 through March 2021. The study will provide the Institute for Museum and Library Services (IMLS) with a better understanding of museum, library, community, and field-level outcomes associated with the Catalyzing Communities Initiative (CCI) and aspects of the initiative and its grantees that mostly likely contributed to and inhibited success.

IMLS plays an essential role in ensuring that the public dollars invested in museums and libraries benefit the communities they serve. To remain vital to the communities they serve, museums and libraries are increasingly expected to contribute to community wellbeing through activities that promote community assets and address community needs. Making such contributions, however, requires knowledge and skills for community engagement that are not typically part of educational programs that prepare people for professions in these institutions. In addition, organizational structures and processes in museums and libraries are not typically set up to support and facilitate such engagement.

In recognition of this capacity-building opportunity, IMLS designed CCI, a grant program for museums, libraries, and their community partners interested in enhancing the benefits of their collaborative, community-based efforts (i.e., asset-focused, community-driven collaboration) through (1) intensive training to build museums and libraries capacity for asset-based community engagement¹; (2) consulting to support implementation of the project grants in local communities, and (3) creating peer opportunities for the museum and library grantees, along with their core partners, to learn from each other and share innovative ideas and promising practices. One- to two-year grants were awarded to 12 organizations in each of two consecutive cohorts, the first in the fall of 2017 and the second in the fall of 2018. Grantee organizations are either museums, libraries, or other institutions that proposed to collaborate closely with museums and libraries.

This proposed evaluation supports the Agency’s FY 2018-2022 Strategic Plan, which frames how the Agency plans to meet the essential information, education, research, economic, cultural, and civic needs of the American public. Under the Strategic Plan, IMLS’s objectives include “adapt to the changing needs of the museum and library fields by incorporating proven approaches as well as new ideas into IMLS programs and services” (4.2), “encourage library and museum professionals and institutions to share and adopt best practices and innovations” (2.2), and “identify trends in the museum and library fields to help organizations make informed decisions” (2.3).

This proposed study will be the first evaluation of the entire CCI initiative. It is a summative assessment of CCI implementation and initial outcomes across the two cohorts of grantees and their 24 diverse projects. This assessment builds on the 24 grantees’ evaluations of their own individual projects. It uses a set of common indicators to assess implementation and interim outcomes among museums, libraries, local networks, and communities across all grantee projects.

This proposed evaluation is designed to inform future IMLS strategies, both within the CCI program and in other capacity-building programs. It is also intended to yield findings that are applicable for others interested in building museum and library capacity to collaborate with other institutions and community members for taking on community-wide concerns. In addition, the data collection will help to understand what lessons can be drawn across a diverse set of projects in local communities that can inform the field about using approaches such as CCI. Finally, the long-term goal of the study is to develop approaches, tools and training for museums and libraries to serve as more effective community partners as a benefit to the public.

IMLS will formally review these findings and consider the relevance to enhance and strengthen the various ways it invests in the museum and library sectors (grant-making, technical assistance, capacity-building, convenings, etc.). The Agency's Five-year Strategic Plan includes a new commitment to build the capacity of museums and libraries to support community well-being and lifelong learning for all Americans. While the universality of the various findings from this study are currently unknown, it will inform future investment approaches by the Agency in technical assistance and cohort style grant making across the Agency's various funding programs. In addition, this study is generating findings specific to the sector’s community engagement work, identifying needs for technical assistance relating to evaluation and community development tools.

IMLS plans to widely share these results with leaders and practitioners in the museum and library sector in hopes that they will use the findings to enhance or begin their community focused work with the themes that emerge from the Community Catalyst evaluation in mind. 1) Moving beyond traditional conference presentations is an obvious step, the agency also plans to post usable resources gleaned from grantees' work on its website much as was done for "Community Salute" (Veterans study in 2016/2017). 2) Further understanding the readiness of the sector to adopt this type of work will also be facilitated by "listening sessions" with local communities (e.g., bringing together museums, libraries, funders and partners within a city or county region) to understand and work through the approaches within their own contexts. 3) We also hope to urge federal partners and non-federal grant making agencies to consider museums and libraries as potential grantees and partners, and that they consider using frameworks grounded in asset-based community development and cohort style grant making.

A2. Purposes and Uses of the Data/Information

The proposed evaluation study design reflects the Agency’s understanding of the program, as reflected in the CCI theory of change (see Figure 1, attached as a separate file). This theory of change (TOC) was created with Agency staff involved with this initiative in September 2018 and builds on the release of IMLS’s 2016 publication “Strengthening Networks, Sparking Change: Museums and Libraries as Community Catalysts.” The theory of change suggests that by providing funding for local library- and museum-involved collaborative projects and ongoing training and technical assistance to build asset-focused, community-driven collaboration capacity, and by supporting learning and diffusion beyond grantees;

• there will be changes in libraries’, museums’, and their partners’ mindsets, knowledge, skills, and practices needed to elicit community priorities for social change and to build on community assets in co-designing and implementing solutions;

• which will, in turn, lead to strengthened local networks and increased empowerment and social well-being within local communities, as well as a higher likelihood of engagement in asset-focused, community-driven collaboration (and systems-level support for such) beyond the grant period.

The TOC shapes the scope of the evaluation’s planned assessment of the CCI’s implementation and outcomes to date. It does so by assessing how this particular capacity-building model has used third-party technical assistance (TA) providers to build capacity in asset-focused, community-driven collaboration within cohorts of grantees supports museums’ and libraries’ engagement with partners in their communities on identified community-wide issues of concern (see Figure 1). The investigation aims to answer the following five questions:

1. To what degree and in what ways did CCI-participating museums, libraries and their grantee partners develop capacity to be community catalysts?

2. In what ways did CCI-participating museums, libraries and their grantee partners change practices to better engage their communities in co-creating and implementing community change?

3. To what degree and in what ways were CCI-participating museums, libraries and their grantee partners able to create a local ecosystem that supports community social change?

4. To what degree and in what ways do CCI communities experience positive social change?

5. How did outcomes vary across types of CCI supports/inputs used, cohorts, or characteristics of involved library/museums, partners, or communities?

It is important to contextualize what this study will and will not be able say with respect to the relationships among capacity building, asset-focused, community-driven collaboration practice changes and other outcomes for museums/libraries and local networks and communities. The mixed methods design attempts to identify contributions of the CCI approach to these outcomes but is not causal in nature. We do not make comparisons to any museums and libraries that are not implementing CCI, or attempt to prove that one or more aspects of an asset-focused, community-driven collaboration approach will lead to changes in outcome. Instead, the study attempts to unpack how asset-focused, community-driven collaboration plays out in the 24 settings where it is being implemented, including exploring where it worked most successfully and where it faced significant challenges. The study explores contribution within specific case examples by gathering multiple sources of evidence for contribution and exploring strength of evidence for alternative drivers of outcomes seen achieved in CCI grant projects. When publishing the results of this study it will be made clear in those publications that IMLS is using new, exploratory methods that will help provide valuable information to form the basis for stronger evaluation approaches going forward, but that the agency is not drawing definitive conclusions from these methods at this time.

To the degree that a preponderance of evidence supports CCI contribution and rules out alternative drivers of change, we can be more confident about the nature of CCI’s potential impact. Such a design provides directional information to IMLS and others about the strengths and limitations of this particular capacity-building model, as well as conditions that tend to support and hinder success, as they consider future strategies.

A2.1 Who Will Use the Information

The study will provide rich information to CCI participants (i.e., museums, libraries, and local partners), the museum and library fields, IMLS, other funders in the museum and library fields, and policy makers about when and how museums and libraries contribute to catalyzing positive community outcomes. CCI participants may use the results to inform course corrections and communications to support of CCI-funded project sustainability or future change efforts in their communities. The results may also be used by leadership in other museums and libraries to inform decisions regarding investments in professional development and collaborative community change efforts within in their own localities. IMLS may additionally use these results to inform CCI strategies, and they and others may use the findings to inform future investments. Study findings will be shared with the public through various media and press outlets that include but are not limited to the IMLS and partner websites and other media channels as well as through field level webinars and presentations.

A.3. Use of Information Technology

IMLS takes its responsibility to minimize burden on respondents very seriously and designed this project with that goal in mind. All administrative documents are in a secure digital format, accessible to only the ORS investigator team. The evaluation systematically begins the investigation in reviewing digital administrative documents before proceeding to collecting additional information of grantees and their partners through surveying and interviewing.

By designing web-based surveys using Survey Monkey for museum/libraries and their local partners, the Agency has eliminated hundreds of hours of labor that would have been required to administer a paper-based on-site survey since it becomes much more likely to survey every member organization of a participating partner network due to minimal costs associated with adding participants. Thus, the electronic nature of the museum/library and local community partner surveys provides the most efficient mechanism for IMLS to capture responses from museums, libraries, and their partners. The electronic surveys, and all communication about them, will be compliant with Section 508 of the Rehabilitation Act.

In addition, the study team will be using existing administrative data from CCI third-party providers (site consultants) through a data sharing agreement accessible via a secure SharePoint site. The ORS team will collect interview data from all participating project team via remote video- and tele-conferencing systems and will be audio-recorded and transcribed. Audios and transcripts will be stored on a secure SharePoint site accessible only to the ORS investigators and will be destroyed at the end of the study.

The results of the project will be shared with the library and museum fields and the public via the IMLS website.

A.4. Efforts to Identify Duplication

The current study is the first cross-grantee evaluation of IMLS’s CCI initiative. As a result, there is no similar ongoing data collection being conducted that duplicates the efforts of this proposed study.

While there are numerous studies and evaluation efforts related to community engagement and community transformation (Center for the Study of Social Policy, 2010, 2015, 2017), a scan of existing literature conducted by the evaluators confirmed that there is little research on capacities needed for museums and libraries to engage in these efforts, nor on the practices changes they make, and the outcomes accruing in their local communities.

Further, as referenced above, the evaluators will be reviewing documents provided from IMLS and this agency’s two third-party technical assistance providers (a cooperator and an EPA team hired through an interagency agreement). We are including as much of this data as possible in order to avoid duplication and minimize burden to respondents.

A.5. Methods Used to Minimize Burden on Small Businesses

There are no small business entities or other small entities involved in this data collection.

A.6. Consequences of Less Frequent Data Collection

This is a voluntary data collection across two years. This study will provide IMLS with quantitative and qualitative measures to gain valuable insights into the relationships between participation in the CCI initiative and the outcomes for museums, libraries and their local communities. Without this evaluation, IMLS would have no methods for analyzing and assessing the contributions of its program in helping inform future policy choices.

Conducting the collection less frequently or with fewer CCI programs would not only impede the Agency’s ability to track the evolution of this initiative, it also would also deprive program grantees, their partners and other interested parties with an opportunity to learn more about the contributions of the CCI approach in the museum and library field.

A.7. Special Circumstances

The proposed data collection activities are consistent with the guidelines set forth in 5

CFR 1320.6 (Controlling Paperwork Burden on the Public-General Information Collection

Guidelines). There are no special circumstances that require deviation from these guidelines.

A.8. Consultations Outside the Agency

IMLS received one public comment from the Federal Register notice dated December 3, 2018.

To help ensure rigor and credibility of the evaluation, this investigation is engaging with an Evaluation Steering Team (EST) to provide feedback on the investigation as it proceeds from design to implementation as well as in disseminating findings. The EST reviewed the evaluation design and tools for this submission on November 5 and will continue to provide guidance on clarifying evaluation questions and outcomes, as well as consideration for ensuring methods will yield culturally relevant results and assisting with dissemination of evidence.

The proposed EST is comprised of nine external experts and 5 members from IMLS and its TA Providers. This group brings a breadth of experience in the museum/library field, community engagement processes, and mixed methods/qualitative evaluation designs. It will meet four additional times over the course of the evaluation in reviewing preliminary and final findings. These members also will be on call to provide real-time guidance on evaluation issues as they arise, including selection of case study sites.

A.9. Payments or Gifts to Respondents.

No payments or gifts will be given to respondents.

A.10. Assurance of Confidentiality

ORS Impact has institutional policies and processes that ensure we (1) review our projects for compliance to human subject protections, (2) protect the identifiable information of evaluation participants to the maximum extent allowed by contractual obligations, and (3) communicate to evaluation participants, prior to data collection, any risks and the volunteer nature of participating in the evaluation.

The ORS Impact project team will submit for internal IRB review by a senior consultant certified in human subjects review protections and the CEO:

• the purpose of the data collection and data collection methods being used;

• the respondent populations and how they will be identified and accessed;

• whether the data will be anonymous, confidential, or neither, and if the data are confidential or neither, explanation of why identifiers are necessary;

• how data will be stored (e.g., electronic files, hard copies);

• who will have access to data and for how long; and

• potential risks and burdens of the project to participants.

At the beginning of the surveys and all qualitative data collections, participants will receive written or verbal assurance that their participation is voluntary, that they can opt out at any time, that their individual responses will not be reported except in aggregate and without personally identifiable information, and that any information shared outside of ORS Impact will not link their responses to their individual or organizational identities unless required by federal law. In written evaluation summaries results will be reported in aggregate form only. In cases where we want to include a quote from an individual respondent for clarity or illumination, we will seek permission from that individual to use the quote.

ORS Impact stores all its evaluation data on a local secured server housed on our premises. The server can only be accessed through an encrypted VPN portal. The system is only accessible to ORS Impact employees. We will add a layer of access security to targeted data folders whose access is restricted to named individuals on a need to know basis.

We will protect personally identifiable information by storing names and other PII separate from the actual data collected. We will assign linking codes to data elements for each organization and individual respondent (e.g., G01P01 to represent the first participant for the first Grantee). The list collating identification numbers with names will be stored in a separate secured folder for which only the third-party evaluation team will have access.

With respect to collecting data from community beneficiaries in focus groups (see Part B Respondent Universe/Samples Set 4), we are likely to have access to sensitive information related to participation in particular project grant interventions or related to community member demographics.

• Focus group participants will be asked to provide written informed consent after being apprised of the potential risks and benefits of the study. We anticipate any potential risks to be minimal.

• We will not identify any individual responses who participate in focus groups to the grant team, IMLS or in reporting.

• Data, including demographic characteristics will be reported only in aggregate by Grantee.

• All focus group recruitment information will be kept in an encrypted secure folder and will be destroyed at the end of the evaluation.

• We will not collect data from individuals under the age of 18. We will not collect data from anyone else classified as a member of a vulnerable population without IRB approval.

Data sets provided to IMLS at the end of the study will not contain any personally identifying information (PII)—such as name or address of respondents or their organizational affiliation—that could permit disclosure or identification of respondents, directly or by inference. We will destroy all personally identifiable information at the end of the study.

A.11. Justification for Sensitive Questions

As stated above in A.10, we do not intend to ask sensitive questions during the course of the study. However, some data collected from community member beneficiaries may be deemed sensitive because of the intervention in which they participate or by nature of their demographic characteristics. We will potentially be collecting data from these individuals as part of community focus groups to understand how CCI is benefitting or not the intended beneficiaries. This information will be used to triangulate the statements made by project partners and third party TA providers about the degree to which the partners are shifting in their approaches to community-wide collaboration and for assess their perceptions about the library/museum as a trusted community ally. To protect this sensitive data, we will adhere to the procedures described in A.10.

A.12. Estimates of Hour Burden to Respondents/Table

(See attached Excel sheet for detail by instrument, total burden listed below)

Number of respondents Estimated response time* Total burden hours

599 52.5 minutes 668

A.13. Estimates of Cost Burden to Respondents

Research participants do not incur any costs other than their time responding.

A.14. Estimates of Cost to Federal Government

The total one-time contracted cost to the Federal Government for this project is $603,100, representing an annualized cost of $241,244 for a 30-month project.

A.15. Reason for Program Changes or Cost Adjustments

This is a new information collection request.

A.16. Project Schedule

Table A.4 below shows the estimated evaluation timeline from September 2018 through March 2021.

Table A.4 | CCI Evaluation Timeline (Q1 = Oct-Dec 2018; Q10 = Jan-Mar 2021)

	Q1	Q2	Q3	Q4	Q5	Q6	Q7	Q8	Q9	Q10
Evaluation design and OMB review	X	X
Engage Evaluation Steering Team (EST)	X				X				X
Administrative data review
Grant project team interviews			C1			C2			C2
Museum/library surveys			C1 C2			C2
Local network partner surveys			C1 C2			C2
Technical Assistance Provider interviews			C1			C2			C2
Case studies					C1				C2
Provider focus groups				X						X
Learning and results validation with IMLS				X		X		X		X

Note: C1 = Cohort 1, C2 = Cohort 2

A.17. Request to Not Display Expiration Date

IMLS will display the expiration date of OMB approval and OMB approval number on all instruments associated with this information collection, including forms and questionnaires.

A.18. Exceptions to the Certification

No exceptions are necessary for this information collection.

1 All grantees and their lead project implementation partners receive training and technical assistance throughout their respective grant periods in Asset-Based Community Development (ABCD), an “approach focused on discovering and mobilizing the resources that are already present in a community” (p. 12, Green, Moore, & O’Brien, 2008, When People Care Enough to Act).

8