CCI Research Questions by Method Table B

CCI Question by Method Table B.1.docx

Community Catalyst (CCI) Program Evaluation: The Roles of Libraries and Museums as Enablers of Community Vitality and Co-Creators of Positive Community Change

CCI Research Questions by Method Table B

OMB: 3137-0113

Document [docx]
Download: docx | pdf

CCI Evaluation Questions, Outcomes and Indicators

Definitions

  • Community organizations: organizations and institutions within the local communities that are involved in the design and/or implementation of local CCI-funded project

  • Community members: residents, citizens, actors, members of associations that are involved in the design and/or implementation of local CCI-funded project

  • Community partners: Community organizations and community members (per above definitions)

  • Museums and libraries: All museums and libraries involved in CCI-funded projects—not necessarily grantees

  • Grantee partners: Grantees that partner with museums and libraries within their CCI-funded projects but are not museums or libraries

CCI Outcomes Questions

  1. To what degree and in what ways did CCI-participating museums, libraries and their grantee partners develop capacity to be community catalysts?

  1. In what ways did CCI-participating museums, libraries and their grantee partners change practices to better engage their communities in co-creating and implementing community change?

  2. To what degree and in what ways were CCI-participating museums, libraries and their grantee partners able to create a local ecosystem that supports community social change?

  3. To what degree and in what ways do CCI communities experience positive social change?

  4. How did outcomes vary across types of CCI supports/inputs used, cohorts, or characteristics of involved library/museums, partners, or communities?

Table B.1 | Changes in CCI Grantee Capacity, Practices, and Local Networks/Communities (EQ #1, #2, #3, #4, and #5)

Evaluation Question/Associated Outcome

Example Indicators

Data sources

  1. To what degree and in what ways did CCI-participating museums, libraries and their grantee partners develop capacity to be community catalysts?

Changes in beliefs, attitudes, knowledge and skills among those involved in the CCI-funded project

  1. Increased awareness of power dynamics in communities and openness to shifting power dynamics in community change efforts


Note: early change after initial training/TA

  • Orientation and commitment toward shifting power from institutions to community members

  • Awareness of power differentials in the community

  • Understanding root/systemic causes of power differentials

  • Understanding of potential roles for the library/museum in efforts to shift power differentials/address root causes thereof

  • Administrative data including ABCD Self-Assessment Toolkit

  • TA provider notes and interviews

  • Project team interviews, museum/library surveys, partner surveys, case studies

  1. Increased understanding of assets and networks, and belief in benefit of using an asset-based approach in local community change efforts


Note: early change after discovery processes and additional TA

  • Understanding of assets in local community

  • Understanding of key actors and influencers within networks in local community

  • Understanding of the historical and cultural context of local community

  • Understanding of what community members care about in local community (i.e., community member priorities)

  • Understanding of areas of local “common ground” between community member priorities and outcomes that institutional partners are working toward

  • Understanding of roles that identified assets can play within community change efforts and how to apply those in own communities

  • Belief in benefit of using an asset-based approach in community change efforts

  • Belief in benefit of adapting strategy to address existing community member priorities

  • Administrative data including asset maps

  • TA provider notes and interviews

  • Project team interviews, museum/library surveys, partner surveys, case studies

  1. Increased capacity to convene diverse stakeholders and facilitate co-creation and joint implementation of a common agenda (intermediate-term/during project implementation)

  • Positive orientation toward collaboration and relationship-building as primary mechanism for community change

  • Strategies and skills for

    • guiding diverse partners in dialogue and working together

    • building and sustaining trust and empathy among community members and institutional partners (e.g., active listening, appreciative inquiry)

    • authentically engaging community members at all stages of community change efforts

  • TA provider notes and interviews

  • Project team interviews, museum/library surveys, partner surveys, case studies

  1. Increased flexibility/ agility/ adaptability to respond to changing nature/ contexts of grantee projects (intermediate-term/during project implementation)

  • Openness to altering course in response to changing community priorities and contexts

  • Sense of accountability to community members versus institutional partners

  • Strategies and tools for capturing, responding to, and adapting to changing community priorities and contexts

  • TA provider notes and interviews

  • Project team interviews, museum/library surveys, partner surveys, case studies

Organizational systems changes among museums, libraries, grantee partners

  1. Increased alignment of organizational priorities and expertise with co-designed, jointly implemented, asset-focused, community-driven collaboration

  • Leadership support and vision for asset-focused, community-driven collaboration (perceived and explicit/public among which levels of leadership/staffing hierarchy)

  • Resources dedicated to asset-focused, community-driven collaboration (e.g., investment in professional development, staff time allocation, part of budget vs dependent on grants)

  • Documented institutional vision, mission statement, and/or strategic plan aligned with/prioritizing asset-focused, community-driven collaboration

  • TA provider notes and interviews

  • Project team interviews, museum/library surveys, case studies

  1. Increased structures and processes supporting authentic engagement of community members

  • Empowered decision-making groups comprised of diverse partners

  • Standing meetings and events at times and locations convenient for community members and inclusive agenda structures

  • Internal communication structures/processes to support effective, bi-directional, and culturally responsive information flow and engagement with the community

  • Flexible policies and practices to address community-driven programs

  • TA provider notes and interviews

  • Project team interviews, museum/library surveys, case studies

  1. In what ways did CCI-participating museums, libraries and their grantee partners change practices to better engage their communities in co-creating and implementing community change?

  1. Increased interactions with community members and non-traditional partners outside of the museum or library (informal discovery practices)

  • Visits to community-driven events and gathering places (in contrast to only institutional events for the community)

  • Interactions with community associations, leaders, and individuals

  • Administrative data including power ladders

  • Grantee interviews, surveys and case studies

  1. Increased engagement in discovery processes with community members (asset mapping, learning conversations) (formal discovery practices)

  • Use of asset mapping to identify a wide range of community assets from individual, associations, institutions, places, and culture

  • Facilitation of community listening sessions to identify community priorities

  • Engagement of additional relevant cross-sector Institutional partners in dialogue with community members

  • Administrative data including asset maps

  • TA provider notes and interviews

  • Grantee interviews, surveys and case studies, including social network analysis of partner relationships

  1. Increased co-creation and joint implementation of a common agenda with community members and cross-sector partners, within CCI-funded project

  • Community members have lead/impacted decision-making in implementation of CCI-funded projects

  • There is a common agenda that was co-created by diverse community partners around community-driven priorities (agenda = common vision/outcome goal and understanding of what assets should be brought to bear to help realize vision/plan)

  • Solutions are assets-focused and emerge from community member strengths and priorities

  • Institutional practices support community action (versus drive the efforts)

  • Community partners implement and support strategies and solutions aligned with common agenda

  • Community partners use communication processes and structures to provide input, and support collaboration/ engagement

  • Community partners use common indicators to track progress

  • Community partners collectively engage in ongoing evaluation, data use, and learning to assess progress and impact

  • Administrative data including asset maps

  • TA provider notes and interviews

  • Grantee interviews, surveys and case studies

  • Partner surveys


  1. Increased planning for engagement in efforts that include co-creation and joint implementation of a common agenda with community members and cross-sector partners, beyond CCI-funded project

  • Plans for integrating asset-focused, community-driven collaboration into other/future community change efforts

  • Rest same as C but beyond CCI-funded project

NA

Changes in local ecosystems

  1. To what degree and in what ways were CCI-participating museums, libraries and their grantee partners able to create a local ecosystem that supports community social change?

  1. Increased number of connections/ partnerships among museums, libraries, other organizations, and community members, particularly including citizen associations, historically under-represented groups, civic leaders, and local funders

  • Number and type of partners (e.g., non-institutional and institutional partners, different sectors)

  • Number of partners that are citizen associations, historically under-represented groups, civic leaders, and local funders

  • Network characteristics including density of the network (degree of cohesion/inter-connectivity in the network), degree of centralization (degree to which activity is centered in a few organizations (high) or spread across organizations (low)), and size of the network

  • Administrative data: Asset maps, TA provider notes

  • Museum/library and partner surveys—social network questions

  • Project team interviews

  1. Deeper connections/ partnerships among museums, libraries, other organizations, and community members, particularly including citizen associations, historically under-represented groups, civic leaders, and local funders

  • Increased value of the network and partners (including shifts in power and influence of partners, level of involvement and resource/asset contribution or partners)

  • Increased trust (including reliability, openness to discussion/compromise, and support of the collaborative mission)

  • Increased collaboration (sharing information and resources, short-term collaboration on discrete projects, long-term partnership)

  • Museum/library and partner surveys—social network questions

  • Project team interviews

Changes in local communities

  1. To what degree and in what ways do CCI communities experience positive social change?

  1. Increased agency/empowerment among community members

  • Sense of agency among community members (e.g., sense of decision-making authority, sense of power to affect community change)

  • Case studies

  1. Increased social well-being among community members

[Community outcomes are identified by local projects and vary according to project goals]

  • Administrative data: local evaluation reports

  • Case studies

  1. Increased perceptions that museums and libraries are trusted and important allies in strengthening communities

  • Perceptions that local museum/library are trusted allies in community change efforts

  • Perceptions that local museum/library are important/effective allies in community change efforts

  • Case studies

  1. Increased local investment in community member-led community transformation

  • Perceptions of local investor interest (monetary and non-monetary/in-kind

  • Case studies

  1. How did outcomes vary across types of CCI supports/inputs used, cohorts, or characteristics of involved library/museums, partners, or communities?

Variation factors

  • Cohort: 1 or 2

  • Grantee type: Museum, library, or other

  • Uptake of CCI TA, tools, peer learning opportunity (CoP)

  • Number of partners: small, medium, large (based on analytic rubric)

  • Focus area of work (e.g., early learning, social service)

  • Leadership/institutional support for library/museum engagement in community-driven change: high, med, low (based on analytic rubric)

  • Previous history with and existing practices of collaborative community engagement

Administrative data: Grant reports/ applications


CCI Implementation

  1. How did grantee project teams vary in their use of CCI supports/inputs?

    1. Which activities mattered the most and to whom?

    2. What attributes of the model require fidelity and what could be adapted to local contexts?

  2. What are the benefits of using a cohort approach, including peer learning networks, in this work?


Table B.1 | CCI Implementation (EQ#6 and #7)

CCI Supports

Dimensions to Assess

Data sources

  1. How did project teams vary in their use of CCI supports?

    1. What activities mattered most to whom?

    2. What attributes of the model require fidelity and what could be adapted to local contexts?

Training

In-person grantee convenings/workshops

  • Presentations by TA providers

  • Presentations by other grantees

  • Large group discussions

  • Small group discussions

  • Networking opportunities

  • Hands on practice/demonstration activities (e.g., how to implement specific tools and practices)

  • Satisfaction with quality/utility component

  • Learning from component—in what specific ways did it build capacity?

  • Application of component—how did grantees apply in own project?

  • Relative instrumental value/benefit of component—how much did it contribute to their ability to implement their project effectively?

  • Administrative data: Grantee feedback on workshop sessions

  • Museum/library and partner surveys additional items for those receiving direct TA support

  • Project team interviews

Webinars

Coaching/ TA Support

Site consultants/coaches

  • Direct benefits of working with coach (learning from direction, learning from reflection, resources, connections, support)

  • Map contribution of coach efforts to capacity outcomes (to what degree did coaching contribute to. . .?)

  • Satisfaction with overall coaching, frequency/amount of time, quality

  • Museum/library and partner surveys additional items for those receiving direct TA support

  • Project team interviews

  • TA provider interviews

EPA TA providers

Tools and resources

  • Articles and white papers

  • Asset mapping

  • Power ladders

  • Journey maps

  • Theories of change

  • Which tools were used?

  • If used, for what purpose?

  • If used, how did the tool support their project work? (qualitative and direct connection to desired practices)

  • If not used? Why?

  • Relative utility and ease of use for each tool?

  • Relevance of each tool for local project?

  • Cultural appropriateness of each tool?

  • What other tools would be useful?

  • Administrative data: TA provider notes and work product artifacts

  • Museum/library and partner surveys additional items for those receiving direct TA support

  • Project team interviews

  • TA provider interviews

  1. What are the benefits of using a cohort approach, including peer learning networks, in this work?

Community of practice

(Also some components of in-person convenings and workshops)

Description of network

  • Changes in number of connections


Quality of network

  • Social value—connect to/benefit from other grantees?

  • Increased trust

  • Increased sharing resources and information

  • Increased collaboration

  • Feel part of bigger thing


Impact on Project Implementation

  • Learning from component—in what specific ways did it build capacity?

  • Application of component—how did grantees apply in own project?

  • Relative instrumental value/benefit of component—how much did it contribute to their ability to implement their project effectively?


Satisfaction

  • Satisfaction with quality/utility of component

  • Administrative data: Grantee feedback on workshop sessions


  • Museum/library and partner surveys additional items for those receiving direct TA support

  • Project team interviews





  • Administrative data: Cohort network mapping artifacts, TA notes

  • Project team interviews

CCI Contribution

  1. What conditions support or inhibit development of grantee capacity and changes in practices related to asset-focused, community-driven collaboration?

  2. To what degree does CCI contribute to changes in museums, libraries, and their local community partners?

Table B.1 | CCI Contribution to Outcomes (EQ #8 and EQ #9)

Evaluation Questions and Constructs

Indicators

Data sources

  1. What conditions support or inhibit development of museum, library, and grantee capacity and changes in practices related to asset-focused, community-driven collaboration? [Note: the constructs and indicators will be emergent, but we provide examples below]

Institutional characteristics

  • Leadership/institutional support for library/museum engagement in community-driven change: high, med, low (based on analytic rubric)

  • Resource allocation

  • Institutional readiness for asset-focused community-driven collaboration

  • Administrative data: TA provider notes, work product summarizing reflective action, grantee and TA provider reports, local theories of change

  • Project team interviews

  • Process tracing focus groups and interviews (during case studies)

Context factors

  • Community demographics

  • Community institutions

  • Funding environment (including business model)

  • Policy environment

  • Administrative data: TA provider notes, work product summarizing reflective action, grantee and TA provider reports, local theories of change

  • Project team interviews

  • Process tracing focus groups and interviews (during case studies)

Support and technical assistance

  • Exposure to TA

  • Utility of TA

  • Uptake of TA

  • Use of tools

  • Peer support


Collective action

  • Quality of local CCI implementation

  • Commitment of partners

  • Buy-in from community/ partners


  1. To what degree does CCI contribute to changes in museums, libraries, and their local community partners?

Strategies, outcomes, and alternative drivers from initiative and local theories of change (varies across grantee)

Emergent indicators through

  • Process tracing analysis

  • Thematic analysis

  • Qualitative comparative analysis

  • Local theories of change

  • Rubric scores (including data from surveys, interviews, focus groups, and administrative data)

  • Process tracing results

Internal and Field Learning Questions (not to be included in PRA)

Answers to the learning questions will emerge through analysis of the questions above, along with strategic learning debriefs with the IMLS program team and the Evaluation Steering Team.

Table B.1 | Museum and Library Sector (EQ #10-13)

  1. What cross-cutting lessons can be drawn about the role of capacity-building in helping build effective community engagement models that libraries and museums can use to work with diverse communities?

  1. How can CCI efforts inform grantmaking and field building strategies?

  1. Did the CCI Cohort Evaluation support changes in IMLS learning culture?

  1. What are the implications of CCI for scaling the model across agency investment areas?


File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified0000-00-00
File Created0000-00-00

© 2024 OMB.report | Privacy Policy