Community Catalyst Initiative Evaluation: Case Study Protocol
The purpose of the case studies in Stage 2 is to collect additional data for three purposes:
Collect evidence on local theories of change to test the contribution CCI-aligned project implementation practices to other changes in museums, libraries, and their local partners and communities (using process tracing methodology)
Elicit community perceptions and experiences with local project implementation
Collect additional data to explore emergent hypotheses that arise from Stage 1 data collection
Because the case studies will be highly dependent on findings that emerge from Stage 1 and local project contexts, we have created a case study protocol that lays out the general purpose of each data collection method, as well as the topics and high-level questions that will be addressed by each. To minimize burden, each data collection method proposed for the case studies will be tailored for specific case sites based on the knowledge we generated during Stage 1 review of administrative data, surveys, and interviews.
Our case study sample will be drawn from the 24 grantee projects in Cohort 1 and Cohort 2. Based on Stage 1 data analysis and consultation with the CCI Evaluation Steering Team, we will purposively select eight sites using the following criteria:
Local project implementation that is strongly aligned with CCI capacity-building efforts, as determined by scores on an asset-focused, community-driven collaboration rubric
Evidence of achieving community outcomes and/or making system changes in support of sustained community-driven collaboration (beyond the funded project)
Variation across grantee characteristics (e.g., Cohort 1 vs Cohort 2, museums vs libraries vs non-library grantees, size of project/community reach).
Once Stage 1 analysis is completed, we will rank each grant project with respect to their asset-focused, community-driven collaboration rubric scores and the degree to which they are achieving community and systems outcomes. From this ranking, we will select the top 12 for additional vetting. We will engage in consultation with IMLS and the Evaluation Steering Team to make the final determination of the eight case study sites to invite for participation in Stage 2, taking into account variation across project types and characteristics. The remaining four sites will serve as alternates if any of the eight declines to participate. Alternates will be chosen from the remaining four sites based on strength of implementation and outcomes as well as an effort to maximize variation across partner type.
We will collect additional qualitative data at each of the case sites using the following methods:
A 60-75-minute interview with the primary project lead(s) to verify their local theory of change (created as part of Stage 1 analysis) prior to the site visit (protocol attached for OMB review)
Focus groups with project teams and community partners (up to 15 people per group) to explore linkages and outcomes for process tracing (protocol attached for OMB review)
A focus group with up to 10 community members to understand community experience and perceptions of local project implementation, as well as perceptions of the involved museum(s)/library(ies) as a trusted community ally (protocol attached for OMB review)
Review of critical events and milestones around local project implementation using administrative data
Interviews with IMLS-contracted third-party TA providers who worked with the grantees (to be developed after Stage 1 analysis)
Follow-up conversations with project team members after the site visits to verify process tracing data (if needed, developed after preliminary Stage 2 analysis)
All data collection tools will be tailored to the individual cases based on what we already know from Stage 1 data collection and analysis. Four case studies will be conducted in the Fall of 2019 (Cohort 1) and four in Fall of 2020 (Cohort 2). Copies of interview and focus group protocols to be used in collecting these data follow.
We will communicate the following stipulations to all participants contributing to case study data collection:
Focus group participants will be asked to provide written informed consent after being apprised of the potential risks and benefits of the study. We anticipate any potential risks to be minimal.
We will not identify any individual responses who participate in focus groups to the grant team, IMLS or in reporting.
Data, including demographic characteristics will be reported only in aggregate by Grantee.
All focus group recruitment information will be kept in an encrypted secure folder and will be destroyed at the end of the evaluation.
We will not collect data from individuals under the age of 18. We will not collect data from anyone else classified as a member of a vulnerable population without IRB approval.
Prior to conducting on-site data collection, we plan to conduct interviews with the designated lead of each of the eight case study sites (8 total). This interview will focus on verifying the project’s theory of change (which we will develop based on Stage 1 analysis) and deepening our understanding of the different components of the change sequence. We may additionally conduct follow-up calls with the designated project after onsite data collection to verify or further validate evidence supporting process tracing hypotheses.
The protocol below includes high-level questions designed to elicit responses that will allow us to (1) refine each case study site’s observed theory of change, (2) surface drivers beyond CCI-aligned project implementation practice that might be associated with the changes identified, and (3) understand the nature of how CCI-aligned implementation practice is valued, adhered to, and contributing to desired changes. Specific follow-up probes will be tailored to sites based on emergent learning from Stage 1.
Note to interviewer: Tailor the interview protocol by selecting questions to expand, clarify, and build on the knowledge generated through the Stage 1 analysis. We do not want to collect redundant information—rather each step of the Stage 2 data collection process should build on what was previously learned in Stage 1. The pre-interview should be scheduled approximately one week prior to the site visit. The interview should last approximately 60-75 minutes.
We are very excited that [name of project] has agreed to participate in the case study part of the CCI cohort evaluation, and we appreciate you taking the time to help us prepare for the upcoming site visit. At the site visit we’ll be talking with a wide range of stakeholders about how [name of project] was implemented to create conditions for meaningful community change and this conversation will help us make the best use of everyone’s time.
Today we’ll focus on three areas. First, we want to confirm our understanding of how the project is using asset-focused, community-driven collaboration, particularly in interactions with community members. Second, we will walk through the theory of change showing how the project contributes to changes in community organizations, networks, and social well-being, which we created based on analysis of the earlier interviews and other documentation. Finally, will talk through any remaining logistics for the site visit.
We have provided three documents to help guide our conversation:
An implementation practices memo detailing our understanding of how the project is using asset-focused, community-driven collaboration
A theory of change visual and narrative describing how we understand change to happen in the project, including stated hypotheses about change happened
A description of the site visit structure, including the different meetings and stakeholders that we would like to convene
We appreciate you taking the time to share your valuable insights and knowledge with us.
Please do not hesitate to ask for more information if any of my questions are not clear.
Our call should take approximately 60-75 minutes. We would like to record the call for transcription purposes. Is this okay with you?
Any questions before we begin?
Notes to interviewer: In this section, we want to confirm our understanding of how the project is using asset-focused, community-driven collaboration. Review the implementation practices memo to tailor this section – confirm our understanding and ask questions that help expand on what was identified in the analysis (e.g., probe in areas where evidence was thin) and use language familiar to the those in the project.
Using the implementation practices memo created for the site, walk them through the description of each condition and ask the following questions:
To what degree does this description accurately reflect how you used asset-focused, community-driven collaboration in your project?
What additional examples or description should we add or change to help people understand how implementation looks in your project?
If you think about a continuum of implementation practice where 1 is developing/in planning; 2 is pilot or early implementation, and 3 is full or mature implementation, where would you place your project on this aspect of asset-focused, community-driven collaboration?
Note to interviewer: In this section we want to verify that the local theory of change we created resonates with the project team before using it in the focus group during the site visit.
In this set of questions, we’d like to walk through the theory of change we developed in the first stage of the study. We have analyzed your survey data, project-related documents, and interviews conducted with your team and with your site consultant to develop a theory of change about how change is occurring in your CCI-funded project. The theory of change is aligned to the initiative-level TOC and focuses on a few areas that, from our review, seem to be showing significant changes.
Note to interviewer: Refer them to the TOC (sent to them prior to the call). Walk through the theory of change, starting with the outcomes to the far right (community and local network changes) and working backwards through system, practice, and capacity changes, and finally CCI support. The potential four community and local network changes are:
Increased and deepened connections
Increased agency and empowerment of community members
Increased social well-being (as defined by local project goals)
Increased investment/ commitment to community transformation
Ask the following questions:
To what degree do you feel like this is sheet is a good representation of areas where the initiative has made significant/important progress?
Let’s unpack this theory of change to make sure we understand how [project name] might have led to specific changes in your community.
How meaningful are these changes (priority goal of the initiative, magnitude, on-track, etc)?
How are these changes measured?
In what ways should we refine how we have portrayed/described these changes?
Would project stakeholders (e.g., community partners, community members) understand the importance/significance of these changes?
Let’s unpack these changes in a little more depth and see if we can understand what other changes created conditions for the system changes and local network in your community to happen. Let’s start with these changes [refer them to the appropriate place on the TOC].
Organizational systems changes include:
Increased alignment of organizational values and expertise with co-designed, jointly implemented, asset-focused, community-driven collaboration
Increased organizational structures and processes supporting authentic engagement of community members
Practices changes include:
Increased interactions with community members and non-traditional partners outside of the museum or library (informal)
Increased engagement in discovery processes with community members (asset mapping, learning conversations) (formal)
Increased co-creation and joint implementation of a common agenda with community members and cross-sector partners, within CCI-funded project
Practice changes applied beyond CCI-funded project
We’ve identified several changes in practices and organizations that seem to precede these local community changes (refer them to the appropriate place on the TOC document) and to be conditions that helped produce the community change(s) (talk through each set of changes and how the connect to local community changes and to each other, including a brief description of each).
To what degree do these changes seem to be the right ones that led to local community changes?
What if any important changes seem to be missing?
How would you re-order these changes to show what happened?
In what ways should we refine how we have portrayed/described these changes?
Would initiative stakeholders understand the importance/significance of these changes?
Note to interviewer: In these questions we are looking to identify the links between CCI-aligned project implementation and changes above
Let’s transition now to thinking about how these changes came about. While it can be difficult to tease out if it was your project implementation efforts that produced these changes, we’d like to try and unpack if and how asset-focused, community-driven practices are changing the landscape and systems to enable and sustain community change. We realize you may not feel comfortable saying that your practices “caused” something to change, but we would like to explore with you in what ways they intended to change things, how much they changed, and whether other factors contributed to the change as well.
How well does this list of strategies represent the work conducted to achieve the changes we just talked about?
Are there key strategies missing that we should add?
In what ways should we refine our description of the strategies?
How did/does the asset-focused, community-driven collaboration practices support the implementation of these strategies?
For each asset-focused, community-driven collaboration practice, ask them to rate them as 1, 2, or 3 in importance, with 1 – not important, 2 – somewhat important, and 3 – very important, and then ask the following questions.
To what degree is our description of the practice accurate and complete?
What specific aspects of the practice were most important to implementing the strategy?
[In these questions we are looking for what else contributed to the change – outside of asset-focused, community-driven collaboration practice and other project strategies. This is a key set of questions.]
How likely is it that this change would have happened if the project had not been in place? How strong of an influence do you think the project played?
What else was going on locally – for example, other initiatives or other policies – that could have contributed to these local community changes? (changes in policy environment, funding etc.)
Is there anything else you’d like to tell me about how change happened in this area?
Now that we have gone through the theory of change, I’d like to ask you to look at the hypotheses under the graphic about how change occurred.
Do these hypotheses seem plausible?
How might you refine these hypotheses to align them more closely to the work of the project and the changes you are seeing?
Thank you very much for helping me refine and verify the theory of change. Your input has been very helpful in clarifying our understanding of how your CCI-funded project has worked to create conditions that lead to positive outcomes for your community.
In the remaining few minutes that we have together, I’d like to work through some logistics for our upcoming sites visit.
Confirm the different meetings and stakeholders who will be needed at each
Describe the purpose of each meeting
Make clear who would be helpful to talk with (e.g., someone who can speak to equity)
Share sample agenda for site visit
Set timelines for recruiting and confirming participants
Discuss any logistics (including meeting space,,etc.)
Touch base on MOU and any questions remaining
Thank you so much for your time, energy, and willingness to go deep into the details of your initiative. Your contribution means a lot to not only us but also the field of museums and libraries engaging in community transformation efforts.
The purpose of the process tracing dialogue focus group is to collect participatory data about the observed links between asset-focused, community-driven collaboration practice and desired outcomes. The data from these focus groups will be used in conjunction with other data to evaluate the strength of evidence that links between strategies and outcomes in the initiative’s theory of change are necessary and sufficient as explanation for CCI impact on museums, libraries, and local communities.
Two focus group facilitators are needed for this data collection. One facilitates the dialogue and one records the discussion on chart paper for the participants to see. The focus group will also be audio recorded.
We will be selecting a group of project team members and key partners to engage in a “process tracing dialogue”— a structured focus group aimed at assessing the local theory of change for how CCI capacity-building led to changes in practice, the local network, and community members, as well as identifying alternative explanations for change. We anticipate that each focus group will include up to 15 participants (8 x 15 = 120 total participants). Criteria for inclusion in the focus groups includes significant involvement in the local CCI-funded project, and representation across a range of roles, including project leaders, institutional leaders, staff involved in direct community engagement efforts, and community leaders closely involved in the project.
Introduce the purpose/frame of the meeting
Lead introductions of all participants including evaluation team
Define key terms – talk about what we mean by systems change, about how they’ll see it constructed
Share the value back to them: What their site will get from this process?
Walk them at a high level through the storyline (the local theory of change). Let them know where it came from and why it’s important, and explain that we’ll be unpacking it and challenging it.
Visually put up the pieces of the theory of change on the wall as you’re talking them through it (all pre-populated on large color-coded sticky notes):
White: Local community changes
Yellow: Organizational and systems changes
Orange: Practice changes
Pink: Strategies
Blue: CCI support
Red: Other drivers
Ask people to react to all of it – what resonates, what might you state differently?
When something resonates – acknowledge it, point to where it is on the visual, but don’t change anything
When something doesn’t resonate, check in with the room – is there a sense of agreement that the change is needed?
If yes: Make the change using the appropriately colored sticky note (note-taker: note this shift in the causal relationship has happened)
If no: Unpack it, try to understand if you’re dealing with a difference of definition/description or a true difference in belief about what drove the change. If the former, try to clarify to get everyone on the same page. If the latter, get people to process whether this is about magnitude of contribution OR about whether it was a contribution at all (are we rejecting part of the causal chain or are we suggesting it isn’t as significant because of some other driver)
Depending on response, you may be adding a new driver or reframing the one already up.
Continue to refine until there is a general calming of energy in the room – people are feeling good about the flow on the wall. Hopefully some things have been added, moved, and removed.
Test the strength of the relationships through a series of questions for each part of the theory represented by the different sticky notes (except for the alternative drivers)
Looking at what is on the wall, are there any changes/strategies that could be removed and we would likely have seen the community level change anyway?
If there is strong agreement in the room, pull the change/strategy out of the visual and move it lower on the wall – still there, but not in the causal flow anymore.
If there are mixed feelings, leave it where it is and move on.
Repeat for each set of sticky notes
Explore the alternative drivers
Let’s look only at the red stickies for a moment. Let’s imagine that even though all of this good work happened, none of it was the reason that the community changes occurred. What else could possibly explain this result happening?
As other possible drivers are named, add them to the visual using red sticky notes, pausing to see if there is a shared sense of each one being important.
For each red sticky up, both ones already on the wall and new ones added, ask:
Was this something that caused the community changes or mediated them – e.g. sped them up or slowed them down but didn’t cause them directly.
Introduce the idea that now that we have a story with many different drivers, we want to understand the “weight” of the influence of the key actions they took compared to other drivers.
Let them know they have 20 “weights” to distribute across the various drivers. The goal should be to place the greatest weight on the things that were necessary for the local community changes to happen. Have individuals distribute their weights on the TOC elements on the wall.
Discuss the pattern of weights and ask them to share why they distributed their dots.
Have them work as a group to collectively distribute 20 weights on the heaviest contributors
After all the weights are placed, ask for a moment of reflection to check and make sure they are in the right places (thumbs up, down sideways). Debrief if thumbs down/sideways
Divide the group into two smaller groups
Have one group develop a narrative making the case that their efforts contributed to community changes focusing on the previous discussions and weighting exercise
Have one group develop a narrative making the case that something else other than their efforts contributed to community changes focusing on the previous discussions and weighting exercise.
Have the groups share their narratives with each other and debrief.
Thank them for the dialogue
Remind them of the products they will get back
Encourage them to follow-up with the key site contact (their end and ours) with follow-up questions
The primary purposes of this focus group are to (1) triangulate the reports of project team members and community partners and third-party TA providers with reports of community members about the degree to which participating museums and libraries are shifting to assets-focused, community driven collaboration, and (2) assess community member perceptions about the library/museum as a trusted community ally. The intent of the focus groups is not to elicit results that are generalized across all community members in all projects. Rather, the purpose is to collect information from community members who have reasonable knowledge and exposure to how the library/museum works in partnership with the community and can speak to aspects of their approach that build on community priorities, assets, and solutions, and shift power toward community members.
Two focus group facilitators are needed for this data collection. One facilitates the dialogue and one records the discussion on chart paper for the participants to see. The focus group will also be audio recorded.
We will recruit individual community members for a structured focus group during the case study onsite data collection. We anticipate that each focus group will include up to 10 participants (8 sites x 10 participants per site = 80 participants). Participants will be identified in consultation with the grantee project team and their third-party TA consultant. Community members will be considered for inclusion in the focus group if they helped develop the project plan, they helped implement the project plan, or they participated in a project-generated activity. We will work with site consultants and project teams to intentionally identify participants who have a range of reactions to the CCI-funded project and the library/museum—from positive to negative to mitigate positive response bias. All participants in the community focus groups must also be age 18 or older so that we are not collecting sensitive data from minors.
Prior to beginning the focus group, a facilitator will make sure participants understand the benefits and risks of participating, by reading aloud the consent form (on the final page of this protocol) and addressing any questions they may have prior to signing the consent form.
What have you done as a part of [project name]? What did you think about the experience?
How was it similar or to or different from experiences you may have had with other efforts to make changes in your community?
How have other community members been involved in the project?
Are resources provided to support participation of community members?
Historical Experiences with Involved Museums/Libraries
Did you have anything to do with [involved museums/libraries] before [project]? What was that like?
Have you heard about other efforts by [involved museums/libraries] to help community members get involved in what they are doing?
Use chart sheets and dots to do group polling and rating on the following items with respect to how true they think the statement represents their museum, library or both. These polls will provide inputs for the focus group dialogue.
[Name of institution] is an important resource in the community
[Name of institution] is a safe, open, and trusted organization
[Name of institution] engages the community in developing programming or other efforts in a way that feels truly welcoming of community members’ ideas
[Name of institution] follows where community members lead in addressing important local issues
[Name of institution] helps community members gather together and build relationships
[Name of institution] does not understand what community members have to offer or what we really want to happen
[Name of institution] makes decisions about what to do on its own
Once participants have had an opportunity to rate each of the items, the facilitator processes the responses for each item using the following prompt:
It looks like most of the responses are [positive, negative, mixed]. Would people like to share why they rated the statement as they did? Probe for specific examples.
How could [Name of Institution] do a better job of supporting community members to work toward making the changes in your community that you think are most important?
Are there any people or organizations that were not involved in this effort who should be involved in future efforts like it? Why?
Agreement and Consent
For Participation in the Community Catalyst Initiative Evaluation
This form asks for your participation a focus group that is part of the evaluation of the Community Catalyst Initiative (CCI). CCI is a grant program of the Institute for Museum and Library Services (IMLS), which provided funding and support for [project name].
The purpose of the evaluation is to see how well CCI is working for communities and if there are ways the program can be better. ORS Impact, an independent consulting firm, is leading the evaluation of CCI and running the focus groups.
If you agree to take part in the evaluation, you will be asked to do the following within this focus group:
Answer questions about your experience of the project and your views on the museums and libraries that worked on the project, and participate in discussion of these with other community members
Allow the focus group discussion to be audiotaped
Allow the facilitators to record information describing the make-up of the focus group participants, including your gender, race/ethnicity, and the area in which you live
The information listed above will help IMLS and project partners understand if and how CCI supported libraries, museums, other organizations, and community members to make desired changes in their communities, and what may have made this easier or harder. Reports about the evaluation will not have your name or any information that would make it possible to identify you. Audio recordings and notes will be stored securely and only ORS Impact staff members will be able see it, unless it is required by law, based on a specific request.
Being part of the evaluation is up to you. Your decision about whether or not to take part will not change your relationship with [project name] or anyone else involved in it.
If you have any questions about being part of the CCI evaluation, you can ask them now or contact the ORS Impact facilitators (Terri Akey and Jenn Beyers) at: ORS Impact, 1100 Olive Way, Suite 1350. Seattle WA 98101, T: (206) 728-0474 x247 (Terri) or x231 (Jenn).
I have read the information above. I asked questions and received answers.
I AGREE to take part in the evaluation and my signature is below.
I
DO NOT AGREE
to take part in the evaluation and my signature is
below.
Participant
Signature: ____________________________________________ Date:
_________
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |