Final Study Plan

2M Research (2M) and our partner, Metris Arts Consulting (collectively “the 2M team”), are contracted with the National Endowment for the Arts (NEA) to perform a study of the Our Town creative placemaking grant program. The contract (No. C16-51) consists of two phases. In Phase I, which ended in August 2017, the 2M team worked closely with NEA to develop

a Theory of Change (TOC) that describes how and why the projects Our Town funds produce economic, social, physical, and systems change in local communities as well as how each of these changes contributes to broader systems-level changes in community development throughout the United States;
a Logic Model (LM) that details the specific inputs, activities, outputs, and outcomes of Our Town projects as well as the community contexts in which they occur; and
a Measurement Model (MM) that specifies indicators and potential data sources for each concept in the LM.

The purpose of Phase II, which began in December 2017, is to implement a study to validate the TOC, LM, and MM we developed in Phase I as well as to understand and recommend any adjustments to the models, especially as they relate to the systems change that Our Town projects foster, based on the results of the study.

In this document, we outline our plan to implement a mixed-methods study that includes a web survey of all current and past Our Town grantees and case studies of three grantees that have implemented or are implementing promising practices that foster systems change or have sustained support and recognition of arts, design, and cultural strategies as integral to community planning and development. The results of both the case studies and the web survey will help the 2M team validate the TOC, LM, and MM from Phase I of the contract and recommend adjustments to improve the models.

Study Research Questions

To address the Phase II objectives, the 2M team identified four research questions. We detail the research questions and their alignment with the Phase II objectives in Table 1. We also detail in Table 1 how the 2M team will answer each research question with evidence from the case studies and web surveys.

Table 1. Alignment of Phase II Objectives, Research Questions, and Case Study and Web Survey Evidence

Phase II Objectives	Study Research Questions	Case Study Evidence	Web Survey Evidence
OBJ 1. Implement a study to validate the TOC, LM, and MM we developed in Phase I	RQ1. How well do the TOC, LM, and MM align with Our Town grant projects?	Project-level LMs for each site that are shared with the grantees after the site visits that demonstrate the application of LM tools	Representative quantitative data that show the frequency and percentage of grantees that align with TOC/LM categories and MM indicators and the frequency and percentage that do not align
OBJ 2. Understand and recommend any adjustments to the models, especially related to the systems change that Our Town projects foster, based on the results of the study	RQ2. How are the Our Town project community contexts, inputs, and activities associated with the proposed outcomes? Are there certain contexts, inputs, activities, or outcomes that, when compared across grantees, are proposed to more readily lead to systems change?	Rich qualitative data on the common factors, including project inputs, activities, and outcomes, that are associated with systems change	Results of statistical tests of how project contexts, inputs, activities, local community change outcomes, and systems change outcomes are associated with one another
	RQ3. What are the outcomes (positive and negative) of various types of Our Town projects not anticipated in the TOC, LM, or MM?	Rich qualitative data on the types of systems change that projects foster	Representative qualitative data that can identify outcomes, especially, but not limited to, systems change, and other areas that are not represented in the TOC, LM, or MM
	RQ4. What adjustments, if any, are recommended to the TOC, LM, and MM based on study findings?	Synthesis of all evidence

The first research question— “How well do the TOC, LM, and MM align with Our Town grant projects?”—ties directly to the first objective of the study, which is to validate the TOC, LM, and MM. The 2M team will use the evidence from case studies and a web survey of Our Town grantees to understand how well elements of grantees’ projects align with the Phase I models. If there is a high degree of alignment, NEA can demonstrate the validity of the TOC, LM, and MM because they realistically represent what happens with projects “on the ground” in their target communities. We expect to find evidence of alignment, but the study results will also inform how we enrich and improve the Phase I models.

The second research question—"How are the Our Town project community contexts, inputs, and activities associated with the proposed outcomes? Are there certain contexts, inputs, activities, or outcomes that, when compared across grantees, are proposed to more readily lead to systems change?”—ensures that the 2M team will identify relationships between project community contexts, inputs, activities, and outcomes (especially key factors associated with systems change outcomes) that can show us how to enrich and improve the TOC, LM, and MM. Currently, the Phase I models provide a menu or comprehensive list of inputs and activities grantees can use, the outcomes to which projects contribute, and the community contexts in which projects occur; however, they are less specific on the connections between each element. The qualitative case study evidence and quantitative evidence from the statistical tests of the survey data can identify these connections and illustrate how each model can better represent them. Moreover, while the 2M team and NEA worked to define specific types of systems change that Our Town projects foster, this domain needs additional refinement in the Phase I models.

The third research question—“What are the outcomes (positive and negative) of various types of Our Town projects not anticipated in the TOC, LM, or MM?”—ensures that the 2M team can provide evidence on unanticipated types of systems change Our Town grantees pursue and produce and how the TOC, LM, and MM can better reflect them. In addition, the question ensures that the other outcome areas in the models, which include economic, physical, and social change, accurately reflect the types of change that Our Town projects realize. Qualitative evidence from both the case studies and the web survey (grantee responses to “other – please specify” categories and open-ended questions) will address this question.

Finally, the fourth research question—"What adjustments, if any, are recommended to the TOC, LM, and MM based on study findings?”—reflects the 2M team’s intention to adjust and improve the Phase I models through a synthesis of the evidence from the case studies and web survey. An important piece of the Final Study Report (we describe this deliverable in the Reporting Plan section later in this draft report) will be the 2M team’s recommended adjustments to the Phase I TOC, LM, and MM.

Mixed-Methods Study Design

To collect evidence to validate and improve the Phase I models, the 2M team developed a mixed-methods study design. NEA requires both quantitative and qualitative (mixed-methods) evidence to better understand the complete picture of its Our Town grant projects. For the qualitative component of the study, the 2M team will work to understand the context within which local communities are implementing Our Town projects and the components that seem to be associated with systems-level changes. For the quantitative component, the 2M team will collect more specific details from all past and current Our Town grantees to more broadly understand and validate or improve the Phase I TOC, LM, and MM. Ultimately, the 2M team will strive to synthesize findings from all the data-collection efforts to inform further development of the Our Town TOC, LM, and MM. We provide an overview of the study design and the usefulness of the mixed-methods approach in Figure 1.

Figure 1. Overview of the Study Design

We organize the remainder of this report as follows. First, we describe in greater detail the case study and web survey data-collection methods and analytical approaches. Next, we provide a detailed timeline for study completion. We end with our approach to the protection of human subjects involved in the study, our communications plan, and a reporting plan in which we describe how the 2M team will report the results of the analysis to NEA.

Grantee Case Studies

To gain a deeper understanding of site contexts and Our Town grantees, the 2M team will conduct three in-person site visits, with two members of the 2M team for each case study. The site visits will include observations, participation, and interviews to collect comprehensive qualitative data about the site context, activities, and outcomes. The purpose of the case studies is to:

glean rich qualitative data on the common factors, including project inputs, activities, and outcomes, that are associated with systems change;
collect information about systems change outcomes that helps refine and improve the systems change categories in the project-level LM developed in Phase I of the Our Town contract;
develop project-level LMs for each site that are shared with the grantees after the site visits that demonstrate the application of LM tools; and
enliven the Final Study Report with rich information (e.g., quotes, photographs) from each site.

Case Study Selection Process

The 2M team will conduct the case studies with three grantees: City of Boston, Thunder Valley Community Development Corporation (CDC), and Forklift Danceworks. We selected each grantee through a rigorous and collaborative process involving both NEA and the 2M team. First, the 2M team developed the following sampling criteria to help NEA identify potential grantees for the case studies:

A systems change focus
At least one rural site
Projects with sufficient time for systems change to occur and be evident

NEA used each criterion to nominate five grantees. In addition, NEA was cognizant of grantees that have already been studied or may otherwise be overexposed and tried to avoid them for the case study sample. Through reconnaissance calls conducted with each grantee nominated, the 2M team investigated the extent to which systems change has occurred and queried whether the project involved numerous stakeholders and project partners of different types to ensure an adequate interviewee pool.

Based on the reconnaissance call findings, the 2M team evaluated each grantee in terms of systems change focus (low to high), indicating the extent to which systems change has occurred and the availability of evidence of that change. In addition, based on the reported partners and partner organizations the grantee identified as potential interviewees, we scored each grantee (low to high) to indicate the availability and diversity of grantee partners. We present the data we collected during each grantee reconnaissance call, other factors we considered in sample diversity, and our ranked recommendations for case study selection in Table 2.

Table 2. Findings from Reconnaissance Call and Case Study Selection Recommendations

Rank	Grantee Name	Funding Years	Region	Location	Systems Change Score	Number/ Diversity of Partners Identified	Other Considerations
1	City of Boston	9/2015– 9/2016	Northeast	Boston, MA	High	High	Activities are similar to Metro Arts and People’s Emergency Center; potentially overexposed
2	Thunder Valley CDC	10/2015– 9/2017	Midwest	Porcupine, SD	Medium	Medium	Only rural site; Native American focus
3	Metro Arts	10/2015– 9/2017	Southeast	Nashville, TN	Medium	High	Activities are similar to City of Boston; equity focus
4	People’s Emergency Center (PEC)	4/2014– 3/2015	Northeast	Philadelphia, PA	Medium	Medium	Activities are similar to City of Boston; wide breadth of program activities
5	Forklift Danceworks	5/2017– 5/2019	Southwest	Austin, TX	Low	High	Current grantee: only emerging evidence of systems-change Greater focus on civic engagement than systems change

The 2M team ranked the City of Boston, Thunder Valley CDC, and Metro Arts, respectively, as the recommended grantees to participate in case studies. The recommendations were based on systems change score, number and diversity of partners identified, grantee funding years (e.g., time since completion versus ongoing project), and geographic diversity in addition to other considerations, including diversity of project activities, urbanicity (e.g., rural versus urban), grantees that may be overexposed, and population or program focus. For example, because of proximity and region, the 2M team advised against including both City of Boston and PEC.

The 2M team discussed the recommendations with NEA and, due to a concern of strong similarity between the City of Boston and Metro Arts projects, decided to proceed with Forklift Danceworks instead of Metro Arts as the third case study in addition to City of Boston and Thunder Valley CDC. The Forklift Danceworks project is an ongoing project and thus will have less evidence of systems change than the other case studies; however, the 2M team will be able to identify factors that may indicate a strong likelihood of systems change. Moreover, as an ongoing project, the 2M team will have the opportunity to perform participant observation of the project’s activities and investigate the implementation of the project as it occurs on the ground.

Case Study Data Collection

Site Visit Planning

The 2M team began site visit planning during the reconnaissance calls we conducted to identify the three case studies. We used the calls to gather information about systems change and query grantees about the number and diversity of partners and stakeholders involved to ensure an adequate pool of interviewees and to discuss dates for participation or observation, such as events and exhibits. The 2M team will compile notes from each call with existing background (e.g., publicly available reports, videos) and administrative data (e.g., the final descriptive report and grant application) to enrich the 2M team’s understanding of the site context, activities, and outcomes.

At least 3 weeks prior to the scheduled site visit, the 2M team will conduct a site visit planning call with each grantee. Prior to the call, the 2M team will identify any existing gaps in knowledge and probe the grantee for additional information as needed. The 2M team will also use the site visit planning call to collect contact information for potential interviewees, their potential contributions and/or perspective, and to develop a preliminary site visit agenda. The 2M team will aim to collect contact information for at least one individual per respondent type. To inform the selection of interviewees, the 2M team will employ Metris’ equity reflection questions—for instance, “Who is in my sample, and what do I need to know about them?” and “What is their vested interest in the results of this research/study?”¹ The 2M team will use grantee information gathered through the reconnaissance call, site visit planning call, and background document review to develop a project-level LM for each grantee. The LM will reflect project-level context, inputs, activities, and outputs to illustrate the ways in which they aggregate and work to influence systems change.

Instrument Development

In collaboration with NEA, the 2M team will develop discussion guides. The rationale for using a discussion guide as opposed to semi-structured interview guides that require asking respondents the same questions in the same way is that they allow flexibility so that we can tailor questions to the context and content of the intervention under study.² As an example, the 2M team plans to ask about partnerships in implementing Our Town initiatives, but because partners will vary across sites and in their type and level of involvement in the project, the lead interviewer will need to tailor questions about partnerships specific to each grantee. Discussion topics and questions also vary according to respondent type.

To develop the discussion guide, the 2M team will again employ Metris’ equity reflection questions, such as “Have I vetted the instrument as appropriate to the participants’ culture?” and “Does my introductory language make respondents feel comfortable, informed, respected?” We will adapt the discussion guides to fit each individual grantee’s specific initiatives. Topics to be addressed across sites include

contextual factors that influenced implementation of the Our Town initiatives,
descriptive information about partner involvement,
successful and unsuccessful activities conducted for the intervention,
challenges to implementing initiatives targeting systems change, and
lessons learned throughout implementation.

For each site, the 2M team will tailor the discussion guides based on the respondent type, relevant arts tactics and strategies, and other context-specific factors to fully and accurately capture the relevant data. Table 3 presents the types of respondent categories for interviewees and examples of their roles in the program or community. The purpose of conducting interviews with multiple respondent types is to capture varied perspectives to identify consensus and differences.

Table 3. Respondent Types and Examples of Site Visit Interviewees

Respondent Type	Example Role
Program partner and sector	Staff of partner organizations by partner sector (community development corporation, local arts agency, library, etc.)
Grantee key staff	Project director (PD), authorizing official, project staff
Artists/designers	Community or project artists
Community-based stakeholder	Municipal representative, neighborhood resident, program participant, local business owner, project volunteer

Conducting Grantee Site Visit

When arrangements have been made with the grantees and interviewees, 2M team members will plan to visit each site for up to 2 days to meet with the respondents who agree to be interviewed. The 2M team will be responsible for scheduling and confirming the interviews and securing a location for the meetings. To minimize the burden of data collection on interviewees, the 2M team will meet interviewees at a convenient location of their choosing.

During the site visit, the 2M team will conduct up to seven 1-hour interviews with project stakeholders, including the PD, staff, community partners, and program participants. The 2M team will audio-record and transcribe each interview. At the start of each interview, we will obtain verbal consent to record the discussion, explaining that the 2M team will use the recording only to ensure an accurate representation of the information shared. We will share de-identified transcripts with NEA and delete all recordings upon completion of the study. For the Forklift Danceworks grantee only, per NEA, we will share raw interview transcripts with GO Collaborative Austin, an NEA-funded project evaluator. The purpose of data-sharing is to reduce the participant burden by reducing two topically similar interviews to only one. For all case study reports, interviewee names will not appear in any reports, though respondent types, including partner agency sectors (i.e., community development corporation, local arts agency, library), may be listed as relevant. 2M will provide a digital copy of the case study to respondents, upon approval by NEA, in appreciation for their participation, which will later be publicly available.

If possible, we will schedule the site visit around grantee activities such as festivals, events, or exhibits so that the 2M team can observe program activities and collect more nuanced data about arts tactics and implementation. The 2M team will methodically collect participant observations and field notes so that we can incorporate contextual information about program context, activities, and outcomes as well as the relationships between each. As available, the 2M team will capture and collect photographs from leading and partnering agencies as evidence of activities.

Post-Site Visit Activities

Project-Level Logic Model Development

Based on the site visits, the 2M team will revise the preliminary project-level LMs to accurately reflect the community context and grantee goals, inputs, activities, outputs, and categories of outcomes including social, economic, and physical outcomes as relevant. The 2M team will then schedule interactive sessions with the grantee PD and up to two additional stakeholders to review the LMs for accuracy. The 2M team will incorporate any feedback obtained during the sessions to finalize the LMs. We will provide the finalized project-level LMs to the grantee PD.

Case Study Analytical Approach

As we complete the site visits and transcribe the data, the 2M team will import the qualitative and visual data collected during each of the site visits into the NVivo data analysis software for analysis. We will analyze the data to investigate the case study objectives and research questions, as outlined in Table 4.

Table 4. Case Study Objectives and Research Questions

Case Study Objectives	Case Study Research Questions	Analytical Approach
Glean rich qualitative data on the common factors, including project goals, inputs, activities, and outputs, that are associated with systems change	Which factors are associated with systems change (project goals, inputs, activities, and outputs)? What trends surface across projects?	Identify and define factors (i.e., inputs, activities, outcomes) linked to systems change, drawing out common trends
Collect information about systems change outcomes that helps refine and improve the systems change categories in the program-level LM developed in Phase I of the Our Town contract	How do case study stakeholders conceptualize and articulate systems change-related outcomes? How do these outcomes align with the categories proposed in the program-level LM?	Collect new constructs for systems change outcomes and refine those presented in the Phase I work
Develop project-level LMs for each site that 2M shares with the grantees after the site visits to demonstrate the application of LM tools		Populate project-level LMs by using background materials and site visit findings. Convene interactive sessions with grantees to vet and revise project-level LMs after site visits
Enliven the Final Study Report with rich information (e.g., quotes, photographs) from each site		Develop case studies with engaging narratives and compelling images while addressing the overarching research questions

By using a deductive approach, the 2M team will develop a codebook, or dictionary of concepts, for use in organizing the data for analysis. The codes will arrange ideas logically to map to research questions and components of the LM to ensure that the qualitative data address the case study objectives and research questions. We will code the qualitative data for content and the visual data, as available, for content and/or context to increase the breadth of the data.

The 2M team will analyze the data to identify additional emergent themes and generate additional codes, as needed. Using a combination of deductive and inductive coding, we will organize data to represent existing categories in the Phase I models as well as lend flexibility to identify and define additional categories within the components, as needed. Qualitative analysis will enable the 2M team to capture the grantee’s and stakeholder’s conceptualization and articulation of the LM components. Table 5 provides example coded text and a corresponding theme as well as the corresponding research question addressed.

Table 5. Example Theme by Research Question and Coded Text

Case Study Research Question	Subtheme	Theme	Code	Example Text¹
How do case study stakeholders conceptualize and articulate systems change-related outcomes?	Artists hire other artists (local, people of color)	Expand reach	Systems Change Outcome	“We noticed that if you give funding to an artist, artists fund other local artists. So we started to notice this cascade effect of paying one and then three to four more involved in the neighborhood, often of color or underrepresented.”
Which factors are associated with systems change (project goals, inputs, activities, and outputs)?	Communities engaged in dialogue; Artists engaged in dialogue	Civic engagement	Social Change Outcome	“The community is catalyzed and wants to talk more about it and help support. People are ready to have conversation after the performance. Artists want to know how to keep dialogue going—this project is the way to keep that going.”
	Presence of arts intermediary to provide artist technical assistance	Social and human capital	Community Context	“The Intermediary is not well studied, in this case that’s us. Normally, arts orgs spend all of our time making sure that there is technical skills training for artists. Not every community has a community arts org that could play the connective tissue like we do.”
¹Example text from reconnaissance call with Our Town grantees

Using queries and cross-tabulations, the 2M team will analyze data to identify relationships between and interactions of the components of the LM at each site to identify factors associated with systems change. The 2M team will code each grantee as a stand-alone case; upon completion of all case studies, we will analyze the data to develop a cross-case study. We will mirror the coding and analytical approach across each grantee so that we can easily analyze the data across case studies. The 2M team will synthesize the queried data to report the qualitative findings of the interviews, observations, and visual data based on the emergent themes in the cross-case study.

Web Survey

In addition to the nuanced and context-specific information that the case studies provide, NEA requires general information about Our Town grantees that can validate the Phase I TOC, LM, and MM as well as identify ways to improve the models so that they reflect how Our Town projects happen in local communities (see Figure 1). The 2M team will implement a web survey of all current and past Our Town grantees (approximately 466) for which NEA has up-to-date contact information to meet this need. The purpose of the web survey is to

collect descriptive data that shows the proportion of grantees with projects that do and do not align with the Phase I LM categories and MM indicators;
test how project community contexts, inputs, and activities are associated with the respondent’s perceived local community change outcomes and assess whether each LM element is associated with the perceived systems change outcomes; and
collect representative qualitative data that can identify outcomes, especially systems change, and other areas that are not represented in the Phase I models and develop recommendations for adjustments.

Web Survey Development

We organized the web survey into domains that correspond to the elements in the Phase I LM. Table 6 shows the number of close-ended (n = 27), “other-specify” (n = 16), and open-ended (n = 2) questions that address the concepts in each survey domain.

Table 6. Overview of the Survey Instrument Domains

Domain	Description of Concepts Captured
Descriptive Information	The name and zip code of the awardee organization, the NEA grant ID, and the award year
Project Activities and Strategies	Five close-ended questions that capture whether grantees use the creative placemaking strategies and arts engagement, cultural planning, design, and artist and creative industry support arts tactics in the Phase I models in their Our Town projects; one “other-specify” question that allows grantees to voice tactics they use that are not represented in the Phase I models
Project Inputs	Three close-ended questions to capture how leadership and partners are involved in Our Town projects; two “other-specify” questions that allow grantees to specify a partner type and a partner role not defined in the survey; one close-ended question asking grantees to estimate the number of partners involved in their project
Community Context	Three close-ended and two “other-specify” questions to capture the types of communities in which Our Town projects occur, including information about the geography, urbanicity, and dynamic of change that describe Our Town communities
Systems Change	Two open-ended questions to capture, from the perspective of the grantees, how Our Town projects foster sustained support and recognition of arts, design, and cultural strategies as integral to community planning and development; two close-ended questions that ask about the laws, policies, and regulations that enable Our Town project activities and that improve as a result of Our Town project activities improve; two “other-specify” questions to allow grantees to voice other laws, policies, and regulations not included in the survey
Local Community Change	Thirteen close-ended questions that capture whether grantees perceive the economic, physical, and social outcomes and indicators represented in the Phase I models in their Our Town projects as well as to capture change in the outcomes since the projects began; nine “other-specify” questions to allow grantees to voice perceived outcomes and indicators that are not represented in the Phase I models

Cognitive Testing

Per the Paperwork Reduction Act (PRA), before finalizing the web survey instrument, it must first pass through clearance with the Office of Management and Budget (OMB) to ensure minimal burden to Our Town grantees and a maximum level of quality and rigor for the survey. From May 16, 2018 to August 3, 2018, the 2M team performed cognitive testing of the Our Town Implementation Study web survey instrument with seven Our Town program grantees. The purpose of the cognitive testing was to detect issues with the usefulness, clarity, and readability of the items in the web survey as well as to estimate the average time to complete the survey. After performing the cognitive testing, 2M met with NEA to discuss recommendations to improve the web survey instrument.

To identify cognitive testing respondents, NEA provided 2M with a list of grantees that met the following criteria:

At least one current grantee
At least one past grantee
At least one tribal grantee
At least one rural grantee
At least one urban grantee
At least one grantee per arts tactic category – Arts Engagement, Cultural Planning, Design, and Artist/Creative Industry Support
A diversity of grantee partnerships, i.e., led by a government agency, led by an arts/cultural nonprofit community organization

Table 7. Cognitive Testing Respondents

Respondent	Type	Survey Test	Interview
1	Past, urban, nonprofit, artist/creative industry support	Complete	Refused
2	Current, rural, tribal, cultural planning	Complete	Email
3	Current, rural, nonprofit, design	Complete	Refused
4	Past, urban, nonprofit, engagement	Complete	Phone
5	Past, urban, municipal government, artist/creative industry support	Complete	Phone
6	Past, urban, nonprofit, cultural planning	Complete	Phone
7	Current, rural, nonprofit, arts engagement	Complete	Phone

2M programmed the draft web survey by using the survey software Confirmit. We sent the selected cognitive testing respondents a link to take the survey in Confirmit in a pre-test mode, which allows respondents to take notes for each question related to level of effort, comprehension, clarity, usefulness, and other thoughts. In addition to respondent notes, 2M assessed the amount of time respondents spent on each question to identify questions that may have issues. After the respondent submitted the survey online, 2M received a notification and reviewed the respondent notes and other information. A 2M interviewer then reached out to the respondent by telephone and/or email to schedule a 30-minute cognitive interview.

2M conducted the interview by telephone (one respondent requested to answer questions via email) to review the respondents’ notes and discuss any issues that arose with specific questions during the survey. The interview also included general questions about the timing of the survey and the clarity of the survey questions. 2M went through a paper version of the survey with respondents and asked them to elaborate on specific comments so that 2M could compare respondents’ perceptions to the intent of the questions and discuss any suggestions for improved clarity.

OMB Clearance Package

After cognitively testing the instrument, the 2M team will work closely with NEA and the PRA liaison to develop a clearance package (Parts A and B) for the web survey. Because the OMB clearance process is multifaceted and often takes significant time to obtain, the 2M team will develop the materials well in advance of any planned data collection to help NEA facilitate clearance of the package (see Study Timeline for details). The 2M team’s experience with OMB clearance on other Federal Government projects has demonstrated that by preparing the package diligently and attentively, OMB will require at most only minor modifications after the initial submission. The 2M team has found that working closely with the agency’s liaison to OMB enables us to identify potential clearance problems early, thus expediting OMB’s approval.

Prior to final submission of the OMB clearance package, the 2M team will develop and finalize the data collection tools they will use to collect information for the web survey. These tools include the survey instrument and respondent communication materials (e.g., instructions, introductory or follow-up emails). The 2M team will also develop a report of the results of the cognitive testing.

The 2M team helped NEA develop a 60-Day Federal Register (FR) Notice (published April 16, 2018). The notice informed the public of NEA’s intent to ask for clearance to collect information and solicited public comments for a 60‑day period (one comment received). Concurrent with developing the 60‑Day FR Notice, the 2M team developed a draft of the OMB clearance package that complies fully with NEA’s clearance requirements. The clearance package includes the Supporting Statement (Parts A and B) for the information collection and all supporting documents. We included all information necessary to complete OMB Form 83-I (PRA Submission).

The 2M team also prepared responses to the public comment that arose from the 60‑Day FR Notice and included it in the clearance package. We have also developed the 30-Day FR Notice. This second notice informs the public that NEA has submitted the clearance request to OMB.

During the OMB review period, the 2M team will help NEA prepare responses to any questions from the public or OMB, and then prepare a final, revised clearance package (as necessary) that incorporates any required changes. After OMB grants approval and assigns an OMB control number to the information collection, we will include this number and the clearance expiration date on all data collection tools.

Web Survey Data Collection

The 2M team will administer the web survey to all current and past Our Town grantees for which NEA has up-to-date contact information (approximately 388 grantees, 7 of which have completed cognitive testing). Figure 2 provides an overview of the process the 2M team will follow to administer the survey, as well as the target number and percentage of grantees that are in the response, nonresponse, or ineligible categories of the total grantee population at each step. First, NEA will send an email that invites grantees to take the survey and introduces the 2M team. This initial email from NEA is anticipated to be a highly effective strategy to increase response rates. Next, the 2M team will follow up with grantees by phone and with an email that provides a link to the survey. As Figure 2 shows, it is anticipated that several of the emails NEA sends out will be returned as undeliverable. The contractor will attempt to correct email addresses during the follow-up phone call. If the point of contact (POC) no longer works at the grantee organization, or the grantee organization itself is no longer in operation, the contractor will consider the grantee ineligible to take the survey and drop that grantee from the survey sample.

Figure 2. Web Survey Flowchart

Note. Figure 2 contains the target sample sizes and response rates at each step in the survey data collection process. The final numbers and percentages will depend on how many grantees are ineligible to take the survey, and how many respond to the survey.

The target response rate within 2 weeks of the initial email containing the survey link is 40 percent. The 2M team anticipates that about 5 percent of grantees will decline to take the survey at this stage. The 2M team will cease contact with grantees that decline the survey and add them to the nonresponse sample. We also anticipate that about 2 percent of grantees will be found ineligible in the first 2 weeks of data collection.

Two weeks after sending the initial email with the link to the survey, the 2M team will begin to email and call grantees to remind them to take the survey. We will call each grantee up to 3 times. In addition to the initial reminder email, the 2M team will continue to email automated reminders to grantees that have no activity on the web survey for 1 week or more. The target response rate by week 8 of data collection (6 weeks after the first email reminder) is 40 percent. Again, we anticipate that about 1 percent of grantees will decline the survey, and about 1 percent will be found ineligible (see Figure 2).

Next, 8 weeks after the initial email with the link to the survey, NEA will send an email reiterating the importance of the survey to grantees that have not responded; this email will include a link to the survey. The 2M team anticipates that this email will have a significant impact on the response to the survey because grantees are more likely to respond to NEA than to 2M. The target response rate from the NEA email is 40 percent. We also anticipate that about 1 percent of grantees will refuse the survey at this point in the process.

Finally, 9 weeks after the initial email, 2M will call the grantees that have still not responded to the survey. The target response rate for the final grantee calls is 15 percent (72 percent remain nonresponses, and 13 percent identify as ineligible).

The process outlined in Figure 2 shows the steps to arrive at the target response rate of 80 percent for the web survey. With 381 grantees in the total response population, 2M would reach this target with 64 percent (or 245 grantees) of the total population of grantees completing the survey, 16 percent of the total population refusing (or not responding to) the survey, and 20 percent of the total population identifying as ineligible to take the survey.

In the event that our target response rate of 80 percent is not met, the 2M team will conduct a nonresponse bias analysis to ensure there are no significant differences in the response rate of different types of respondents. The nonresponse bias analysis, if applicable, will analyze the response rate based on the following grantee characteristics: (1) the type of lead organization (government agency or nonprofit), (2) the status of the grant (current or past), (3) the budget of the project, and (4) grantee urbanicity (urban/rural/tribal). The nonresponse bias analysis can identify any difference between the total population and survey response sample in terms of these characteristics. If the 2M team finds a significant difference, we will then weight the survey data to be more representative of the total population. For example, if the nonresponse analysis finds that the proportion of past grantees in the survey sample is half of what the proportion is in the total population, we would weight the responses of past grantees twice as much as current grantees in the survey sample to compensate. The survey weights will correct for any difference in the characteristics between the grantee population and the survey response sample, as well as reduce bias in the survey. The 2M team will perform this step, if needed, prior to conducting any of the analyses of the survey data that we describe below.

Web Survey Analytical Approach

We provide an overview of our approach to the web survey data analysis in Figure 3. Our analysis of the survey data consists of five steps. First, the 2M team will import qualitative survey data from Confirmit into NVivo and code the “other-specify” response categories and the open-ended questions to identify and operationalize common themes. Second, we will import quantitative survey data from Confirmit and a coding spreadsheet of qualitative data from NVivo to Stata. Third, after preparing the data for statistical analysis, we will perform descriptive analysis of the responses to the closed-ended survey questions to understand the frequency of selection of each of the survey items. Fourth, the 2M team will perform statistical tests of association between the community contexts, inputs, activities, and outcomes captured in the survey. Finally, the 2M team will synthesize the results of all previous analyses to develop recommendations for NEA. We map the analyses in Figure 3 to the survey objectives and research questions in Table 8.

Figure 3. Steps of the Web Survey Analysis

Table 8. Survey Objectives and Research Questions

Survey Objectives	Survey Research Questions	Analytical Approach
1. Collect descriptive data that show the proportion of grantees with projects that do and do not align with the Phase I LM categories and MM indicators	RQ1a. Do Our Town grantees use the creative placemaking strategies and arts tactics and pursue the local community change outcomes specified in the Phase I LM? RQ1b. Do Our Town grantees capture evidence of local community change outcomes using indicators from the Phase I MM?	Descriptive analysis (Step 3)
2. Test how project community contexts, inputs, and activities are associated with the respondent’s perceived local community change outcomes and assess whether each LM element is associated with the perceived systems change outcomes	RQ2a. What are the associations between the elements of the Phase I LM (i.e., community contexts, inputs, activities, perceived local community/systems change outcomes)? RQ2b. What key LM elements are associated with perceived systems change outcomes? RQ2c. Do certain LM elements have a relationship with systems change that varies in different community contexts?	Statistical tests of association (Step 4)
3. Collect representative qualitative data that can identify outcomes, especially systems change, and other areas that are not represented in the Phase I models and develop recommendations for adjustments	RQ3a. What economic, social, physical, and systems changes do grantees pursue that the Phase I models do not capture? RQ3b. What indicators do grantees use as evidence of local community change that the Phase I MM does not include?	NVivo coding (Step 1)

Step 1: NVivo Coding

As a first step in the analysis of the web survey data, the 2M team will import qualitative data from open-ended questions and “other-specify” responses into NVivo. Then, using an inductive coding process, the 2M team will identify common themes for each open-ended and “other-specify” response. The 2M team will begin by reviewing a small, representative sample of the responses to identify a set of “codes” that represent the underlying concepts in the data. After this initial review, members of the 2M team will meet as a group to compare codes; consolidate codes that capture the same concept; and where possible, identify “parent” codes that represent broader themes, and “child,” or even “grandchild,” codes that represent more specific details within the broader codes. After agreeing on the initial set of codes (i.e., coding scheme), the 2M team will review all remaining data and organize them into the codes. Importantly, the coding scheme, while it guides the organization of the data, will be a living document that the 2M team can add to and revise as we review more data. In this way we “let the data speak for themselves” and identify codes intuitively and iteratively. After reviewing all data, the 2M team will collaboratively synthesize the data across all codes into a set of common themes that will represent concepts that grantees describe but that are not reflected in the Phase I models. We will report the high-level themes as well as the more granular codes and subthemes we identify to NEA to ensure our process is transparent and clear. The themes describing the local community change and systems change will offer data on the outcomes and indicators of various types of Our Town projects not anticipated in the TOC, LM, or MM. All themes will inform recommendations for adjustments to the Phase I models.

After coding the qualitative survey data, the 2M team will operationalize each high-level theme so that we can include it in the statistical analyses. We will operationalize each theme as a 0/1 indicator variable, where “0” indicates that the grantee did not discuss the theme and “1” indicates that the grantee did discuss the theme. NVivo can produce a spreadsheet with 0/1 indicators for each coding theme that we can easily import into any statistical software.

Step 2: Stata Data File

In the next step of the analysis of the web survey data, the 2M team will import into a Stata data file the indicator variables for each qualitative theme and the quantitative survey data from close-ended questions from Confirmit. We will operationalize the close-ended survey data as follows:

For items that are “select one” or “select all,” we will assign a 0/1 indicator for each response category, where “0” indicates that the respondent did not select the category and “1” indicates that the respondent did select the category.
We will assign yes/no items a 1 if a respondent selects “Yes” and a 0 if a respondent selects “No.”
We will assign a 0 to the lowest or most negative response in a Likert-type scale, a 1 for the next response, and so on. For example, we would score an item measured with strongly disagree to strongly agree as “strongly disagree” = 0, “disagree” = 1, “agree” = 2, and “strongly agree” = 3.
For questions that ask grantees to describe a concept before and after their Our Town project (i.e., “before and after” questions), 2M will create change variables that are the difference between the grantee response for the after period and the response for the before period. For example, if a grantee answers “disagree” (1) for the before period and “agree” (2) for the after period, the change variable for that concept would equal 1 (2-1); if a grantee answers “strongly disagree” (0) for the before period and “strongly agree” (3) for the after period, the change variable for the that concept would equal 3 (3-0); and so on.

The 2M team will use the Stata data file for the descriptive and statistical tests of association.

Step 3: Descriptive Analysis

The third step of the web survey analysis—the descriptive analysis—tests alignment between the grantees and the Phase I models, which will help NEA and the 2M team understand how often grantees used the concepts represented in the TOC, LM, and MM. The most important evidence for alignment is the frequency and percentage of grantee responses to the “other” categories in the survey. If a grantee selects “yes” for an “other” category pertaining to the creative placemaking strategies, arts tactics, or local community change outcomes, the grantee does not align with the Phase 1 models for that concept because the grantee is indicating that none of the specified categories fit their project. The overall frequency and percentage of “yes” responses to each “other” category will thus inform NEA of the degree to which the Our Town grantees do not align to the Phase 1 models.

In addition to the analysis of the responses to the “other” categories, the 2M team will also analyze the frequency and percentage of responses to all close-ended questions and all identified qualitative themes. For close-ended questions that ask grantees to think about a concept before and after the project, e.g., the local community change survey domain, the 2M team will provide a side-by-side comparison of the survey responses that correspond to the timeframe before the project and the responses that correspond to the timeframe after the completion of the project. In addition, we will report the frequency and percentage of the change variables described above, which will provide the frequency and percentage of grantees that provide a more positive response for the period after the project (e.g., moving from “disagree” to “agree” from before to after the project) as well as those that provide a more negative response (e.g., moving from “agree” to “disagree” from before to after the project). These data will provide useful information to NEA on how Our Town projects tend to happen on the ground. This information can inform decisions to revise and improve the Phase 1 models. Specifically, it can

identify common themes (qualitative themes that large percentages of grantees discuss in response to survey questions) that NEA should consider for inclusion in the Phase 1 models;
inform NEA of the distribution of the community contexts, inputs, activities, and outcomes among the Our Town grantee population; and
provide some evidence of the benefits/changes grantees perceive their projects produce. This information will come primarily from the local community change survey domain, which includes “before and after” questions. As we state below, this evidence should be interpreted with caution as it represents grantee perceptions of change rather than change itself.

Finally, the 2M team will utilize descriptive analysis to assess survey items with missing data (no response). For any item that is missing for 5 percent or more of grantees, 2M will compare respondents with missing data for the item with respondents with complete data for the item by using data from other sections of the survey to understand whether respondents with particular types or combinations of characteristics tend not to answer the question. We will report to NEA the results of the missing data analyses and any implications the missing data have for the interpretation of results.

Step 4: Statistical Tests of Association

In addition to descriptive analysis, the 2M team will perform statistical tests of association to understand the evidence answering the second research question. First, the 2M team will perform pairwise correlations³ between all close-ended questions and operationalized qualitative themes in the Stata dataset. A “pairwise correlation” is a statistic that estimates the degree of association between two variables. The statistic can range from −1 to 1, where −1 is a perfect disassociation between the variables (a change in one is associated with the opposite change in the other), 0 is no association (no relationship at all between the two variables), and 1 is perfect association (a change in one is associated with the same change in the other). The value of the correlation statistic indicates the degree to which the concepts occur together (or not). Like other statistics, it is also possible to test the statistical significance of the correlation statistic. The statistical significance provides evidence that the finding is not the result random chance but rather represents a true relationship

The 2M team will assess three scenarios with different interpretations with the pairwise correlations:

A correlation statistic greater than 0 that is statistically significant, with 90 percent confidence
A correlation statistic less than 0 that is statistically significant, with 90 percent confidence
A correlation, regardless of the value, that is not statistically significant, with at least 90 percent confidence

The first scenario indicates a positive association between two concepts in the survey, which means that the two concepts often occur together. In contrast, the second scenario indicates a negative association or a disassociation between two concepts in the survey, which means that the two concepts often do not occur together. Finally, the third scenario indicates a finding that could be the result of random chance. The 2M team will consider findings under the third scenario as evidence of no relationship between the two concepts.

In addition to the pairwise correlations, 2M will test multivariate regression models⁴ for each qualitative theme that captures systems change outcomes. This analysis can determine the factors, or concepts in the survey, that are mostly likely to contribute to (or impede) systems change. Although the pairwise correlations can indicate associations between systems change outcomes and other concepts captured in the survey, the multivariate regression analysis can estimate the associations while controlling for confounding factors. For example, a pairwise correlation may indicate that a systems change outcome is strongly associated with projects in urban areas, but in truth this association results because projects in urban areas tend to have more partners, which is also strongly associated with the systems change outcome. A multivariate regression can test whether the association between the systems change outcome and urban areas is still found regardless of the number of partners a project may have.

Second, multivariate regressions can see how associations between systems change outcomes and other concepts in the survey change in different contexts. Continuing with the example, the 2M team may use a multivariate regression to understand how the association between the number of partners and a systems change outcome changes in urban, rural, and suburban contexts.

The 2M team will test two forms of multivariate regression models:

Multivariate regression models that capture associations between each systems change outcome identified in the qualitative analysis and all other concepts captured in the survey (includes community contexts, inputs, activities, and local community change outcomes)
Multivariate regression models that show how the associations change in different community types

Both models provide evidence of the factors associated with different types of systems changes.

Step 5: Synthesis of Findings

In the final step of the analysis of survey data, the 2M team will synthesize the findings from all previous analyses into key takeaways and recommendations for improvements to the Phase I TOC, LM, and MM. In particular, 2M anticipates the following categories of recommendations:

Recommendations for specifying types of systems change outcomes, informed by the qualitative analysis of open-ended systems change questions
Recommendations for representing relationships between contexts, inputs, activities, and local community change and systems change in the TOC and LM, informed by the statistical tests of association
Recommendations for adjustments to contexts, inputs, activities, or local community change outcomes and indicators informed by the descriptive analysis and qualitative analysis of “other-specify” questions

Limitations of the Survey Analytical Approach

Finally, NEA should note two limitations to the web survey analytical approach. First, the outcomes, including systems change, that the survey captured are mostly based on grantee perceptions rather than empirically observed change. Grantees should have a good idea of the types of change their project may lead to, but NEA should not use the results from the survey analysis as evidence of the observed impacts of the Our Town project. Rather, NEA should use the evidence to understand and describe how Our Town projects happen in their local communities and the types of change they may produce.

Second, the usefulness of the statistical tests of association, especially the regression analysis, will depend on how comprehensively grantees respond to the survey. If, for example, very few grantees respond to the systems change questions in detail, the analysis may not be able to find meaningful relationships between the elements of Our Town projects (i.e., contexts, inputs, activities, and local community change outcomes) and systems change. The 2M team can pursue other analysis options if the systems change data are not sufficiently detailed or only a small number of grantees provide rich information. For example, 2M can identify the grantees that do respond in detail to the systems change questions and qualitatively describe and compare their attributes to get some understanding of the factors related to systems change. We intend to pursue the regression analysis first because it can quickly show significant and meaningful relationships in the data; however, we will be cognizant of the limitations of the analysis and ensure, regardless of the degree of the grantee responses, that NEA will understand the characteristics of grantees that pursue and foster systems change.

Study Timeline

Table 9 details our Phase II timeline.

Table 9. Detailed Phase II Timeline

Week	Task	Activity	Who	Status Notes	Work Period	Done
0	Kickoff Meeting (C.3.2.1)					
1		Meeting materials delivered to NEA	Murdoch		12/7/2017–12/8/2017	
1		Convene Kickoff Meeting	Holden, Murdoch, Morrissey, NEA		12/12/2017	
2		Kickoff Meeting Memorandum	Murdoch, Morrissey		12/12/2017–12/19/2017	
	Work Plan (C.3.2.2)
–		Recurring Meeting Minutes	Murdoch, Morrissey		3 business days after meeting	–
–		Monthly Progress Reports	Murdoch		15th of each month, beginning 1/15/2018	–
3		Draft Work Plan	Murdoch, Holden		12/7/2017–12/22/2017	
4		Final Work Plan	Murdoch, Holden		1/3/2018	
	Study Plan (C.3.2.3)
6		Project Timeline, including key meetings with NEA Design team	Murdoch		1/17/2018	
6		Revised Research Questions	Murdoch, Morrissey		1/19/2018	
7		Case Study Selection Process and Approach	Murdoch, Morrissey, Nicodemus (Metris)		1/26/2018	
9		Draft Survey Instrument	Murdoch, Morrissey		1/19/2018–2/9/2018	
9		Identify potential candidates for Case Studies	NEA		1/26/2018–2/12/2018	
11		Revision #1 Survey Instrument	Murdoch, Morrissey		2/9/2018–2/21/2018	
12		Reconnaissance calls with grantees (five)	Morrissey, Nicodemus (Metris)		2/20/2018–2/26/2018	
12		Submit recommendations for grantees for Case Studies	Murdoch, Morrissey, Holden, NEA, Technical Working Group (TWG)		2/28/2018	
12		Draft Study Plan, including Revision #3 Survey Instrument	Murdoch, Morrissey, Holden		1/3/2018–3/9/2018	

14		NEA initial feedback on Study Plan	NEA		3/5/2018–3/19/2018	
16		Survey sample data points identified by 2M	Murdoch		3/28/2018	
16		Meeting to discuss final Case Study selection and final revisions to Survey Instrument before cognitive testing	NEA, Murdoch, Nicodemus (Metris)		3/30/2018	
17		Revision #1 of Study Plan	Murdoch, Morrissey		3/19/2018–4/2/2018	
17		Convene PRA planning call	NEA, Murdoch		4/5/2018	
18		Initial Site Visit Discussion Guides	Nicodemus (Metris); Morrissey		4/11/2018	
18		NEA feedback on Study Plan Revision #1	Murdoch, Morrissey		4/2/2018– 4/9/2018	
18		60-Day FR Notice	NEA		4/10/2018	
19		Revised Survey Instrument #4	Murdoch		3/19/2018–4/20/2018	
34		Cognitive testing	Murdoch, Morrissey, RAs		5/16/2018–8/3/2018	
38		Cognitive Testing Report	Murdoch		8/6/2018–8/27/2018	
38	Key Deliverable #2	Final Study Plan	Murdoch, Morrissey, Holden	Up to three revisions	3/2/2018–8/27/2018	
38		Draft Supporting Statements for OMB Survey Clearance Package	Murdoch, Morrissey, Holden		6/1/2018–8/27/2018	
42	Key Deliverable #3	Final OMB Survey Clearance Package	Murdoch, Holden		8/27/2018–9/24/2018
42		30-Day FR Notice	NEA		9/24/2018
	Implementation (C.3.2.4)
20		Develop project-level LMs (three)	Morrissey, Nicodemus (Metris)		4/2/2018–4/27/2018	
22		Site visit planning call with grantees (three)	Morrissey, Nicodemus (Metris)		4/30/2018–5/9/2018	
–		Develop tailored Site Visit Discussion Guides	Morrissey, Nicodemus (Metris), Holden		1-2 weeks before site visit	
38		Conduct in-person site visits with grantees (three)	Nicodemus (Metris), Morrissey		5/14/2018–8/31/2018	
42		Qualitative data analysis	Nicodemus (Metris), Morrissey		Beginning after first site visit through 9/28/2018
–		Complete Case Studies (three)	Nicodemus (Metris), Morrissey, Holden		6-8 weeks after site visit
42		Interactive grantee sessions to review project-level LMs (three)	Morrissey, Nicodemus (Metris)		3-4 weeks after site visit
42		Provide deidentified case study interview transcripts	Morrissey, RAs		9/28/2018	
49		Cross-Case Study (one)	Nicodemus (Metris), Morrissey, Holden		8/6/2018–11/16/2018
49		Submit draft of Chapter 4 (Case Studies) of Study Report	Nicodemus (Metris), Morrissey, Holden		8/6/2018–11/16/2018
59		Obtain OMB approval, including 30-day announcement	OMB	120 days from submission of the package	1/22/2019
71		Survey implementation	Murdoch, RAs		1/22/2019–4/19/2019
74		Provide a Microsoft Excel spreadsheet of survey findings specific to LM components	Murdoch		4/22/2019–5/6/2019
76		Complete survey data analysis and submit draft of Appendix B (Web Survey) of Study Report	Murdoch, Holden		4/22/2019–5/20/2019
	Final Study Report Activities
80		Submit 1^st draft Study Report	Murdoch, Holden, Morrissey		5/20/2018–6/10/2019
82		Feedback on 1^st Study Report	NEA		6/11/19–6/24/2019
83		Submit 2^nd draft Study Report	Murdoch, Holden, Morrissey		6/25/19–7/1/2019
85		Feedback on 2^nd Study Report	NEA		7/2/2019–7/15/2019
86		Submit 3^rd draft Study Report	Murdoch, Holden, Morrissey		7/16/2019–7/22/2019
87		Feedback on 3^rd Study Report	NEA		7/23/2019–7/29/2019
88	Key Deliverable #4	Submit Final Study Report	Murdoch, Holden		7/30/2019–8/5/2019

Protection of Human Subjects

Case Studies

For all recorded interviews, the 2M team will review the informed consent notice with interviewees to ensure that all participation is voluntary and that interviewees understand NEA’s desire to share the case studies with a broad audience. For each case study, the grantee name and project location will be identified, but interviewees will be given anonymity. To increase the anonymity of interviewees, in all reporting materials, interviewees will be referenced according to their respective respondent project role (e.g., neighborhood resident, local business, volunteer) or partner sector rather than by name or position. There is moderate risk of loss of anonymity for participants. Grantee staff may be able to identify individuals based on their reported role because grantee staff will initially provide all contact information; this will be explicitly described to the interviewee in the informed consent script. For the Forklift Danceworks grantee, raw transcripts will be provided to GO Collaborative Austin, an NEA-funded evaluator, for internal purposes only.

In the case study interview transcripts, all personally identifiable information (PII), including names, will be redacted to ensure confidentiality. 2M will report to NEA the number and type of respondents interviewed at each site. For any visual data collected during site visits, the 2M team will obtain written consent before including human subjects. In addition, reports may include publicly released photographs and visuals.

Web Survey

The web survey will not include any PII and thus poses minimal risk to the respondents. The 2M team will provide all respondents with a description of the importance of the survey and a notification that their response to the survey is voluntary. As a part of OMB clearance, we will help NEA respond to public comments on the data collection and ensure the usability, readability, and clarity of the survey through cognitive testing. The 2M team will store all survey data on a secure data server and ensure that we deliver the data to NEA securely. We will work with NEA to identify the best method of data transfer; however, we have found, for data without PII, that setting up a Microsoft SharePoint site and giving the client a login to the site works best.

Communications Plan

The 2M team developed a plan for all communications with Our Town grantees during Phase II of the Our Town Implementation Study. Generally, we request that NEA contact grantees first by email to introduce the 2M team and the purpose of the study. After the initial email from NEA, the 2M team will handle all communication with grantees. If a grantee contacts NEA directly with questions or concerns about the study, NEA may direct the grantee to the Our Town Implementation Study project manager, James Murdoch, or, if NEA deems it necessary, the PD, Debra Holden. We provide more details about our plans for communication with grantees for the case studies and web survey below.

Case Studies

The 2M team provided NEA with scripted emails for initial introduction of the 2M team to each grantee for participation in reconnaissance calls (Thereafter, the 2M team will communicate with grantees directly during the site visit planning call and ad hoc correspondences for site visit planning, follow-up, and post-site visit interactive sessions for LM review. The site visitors will lead all communications with grantees regarding the case studies.

Once the three grantees for each case study are finalized, the 2M team will send a study declination email to the grantees not selected and, at least 3 weeks prior to the site visit, an email requesting a 20- to 30-minute phone call with staff of each grantee selected as a case study to collect contact information, create a preliminary site visit agenda, and discuss program activities, as needed. After the site visit planning call, 2M will ask the grantee to send an email to introduce 2M and the purpose of the interview to each potential interviewee. The 2M team will coordinate and schedule up to seven interviews to decrease grantee burden and will interact directly with potential interviewees after the introduction and purpose email to increase anonymity.

During the site visit, grantee staff will be the primary POCs for the 2M team onsite, but the 2M team will have the contact information of each interviewee and will operate autonomously as the schedule allows. The 2M team will review the informed consent notice prior to each interview to ensure that participation is voluntary. 2M will implement tailored versions of the discussion guides approved by NEA. After the site visit, the 2M team will send a thank you email to all interviewees.

After the site visit, the 2M team will send an email to schedule an interactive video session, as available, with each grantee and up to two additional stakeholders to review the project-level LM. During the call, the 2M team will provide a step-by-step review of the LM and collect any feedback or revisions the grantee staff offer. The 2M team will send the final project-level LM to the grantee 2 weeks after the interactive call.

Web Survey

Once we finalize the grantee candidates for the cognitive testing of the web survey, NEA will send an email to each candidate inviting them to participate in the cognitive testing. The email will ask grantees to confirm with NEA and 2M whether they will participate in the cognitive testing. Once the 2M team receives five confirmations, the 2M team will send the remaining grantees an email informing that they were not selected for cognitive testing. The grantees (up to five) that are selected for cognitive testing will receive an email that provides instructions on how to access the survey online in Confirmit.

After cognitive testing and OMB approval, the 2M team will ask NEA to send past and currently funded Our Town grantees for whom they have up-to-date contact information an email invitation that describes the importance of the study and introduces the grantees to the 2M team. The 2M team will then follow up with an email to the grantees that provides instructions on how to access the survey online in Confirmit. In addition, the email will provide a toll-free number and email address that grantees can use to ask questions about the survey and receive technical assistance if they have trouble accessing, answering, or using the survey. Then, 2 weeks after the initial email, the 2M team will send a follow-up email to grantees that have not yet started the survey online. In addition to these emails, 2M will use the Confirmit system to track the progress of grantees in the survey. If a grantee starts a survey but does not finish, 2M will send an automated reminder to complete the survey after 1 week of inactivity. As with the invitation and reminder emails, the automated reminder will include the toll-free number and email address that respondents can use if they are having any issues with the survey. Eight weeks after the initiation of the survey data collection, the 2M team will ask NEA to send a final email to nonresponding grantees reiterating the importance of the survey and asking grantees to complete the survey online. Throughout the survey data collection, the 2M team will call nonresponding grantees up to three times (each time an email is sent, excluding the automated reminders) to ask grantees to complete the survey online. At the end of the 12th week of data collection, 2M will close the web survey.

Reporting Plan

The 2M team will report all findings of the study in the Final Study Report. This final deliverable for Phase II of the Our Town Implementation Study will include the following details:

An executive summary of the study implementation and Phase II findings
Introduction and background
Discussions of key findings from data analyses by research question
Recommended revisions to the TOC, MM, and LM based on study findings
Conclusions and recommendations about future Our Town guidelines development, grantee reporting requirements, and performance metrics
For Appendix A, key case study findings and narratives
For Appendix B, detailed web-based survey findings, including tables of descriptive statistics for each survey question and the results of the statistical tests of association
For Appendix C, a full description of the study methods, including research questions, design and methods, and materials and instruments used in data collection

The 2M team will submit the chapters of the report in sections for iterative review by NEA, as we did for the Framing Document during Phase I. We will use survey and case study analyses to present findings as they relate the study research questions and overall validation of Phase I Our Town tools. The survey and case study data provide different, but complementary, data that will inform the 2M team’s recommendations for revisions to the TOC, LM, and MM. In particular, the case studies will provide in-depth evidence of systems change and the factors that foster it, both of which can inform the Phase 1 models, while the survey will provide more generalized evidence of how Our Town projects occur on the ground that both validates and informs the Phase 1 models. As in Phase I, the 2M team is dedicated to providing visual and graphical representations of data that can easily be shared with a wider audience.

In addition to the Final Study Report, the 2M team will provide NEA with several deliverables specific to the case studies and web survey, which we describe below.

Case Studies

The 2M team will produce a single Case Study consisting of 2,500 to 3,000 words for each site to describe the grantee and outcomes of the Our Town project and to provide greater context to the survey findings; 2M will work with NEA to ensure that the case studies, or specific sections of the case studies, are appropriate for public audiences. The 2M team will organize the case studies using a similar structure across case study reports addressing the case study objectives. The case studies will be written as a narrative and include direct quotations with the respective respondent role.

Upon completion of the three case studies, the 2M team will produce a Cross-Site Analysis Report to demonstrate commonalities and differences between the activities, outcomes, and key contexts of the three sites as they relate to the research questions. Specifically, the report will synthesize high-level findings to depict variations and similarities between sites in systems change outcomes and the associated factors. The 2M team will prepare the document for a public audience and include visualizations, photographs, and graphics, as available, to enliven the report.

As previously discussed, the 2M team will develop project-level LMs for inclusion in each case study. We will review the project-level LMs with the grantee after the site visit during an interactive video conference to ensure accuracy and make any necessary revisions. We will include the final project-level LM in each case study to demonstrate the application of the Phase I project-level LM and any suggested recommendations.

Web Survey

The 2M team will provide NEA with a database (both Excel and Stata) of all survey data. The database will include

raw survey data responses, including text to open-ended and “other-specify” questions;
operationalized survey data as 0/1 indicators and ordinal scales (Likert-type questions);
Stata syntax that the 2M team used to produce all tables, figures, and other numbers that come from the web survey in the Final Study Report; and
a codebook describing all variables in the database.

The 2M team will report the web survey findings in a chapter of the Final Study Report. We will organize the chapter based on the survey objectives and research questions, providing the evidence from the web survey that answers each question. We will report the results of the descriptive analysis and statistical tests of association in tables and figures as well as in narrative. We will report the results of the qualitative analysis with narrative, using representative quotes of the themes we identify to increase clarity.

1 Metris Arts Consulting, (2017, September). Equity reflection questions. Retrieved from https://metrisarts.com/wp-content/uploads/2017/09/equity_reflection_questions_092217.pdf

2 Patton, M. Q. (2002). Qualitative research and evaluation methods. Thousand Oaks, California: Sage Publications.

3 The variables in the Stata dataset will be binary (0/1 indicators), ordinal (Likert-type questions), or in rare cases (e.g., number of partners), continuous. The 2M team will use the appropriate estimation technique for each variable. Specifically, the 2M team will estimate pairwise correlations between binary variables with tetrachoric correlations and all other correlations with Spearmen’s rank correlation coefficient.

4 All models will be logistic regression models with the 0/1 systems change outcome themes as the dependent variables.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Subject	C16-51
Author	James Murdoch, MPP
File Modified	0000-00-00
File Created	2021-01-20