Cognitive Testing Report

From May 16, 2018 to August 3, 2018, 2M Research (2M) performed cognitive testing of the Our Town Study web survey instrument with seven Our Town program grantees. The purpose of the cognitive testing was to detect issues with the usefulness, clarity, and readability of the items in the web survey as well as to estimate the average time to complete the survey. After performing the cognitive testing, 2M met with NEA to discuss recommendations to improve the web survey instrument. This report summarizes the comments of the seven Our Town grantees as well as the revisions to the survey that NEA and 2M implemented based on the comments. The report is organized into two main parts. Part 1 describes the comments received during the cognitive testing process and resulting modifications to the survey. Part 2 provides the revised web survey instrument, with revisions highlighted in yellow. The report also includes an appendix with the full comments the cognitive testing respondents provided on the survey.

Sample

To identify cognitive testing respondents, NEA provided 2M with a list of grantees that met the following criteria:

At least one current grantee
At least one past grantee
At least one tribal grantee
At least one rural grantee
At least one urban grantee
At least one grantee per arts tactic category – Arts Engagement, Cultural Planning, Design, and Artist/Creative Industry Support
A diversity of grantee partnerships, i.e., led by a government agency, led by an arts/cultural nonprofit community organization

Using these selection criteria, 2M intended to collect cognitive testing data from a diverse sample of grantees to gain grantee input from a variety of perspectives. Purposefully sampling diverse grantees will help to identify any gaps in the survey responses or other areas for improvement and will help to ensure the survey is appropriate for a range of respondents.

Table 1 describes the seven grantees with which 2M performed cognitive testing of the web survey instrument.

Table 1. Cognitive Testing Respondent Sample

Respondent	Type	Survey Test	Interview
1	Past, urban, nonprofit, artist/creative industry support	Complete	Refused¹
2	Current, rural, tribal, cultural planning	Complete	Email
3	Current, rural, nonprofit, design	Complete	Refused¹
4	Past, urban, nonprofit, engagement	Complete	Phone
5	Past, urban, municipal government, artist/creative industry support	Complete	Phone
6	Past, urban, nonprofit, cultural planning	Complete	Phone
7	Current, rural, nonprofit, arts engagement	Complete	Phone
¹Two respondents refused cognitive testing interviews due to time constraints; however, both individuals provided adequate comments within the open-ended questions in the survey to support cognitive testing analysis.

Description of data collection process

2M programmed the draft web survey by using the survey software Confirmit. We sent the selected cognitive testing respondents a link to take the survey in Confirmit in a pre-test mode, which allows respondents to take notes for each question related to level of effort, comprehension, clarity, usefulness, and other thoughts. In addition to respondent notes, 2M assessed the amount of time respondents spent on each question, distinct from the amount time respondents spent providing notes, to identify questions that may have issues. After the respondent submitted the survey online, 2M received a notification and reviewed the respondent notes and other information. A 2M interviewer then reached out to the respondent by telephone and/or email to schedule a 30-minute cognitive interview.

2M conducted the interview by telephone (one respondent requested to answer questions via email) to review the respondents’ notes and discuss any issues that arose with specific questions during the survey. The interview also included general questions about the timing of the survey and the clarity of the survey questions. 2M emailed a Microsoft Word version of the survey to the respondents prior to the cognitive interview so that respondents were able to re-read specific survey questions, as needed. 2M asked respondents to elaborate on specific comments made to the online survey so that 2M could compare respondents’ perceptions to the intent of the questions and discuss any suggestions for improved clarity.

Part 1: Modifications to the Web Survey Instrument

The comments respondents provided on the web survey instrument online as well as in the cognitive interview were generally positive. All respondents felt the survey was highly relevant to Our Town grantees and the larger field of creative placemaking, and that the survey was comprehensive of how Our Town projects operate. Most of the comments suggesting revision focused on specific issues of clarity. The comments that NEA and 2M utilized to revise the survey are summarized below. The appendix of this report includes the detailed comments from each respondent verbatim.

Modifications Based on General Comments on the Survey

Table 2 summarizes the general comments on the web survey instrument 2M received from the seven cognitive testing participants. In addition, it provides the revisions to the survey instrument that 2M and NEA implemented based on the respondent comments.

Table 2. General Comments and Revisions to the Web Survey

Comments	Revisions
Most respondents said that the survey felt long. Average time spent was about 45 minutes to an hour, including comments.	Add a progress bar indicating percent complete. Several revisions in Table 3-7 below also reduce the survey length.
Many respondents said they thought grantees may hesitate to answer negatively (e.g., “disagree” or “strongly disagree”) to questions because they do not want NEA to think badly of their projects.	Add statement to beginning of survey explaining the importance of honest answers and that NEA is interested in grantees’ perceptions Add a statement to every survey heading/section page that answers will not affect current or future grants Add statement to every survey heading/section page that responses will help improve the Our Town program to emphasize the importance of the study
All respondents were positive about the survey overall. All respondents agreed that the survey is very relevant and understandable. One respondent indicated the N/A option was often confusing (e.g., question 1-1b: if your grant was not designed to produce the change, do you answer “N/A” or “disagree”?).	Offer “N/A” option only in specific instances and, if included, ensure “N/A” is clearly defined

Modifications Based on Comments on Specific Survey Questions and Items

The web survey instrument has several domains that pertain to different aspects of an Our Town project. The following tables summarize the comments and revisions to the survey instrument for Local Community Change (Table 3), Systems Change (Table 4), Project Community Context (Table 5), Project Inputs (Table 6), and Project Activities and Strategies (Table 7).

Table 3. Specific Comments and Revisions to the Local Community Change Domain

Comments	Revisions
Most respondents suggested that these questions are difficult because the concepts the survey asks about are not typically measured/tracked. This particularly applies to new grantees or grantees with projects that haven’t started yet because no/limited activities have occurred. The respondents also expressed the need to qualify/explain to NEA why particular concepts are not captured in their project.	Add question before the domain that asks, “have you completed most of your project activities” with “Yes” and “No” (Y/N) response options. Respondents that answer “no” will not be asked the detailed “after” questions for the Local Community Change domain.
One respondent suggested starting with social change as she felt many of the other changes begin with social change.	To minimize nonresponse to any particular local change category (we anticipate fewer respondents will answer items at the end of the survey due to survey fatigue), 2M will program the web survey to randomize the order of economic, social, and physical change. Sometimes social change will come first and sometimes not.

Table 4. Specific Comments and Revisions to the Systems Change Domain

Comments	Revisions
Three respondents suggested that the questions are worded to produce yes/no answers rather than an open-ended response.	Rephrase questions to be more close-ended
Question 2-2 and 2-3 were confusing for four respondents; clarification is needed on what is being “sustained” and what is meant by “sustained.” Respondents liked the specific examples provided in Question 2-6 NEA and 2M discussed that questions 2-1, 2-2, 2-3, and 2-5 are too similar to 2-6	Remove questions 2-1, 2-2, 2-3, and 2-5, as they are too similar to question 2-6. Update question 2-6 (question 5-2 in the revised instrument) to define “sustained” as beyond the Our Town grant period.
One respondent suggested including close-ended questions for each type of systems change that allow grantees to specify the degree of the change, to increase comprehension/ease of answering.	No action taken. Open-ended Systems Change information is important because it can help to refine the concept in the program theory of change and logic model
Two respondents suggested question 2-7a and 2-7b would be easier to answer with a neutral option, e.g., “neither agree nor disagree.”	Revise questions 2-7a and 2-7b (questions 5-3a and 5-3b in the revised instrument) to include a neutral response option.

Table 5. Specific Comments and Revisions to the Project Community Context Domain

Comment	Revision
For question 3-3, one respondent suggested splitting population and business into two separate questions, rather than using “and/or.”	Revise question 3-3 (question 4-3 in the revised instrument) to ask only about population change and not business change in the target community.

Table 6. Specific Comments and Revisions to the Project Inputs Domain

Comments

Revisions

Question 4-1 and 4-2 were confusing for five respondents:
- The term “partner” can mean so many different things.
- Larger partners (e.g., a city or a university) may have certain agencies/ departments that play very different roles.
- It is hard to remember the exact number, especially using a broad definition of partners that includes individuals and organizations.
Respondents indicated question 4-3 was not well organized and was difficult to understand and answer.

Revise the partner questions to include an open-ended question about the top partners (up to five) involved in a respondent’s project, a close-ended question about the role of each of the partners, and a question asking respondents to estimate the number of additional partners involved in their projects.

Table 7. Specific Comments and Revisions to the Project Activities and Strategies Domain

Comment	Revision
All respondents indicated that grantees will almost always “strongly agree” or “agree” to all creative placemaking strategies (question 5-2).	Revise question 5-2 (question 2-2 in the revised instrument) to ask respondents to rank the creative placemaking strategies.

Part 2: Revised Web Survey Instrument

OMB Number: XXXX-XXXX

Expiration Date: XX/XX/XXXX

Study of the Our Town Program

Grantee Web Survey

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is XXXX-XXXX, expires XX/XX/XXXX. The time required to complete this information collection is estimated to average 30 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of time estimates or suggestions for improving this form, please contact: National Endowment for the Arts, 400 7th Street SW, Washington, DC 20506-0001. Attention: Sunil Iyengar.

~~The National Endowment for the Arts (NEA) is conducting a national web survey of all current and past~~ ~~Our Town~~ ~~grantees to gain a better understanding of how~~ ~~Our Town~~ projects take place in local communities and the types of change they foster. As one of the first large-scale surveys of a national creative placemaking program, the results of this study can provide important evidence for support and recognition of arts, design, and cultural strategies as integral to community planning and development. As a grantee of the ~~Our Town~~ ~~program, your responses to the web survey will help ensure that the results of our study accurately represent how~~ ~~Our Town~~ projects work, inform how NEA can improve and increase the effectiveness of its creative placemaking grants, and contribute to the growing evidence base for the importance of creative placemaking programs such as ~~Our Town.~~

2M Research (2M) was contracted by NEA to conduct this survey as a part of the investigation called the “Our Town Program Implementation Study.” If your organization received more than one Our Town grant, please answer this survey thinking about the most current grant project you are working on. Many of the questions in this survey ask about your perception of how your Our Town project is working and the types of changes to which you think your project is contributing. NEA is very interested in your perception of how Our Town projects unfold in local communities. Please answer all questions honestly, responses provided will be used to improve the program. 2M will keep the information you provide in this survey private, to the extent provided by law. We will not attribute any data to specific survey respondents. We will redact any personally identifying information (such as names) and will provide data from this study to NEA. Your responses to this survey will in no way impact the outcome of any present or future NEA grant applications, contract proposals, or cooperative agreement proposals from your institution or institution’s partners.

Please note, if you cannot complete the survey in one sitting, you can save it and complete it at a later date. You may also share the login credentials with appropriate personnel as you deem necessary in order to report accurate information. The survey will take about 30-45 minutes to complete.

Please note that your participation in this study is completely voluntary, and you may stop at any time if necessary. We thank you for your participation and cooperation in this very important study.

Domain 1. Descriptive Information

Please review the following descriptive information for the primary awardee of the Our Town grant that this survey pertains to.

1-1. Please select “Yes” if the information is correct. Please select “No” if the information is incorrect and enter the correct information.

Descriptive information	Yes	No	Please enter correct information (if “No” selected)
Awardee organization name: [WEB SURVEY POPULATES]	☐	☐
Grant ID: [WEB SURVEY POPULATES]	☐	☐
Award year: [WEB SURVEY POPULATES]	☐	☐
5-digit zip code of the awardee organization: [WEB SURVEY POPULATES]	☐	☐

Domain 2: Project Activities and Strategies

Please answer the following questions about your Our Town project’s activities, including aspects of the events, programs, or other strategies you [will complete/completed] as a part of your Our Town intervention during the time that your project [is/was] funded by NEA.

NEA is very interested in your perception of how Our Town projects unfold in local communities. Please answer all questions honestly, responses provided will be used to improve the program. Your responses to this survey will in no way impact the outcome of any present or future NEA grant applications, contract proposals, or cooperative agreement proposals from your institution or institution’s partners.

Activities

2-1. Please select “Yes” if the category reflects activities your Our Town project [uses/used]. Please select “No” if the category does not reflect activities your Our Town project [uses/used].

[IN THE WEB VERSION DEFINITIONS OF THESE ITEMS APPEAR IF A USER HOVERS OVER THE ACTIVITY.]

Arts Engagement

*2-1a. The activities that my Our Town* project [uses/used] include…**	Yes	No
Artist residency: A program designed to connect artists with the opportunity to bring their creative skillsets to non-arts institutions, including government offices, businesses, or other institutions	☐	☐
Arts festivals: Public events that gather people, often in public space or otherwise unexpected places, to showcase talent and exchange culture	☐	☐
Community co-creation of art: The process of engaging stakeholders to participate or collaborate with artists/designers in conceiving, designing, or fabricating work(s) of art	☐	☐
Performances: Presentations of a live art work (e.g., music, theater, dance, media)	☐	☐
Public art: A work of art that is conceived for a particular place or community, is broadly accessible, and often involves community members in the process of developing, selecting, or executing the work	☐	☐
Temporary public art: A work of art that is conceived for a particular place or community, is meant for display over a finite period of time, is broadly accessible, and often involves community members in developing, selecting, or executing the work	☐	☐

Cultural Planning

*2-1b. The activities that my Our Town* project [uses/used] include…**	Yes	No
Cultural planning: The process of identifying and leveraging a community’s cultural resources and decision making (e.g., creating a cultural plan, or integrating plans and policies around arts and culture as part of a city master planning process)	☐	☐
Cultural district planning: The process of convening stakeholders to identify a specific geography with unique potential for community and/or economic development based on cultural assets (e.g., through designation, branding, policy, plans, or other means)	☐	☐
Creative asset mapping: The process of identifying the people, places, physical infrastructure, institutions, and customs that hold meaningful aesthetics and historical and/or economic value that make a place unique	☐	☐
Public art planning: The process of developing community-wide strategies and/or policies that guide and support commissioning, installing, and maintaining works of public art and/or temporary public art	☐	☐

Design

*2-1c. The activities that my Our Town* project [uses/used] include…**	Yes	No
Artist-/designer-facilitated community planning: Artists/designers leading or partnering in the creative processes of visioning, and for solutions to community issues	☐	☐
Design of artist space: Design processes to support the creation of dedicated spaces for artists to live and/or to produce, exhibit, or sell their work	☐	☐
Design of cultural facilities: Design processes to support the creation of a dedicated building or space for creating and/or showcasing arts and culture	☐	☐
Public space design: The process of designing elements of public infrastructure, or spaces where people congregate (e.g., parks, plazas, landscapes, neighborhoods, districts, infrastructure, and artist-produced elements of streetscapes)	☐	☐

Artist and Creative Industry Support

*2-1d. The activities that my Our Town* project [uses/used] include…**	Yes	No
Creative business development: Programs or services that support entrepreneurs and businesses in the creative industries, or help cultivate strong infrastructure for establishing and developing creative businesses	☐	☐
Professional artist development: Programs or services that support artists professionally, such as through skill development or accessing markets and capital	☐	☐

2-1e. Please describe any other arts, culture, or design activities that your Our Town project [uses/used].

Creative Placemaking Strategies

2-2. Please rank the following statements in terms of how well the statement describes the intention of your Our Town project.

*My Our Town* project is best described as an initiative that [is/was] intended to…**	First	Second	Third	Fourth
Illuminate: Bring new attention to or elevate key community assets and issues, voices of residents, local history, or cultural infrastructure	☐	☐	☐	☐
Energize: Inject new or additional energy, resources, activity, people, or enthusiasm into a place, community issue, or local economy	☐	☐	☐	☐
Imagine: Envision new possibilities for a community or place—a new future, a new way of overcoming a challenge, or a new way of approaching problem-solving	☐	☐	☐	☐
Connect: Bring together communities, people, places, and economic opportunity via physical spaces or new relationships	☐	☐	☐	☐

Domain 3: Project Inputs

Please answer the following questions about your Our Town project’s inputs, or the resources you [use/used] to carry out your Our Town intervention, during the time that your project [is/was] funded by NEA.

~~4-1. How many partners [are/were] engaged in your~~ ~~Our Town~~ ~~project? By partners, we many any individual or organization that had a role in your project.~~

~~Number of partners: _____~~

~~4-2. For each partner you listed in item 2-1, please provide the name of the organization.~~

~~Name of partner organization 1:~~ ~~_____~~

~~[WEB SURVEY WILL HAVE THE SAME NUMBER OF PARTNER NAME ITEMS AS THE NUMBER OF PARTNERS THE RESPONDENT INDICATED IN 4-1.]~~

~~4-3. We would like to hear about the individuals and organizations involved in your~~ ~~Our Town~~ project and their roles. Please select the role or roles that best fit the individual or organization type involved in your project. If the individual or organization type [is/was] not involved in your project, please select “N/A” (SELECT ALL THAT APPLY).

Individuals

	*~~Individuals Involved in Your~~ ~~Our Town~~ ~~Project~~*
~~Project Roles of Individuals~~	~~The Mayor, County Commissioner, Town Manager, or Tribal Leader~~	~~Artist~~	~~Community Resident~~
~~Establish Vision of the Project~~	☐	☐	☐
~~Implement Project Activities~~	☐	☐	☐
~~Advocate for Project~~	☐	☐	☐
~~Communicate Successes of the Project~~	☐	☐	☐
~~Provide Access to Important Networks~~	☐	☐	☐
~~Fund Project~~	☐	☐	☐
~~Provide Physical Space for Project Activities~~	☐	☐	☐
~~Provide Subject Matter Expertise or Training~~	☐	☐	☐
~~Navigate Local Ordinances or Other “Red Tape”~~	☐	☐	☐
~~N/A (not involved in project)~~	☐	☐	☐
~~Other (please specify)~~

Government Agencies

	*~~Agencies Involved in Your~~ ~~Our Town~~ ~~Project~~*
~~Project Roles of Agencies~~	~~City/Town Government Agency~~	~~County Government Agency~~	~~Federal Government Agency (other than NEA)~~
~~Establish Vision of the Project~~	☐	☐	☐
~~Implement Project Activities~~	☐	☐	☐
~~Advocate for Project~~	☐	☐	☐
~~Communicate Successes of the Project~~	☐	☐	☐
~~Provide Access to Important Networks~~	☐	☐	☐
~~Fund Project~~	☐	☐	☐
~~Provide Physical Space for Project Activities~~	☐	☐	☐
~~Provide Subject Matter Expertise or Training~~	☐	☐	☐
~~Navigate Local Ordinances or Other “Red Tape”~~	☐	☐	☐
~~N/A (not involved in project)~~	☐	☐	☐
~~Other (please specify)~~

Institutions

	*~~Institutions Involved in Your~~ ~~Our Town~~ ~~Project~~*
~~Project Roles of Institutions~~	~~Private~~ ~~Foundation~~	~~School or School District~~ ~~(K–12)~~	~~College/ University~~	~~Library~~	~~Museum~~
~~Establish Vision of the Project~~	☐	☐	☐	☐	☐
~~Implement Project Activities~~	☐	☐	☐	☐	☐
~~Advocate for Project~~	☐	☐	☐	☐	☐
~~Communicate Successes of the Project~~	☐	☐	☐	☐	☐
~~Provide Access to Important Networks~~	☐	☐	☐	☐	☐
~~Fund Project~~	☐	☐	☐	☐	☐
~~Provide Physical Space for Project Activities~~	☐	☐	☐	☐	☐
~~Provide Subject Matter Expertise or Training~~	☐	☐	☐	☐	☐
~~Navigate Local Ordinances or Other “Red Tape”~~	☐	☐	☐	☐	☐
~~N/A (not involved in project)~~	☐	☐	☐	☐	☐
~~Other (please specify)~~

Organizations and For-Profit

	*~~Organizations and Companies Involved in Your~~ ~~Our Town~~ ~~Project~~*
~~Project Roles of Institutions~~	~~Nonprofit Arts Organization~~	~~Nonprofit Community Organization~~	~~Religious Organization~~	~~Media Organization~~	~~Nonprofit Organization other than those identified~~	~~For-Profit Company~~
~~Establish Vision of the Project~~	☐	☐	☐	☐	☐	☐
~~Implement Project Activities~~	☐	☐	☐	☐	☐	☐
~~Advocate for Project~~	☐	☐	☐	☐	☐	☐
~~Communicate Successes of the Project~~	☐	☐	☐	☐	☐	☐
~~Provide Access to Important Networks~~	☐	☐	☐	☐	☐	☐
~~Fund Project~~	☐	☐	☐	☐	☐	☐
~~Provide Physical Space for Project Activities~~	☐	☐	☐	☐	☐	☐
~~Provide Subject Matter Expertise or Training~~	☐	☐	☐	☐	☐	☐
~~Navigate Local Ordinances or Other “Red Tape”~~	☐	☐	☐	☐	☐	☐
~~N/A (not involved in project)~~	☐	☐	☐	☐	☐	☐
~~Other (please specify)~~

~~4-4. Please describe any other individuals or organizations involved in your project and how their role [is/was] important to the success of your~~ ~~Our Town~~ ~~project.~~

3-1. Please name up to five partners that [will have/had] a critical role in your Our Town project. Please provide the name of the partner organization rather than a specific individual unless the partner is an individual that is not affiliated with an organization (e.g., an individual artist or a community resident).

Partner 1 Name: _____

Partner 2 Name: _____

Partner 3 Name: _____

Partner 4 Name: _____

Partner 5 Name: _____

3-2. Please select the category that best describes each partner you named in item 3-1.

[NOTE: WEB SURVEY WILL POPULATE THE NAME OF EACH PARTNER IN THE COLUMNS OF THE TABLE]

	Partners
Description	[Partner 1 NAME]	[Partner 2 NAME]	[Partner 3 NAME]	[Partner 4 NAME]	[Partner 5 NAME]
The Mayor, County Commissioner, Town Manager, or Tribal Leader	☐	☐	☐	☐	☐
Artist	☐	☐	☐	☐	☐
Community Resident	☐	☐	☐	☐	☐
City/Town Government Agency	☐	☐	☐	☐	☐
County Government Agency	☐	☐	☐	☐	☐
Federal Government Agency (other than NEA)	☐	☐	☐	☐	☐
Private Foundation	☐	☐	☐	☐	☐
School or School District (K-12)	☐	☐	☐	☐	☐
College/University	☐	☐	☐	☐	☐
Library	☐	☐	☐	☐	☐
Museum	☐	☐	☐	☐	☐
Nonprofit Arts Organization	☐	☐	☐	☐	☐
Nonprofit Community Organization	☐	☐	☐	☐	☐
Religious Organization	☐	☐	☐	☐	☐
Media Organization	☐	☐	☐	☐	☐
Nonprofit Organization other than those identified	☐	☐	☐	☐	☐
For-profit Company	☐	☐	☐	☐	☐
Other (please specify)

3-3. Please select the role or roles that best fit each partner you named in item 3-1.

[NOTE: WEB SURVEY WILL POPULATE THE NAME OF EACH PARTNER IN THE COLUMNS OF THE TABLE]

	Partners
Project Roles	[Partner 1 NAME]	[Partner 2 NAME]	[Partner 3 NAME]	[Partner 4 NAME]	[Partner 5 NAME]
Establish Vision of the Project	☐	☐	☐	☐	☐
Implement Project Activities	☐	☐	☐	☐	☐
Advocate for Project	☐	☐	☐	☐	☐
Communicate Successes of the Project	☐	☐	☐	☐	☐
Provide Access to Important Networks	☐	☐	☐	☐	☐
Fund Project by Cash Donation	☐	☐	☐	☐	☐
Provide Physical Space for Project Activities	☐	☐	☐	☐	☐
Provide Subject Matter Expertise or Training	☐	☐	☐	☐	☐
Navigate Local Ordinances or Other Regulations	☐	☐	☐	☐	☐
Other (please specify)

3-4. Please estimate the number of additional partners involved in your project. By “partner” we mean any individual or organization with a role in your project.

Estimated number of additional partners: _______

Domain 4. Project Community Context

Please answer the following questions about your Our Town project’s target community. By “target community” we mean the geographic region or area where you [implement/implemented] your Our Town intervention during the time that your project [is/was] funded by NEA.

4-1. Please indicate which category best describes your perception of your Our Town project’s target community at the time your organization applied for the Our Town grant. (SELECT ONE)

☐ A region that spans multiple counties, cities, or towns

☐ A county

☐ A city or town

☐ Multiple neighborhoods within a city or town

☐ A neighborhood

☐ A Tribal reservation area

☐ Other (please specify):

4-2. Please indicate which category best describes your perception of the density of the target community of your Our Town project at the time your organization applied for the Our Town grant. (SELECT ONE)

☐ Urban

☐ Suburban

☐ Rural

☐ Mixed, the target community includes some combination of urban, suburban, and/or rural areas

☐ Other (please specify):

4-3. Please indicate which category best describes your perception of the dynamic of change in the target community of your Our Town project (1) at the time your organization applied for the Our Town grant and (2) now.

	My perception of the target community
Dynamic of Change Category	1. At the time my organization applied for the Our Town grant (SELECT ONE)	2. Now (SELECT ONE)
Displacing: Influx of population ~~and/or businesses~~ causing concerns that the current population ~~and/or business~~ would be forced to leave the target community	☐	☐
Growing: Influx of population ~~and/or businesses~~ without displacement concerns	☐	☐
Stable: No change in population ~~or businesses~~	☐	☐
Declining: Loss of population ~~and/or businesses~~	☐	☐

Domain 5. Systems Change

Please answer the following questions about the sustained impacts of your Our Town project. Please answer the following questions with as much detail as possible.

~~2-1. After the completion of your~~ ~~Our Town~~ ~~project, do you [expect to see/see] arts, culture, and design strategies incorporated in your community’s approach to community challenges? Please describe.~~

~~2-2. Is there evidence that the project [will be/has been] sustained? Please describe.~~

~~2-3. Is there evidence that partnerships [will be/have been] sustained? Please describe.~~

5-1. Please describe any evidence you [expect to have/have] that others [will be/have been] replicating a model established by this project.

~~2-5. [Do you expect your project to produce/did your project produce] change that will last a long time? Please describe.~~

5-2. Please describe the evidence that you [expect to have/have] that tells you your project will produce change that will last beyond the Our Town grant period. For example, your project may [forge/have forged] partnerships that increase the capacity for work beyond the project; utilize arts, culture, and design to address community challenges in a way that is scalable or that can be replicated by others; or [create/have created] long-term funding streams, policies, and practices that have become organizational norms. These are just examples. We are interested in what you think is relevant.

5-3a. Please indicate your level of agreement with each of the following statements regarding how the following laws, policies, or regulations [impact/impacted] the implementation of your Our Town project’s activities in the target community. Please select “N/A” if the law, policy, or regulation did not exist at the time your organization applied for the Our Town grant.

*At the time my organization applied for the Our Town* grant…**	Strongly Agree	Agree	Neutral	Disagree	Strongly Disagree	N/A	Specific Examples or Description (optional)
Local development regulations existed that [enable/enabled] implementation of my project’s arts, culture, and design activities	☐	☐	☐	☐	☐	☐
Local funding or incentives existed that [enable/enabled] implementation of my project’s arts, culture, and design activities	☐	☐	☐	☐	☐	☐
Local public space regulations existed that [enable/enabled] implementation of my project’s arts, culture, and design activities	☐	☐	☐	☐	☐	☐
Local zoning ordinances and/or property codes, including fire code, existed that [enable/enabled] implementation of my project’s arts, culture, and design activities	☐	☐	☐	☐	☐	☐
Planning and design activities involving resident participation existed that [enable/enabled] implementation of my project’s arts, culture, and design activities	☐	☐	☐	☐	☐	☐
Social policies benefiting or giving voice to underrepresented constituencies in the target community existed that [enable/enabled] implementation of my project’s arts, culture, and design activities	☐	☐	☐	☐	☐	☐
Tax laws existed that [enable/enabled] implementation of my project’s arts, culture, and design activities	☐	☐	☐	☐	☐	☐
Transportation policies existed that [enable/enabled] implementation of my project’s arts, culture, and design activities	☐	☐	☐	☐	☐	☐
Other laws, policies, or regulations existed that [enable/enabled] implementation of my project’s arts, culture, and design activities	☐	☐	☐	☐	☐	☐
Other-Please specify:

5-3b. Please indicate your level of agreement with each of the following statements regarding the evidence you have now that your Our Town project’s activities [contribute/contributed] to improvements in the following laws, policies, or regulations that can enable future arts, cultural, and design activities in the target community. Please select “N/A” if policy did not exist in the grantee’s project target community.

*I have evidence now that my Our Town* project’s activities [contribute/contributed] to improvements in…**	Strongly Agree	Agree	Neutral	Disagree	Strongly Disagree	N/A	Specific examples or description (optional)
Local development regulations that can enable future arts, cultural, and design activities	☐	☐	☐	☐	☐	☐
Local funding or incentives that can enable future arts, cultural, and design activities	☐	☐	☐	☐	☐	☐
Local public space regulations that can enable future arts, cultural, and design activities	☐	☐	☐	☐	☐	☐
Local zoning ordinances and/or property codes, including fire code, that can enable future arts, cultural, and design activities	☐	☐	☐	☐	☐	☐
Planning and design activities involving resident participation that can enable future arts, cultural, and design activities	☐	☐	☐	☐	☐	☐
Social policies benefiting or giving voice to underrepresented constituencies in the target community that can enable future arts, cultural, and design activities	☐	☐	☐	☐	☐	☐
Tax laws that can enable future arts, cultural, and design activities	☐	☐	☐	☐	☐	☐
Transportation policies that can enable future arts, cultural, and design activities	☐	☐	☐	☐	☐	☐
Other laws, policies, or regulations that can enable future arts, cultural, and design activities	☐	☐	☐	☐	☐	☐
Other-Please specify:

Domain 6: Local Community Change

Please answer the following questions about the impacts of the Our Town activities you [implement/implemented] on the target community’s economic, physical, and/or social environments. By “target community” we mean the geographic region or area where you [implement/implemented] your Our Town intervention during the time that your project [is/was] funded by NEA.

6-1. Have you completed most of your project activities?

☐ Yes

☐ No [WEB SURVEY WILL SKIP 6-1d, 6-2d, and 6-3d]

[NOTE: THE WEB SURVEY RANDOMIZES THE ORDER OF SOCIAL, ECONOMIC AND PHYSICAL CHANGE]

Social Change

The items in this section are all social changes, which include improvements to social relationships, civic engagement and community empowerment, and/or amplifying community identity.

6-1a. [Are/Were] your Our Town project’s activities designed to contribute to social changes?

☐ Yes

☐ No [WEB SURVEY WILL SKIP TO 6-2b.]

6-1b. Please indicate your level of agreement with the following statements regarding the social changes your Our Town project’s activities [are/were] designed to contribute to.

*My Our Town* project activities [are/were] designed to contribute to…**	Yes	No
Civic Engagement: Increases in active participation in efforts that impact public life and the community, not only one’s own self-interests.	☐	☐
Social Cohesion: Increases in the level of working trust and mutual support within a community that facilitates collective action	☐	☐
Community Attachment: Increases in pride, interest, and satisfaction with the physical and social/cultural aspects of one’s neighborhood	☐	☐
Other Social Change	☐	☐
Other-Please specify:

[WEB SURVEY SKIPS TO 6-2 IF RESPONDENT DID NOT ANSWER “YES” TO ANY OF THE SOCIAL CHANGE ITEMS IN 6-1b, EXCLUDING THE “OTHER” CATEGORY.]

[NOTE: Removed N/A response option] 6-1c. You indicated “Yes” in 6-1b that your Our Town project activities [are/were] designed to lead to [OUTCOMES RESPONDENT SELECTED “YES” WILL APPEAR HERE.]. Please indicate your perception of the need for the following specific forms of these social changes in the target community at the time your organization applied for the Our Town grant.

[NOTE: RESPONDENT ONLY SEES THE INDICATORS THAT CORRESPOND TO AN OUTCOME THEY SELECTED “YES” FOR IN 6-1b.]

	*Degree of need in the target community at the time your organization applied for the Our Town* grant**
Specific Social Changes	Strong Need	Moderate Need	Mild Need	No Need
Civic Engagement: People actively participating in community institutions (e.g., schools, neighborhood meetings, public hearings, school boards, civic activities)	☐	☐	☐	☐
Civic Engagement: Individuals or community organizations partnering to address community issues	☐	☐	☐	☐
Social Cohesion: Trust between residents in the community	☐	☐	☐	☐
Social Cohesion: Support systems for community organizing and other collective action of people and organizations in the community	☐	☐	☐	☐
Social Cohesion: Residents that imagine positive options for their future	☐	☐	☐	☐
Social Cohesion: Residents that feel they have the skills and confidence to generate opportunities for themselves	☐	☐	☐	☐
Community Attachment: Residents that feel welcome in the community	☐	☐	☐	☐
Community Attachment: Residents that feel like they belong in the community	☐	☐	☐	☐
Community Attachment: Residents that value and talk about rich traditions and/or histories in the community	☐	☐	☐	☐
Community Attachment: Residents that say they wish to remain in the community for a number of years	☐	☐	☐	☐
Other Social Change	☐	☐	☐	☐
Other-Please specify

[NOTE: Removed N/A response option] 6-1d. You indicated “Yes” in 6-1b that your Our Town project activities [are/were] designed to lead to [OUTCOMES RESPONDENT SELECTED “YES” WILL APPEAR HERE.]. Please indicate your perception of the degree to which your Our Town project’s activities [impact/impacted] the following specific forms of these social changes in the target community based on the evidence you have now.

Please note, we do not expect projects to affect a large number of indicators.

	Degree of impact of your project’s activities on the change based on the evidence you have now
Specific Social Changes	Strong Impact	Moderate Impact	Mild Impact	No Impact
Civic Engagement: Increases in the number of people actively participating in community institutions (e.g., schools, neighborhood meetings, public hearings, school boards, civic activities)	☐	☐	☐	☐
Civic Engagement: Increases in the number of community leaders and other individuals actively working to improve the community	☐	☐	☐	☐
Civic Engagement: Increases in the number of individuals or community organizations partnering to address community issues	☐	☐	☐	☐
Social Cohesion: Increases in the level of trust between residents in the community	☐	☐	☐	☐
Social Cohesion: Improvements to support systems for community organizing and other collective action of people and organizations in the community	☐	☐	☐	☐
Social Cohesion: Increases in the number of residents that imagine positive options for their future	☐	☐	☐	☐
Social Cohesion/: Increases in the number of residents that feel they have the skills and confidence to generate opportunities for themselves	☐	☐	☐	☐
Community Attachment: Increases in the number of residents that feel welcome in the community	☐	☐	☐	☐
Community Attachment: Increases in the number of residents that feel like they belong in the community	☐	☐	☐	☐
Community Attachment: Increases in the number of residents that value and talk about rich traditions and/or histories in the community	☐	☐	☐	☐
Community Attachment: Increases in the number of residents that say they wish to remain in the community for a number of years	☐	☐	☐	☐
Other Social Change	☐	☐	☐	☐
Other-Please specify

Economic Change

The items in this section are about economic changes, which include economic improvements of individuals, institutions, or the community. By economic changes, we mean growth or strengthening of the local labor force and businesses as well as local tourism.

6-2a. [Are/Were] your Our Town project’s activities designed to contribute to economic changes?

☐ Yes

☐ No [WEB SURVEY WILL SKIP TO 6-3.]

6-2b. Please indicate ‘Yes’ or ‘No’ with the following statements regarding the economic changes your Our Town project’s activities [are/were] designed to contribute to.

*My Our Town* project’s activities [are/were] designed to contribute to…**	Yes	No
Local Business Growth: Increases in business activity and business diversity in the community	☐	☐
Job Creation/Labor Force Participation: Increases in the number and type of jobs available to community residents	☐	☐
Professional Development/Training: Increases in artists/residents’ knowledge, skills, and employability	☐	☐
Preventing Displacement: Decreases in the displacement of low-income residents that desire to remain in the neighborhood/area	☐	☐
In-Migration: Increases in residents in the community	☐	☐
Tourism: Increases in visitors to the community	☐	☐
Other Economic Change	☐	☐
Other-Please specify:

[WEB SURVEY SKIPS TO 6-3 IF RESPONDENT DID NOT ANSWER “YES” TO ANY OF THE ECONOMIC CHANGE ITEMS IN 6-2b, EXCLUDING THE “OTHER” CATEGORY.]

[NOTE: Removed N/A response option] 6-2c. You indicated “Yes” in 6-2b that your Our Town project activities [are/were] designed to lead to [OUTCOMES RESPONDENT SELECTED “YES” WILL APPEAR HERE.]. Please indicate your perception of the need for the following specific forms of these economic changes in the target community at the time your organization applied for the Our Town grant.

[NOTE: RESPONDENT ONLY SEES THE INDICATORS THAT CORRESPOND TO AN OUTCOME THEY SELECTED “Yes” FOR IN 6-2b.]

	*Degree of need in the target community at the time your organization applied for the Our Town* grant**
Specific Economic Changes	Strong Need	Moderate Need	Mild Need	No Need
Local Business Growth: New (recently opened) local businesses	☐	☐	☐	☐
Local Business Growth: Resident spending at local businesses	☐	☐	☐	☐
Local Business Growth: Positive resident, visitor, and/or employer perceptions of the local business climate	☐	☐	☐	☐
Job Creation/Labor Force Participation: Employment opportunities in the community	☐	☐	☐	☐
Job Creation/Labor Force Participation: Internship opportunities in the community	☐	☐	☐	☐
Job Creation/Labor Force Participation: Employment at my Our Town project’s partner organizations	☐	☐	☐	☐
Professional Development/Training: Professional networks of artists or residents in the community	☐	☐	☐	☐
Professional Development/Training: Artists or residents with strong professional skills in the community	☐	☐	☐	☐
Preventing Displacement: Affordable and/or mixed-income housing policies in the community	☐	☐	☐	☐
Preventing Displacement: Organizations in the community that serve different demographic groups (race/ethnicity/low income/children/families)	☐	☐	☐	☐
In-Migration: High-quality public space and community facilities that can attract new residents	☐	☐	☐	☐
In-Migration: People living and staying in the community	☐	☐	☐	☐
Tourism: People visiting the community more than once in a year	☐	☐	☐	☐
Tourism: People visiting multiple destinations when they come to the community	☐	☐	☐	☐
Other Economic Change	☐	☐	☐	☐
Other-Please specify:

[NOTE: Removed N/A response option] 6-2d. You indicated “Yes” in 6-2b that your Our Town project activities [are/were] designed to lead to [OUTCOMES RESPONDENT SELECTED “YES” WILL APPEAR HERE.]. Please indicate your perception of the degree to which your Our Town project’s activities [impact/impacted] the following specific forms of these economic changes in the target community based on the evidence you have now.

Please note, we do not expect projects to affect a large number of indicators.

[NOTE: RESPONDENT ONLY SEES THE INDICATORS THAT CORRESPOND TO AN OUTCOME THEY SELECTED “Yes” FOR IN 6-2b.]

	Degree of impact of your project’s activities on the change based on the evidence you have now
Specific Economic Changes	Strong Impact	Moderate Impact	Mild Impact	No Impact
Local Business Growth: Increases in the number of new (recently opened) local businesses	☐	☐	☐	☐
Local Business Growth: Increases in resident spending at local businesses	☐	☐	☐	☐
Local Business Growth: Increases in resident, visitor, and/or employer perceptions of the local business climate	☐	☐	☐	☐
Job Creation/Labor Force Participation: Increases in employment opportunities in the community	☐	☐	☐	☐
Job Creation/Labor Force Participation: Increases in internship opportunities in the community	☐	☐	☐	☐
Job Creation/Labor Force Participation: Increases in employment at my Our Town project’s partner organizations	☐	☐	☐	☐
Professional Development/Training: Increases in the professional networks of artists or residents in the community	☐	☐	☐	☐
Professional Development/Training: Increases in the professional skills of artists or residents in the community	☐	☐	☐	☐
Preventing Displacement: Improvements to affordable and/or mixed-income housing policies in the community	☐	☐	☐	☐
Preventing Displacement: Increases in organizations in the community that serve different demographic groups (race/ethnicity/low-income/children/families)	☐	☐	☐	☐
In-Migration: Increases in high-quality public space and community facilities that can attract new residents	☐	☐	☐	☐
In-Migration: Increases in the number of people living and staying in the community	☐	☐	☐	☐
Tourism: Increases in the number of people visiting the community more than once in a year	☐	☐	☐	☐
Tourism: Increases in the number of people visiting multiple destinations when they come to the community	☐	☐	☐	☐
Other Economic Change	☐	☐	☐	☐
Other-Please specify

Physical Change

The items in this section are all physical changes, which are improvements that occur to the built and natural environment.

6-3a. [Are/Were] your Our Town project’s activities designed to contribute to physical changes?

☐ Yes

☐ No [WEB SURVEY WILL SKIP TO END OF SURVEY]

6-3b. Please indicate your level of agreement with the following statements regarding the physical changes your Our Town project’s activities [are/were] designed to contribute to.

*My Our Town* project’s activities [are/were] designed to contribute to…**	Yes	No
Beautification and/or Enhancement of the Physical Environment: Physical improvements to the community that improve its aesthetic appearance	☐	☐
New Construction and Redevelopment: Physical improvements to the community that address decline and improve infrastructure	☐	☐
Other Physical Change	☐	☐
Other-Please specify:

[WEB SURVEY SKIPS TO END OF SURVEY IF RESPONDENT DID NOT ANSWER “YES” TO ANY OF THE PHYSICAL CHANGE ITEMS IN 6-3b, EXCLUDING THE “OTHER” CATEOGRY.]

[NOTE: Removed N/A response option] 6-3c. You indicated “Yes” in 6-3b that your Our Town project activities [are/were] designed to lead to [OUTCOMES RESPONDENT SELECTED “YES” WILL APPEAR HERE.]. Please indicate your perception of the need for the following specific forms of these physical changes in the target community at the time your organization applied for the Our Town grant.

[NOTE: RESPONDENT ONLY SEES THE INDICATORS THAT CORRESPOND TO AN OUTCOME THEY SELECTED “YES” FOR IN 6-3b.]

	*Degree of need for the change in the target community at the time your organization applied for the Our Town* grant**
Specific Physical Changes	Strong Need	Moderate Need	Mild Need	No Need
Beautification and/or Enhancement of the Physical Environment: Improvements to storefronts, sidewalks, street plantings, lighting, and/or building conditions in the community	☐	☐	☐	☐
Beautification and/or Enhancement of the Physical Environment: Improvements to the perception of cleanliness in the community	☐	☐	☐	☐
Beautification and/or Enhancement of the Physical Environment: Support for public spaces in the community that are free from traffic, crime, and noise	☐	☐	☐	☐
New Construction and Redevelopment: Support for community anchors such as cultural facilities, public space, or artist space	☐	☐	☐	☐
New Construction and Redevelopment: Accessibility to public transportation	☐	☐	☐	☐
New Construction and Redevelopment: Community accessibility and safety for pedestrians and cyclists	☐	☐	☐	☐
New Construction and Redevelopment: Repurposing or revitalizing abandoned buildings or other blighted features of the community	☐	☐	☐	☐
Other Physical Change	☐	☐	☐	☐
Other-Please specify

[NOTE: Removed N/A response option] 6-3d. You indicated “Yes” in 6-3b that your Our Town project activities [are/were] designed to lead to [OUTCOMES RESPONDENT SELECTED “YES” WILL APPEAR HERE.]. Please indicate your perception of the degree to which your Our Town project’s activities [impact/impacted] the following specific forms of these physical changes in the target community based on the evidence you have now.

Please note, we do not expect projects to affect a large number of indicators.

[NOTE: RESPONDENT ONLY SEES THE INDICATORS THAT CORRESPOND TO AN OUTCOME THEY SELECTED “YES” FOR IN 6-3b.]

	Degree of impact of your project’s activities on the change based on the evidence you have now
Specific Physical Changes	Strong Impact	Moderate Impact	Mild Impact	No Impact
Beautification and/or Enhancement of the Physical Environment: Improvements to storefronts, sidewalks, street plantings, lighting, and/or building conditions in the community	☐	☐	☐	☐
Beautification and/or Enhancement of the Physical Environment: Improvements to the perception of cleanliness in the community	☐	☐	☐	☐
Beautification and/or Enhancement of the Physical Environment: Support for public spaces in the community that are free from traffic, crime, and noise	☐	☐	☐	☐
New Construction and Redevelopment: Support for community anchors such as cultural facilities, public space, or artist space	☐	☐	☐	☐
New Construction and Redevelopment: Increases in accessibility to public transportation	☐	☐	☐	☐
New Construction and Redevelopment: Increases in community accessibility and safety for pedestrians and cyclists	☐	☐	☐	☐
New Construction and Redevelopment: Repurposing or revitalizing abandoned buildings or other blighted features of the community	☐	☐	☐	☐
Other Physical Change	☐	☐	☐	☐
Other-Please specify

Thank you for your participation. Your survey is now complete.

Appendix: Detailed Respondent Comments¹

Survey Question	Respondent 1	Respondent 2	Respondent 3	Respondent 4	Respondent 5	Respondent 6	Respondent 7
1-1a.		ok	Readable, clarity, useful for our situation and programmatic intentions	'Sounds awkward, perhaps should read: "project activities" without apostrophe.	This is clear. It is good that you used the same term as in the grant guidelines (economic change is term used by NEA in grant description)	The planning phase of Turn to the River was not intended to have direct economic impact, though the overall project, once implemented, is. So I am reporting on the planning phase only, which received an Our Town grant. Everything is very readable, clear, but I am wondering if you might want to have an opportunity for an addendum such as my sentence above – since in the long run it will lead to economic development	Our Town grants are made to, " integrate arts, culture, and design activities into efforts that strengthen communities by advancing local economic, physical, and/or social outcomes." So, economic changes should be an outcome, perhaps the question should ask how did arts and culture activities contribute to economic changes or how were those changes tracked, evaluated, etc.?
1-1b.	I am confused by the N/A vs disagree. Projects were either designed to do it or not designed to do it. I was confused on how to answer for my grant. For example, I put NA b/c my grant was not designed to increase job creation so shouldn't I answer disagree instead of N/A?	'Preventing displacement, not everyone lives in a conventional neighborhood, particularly rural residents. Could say "neighborhood/area" or something similar	Good breakdown of possible impacts; specifying "Other Economic Change" it now occurs to me that these topics imply buy, hire and inspire local changes, but those options be included	"In-Migration" is not a layperson's term "Tourism: Increases in visitors to [AND INTEREST?] in the community Does business diversity refer to types of businesses or businesses operated by people from diverse backgrounds? Increases in residents may be difficult to measure during an OT grant period.	This is a good list as it covers main areas for economic advancement. Some activities reach two or more goals, such as cultural asset management which can be used to help prevent displacement and also cultural tourism. One area you could add, which is a topic in the field has to do with reaching targeted groups to provide equity and access. Most projects I imagine cannot actually create permanent jobs, but through training and internships can increase diversity and opportunity. A question could be "Diversity Increases diversity in employment to reach targeted groups."		When reading this question, I think some of these "contributions" will happen during planning and implementation and some happen after the fact, perhaps years after, so they may be difficult to track right after the project is completed.
1-1c.	Good.	'ok	Excellent. I see the usefulness of the transitions between each section in helping grantees reflect on their projects, goals, objectives and effectiveness.		This is quite thorough. I do like how you have broken down each area into more specific components. You could add a question about education involving targeting underserved groups. Such as: Education "Increases opportunity for targeted underserved groups"		I find this page difficult because these changes will be challenging to track. Does the NEA provide a checklist to the grant applicants, or survey templates to create before and after baseline comparisons of data? How can you track perceptions? A community may be broader than the geographic area where a project takes place.
1-1d.	I would like to see a don't know or unsure rather than a N/A	It might be nice to have the respondent indicate how far into the process/what stage of the process they're in. It might help to show why there is "no impact" for certain projects.	I feel like there are so many issues and details behind these forms of economic changes that must be addressed and discussed to arrive at an understanding of the dynamics that foster effective and enduring creative placemaking. That said, readability, clarity and usefulness are okay, but leave lots of key questions unexplored. I guess it's the how do we get to these places of systemic change. That is what our organization and community is dealing with on a daily basis.	Grantees may need to prompted to select these impacts upon application so they know to evaluate these quantifiable measures.	This is very clear. Our project is designed to help prevent displacement by providing low cost studio space (primary goal is arts education and professional development). As the answers here did not specifically address this activity, I included in the "Other" box		I think the way this question is worded makes it easier to answer than the previous question.
1-2a.	Does physical changes include beautification like murals? If so, then question is unclear.	This is confusing for me. Our activities are all planning - related at this point. But that plan will contribute to future physical changes. Should I answer yes to this question?	Fine		This is very straightforward
1-2b.			It is challenging to answer these generalized questions without feeling frustrated at the lack of opportunity to delve into the conversation that the topics beg us to have. I would like for this survey to be taken by our board of directors and key stakeholders in our community. Most likely they would not agree with some or perhaps all of my perceptions, and that is one of the challenges faced by practitioners of creative placemaking. Seeing the connections and envisioning where processes will take us in the public realm. Possibly influencing the private and personal for some people.		Again, this is clear. My only comment would that you could add another line specifying whether the project resulted in plans, or actual construction, or both. An additional question could be "Plans and designs: to develop a strategic plan for an arts facility." As NEA funds cannot be used for capital improvements, many projects result in plans. This would also clarify that the project did or did not include any new construction or improvements that were done, such as streetscape improvements.	Again, our project was a plan, not intended to impact physical changes until implemented. Everything is very readable, clear, but I am wondering if you might want to have an opportunity for an addendum such as my sentence above – since in the long run it will lead to physical changes
1-2c.		'It's a little distracting that the primary physical change is bolded on each line. It would be preferable to display the main type of physical change first then have sub-headings, or possible omit altogether. I'm not sure that they assist the respondent in any way...I realize that things can be finicky in terms of how they are displayed.	Okay		This is thorough. One area to ask, however, could be concerning the purpose of the development. Is the development designed to increase the arts infrastructure of a community? Or is it to retain a "sense of place" in a rapidly changing area. These two purposes are different – some projects are designed to provide access and equity to areas where there are limited arts organizations. For example, for our project, the purpose of creating an arts incubator is to help develop cultural facilities and opportunities in an area where there are much fewer arts and cultural spaces than other neighborhoods such as Downtown LA. The other purpose for some projects is to retain a sense of place. For example, our first Our Town grant took place in Watts. The idea was to redevelop an existing historic train station to design a visitors center to celebrate and preserve the history of the community. Getting at the purpose for the redevelopment in terms of community identity and to ensure existing residents maintain a sense of belonging in their communities would be interesting. As well as how many projects are designed to change the community to increase the number of cultural sites in a community. It may also be interesting to ask specifically for the general type of development. Is the purpose of the development arts education, or services to artists, or performing arts such as a theatre?		I think it would be helpful to supply grant applicants with these questions prior to executing their projects
1-2d.			I find these forms of physical changes lacking in the reality of nuanced interactions that occur to bring them about. To me, that is the important part of the conversation that must be dissected and explored collaboratively		These questions are all pertinent and important. One area not mentioned, however, involves funding capital projects. (although I see it is touched on later in the survey as to which type of entities helped provide funding). As NEA funds don't cover construction, strong public / private partnership are needed as you know
1-3a.	You may want to give examples of social changes like increased interaction, mover overlap between communities, opportunities to create art together, expand networks etc.	Ok	Fine: it looks like I was getting ahead of myself. At this point, I would have had an easier time answering the previous questions if the social part came first. My perception is that this is the underpinning topic for evaluation that everything else rests upon.		This is clear	I like your offering a broad range of options that one would include in thinking about this question: social relationships, civic engagement and community empowerment, and/or amplifying community identity. It is useful, clear and readable.
1-3b.	Good.	Ok	I feel that the phrase "not only one's own self –interest a bit too judgmental. In terms of creative placemaking, sustainability and risk taking/resilience, our society would do well to gain insight into the relationship between self- interest and cultural/social/ economic survival of the whole. We are moving away from dualistic thinking and that phrase reinforces the us-vs them mentality			I like the range of options and think everything is readable, clear and useful. I added another possible option that might address the overall sense that things are possible – beyond building trust, pride and interest, the impact of this type of project has changed the belief system among many here - I.e that things are possible that they thought impossible. Not sure how to word it or what to call it, but it’s a slightly different outcome that you might be able to articulate in a way that will resonate within many communities.	Is the NEA interested in tracking the social changes that happen in the process of building a project, or only after the project is completed, should there be a question about equity in the process?
1-3c.	Good	Again, not sure that the bolded text will be helpful for a respondent. For me, it's just distracting – maybe it could be placed after the actual text that guides us how to respond	Becoming familiar with Community Capitals through the work and writings of Flora and Flora, I've become more appreciative of the challenges shared by rural and urban communities. The difference is that with small communities there is much more familiarity and impact from social unrest, so for example, while the above social changes were all present to a great degree in our community. I indicated in my choices that there was a substantial need at the time of applying for the Our Town grant, because of the intense blaming, criticizing, and siloing entrenched in our fabric. My decade of work has been devoted to implementing public projects and processes in collaboration and partnerships that would gradually transition the impassioned expressiveness toward positive engagement. It takes many people working on this diligently over time and that is, I feel, the case in our community		Again these questions are very thorough. You could add a few areas: Under collective efficacy, you could ask about increases in after-school programs for families. Our project involves creating an outdoor sculpture museum along with public programming to bring people together. Pacoima is largely Latino. Other urban communities in LA have seen identity changes due to local migration, so the question about level of trust is important. You could add a question specifically about the existing arts and cultural infrastructure.	Breakdown into multiple details is thought provoking and useful
1-3d.	Again - need an unsure category rather than N/A		It is difficult to indicate an accurate level of impact. In reality less people are going to public meetings because they are not generally complaining and blaming as much – we now have a skilled borough manager who is a proponent of creative placemaking. We also have a professional planner of similar skill and she is getting an assistant this year. Systemic changes are necessary in order to shift social dynamics. So rather than more people going to meetings, for the first time in many years young people are beginning to step up to the plate to serve on boards, commissions and run for public office. Nonprofit service organizations are seeing more youthful leadership, and young people are starting businesses. Some even moving here from other places to start businesses. The numbers in each of these instances are small. We are a community of 2,500 people. A handful of people can make a substantial difference in community dynamics (this goes both ways...positive and negative). I would like to think that our creative placemaking strategies have played a part in these changes, small though it may be.	Grantees might need to be provided standardized tools to measure these kinds of changes in attitudes and engagement. Applicants will be challenged to demonstrate these changes without more framework.	One additional question or area could ask about how the project increased the agency of the marginalized communities. It could be "Community Attachment: Increases opportunities for targeted groups to participate in municipal community development and planning projects." The community organization question, however, may get at this.	Again there is a discrepancy between our plan and what has happened since, in other phases of our project – I wonder if you might want to include a question so survey takers can let you know if their project is in a particular phase
2-1.	This is actually a yes or no question and should read "How do you see arts, culture..." or How has arts culture etc been incorporated?		This section is clear and useful	These changes might be hard to accurately reflect upon in a grant period.	You may wish to say "in which ways" so as to not get yes/no answers	Because our project has been ongoing in different phases since the initial Our Town project, it is hard to distinguish exactly what came out of that phase, so you might want to also include a question about whether this initial funded project led to continuous involvement and efforts that have yielded additional results.	Should this question ask how arts and culture were used to influence design strategies, project outcomes, and community challenges? Or what was the most successful strategy you used in this project that is replicable and can be implemented in the future?
2-2.	This is a yes/no question and unlikely to get the kind of information you want. Perhaps something like Describe how the project has or has not been sustained.	Sometimes, it's not the project that will be sustained, but the goals and objectives that are identified as a result of the project – but I get what was being asked here	Good question. It fits any condition/status and welcomes reflection on future outcomes	What does sustained mean? Beyond the grant period?	This is good	It is clearly written, but I'm not certain exactly what you want to learn from this.	Should this question clarify what is being sustained either social, physical, or economic?
2-3.	Same comment as last time. What partnerships - if any - have been sustained?		Open-ended questions might elicit more description by providing a little more parenthetical guidance.	'Difficult and broad question to answer. Maybe this needs more examples to say yes or no to.	To me this is clear. Some people may have different ideas about what this means. For example, it could mean that the partnership is continuing, or that the City or other partner have shifted to a more advisory role. Although clear to me, you could denote "evidence," as in is there continued programming as a result, new programming, or if the project is complete, what sustained impact is anticipated.	Clear, readable, useful
2-4.	Maybe add, if so, please describe.	'Might be better to ask if there are aspects of the project/model that can be replicated in other communities, etc	Fine	Who might the others be?	This is straightforward	Good question, clear	What are the model practices used in this project that will be replicated in your community?
2-5.	Very speculative question. Perhaps something along the lines of Do you see evidence such as strengthened partnerships, investment of additional resources from other parties etc etc that indicate that your project may have lasting impact?	Ok	Important question. Clear and useful.	This question seems strangely worded.	This is clear	Good question, clear	Should this question be more direct in terms of the length of time the project will create an impact? And should it break down social, physical and economic?
2-6.			This is a difficult question to answer. It is helpful to have examples. One cannot always be certain what action causes change. The public realm is a highly complex system of interactions and forces		I like how you have examples here. These questions will pick up threads	It’s a little confusing – takes a few times reading it. Could you break this one out into bullet points as you did for the others, and then leave room for people to add their own as well?
2-7a.		This set of questions is kind of confusing. Some of these things exist but don't generally move the needle in one way or another. "enabling" feels like the policy or law should actually help rather than not hinder…	Okay. Difficult to give examples as Alaska governments are short on relevant policies and codes	This prompt is long and difficult to read: "Please indicate your level of agreement with each of the following statements regarding how the following laws, policies, or regulations impacted the implementation of your Our Town project’s activities in the target community."	This is a good list. It includes specifics that connect to Our Town goals	Clear
2-7b.	I wouldn't mind a mid-point neutral option here. And also a don't know rather than N/A. For some of the question it is NA, for others, I don't know.	'Check grammar on instruction in the grey bar. Missing an 'on'	It's really too early in our project's evolution to see changes in policy, laws and regulations		Again, this is a good list that gets at specific outcomes. I am curious to know if any Our Town projects were able to effect tax laws or actual transportation policy. One area you may want to define is what is meant by some of these terms. For example, changes in transportation policies could be construed as informal policy changes, such as "art buses" as opposed to actual written municipal policy.
3-1.	Fine	Ok	Would suggest category: A county or borough		This is clear	Seems very clear and direct
3-2.		Ok	Fine		Good	Clear	Do the options need to be defined with actual population numbers?
3-3.			It would be clearer to have two separate categories: Growing population Growing businesses not and/or. We have stable population and growing businesses at this point	These changes would likely never be measurable by grantees during an OT grant period	This is interesting. You may want to put "choose one" or which categories...as people tend to read quickly and may try to click more than one
4-1.	I wonder if a scale of like 1 - 5, 6 - 10, 11 - 15 would be useful because I don't remember the exact number of partners and I am sure I am not the only one! Returning from successive page - maybe note that you will be asked to list partners on next page	Why is there an underscore mark up there?	Not sure why "Number of partners:" is at the lower left here. Defining critical partners is difficult in a project that centers on community engagement. The two main ones are the borough and the Chilkoot Indian Association. A few key local businesses are also critical to the success of our project	The NEA tends to consider partners organizations, not individuals. This question may result in inflated answers counting every artist or consultant as a participant/partner.	Not sure if you want to make distinction between main project partners and additional project partners from the community; as well as if partners are arts organizations, government, or a social service field. If it helps, you could also do at time of app and after, as often additional partners are picked up during implementation	I am not sure how to answer this project. For example we had the City as a partner, but within the city there were multiple departments and they addressed very different aspects and so would they count as multiple partners or one partner? Same with Universities. I gave a number that does not include volunteer citizens – was not sure if you were looking for entities / organizations or what. I think this question is not clear. Maybe you could either provide categories or clarify what you are looking for exactly. I can count partnerships in a number of ways. For example some entities were ongoing partners throughout the whole 2 years of the grant funded project. Others held events in conjunction and were just involved for a short time.	Who are the partners that are critical to success would be more telling? Our include: Town of Jackson, Jackson Community Pathways, National Elk Refuge, US Fish and Wildlife Service, WyDOT, local businesses, and residents.
4-2			4.1 and 4.1 would be clearer if the dangling "Number of partners:" was a fill-in rather than in the request for feedback		You could, if it helps get at useful information, also ask type of agency	It is confusing – there are only 10 slots, but room for any number to be listed as partners in the previous page. Again, not sure of what you define as a partner	Maybe list this with the previous question
4-3a.	My community partners played a big role in all of these things and it may be helpful to include a 4^th column for partner	Project has all of these individuals on its advisory team	N/A and Other are a bit confusing		Not sure if you specifically want to see specific involvement of Mayor or highest ranking official. One area not noted is whether an LAA Director is directly involved. Larger urban areas, such as LA, would normally see the Mayor supporting projects, but day to day implementation to the LAA. This could also help tell how involved the local arts organization involved was	There are many other individuals that do not fit in this category, such as other city leaders or employees besides the mayor; other non-profit leaders, economic development entities, media members, etc	I think this is confusing, perhaps there should be a column for the project leader
4-3b.			Our town of Haines and borough are consolidated. The project is in the townsite and in the borough		As all Our Town grants require a partnership, I imagine most answers will be either City or County. A question somewhere could ask if the NEA grant was leveraged to provide private foundation or new public funding for capital improvements. It would also be interesting to see if the project did result in plans, what is the expectation for development, id the project anticipated to be developed in the next 5 – 10 years, 10 – 15 years, etc. A question could also be asked if the project resulted specifically in new partnerships, and if so, what kind – programmatic partners or funding partners	Pretty clear
4-3c.	Should other non-profit institutions have a role here too? Again, I had numerous non-profit community partners that are institutions		All other institutions are part of the community engagement fabric and are primarily supportive		I imagine there may be some overlap between what each partner does in these questions. Not sure if you wish to ascertain what roles each plays.	Seems clear
4-3d.			This question is somewhat problematic for our project – determining roles of entities that are primarily supportive		You may wish to add a column for local arts agencies. Our Town partnerships sometimes, as in our case, have the City's local arts agency as the primary partner, and a non-arts nonprofit partner. In many cases, a nonprofit arts organization may be an ancillary partner, but not a primary partner. Also, you could add a column for nonprofit social service organization. If the government partner is the arts agency, a non-arts nonprofit may be utilized	Seems clear and readable
4-4.	N/A	N/A	It is sometimes difficult to separate individuals, institutions and roles	This question seems to duplicate the roles that grantees outline in their grant proposals	This is good to allow people to provide additional information	I was a little unclear about this, but it did generate a quick response, but I'm not sure it is what you were looking for
5-1a-d.	Good		Clear		This is a good way of breaking down types of activities. I like it!	It is clear, etc. Our project encouraged a group of local artists to form and create a public art project. While not a part of our project, it seemed to be in direct response to our challenge of reconnecting downtown with the river.
5-1e.	No suggestions		Clear	N/A	You could also ask for any other categories of support that orgs may wish to add	Seems clear
5-2.	Good		I hope it's fine to check all		This is good, but I imagine most people will say their projects hit these areas in some way	Good ways to describe what happens from Our Town projects
Additional Comments		This survey could use a progress bar. I wonder if it could be split up into different parts, it's really long, and I question how many people will get all the way through it. It was really tough for me to get through	I'm wondering if it is possible to have a copy of the survey I filled out with the questions and responses. I would like to show this to our board and key stakeholders			Crystal clear

1 The survey question numbers in this table reflect the version of the survey used for cognitive testing, which used a different order for the survey domains than the revised survey above. In the cognitive testing version of the survey the domains were Domain 1 – Local Community Change (Domain 6 in the revised survey), Domain 2 – Systems Change (Domain 5 in the revised survey), Domain 3 – Project Community Context (Domain 4 in the revised survey), Domain 4 – Project Inputs (Domain 3 in the revised survey), Domain 5 – Project Activities and Strategies (Domain 2 in the revised survey), and Domain 6 – Descriptive Information (Domain 1 in the revised survey).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	James Murdoch, PhD
File Modified	0000-00-00
File Created	2021-01-20

Cognitive Testing Report

Attachment C - Cognitive Testing Report_FINAL.docx

Our Town Program Proposed Collection