IMLS Museums for America Evaluation Study

OMB Package

IMLS Museums for America Evaluation Study

Contract 2009-057

.

July 27, 2010

OMB Package

IMLS Museums for America Evaluation Study

Contract 2009-057

Submitted to:

Carlos Manjarrez

Office of Policy, Planning, Research & Communication
Institute of Museum and Library Services
1800 M Street NW, 9th Floor
Washington, DC 20036-5802
Phone: 202-653-4647
Fax: 202-653-4611
email: [email protected]

Submitted by:

RMC Research Corporation

1000 Market Street, Building 2

Portsmouth, NH 03801

Phone: 603.422.8888

Fax: 603.436.9166

email: [email protected]

July 27, 2010

Contents

Supporting Statement for Paper Work Reduction Act Submissions

Overview 1

Purpose 1

Methodology 1

Data Collection 2

Pilot Testing 3

Timetable 3

Appendices

Appendix A 5

Online Survey Outline (revised) 6

Online Survey Flow Chart (revised) 8

Online Survey Questions (revised) 11

Appendix B 33

Case Study Outline 34

Case Study Protocols 36

Justification A & B separate documents

Supporting Statement for Paper Work Reduction Act Submissions

Overview

This document has been prepared to support the clearance of procedures and data collection instruments to be used in the Institute of Museum and Library Services’ (IMLS) study of its Museums for American grant program. This study falls under the IMLS contract 2009-057.

Part A, “Justification,” and Part B, “Information Collection Using Statistical Methods,” respond to instructions in SF93. Appendices contain copies of the instruments and descriptions of procedures for which clearance is sought.

Purpose

The Institute of Museum and Library Services (IMLS) is the primary source for federal support for the nation’s libraries and museums (federal law 20 U.S.C. Section 9171). Museums for America (MFA) is the Institute’s largest grant program for museums and is designed to support projects and ongoing activities that build museums’ capacities to serve their communities. The program serves all museum types (aquariums, arboretums, botanical gardens, art museums, youth and children’s museums, general museums, historic houses and sites, history museums, nature centers, natural historical and anthropology museums, planetariums, science and technology centers, specialized museums, and zoological parks), with the exception of for-profit and federally funded institutions. The Museums for America grant program supports high-priority activities that advance an institution’s mission and strategic goals. From 2004 to 2007 IMLS solicited applications and awarded grants in three categories: Serving as Centers of Community Engagement, Sustaining Cultural Heritage, and Supporting Lifelong Learning. More recently awards are being made in refinements of the earlier categories: Engaging Communities (Education, Exhibitions, and Interpretation); Building Institutional Capacity (Management, Policy, and Training); and Collections Stewardship. 

In response to its legislative authority to conduct analyses on the impact and effectiveness of museum and library services (20 U.S.C. Chapter 72, Section 9108), IMLS proposes to assess the effectiveness of its Museums for America grant-making program. Data from the study will be used to inform IMLS of 1) applicant and grantee profiles, 2) the applicants’ view of the application process, 3) funded project and implementation activities, and 4) organizational and community affects from funded projects that have completed. Evaluation findings will contribute to enhancing the MFA application process and the grant-making itself. In addition, the study results will highlight the overall accomplishments of the grant program has on the museum field.

Methodology

RMC Research Corporation has designed a two-part data collection effort. The first is an online survey. The survey will solicit information from all applicant institutions (approximately 644 institutions) that applied for Museums for America grants for the 2008 to 2010 funding cycles. Questions will include perceptions of outreach, grant requirements and assistance, application outcomes for an institution, and the role of IMLS funding. A subset of the 644 applicants (395 institutions) will also be asked questions about perceptions of unfunded applications. Additional survey questions will solicit data on the project implementation process from grantees with completed projects or projects in process for at least a year (approximately 761institutions). These items will include questions about partnerships formed and grant activities. An additional set of questions on the effects of grantee activities will be addressed to the grantees with completed projects (a subset of the above totaling approximately 589 institutions). If a grantee’s project was completed three or more years ago (a subset of the above includes 304 institutions), the grantee will be asked if there were any long-term effects on the institution or community from the funded project. See Appendix A for the online survey outline, flow chart, and survey design.

Six case studies will be conducted at selected MFA grantee sites. These case studies will provide a nuanced view of the trajectories of Museums for America grantee activities through the close study of six museums with completed MFA projects. Qualitative data gathering at these institutions will allow for targeted and specific inquiries into the factors associated with effects both within the institution and in the surrounding community. These data will provide an opportunity to investigate the connections among institutional and community characteristics, program implementation choices, and effects.

The case studies will be based on interviews and focus groups conducted with museum administrators, program staff, and participants as appropriate to each program, accompanied by extensive review of grant applications, annual reports and other documentation, media coverage, and first-hand observations. Two-person teams of RMC staff will conduct each site visit, anticipated to take one-and-a-half days to complete, with extensive pre-visit planning contact, and follow-up telephone interviews as needed.

Case study site visit selection will occur in close consultation with the IMLS. It is anticipated that two studies will be conducted in each of the three grant categories (Serving as Centers of Community Engagement, Sustaining Cultural Heritage, and Supporting Lifelong Learning). In seeking maximum variability among case studies, selection criteria may also include institution size, location (geographic and urban or rural), audience(s) served, and exemplary successes. RMC intends to conduct three (one from each grant category) of the six site visits as pilot cases prior to OMB approval. See Appendix B for the case study outline and protocols.

Because of staff turnover and the potential for new staff’s unfamiliarity with the MFA application process and in order to secure adequate representation to examine the application process by institution size and type, the entire universe of about 644 MFA applying institutions from the 2008-2010 funding cycles will be solicited for the online survey about the application experience, rather than a representative sample. A survey of all applicants will allow the Institute to garner valuable information about the application process (including its potential value to non-funded museums in clarifying their mission or strengthening their organizations). All 761 institutions awarded grants from 2004 to 2009 will be asked additional questions concerning project implementation, with subsets of this larger group asked to complete questions about short and long-term effects of their grant (with 285 of these completing one additional section and 304 completing two) . These latter questions will focus on the institutions’ perceptions of the value of the grants on their operations and their communities.

Data Collection

Online Survey

With the approval of the survey by OMB and the completion of the online forms and database, RMC will begin the implementation of the online survey. Respondents will complete the surveys online using a code and password. The sequence will be as follows:

1. IMLS will send out an announcement via newsletter of the study including the importance of participation to all Museums for America applicants.

2. RMC will send invitations to Museums for America applicants via letters addressed to the current director; invitations will direct recipients to the url of the online survey form and contain the code and password to access the survey.

3. Email reminders will be sent to all non-responding applicants.

4. To ensure a high response rate of completed project grantees within the desired analytic characteristics (e.g., institutional types and sizes), follow-up phone calls will be made to non-responding grantees.

5. Completed surveys will be automatically saved and logged into RMC’s secure database.

6. Any outlying or questionable survey data will be verified via email or phone to ensure data integrity.

Case Studies

RMC Research Corporation has developed a menu of case study protocols in order to choose the most appropriate depending on the type of project a grantee undertook. Case study data will be analyzed as they are collected in order to identify the range of responses for each indicator.

Measures to ensure reliability of the qualitative data include: a) consistent note-taking, b) exposure to multiple and different situations using triangulated methods, and c) comparing emerging themes with findings from related studies. Validity will be addressed as follows:

• Face validity (i.e., whether observations fit a plausible frame of reference);

• Internal validity through pre-testing instruments, rigorous note-taking and methods, peer debriefing, and member checks or participant verification;

• External validity through “thick description” and comprehensive description of our methods so that others can determine if our findings compare with theirs.

Pilot Testing

In order to minimize any technical problems and to clarify minor wording or language issues that could cause confusion, RMC will pilot-test the online survey with four applicants and four grantees. The pilot testing will take place after the online survey has been approved by the OMB. In order to avoid duplication of effort, data collected through pilot testing will be used as survey data.

Timeline

The following chart details the evaluation timeline by task.

Task Description

Jan '10

Feb

Mar

Apr

May

Jun

Jul

Aug

Sep

Oct

Nov

Dec

Jan '11

Develop Evaluation Instruments and OMB package

Develop survey and case study instruments

X

X

Draft OMB documents

X

Submit OMB draft package to IMLS

X

Revise and submit final package to IMLS

X

OMB Submission and Review

X

X

X

X

X

Develop Online Survey & Database

Obtain grantee data and contact information from IMLS

X

X

Create data base of grantees and grants and contact info

X

Develop online survey forms

X

X

X

Pilot respondents to test online version

X

Finalize forms and database

X

Collect Data

Conduct Case Study Visits (n=6)

X

X

Prepare and contact grantees for online survey announcement

X

Collect online survey data from applicants and grantees

X

X

Send reminders to complete online survey

X

Collect & verify data in database

X

X

Analyze Data

Analyze existing administrative applicant and grantee data

X

X

Analyze survey data for preliminary findings

X

Review preliminary findings with IMLS

X

Conduct final analyses

X

Preparation of Final Deliverables

Write draft report

X

X

Write data file documentation

X

Submit draft report and documentation to IMLS for review

X

Revise report and documentation

X

Design and produce copies of exec summary

X

X

Submit final report, exec summary, and data files to IMLS

X

Send exec summary to responding grantee institutions

X

IMLS Museums for America Evaluation Tasks and Timeline

Appendices

Appendix A

Online Survey Outline (revised)

Online Survey Flow Chart (revised)

Online Survey Questions (revised)

Appendix B

Case Study Outline

Case Study Protocols

Appendix A

Online Survey Outline (revised)

Online Survey Flow Chart (revised)

Online Survey Questions (revised)

MFA Online Survey Outline (revised)

July 8, 2010

NOTE: Large institutions serving as a parent to multiple museums that have applied to the MFA program will be identified. Each museum will be considered a separate, unique entity. Each museum will be asked to participate in the survey and respond to it’s MFA experiences.

There are six sections to the full survey. All surveyed museums will be asked to complete Part A on background information. However, Part B and C will be completed by museums applying for the 2008-2010 funding cycles. Parts D-F will be completed by a subset of museum grantees. Museums receiving multiple grants will be requested to complete information about ONLY the oldest of theses grants in an effort to get the richest data on grant effects over time. First time grantees with projects that have been in process for less than a year will not be asked to complete Parts D-F.

Introduction and Instructions

PART A: ALL SURVEYED MUSEUMS (~1011 museums)

Museum Background

• Institutional Type, governance, maturity

• Experience with other grant programs

PART B. ALL APPLICANTS from 2008-2010 Funding Cycle (~644 museums)

Overall Application Process

• All Pre-Populated Records of Application History (2008-2010)

• Application Process Experience

• Application Process Outcomes

PART C: ANY APPLICANT from 2008-2010 Funding Cycle NOT FUNDED (~ 395 museums)

Overall Unfunded Project Experience

• Pre-Populated Records of Unfunded Application History (2008-2010)

• Selection and Award Process

• Reapplication History

PART D: FUNDED GRANT PROJECT ACTIVITIES and STATUS (~761 museums)

Single Grant Project Descriptions

• Pre-Populated Record of Funded Grant\ Knowledge of the project

• Grant Project Background

  • Partnerships

  • Activities

  • Exhibitions

  • Collections

  • Technology and Online Resources

  • Organizational Development

  • Groups Served by the Project

  • Status of Activities

PART E: SHORT TERM EFFECTS and REFLECTIONS (~589 museums)

NOTE: This section on individual funded project effects will provide data for identifying patterns of effects across analytic characteristics such as the size of grants, and size of museum. In addition, project effects will be examined by the different types of activities the institution engaged in during the project. The project effect data are NOT intended to trace any direct causal influence from each project activity.

Single Grant Project Effects

• Project Effects

  • Organizational

  • Community

  • Effects due to Partnerships

• Sustainability and Reflection Questions

PART F: LONG TERM EFFECTS (304 museums with projects completed 3 or more years ago)

Single Grant Project Long-Term Effects

• Organizational, Community, and Unexpected Effects

MFA Evaluation

Online Survey Flow Chart (revised)

There are two distinct survey research studies being conducted for IMLS with regard to the Museums for America grant program. The first study examines the perceptions of the MFA application process and the experiences of having unfunded applications. Only museums applying recently, for the 2008 to 2010 funding cycle will be contacted (Section B). Those applicants notified of any unfunded applications from this submittal time will be ask additional option questions with regard to this experience (Section C). All museums participating in this study will be asked background questions (Section A).

The second study of inquiry addresses the activities associated with funded grant projects (Section D), short term effects from a completed project (Section E), and long term effects from a project finished three or more years ago (Section F). Museum staff will be asked about only one awarded project based on the following criteria:

1. had a completed project in 2007 or earlier (Section D,E,F), or

2. had a recent completed project in 2008-2010 (Section D, E), or

3. have a project in progress for at least a year (Section D).

All museums participating in this study will be asked background information (Section A).

The table below presents the estimated number of museums asked to participate in one or both of the studies depending on their status of application for the 2008-2010 funding cycle AND if there is a funded grant in progress for at least a year or had a completed project. Each variation of the survey sample pool is described and the number of museums asked to answer the appropriate survey section is indicated.

The Number of Museums Surveyed by Survey Section

Description of Sample Pool		Application Study (2007-2009 Applicants only)		Project Activities and Effects Study on ONE Awarded Project
	A. Museum Background	B. Perception of Application Process	C. Perception of Unfunded Application(s)	D. Project Activities ( 1 project completed 2007 or earlier OR completed 2008 to present or 1 project in progress)	E. Short Term Effects ( 1 project completed 2007 or earlier OR completed 2008 to present)	F. Long Term Effects (1 project completed 2007 or earlier)
Applied 2007-2009 and at least one unfunded	250	250	250
Applied 2007-2009 and at least one unfunded; only grant ever received is still in process	34	34	34	34
Applied 2007-2009 and at least one unfunded; have a recent grant completed only	45	45	45	45	45
Applied 2007-2009 and at least one funded; have a completed grant finished in 2007 or before.	66	66	66	66	66	66
Applied 2007-2009 and all funded; have grant project in process	106	106		106

Description of Sample Pool		Application Study (2007-2009 Applicants only)		Project Activities and Effects Study on ONE Awarded Project
	A. Museum Background	B. Perception of Application Process	C. Perception of Unfunded Application(s)	D. Project Activities ( 1 project completed 2007 or earlier OR completed 2008 to present or 1 project in progress)	E. Short Term Effects ( 1 project completed 2007 or earlier OR completed 2008 to present)	F. Long Term Effects (1 project completed 2007 or earlier)
Applied 2007-2009 and all funded; have recent project completed	80	80		80	80
Applied 2007-2009 and all funded; have a completed grant finished in 2007 or before.	63	63		63	63	63
Did not apply 2007-2009; have grant project in process	32			32
Did not apply 2007-2009; Have a recent grant completed	160			160	160
Did not apply 2007-2009; Have a completed grant finished in 2007 or before	175			175	175	175
TOTAL Museums	1011	644	395	761	589	304

Online Survey Questions (revised)

Museums for America Evaluation

July 12, 2010

Introduction and Instructions

Your museum is participating in a study on the Institute of Museum and Library Service’s Museums for America Grant Program, conducted by RMC Research Corporation. Your responses will help us evaluate and improve the Museums for America Grant Program. The survey is divided into six (6) sections which include questions on the application process, grant project activities, and grant project outcomes.

All museums will be asked to complete Part A on background information. However, Part B -F will be completed by a subset of museums. You will be directed to the appropriate sections of the survey for your institution according to your museum’s status as a funded or unfunded applicant as well as the year of your grant application or award.

This research is sponsored by the Institute of Museum and Library Services. Your museum was selected for participation in this survey because of its status as a Museums for America Grant Applicant. This is a confidential survey. Information in the report will be aggregated and individual persons and individual organizations will not be identified. Respondents will provide information to RMC’s secure website where it will be downloaded onto a password-protected service in RMC’s office in Portsmouth, New Hampshire. All questions are optional, so you are free not to answer any questions you do not want to answer. We expect completing the survey will take from seventeen (17) to forty-five (45) minutes, depending upon the number of sections relevant to your museum.

PART A: MUSEUM BACKGROUND (COMPLETED BY ALL SURVEYED MUSEUMS)

A1. Which of the following most closely describes your museum? (select one)

	Aquarium		Historic House/Site		Science/Technology Museum
	Arboretum/Botanical garden		History Museum		Specialized Museum (INFO)
	Art Museum		Natural History/Anthropology Museum		Zoo
	Children’s /Youth Museum		Nature center
	General Museum (INFO)		Planetarium		Other; please specify:_____________________

(INFO) A museum with collections representing two or more disciplines equally (e.g., art and history)

(INFO) A museum with collections limited to one narrowly defined discipline (e.g., textiles, stamps, maritime, ethnic group, tribal)

A2. Which of the following most closely describes your museum’s governance? (select one)

	College, university or other academic entity		Federal		Local (county or municipal)
	Non-profit, non-governmental organization or foundation		State
	Native American Tribe/Native Hawaiian Organization

A2a. If your museum is governed by a college, university, or other academic entity, which of the following most closely describes your governance? (select one)

	Community college		Private four-year college or university
	Historically Black college or university		State four-year college or university
			Other; please specify: ___________________

  

A3. What year was the museum first open to the public? ______

A4. How many people have held the museum’s leadership role (e.g., director, curator, president) over the past seven years (since 2004)? ______

A5. How many current staff members does the museum have?

___ a. full-time employees

___ b. part-time employees

___ c. unpaid staff (volunteers, docents, Board members)

Other Grant Programs

A6. How many times has your museum applied for any IMLS grant in the last 12 months? ______ Don’t Know

A71. Has your museum applied for any OTHER IMLS grants listed below?

A72. If yes, were you awarded a grant?

A73. If yes, was it before or after your MFA application?

Other IMLS Grant Programs	A71 Did your institution apply?			A72. If yes, were you awarded a grant?		A73 If funded, was it before or after MFA application?
	No	Yes	Don’t Know	No	Yes	Before	After
a. 21st Century Museum Professionals	Skip to b		Skip to b	Skip to b
b. Connecting to Collections: Statewide Implementation Grants	Skip to c		Skip to c	Skip to c
c. Connecting to Collections: Statewide Planning Grants	Skip to d		Skip to d	Skip to d
d. Conservation Assessment Program	Skip to e		Skip to e	Skip to e
e. Conservation Project Support	Skip to f		Skip to f	Skip to f
f. Museum Assessment Program	Skip to g		Skip to g	Skip to g
g. National Leadership Grants	Skip to h		Skip to h	Skip to h
h. Museum Grants for African American History & Culture	Skip to i		Skip to i	Skip to i
i. Native American/Native Hawaiian Museum Services,	Skip to j		Skip to j	Skip to j
j. American Heritage Preservation Grants	Skip to 3.4		Skip to 3.4	Skip to 3.4

A8. Has your museum applied for any federal agency grants other than to IMLS?

No

Yes

Don’t Know

PART B. APPLICATION PROCESS

Pre-Populated Record <Fields> To Appear On The Online Survey

B1. Are you knowledgeable about each of the following MFA applications?

 <Institution/Museum Name>  <OrgUnit>

All MFA applications submitted to IMLS for 2008-2010 funding cycle by first submit date to last

Knowledgeable about?		Fiscal Year	Project Title	Funded/ Unfunded Status	Contact Name	Contact Title
No	Yes
No	Yes
No	Yes
No	Yes

Application Process Experience 

In order to improve the MFA application process, IMLS is interested in your experience(s) with the process.  Please take into account all MFA applications your museum has submitted to IMLS over the past three years.      

B2. The IMLS Museums for America grant program requires museums to commit to a 1:1 cost sharing for the proposed project. Over the past three years, did your institution ever CONSIDER applying for a MFA grant but DID NOT APPLY due to the cost sharing requirement?

No

Yes

Don’t Know

B3. For the MFA application(s) the museum did submit to IMLS, did the cost-sharing requirement have an influence on any of the following with respect to your proposed project(s):

	No	Yes	Don’t Know
Amount of time to raise the cost sharing funds
Type of MFA grant applied for
Duration of the proposed project
Scope of the proposed project
Staffing costs of the proposed project
Non-labor costs of the proposed project

B4. Which of the following description(s) apply to the person(s) who wrote your MFA application(s)? (select all that apply)

	Dedicated staffed grant writer (paid or unpaid)		Administrative Leadership (e.g., Executive Director, Head Curator, President)
	Hired grant writer(s) as consultant		Other staff member(s) (e.g., program coordinator)
	Other: please specify________________________		Don’t Know

B5. Were you aware of the IMLS outreach activities listed below?

B6. If yes, did you participate in any of the activities?

B7. If yes, how helpful were the activities in completing your MFA application?

	B.5. Aware		B.6 If aware, did you participate		B.7. If participated, how helpful was the activity?
	No	Yes	No	Yes	Not at all helpful	Somewhat helpful	Very helpful
a. Information session(s) at conferences/meetings	Skip to b		Skip to b
b. Information from national, regional, or state associations regarding MFA application	Skip to c		Skip to c
c. Audio conference calls prior to application deadlines	Skip to d		Skip to d
Id. ndividual counseling through phone calls, emails, or in-person visits	Skip to 4.7		Skip to 4.7

B8. Were you aware of the following MFA resources posted on the IMLS website?

B9. If yes, did you utilize any of the resources?

B10. If yes, how helpful were the resources in completing your application?

	B8 Aware		B9 If aware, did you utilize?		B10 If yes, how helpful was the activity?
	No	Yes	No	Yes	Not at all helpful	Somewhat helpful	Very helpful
a. Sample MFA applications and tips on IMLS website	Skip to b		Skip to b
b. Outcome-based evaluation resources	Skip to 4.10		Skip to 4.10

B11. To what extent do you disagree or agree with the following statements? 

Topic	Statement	Strongly Disagree	Disagree	Agree	Strongly Agree	Don’t Know or Not Applicable
Access to MFA grant information	Information about the MFA grant opportunities was easy to obtain.
Clarity of MFA application	The guidelines and requirements were clear.
	Instructions on budget ( direct, indirect costs, cost-sharing) were clear.
	The evaluation requirements were clear.
	Instructions on submitting an application were clear.
	The MFA grant-review process was clear.
Grants.gov	Downloading the Grants.gov application from the website was easy.
	Submitting the online applications through the Grants.gov was easy.
	IMLS staff was helpful in answering questions about using the Grants.gov website.
	Submitting an application by hand and mail was easier than using grants.gov

B12. Including yourself, has anyone on your staff served as a:

	No	Yes	Don’t Know
Field reviewer for the MFA grant program
Grant review panelist for the MFA grant program

Application Process Outcomes

B13. IMLS is interested in hearing if the application process itself, regardless of funding, had any effects on your museum. To what extent did the application process prompt your museum to : 

	Not at All Effected	Somewhat Effected	Effected A Lot	Don’t Know
Create a mission/strategic plan for the first time
Revise your mission/strategic plan
Increase awareness of different programming
Increase awareness of your resources/strengths
Reframe your project in order to fit into the categories in IMLS guidelines
Adopt new ways of integrating outcomes-based planning in the project development and evaluation processes
Explore new external partnerships
Collaborate among departments/staff
Explore new technologies
Improve your institution's ability to apply for other (non IMLS) funding
Other; please specify:

B14. Is the MFA application process, more difficult (more complex, requires more time), the same or less difficult (less complex, requires less time) than other types of grant applications?

	More difficult	Same	Less difficult	Don’t Know
Other Federal Grants
Other Government Grants (State, Local, County, etc)
Private Grants

B15. Were you aware that IMLS DOES NOT track reapplications, therefore each application, whether a reapplication or not, is treated individually and on its own merits?

No

Yes

B16. How likely is it that your museum will apply for another Museums for America grant in the future?

Not at all likely

Somewhat Likely

Very Likely

Don’t Know

B16a. If not at all likely, why not? (Check all that apply)

	Already have an application or project in process
	Our Mission or strategic is no longer current
	Unavailable resources to plan and write an application
	Unavailable or inefficient resources to carry out a project if awarded
	Finding funds/resources for cost sharing is too difficult
	Funding is not needed at this time
	Don't see ourselves as competitive/ discouraged by previous failed applications
	The application process requires hardware and/or software not available to us
	Other; please specify: ____________________________________________

B17. What recommendations do you have for improving the MFA application process?

SYSTEM NOTE: IF MUSEUM HAD ANY APPLICATIONS UNFUNDED from 2008-2010 funding cycle Continue with Section C.

SYSTEM NOTE: IF MUSEUM HAD ALL APPLICATIONS 2008-2010 FUNDED Skip to Part D IF museum has a project in process for at least one year OR a completed project.

SYSTEM NOTE: IF MUSEUM HAD ALL APPLICATIONS 2008-2010 FUNDED EXIT IF museum DOES NOT have a grant in progress for at least one year OR a completed project.

PART C: OVERALL UNFUNDED MFA APPLICATION REACTIONS

COMPLETED BY MUSEUMS with ANY UNFUNDED MFA APPLICATIONS from 2008-2010 funding cycle

Unfunded MFA Project(s)

• Unfunded project questions ask about overall experience (not app specific)

Fiscal Year	Project Title	Contact Name	Contact Title

Application Experience

C1. To what extent do you disagree or agree with the following statements:

	Strongly Disagree	Disagree	Agree	Strongly Agree	Don’t Know or Not Applicable
We believe our application(s) was/were given full and fair consideration.
The reviewers’ comments were useful for redesigning the project.
IMLS staff was helpful in providing feedback.

C2. Did your museum do any of the following with any of the unfunded MFA applications:

Did you:	No	Yes	Don’t Know
redesign an unfunded project and resubmit the application to the IMLS-MFA program?
If YES, did you receive IMLS-MFA funding for the redesigned project?
resubmit a MFA unfunded project to a different IMLS grant program?
If YES, did you receive IMLS funding from a non-MFA grant program?
submit a MFA unfunded project to a non-IMLS funder?
If YES, did you receive funding from the non-IMLS funder?

C2a. If your museum did not receive any outside funding for unfunded MFA projects, were you able to execute any version of a MFA unfunded project?

No

Yes

Don’t Know

SYSTEM NOTE: IF NO FUNDED PROJECTS in PROCESS OVER A YEAR OR NO COMPLETED PROJECTS - EXIT

Thank you for taking the time to reply to this survey.

Please hit the SUBMIT button if you are done.

SYSTEM NOTE: IF ANY FUNDED PROJECTS in PROCESS OVER A YEAR OR COMPLETED PROJECTS continue with Part D.

PART D: FUNDED GRANT PROJECT ACTIVITIES and STATUS

COMPLETED BY MFA GRANTEES with a project in process for at least a year or a completed project.

Pre-populated fields for ONE targeted funded project.

SYSTEM NOTE: Table below will be presented at the top of each new survey page for Section D, E, F

Knowledgeable about?		Fiscal Year	Project Title	Estimated Completion Date	Number of Extensions	Contact Name	Contact Title
No	Yes

Single Grant Project Descriptions

Grant Project Background

D1. Please identify the types of partner organizations (INFO involved in this MFA project.

(INFO) Partner organizations are defined as any outside organization which expended cash or in-kind resources on the project. In-kind resources include s any payment made in the form of goods and services, rather than cash.

	Other Museums; please identify museum type _(DROP DOWN MENU)_________________________
	Library

Government organizations
	State government
	Local or County government
	City Government
	Other; please describe ___________________________________

Non-Government Organizations/Community Organizations
	Community Health facility (hospital, mental health facility, health clinic, etc)
	Family Services Organization (day care, YMCA, family services center, etc)
	Youth Organization (Boys and Girls Club, 4H Club, Afterschool Program)
	Senior Services
	Arts and Culture Organization
	Legal Services Organization (family court, legal aid organization, etc)
	Local Media
	Civic Organization (Kiwanis, Chamber of Commerce, etc)
	Local Business
	Other Non-Profit Organization; please describe_________________________________
	Other; please describe

Education
	Elementary School
	Secondary/High School
	4 Year College or University
	Community College
	Other; please describe

Policy and Research Organization; please identify ______________

Online/ Technology Partner; please identify ______________________

	Other; please describe _____________________________________
	NO partner organizations were\are involved

D2. Indicate which of the following types of activities (Programming, Exhibitions, Collections, Technology and Online Resources, or Organizational Development) your museum conducted or is in the process of doing as part of this MFA project.

	No	Yes	Don’t Know
Programming (e.g.; concerts, broadcasts, lectures, workshops, education programs)
Exhibitions (e.g.; new, upgraded, traveling exhibits)
Collections (e.g.; digitizing collections, inventory, collection management, moving items)
Technology and Online Resources (e.g.; online media, database creations, purchasing technology, audio)
Organizational Development (e.g.; hiring, staff development, management systems)

SYSTEM NOTE: IF YES to ABOVE GO TO APPROPRIATE SECTION; IF NO SKIP SECTION

D2a. Programming Activities– If this MFA project included programming activities, please indicate for each activity whether the museum developed a new activity, OR enhanced an existing activity (e.g., expanding for different age groups, upgrading materials) OR continued an existing activity.

	Select one - Radio button for selection
	Develop new programs or materials	Enhance existing programs or materials	Continue existing programs or materials
Concerts
Film festivals
Live Performances
Broadcasts
Demonstrations and Workshops
Lectures
Community discussion groups
Education programs at our institution
In-school programs
Afterschool programs
Curriculum guides
Interpretive programs or materials
Classes or institutes
Training sessions
Conferences
Internships, mentoring or apprenticeships opportunities
Other; please describe

No Programming Activities

D2b. Exhibitions (select all that apply)

	Upgrade/expanded current exhibit, including making current exhibit accessible (e.g., multi-languages, Braille)
	Develop concept for new exhibit
	Research new exhibit
	Plan new exhibit
	Fabricate a new exhibit
	Create traveling exhibition
	Other, please describe

No Exhibitions

D2c.. Collections (check all that apply)

	Digitize collections
	Convert non-digital content to digital content
	Repurpose digital content (INFO)
	Create new digital content

(INFO) Utilized digital content in a different way (e.g., used for education programs)

D2c1. If the MFA project involved digitizing the museum’s holdings, what portion of the collections did/would the project cover? (select one only)

			Less than 10% of collections
			Between 10-25% of collections
			Between 25%-50% of collections
			Between 50% and 75% of collections
			Over 75% but less than 100% of collections
			100% of our collections
			Not Sure
	Inventory collections
	Move collections
	Create public collections finding guides
	Create new collections management guidelines/procedures
	Implement new collections management system
	Other; please describe

No Collection Activities

D2d. Technology and Online Resources (select all that apply)

	Update or create new website
	Create online access to collections records or information
	Purchase technology equipment (e.g., computers, digital cameras, scanner, voice recorder)
	Consolidate multiple databases
	Develop searchable online database
	Upgrade, purchase or install new software (e.g., new collections management software)
	Create audio tour
	Create online exhibition
	Develop high-tech interactive exhibition
	Other; please describe

No Technology Activities

D2e. Organizational Development, including Staffing Resources (select all that apply)

	Provide staff, volunteer and/or docent training, including in use of new technology and online resources
	Hire full-time staff (e.g., program coordinator, director, curator, educator)
	Hire part-time staff (e.g., program coordinator, director, curator, educator)
	Hire a consultant (e.g., interpretive, planning, education, exhibition, web)
	Contract for services (fabrication, design, security, etc)
	Train interpreters, volunteers or docents
	Create or expand interpreter, docent or volunteer program
	Support a research and evaluation program, including conducting surveys
	Develop key management plans (e.g., comprehensive interpretive plan, emergency plans, collections management plans)
	Other; please specify

No Organizational Development

D3. Which of these groups were\are directly served by this project? (select all that apply)

	Paid staff; please describe
	Unpaid staff (docents, interns, volunteers); please describe
	Adult learners
	Seniors
	Parents
	Youth
	Teachers
	Students
	Administrators
	Professionals
	Residents of particular neighborhood or community/geographic area within a community
	Policymakers
	Persons with disabilities
	Low income citizens
	Specific racial/ethnic communities
	Families
	Not group-specific
	Other; please specify

D4. Age groups served by your grant project (select all that apply)

	Preschool
	Grades K-5
	Grades 6-8
	Grades 9-12
	Post high school/Young adults (18-25)
	Adults (26-64)
	Seniors (65 and older)
	Multi-age/Family
	Other, please specify

Status of Activities

Status of activities –the list of activities that grantees check off in the previous section will be populated to this section.

D5. What is the completion status of the project’s activities?

Example:

<populated with specified activities>	Will not do as planned		Still in progress	Completed
Afterschool programs
Curriculum guides
Train interpreters or docents
Establish a docent program and recruit docents
Implement a research and evaluation program, including conducting surveys

D6. Were any other activities conducted or planned?

Yes

No

If yes, please list activities and current status of each below

	Completion Status
Other activities:	Will not do as planned	Still in progress	Completed

D7. Taking into consideration the percentage of resources (money, staffing) spent\being spent on this project, what was/is the main emphasis of the project. (Select only one)

	Programming (e.g.; concerts, broadcasts, lectures, workshops, education programs)
	Exhibitions (e.g.; new, upgraded, traveling exhibits)
	Collections (e.g.; digitizing collections, inventory, collection management, moving items)
	Technology and Online Resources (e.g.; online media, database creations, purchasing technology, audio)
	Organizational Development (e.g.; hiring, staff development, management systems)

SYSTEM NOTE:

IF this project is in progress - EXIT

IF this project is completed – continue to Section E.

OTHER COMMENTS:

As a thank you for your time and interest in participating in this study, IMLS would like to send you a copy of the Museums for America Grant Evaluation Executive Summary.

If interested please fill out the following information:

Name:

Title:

Organization:

Address: City State Zip

Thank you for taking the time to reply to this survey.

Please hit the SUBMIT button if you are done.

PART E: SHORT TERM EFFECTS AND REFLECTIONS

COMPLETED BY MFA GRANTEES WITH A COMPLETED GRANT.

Single Grant Project Effects

E1. Please review the list of effects this MFA project may have had on the organization or community.

Select each effect resulting from the MFA project.

ORGANIZATIONAL EFFECTS (select all that apply)

Programming
	Grant-funded programming continued beyond grant
	Sustained grant-funded programming
	Did not sustain grant-funded programming
	Sought additional funding to sustain grant-funded programming
	Developed related programming to expand value of grant-funded program
	Expanded our range of possibilities for programs or exhibitions
	Other; please describe:
	No changes in programming/not applicable to grant

Organizational Capacity
	Enhanced staff capacity in program development
	Enhanced staff capacity in creating new kinds of exhibitions
	Enhanced staff capacity in reaching new or larger audiences
	Enhanced staff capacity in forming external partnerships
	Enhanced staff capacity in working internally across departments
	Enhanced staff capacity in using outcomes based evaluation
	Improved staff cohesion and commitment to mission
	Greater alignment of staff responsibilities with mission
	Greater board involvement
	Helped institution fulfill its mission (e.g., meet certain benchmarks)
	Increased ability to attract outside funding
	Decreased ability to attract outside funding
	Other; please describe:
	No effect on our organization/not applicable to grant

COMMUNITY EFFECTS (select all that apply)

Audiences
	Reached new audiences (e.g., youth, families, minorities)
	Increased commitment by existing audiences
	Changes in regular audience participation
	Increased audience access (e.g., expanded hours, mobile programming)
	Other; please describe:
	No effect on audiences/not applicable to grant

Community Profile
	Changed institutional identity
	Strengthened museum’s public image
	Raised institution’s prestige in its community
	Increased local media coverage of institution’s activities
	Increased visibility of institution as a center of community learning
	Other; please describe:
	No change to our profile/not applicable to grant

Partnerships
	Improved skills in building partnerships
	Strengthened ongoing partnerships
	Developed new partnerships
	With schools or Institutes of Higher Education
	With youth organizations
	With other museums/historical societies
	With other community organizations
	Developed new types of partnerships
	Decreased ability to attract partners
	Increased ability to attract more partners
	Other; please describe:
	No effect on partnerships/not applicable to grant

EFFECTS DUE TO THE PARTNERSHIP(S) (select all that apply)

	Brought in new audiences
	Brought in new resources not normally available to our organization
	Brought in new staff, paid or unpaid
	Brought in new volunteers or memberships
	Raised community awareness of our organization
	Other; please describe:
	No effect from partnerships/not applicable to grant

E2. Are there any other effects not covered above? If yes, Please describe

__________________________________________________________________________

E3. If the grant project enabled new programming, to what extent has your organization been successful in continuing the programs?

Not at all successful

Somewhat successful

Very successful

Not applicable

E4. If the grant enabled you to bring in new audiences, to what extent has your organization been successful in sustaining these new audiences?

Not at all successful

Somewhat successful

Very successful

Not applicable

E5. Was this MFA grant from IMLS larger, smaller, or about the same as other grants you may have received?

Larger

Smaller

About the same

Not applicable

E6. Were the MFA grant funds sufficient to complete the planned project activities?

No

Somewhat

Yes

Don’t Know

SYSTEM NOTE (NOT DISPLAYED):

IF this project ended less than 3 years ago - EXIT

IF this project ended more than 3 years ago – Continue to Section F

OTHER COMMENTS:

As a thank you for your time and interest in participating in this study, IMLS would like to send you a copy of the Museums for America Grant Evaluation Executive Summary.

If interested please fill out the following information:

Name:

Title:

Organization:

Address: City State Zip

Thank you for taking the time to reply to this survey.

Please hit the SUBMIT button if you are done.

PART F: LONG-TERM EFFECTS

COMPLETED BY grantEEs WHERE PROJECT WAS completed 3 or more years ago

Single Grant Project Long-Term Effects

F1. Please describe any long-term effects(s) that the MFA project has had on your organization.

F2. Please describe any long-term effects(s) that the MFA project has had on your community or community relationships.

F3. Please describe any UNEXPECTED long-term effects(s) that the MFA project has had on your organization, community or community relationships.

EXIT

OTHER COMMENTS:

As a thank you for your time and interest in participating in this study, IMLS would like to send you a copy of the Museums for America Grant Evaluation Executive Summary.

If interested please fill out the following information:

Name:

Title:

Organization:

Address: City State Zip

Thank you for taking the time to reply to this survey.

Please hit the SUBMIT button if you are done.

Appendix B

Case Study Outline

Case Study Protocols

Case Study Outline

Museums for America Evaluation

Overview

Case studies will provide a nuanced view of MFA successes through close study of a select group of completed MFA grants. These case studies will be based on a series of qualitative data collection activities conducted at selected institutions, and will allow for targeted and individual inquiry into the factors producing project effects and best practices, both within the institution and in the community.

The case studies will be based on interviews and focus groups with museum administrators, program staff, and participants as appropriate to each program, as well as extensive review of grant applications, annual reports and other documentation, media coverage, and first-hand observations. Case studies will provide an opportunity for investigating the connections among institutional and community characteristics, program implementation choices, and effects.

Two-person teams of RMC staff will conduct each site visit. Each visit is expected to take one-and-a-half days to complete, with extensive pre-visit planning contact, and follow-up telephone interviews as needed.

Site Selection

Six case studies will be conducted, and will include a mix of each of the three MFA grant categories (Serving as Centers of Community Engagement, Sustaining Cultural Heritage, and Sustaining Lifelong Learning). In seeking maximum variability among case studies, selection criteria may include:

• A range of project activities

• Institution size, type and location variety

• Project director and staff still employed at the museum

A case study selected process will be conducted by RMC under the direction of IMLS. As part of this process, RMC will review relevant grantee reports. RMC will make the final selection based on recommendation of 10-12 projects/institutions by IMLS program officers.

Final site selection will be based on review of the significant factors leading to the particular selections, i.e. the successes or outstanding features of each of the selected grant experiences.

Visit Preparation

Pre-visit discussions will be conducted with the project director and/or other appropriate institution representation to identify optimal interview and focus group subjects and to develop a schedule for the visit that will use the available time to maximum effect.

Possible on-site interviews (approximately six per site) will be conducted with the institution’s project director, executive director, and management or administrative person, project staff, partners, and/or audience members. These may include both paid and unpaid staff and volunteers. It is anticipated that individual interviews will be employed for data collection. However, in the case of multiple partner or community member respondents, small focus groups (maximum six participants) may be convened. Focus group subjects will be drawn, again as appropriate, from community and audience members, education and public programming staff, volunteer staff and interns, community activists, and representatives of school and other partner organizations. RMC will also review available documentation of program effects, such as attendance patterns and histories.

Protocols

RMC will develop protocols for the following categories of interviewees:

• Project Manager(s)

• Museum Leadership

• Project Staff/Educators

• Partners

• Community/Audience Members

The interview and focus group protocols will be tailored to the specifics of the institution and grant project prior to each site visit.

Case Study Protocols

Project Manager Interview

Interviewee Name: _______________________ Title: ______________________

Institution: ___________________________ Date:_________ Interviewer: ____

Application Process

1. Could you share with us the history of this project? How did the idea originate?

2. How did you decide to apply for the IMLS MFA funding? Did you apply more than once? If you applied more than once, did you get feedback and was it helpful?

3. How would you characterize the application process compared with other grants you have applied for? How does it compare to other Federal grants you’ve applied for?

4. Did you find the application process valuable? If yes, what was most valuable about it (e.g. what did you learn about your institution, project development, etc.)?

5. What was most challenging about completing the application?

6. How did you approach the cost-sharing requirement?

Project Implementation and Sustainability

1. To what extent was the project successful in advancing your institution’s mission? How does this project align with your institution’s mission?

2. What aspects of the project were most successful?

3. What challenges did your institution encounter in implementing the project?

4. What was the effect or value to your institution of partnerships developed through the project? Have these partnerships been maintained, and if so, how?

5. What does sustainability mean to you in the context of this project? (e.g., exhibits or programming continue, new audiences engaged, continuation of new offerings, access to collections or other resources, etc.) How was this project or component parts supported after the grant period?

Community Effects

1. How did the target audience and/or your existing audience respond to the program? What trends did you observe over time?

2. How has this project changed the museum’s reputation in the community?

3. How has the community’s involvement with your museum continued or grown as a result of the program?

Organizational Effects

1. In what areas, if any, has the capacity of your staff and/or institution been improved (e.g., growth in knowledge, skills, efficiency, programming)? To what extent were these capacity changes the result of new staff, professional development, strategic planning, budgeting changes, or new technologies?

2. To what extent, if any, did the project support the development of new relationships within the institution, such as across departments?

3. Have there been any outcomes of this improved capacity? If so, what? (e.g. additional new programs or initiatives)

4. Did the project result in any changes in how individuals in the organization perceive the institution’s mission and/or value to the community?

Planning and Evaluation

4. To what extent were the outcomes-based planning and evaluation requirements familiar to your or your institution?

5. Do you view planning and evaluation differently as a result of this project? Please explain your response.

6. To what extent are you integrating outcomes-based planning and evaluation in subsequent projects?

Summary

1. Overall, what would you say is the most important effect of having received this grant from IMLS?

2. Were there any surprises or other effects you did not expect or foresee?

3. How would you describe the value of the MFA program for institutions like yours?

4. Any other comments?

Project Staff

Interviewee Name: _______________________ Title: ______________________

Institution: ___________________________ Date:_________ Interviewer: ____

1. Describe your involvement with this project. What was your role?

2. How would you describe the implementation of this project? Did it go smoothly?

3. What new skills or knowledge did you gain from this project?

   1. How have you been able to use these skills or knowledge in subsequent projects?

4. Did the grant have an influence on how you do your work? If so, explain.

5. Are there any other ways in which the grant changed how you do your job, your responsibilities, or improved your ability to do your job?

6. To what extent were you involved in evaluation aspects of the project?

   1. If you were involved in these, were the IMLS requirements valuable?

   2. What challenges did you face in meeting the evaluation requirements? How might these requirements better fit the needs of the MFA projects?

   3. What did you learn from the experience?

7. Are there other ways in which this grant affected your institution’s ability to serve the public?

   1. What kind of community feedback have you received about the project?

8. What other aspects of the project have been sustained since the grant period ended? Please describe. How were these projects supported after the grant period?

Summary

9. Overall, what would you say is the most important effect of having received this grant from IMLS?

10. Were there any surprises or other effects you did not expect or foresee? Please describe.

11. How would you describe the value of the MFA program for institutions like yours?

12. Any other comments?

Museum Leadership Interview

Interviewee Name: _______________________ Title: ______________________

Institution: ___________________________ Date:_________ Interviewer: ____

Overall Effects

1. Please describe the value of this grant for your institution in terms of your mission (e.g., offerings, audience, efficiency).

   1. How does this grant relate to what you consider your institution’s strength in serving the community?

   2. How does this grant relate to what you consider its greatest challenges?

2. What aspects of the project were most successful?

3. What challenges did your institution encounter in implementing the project?

4. How has this project affected what your institution does?

   1. [If a conservation grant, probe about conservation practices after grant. What changed?]

   2. [If an outreach grant, probe about outreach practices after grant. What changed?]

   3. Etc.

Community Effects

5. Did this project change the museum’s reputation in the community? If so, how?

6. How has the community’s involvement with your museum continued or grown as a result of the program?

7. Has the museum’s level of engagement with the community changed over time? If so, how? To what do you attribute that change?

IMLS Grantee

8. Are you regular applicants to IMLS or other federal funders? Was this the first IMLS grant your institution has received or one of many? First MFA grant?

9. How important was receipt of this funding for the institution?

   1. How did the size of the grant compare with other grants received?

   2. What kind of prestige is conveyed by receipt of the grant?

   3. Were you able to leverage the funds or the prestige in additional ways?

10. In what ways did the application process or the implementation of the grant affect how you understand your mission or how you understand how you can achieve it?

11. How would you describe the value of the MFA program for an institution like yours?

12. What challenges have you faced in sustaining the project?

13. What new opportunities for sustaining the project have emerged?

Planning and Evaluation

14. To what extent were the outcomes-based planning and evaluation requirements familiar to your or your institution?

15. Do you view planning and evaluation differently as a result of this project? Please explain your response.

16. To what extent are you integrating the outcomes-based planning and evaluation by IMLS in subsequent projects? Please describe current efforts.

Community Member Interview

Interviewee Name: _______________________ Title: ______________________

Institution: ___________________________ Date:_________ Interviewer: ____

1. How were you connected to the IMLS MFA grant project (attended exhibit or program, community partner, volunteer or docent at the museum, etc.)?

2. Have you had any dealings with the museum in the past? If yes, what types of activities or programs did you conduct with the museum?

3. What did you like most about this project?

4. What did you learn from your involvement in this project?

5. What value did this project provide to the community?

6. Did the project have an effect on your impression of the museum? If yes, please describe.

7. Did the project have an effect on your attendance at museum events or involvement in museum activities?

8. How would you characterize the relationship between the museum and the community?

9. Do you know of other community groups that participate in programming with the museum? If so, what types of activities are they engaged in?

10. Has the museum’s level of engagement with the community changed over time? If so, how? To what do you attribute that change?

Partner Interview

Interviewee Name: _______________________ Title: ______________________

Institution: ___________________________ Date:_________ Interviewer: ____

1. What was your institution’s role in the MFA project? To what extent were you involved in developing the project? To what extent were you involved in the grant application process for this project?

2. What was your interest in participating in this project? In what ways was this project valuable for your institution?

3. Was this the first time you worked with the museum? If so, did this project represent a change in the types of collaboration you have engaged in previously?

4. How was the partnership structured?

5. How has your relationship with the museum changed as a result of this project?

6. Have you continued to work together as a result of the project? If no, why not? If yes, is this work on the same project or on a new project?

7. Do you anticipate future partnerships with the museum? If so, what kinds of partnerships you would be interested in pursuing? What aspects of the partnership would you continue? Do differently?

8. How would you characterize the relationship between the museum and community organizations such as yours?

9. Do you know of other organizations that participate in programming with the museum? If so, what types of activities are they engaged in?

10. Has the museum’s level of engagement with the community changed over time? If so, how? To what do you attribute that change?