|
|
|
IMLS Evaluation of Four Grant Programs Serving
Native American, Native Hawaiian, and Alaska Native Communities
This mixed-method evaluation is designed to determine how well IMLS's grantmaking aligns with the needs of communities served by four specific grant programs; to lay a foundation for improving the quality, reach, and impact of the agency’s grant programs in the future; and to inform efforts to increase the organizational capacity of eligible applicants to submit high-quality grant applications and of grantees to complete their award responsibilities successfully. The grant programs are Native American Library Services: Basic Grants; Native American Library Services: Enhancement Grants; Native American/Native Hawaiian Museum Services Program; and Native Hawaiian Library Services.
The evaluation’s primary information collection efforts will capture application and program experiences with the four grant programs from FY2015-FY2021. This seven-year time period was determined to best balance recency (to support recall of respondents) and longitudinal analysis to enable a meaningful examination of trends. The evaluation team will be able to explore whether application types or program experiences changed before and during the COVID-19 pandemic, as the data will capture multiple years of both conditions.
The proposed evaluation includes three primary information collection activities: (A) a web-based survey of grantees and eligible non-applicants, (B) semi-structured interviews with grantees, unsuccessful applicants, eligible non-applicants, service and intertribal organizations, tribal leadership, funders, and IMLS staff, and (C) a virtual convening of a mix of grantees, unsuccessful applicants, and eligible non-applicants. This primary data collection will be coupled with a secondary analysis of publicly available information about tribal and Native Hawaiian communities and IMLS administrative data on applications, awarded grants, and interim and final grantee performance reports. The next section describes the universes for the primary data collection, followed by more details on each of the three information collection activities.
For the information collection methods proposed in this study, the evaluation team will use two universes as described below:
Universe One is comprised of organizations or departments such as native libraries, museums, cultural centers, and archives affiliated with entities eligible for funding through one or more of the IMLS grant programs.
Universe Two is comprised of supporting institutions and investors in this field, specifically intertribal organizations, discipline-specific service organizations, and funders.
Both universes are censuses of relevant organizations, so there is no sampling utilized to build these universes.
Eligible entities for the four grant programs serving Native American, Native Hawaiian, and Alaska Native communities include federally recognized Native American tribes (including Alaska Native villages or corporations as defined in, or established pursuant to, the Alaska Native Claims Settlement Act, 43 U.S.C. 1601 et seq.) and 501(c)(3) nonprofit organizations that primarily serve and represent Native Hawaiians (as the term is defined in 20 U.S.C. § 7517). We will use the terms "eligible entities" and "tribes and nonprofit organizations serving primarily Native Hawaiians" interchangeably in this and associated documents.
A single eligible entity may operate more than one type of organization or department, such as a library, a museum, a cultural center, or an archive. Consequently, certain eligible entities may apply to two or more grant programs, depending on their ability to meet additional eligibility criteria. For example, two of the grant programs (i.e., Native American Library Services: Basic Grants and Native American Library Services: Enhancement Grants) also require that an eligible entity be able to document the existence of a library that meets, at a minimum, three additional criteria: (1) has regularly scheduled hours, (2) has staff, and (3) has materials available for library users.
Organizations such as schools, tribal colleges, or departments of education are not eligible to apply or receive these IMLS grants on their own, although in certain programs they may be involved in the administration of a grant, and their staff may serve as project directors in partnership with an eligible applicant. Individuals serving in this capacity will be well-positioned to answer the overall evaluation questions (Table B.2) and therefore are included in Universe One.
Because the exact number of organizations or departments such as libraries, museums, cultural centers, and archives associated with eligible entities is currently unknown, we have estimated the size of Universe One (Table B.1) by beginning with the known number of Bureau of Indian Affairs-designated federally recognized tribes (n=574) and U.S. Department of Interior-designated eligible nonprofit organizations (n=114) serving Native Hawaiian populations (Total n = 688). Next, we factored in the percentage of tribes and nonprofits serving primarily Native Hawaiians that submitted applications to IMLS grant programs in the past. To check our estimate, we have included a question in the survey instrument (Appendix B, Question 6) for respondents to indicate the existence of other libraries, museums, cultural centers, and/or archives associated with their tribe or nonprofit. This will help IMLS better understand the entire universe and allow the evaluators to check for respondent bias.
Table B.1. Estimation of the NUMBER OF ELIGIBLE ENTITIES IN Universe One
|
Number |
|||||
Total Eligible Entities |
688 |
|
|
|||
|
574 |
|
|
|
|
|
|
114 |
|
|
|
|
|
Tribes and nonprofits serving primarily Native Hawaiians estimated to submit ≥ 1 application per year (33.1% of Eligible Entities)* |
228 |
|
|
|||
Estimated Universe One Count |
916 |
|
||||
*From IMLS grant records from 1998-2021, the percentage of tribes and from 2005-2021, nonprofit organizations serving primarily Native Hawaiians that applied to more than one of the four specified IMLS grant programs across the Office of Library Services and the Office of Museum Services.
Across these two populations comprising Universe One, the evaluation will focus on the following three cohorts:
Grantees: organizations that have received at least one grant from one of the IMLS grant programs from FY2015-FY2021;
Unsuccessful Applicants: organizations that have applied at least once and have never received a grant from FY2015-FY2021; and
Eligible Non-applicants: eligible federally recognized tribes and nonprofit organizations serving primarily Native Hawaiians that did not apply to any of the four grant programs from FY2015-FY2021.
Universe Two is composed of organizations that actively fund projects, provide technical assistance, and/or advocate on behalf of indigenous populations. These include intertribal organizations, discipline-specific service organizations, and public and private funders. To estimate Universe Two, we reviewed Candid’s database of funders who are actively funding indigenous populations and found 235 public agencies and private foundations that have given over $1M in grants during the time period of focus (FY2015-FY2021). In addition, the National Congress of American Indians (NCAI) has a directory of approximately 53 intertribal organizations and relevant national Indian organizations. In addition, evaluators estimated 25 discipline-based service organizations, resulting in a total universe estimate of 313 organizations.
As indicated above, we propose three primary information collection activities: (A) a web-based survey, (B) semi-structured interviews, and (C) a virtual convening. Table B.2 provides a detailed crosswalk of evaluation objectives, goals, questions, and the information collection activities. Following this crosswalk, we provide details on how each universe is involved in the information collection.
Table B.2. Evaluation Objectives and Associated Evaluation Questions
# |
Objective |
Goal |
Questions |
Evaluation Instruments (G=grantee, UA=unsuccessful applicant, N=non-applicant, F=funder, S=service or intertribal orgs, T=tribal leadership) |
1 |
Program Alignment with Community Need |
Assess how well IML’s grantmaking aligns with the needs of communities served by the four specific grant programs |
What are the current and various needs of Native American tribes and nonprofit organizations serving primarily Native Hawaiians that are eligible to apply for IMLS funds? What are the top priorities of these eligible entities with respect to the type and size of the organization and the characteristics of the communities and/or nations? How do IMLS grantmaking goals, objectives, and grant requirements align with the community needs and priorities? Are the current funding allocations sufficient to support the types of grants communities are interested in applying for? How has the COVID pandemic influenced interests related to grant applications? |
Surveys: Q2, Q6-9, Q16-19, Q30, Q36 Semi-structured Interviews: G2, G3, G4, G5, G15, G16, G18 UA2, UA3, UA10, UA11, UA13 N2, N3, N8, N9, N11 F4, F6 S1, S2, S5 T2, T3, T6, T7 Convening: Dream, Design Secondary data sources: IMLS administrative data, literature review, census data, program files (applications, grantee performance reports) |
2 |
Program Reach |
Identify the gaps in reaching potential applicants. |
How many potentially eligible Native American tribes and nonprofits serving primarily Native Hawaiians exist? What are the differences in how they are organized and governed? How many Native American tribes and nonprofits serving primarily Native Hawaiians have applied for an IMLS grant? How many have been awarded a grant? What organization types (e.g., cultural center, museum, library, community center) are represented in applications for IMLS funds? |
Surveys: Q2, 6-15 Semi-structured Interviews: G1 UA1 N1 F2, F3 S1 T1, T4, T5 Convening: N/A Secondary data sources: IMLS administrative data, literature review, census data, program files (applications, grantee performance reports) |
3 |
Organizational Capacity & Experience of Applicants |
Gain an appreciation for eligible entities’ capacities, defining limitations and constraints to seek and secure IMLS grant funds and implement grant projects. |
What organizational constraints keep tribes and organizations serving primarily Native Hawaiians from applying to IMLS? What capacity do tribes and organizations serving primarily Native Hawaiians need to demonstrate to apply for and be awarded an IMLS grant? How does the IMLS grantmaking process address the capacity limitations of potential applicants? What can IMLS do to better connect to all potential applicants? How can IMLS address applicant capacity issues? How might IMLS consider expanding its reach to tribes and organizations serving primarily Native Hawaiians? How can IMLS better communicate and partner with communities to increase the number and improve the quality of applications? Who should be the key partners and what are the best opportunities to build new relationships? What are the barriers and challenges that entities face when applying for IMLS grants? What is the level of understanding of eligibility and of the processes associated with preparing an application, reviewing an application, and carrying out an award? What do successful vs. unsuccessful applications look like within each grant program and across all grant programs over time? |
Surveys: Q4, Q5, Q20-31, Q35 Semi-structured Interviews: G7, G10, G11, G12, G13, G16 UA5, UA6, UA7, UA9, UA11, UA12 N4, N5, N6, N7, N9 F3, F4, F5, F6 S3, S4 T4, T5, T6 Convening: Discovery, Design, Destiny Secondary data sources: IMLS administrative data, literature review, program files (applications, grantee performance reports) |
4 |
Eligible Applicant Universe |
Conduct comparisons of those who are being reached to understand the differences among those who: apply and are selected (“grantees”), apply and are not selected (“unsuccessful applicants”), and do not apply (“eligible non-applicants"). |
Are there any key factors that distinguish those who have applied and obtained grants vis-à-vis those who have not applied (e.g., size of institution, size of grant, geography, type of applicant, age of institution)? Are there any key factors that distinguish between those who have applied and received grants vs. those who have applied but did not receive grants? |
Semi-structured Interviews: N/A Surveys: N/A Convening: N/A Secondary data sources: IMLS administrative data, program files (applications, grantee performance reports)
|
5 |
Peer Review Process |
Evaluate the peer review process to determine alignment with core cultural principles and examine the effectiveness of the ranking system. |
How does the IMLS peer review process incorporate cultural acknowledgment with respect to Native American and Native Hawaiian grant applicants? How effective is peer review feedback to applicants as they revise applications and seek IMLS funding in subsequent funding cycles? |
Surveys: Q22 Semi-structured Interviews: G10 UA8 S4 Convening: N/A Secondary data sources: IMLS admin records, literature review |
6 |
Project Implementation and Grant Administration |
Gain an understanding of the grant implementation process, monitoring, and performance and financial reporting |
How effective has IMLS been in designing and administering grantmaking across the separate programs in OLS and OMS? What seems to work well and what causes challenges? To what extent did the grantees report having built organizational capacity through new staffing, skills building, ability to take care of collections, etc., relative to their projects (e.g., leadership and staffing, organizational structures & systems, stewardship and quality of the library or museum collection, accessibility of library or museum collection)? To what extent did the grantees report having built capacities due to funded project (e.g., developing new partnerships with other grantees, using grant to attract new funding sources)? In what ways have grantees asked for approval of changes or extensions to their awards and why did they ask for changes? How often do changes need to be made? What are the internal processes needed to implement the changes? How well does IMLS respond to change requests from the grantee perspective? What capacity do Native American/Alaska Native tribes and organizations serving primarily Native Hawaiians need to successfully implement an IMLS grant? What characteristics do high performing grantees have in common (e.g., size of institution, size of grant, geography, type of applicant, age of institution, awarded multiple IMLS grants)? |
Surveys: Q16-18, Q20-34, Q26, Q37 Semi-structured Interviews: G5, G6, G7, G17 UA5, UA6, UA7, UA8, UA12 N4, N6, N7 S4 T5 Convening: Discovery Secondary data sources: IMLS administrative data, literature review, census data, program files (applications, grantee performance reports)
|
7 |
Funding Ecosystem |
Assess the niche of IMLS’s grantmaking as a federal government entity given its small footprint relative to other federal programs and identify how it can both focus efforts to leverage the most substantial outcomes and better align its priorities with those from Congress and the White House |
How are other federal and non-federal entities meeting the funding and other needs of these communities, and what is IMLS’s niche relative to their work? What can IMLS learn from them?
|
Surveys: N/A Semi-structured Interviews: G9, G14, G15 UA4, UA9, UA10 N5, N6, N7, N9 F1, F2, F4, F5 S4 Convening: Dream Secondary data sources: Literature review |
8 |
Post Grant Measurement of Outcomes |
Understand the impact that IMLS-funded projects are having on grantees and their respective constituencies, including understanding the types of outcomes IMLS-funded projects have had based on the grant award size, with particular emphasis on gathering insights from grantees who have received smaller grants |
How have the IMLS grant programs made a difference in the capacity of the tribes and their organizations to:
How do the accomplishments differ based on grant program and award size? If the grantee had not received financial support from IMLS, would the project have been realized? What funding sources would have been accessed? Would the timeline for implementation have been met? Are there any promising practices/models that can be shared with other communities that may benefit from lessons learned? How does IMLS define existing measures of performance and/or outcomes? How do grantees experience the reporting requirements? How does IMLS staff utilize the information collected? |
Surveys: Q16,-18, Q19, Q21-Q31 Semi-structured Interviews: G5, G6, G7, G8, G9, G15 UA4 F5 S4 T5 Convening: Discovery Secondary data sources: IMLS administrative data, literature review, program files (applications, grantee performance reports)
|
The proposed survey will determine how well the four IMLS grant programs align with community needs; assess experiences with the application process, project implementation, and grant management; and explore the building of organizational capacities through projects supported by the grant programs.
The survey will be fielded to grantees and eligible non-applicants and tailored so that different respondent groups will have access to different modules in the survey (Table B.3 and Appendix B).
Table B.3. Survey Modules by Respondent Group
Module |
Respondent Group |
|
Grantee |
Eligible Non-applicant |
|
Organizational Characteristics |
X |
X |
Familiarity with IMLS |
X |
X |
Organization Programs |
X |
X |
IMLS Application Experience |
X |
|
IMLS Grantee Experience |
X |
|
Grants and Fundraising |
X |
X |
Current Support |
X |
X |
IMLS administrative data contains contact information for grantees, and IMLS will send a respondent contact letter (Appendix A) to increase response rates for this respondent group, followed by a letter from Kituwah Services. For eligible non-applicants, the evaluation team will build a non-applicant contact list based on the inputs as indicated above. Given that these respondents will not have had previous contact with IMLS, the evaluation team expects a high nonresponse rate. See Part B, Section 3 for more information on nonresponse mitigation.
The evaluators plan to interview 50 respondents combined across both Universes One and Two. Table B.4 provides an overview of the information to be collected in interviews with each respondent type across the two universes.
For Universe One, we propose conducting 28 interviews (12 grantees, 6 unsuccessful applicants, and 10 eligible non-applicants). This number is consistent with an estimated saturation point of approximately ~30 interviewees across qualitative studies of varying sample sizes over 300 respondents.1 Conducting 28 interviews also allows the evaluation team to capture perspectives across geographic regions and community sizes, and organization or department types.
For Universe Two, we propose conducting 22 interviews with other external stakeholders to collect information on community needs, organizational capacity, applicant experience, project outcomes, and the funding ecosystem in which IMLS operates in order to better obtain a holistic view of the museum and library fields, respectively. This includes funders from public agencies and small and large private foundations (n=8); service and intertribal organizational leaders across library and museum disciplines (n=8); and tribal leaders across geographic regions and community sizes (n=6). The evaluation team also plans to conduct interviews with 10 IMLS staff.
Table B.4. Questions by Interviewee Type
Universe |
Interviewee Type |
Target Number of Respondents |
Information to be Collected |
Universe One |
Grantees |
12 |
|
Unsuccessful Applicants |
6 |
|
|
Eligible Non-applicants |
10 |
|
|
Universe Two |
Funders – Public Agency and Private Foundation |
8 |
|
Service and Intertribal Organizations |
8 |
|
|
Tribal Leadership |
6 |
|
|
IMLS Staff |
10 |
|
The interviews will each be 60 minutes in length for all respondent groups. The interviews will be semi-structured, using pre-prepared guiding language, questions, and prompts. The interviews will allow the evaluation team to learn more from respondents about their organizational characteristics, the contributions of IMLS funding to their organizations, the sustainability of funded projects, experience with the grant application process and grant management, organizational experience with grants and fundraising, and the influence of COVID-19 on their organizations. The grantee and eligible non-applicant interviews may be informed by their relevant survey responses (if applicable) and adapted on the spot to elicit the most useful insights. See Appendix C for the semi-structured interview guides.
The third information collection effort will be a virtual convening, “Appreciative Inquiry Summit,” with up to 30 individuals who represent eligible entities for IMLS grant programs, including grantees, unsuccessful applicants, and eligible non-applicants. The convening will be approximately three hours, including both large group discussion and small group breakout sessions.
The convening will align with indigenous methods of storytelling and center on an Appreciative Inquiry model, a participatory form of inquiry that focuses on organizational strengths and defining opportunities intended to shape future actions.2 Appreciative Inquiry deliberately focuses on what is working well and how things could be made even better. See Appendix D for an outline of the convening structure.
Below we describe the procedures for the information collection via the survey, semi-structured interviews, and virtual convening.
Information Collection Method: The evaluation team constructed one survey instrument tailored to two respondent groups: grantees and eligible non-applicants (Appendix B). The evaluation team had intended to select the entire Universe One of grantees, unsuccessful applicants, and eligible non-applicants for surveying, but analysis of IMLS administrative data revealed only six unsuccessful applicants (unique organizations that submitted at least one application between FY2015-FY2021 but were never selected for a grant award). Given this small sample size, we will not include unsuccessful applicants in the survey but will use the semi-structured interview method to ensure their experience is represented in the evaluation results.
For the other two Universe One cohorts (i.e., grantees and eligible non-applicants), there will be no sampling for the survey, as we will administer a census survey to all members of each cohort. The specific selection and collection procedures for each cohort are described next.
The Grantee Cohort will include those organizations that submitted applications and were awarded funds between FY2015 and FY2021 and have completed at least 9 months of implementation work at the time of the survey (August 2022). According to IMLS records, there were 1,653 grants awarded between FY2015 and FY2021 across all four grant programs (see Study Design Overview) to 331 grantees representing federally recognized tribes and nonprofit organizations serving primarily Native Hawaiians. (Some organizations received more than one grant during the evaluation period.3 )The survey begins with a screening question to ensure the person completing the survey is knowledgeable about the grant, preferably the individual who led project implementation (i.e., the Project Director) and is most familiar with project goals and outcomes. Based on response rates from previous IMLS evaluations, we anticipate a 50 percent response rate. With 331 distinct grantees that are federally recognized tribes or nonprofit organizations serving primarily Native Hawaiians and a response rate of 50 percent, this will yield approximately 164 respondents.
Eligible Non-applicant Cohort: As outlined above, Universe One consists of an estimated total of 916 organizations. After the subtraction of 331 grantees and 6 unsuccessful applicants, the survey will be sent to the remaining estimated 579 eligible non-applicants and include initial screening questions to ensure its completion by a knowledgeable individual from each organization that is eligible but has never applied to one of the grant programs. While we will utilize techniques to improve response rates as described in the next section, we anticipate a relatively low response rate of 25-35%, based on assumptions that there will be limited perceived incentives to participate (i.e., the respondents may not appreciate that their participation could help them understand the various grant opportunities available to them) and lack of a personal connection to the grant programs and staff. This estimated response rate also parallels the 32 percent response rate of non-applicants for IMLS’s 2020 evaluation of its Museum Grants for African American History and Culture program.4 The team will address universe bias by using validation techniques to ensure the non-applicant contact list is as representative as possible from the start.
To build the non-applicant contact list, the evaluation team will rely on contacts from key partner associations, such as the Association of Tribal Archives, Libraries, and Museums (ATALM). Using additional data sources and subject matter expertise, the team will determine any gaps in the expected representation of non-applicants. Based on the assessment of those gaps, the team will reach out to complimentary service and intertribal organizations to augment the non-applicant list and ensure it reflects the composition of their networks. The team will also use distribution estimates (e.g., libraries, museums, and institutions that are both) from tribal-focused published reports, such as the ATALM 2010 Sustaining Indigenous Culture report,5 as a benchmark to further examine the non-applicant distribution list. Lastly, and as noted above, at the conclusion of the collection period, the evaluation team will also cross-check the distribution estimates against the survey instrument responses (Appendix B, Question 6) related to listing all cultural institutions. The evaluation team will note any limitations when assessing the findings if there is over- or under-representation of a key group in the responses.
The survey will be administered online via the SurveyMonkey® platform with access to the respondent data limited to Kituwah Services, LLC and their subcontractors, Seminole Heritage, and Urban Institute. IMLS will send an email message to grantees and eligible non-applicants explaining the evaluation and the request for information collection (Appendix A). Approximately one week later, Kituwah Services will send an email message to invite them to take the survey using an individualized link. The survey is designed to be completed online and accessible through multiple internet platforms and devices such as computers, tablets, and smartphones. A PDF version will be available for download for informational purposes only. The survey completion will vary by respondent type: approximately 30 minutes (0.50 hr) for grantees and approximately 15 minutes (0.25 hr) for eligible non-applicants. The evaluation team will send an automated follow-up email to non-respondents after two and after four weeks. The survey will close after six weeks.
Stratification and Sample Selection: As stated above, there will be no sampling for the survey. The complete universes of grantees and eligible non-applicants will be contacted to assist with identify the appropriate individuals to contact. The unsuccessful applicants will not be surveyed because of their small sample size.
Estimation Procedure: N/A
Degree of Accuracy Needed: N/A
Unusual problems requiring specialized sampling procedures: There are no anticipated unusual problems requiring specific sampling procedures in this evaluation.
Any use of periodic (less frequent than annual) information collection cycles to reduce burden: This information collection will only occur once.
Information Collection Method: The evaluation team will conduct 50 external stakeholder interviews in Universe One and Universe Two to gain deeper, customized insights into the stakeholders' experiences relative to these grant programs. Table B.4 above outlines the information to be collected by respondent type and relevant universe.
The evaluation team will identify the appropriate personnel at the respondent organization to participate in the interview through targeted outreach.
We will conduct outreach to each above-noted interviewee as follows:
In Universe One, we will conduct interviews with individuals at grantee organizations who led project implementation as they will be most familiar with project goals and outcomes.
To address unsuccessful applicants, we will conduct interviews with individuals who led the application process. Should the target interviewee be unavailable, the evaluation team will identify another individual at the organization that is similarly knowledgeable on the project/application process. If no such individual exists, the organization will be marked as “Unable to interview – no knowledgeable respondent at the organization.” The evaluation team will then select another group from the given organization respondent type category.
To address eligible non-applicants, we will conduct interviews with the individuals who would lead the project implementation if they were to apply for and receive a grant.
In Universe Two, we will select interviewees associated with funders and other stakeholders based on those who in the organization have the greatest familiarity with IMLS and funding for indigenous populations as identified through the initial contact requests. If no such individual exists, the organization will be marked as “Unable to interview – no knowledgeable respondent at the organization.”
Stratification and Sample Selection: The team will conduct 28 interviews with contacts from Universe One and 22 interviews with contacts from Universe Two. The evaluation team will also conduct interviews with IMLS staff.
Universe One: Interviewees for Universe One (28 in total) will be selected through purposeful sampling based on cohort classification; type of organization or department (e.g., museum or library); and geographic region. Other characteristics such as organizational capacity and operating budget will be considered as data become available through survey information collection. Table B.5 presents the distribution of interview sampling.
Table B.5. Semi-Structured Interviews FOR Universe One sampling
|
Grantee |
Unsuccessful Applicant |
Non-Applicant |
Type of Organization or Department |
|||
Library |
6 |
4 |
5 |
Museum |
6 |
2 |
5 |
Sub-Total |
12 |
6 |
10 |
Total |
28 |
||
Geographic Region |
|||
Eastern Region |
2 |
0 |
2 |
Eastern Oklahoma, Great Plains, Midwest, Southern Plains |
3 |
1 |
2 |
Navajo, Rocky Mountain, Southwest, and Western |
3 |
0 |
2 |
Alaska, Northwest, and Pacific |
2 |
0 |
2 |
Hawaii |
2 |
5 |
2 |
Sub-Total |
12 |
6 |
10 |
Total |
28 |
||
Universe Two: External interviewees for Universe Two (22 in total) will be selected through purposeful sampling: 8 public agency funders and private foundations, 8 service or intertribal organizations (organizations dedicated to serving indigenous people and aligned with the mission and purpose of IMLS programs), and 6 tribal leaders (Table B.6). We will identify the funders and service or intertribal organizations through the grantee survey, consultation with subject matter experts, conversations with IMLS staff, and a literature review to include organizations such as Indigenous Peoples Museum Network (IPMN), Association of Tribal Archives, Libraries, and Museums (ATALM), and the United South and Eastern Tribes (USET). The evaluation team will develop a targeted list of identified stakeholders and share the list with IMLS to identify those who have a clear conflict of interest (and therefore will be dropped from consideration) or who are not the most appropriate respondent in their organization (in which case we will work to identify the most appropriate respondent using the same group listed above).
Table B.6. Semi-Structured Interviews FOR Universe Two sampling
Stakeholder Organization Type |
Number of Semi-structured Interviews |
Public Agency Funder |
3 |
Small Private Foundation |
2 |
Large Private Foundation |
3 |
Intertribal Organizations |
2 |
Discipline-Based Service Organizations (Libraries) |
2 |
Discipline-Based Service Organizations (Museums) |
2 |
Tribal Leaders |
6 |
Total |
22 |
Lastly, the evaluation team will select up to 10 IMLS staff to interview, representing a variety of roles within the organization, such as Program Officers in the Office of Library Services and the Office of Museum Services.
Estimation Procedure: N/A
Degree of Accuracy Needed: N/A
Unusual problems requiring specialized sampling procedures: There are no anticipated unusual problems requiring specific sampling procedures in this evaluation.
Any use of periodic (less frequent than annual) information collection cycles to reduce burden: This information collection will only occur once.
Other Notes: We selected the sample targets identified in Table B.5 and Table B.6 to provide a reasonably representative perspective from each stakeholder organization type with weighting towards collecting insights from grantees. Kituwah Services will provide updates on interview administration and analysis progress to the IMLS Contracting Officer Representative via bi-weekly updates and to other key IMLS staff, such as Program Officers from the Office of Library Services and the Office of Museum Services, during monthly check-ins. Interviews will only be administered once.
Information Collection Method, Stratification and Sample Selection: The evaluation team will host a virtual convening, “Appreciative Inquiry Summit,” with up to 30 individuals from Universe One who represent eligible entities for IMLS grant programs, including grantees, unsuccessful applicants, and eligible non-applicants. The purposeful sample of participants will consider organization size, tribal population, and geographic region (see Table B.7).
We will conduct the convening using a video conference platform such as Zoom or Microsoft Teams. The meeting will be approximately three hours long and will include both large group discussion and small group breakout sessions.
The convening will align with indigenous methods of storytelling and center on Appreciative Inquiry, a participatory form of inquiry that focuses on organizational strengths and defines opportunities to help shape future actions. Appreciative Inquiry deliberately focuses on what is working well and how things could be made even better, by using the 4-D model: Discovery, Dream, Design, and Destiny.6,7
Table B.7. Virtual Convening Sampling
|
Grantee |
Unsuccessful Applicant |
Non-Applicant |
|
|
|
Type of Organization or Department |
|
|
|
|||
Library |
8 |
1 |
6 |
|
|
|
Museum |
8 |
1 |
6 |
|
|
|
Sub-Total |
16 |
2 |
12 |
|
|
|
Total |
30 |
|
|
|
||
Geographic Region |
|
|
|
|||
Eastern Region |
3 |
0 |
3 |
|
|
|
Eastern Oklahoma, Great Plains, Midwest, Southern Plains |
3 |
1 |
3 |
|
|
|
Navajo, Rocky Mountain, Southwest, and Western |
3 |
0 |
3 |
|
|
|
Alaska, Northwest, and Pacific |
3 |
0 |
3 |
|
|
|
Hawaii |
2 |
1 |
2 |
|
|
|
Sub-Total |
14 |
2 |
14 |
|
|
|
Total |
30 |
|
|
|
||
Estimation Procedure: N/A
Degree of Accuracy Needed: N/A
Unusual problems requiring specialized sampling procedures: There are no anticipated unusual problems requiring specific sampling procedures in this evaluation.
Any use of periodic (less frequent than annual) information collection cycles to reduce burden: This information collection will only occur once.
We will request surveys from organizations that applied to any of the four grant programs between FY2015 and FY2021. This is intended to reduce recall bias from older applications and to acknowledge staff turnover, assuming an increased likelihood that staff involved in the original applications might not still be with the organization. Given the recent – and in some cases ongoing –relationship between the grantee population and the IMLS grant programs, IMLS will email an introductory “Respondent Contact Letter” to all potential respondents at the start of the survey period. The intent of the email is to encourage them to respond by outlining the potential benefit of their participation in the survey. The evaluation team will also conduct personalized outreach via email and phone to encourage the participation of non-respondent grantees (Appendix A). We anticipate a response rate for grantees of approximately 50 percent, resulting in approximately 164 completed surveys.8
We will send a survey to all non-applicant entities that are eligible for IMLS funding and included in a database assembled by the evaluation team that consists of federally recognized Indian tribes (including Alaska Native villages or corporations) and nonprofit organizations that primarily serve and represent Native Hawaiians (as the term is defined in 20 U.S.C. § 7517). Unlike the grantee survey, the evaluation team anticipates lower response rates due to assumed lesser familiarity with IMLS and its grant programs and so with fewer incentives to participate. To mitigate this, the team will partner with the leading associations in the field – including organizations such as Indigenous Peoples Museum Network (IPMN), Association of Tribal Archives, Libraries, and Museums (ATALM), and American Indian Alaska Native Tourism Association (AIANTA) – to secure their assistance in promoting the survey.
The evaluation team also will issue a press release to share with relevant tribal news outlets. The survey language will be concise, emphasizing the short time required to complete it (under fifteen minutes) and the potential benefit to them and to organizations like them. We will send at least two follow-up emails to engage target respondents. The team also will leverage word-of-mouth referrals to boost response rates. Based on a previous IMLS evaluation of the Museum Grants for African American History and Culture program, we estimate a response rate of eligible non-applicants to be approximately 30 percent or 174 completed surveys.9 We will contact organizations such as ATALM, AIANTA, IPMN, and others to secure their assistance in promoting the survey, including through their social media channels. The evaluation team will issue a press release to share with relevant tribal news outlets.
To measure the extent of response bias for each survey group, we will compare respondents and non-respondents based on key organizational characteristics such as budget size, governance, organization type, geographic region, staff size, etc. We will identify these characteristics using data collected as part of the IMLS application for the applicant group. For non-applicants, we will use secondary data from discipline-based service organizations and intertribal organizations to estimate these characteristics. We will use weighting adjustments if respondents’ characteristics differ significantly from those of non-respondents. It is important to note the possibility that eligible non-applicant institution response rate will be lower than the applicant institution response rate because, by virtue of having not applied for these grants, non-applicants may face barriers that limit their ability to engage with or apply to IMLS.
Response rates are expected to vary across different interviewee groups. We will recruit interviewees on a rolling basis until we have reached our target for that group, totaling 50 across all groups (58 including IMLS staff interviews).
To encourage participation, IMLS will send an introductory Respondent Contact Letter to all selected informants at the start of the information collection to encourage participation (Appendix A). Invitations to participate in an interview will be sent by the evaluation team within two weeks of that outreach. The evaluation team will send a follow-up email one week later and send a second email, accompanied by a phone call, three weeks after the interview launch period, as needed. Response rates are indicated in Table B.8. We will invite respondents in each group to interview on a rolling basis and will keep sampling until we reach the target response goal for that group.
TABLE B.8. INTERVIEW RESPONSE RATE ESTIMATES
Respondent Type |
Universe |
Target Responses |
Anticipated Response Rate (incudes attrition)10 |
Estimated Number of Individuals Needed for Outreach to Achieve Target (target responses/ anticipated response rate) |
Grantees |
331 |
12 |
50% |
24 |
Unsuccessful Applicants |
6 |
6 |
100% |
6 |
Eligible Non-applicants |
541 |
10 |
50% |
20 |
Public Agency Funders and Private Foundations |
235 |
8 |
50% |
16 |
Service and Intertribal Organizations |
50 (est.) |
8 |
50% |
16 |
Tribal leaders |
574 |
6 |
50% |
12 |
IMLS Staff |
10 (est.) |
10 |
100% |
10 |
Among grantees, unsuccessful applicants, and eligible non-applicants, there may be a bias towards larger, more-resourced institutions due to their capacity to more likely participate in a 60-minute interview. To account for this, the evaluation team will monitor the characteristics of interviewees as they are being scheduled and subsequently target outreach to participants that may be underrepresented in the interview pool to maximize representation whenever possible.
We will hold an Appreciative Inquiry Summit inviting up to 30 individuals who represent eligible entities for IMLS grant programs, including grantees, unsuccessful applicants, and eligible non-applicants. We will select participants through purposeful sampling based on the type of organization or department (e.g., library or museum); organization size based on staff and budget; and geographic region as outlined in Table B.7 to ensure all populations are represented.
The evaluation surveys have been subjected to several rounds of internal testing using evaluation team staff and IMLS personnel, including senior survey methodologists. Eight experts in the field, similar to but distinct from those in our target sample, reviewed the materials. This review was designed to troubleshoot potential technical issues and reduce the time burden on respondents.
In the proposed mixed-methods approach, the evaluation team will use a variety of methods to analyze the quantitative and qualitative data.
Quantitative Methods: For the survey data, the evaluation team will generate descriptive statistics for each question both in the aggregate and within groups by budget size, organization type, and geographic region. For survey questions that use scales, the team will look holistically at distributions to interpret results and will also use measures of central tendencies (mode, median, mean) to best understand trends among all respondents and subgroups. After reviewing all descriptive statistics across questions and subgroups, the evaluation team will elevate highlights and core findings.
Qualitative Methods: For the open-ended survey data, interview data, and convening data, the evaluation team will use a grounded theory approach to generate themes from the qualitative data inputs. While the interviews are being conducted, the interviewers and convening facilitators will meet to discuss emerging themes. The evaluation team will use qualitative research software (NVivo or Dedoose) to systematically capture codes across interviews. After the initial coding has taken place, the team will reconvene and assess the coding structure and iterate as needed to reflect additional emerging themes in ongoing interviews. Once all survey data are collected and the interviews and convening are complete, the evaluation team will aggregate all codes and consolidate core findings.
When the separate quantitative and qualitative data analysis is complete, the evaluation team will review and triangulate the results together to identify emerging themes and capture areas of nuance across all evaluation questions to include in the interim and final reports.
The agency responsible for receiving and approving contract deliverables is:
Office of Research and Evaluation
Institute of Museum and Library Services
955 L’Enfant Plaza North, SW
Suite 4000
Washington, DC 20024
Person Responsible: Emily Plagman, Expert Consultant, [email protected] (202-653-4763)
The organization responsible for survey design, information collection, and data analysis is:
Kituwah Services, LLC
1158 Seven Clans Lane
PO Box 366
Cherokee, NC 28719
Christopher M. Shrum, Ph.D., MPA – Principal Investigator
William Medcalf – Project Manager
2 Shepherd and Graham (2020) point out that the importance placed on context and conditions and the testing of assumptions is critical to evaluation in Indigenous contexts. They go on to suggest “realist evaluation in combination with appreciative inquiry could be one practical way of bridging Western and Indigenous approaches.” Shepard, R. P. and Graham, K. A. H. (2020) Identifying Key Epistemological Challenges Evaluating in Indigenous Contexts: Achieving Bimaadiziwin through Youth Futures. Canadian Journal of Program Evaluation. 34.3, 442-463.
3 Grants awarded tallied from Annual Performance Reports FY2019 and FY2021 (https://www.imls.gov/sites/default/files/2022-02/2021apr.pdf and https://www.imls.gov/sites/default/files/publications/documents/2019apr.pdf). All publicly available awarded grants data (FY2015-FY2021) were aggregated for each of the four grant programs. Data were accessed on January 26, 2022.
4 See: Nurture, Sustain, Expand. A Retrospective Evaluation of the Museum Grants for African American History and Culture Program (imls.gov).
6 See https://www.betterevaluation.org/en/plan/approach/appreciative_inquiry for more on Appreciative Inquiry.
7 Coghlan, A.T., Preskill, H. and Catsambas, T.T. (2003). An overview of appreciative inquiry in evaluation. New Directions for Evaluation, 2003 (100), 5-22.
8 This is anticipated to be higher than the average response rates to computer-based surveys, which generally hover around 20 to 30 percent (see Manfreda, Katja Lozar, Michael Bosnjak, Jernej Berzelak, Iris Haas, and Vasja Vehovar. 2008. “Web Surveys versus other Survey Modes: A Meta-Analysis Comparing Response Rates.” International Journal of Market Research, 50(1), 79–104. https://doi.org/10.1177/147078530805000107).
910We estimate a higher response rate for this survey based on past Urban Institute experience conducting similar evaluations of recent grantees as well as techniques described in this Part B on managing non-response rates (drawing from, among others, National Research Council 2013. Nonresponse in Social Science Surveys: A Research Agenda. Washington, DC: The National Academies Press. https://doi.org/10.17226/18293). The evaluation of the Museum Grants for African American History and Culture (AAHC) program had a 53% response rate for grantees, so we adjusted the response rate to reflect this as a baseline.
10 Response rates were estimated using the following logic and evidence:
Grantees and applicants: The Center for Effective Philanthropy reviews of American Foundations tend to have a 40% response rate for grantees. This low response is associated with challenges many organizations have in communicating with them. Based on our history with high response rates for active grantees in other evaluations, and the fact that this is an interview request with active follow-up, we expect a higher response rate in this project, particularly for more recent cohorts.
Funders: The Center for Effective Philanthropy reported sending surveys to 163 CEOs of foundations that had used CEP’s Grantee Perception Report, in 2006, asking about the types of support they provide to grantees, and why; about half responded. Given that this outreach will be more targeted to a more bespoke set of funders who are likely to have heard of IMLS and the AAHC program, we anticipate a higher response rate. We also draw on the fact that we’re asking for interviews and not survey responses which we think we may have more success with since it gives funders an opportunity to talk about their own work and their perspectives on the field.
Other stakeholders: There is relatively limited evidence to reliably estimate response rates for field experts with estimates based on existing Urban Institute projects interviewing key stakeholders (see, for example: https://www.huduser.gov/portal/sites/default/files/pdf/PayforSuccess.pdf), consultation with Urban Institute’s Subject Matter Expert and Urban’s senior survey methodologist, and the fact that interviewees will likely have good familiarity with IMLS and the AAHC program. As such, we anticipate a relatively high (75%) response rate for this group.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Eldridge, Matthew |
| File Modified | 0000-00-00 |
| File Created | 2022-06-23 |