Part A - OMB PRA package for AAHC Evaluation REVISED (04.17.20)

Supporting Statement for Paperwork Reduction Act Submission
Evaluation of the African American History and Culture Grantmaking Program
OMB # 3137-XXXX

Part A. Justification

A1. Necessity of the Information Collected
The study is being instituted as part of IMLS’s statutory mission to conduct analyses, identify
trends, and measure the impact of its programs.1 The study is a new data collection request,
and the data to be collected are not available elsewhere. The data collection activities are
planned for April 2020 through June 2020.
The Institute of Museum and Library Services (IMLS) has contracted the Urban Institute to
conduct an evaluation of the African American History and Culture (AAHC) grant program. The
goals of the AAHC program are to build the capacity of African American museums and support
the growth and development of museum professionals at such museums. The AAHC was
created by an Act of Congress in 2003 (20 U.S. Code § 80r–5) – the same Act that created the
Smithsonian National Museum of African American History and Culture (NMAAHC).
Over the course of 14 grant cycles2, the AAHC program has awarded funds to over 180 projects
that nurture museum professionals, build institutional capacity, and increase access to museum
and archival collections at African American museums and Historically Black Colleges and
Universities (HBCUs). IMLS accepts applications from museums of all sizes and geographic areas
whose primary purpose, as reflected in their mission, is African American life, art, history,
and/or culture.
The AAHC program is organized around a logic model (The Urban Institute’s interpretation of
which is found in Appendix E) with inputs supporting specific activities, leading to outputs that
drive short-term, intermediate, and long-term outcomes; this logic model will serve as the
frame for evaluating the role of this program in building capacity of its African American
museum and HBCU grantees.
This proposed study will be the first evaluation of the AAHC program and its contributions to
grantees’ outcomes. Coupled with a secondary data collection effort which includes reviewing
publicly available information and IMLS administrative data on applications and awarded grants
as well as grantee intermediate and final reports, this evaluation includes three specific primary
1

https://www.imls.gov/sites/default/files/mlsa_2018_asamended.pdf. See 20 U.S.C. § 9108.

2

The most recent grants were awarded in May 2019.

1

data collection activities: (A) survey of grantees, (B) survey of nonapplicants, and (C) interviews
with funders, applicants, grantees, IMLS staff, and other stakeholders.
Grantee survey
This survey will chiefly evaluate the performance of the AAHC program by understanding the
role of the program in enhancing grantee capacity. See Objective 2, Section A2. This survey will
ask questions that aim to capture the activities and short-term outcomes (as listed in Appendix
E, the AAHC Program Logic Model) of recent grantees.
This survey will be sent to grantees selected in 2014-2019 who have completed at least nine
months of project work in order to ensure accuracy in survey results (given recall bias and high
probability of staff turnover). See Appendix B for survey instrument.
Nonapplicant survey
Nonapplicants are defined as organizations which are eligible for AAHC funding based on the
program’s eligibility criteria but which have never applied for a grant (those who have applied
at least once and have never received funding are considered “applicants”).
A subset of AAHC program nonapplicants will receive a short survey that will ask questions
about their awareness of the AAHC program, barriers to application, and current funding
sources. It’s anticipated that there will be a high non-response rates among this population. To
mitigate this, the team will partner with the leading association in the field – the Association of
African American Museums (AAAM) – to secure their assistance in promoting the survey,
including through their social media channels. The survey language itself will be concise and
emphasize the short time required to complete it (under five minutes) and the potential benefit
to their field and organizations like them. Multiple attempts will be made to engage target
respondents (at least two follow-up emails). The team will also leverage word-of-mouth
referrals to boost response rates. See Appendix C for Nonapplicant Survey.
Interviews
Interviews with up to 50 respondents will collect rich information on program barriers, results,
opportunities, and needs. These interviews will each be approximately 45 to 60 minutes in
length and will be semi-structured, using guiding language, questions, and prompts. The
interviews will be customized to the specific respondent, informed by their responses to the
survey (if applicable), and adapted on the spot to elicit the most useful insights.
Specifically, it will collect detailed information on, depending on the respondent group,
experiences with the AAHC application process, barriers to participating and implementing the
grant, and the program’s administrative management; perspectives on the AAHC program’s
alignment with needs in the field; contributions of the AAHC program to specific grantee
outcomes, including both short-term and long-term outcomes; and opportunities for program
improvements. See Appendix D for the semi-structured interview guides.
2

A2. Purposes and Uses of the Data
The purpose of this evaluation is to provide insights into AAHC program management and
outcomes supported by the program. This is not an audit of grantees or their individual
performances. The three core evaluation objectives and their associated research questions
are:
1. Develop a thorough understanding of the eligible population of African American
museums and Historically Black Colleges and Universities (HBCUs), the characteristics of
successful grantees compared with this wider population of institutions, and what
potential steps could be taken to expand the pool of eligible, prospective grantees.
Research questions:
a. What is the universe of African American museums and HBCUs? What share have
participated in the AAHC program?
b. What is the universe of government and philanthropic funders of AA museums on a
national-level scope?
c. Are there any key factors that distinguish between grantees and nonapplicants?
d. Are there any key factors that distinguish between applicants and grantees?
e. What might IMLS do to better connect to all potential applicants? What tradeoffs might
it need to make to achieve this result given any other competing objectives?
f.

How has this program adapted to evolving needs of organizations eligible to receive an
AAHC grant from IMLS?

2. Evaluate the performance of the AAHC program, including understanding the role of the
AAHC program in enhancing grantee capacity, an analysis of portfolio performance, an
understanding of AAHC program’s administrative practices, and the potential for
embedding evaluation into the program.
Research questions:
a. How has this IMLS AAHC program made a difference in the capacity of the nation’s
African American museums and HBCUs?
b. Are there certain parts of the AAHC grant portfolio that have performed better? For
instance, how has the performance of small grants for small institutions compared to
large grants for large institutions?

c. How have IMLS administrative practices for the AAHC program influenced:
i. Participation of applicants? Why?
3

ii. Grantees’ implementation of project awards? Why?
d. How can program evaluation be a more integral part of the grant program and not an
optional or additional feature for grantees?
e. What has happened to the museums (and staff) that received funding in the early
iteration of the program? What have they accomplished post-grant?

3. Interpret the programmatic evaluation’s findings in relation to AAHC program goals, as
outlined in its enabling legislation.3
Research questions:
a. How has the AAHC grant program, now in its 13th year, performed overall in
meeting its legislative goals?
b. What are the barriers to the AAHC program achieving better outcomes?

This research is intended to improve IMLS’s understanding of the contributions of the AAHC
program to grantee outcomes, inform its efforts to strengthen the program, expand the pool of
applicants in future years, and enable it to effectively communicate the program’s
contributions to prospective applicants, policymakers, and other audiences.
The stakeholders likely to be interested in the findings from this research are diverse and many.
Audiences include IMLS, the AAHC-eligible population (especially grantees and past or
prospective applicants), the public, Congress, the Administration, the Smithsonian NMAAHC,
funders of eligible entities, and other local and national partners. The broader Association of
African American Museums (AAAM) community and stakeholders are likely to also be
interested in the evaluation’s findings. The evaluation will be structured and executed with this
large, diverse stakeholder audience in mind and the evaluation team will look for opportunities
to effectively convey insights and stories targeted for their consumption and use.

A3. Use of Information Technology
IMLS takes its responsibility to minimize burden on respondents very seriously and designed
this project with that goal in mind. All administrative documents are in a secure digital format,
accessible to only the Urban Institute research team. The evaluation systematically begins the
investigation in reviewing digital administrative documents before proceeding to collecting
additional information of grantees and other stakeholders through surveying and interviewing.
3

National Museum of African American History and Culture Act (2003), 20 U.S. Code § 80r–5, B.

4

By designing web-based surveys using Qualtrics (a survey software tool), the Agency has
eliminated many hours of labor that would have been required to administer using a different
mode. Qualtrics is a user friendly, customizable qualitative data collection tool that enables
detailed analyses of survey responses. The electronic surveys, and all communication about
them, will be compliant with Section 508 of the Rehabilitation Act. Respondents will be given
the option of receiving a paper survey.
The Urban Institute team will collect interview data via telephone, with concurrent notetaking
(a rough transcript) and audio recording (via 8x8 teleconference software or a handheld audio
recorder placed next to the speaker). Before recording, the Urban Institute team will obtain the
interviewee’s consent to be recorded. Audio files will be stored on a secure drive accessible
only to the Urban Institute investigators and will be destroyed at the end of the study.
The results of the project will be shared with the target audience (existing and prospective
grantees, policymakers, key partners, funders, academics, and others) and the public via a
dissemination strategy, with the final report accessed via the IMLS website and the Urban
Institute website.
A4. Efforts to Identify Duplication

There are no previous efforts to collect the same information from these respondents. For
example, no existing data source includes information from eligible nonapplicants about why
they have chosen not to apply.
The evaluation is initially analyzing secondary data, particularly IMLS administrative project
grant reports, to inform subsequent collection of primary data in avoiding duplication of
information. These secondary documents provide important pieces of evidence about funded
activities, challenges encountered in implementing projects, and outcomes observed.
However, they provide insufficient information alone to address the evaluation’s research
questions and are not structured to ask consistent questions related to the AAHC program’s
processes and contributions to project outcomes. To meet the goals of the evaluation,
additional, new data collection is required.

A5. Methods Used to Minimize Burden on Small Businesses
There are no small businesses involved in this data collection.
Some survey and interview respondents are expected to be from small entities (e.g., small
museums with limited personnel capacity). In order not to overly burden these entities, the
evaluation will provide clear, concise instructions and ensure the data collection process limits
requests for personnel time or the need to collect additional, follow-up information. The
5

instruments are designed to minimize respondent burden: the grantee survey is designed to
take 30 minutes, the nonapplicant survey is designed to take less than 5 minutes, and the
interviews will each take 60 minutes or less. Only one person from each organization will be
asked to complete a survey or participate in an interview. The grantee survey was reviewed by
three experts in the field (not drawn from the sample population but reflecting a similar profile)
to trouble-shoot technical issues and reduce the time burden on respondents.
To further reduce burdens, interviews will be conducted by phone, scheduled at the
convenience of the interviewee, and kept to a minimal amount of time (and not longer than 60
minutes).
A6. Consequences of Less Frequent Data Collection
This is a voluntary data collection effort undertaken over several months. The study will provide
IMLS with insights into the performance of the AAHC program and its contributions to grantee
outcomes. Without this evaluation, IMLS would have no comprehensive, reliable information to
answer key questions on program performance with which to report to key stakeholders
including eligible organizations, policymakers, and other partners. Conducting the collection
less frequently (essentially forgoing the collection) would impede the evaluation’s ability to
provide meaningful insights, depriving grantees and other key stakeholders the opportunity to
learn about the contributions and performance of the AAHC program.
A7. Special Circumstances
The proposed data collection activities are consistent with the guidelines set forth in 5 C.F.R.
Part 1320 (Controlling Paperwork Burdens on the Public). There are no special circumstances
that require deviation from these guidelines.
A8. Consultations Outside the Agency
No comments were received on the Notice of Proposed Information Collection Request, 84 FR
44942, pp. 44942-44943, dated August 27, 2019.
IMLS published a notice in the Federal Register with a 30-day public comment period to
announce this proposed information collection being submitted to OMB on February 5, 2020,
85 FR, pp.6583-6584.
The evaluation design, including data collection instruments, has been developed with the
Urban Institute evaluation team, including senior methodologists, and a review by Urban’s
Institutional Review Board. The evaluation study plan and draft instruments have also been
reviewed by a subject matter expert on African American museums, who provided
recommendations to improve the availability of additional data, the clarity of questions and
instructions, and the need for specific data elements.
6

The survey instrument was reviewed by three experts in the field, similar to but distinct from
those in our target sample. This review led to the revision and consolidation of several
questions to improve the survey’s clarity and reduce the time burden on respondents.
A9. Payments of Gifts to Respondents
No incentives, or other payments or gifts, will be offered to survey or interview participants.
A10. Assurance of Confidentiality
Before beginning any survey or interview, stakeholders will be provided an explanation of the
purpose of the evaluation and how their responses will be used. Respondents will be told that
their individual responses will be de-identified, and will be publicly reported only in the
aggregate. However, they will also be told that unique responses could be potentially
identifying, and the Urban Institute evaluators cannot promise anonymity. In cases where
Urban Institute’s evaluators want to include a quote from an individual respondent for clarity or
illumination, they will seek permission from that individual to use the quote.
Access to data will be password-controlled and limited to those staff involved in fielding the
surveys and interviews and who have signed a confidentiality agreement. All survey and
interview data will be saved to an encrypted network drive, with access limited to Urban
Institute staff with a need to work with raw data, and who have signed the confidentiality
agreement. Access will only be available on-site or secure remote access, through passwordprotected computers.
The survey research instruments and interview protocols have been reviewed and approved by
the Urban Institute’s Institutional Review Board (IRB) prior to initiating any research. Urban
Institute’s IRB operates according to the Common Rule on the Protection of Human Subjects
found in Title 45 of the Code of Federal Regulations, Part 46 (45 C.F.R. Part 46). The information
requested under this collection is protected and held private in accordance with 42 U.S.C.
§ 1306, 20 C.F.R. Parts 401 and 402, 5 U.S.C. § 552 (Freedom of Information Act), 5 U.S.C.
§ 552a (Privacy Act of 1974) and OMB Circular No. A-130. The IRB submission included the
following information:
•
•
•
•
•
•

the purpose of the data collection and data collection methods being used;
the respondent populations and how they will be identified and accessed;
whether the data will be anonymous, confidential, or neither, and if the data are
confidential or neither, explanation of why identifiers are necessary;
how data will be stored (e.g., electronic files, hard copies);
who will have access to data and for how long; and
potential risks and burdens of the project to participants.

7

Data sets provided to IMLS at the end of the study will not contain any personally identifying
information (PII)—such as name or address of respondents or their organizational affiliation—
that could permit disclosure or identification of respondents, directly or by inference. The
Urban Institute will destroy all personally identifiable information at the end of the study.
A11. Justification for Sensitive Questions
No questions of a sensitive nature will be included.
A12. Estimates of Hour Burden to Respondents
Table A.1 and Appendix F present the projected burden hour estimates for data collection for
the grantee and nonapplicant web-based surveys as well as for the interviews with grantees,
applicants, and other stakeholders. The estimates included in Table A.1 are based on estimates
for the time needed to complete these data collection activities. For the surveys which are
sampling a larger population, the evaluation is using estimated response rates as presented in
Table B.1. The evaluators assume that the contact person for grantees, nonapplicants,
applicants, and other stakeholder categories are similar and each earn approximately $28.12
per hour. 4 It is assumed that the contact person for funders will earn approximately $59.56 per
hour.5
All tools will be single use (i.e. invited respondents will only be asked to complete it once),
however some grantee survey respondents will be invited to voluntarily participate in the semistructured interview.
The grantee survey will take up to 30 minutes to complete (validated through the review pilot)
and the evaluators will invite all grantees from 2014-2019 to complete it (there are
approximately 74 distinct grantees, of which 56, or 75 percent, are expected to complete the
survey). 6 This equates to a 28-hour time burden for this tool at a total cost of $787.36.
The nonapplicant survey will take approximately 5 minutes to complete. The evaluators will
invite 300 organizations to complete the survey and anticipate that up to 150 will respond,

“Occupational Employment Statistics: Occupational Employment and Wages, May 2018 – 25-4012 Curators,” Bureau
of Labor Statistics, accessed December 13th, 2019, https://www.bls.gov/oes/current/oes254012.htm

4

“Occupational Employment Statistics: Occupational Employment and Wages, May 2018 – 11-1021 General and
Operations Managers,” Bureau of Labor Statistics, accessed December 13th, 2019,
https://www.bls.gov/oes/current/oes111021.htm

5

As described in Part B, question B1, The evaluators derived this response rate estimate based on Urban Institute’s
past experience with surveys of grantees on behalf of the grant program. This high response rate is due to the
familiarity of grantees with the program and their interest in and awareness of benefits in responding to the request.
Limiting the survey to recent grantees (post-2014) also improves the response rate.

6

8

equating to an approximately 12-and-a-half-hour time burden for this tool at a total cost of
$350.09.
The evaluation plan is for 50 interviews total, across all groups.7 The interviews will take
approximately 1 hour each, equating to a 50-hour time burden for this tool at a total cost of
$1,447.40. (Time burdens for IMLS staff interviews are included in Section A14, Estimates of
Costs to the Federal Government).
TABLE A.1
Burden Estimates
Predicted or
Targeted
Responses 8
56

Burden
Hour Per
Response
0.5

Total
Burden
Hours
28

Hourly Cost
per
Response
$28.12

Cost
$780.33

150

0.083

12.45

$28.12

$350.09

Interviews with
applicants

5

1.0

5

$28.12

$140.60

Interviews with
grantees

33

1.0

33

$28.12

$927.96

Interviews with
funders

4

1.0

4

$59.56

$238.24

Interviews with
other stakeholders

5

1.0

5

$28.12

$140.60

Instrument
Web-based
Grantee Survey
Web-based
Nonapplicant
Survey

Total

256

90.20

$2,577.82

A13. Estimates of Cost Burden to Respondents
There are no additional cost burdens to respondents beyond the labor cost of burden-hours
described in item A12 above.

Response rates for interviewees are provided in Part B, question B3, and vary from 50 percent for applicants to 90%
for post-2014 grantees.

7

For the survey, the responses are presented as estimated (i.e., total population multiplied by the estimated response
rate). For the interviews, the responses are presented as targeted (i.e., the population will be randomly sampled until
the target number is hit).

8

9

A14. Estimates of costs to the Federal Government.
The total one-time estimated cost of this evaluation to the Federal Government is $275,764,
with the third-party evaluation contract of $227,317 and IMLS staff costs of $48,447 (693
projected hours across 10 IMLS employees).
A15. Reasons for Program Changes or Cost Adjustments
This is a new information collection request.
A16. Project Schedule
Grantee Survey
The Urban Institute will conduct the Grantee Survey a single time, beginning approximately in
April 2020 or as soon as OMB approval is received and ending approximately 1 month later. The
start of data collection will be preceded by an introductory Respondent Contact Letter to
respondents from IMLS to explain the importance of the evaluation, participation in data
collection, and the role of the Urban Institute (Appendix A).
The Urban Institute will contact target grantees to initiate the survey, with information
reiterating the purpose of the survey and use of their responses (Appendix B). This email will
direct respondents to the survey on Qualtrics, which will include introductory language and
consent language. Respondents must indicate their consent to complete the survey. The survey
instrument language is in Appendix B.
Survey results will be analyzed following completion of the survey and preliminary analysis is
expected to be completed approximately 2 weeks after end of the survey data collection.

Nonapplicant Survey
The Urban Institute will conduct the nonapplicant survey a single time, beginning in April 2020
or as soon as OMB approval is received and ending approximately 1 month later. This will be
the only time that this target population will be asked to provide data. The Urban Institute will
make initial contact with these respondents since there may not be name recognition for IMLS
(or Urban) (Appendix C). This outreach will briefly explain the purpose of survey, use of data,
value/benefit to respondents, and the short time commitment. This email will also include a
link to the survey. Other methods of disseminating the survey will be used, such as leveraging
personal connections (word-of-mouth) of the project’s SME and IMLS’s partners including the
social media presence of the Association of African American Museums.
10

Key Respondent Interviews
The Urban Institute will conduct up to 50 single-time interviews with respondents from across
funders, applicants, grantees, IMLS staff, and other stakeholders. This effort will begin in April
2020 or as soon as OMB approval is received and ending approximately 2 month later. The start
of data collection will be preceded by an introductory Respondent Contact Letter to
respondents from IMLS to explain the importance of the evaluation, participation in data
collection, and the role of the Urban Institute (Appendix A). (Grantees who were invited to
participate in the 2014-2019 grantee survey will only receive that first Respondent Contact
Letter).
The Urban Institute will contact target respondents to initiate the survey, with information
explaining the purpose of the interview,
The Urban Institute will contact target respondents to participate in an interview, with
information reiterating the purpose of the interview, the use of their responses, and the types
of questions that will be asked and inviting them to participate by indicating their availability
(Appendix D). These interviews will be semi-structured and will be tailored and adapted to each
respondent’s unique context and background as well as the natural flow of the interview.
Indicative interview questions showing areas of interest for each interview population are
included in Appendix D.
Interview data will be analyzed on a rolling basis and analysis is expected to be completed
approximately 2 weeks after end of the interview data collection period.
Report and Publication
In July 2020, draft findings from the primary data collection will be included in an interim report
(the project’s second interim report 9) that’s shared with IMLS and formally presented in-person
at a briefing and discussion. Feedback from IMLS and identified subject matter experts will be
integrated into the report and a final draft of this interim report will be delivered to IMLS in
August 2020.
The findings from both interim reports (i.e., findings from both the primary and secondary
research) will be presented in a final project report, designed to address all of the evaluation’s
research questions. A draft of this final report will be shared with IMLS in September 2020
followed by an in-person briefing and discussion of findings. After two rounds of review and
revision, a final report will be shared with IMLS in November 2020, ready for publication.
TABLE A.2
Project timeline
The first project interim report will reflect findings from the secondary data collection and analysis. This first report
will be delivered to IMLS in February 2020 in draft form and finalized, after IMLS feedback in March 2020.

9

11

Project phase

Task

Due

Launch and
evaluation design

Initial project launch

9/15/2019

Project Management Workplan submitted to IMLS

10/1/2019

Formal project kickoff with IMLS and Urban Institute

10/07/2019

Approval of Evaluation Study Plan (ESP)

12/16/2019

PRA submitted to OMB for approval

1/31/2020

Provide IMLS with complete list of databases andmerge,
framework of data points to collect, data requested from IMLS,
and frame for reviewing literature *

12/09/2019

Launch secondary research (after approval of ESP)

12/13/2019

Draft consolidated database of eligible population (with
characteristics)

1/17/2020

Completed review of secondary literature (including IMLS
admin data)

1/17/2020

Analysis of consolidated database and literature review for
Objective 1

1/24/2019

Analysis of secondary research insights relevant for Objectives
2 and 3

2/5/2019

Draft first interim report and presentation material due to IMLS

2/12/2020

Presentation of first interim report at IMLS*

03/05/2020

Final survey and interview protocols due to IMLS *

12/13/2019

PRA submitted to OMB

5/15/2020

Launch primary research collection (anticipated OMB approval
date)

4/27/2020

Launch survey

5/7/2020

Survey ends

5/28/2020

Analysis of survey results

6/11/2020

Develop target interview list and share with IMLS

4/30/2020

Launch scheduling of interviews

5/14/2020

Conclude interviews

7/10/2020

Analysis of interview results

7/24/2020

Draft second interim report and presentation material due to
IMLS

8/7/2020

Secondary
research

Primary research

12

Final work
products and
dissemination

Presentation of second interim report at IMLS*

8/21/2020

Informal listening opportunity with grantees at AAAM
conference

8/8/2020

Delivery of final draft report and draft presentation in-person
presentation

10/20/2020

Presentation of findings at IMLS

11/03/2020

Feedback on presentation materials

11/03/2020

Amended presentation materials

11/10/2020

Round 1 of feedback on report

11/03/2020

Response to round 1

11/17/2020

Round 2 of feedback on report

12/01/2020

Response to Round 2 (i.e. final report)

12/15/2020

A17. Request to Not Display Expiration Date
IMLS will display the expiration date of OMB approval and OMB approval number on all
instruments associated with this information collection, including forms and questionnaires.
A18. Exceptions to the Certification
No exceptions are necessary for this information collection.

13