3145-CCI Supporting Statement Part A

Supporting Statement for OMB
Clearance Request
Parts A and B
Evaluation of the Centers for
Chemical Innovation
(CCI) Program

Executive Summary: Evaluation of the Centers for Chemical
Innovation (CCI) Program
Introduction

The National Science Foundation (NSF) established the Centers for Chemical Innovation (CCI) Program
(formerly known as Chemical Bonding Centers) in 2004 to support research centers focused on major,
long-term fundamental chemical research challenges. The goals that NSF set forth for the CCI Program
include that Centers will (a) produce transformative research, leading to innovation, and attract broad
scientific and public interest; (b) be agile structures that can respond rapidly to emerging opportunities
through enhanced collaborations; and (c) integrate research, innovation, education, broaden participation,
and informal science communication. The program supports research centers based at academic
institutions or non-profit, non-academic organizations via a two-phase funding model. Phase I centers
receive a total budget of $1.8 million and are funded for three years. 1 The awardees are expected to
engage in research and activities leading to broader impacts. 2 In addition, Phase I CCIs develop the
policies, programs, and infrastructure necessary for a much larger Phase II center. Within 15 months of
the Phase I award, grantees submit a strategic plan (a key deliverable for this phase), describing all
aspects of CCI operations, which is reviewed by NSF and external reviewers. Phase I CCIs are eligible to
compete for Phase II awards, with a total funding level of up to $20 million for five years, and a
possibility of one competitive renewal of the same size. Phase II CCIs are expected to build upon the
results achieved in Phase I, to ultimately address important challenges in fundamental chemistry, and to
contribute to society.
Between 2004 and 2017, the CCI Program funded 28 Phase I grants. Among these, nine successfully
proceeded to Phase II, and six of the nine received competitive renewals.
The National Science Foundation contracted with Abt Associates (Abt) and its partner, NET ESolutions
Corporation (NETE), to conduct an evaluation of the outcomes of the CCI Program and implementation
factors (e.g., Center-level structures and operations, program-level funding model) contributing to
research, collaboration, and broader impact activities and outcomes. This is an application to the Office of
Management and Budget under the Paperwork Reduction Act for approval of data collection instruments
associated with the evaluation. The instruments in this application include: (1) CCI Principal Investigator
(PI) and Co-Investigator (Co-I) Survey; (2) CCI Phase II Principal Investigator (PI) and Co-Investigator
(Co-I) Interview; (3) CCI Graduate Student and Postdoctoral Researcher Survey; and (4) CCI Center
Industry Partners Interview. The CCI Program has been in existence for nearly 15 years but has
1
2

https://www.nsf.gov/pubs/2018/nsf18555/nsf18555.pdf
The American Innovation and Competitiveness Act (2017) defines broader impacts as 1) increasing the
economic competitiveness if the U.S., 2) advancing the health and welfare of the American public, 3)
supporting the national defense of the U.S., 4) enhancing partnerships between academia and industry in the
U.S., 5) developing an American STEM workforce that is globally competitive through improved kindergarten
through grade 12 STEM education and teacher development, and improved undergraduate STEM education and
instructions, 6) improving public scientific literacy and engagement with science and technology in the U.S.,
and 7) expanding participation of women and individuals from underrepresented groups in STEM. See
legislation: S. 3084 (114th) American Innovation and Competitiveness Act (2017). Section 526(a) of the
America COMPETES Reauthorization Act of 2010 (42 U.S.C. 1862p–14(a)42 U.S.C. 1862p–14(a)42 U.S.C.
1862p–14(a)42 U.S.C. 1862p–14(a)42 U.S.C. 1862p–14(a))
2

never been systematically evaluated. As this is the first assessment of the CCI program, new data
collection is necessary to provide critical evidence to assess the CCI Program’s progress in achieving its
goals, to communicate the outcomes of the program, and to inform improvements in CCI Program and
Center-level design and operation. Across the NSF, the evaluation will also inform planning decisions
about the center-based research concept and phased funding mechanisms. Additionally, the evaluation
findings will be used to communicate the outcomes of the CCI program to the wider chemistry
community.

3

Part A: Justification
A.1. Explain the circumstances that make the collection of information necessary

This Information Collection Request (ICR) seeks to establish a new information data collection,
consisting of two surveys and two interview protocols to support the Evaluation of the Centers for
Chemical Innovation (CCI) Program.
The goals of the CCI Program can be divided into two categories that correspond to the NSF merit review
criteria of Intellectual Merit and Broader Impacts: 1) advance knowledge in chemistry (Intellectual Merit)
and 2) benefit society and contribute to desired societal outcomes (Broader Impacts). The CCI Program
directly contributes to NSF’s strategic goals as outlined in the report “Investing in Science, Engineering,
and Education for the Nation’s Future – NSF Strategic Plan for 2014-2018.” The CCI Program
principally aligns with Strategic Goal 1, Objective 1: to transform the frontiers of science and engineering
by investing in fundamental research to ensure significant advances across science, engineering, and
education. The program also aligns with Strategic Goal 1, Objective 2: to integrate education and research
to support development of a diverse STEM workforce and cutting-edge capabilities. The broader impacts
of innovation, higher education and professional development, broadening participation, and informal
science communication also align with Strategic Goal 2, Objective 1: to stimulate innovation and address
societal needs through research and education, specifically by strengthening the links between
fundamental research and societal needs through investments and partnerships.
The National Science Foundation contracted with Abt Associates (Abt) and its partner, NET ESolutions
Corporation (NETE), to conduct an evaluation of the outcomes of the CCI Program and implementation
factors (e.g., Center-level structures and operations, program-level funding model) contributing to
research, collaboration, and broader impact activities and outcomes. The evaluation will cover CCI Center
awards starting in fiscal year 2004, with an emphasis on Phase II Centers. The CCI Evaluation aims to
address five key research questions:
1. What are the important contributions of the CCI Program to our current understanding of
fundamental chemistry?
2. How successful have the CCI centers been at transferring their basic research results into societal
or economic benefits (innovation)?
3. What are the contributions of the CCI Program in the areas of workforce development (education
and professional development), broadening participation, and informal science communication?
4. How effective are the center structures and operations in achieving the program’s goals?
5. How effective is the two-phase funding model for the CCI Program?
The evaluation design combines data collection from secondary sources, including publications and
administrative data, with online surveys and phone interviews, which are the subject of this ICR.
Combined, the data from primary and secondary sources will give a more complete picture of the CCI
Program- and project-level processes, outcomes, and lessons learned than could be possible with either
source alone.
Overview of Study Design and Data Collection Strategy

To address the key research questions, Abt proposed a mixed-methods evaluation, which will rely on both
extant administrative and public data as well as new data, as follows.
4

Administrative and public data includes the following:
a. Coding of grantee-produced administrative data (including grant proposals and annual Research
Performance Progress Reports (RPPRs)), to capture intended and realized management
structure(s), accomplishments and outputs, activities, participants and collaborators, and changes
in plan (if any) occurred, primarily from Principal Investigators’ perspectives.
b. Coding of NSF-produced administrative data (including oversight/funding memos, site visit
reports, internal interim reviews, and committee of visitor reports), to capture some external-tothe-Center perspective on grant accomplishments, challenges, and recommendations.
c. Analysis of public publication records (bibliometrics) for CCI PIs and Co-Investigators as well as
for a comparison group of PIs on individual-investigator grants funded by NSF’s Chemistry
Division, to measure research productivity and research networks.
This ICR seeks approval for data collection for the following: (1) CCI Principal Investigator (PI) and CoInvestigator (Co-I) Survey (PIs); (2) CCI Phase II Principal Investigator (PI) and Co-Investigator (Co-I)
Interview; (3) CCI Graduate Student and Postdoctoral Researcher Survey; and (4) CCI Center Industry
Partners Interview. Details on each component of the data collection appear below. The surveys and
interview protocols were previously tested under a Fast Track clearance, OMB Control #3145-0215.
1. CCI Principal Investigator (PI) and Co-Investigator (Co-I) Survey. Surveys of CCI Phase I and
Phase II PIs and Co-Is will be used to understand the role of the center in facilitating achievement
research, collaboration, and broader impacts, to assess grantee satisfaction with the center
structure and a two-phase funding model, to document outcomes, and to describe challenges
encountered. The survey request and reminder emails, consent language, and survey instrument
appear in Appendix A.
2. CCI Phase II Principal Investigator (PI) and Co-Investigator (Co-I) Interview. Interviews of
Phase II CCI PIs and Co-Investigators will be used to further explore the data emerging from the
survey. Up to 27 CCI Phase II Center PIs and Co-Investigators will receive requests to complete
both a survey and an interview. The interview request email, consent language, and omnibus
interview protocol appear in Appendix B, with notation of items to be delivered to PI and to CoInvestigators.
3. CCI Graduate Student and Postdoctoral Researcher Survey. Surveys of graduate students and
postdoctoral researchers who contributed to and/or received financial support under Phase I and
Phase II CCI center activities will be used to understand the role of CCI in education, training,
and career development. The survey request and reminder emails, consent language, and survey
instrument appear in Appendix C.
4. CCI Center Industry Partners Interview. Interviews of industry partners that contributed to Center
activities will be used to capture knowledge exchange with CCIs and other benefits of partnership
as well as perspective on CCI contributions to chemistry and society. This data collection will
inform NSF about whether CCI is enhancing partnerships between academia and industry and
therefore meeting broader impact goals. The interview request email, consent language, and
interview protocol appear in Appendix D.

5

New data collection from primary sources is necessary for the following reasons:
•

Because the RPPRs are standardized across all NSF programs, PIs have some degree of latitude
in reporting on their activities, accomplishments, outputs, and broader impacts. Thus, what is
reported and the nature and level of detail is not systematic across all Centers or reporting
periods and do not provide a complete dataset.

•

Information included in administrative data covers only the award period and, therefore, does not
fully address the contribution of the program to longer-term outcomes accumulating over time,
such as societal or economic benefits, workforce development, public education, or the perceived
effectiveness of the two-phase funding model in advancing science, collaboration, and broader
impacts.

•

Surveys and interviews with CCI PIs and Co-Investigators would allow an examination of the
participant experience, the role of the Centers in advancing science, challenges encountered,
lessons learned, effective center structures and management, and other topics.

•

Surveys of graduate students and postdoctoral researchers are the only source of data to examine
in detail the experience of these groups and the role of CCI in their career development. This is an
important goal of the program, but limited information on this topic is available from
administrative data.

•

The role of CCIs in advancing industry connections and economic development will be explored
in interviews with industry partners. This is an important goal of the program, but limited data on
this topic are available from administrative data, and only from the perspective of the Center PI.

Topics covered in the PI/Co-Investigator survey and interview that are not fully available in the
administrative data include:
• Characteristics of research projects
• Collaborations
• Benefits of participation to research progress, workforce development, broadening participation,
and public engagement
• Benefits to industry and research community
• Changes in publication behavior and personal visibility
• Satisfaction with and role of center structure, management and organizational strategies
• Sustainability of activities and accomplishments
• Challenges encountered because of center structure
• Role of and satisfaction with Phase I
• Benefits and limitation of a two-phase model
Topics covered in the graduate student/postdoc survey that are not available in administrative data
include:
• Research and professional development opportunities made available through CCIs
• Mentorship received
• Work in a lab/research group of CCI partner
• Center activities in which individual participated
• Satisfaction with CCI participation

6

•
•
•
•

Benefits of CCI participation
Reasons for leaving CCI project
Career status and plans and influence of CCI on career goals
Demographic characteristics

Topics covered in the industry partners interviews that are not available in administrative data include:
• Benefits to industry
• Center activities in which individual participated
• Benefits of CCI participation
• Collaborations
• Sustainability of activities and accomplishments
• Challenges encountered because of center structure

Exhibit 1 shows how the information being collected from surveys and interviews is linked to the primary
research questions and their intended use. The interview and survey protocols can be found in Appendices
A-D. The program logic model, which informs the evaluation questions, administrative data coding
schema, and new data collection instruments and protocols appears in Appendix E.

7

Exhibit 1: Research Questions Linked to New Data Collection Strategies
Research question
PI/Co-Investigator survey
Graduate
CCI PI interviews
student/postdoc survey
1. What are the important
Q4 – interdisciplinary and
Q4 – change in scientific
contributions of the CCI
transformative characteristics
program and productivity
Program to our current
of funded projects
due to CCI participation
understanding of
Q5-7 – role of pre-existing
Q5 – types of research best
fundamental chemistry?
collaboration and contribution
suited to the center model
of participation in CCI to
Q6 – most important
collaboration
scientific accomplishments
of CCIs
Q8-10 – program role in
Q8 – how did CCI
advancing research,
demonstrate leadership in
improving personal
the field
visibility/productivity, and
Q12 – reputation of CCI in
benefiting the research
and benefits to the chemistry
community through
community
capacity/infrastructure
building
2. How successful have
Q14 – program benefits to
Q6 – most important
the CCI centers been at
industry
economic benefits resulting
transferring their basic
from CCI
research results into
societal or economic
benefits (innovation)?

3. What are the
contributions of the CCI
Program in the areas of
workforce development,
broadening participation,

Q11-13 – program role in
workforce development,
broadening participation, and
informal science
communication

Q7 – mentorship during
CCI participation
Q8 and 12 – role of CCI
participation in career
choices

Q6 – most important
workforce development and
public education benefits
resulting from CCI

Industry partner
interviews

Intended Use
NSF CCI Program
Officers, Division of
Chemistry Leadership and
Staff: Understand and
communicate progress in
achieving goals and
outcomes of the CCI
Program
Broader Chemistry
Community: Communicate
outcomes of the program

Q1-3 – nature and extent
of partnership
Q4 – benefits of
partnership to industry

NSF CCI Program
Officers, Division of
Chemistry Leadership and
Staff: Understand and
communicate progress in
achieving goals and
outcomes of the CCI
Program
Broader Chemistry
Community: Communicate
outcomes of the program
NSF CCI Program
Officers, Division of
Chemistry Leadership and
Staff: Understand and
communicate progress in

8

Research question

PI/Co-Investigator survey

and informal science
communication?

4. How effective are the
center structures and
operations in achieving
the program’s goals?

5. How effective is the
two-phase funding model
for the CCI Program?

Q15-16 – role of center
structure in success
Q17-19 – satisfaction with the
center and challenges
experienced

Q20-25 – role of Phase I and
benefits/limitations of twophase model

Graduate
student/postdoc survey
Q9-11 – current career
status
Q13-16 – CCI experiences
and satisfaction with these
experiences
Q17-18 – role of CCI
participation in quality of
training and skill
development
Q19-20 – demographic
characteristics

CCI PI interviews

Industry partner
interviews

Intended Use
achieving goals and
outcomes of the CCI
Program
Broader Chemistry
Community: Communicate
outcomes of the program

Q1-3, Q9 – reason to
participate in CCI and history
of partnership
Q10 – evolution of the CCIs
Q11 – sustainability of CCIs
Q13 – most and least
effective organizational
components of CCIs
Q14 – participation
challenges
Q15-16 – challenges and
advantages of center model
Q17-18 – strengths and
weaknesses of two-phase
model

Q5 – sustainability of
partnership
Q6 – participation
challenges

NSF Directorates: Inform
planning decisions about
the center-based research
concept
CCI Program Officers:
Inform CCI Program
design and operations
CCI Center PIs and Coinvestigators: Inform
Center- and Project- level
design and operations
NSF Directorates: Inform
planning decisions about
the center-based phased
funding mechanisms

CCI Program Officers:
Inform CCI Program
design and operations
Note: Questions 1-3 in the PI/Co-Investigator survey and 1-4 in the graduate student/postdoctoral researcher survey are for verification purposes and to enable
subgroup comparisons of the data.

9

A.2. Indicate how, by whom, and for what purpose the information is to be used

The CCI Program has been in existence for nearly 15 years but has never been systematically
evaluated. The CCI Program is a significant investment for NSF and it is critical that the findings of the
evaluation are as widely used as possible. The following knowledge will be generated:
•

For CCI Program Officers, Division of Chemistry leadership, and other Directorates at NSF: the
contribution of the program to advancing fundamental chemistry, improving connections to
industry, and providing professional development and career opportunities to junior scholars;
experiences of participants; effective and ineffective center structure and management processes;
and the benefits/limitations of this unusual two-phase funding model.

•

For CCI Center PIs and Co-Investigators: more and less effective center structure and
management processes.

•

For the broader chemistry community: the contribution of the program to advancing
fundamental chemistry, improving connections to industry, and providing professional
development and career opportunities to junior scholars.

•

For the evaluation community at NSF and beyond: the strengths and weaknesses of various
evaluation strategies for examining the processes and outcomes of large research centers.

As described in Exhibit 1 above, the new data collection from primary sources enables NSF CCI program
officers and Division of Chemistry leadership and staff to more comprehensively examine the processes,
outputs, and outcomes of Centers toward determining the CCI Program’s progress in achieving its goals,
to communicate the outcomes of the program, and to inform improvements in CCI Program and Centerlevel design and operations. The new data collection from primary sources will also inform planning
decisions about the center-based research concept and phased funding mechanisms across NSF
directorates and divisions. Additionally, the findings of the new data collection from primary sources will
be used to communicate the outcomes of the CCI program to the wider chemistry community.
A.3. Describe whether, and to what extent, the collection of information involves the use of
automated, electronic, mechanical, or other technological collection techniques or other
forms of information technology

Abt will conduct the survey using an online, internet-accessed survey software. This will allow
respondents to take the survey at any time within the window of the data collection period and to submit
their responses electronically. There are no paper forms involved in the survey. Burden-reducing features
include:
• Secure personalized access. Each grantee will receive a customized link to the survey. The
survey software allows respondents to save responses and return to the survey later to finish at
their convenience.
• Automated skip patterns. Skip logic embedded in the survey minimizes respondent burden by
omitting non-applicable questions.
• Automated validation checks. The software will check for allowable ranges for numeric
questions, minimizing out of range or unallowable values. This reduces entry errors that may
require follow-up contacts to gather correct information.

10

•

•

Close-ended questions. These types of questions reduce burden on respondents and facilitate
data analysis. A small number of “other” options are included to make sure that respondents have
an opportunity to enter information which does not fit pre-existing options.
Response not required. Most questions will not require a response in order for the respondent to
proceed, save, or submit. A limited number of items designed with survey logic to automate skip
patterns will require responses.

NSF will send an advance e-mail to PIs to inform them of the evaluation and that they, Co-Investigators,
graduate students, and postdoctoral researchers will be receiving surveys via email from the evaluation
team. All communications with respondents, including survey launch and follow-up and interview
scheduling, will be conducted by email to keep respondent burden to a minimum. Survey responses will
be tracked in real time to help guide the follow-up strategy. Data collected in the survey will be analyzed
using a semi-automated process, whereby response frequencies will be generated using SAS software.
Interviews will be digitally recorded with respondent permission and transcribed. Abt will use the
transcripts for accurate recall and efficient content analysis of the interviews. Interview respondents will
not be asked to complete any forms.
The results of the information collection will be made available to the public over the internet (subject to
technical peer review, see A.16 for details). Privacy, confidentiality and data security related to this
information collection are detailed in Section A.10.
A.4. Describe efforts to identify duplication. Show specifically why any similar information already
available cannot be used or modified for use for the purposes described in item 2 above

NSF and Abt made every effort to avoid duplication and to determine how the research questions could
be addressed while minimizing new data collection. Topics of interest not available from administrative
data are described in section A1. Because this is the first systematic external evaluation of the CCI
Program and no other studies have examined the effectiveness of the program, there are no existing
evaluation datasets or other sources which could be reused.
During the design phase of the project, the Abt evaluation team invested significant effort to review
samples of all available administrative data, including grant proposals, annual Research Performance
Progress Reports, program solicitations, merit review summary statements, committee of visitors reports,
and other documents generated by the program to determine what type of information is consistently
available from these sources. In parallel, Abt reviewed published literature and evaluation studies related
to center-based research evaluation to identify appropriate performance metrics. As a result of these
efforts, a framework was developed which listed all indicators relevant to the study and the sources where
these data could be obtained. This mapping process revealed clear gaps in administrative data which
could be filled using new data collection described in this request.
A.5. If the collection of information impacts small businesses or other small entities, describe any
methods used to minimize burden

Some of the CCI Program PIs and Co-Investigators included in the survey and/or interview sample may
have established small businesses in addition to continuing their academic research. From administrative
data, we estimate this to be a small proportion of the study population. Given that a goal of the CCI
Program is to produce transformative research leading to innovation, to fully evaluate program

11

performance it may be necessary to survey and/or interview individuals that have established small
businesses as an extension of their CCI-funded activities.
A.6. Describe the consequence to Federal program or policy activities if the collection is not
conducted or is conducted less frequently, as well as any technical or legal obstacles to
reducing the burden

This is a one-time data collection for the purposes of program evaluation. The data collection efforts
proposed in this ICR will affect most respondents only once. The nine Phase II Center PIs and a sample
of 18 Phase II Co-Investigators will receive both the survey and an invitation to take part in an interview.
Each of the two types of data collection will occur only once.
This is the first time that data will be collected for the purposes of assessing the effectiveness of the CCI
Program. Without the information collection (or a less frequent collection since this is a one-time data
collection), NSF will not be able to assess whether the CCI Program is achieving it goals related to 1)
advancing knowledge in chemistry (Intellectual Merit) and 2) benefiting society and contributing to
desired societal outcomes (Broader Impacts). Without the surveys and interviews, NSF will not have a
clear understanding of program outcomes, potential areas for improvement in CCI program design and
implementation, and center-based research concept and phased funding mechanisms.
There are no technical or legal obstacles to reducing the burden. The designated Contracting Officer is
authorized to approve changes in any of the requirements under this contract.

A.7. Explain any special circumstances that would cause an information collection to be
conducted in a manner: Requiring respondents to report information to the agency more
often than quarterly; Requiring respondents to prepare a written response to a collection
of information in fewer than 30 days after receipt of it; Requiring respondents to submit
more than an original and two copies of any document; Requiring respondents to retain
records, other than health, medical, government contract, grant-in-aid, or tax records, for
more than 3 years; In connection with a statistical survey, that is not designed to produce
valid and reliable results that can be generalized to the universe of study; Requiring the
use of a statistical data classification that has not been reviewed and approved by OMB;
That includes a pledge of confidentiality that is not supported by authority established in
statute or regulation, that is not supported by disclosure and data security policies that
are consistent with the pledge, or that unnecessarily impedes sharing of data with other
agencies for compatible confidential use; or Requiring respondents to submit proprietary
trade secrets or other confidential information unless the agency can demonstrate that it
has instituted procedures to protect the information’s confidentiality to the extent
permitted by law.

There are no special circumstances for the proposed data collection efforts. None of the above special
circumstances apply. While some questions ask respondents about patents and licenses in a general sense,
they do not request details. Furthermore, none of these responses are required.

12

A.8. If applicable, provide a copy and identify the date and page number of publication in the
Federal Register of the agency's notice, required by 5 CFR 1320.8(d), soliciting comments
on the information collection prior to submission to OMB
Federal Register Requests

On May 18, 2018, a 60-day Federal Register Notice was published at 83 FR 23301. No pertinent
comments were received.
Consultations Outside the Agency

To provide input on all aspects of the CCI evaluation, a distinguished external Technical Working Group
(TWG) was recruited in collaboration with NSF. It is composed of five chemistry experts and three
experts in evaluation, bibliometrics, and science, technology, and innovation (STI) policy. The members
of the TWG have expertise aligning to all of the primary chemistry research disciplines of the Phase II
CCIs; are involved in industry; participate in other NSF-funded center leadership and advisory activities;
and hold numerous honors and awards in innovation, research, and teaching from NSF, scientific
associations, and other organizations. They also have extensive experience in STI evaluation and
bibliometric methods. The following individuals are serving as members of the TWG:
Chemistry Experts
1. Prof. Victor Batista, Professor of Chemistry, Yale University
2. Prof. Emilio Bunel, Director of Division of Chemical Sciences and Engineering, Argonne National
Laboratory
3. Dr. Cynthia Friend, Professor of Chemistry, Department of Chemistry and Chemical Biology;
Professor of Materials Science, School of Engineering and Applied Sciences, Harvard University
4. Prof. Melissa Hines, Professor of Chemistry, Cornell Center for Materials Research
5. Prof. Tara Meyer, Professor of Chemistry, Assistant Dean of Graduate Studies, University of
Pittsburgh
Experts in Evaluation, Bibliometrics, and Science, Technology, and Innovation Policy
1. Dr. Kevin Boyack, President, SciTech Strategies (bibliometrics)
2. Dr. Daryl Chubin, Independent consultant, co-chair of Understanding Interventions, and author of an
evaluation of the NSF Science and Technology Centers, including multiple surveys from which the
CCI evaluation draws 3
3. Dr. Gretchen Jordan, Independent consultant (federal science innovation and program evaluation)
Two TWG meetings have been planned. The first took place on April 6, 2018, to review the evaluation
design and instruments, and the second will take place toward the end of the project to discuss
preliminary findings. For the convenience of the TWG members, both meetings will use a videoconference system. Two weeks prior to the April 6, 2018 meeting, Abt emailed the TWG members the
evaluation plan and a set of questions to provide feedback. An evaluation framework and literature review
were also included as background material. The feedback provided to Abt and other aspects of the study

3

Chubin, D. E., Derrick, E., Feller, I., & Phartiyal, P. (2010). AAAS review of the NSF science and technology
centers integrative partnerships (STC) program, 2000-2009. Washington, DC: American Association for the
Advancement of Science.
13

were discussed during the meeting. Abt will also consult with the TWG members as necessary during the
course of the study.
A.9. Explain any decision to provide any payment or gift to respondents, other than remuneration
of contractors or grantees

No payments to respondents are proposed for this information collection.
A.10. Describe any assurance of confidentiality provided to respondents and the basis for the
assurance in statute, regulation, or agency policy

Abt will take several steps to safeguard respondent information:
1. All staff on the project were instructed in the privacy requirements of the study and signed
statements affirming their obligation to maintain privacy.
2. Access to any data with identifying information will be limited to contractor staff directly
working on the survey and will require individual usernames and passwords.
3. Respondents will be fully informed about the purpose of the study, given assurances of
confidentiality, and told that the data collection is voluntary.
4. Names and other identifying information for survey respondents will be replaced with numerical
identifiers after the data are collected and prior to analysis. A key linking the names to the
identifiers will be kept in a separate location with access for Abt staff on a need-only basis.
5. All data will be reported in aggregated form only.
6. When evaluation data is transferred to NSF, all survey and interview data will be de-identified.
7. Any quotations from responses used in public reporting will be edited to ensure that the identity
of the respondent cannot be ascertained.
Similar procedures will be used for interview data. All data collection instruments and procedures have
been reviewed by Abt’s Institutional Review Board.
Data collected as a result of this study will be used in accordance with criteria established by NSF for
monitoring research and education grants, and in response to Public Law 99-383 and 42 USC 1885c.
A.11. Provide additional justification for any questions of a sensitive nature, such as sexual
behavior and attitudes, religious beliefs, and other matters that are commonly considered
private

The survey of graduate students and postdoctoral researchers collects gender and race/ethnicity
information on a voluntary basis (an option “choose not to provide information” is included). This
information is being collected to determine how effective the CCI Program has been in broadening
participation for individuals underrepresented in science. These data are not available from other sources.
A.12. Provide estimates of the hour burden of the collection of information

Exhibit 2 presents the estimated response burden to participants. Survey estimates were informed by pilot
testing performed under Fast Track Clearance 3145-0215. Abt will administer 211 PI/Co-Investigator and
859 graduate student/postdoc surveys. The PIs and Co-Investigator survey is estimated to take
approximately 20 minutes and the student survey approximately 15 minutes to complete. The resulting
total time burden for all surveys is expected to be 285 hours. Abt will conduct approximately 42
interviews. The estimates for PIs/Co-Investigator interviews is 60 minutes each and for industry partners
14

20 minutes each. The total time burden for all interviews is approximately 32 hours. The total burden for
new data collection for this ICR should not exceed 317 hours.
Exhibit 2: Estimated Response Burden in Hours
Respondent Type
Number of
Respondents
SURVEYS (Total)
PIs/ Co-Investigators
Graduate Students/ Postdocs
INTERVIEWS (Total)
Phase II CCI PIs/ Co-Investigators
Industry Partners
SURVEYS AND INTERVIEWS (Total)

Time Per
Response
(Hours)

211
859

0.33
0.25

27
15

1.00
0.33

Number of
Responses

Total Time Burden
(Hours)

1,070
211
859
42
27
15
1,112

285
70
215
32
27
5
317

Exhibit 3 presents the estimated total cost burden to respondents participating in interviews and surveys,
which totals $9,517. The estimated total cost burden for 1,070 PIs/Co-Investigators and graduate
students/postdocs to complete the surveys is $7,882. The estimated total cost burden of participating in 42
interviews is $1,635.
Exhibit 3: Estimated Cost to Respondents
Respondent Type
Annual
Average
Salary
Hourly
Estimate
Wage
($)
($)
SURVEYS (Total)
PIs/ Co-Investigators*
104,000
50
Graduate Students/ Postdocs**
36,632
18
INTERVIEWS (Total)
PIs/ Co-Investigators*
104,000
50
Industry Partners***
120,000
58
SURVEYS AND INTERVIEWS (Total)

Time Per
Response
(Hours)

Cost per
Response
($)

0.33
0.25

17
5

1.00
0.33

50
19

Number of
Respondents
in Category
1,070
211
859
42
27
15
1,112

Total Cost
for
Responses
($)
7,882
3,587
4,295
1,635
1,350
285
9,517

*Salary estimates for PIs/Co-Investigators are based on the average median salary for all full-time employed chemists (excluding
biochemists) in 2015 according to the National Science Foundation. 4
**Postdoctoral and graduate student salary estimates were averaged from two different sources. NSF reported that the median
salary for postdocs in the physical sciences was $48,000 in 2015 (the most recent data point available through NSF). 5
According to Glassdoor, the national average salary for a chemistry graduate student is $25,264 per year in 2018 in the U.S.
Graduate student salary estimates through Glassdoor were based on 14,894 salaries submitted anonymously by chemistry
Graduate student employees. 6
***The salary estimate for industry partners is based on private, for-profit median salary estimates for U.S. residents employed as
chemists (excluding biochemists) in 2015. 7

4

5

6

7

National Science Foundation. Table 69. Median annual salaries of U.S. residing full-time employed doctoral
scientists and engineers, by occupation and sector of employment: 2015. Retrieved from
https://ncsesdata.nsf.gov/doctoratework/2015/html/SDR2015_DST_69.html
National Science Foundation. Median salaries for recent U.S. Science, Engineering, and Health (SEH) doctorate
recipients in postdoc and non-postdoc positions up to 5 years after receiving degree: 2015. Retrieved from
https://www.nsf.gov/statistics/2018/nsb20181/assets/901/tables/tt03-18.pdf
Glassdoor. Updated May 3, 2018. Chemistry Graduate Student Salaries. Retrieved from
https://www.glassdoor.com/Salaries/chemistry-graduate-student-salary-SRCH_KO0,26.htm
National Center for Education Statistics. Table 69. Median annual salaries of U.S. residing full-time employed
doctoral scientists and engineers, by occupation and sector of employment: 2015. Retrieved from
https://ncsesdata.nsf.gov/doctoratework/2015/html/SDR2015_DST_69.html
15

A.13. Provide an estimate for the total annual cost burden to respondents or record-keepers
resulting from the collection of information

There are no total capital or start-up costs to respondents or record-keepers resulting from the collection
of information. There are also no total operation, maintenance, or purchase of services costs to
respondents or record-keepers resulting from the collection of information other than the time spent
responding to the survey and interviews attached as appendices to this request.
A.14. Provide estimates of annualized costs to the Federal Government

The estimated cost to the Federal Government for the entire CCI Program evaluation study is $927,548.
Exhibit 4 provides the hour and cost breakdown by Phase of the evaluation and by activity.
The estimated cost of only the survey and interview data collection activities included in this request is
$343,888, which includes instrument development and pretesting, drafting the data collection plan and
information collection request, survey data collection and analysis, and interview data collection and
analysis. Indirect as well as direct costs are included in the estimate. There are no significant costs beyond
the normal labor costs for staff.
The total hour and cost breakdown associated with other Phase 1 and Phase 2 activities that do not
involve new data collection (e.g., involving use administrative and public data) are reflected in the table
without additional detail.
Exhibit 4: Overall Cost to the Federal Government
Category
Phase 0 Project Management and Communications Total
Phase 1 Evaluation Design and Planning Total
Instrument Development and Pretesting
Preparation of Data Collection Plan and Information Collection Request
Other Evaluation Design and Planning Activities
Phase 2 Data Collection and Analysis Total
Survey Data Collection and Analysis
Interview Data Collection and Analysis
Other Data Collection and Analysis Activities
Phase 3 Evaluation Reporting Total
PHASE 0-3 Total

Hours
549
1322
510
444
368
2929
856
497
1,576
681
5481

Costs
$110,559
$245,071
$82,229
$82,757
$80,085
$437,541
$113,221
$65,681
$267, 639
$134,377
$927,548

A.15. Changes in Burden

This is a new data collection.
A.16. For collections of information whose results will be published, outline plans for tabulation
and publication. Address any complex analytical techniques that will be used. Provide the
time schedule for the entire project, including beginning and ending dates of the collection
of information, completion of report, publication dates, and other actions

Abt will conduct bibliometric analyses and abstract and code data from grantee-produced and NSFproduced administrative data during the period of May 2018-December 2018. Surveys will be
administered in January 2019, conditional on OMB approval. Interviews will be conducted in February
2019, after we review survey data, conditional on OMB approval.
Analysis of Survey Data. For all survey items, descriptive statistics will be used to summarize responses.
For measures using continuous scales, means and standard deviations will be calculated to describe
16

central tendency and variation. Frequency distributions and percentages will be used to summarize
measures on categorical scales. In addition, cross-tabulations will be used to illustrate differences in
measures between groups or the distribution of measures across subgroups of interest. Tests for
significant differences (e.g., t-test, chi-square, and ANOVA) may be used to compare responses between
subgroups of respondents. Where appropriate, factor analysis will be used to explore and assess
unobserved latent variables based on the correlation among observed variables.
Analysis of Interview Data. Trained researchers will code major themes that emerge from interview data. An
iterative coding process will be used so that codes are regularly refined and new codes are generated in
response to emergent themes within the responses. The data that emerge will be summarized as a
narrative. Within each respondent group, Abt will quantify how many individuals expressed a given view,
to give a sense of the level of agreement. These data will also be used to provide examples and quotes to
supplement the survey data. The qualitative data will provide richer and more detailed information on
more specific topics than could be gleaned from close-ended questions.
Exhibit 5 shows the schedule of activities associated with the entire evaluation study, including the
surveys and interviews included in this request under Phase 2 Data Collection and Analysis Activities.
Exhibit 5: Deliverables and Due Dates
Deliverables
Phase 1 Evaluation Design and Planning
Phase 2 Data Collection and Analysis
Administrative and public (bibliometric) data
collection and analysis
Survey collection start date
Survey collection end date
Survey analysis period
Interview collection start date
Interview collection end date
Interview Analysis period
Phase 3 Evaluation Reporting
Draft Evaluation Report
Final Evaluation Report and Database

Timeline
October 2017 to May 2018
May 2018 to April 2019
May 2018 to December 2018
January 2019
February 2019
February 2019
February 2019
March 2019
March 2019 to April 2019
April 2019 to July 2019
Due 18 months after contract award: April 2019
Due 21 months after contract award: July 2019

NSF provided guidance that the contractor may not publish or in any other way reveal data, information
or the results of the study, or other analyses or NSF internal discussions of data collected or accessed for
the study, without prior written approval from NSF. The contractor shall transfer to NSF all work
products, including web surveys, materials necessary to perform data collection, bounce back messages
from survey email addresses, and any other items associated with and necessary to administer, analyze,
and finalize the surveys, in conformance to OMB requirements associated with confidentiality. All data
collected are the sole property of NSF and will not be disclosed or distributed to any non-NSF sources
without prior written approval from NSF. All reports will be submitted as MS Office compatible
hardcopies and e-files. An electronic copy of the final deliverable(s) will be delivered to the NSF COR.
After the products are delivered, NSF determines whether the quality of the products deserves publication
verbatim by NSF; i.e., NSF typically is the exclusive publisher of the information collected by the
collections. Often it is only after seeing the quality of the information the collection delivers that NSF
decides the format and manner (in its Online Document System or simply a page on the NSF website) in
which to publish.

17

NSF intends to make information about evaluations and findings from evaluations broadly available and
accessible, typically on the internet. NSF intends to release results of evaluations that are not specifically
focused on internal management, legal, or enforcement procedures or that are not otherwise prohibited
from disclosure. Evaluation reports present all results, including favorable, unfavorable, and null findings.
NSF intends to release evaluation results in a timely manner—usually within six months of a report's
completion—and will archive evaluation data for secondary use by interested researchers (e.g., public use
files with appropriate data security protections). After internal technical peer review, the Section Head of
NSF’s Evaluation & Assessment Capability has authority to seek NSF clearance to approve, release, and
disseminate evaluation reports.
A.17. If you are seeking approval to not display the expiration date for OMB approval of the
information collection, explain the reasons that display would be inappropriate

We will display OMB approval information on the data collection, including expiration date, at the
beginning of all data collection instruments. The following statement will be attached to the data
collection instrument:
“The OMB control number for this project is 3145-NEW. Public reporting burden for this collection of
information is estimated to average [PI/Co-Investigator survey: 20; PI/Co-Investigator interview: 60;
Graduate student/postdoc survey: 15; Industry Partner interview: 20] minutes per respondent, including
the time for reviewing instructions, searching existing data sources, gathering and maintaining the data
needed, and completing and reviewing the collection of information. Send comments regarding this
burden estimate or any other aspects of this collection of information, including suggestions for reducing
this burden, to Suzanne H. Plimpton, Reports Clearance Officer, National Science Foundation, 2415
Eisenhower Ave, Alexandria, VA 22314 or send e-mail to [email protected].”
A.18. Explain each exception to the topics of the certification statement identified in Certification
for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

18