3145-CCI Supporting Statement Part B

Section B: Data Collection Procedures and Statistical Methods
Part B of the Supporting Statement for the Evaluation of the CCI study covers Data Collection Procedures
and Statistical Methods. The data collection instruments described in this submission include the Survey
and Interview protocols (See Appendices 1 and 2).
B.1. Describe (including a numerical estimate) the potential respondent universe and any
sampling or other respondent selection methods to be used. Data on the number of
entities (e.g., establishments, State and local government units, households, or persons)
in the universe covered by the collection and in the corresponding sample are to be
provided in tabular form for the universe as a whole and for each of the strata in the
proposed sample. Indicate expected response rates for the collection as a whole. If the
collection had been conducted previously, include the actual response rate achieved
during the last collection.

Since this is a new data collection and we do not have an actual response rate from previous collection
efforts, the estimates for response rates are based on estimates for similar projects that Abt has conducted.
Surveys. Two online surveys are proposed for the study: CCI Principal Investigator (PI) and CoInvestigator (Co-I) Survey and CCI Graduate Student and Postdoctoral Researcher Survey. Both surveys
will be administered to current and recent former U.S.-based participants of all Phase I and Phase II
grants awarded between 2010 and 2016. To minimize possible recall bias and to ensure that respondents
have sufficient experience with CCI, the samples will be limited to PIs and Co-Investigators who were
included in annual reports between the 2012-13 and 2016-17 reporting years and graduate students and
postdoctoral researchers who were included in annual reports between 2014-15 and 2016-17. Applying
these criteria results in samples of 211 PIs/Co-Investigators affiliated with 14 CCIs (5 Phase I-only and 9
Phase I/II) and 859 graduate students/postdocs affiliated with 9 Phase II CCIs. The surveys will be
administered to all individuals in this sample (a census). Since this is not a mandatory collection, we
expect to have some level of nonresponse, approximately 20 percent, resulting in an 80 percent response
rate. Non-respondents will include people who choose not to complete the survey and people for whom it
was not possible to update contact information and thus who could not be reached.
Interviews. The evaluators propose interviews with three types of respondents: PIs on CCI grants, CoInvestigators on CCI grants, and industry partners. Abt will use a convenience sample for interviews.
•

•

For the CCI PI and Co-Investigator interviews, all 9 primary PIs on Phase II grants and a sample of
up to 18 Co-Investigators on Phase II grants will be included. Abt will select Co-Investigators based
on recommendations from the PI, survey data, and administrative data. In particular, the CoInvestigator interview sample will include individuals with a range of project activities and duration
of Center participation. Abt expects a 100 percent response rate.
For the industry partner interviews, Abt will draw a sample based on administrative data and
recommendations provided by each CCI PI for several industry partners that were involved with the
center to such a degree that they would be able to provide useful feedback. The industry partner
sample will include a range of industries, if possible. Approximately 10-15 individuals will be
interviewed. Abt expects a 100 percent response rate.

19

Locating respondents. Respondent contact information will be extracted from the most recent annual
report in which it is available. For any emails that bounce back when Abt sends out the pre-survey
notification (see B.3), an attempt will be made to identify an alternative address through internet searches.
B.2. Procedures for the Collection of Information

B.2.1. Statistical methodology for stratification and sample selection.
The surveys of CCI PIs, Co-Investigators, graduate students, and postdoctoral researchers will include the
census of 211 PIs and Co-Investigators affiliated with 14 CCIs (5 Phase I-only and 9 Phase I/II) listed in
annual grant reports between the years 2012-13 and 2016-17, and 859 graduate students and postdocs
affiliated with 9 Phase II CCIs listed in annual grant reports between the years 2014-15 and 2016-17.
Survey protocols are included in Appendix A and C.
The interviews are not intended to yield representative population estimates and thus respondents will be
selected based on a recommendation of CCI PIs, program data, and survey data. Interview protocols are
included in Appendix B and D.
B.2.2. Estimation procedure.
Not applicable.
B.2.3. Degree of accuracy needed for the purpose described in the justification.
The expected 80 percent survey response rate will lead to estimates with a high degree of accuracy even if
non-response bias is present. Survey responses will be triangulated with administrative data as a further
check on accuracy. Nevertheless, a high degree of accuracy is not required for the purposes described in
this justification. The purpose of the evaluation is to explore the outputs and outcomes of the CCI
program, not assess impact or causation or generalize survey responses to all center-based programs.
B.2.4. Unusual problems requiring specialized sampling procedures.
Not applicable.
B.2.5. Any use of periodic (less frequent than annual) data collection cycles to reduce burden.
Not applicable.
B.3. Methods to Maximize Response Rates and Deal with Nonresponse

Since this is the first evaluation of NSF CCI and there are no prior CCI studies to draw from, the study
team used the knowledge and lessons learned from prior NSF evaluations to design instruments that will
be clear and relevant to CCI participants. The study team completed two previous rounds of pretesting of
the online survey to ensure that participants can complete the survey efficiently without any glitches or
technical issues.
One challenge of this study will be motivating participation and following up with initial non-respondents
to achieve our target response rate of at least 80 percent. The web-based approach will allow us to easily
identify non-respondents for follow-up contact to encourage participation and maximize response rates.
To increase the overall survey response rate, follow-up with respondents will be multi-modal (email and
telephone).

20

Specifically, we will use the following methods to encourage participation (survey and interview requests
and reminders are included with instruments in Appendices A-D):
•

Pre-study notification from NSF to PIs. NSF will send a letter to PIs introducing the team at
Abt Associates and informing them about the goals of the CCI evaluation. This email will add
credibility to the email solicitation that will come from Abt a day later to encourage participation
and increase study response rates.

•

Pre-study notification from Abt to PIs. Abt will email PIs with a list of potential study
respondents to notify and request their participation. We will personalize the email invitations by
addressing each PI by name by using MS Word’s mail merge tool. In addition, we will also
include the name of the NSF contact and a survey administrator contact so that participants may
ask questions about the study.

•

Pre-study notification from Abt to study participants. Abt will email potential study
respondents to inform them about the goals of the CCI evaluation and request their future
participation. Abt will look for updated email addresses for any participants whose emails bounce
back.

•

Survey request email from Abt to study participants. At the designated opening date, the
study team will send an email message to respondents with a unique survey link, detailed
instructions, the closing date, and project staff contact information. Detailed on-screen
instructions will be included. Throughout the data collection cycle, the study will use an email
address to ensure that potential respondents can quickly and easily obtain answers to questions or
concerns.

•

Personalized reminder emails. A personalized reminder email will be sent to study participants
approximately two weeks after the initial invitation to encourage anyone who has not yet
participated and to thank those who completed the survey. Additional reminder emails will be
sent to non-respondents weekly.

•

Final reminder email. A final reminder email will be sent to participants 48 hours before the
close of the survey to again remind those who have not yet participated and to thank those who
completed the survey.

Abt expects a survey response rate of at least 80 percent and will implement follow-up procedures with
non-respondents to increase participation. If survey response is less than 80 percent, two types of analyses
will be performed to assess the implications of non-response. First, the available characteristics of
individuals who completed the surveys (Center affiliation, project role, number of years of Center
participation) will be compared to the characteristics of those who did not. Second, a statistical test using
these baseline characteristics will be performed to predict the probability that a project participant was
located and responded to the survey request. If these analyses point to the possibility of non-response
bias, sampling weights will be created based on the observable baseline characteristics and used in
frequency calculations.

21

B.4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an
effective means of refining collections of information to minimize burden and improve
utility. Tests must be approved if they call for answers to identical questions from 10 or
more respondents. A proposed test or set of test may be submitted for approval
separately or in combination with the main collection of information

The survey instruments and interview protocols developed for this data collection were pilot-tested under
Fast Track Clearance 3145-0215.
The survey instruments were programmed and subjected to two rounds of pretesting. First, Abt
researchers followed a pre-determined script to ensure that all skip patterns and open-ended and multiplechoice questions worked as intended. Any glitches identified through this process were fixed before
piloting the survey on a small number of subjects chosen to represent the study populations. For the pilot
test, 36 graduate students and postdocs and 15 PIs and Co-Investigators were selected using purposive
sampling to participate. These individuals were asked to provide feedback on the clarity and content of
the questions and the time it took them to complete the survey.
Abt received 15 responses from graduate students and postdoctoral researchers and 9 responses from
PIs/Co-Investigators. The median response time (inclusive of pilot feedback questions) was 15 minutes
for both surveys. These burden estimates include the time to answer each question and provide feedback,
and therefore likely exceed the actual time it will take to complete the surveys. Thus, the estimates of
burden provided in section A.12 are conservative. Since we expect some PIs and Co-Investigators,
particularly those who were involved in the program for numerous years, may spend more time
responding to the survey, we estimate their burden at 20 minutes.
All of the CCI pilot testers reported that the survey instructions were easy to follow, and the questions
were asked in a logical order, were clearly worded, and were easy to answer. The total number of
questions was considered appropriate.
Pilot testers identified a handful of survey questions that were not applicable to their experience, or that
could use additional or slightly different options; the surveys were adjusted based on this feedback.
Exhibits 6 and 7 show the comments from the testers and the corresponding revisions to the PI/CoInvestigator and the graduate student/postdoctoral researcher surveys, respectively. Note that identifying
information (center names and primary PIs) in the feedback has been redacted to protect respondents’
privacy.

22

Exhibit 6. PI/Co-Investigator Pilot Feedback and Revisions
Original Question
2. All of the questions in the survey
refer to your participation in the
following NSF Center for Chemical
Innovation (CCI): [CCI name].
Please note that your center may have
been funded under a predecessor to
the NSF CCI program such as
Chemical Bonding Centers (CBC).

□
□
□

Feedback
“I know what you mean here, so I
checked the first one, but I have been a
member of two CCIs: [CENTER A] (as
a PI) and [CENTER B] (as a postdoc).
So, I could click both of the first two
choices.”

Revision
Revised question text and response
options
2. All of the questions in the survey
refer to your participation in the
following NSF Center for Chemical
Innovation (CCI): [CCI name].
Please note that when first established,
the program was called the Chemical
Bonding Centers.

I have participated in this CCI
I have participated in a different
CCI  A

□
□

I have not participated or I am not
sure whether I participated in the
CCI program

□

A. Please select from the list of options
If you participated in more than one
CCI, please select the CCI with which
you have been most extensively
involved.
[pull-down menu of CCIs]

I have participated in this CCI
I have not participated in this CCI,
but I have participated in a
different CCI  A
I have not participated or I am not
sure whether I participated in the
CCI program

A. Please select from the list of options
If you participated in more than one
CCI, please select the CCI with which
you have been most extensively
involved and limit your answers to this
CCI only.
[pull-down menu of CCIs]

3. What is your role in this CCI? Select
one.

“OK, now I'm confused. Internally, in all
communication we have by e-mail, any
member of the [CENTER] than runs a
research group is called a PI. But, I
assume that the NSF might view
[PRIMARY PI] (the director of the
[CENTER]) as the PI and the rest of us
as co-investigators. Otherwise, I don't
know what a co-investigator is. So, I am
clicking "co-investigator" here, but I
definitely think this question needs
clarification. Maybe "directing/head PI
of this CCI" or "co-investigator/co-PI"?”

Deleted question
Project role recorded in RPPRs

5. On a scale of 1-5, where 1 indicates
“not at all” and 5 indicates “very,” to
what extent does the research
conducted by your CCI have the
following characteristics?

“I do not think many would admit
anything below a five or four to those
questions. Who would say their work
was not important and needed more
money? Don't think those are useful
questions”

No change made
Some pilot respondents gave
themselves ratings below four

10. Please indicate whether
participation in the CCI has benefited
your research program.

“I work at a government lab that has
minimal access to students and
analytical equipment used in my work.”

No change made

23

Original Question

Feedback

Revision

11. Have any of the following changes
occurred in your publication patterns,
research interests, and/or professional
visibility since you began participating
in CCI?

“Maybe there should also be a "not
applicable" choice here. We are only in
our second year, and we are just
starting to submit our first [CENTER]
papers. Thus, it is hard to judge the
impact on the quality of the journals we
publish in.”

Revised response options (changed
“Has not changed” to “Has not
changed/Too early to tell”)

12. Which of the following
resources/infrastructure created or
improved by CCI, if any, are being used
by researchers not affiliated with the
center?

“What is the difference between
instruments and equipment? Is an
instrument not equipment? Also, maybe
add information infrastructure. My
group is adopting the electronic data
management system that the
[CENTER] developed. I think it is
great!”

Revised response options

□
□
□
□
□
□
□
□
□

Methods or instruments, research
practice
Reagents
Data
Communication infrastructure
Educational or outreach materials
Facilities or equipment
Lessons learned for how to run a
large center
New partnerships or alliances
Other resources. Please specify:
__

“My response to the above question
only applies to my lab.”

10. Which of the following
resources/infrastructure created or
improved by CCI, if any, are being used
by researchers not affiliated with the
center?

□
□
□
□
□
□
□
□
□
□
□
□

Methods
Reagents
Data
Communication infrastructure
Data management system
Educational or outreach materials
Facilities
Equipment
Lessons learned for how to run a
large center
New partnerships
Other resources. Please specify:
__
I am not aware of any resources
created or improved by CCI that
are being used by researchers not
affiliated with the center

24

Original Question

Feedback

Revision

13. Please indicate whether your CCI
developed or improved the following
infrastructure to develop workforce,
broaden participation of
underrepresented groups in chemistry,
and improve public outreach, and if you
expect to be able to sustain it after the
grant ends.

“This prompt has two clauses/questions
(1. was it developed and 2. will you be
able to sustain it), but there is only one
check box per item. Maybe better to
ask "If this was developed, will you be
able to maintain it when the grant
ends?" That's how I'm answering the
question”

Split question into two
11. Please indicate whether your CCI
developed or improved the following
educational or outreach opportunities.

□

I do not know whether my CCI
developed or improved these types
of opportunities  skip 13b

12. Current grants: Please indicate
whether you expect to be able to
sustain the following programs and
activities after the grant ends.
Completed grants: Please indicate
whether the following programs and
activities are still in place.
14. Please indicate whether the
following improvements have occurred
as a result of CCI funding.

“I am a contract scientist in a
government lab where I have very little
influence on the institutional culture that
is controlled by civil servants and their
management system.”

Added “N/A” response option

15. Please indicate whether the CCI
delivered any of the following benefits
to industry.

“Is this my opinion of whether the
[CENTER] was able to accomplish
each benefit or whether I personally
participated/achieved each benefit? If
the former, there should also be a
column for "not sure". For a lot of these,
I have no idea but that is not an option.”

Added “Uncertain” response option
Added checkbox to skip question
14. Please indicate whether the CCI
delivered any of the following benefits
to industry.

□

“Don't know about increase in sales or
cost savings”

I am not aware of any
partnerships between my CCI
and industry  skip to next
question

“I'm not sure how relevant these are for
[CENTER]. Possibly on the
experimental side but not yet in my
direct projects.”
19. Phase II only: To what extent has
participation in Phase I contributed to
the success of your Phase II center?

□
□
□

Not at all
To some extent
To a considerable extent

“Consider adding an "n/a" option. I did
not participate in Phase I, so there is no
option for me.”
“I did not participate in Phase 1.”

Added checkbox to skip Phase I
impact questions
20. Phase II only: To what extent has
participation in Phase I contributed to
the success of your Phase II center?

□
□
□
□

I did not participate in Phase I 
skip Phase I impact questions
Not at all
To some extent
To a considerable extent

25

Original Question

Feedback

Revision

20. Have any of the following occurred
as a result of your participation in
Phase I? Would these outcomes be
more challenging to achieve under a 1phase center model?

“Consider adding "I did not participate
in Phase I" instead of just "no benefits"”

Addressed above

Overall Feedback

“Add an NA to all questions”

Added N/A
An N/A option encourages low-effort
responses, so we added it only where
we believed a particular question may
not be applicable to certain
respondents.

Overall Feedback

“I think some of the questions need
clarification because different co-PIs
have different roles and involvement,
but overall, I think the survey is good”

No change made
However, we did add “N/A” and
“Uncertain” response options where
warranted.

“I did not participate in Phase 1.”

26

Exhibit 7. Graduate Student/Postdoctoral Researcher Pilot Feedback and Revisions
Original Question
2. All of the questions in the survey
refer to your participation in the
following NSF Center for Chemical
Innovation (CCI): [CCI name].
Please note that your center may have
been funded under a predecessor to
the NSF CCI program such as
Chemical Bonding Centers (CBC).

□
□
□

I have participated/still
participating in this CCI
I have participated/still
participating in a different CCI 
A
I have not participated in the CCI
program

A. Please select from the list of options
If you participated in more than one
CCI, please select the CCI with which
you have been most extensively
involved.
[pull-down menu of CCIs]

Feedback
“I didn't know there were a Chemical
Bonding Centers. Otherwise it is clear
about my particular involvement with
the [CENTER]”

Revision
Revised question text and response
options
2. All of the questions in the survey
refer to your participation in the
following NSF Center for Chemical
Innovation (CCI): [CCI name].
Please note that when first established,
the program was called the Chemical
Bonding Centers.

□
□
□

I have participated/am still
participating in this CCI
I have not participated/am not
participating in this CCI, but I have
participated/am still participating in
a different CCI  A
I have not participated in the CCI
program

A. Please select from the list of options
If you participated in more than one
CCI, please select the CCI with which
you have been most extensively
involved and limit your answers to this
CCI only.
[pull-down menu of CCIs]

27

Original Question

Feedback

11. Which of the following positions are
you most interested in pursuing after
you complete your degree and/or
postdoctoral training? Have your career
goals changed since you began
participating in the CCI?

“This page doesn't reflect nontraditional career paths such as science
writing or consultation.”

□
□
□
□
□
□
□
□

Faculty member in a research
college or university
Faculty member in a 2-year or 4year teaching college
Program officer/academic
administrator
Non tenure-track researcher in a
university or a research institute
Researcher in a government
laboratory
Research and Development
position in industry
Business position in industry or an
entrepreneur
Other, please specify: _____

Revision
Added response option
12. Which of the following positions are
you most interested in pursuing after
you complete your degree and/or
postdoctoral training? Have your career
goals changed since you began
participating in the CCI?

□
□
□
□
□
□
□
□
□

Faculty member in a research
college or university
Faculty member in a 2-year or 4year teaching college
Program officer/academic
administrator
Non tenure-track researcher in a
university or a research institute
Researcher in a government
laboratory
Research and development
position in industry
Business position in industry or an
entrepreneur
Science policy, law, consulting, or
science writing
Other, please specify: ______

28

Original Question
12. Have you spent time working in a
laboratory/research group of a CCI
partner organization (as an intern,
graduate student, visiting scholar, or
similar)?
12A. Which of the following best
describes the organization(s) in which
you worked? Select all that apply if you
worked with more than one type of
organization.
12B. How much time did you work at
CCI partner organization(s) in total?

Feedback
“Not really sure what you mean by
partner organization. Is this the
universities which are part of the CCI,
which are not my own? or something
else? I am going to assume so, and
answer yes. Perhaps explain in more
detail or give examples?”
“It would be easier to answer this page
with options that indicate if you are still
involved with the CCI.”
“Adding an "I'm not sure" option would
be a more accurate answer.”

Not at all valuable

13A. Which of the following best
describes the organization(s) in which
you worked and/or currently work?
Select all that apply if you worked with
more than one type of organization.

13C. How valuable was this experience
to your career development?

Somewhat valuable

□
□
□
□

Very valuable

16. How well do you think participation
in the CCI has prepared/is preparing
you for the following activities?

Revised question text and response
options
13. Have you spent time working in a
laboratory/research group of the CCI
partner organization (e.g., another
university or company involved with
your center) as an intern, graduate
student, visiting scholar, or similar role?

13B. How much time did you work at
CCI partner organization(s) in total?

12C. How valuable was this experience
to your career development?

□
□
□

Revision

“This form is answered to reflect what I
know my specific involvement with the
center has prepared me for. One-onone interactions with research advisors
would reflect different answers to these
same questions”

Not at all valuable
Somewhat valuable
Very valuable
Too early to tell/uncertain

Changed “N/A” response option to
“N/A/too early to tell”

The interview protocols were subjected to a round of pilot testing. Four Phase II Co-Investigators and two
Industry Partners participated in the pilot. The median response time was 60 minutes for Co-Investigator
interviews and 30 minutes for Industry Partners interviews. The questions in the PI/Co-Investigator
interview protocol were clear and relevant to all pilot Co-Investigators. Pilot industry partners had
difficulty providing detailed responses to three of the questions in the Industry Partners interview protocol
given the nature of their involvement with the CCIs. The three questions are:

29

1. What do you see as the most important scientific and engineering accomplishments of the Center? Do
you think these accomplishments would have been possible without the Center structure? If not, how
were these accomplishments enabled by the Center?
2. What do you see as the most important non-scientific accomplishments of the Center (e.g. in
workforce development, knowledge transfer, educating the general public)? Would they have been
possible without the Center? If not, how were these accomplishments enabled by the Center?
3. Do you have a view about the Center organization and processes? Do you think these are effective?
Are there components you would change?
We removed these three questions from the Industry Partners interview protocol. Given that these three
questions represent one-third of the questions in the piloted protocol, we estimate the burden of the final
protocol at 20 minutes (two-thirds of the median pilot response time).
B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The contractor for collection and analysis of data in this study is Abt Associates. Staff from Abt
Associates, the contractor performing the study, have all necessary experience to complete this data
collection. Allan Porowski, the Project Quality Advisor, is an evaluation methodologist who has
substantial experience with survey research. Ellen Bobronnikov and Luba Katz, the Management and
Technical Leads have each designed and administered numerous surveys for NSF as well as other federal
agencies, and Brian Freeman, the Director of Analysis, has a background in statistics and has designed
numerous federally funded surveys and analyzed their data. In addition, the data collection instruments
and procedures have been reviewed by NSF staff familiar with the program and by the external Technical
Working Group described in section A.8.
Abt
Ellen Bobronnikov
Luba Katz
Allan Porowski
Brian Freeman
Jessie Bristol
Alex Silverman

Project Role
Management Lead
Technical Lead
Project Quality Advisor
Director of Analysis
Sr. Analyst
Sr. Programmer Analyst

Email
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]

Phone
617-349-2718
617-349-2313
301-634-1765
617-520-2356
617-520-3085
617-520-3540

NSF
Rebecca Kruse
Lin He

Evaluator, COR
CCI Program Officer

[email protected]
[email protected]

703-292-4211
703-292-4956

Physics Frontiers Centers
Program Officer

[email protected]

703-292-8783

Jean Cottam Allen

30