Download:
pdf |
pdfHigher Education Research and Development (HERD)
Proposed COVID-19 Impact Questions Testing
Report and Recommendations
October 27, 2020
prepared for
National Science Foundation
National Center for Science and Engineering Statistics
by
ICF • 530 GAITHER ROAD • ROCKVILLE, MD 20850
Table of Contents
Introduction ............................................................................................................................................ 3
Methodology........................................................................................................................................... 3
Question Development........................................................................................................................ 3
Testing Timeline .................................................................................................................................. 3
Sample ................................................................................................................................................ 4
Recruitment ........................................................................................................................................ 4
Participants ..................................................................................................................................... 4
Testing Format .................................................................................................................................... 5
Detailed Findings ..................................................................................................................................... 5
Question A: Degree of Disruption to R&D Activities ............................................................................. 5
Question B: Diverted R&D Funds ......................................................................................................... 6
Diverted Funds (Question B1) .......................................................................................................... 6
Percentage of Diverted Funds (Question B2) .................................................................................... 7
Changes to Question B Between Rounds.......................................................................................... 7
Question C: New R&D Funds Received ................................................................................................. 7
Receiving New Funds (Question C1) ................................................................................................. 7
External Sources of New Funds (Question C2) .................................................................................. 8
Changes to Question C Between Rounds.......................................................................................... 8
Results from Webinar Polls...................................................................................................................... 9
Appendices
A. Cognitive Interview Communications
B. Round 1 Cognitive Interview Test Questions
C. Round 2 Cognitive Interview Test Questions
D. Round 1 Cognitive Interview Protocol
E. Round 2 Cognitive Interview Protocol
F. Final COVID-19 Impact Questions
G. FY 2020 HERD Survey Update Webinar Poll Results
ICF
2
Introduction
The National Center for Science and Engineering Statistics (NCSES), within the National Science
Foundation (NSF) conducts the Higher Education Research and Development (HERD) Survey annually.
The HERD survey collects information on the financial resources allocated to R&D by universities and
colleges and is one of several surveys at NCSES that collect comparable information on R&D from
different sectors of the economy (e.g., businesses, nonprofits, and government).
The closures and social distancing requirements resulting from the COVID-19 pandemic disrupted
routine R&D at colleges and universities beginning in March 2020. At the same time, additional funding
to address pandemic-related topics became available and some researchers pivoted their work to these
topics. NCSES wanted to include questions on the FY 2020 HERD survey, due to launch in early
November 2020, that would measure the impacts of the COVID-19 pandemic on higher education R&D
activity. However, questions asking for detailed expenditure amounts would likely take too long to
develop and would be too burdensome for universities and colleges to complete. For those reasons,
NCSES proposed adding a few easy to complete, qualitative questions to assess impacts of the
pandemic.
After initial question design, ICF conducted cognitive interviews with 15 higher education institutions to
get feedback on the proposed questions. Additionally, during the annual HERD survey update webinar,
attendees were asked through a poll how easy it would be to answer the questions should they appear
on the FY 2020 HERD survey. This report summarizes the findings and recommendations based on the
testing and webinar polls.
Methodology
Question Development
Three questions were initially designed by NCSES. These questions asked about the degree of disruption
to the institution’s FY 2020 R&D, and whether R&D funds were diverted or added as a direct result of
the COVID-19 pandemic. ICF’s survey methodologist reviewed the questions and provided feedback. The
questions were then revised based on additional NCSES and ICF discussions.
Following the first nine cognitive interviews, ICF and NCSES revised the questions based on feedback
before conducting six additional cognitive interviews. Minor revisions were also made prior to
presenting the final questions at the annual survey update webinar.
Testing Timeline
Table 1 presents the timeline for testing.
Table 1. Timeline for Testing
Task
Round 1 Invitation E-mails Sent
Round 1 Interviews Occur
Round 2 Invitation E-mails Sent
Round 2 Interviews Occur
Polling of Respondents
ICF
Timeframe
August 20, 2020
August 24–28, 2020
September 11, 2020
September 14–16, 2020
October 7, 2020
3
Sample
The sample was limited to HERD standard form institutions from the three highest quartiles by total
R&D expenditures reported for FY 2019. These institutions have higher levels of research activity and
thus have a greater impact on total R&D. They also have greater insight into issues related to the R&D
interruptions. Institutions that were part of other recent methodological work were excluded from the
sample. Eligible institutions were randomly selected from the three quartile groups. ICF selected a
sample of 60 institutions.
Recruitment
Sampled institutions were invited to participate in an interview to discuss the COVID-19 questions.
Appendix A includes all contact protocols. The e-mail was sent to the HERD primary contact. In the
invitation e-mail, institutions were asked to contact ICF if they were interested and available to
complete an interview by August 28 (for round 1) and by September 16 (for round 2). If respondents
said they were interested, we first sent them a copy of the questions and asked them to review. In the
e-mail they were told, “If you are not the best person to answer these questions, we ask that you invite
that person to the interview, or ask them to contact me to schedule an interview.” Once institutions
reviewed the questions and provided names and availability, we scheduled the interviews.
Table 2 includes the number of invited and interviewed institutions in each quartile.
Table 2. Number of Institutions Invited and Interviewed
Quartile
Type of Control
Invited
Interviewed
2nd
Public
12
3
Private
8
2
3rd
Public
12
3
Private
8
3
4th
Public
12
2
Private
8
2
Total
60
15
Participants
Information on the participating institutions are presented in table 3.
Table 3. Information About Institutions That Participated in Interviews
Institution
Type of Control Quartile
University of Pennsylvania
Private
4th
University of Southern California
Private
4th
Colorado State University, Fort Collins Public
4th
University of Missouri, Columbia
Public
4th
Boise State University
Public
3rd
Illinois Institute of Technology
Private
3rd
Northern Illinois University
Public
3rd
Nova Southeastern University
Private
3rd
University of Tulsa
Private
3rd
Tennessee State University
Public
3rd
St. Edward’s University
Private
2nd
Sam Houston State University
Public
2nd
ICF
FY 2019 Total R&D (in thousands)
$1,506,285
$909,683
$398,477
$278,793
$39,824
$34,334
$26,570
$24,567
$18,648
$15,236
$9,650
$9,445
4
Institution
University of New England
Florida Gulf Coast University
University of North Carolina, Asheville
Type of Control
Private
Public
Public
Quartile
2nd
2nd
2nd
FY 2019 Total R&D (in thousands)
$7,346
$5,907
$4,602
Testing Format
As previously mentioned, participants were sent the COVID-19 questions prior to scheduling the
interview (see Appendix B and C). During the interview, participants were asked questions about their
interpretation of phrases and words of the questions and response options. Interview protocols are
included in Appendix D and E. The interviews lasted approximately 30 minutes. 1
We offered a video conference option to participants, and none chose this option.
Detailed Findings
Before asking participants about the COVID-19 impact test questions, the interviewer asked the
respondents to provide an overview of how research has changed at their institution since mid-March.
Most of the participants said that the institution completely shut down or paused on-campus research
activities for at least a few weeks, beginning in mid-March. Participants mentioned that even when the
campus was closed, many researchers were able to pivot their work to things that could be done
remotely, such as analysis and writing. By June, institutions were beginning to “ramp up” and begin
limited on-campus research again. In many cases, faculty and staff could begin working in labs before
students were allowed to access labs or other on-campus research spaces.
Participants were also asked if there were words or phrases that their institution was using to describe
the effects of the pandemic on research. About half of the participants said that there were not, while
the other half said that they were using words such as “shut down,” “paused,” “suspension of activities,”
or “disruption.”
Below, we summarize the feedback provided by respondents for all test questions during both rounds of
cognitive interviews.
Question A: Degree of Disruption to R&D Activities
Once participants were given a few minutes to read the question, they were asked what they thought
Question A was asking in their own words. Most of the participants understood what this question was
asking; however, during the first round of interviews, participants thought about varied types of R&D
and timeframes. During round 1, Question A only referred to FY 2020 R&D activities, with no other
clarifications about reference periods or types of R&D. Most of the participants in round 1 thought
about on-campus or field research only. Two participants said that they thought about on-campus
research first and then began thinking more broadly to “nonphysical” research that could be done
remotely. Further, participants were asked what timeframe they thought of when reading this question,
Round 1 participants were told that the interview would last 15 minutes, and we attempted to keep those
interviews as close to 15 minutes as possible. However, most of these interviews lasted at least 20 minutes, and
some were closer to 30 minutes. For round 2, all participants were told that the interview would last 30 minutes,
and all were within that timeframe.
1
ICF
5
and answers varied from March through June, spring and fall, and the height of the shutdown at their
institution.
During round 1, response options asked respondents to gauge the level of disruption (e.g., partially shut
down). While some participants thought the response options were understandable, four participants
thought that the categories needed clarification. One participant noted a change in response options
from using shutdown to disruption, and three others commented that the use of “fully,” “partially,” and
“minimally” made it harder to determine which response option to select.
Five of the respondents said that it would be easy to answer this question across the entire institution,
but two said that it would be difficult. Both of these participants were the same participants who said
that they thought of research activities that were not just in labs or field work.
Following the first round of interviews, we made the following changes to Question A. No changes were
made after the second round of interviews.
•
•
•
•
Changed focus from FY 2020 to June 2020. In round 1, most of the participants said that there
was an initial shutdown of activities (beyond R&D) in March and April, but by June, the campus
and R&D activities were “ramping up” again. During round 2, while this wording change caused
some participants to ask, “why June,” participants felt that adding this specific timeframe made
it easier to answer the question. One participant said that while it would be easier to answer, it
would not reflect the entire picture of how R&D activities were impacted by the pandemic.
Added instruction to think about on-campus and off-campus activities. Before the instruction
was added, most participants initially thought of on-campus R&D activities only when answering
this question. During the interviews, some participants said that as they thought about the
question, they considered off-campus R&D, and some only thought about that when asked
during the interview. After the second round of interviews, participants agreed that this
clarification made the question easier to answer. Most of the participants in round 2 said that
they would think primarily of on-campus R&D activities or would need to ask for more
clarification if this instruction was not there.
Changed the response options to reflect performance of R&D activities versus
disruption/shutdown of R&D. Participants during round 1 did not think that selecting a
response would be easy with the response options provided. During round 2, participants said
that the new response options (e.g., “Could not perform any R&D,” “Could perform almost all
R&D”) made sense and would make it easier to respond to this question.
Other minor changes were made for consistency with other questions on the HERD survey.
We moved the instruction “Your best estimate is fine” to the second bullet and changed “fine”
to “acceptable” to be consistent with other questions on the HERD survey. We also changed and
moved the instruction from the “check one” response option to “select one” directly above the
response options.
Question B: Diverted R&D Funds
Diverted Funds (Question B1)
Most participants understood that diverted funds meant the intent of the original funds changed to
something else, COVID-19 related in this case. There were a couple of participants who said that they
were not sure how they would do this in the case of sponsored research, but they did understand the
question and thought it would be easy to answer. Most of the participants said that they did not divert
funds for this purpose.
ICF
6
During round 1, Question B1 did not contain information on types of R&D funds, so participants were
first asked what R&D funds they thought of when reading this question. Five participants said that they
thought of sponsored or externally funded research, two participants questioned whether they should
include institutionally funded R&D, and two participants said that they thought immediately of both
sponsored and institutionally funded R&D.
Five participants were asked if they tracked diverted funds (regardless of whether they diverted funds in
this instance), and three participants said that they would be tracked, but two participants said that they
would not be easily and/or consistently tracked.
Percentage of Diverted Funds (Question B2)
Four participants said that Question B2 (the percentage of funds diverted) would be easy to answer if
they diverted funds, while three participants said it would difficult. However, they also said that the
instruction that “Your best estimate is acceptable” would help in answering the question.
Only one participant said that the response options did not make sense. This participant said that asking
for percentages could be confusing, as it is not clear where the percentage of funds is from (e.g., all
active awards and funds received to date or just expenditures for the fiscal year). This participant did not
have diverted funds. All other participants in round 1 and in round 2 thought that the response options
and ranges of percentages made sense.
Changes to Question B Between Rounds
Following the first round of interviews, we made the following changes to Question B. No changes were
made after the second round of interviews.
•
•
•
Added instruction to think about all R&D funds including sponsored research, unrestricted
gifts, and institutionally financed R&D to Question B1. Before the instruction was added, five
participants said that they only thought about sponsored research. During round 2, three of the
participants said that without this added instruction, they would only think of sponsored
research, and all noted that this instruction was important to include.
Moved response options yes and no under Question B1. At the request of an NCSES survey
methodologist, these options were moved from the right of the question to directly below it to
be more intuitive for the participants.
Minor changes were made to Question B2. We moved the instruction “Your best estimate is
fine” to a second bullet and changed “fine” to “acceptable” to be consistent with other
questions on the HERD survey. We also changed and moved the instruction from the “check
one” response option to “select one” directly above the response options.
Question C: New R&D Funds Received
Receiving New Funds (Question C1)
Participants understood this question to be asking about external funds received, however, there was
confusion during round 1 on what type of R&D should be included and if funds other than R&D that
were received because of COVID-19 should be included. Two participants asked if this was intended to
include Coronavirus Aid, Relief, and Economic Security (CARES) Act funding or other funding that was
received because of COVID-19 but may not have specifically been intended for R&D.
While not all participants said that their institution received R&D funding as a result of the COVID-19
pandemic, those that did said this would be tracked in their systems. A couple of others that did not
ICF
7
receive funding said that they thought this would be tracked, but they were not sure how well it would
be tracked to be able to answer this question. While responding to this, one institution asked if the
funds needed to be spent in FY 2020.
External Sources of New Funds (Question C2)
All participants understood this question asking for the funding agency or sponsor of the new funds
received. Some participants said that the types of sources listed were familiar and similar to other HERD
survey questions. They also said that this question would be easy to answer if they received new funds,
because this is how their institution tracks new funds.
When asked if any sources of funding were missing, the only source that two participants said was
international/foreign and one participant said gifts or donations.
Changes to Question C Between Rounds
Following the first round of interviews, the following changes were made to Question C.
•
•
•
•
•
•
ICF
Changed wording of Question C1 to clarify new funds for R&D could be added as result of
COVID-19 and added the instruction: “This could include R&D that is not specifically related to
COVID-19.” This phrasing was changed and added to because there was confusion during round
1 on what type of R&D should be included and if funds other than R&D that were received
because of COVID-19 should be included. During round 2, there was more clarity because of
these additions, but it did not completely clear up the intent of the question for respondents.
Two round 2 respondents still thought that this could mean funds received because of COVID-19
but not related to R&D, while one thought that it was only referring to R&D specifically related
to COVID-19. Therefore, an additional change was made following round 2.
Added clarity and examples to the instruction: “This could include R&D that is not specifically
related to COVID-19.” Adding this instruction before the second round of interviews helped
participants understand that the new funds did not need to be for COVID-19 research. However,
a couple of participants said that this instruction initially caused confusion and seemed to make
them think about non-R&D funds. Participants from two institutions said that it would be useful
to include an example of this type of research.
Added the timeframe of FY 2020 to Question C1 and added the instruction: “Funds need not
be spent in FY 2020.” The timeframe and instruction were added for clarity. This was in part
because one institution asked if the funds needed to be spent in FY 2020 and because the FY
2020 survey is fielded during institutions’ FY 2021.
Moved response options yes and no under Question C1. At the request of an NCSES survey
methodologist, these options were moved from the right of the question to directly below it to
be more intuitive for the participants.
Removed the instruction to refer to Question 1 of the survey if more information is needed
about the types of sources list as response options on Question C2. NCSES recommended
removing this reference. Because the information in Question 1 includes a lot of text, we did not
want to add it to this question. During round 2, participants were asked if a reference to
Question 1 would be helpful in answering this question, and they all said that it would, even
though some said that it was not necessary for their response. Therefore, this reference was
added back to the final question.
Added “including foreign governments, other universities and gifts” to the Other sources
option on Question C2. During round 1, these were noted by two participants as missing from
the list. During round 2, participants said that there were no sources missing from the list.
8
•
Changed and moved the instruction from “check all that apply” to “select all that apply”
directly above the response options. These changes were made to be consistent with other
questions on the HERD survey.
Results from Webinar Polls
During the annual HERD survey update webinar, NCSES presented the three questions and conducted a
poll for each, asking attendees how easy or difficult it would be to answer each question should they
appear on the FY 2020 HERD survey. See appendix F for the final questions presented during the
webinar. Each proposed question was presented separately on a PowerPoint presentation. Attendees
were given 1 to 2 minutes to read the question before a poll question appeared in a separate window on
the screen. The poll question asked, “How easy or difficult would it be to answer Question [A] for the FY
2020 HERD survey?” Participants were given another minute to respond. For questions B and C, which
both included two parts, attendees were asked to provide feedback about both parts of the questions.
There were 666 attendees from 483 institutions (390 standard form and 93 short form). Not all
attendees responded to each poll question, but some institutions were represented more than once per
poll. Appendix G provides full poll results for each question.
There were 565 attendees (85 short form and 480 standard form) that responded to Question A. Most
participants said Question A would be easy (38.2%) or neither difficult nor easy (37.9%) to answer on the
FY 2020 survey; more short form institutions said it would be easy (45.9%) while standard form
institutions were still split between easy (36.9%) and neither difficult nor easy (39.6%).
There were 543 attendees (79 short form and 464 standard form) that responded to Question B.
Participants were clustered between responding that Question B1 would be easy (29.3%), neither
difficult nor easy (27.1%), or difficult (30.2%) to answer on the FY 2020 survey. Short form institutions
were more likely to respond that it would be easy (38.0%), while standard form institutions were more
likely to say that it would be difficult (31.3%). Institutions were most likely to respond that answering
Question B2 would be difficult (39.2%).
There were 550 attendees (79 short form and 471 standard form) that responded to Question C. Nearly
half (47.8%) responded that answering Question C1 would be easy, and 44.7% said that answering
Question C2 would be easy.
ICF
9
APPENDIX A: Cognitive Interview Communications
Contact Materials for COVID-19 Question Interviews
Invitation
To: Primary contact
Cc: others normally included in survey contacts
From: NSF Higher Education R&D Survey
Re: HERD Survey: Tracking Impacts of the COVID-19 Pandemic on R&D ([[inst_id]])
Dear [[firstname]] [[lastname]]:
Thank you, again, for completing the FY 2019 Higher Education Research and Development (HERD)
Survey. I appreciate your time and efforts.
I am writing today to request your assistance in our attempt to measure the impacts of the COVID-19
pandemic on higher education research and development (R&D). The National Center for Science and
Engineering Statistics within the National Science Foundation is considering adding a few questions on
this topic to the FY 2020 survey. Before we finalize the questionnaire, we would like to get feedback
from current survey respondents about the questions and how impacts of the pandemic on research
activities are being tracked at your instituiton.
If you are willing to help, we would ask you to do two things:
1) Review a draft of the COVID-19 questions and identify the best person(s) at your instituton for
us to interview. Because these are not typical HERD survey questions, you may find that you are
not able to answer these questions on your own. We will send you the questions to review
before the interview, so you have time to discuss them with others at your instituiton.
2) Participate in a 15 minute virtual interview. During the interview, we will ask you, and any
others you would like to invite, about your ability to answer the potential survey questions and
how your institution is tracking the impacts of the pandemic on research activities. We would like
to complete the interviews by August 28.
If you, or someone else from your instituton, is able to help, please contact Cameron Shanton from our
survey staff at ICF at [email protected] or 866-936-9376. If we do not hear from you by August
24, we will contact you again. Cameron will send you the questions to review and follow-up later to
schedule a time for the interview.
Thank you for your help and for your continued participation in the NSF HERD Survey.
Sincerely,
Michael Gibbons
Project Officer, Higher Education R&D Survey
National Center for Science and Engineering Statistics
National Science Foundation
A-1
Invitation Additional Information – Sending Questions to Confirm Participation
To: Person(s) who volunteered to participate
Cc:
From: NSF Higher Education R&D Survey
Re: HERD Survey: COVID-19 Pandemic test questions ([[inst_id]])
Dear [[firstname]] [[lastname]]:
Thank you for agreeing to participate in a brief interview on potential new questions for the FY 2020
Higher Education Research and Development (HERD) Survey about the impacts of the COVID-19
pandemic on R&D activity.
The questions we’ll be discussing during the interview are attached to this email. During the interview,
we’ll ask about your ability to answer the questions and your general understanding of the questions.
We will also ask some questions about how the impacts of the pandemic on research activities are being
tracked at your instituiton.
Please review the questions, discuss with your colleauges, and contact me to schedule a 15-minute
follow-up interview. We would like to complete the interviews by August 28. You can contact me at
[email protected] or 866-936-9376.
If you are not the best person to answer these questions, we ask that you invite that person to the
interview, or ask them to contact me to schedule an interview.
Thank you for your help and for your continued participation in the NSF HERD Survey.
Sincerely,
Cameron Shanton
Data Collection Specialist
Higher Education R&D Survey
ICF
A-2
Invitation – Reminder E-Mail
To: Primary contact
Cc: others normally included in survey contacts
From: NSF Higher Education R&D Survey
Re: Please reply: Tracking Impacts of the COVID-19 Pandemic on R&D in the NSF-NCSES HERD Survey
([[inst_id]])
Dear [[firstname]] [[lastname]]:
A few days ago, we sent you an e-mail asking for your assistance. We are considering adding a few
questions to the FY 2020 Higher Education Research and Development (HERD) Survey about the impacts
of the COVID-19 pandemic on R&D activity. We want to be sure we understand the impacts of the
pandemic on the research endeavor and the abilities of universities to track these impacts. We would
greatly appreciate your input on our question development.
If you would like to speak with us about the effects of the pandemic on R&D at your institution, we
would ask you to do two things:
1) Review a draft of the COVID-19 questions and identify the best person(s) at your instituton for
us to interview. Because these are not typical HERD survey questions, you may find that you are
not able to answer these questions on your own. We will send you the questions to review
before the interview, so you have time to discuss them with others at your instituiton.
2) Participate in a 15 minute virtual interview. During the interview, we will ask you, and any
others you would like to invite, about your ability to answer the potential survey questions and
how your institution is tracking the impacts of the pandemic on research activities. We would like
to complete the interviews by August 28.
If you, or someone else from your instituton, is able to help, please contact Cameron Shanton from our
survey staff at ICF at [email protected] or 866-936-9376. Cameron will send you the questions
to review and follow-up later to schedule a time for the interview.
Thank you for your help and for your continued participation in the NSF HERD Survey.
Sincerely,
Michael Gibbons
Project Officer, Higher Education R&D Survey
National Center for Science and Engineering Statistics
National Science Foundation
A-3
Round 2 invitation to those who expressed interest in round 1 but were not interviewed
Send in reply to last e-mail from respondent.
To: Primary contact
Cc: others normally included in survey contacts
From: NSF Higher Education R&D Survey
Re: HERD Survey: Tracking Impacts of the COVID-19 Pandemic on R&D ([[inst_id]])
Dear [[firstname]]:
In August you mentioned your willingness to help review some questions we were considering adding to
the HERD survey about the impacts of COVID-19 (see e-mail string below). At the time, we had already
scheduled all the interviews we could.
We made revisions to the questions after we completed the first round of interviews and we’d like to
conduct a few more interviews about the questions before we finalize the FY 2020 questionnaire.
If you, or someone else from your institution, is still able to help, you may reach me by e-mail at
[email protected] or call toll-free at 866-936-9376. We would like to complete this next round
by September 16.
Thank you for your help and for your continued participation in the NSF HERD Survey.
Sincerely,
Cameron Shanton
Data Collection Specialist
Higher Education R&D Survey
ICF
A-4
Round 2 invitation to those who were invited in round 1 but never responded
To: Primary contact
Cc: others normally included in survey contacts
From: NSF Higher Education R&D Survey
Re: HERD Survey: Tracking Impacts of the COVID-19 Pandemic on R&D ([[inst_id]])
Dear [[firstname]] [[lastname]]:
In August, we sent you an e-mail requesting your assistance in our attempt to measure the impacts of
the COVID-19 pandemic on higher education research and development (R&D). In that e-mail, we said
that the National Center for Science and Engineering Statistics within the National Science Foundation is
considering adding a few questions on this topic to the FY 2020 survey.
We conducted several interviews already and made revisions to the questions. Before we finalize the
questionnaire, we would like to conduct a few more interviews. If you are willing to help, we would ask
you to do two things:
1) Review a draft of the COVID-19 questions and identify the best person(s) at your
instituton for us to interview. Because these are not typical HERD survey questions, you
may find that you are not able to answer these questions on your own. We will send you
the questions to review before the interview, so you have time to discuss them with
others at your instituiton.
2) Participate in a 30 minute virtual interview. During the interview, we will ask you, and
any others you would like to invite, about your ability to answer the potential survey
questions and how your institution is tracking the impacts of the pandemic on research
activities. We would like to complete the interviews by September 16.
If you, or someone else from your institution, is able to help, please contact Cameron Shanton from our
survey staff at ICF at [email protected] or 866-936-9376. Cameron will send you the questions to
review and follow-up later to schedule a time for the interview.
Thank you for your help and for your continued participation in the NSF HERD Survey.
Sincerely,
Michael Gibbons
Project Officer, Higher Education R&D Survey
National Center for Science and Engineering Statistics
National Science Foundation
A-5
Confirmation of Interview – Sent in round 1 and round 2
To: Person(s) who volunteered to participate
CC: other attendees, including relevant NCSES and ICF staff
From: NSF Higher Education R&D Survey
Re: Confirming interview ([[inst_id]])
Dear [[firstname]] [[lastname]]:
Thank you for agreeing to talk to us about possibly adding questions to measure the effects of the
COVID-19 pandemic on R&D for the FY 2020 Higher Education R&D (HERD) Survey.
The [teleconference/video conference] is scheduled for DayoftheWeek, Month XX, at X:XX AM/PM
(EDT/CDT/MDT/PDT). At that time please [call our conference line at 1-855-423-6338, conference ID is
450349/click on the link below to join the Skype meeting]. Sherri Mamon will be leading the interview. I
would be happy to send you a Microsoft Outlook invitation, if you prefer.
In advance of the interview, we would like to request your agreement with the attached informed
consent. Please either return a signed copy of the attachment or reply to this e-mail stating your
consent.
If you have further questions or need to reschedule, you may reach me by email at
[email protected] or call toll-free at (866) 936-9376.
Thank you very much for your help.
Sincerely,
Cameron Shanton
Data Collection Specialist
Higher Education R&D Survey
ICF
A-6
Reminder of Interview – Sent in round 1 and round 2
This will be sent to institutions one business day before their scheduled interview, but only if they
received the first confirmation at least 2 days before the scheduled interview.
To: Person(s) who volunteered to participate
CC: other attendees, including relevant ICF staff
From: NSF Higher Education R&D Survey
Re: Interview Reminder ([[inst_id]])
Dear [[firstname]] [[lastname]]:
This is just a reminder about our interview scheduled for DayoftheWeek, Month XX, at X:XX AM/PM
(EDT/CDT/MDT/WDT). At that time please [call our conference line at 1-855-423-6338, conference ID is
450349/click on the link below to join the Skype meeting].
You do not need to do anything to prepare for this meeting. [add in if informed consent has not yet been
received, However, please return a signed copy of the attached informed consent or reply to this e-mail
that you consent.]
If you have any questions or need to reschedule, you may reach me by email at
[email protected] or call toll-free at (866) 936-9376.
Thank you very much for your help.
Sincerely,
Sherri Mamon
Qualitative Researcher
Higher Education R&D Survey
ICF
A-7
Consent Form
The National Center for Science and Engineering Statistics (NCSES), within the National Science
Foundation (NSF), designs, conducts, and sponsors surveys on science, engineering, and technology. In
order to produce the best information possible, NCSES routinely evaluates its surveys.
You have volunteered to take part in a study to improve one of the NCSES’s surveys. In order to have a
complete record of your comments, your interview session will be recorded (audio and/or video). We
plan to use the recordings to improve the survey. Staff directly involved in this research project will have
access to the recordings. In addition, a transcription service may have access to the recordings, to
provide us with a written version of your comments. They may also be used for training others to
conduct this type of research, and in presentations to professional audiences. Specific results or quotes
will not be attributed directly to you in any summaries or reports.
This study is authorized by law (42 U.S.C. 1862 Section 3.a.6.). The OMB control number for this study is
3145-0174.
I have volunteered to participate in this study, and I give permission for my recordings to be used for the
purposes stated above.
_________________________
Researcher’s Signature
____________________________
Participant’s Signature
_________________________
Printed Name
____________________________
Printed Name
_________________________
Date
____________________________
Date
A-8
APPENDIX B: Round 1 Cognitive Interview Test Questions
Question A.
To what degree were your institution’s FY 2020 R&D activities disrupted due to the COVID-19
pandemic? Your best estimate is fine.
(check one):
Fully shut down R&D
Almost fully shut down R&D
Partially shut down R&D
Minimally disrupted R&D
Did not affect R&D at all
Question B.
1. In FY 2020, were any R&D funds diverted from their originallyintended research focus as a result of the COVID-19 pandemic?
Yes
No
Go to Question B2
Go to Question C
2. About what percentage of R&D funds did your institution divert for this purpose? Your best estimate is fine.
(check one):
81 – 100%
61 – 80%
41– 60%
21 – 40%
1 – 20%
Question C.
Yes
1. Did your institution receive new funds for R&D related to COVID-19?
No
Go to Question C2
Section complete.
2. What were the external sources of funds? (check all that apply)
If needed, see Question 1 of the survey for more information about what is included in each source.
Federal government
State or local government
Business
Nonprofit organization
Other sources
B-1
APPENDIX C: Round 2 Cognitive Interview Test Questions
Question A.
In June 2020, to what degree were your institution’s expected R&D activities disrupted as a result of the
COVID-19 pandemic?
• Think about all R&D activities, including on-campus and off-campus activities.
• Your best estimate is acceptable.
(Select one)
Could not perform any R&D
Could perform very little R&D
Could perform some R&D
Could perform almost all R&D
Could perform all expected R&D
Question B.
1. In FY 2020, were any R&D funds diverted from their originally-intended research focus as a result of the
COVID-19 pandemic?
• Think about all R&D funds including sponsored research, unrestricted gifts, and institutionally financed R&D.
Yes
No
Go to Question B2
Go to Question C
2. About what percentage of R&D funds did your institution divert for this purpose?
• Your best estimate is acceptable.
(Select one):
81 – 100%
61 – 80%
41– 60%
21 – 40%
1 – 20%
C-1
Question C.
1. In FY 2020, did your institution receive new funds for R&D as a result of the COVID-19 pandemic?
• This could include R&D that is not specifically related to COVID-19.
• Funds need not be spent in FY 2020.
Yes
No
Go to Question C2
Section complete
2. What were the external sources of funds?
(Select all that apply)
Federal government
State or local government
Business
Nonprofit organization
Other sources, including foreign governments, other universities and gifts
C-2
APPENDIX D: Round 1 Cognitive Interview Protocol
COVID-19 Questions to Add to FY 2020 HERD Survey - Interview Protocol
Introduction
[Interviewer introduces herself.] Thank you very much for talking to me today. As you know from our
initial contact, we’re considering adding some questions related to the COVID-19 pandemic and how this
is causing disruptions to R&D activities at colleges and universities.
Before we begin our interview, I want to review some information with you.
• The interview should last approximately 15 minutes.
• There are other people on the phone. Some work with me at ICF on the HERD team and [if
appropriate] others are from NCSES at NSF. They are listening to our discussion today and you
may hear from them towards the end of the interview. Is that okay with you?
• I received your informed consent earlier. Any questions?
• [If informed consent was not received, review form and obtain verbal consent.]
1. For the record, can you tell me your title(s) and your role(s) in the HERD Survey?
You should have received the proposed new questions in an e-mail. If you aren’t viewing them already,
please refer to that attachment for our discussion. If you need me to resend it, let me know.
Effects of the Pandemic
Let’s start broadly.
2. Can you give us a brief sense of how research at your institution has changed over the past few
months (since mid-March)?
3. What words are your institution using to describe the effects of the pandemic on research?
a. [If needed, suggest words] halted, shut down, disrupted, uncertain, decreasing,
impacted, increasing, thriving?
Question A (degree R&D disrupted)
Now let’s take a look at Question A.
4. In your own words, what do you think this question is asking?
5. What were you thinking of as “R&D activities” when reading this question?
6. How useful are the response options?
a. Would you have any trouble deciding between any of the specific options such as fully
shut down, almost fully shut down, partially shut down or minimally shut down?
i. What do each of the categories mean to you?
1. Shut down for a week, a month, some other time period? Currently still
shut down?
2. Or do the categories reflect a percentage of researchers continuing to
work?
3. What would have needed to happen to move to a higher or lower
category?
b. Do you think it will be easy or difficult to generalize across your entire institution? Can
you tell me more about that?
7. What time period were you thinking of?
8. How easy or difficult do you think it would be to report about FY 2020 R&D activity only?
D-1
Question B (R&D funds diverted)
Now let’s move to Question B and start by looking at Question B1.
9. What R&D funds do you think of when reading this question?
a. Did you think about externally sponsored projects? Funds from unrestricted gifts?
Departmental funding or other institutional funds.
10. How would you define “diverted funds” in your own words?
11. Would you find this question easy or difficult to answer? Please explain.
12. Are diverted funds tracked in your institution’s systems?
Now let’s look at Question B2.
13. How easy or difficult would it be to answer this question?
14. Did you see the instruction “Your best estimate is fine”? Would this help you answer the
question?
a. If we did not have that instruction, would you assume to use your best estimate?
15. How would you go about answering this question?
a. Probe on whether or not records are available, who has access to them….
16. Did the ranges in the response options provided make sense to you?
a. Would it be easier with broader ranges of percentages? Smaller ranges of percentages?
Or some other response categories?
Question C (new R&D funds)
Now let’s move to Question C and start by looking at Question C1.
17. What specific funds or types of funds are you thinking about when reading this question?
a. Are you thinking about external funds only? Are you thinking of any unrestricted funds
or just sponsored R&D?
18. What kind of R&D projects were you thinking about when reading this question?
19. What does “R&D related to COVID-19” mean to you?
a. Do you think your institution received any funding that would meet these criteria? [If
yes] Can you tell me more about that?
20. Are these new funds tracked in your institution’s systems?
Let’s look at Question C2.
21. In your own words, what do you think this question is asking?
22. How would you answer this question for your institution?
23. (If respondent selected “other sources”) What sources were you thinking of?
24. Are there any sources of funds that you think should be added to the list?
25. How easy or difficult do you think this question is to answer?
Wrap Up
26. Are there any questions that you think would be more appropriate to ask in our attempt to
capture disruptions to R&D related to the COVID-19 pandemic?
Those are all of the questions I had for you today. Did you have any other comments for us, or things we
should keep in mind?
D-2
Thank you for taking the time to meet with us to discuss the effects of the pandemic on research at your
college/university. We will let you know as soon as we can if these questions (or versions of these
questions) are added to the FY 2020 HERD Survey.
D-3
APPENDIX E: Round 2 Cognitive Interview Protocol
COVID-19 Impact Questions to Add to FY 2020 HERD Survey - Interview Protocol
Introduction
[Interviewer introduces herself.] Thank you very much for talking to me today. As you know from our
initial contact, we’re considering adding some questions related to the COVID-19 pandemic and how this
is causing disruptions to R&D activities at colleges and universities.
Before we begin our interview, I want to review some information with you.
• The interview should last approximately 30 minutes.
• There are other people on the phone. Some work with me at ICF on the HERD team and [if
appropriate] others are from NCSES at NSF. They are listening to our discussion today and you
may hear from them towards the end of the interview. Is that okay with you?
• I received your informed consent earlier. Any questions?
• [If informed consent was not received, review form and obtain verbal consent.]
1. For the record, can you tell me your title(s) and your role(s) in the HERD Survey?
You should have received the proposed new questions in an e-mail. If you aren’t viewing them already,
please refer to that attachment for our discussion. If you need me to resend it, let me know.
Effects of the Pandemic
Let’s start broadly.
2. Can you give us a brief sense of how research at your institution has changed over the past
several months (since mid-March)?
3. What words are your institution using to describe the effects of the pandemic on research?
a. [If needed, suggest words] halted, shut down, disrupted, uncertain, decreasing,
impacted, increasing, thriving?
Question A (degree R&D disrupted)
Now let’s take a look at Question A.
4. In your own words, what do you think this question is asking?
5. What does “in June 2020” mean to you?
a. Are you thinking about the entire month, parts of the month, end of the month?
b. Does the timeframe of June 2020 make it easier or more difficult to answer this
question?
6. Does the instruction to think about all R&D activities including on campus and off campus
activities make it easier or more difficult to answer this question?
a. Do you think it will be easy or difficult to generalize across your entire institution? Can
you tell me more about that?
b. If we didn’t have this instruction, what R&D activities would you think about in
answering this question?
c. Does your institution have R&D activities happening both on-campus and off-campus?
7. How useful are the response options?
a. Would you have any trouble deciding between any of the specific options?
i. What do each of the categories mean to you?
1. Do the categories reflect a percentage of researchers continuing to
work?
2. How do you determine which response option to choose?
E-1
8. How easy or difficult do you think it would be to report about June R&D activity only?
Question B (R&D funds diverted)
Now let’s move to Question B and start by looking at Question B1.
9. Does the instruction to think about all R&D funds including sponsored research, unrestricted
gifts, and institutionally financed R&D make it easier or harder to answer this question? Tell me
more about that.
a. If we didn’t have this instruction, what would you think should be included in this
question?
10. How would you define “diverted funds” in your own words?
11. Are diverted funds tracked in your institution’s systems?
Now let’s look at Question B2.
12. How easy or difficult would it be to answer this question?
13. Did you see the instruction “Your best estimate is acceptable”? Would this help you answer the
question?
a. If we did not have that instruction, would you assume to use your best estimate?
14. How would you go about answering this question?
a. Probe on whether or not records are available, who has access to them….
15. Did the ranges in the response options provided make sense to you?
a. Would it be easier with broader ranges of percentages? Smaller ranges of percentages?
Or some other response categories?
Question C (new R&D funds)
Now let’s move to Question C and start by looking at Question C1.
16. In your own words, what do you think this question is asking?
a. What does “R&D as a result of the COVID-19 pandemic” mean to you?
b. What does the instruction that “This could include R&D that is not specifically related to
COVID-19” mean to you?
c. Do you think your institution received any funding that would meet these criteria? [If
yes] Can you tell me more about that?
17. What specific funds or types of funds are you thinking about when reading this question?
18. What kind of R&D projects were you thinking about when reading this question?
19. The question currently asks about your institutions receiving new funds. Would this question be
easier to answer if it asked if your institution was awarded new funds?
20. Does the instruction “Funds need not be spent in FY 2020” make it easier or more difficult to
answer this question?
21. Are these new funds tracked in your institution’s systems?
Let’s look at Question C2.
22. How easy or difficult do you think this question is to answer?
a. Do you understand the types of funds included in each response option?
b. These sources are the same used in Question 1 of the HERD survey. Do you think more
information about these sources or making a reference to Question 1 would be helpful
in answering this question?
23. How would you answer this question for your institution?
24. (If respondent selected “other sources”) What sources were you thinking of?
E-2
25. Are there any sources of funds that you think should be added to the list?
Wrap Up
26. Are there any questions that you think would be more appropriate to ask in our attempt to
capture disruptions to R&D related to the COVID-19 pandemic?
Those are all of the questions I had for you today. Did you have any other comments for us, or things we
should keep in mind?
Thank you for taking the time to meet with us to discuss the effects of the pandemic on research at your
college/university. We will let you know as soon as we can if these questions (or versions of these
questions) are added to the FY 2020 HERD Survey.
E-3
APPENDIX F: Final COVID-19 Impact Questions
Question A.
In June 2020, to what degree were your institution’s expected R&D activities disrupted as a result of the
COVID-19 pandemic?
• Think about all R&D activities, including on-campus and off-campus activities.
• Your best estimate is acceptable.
(Select one)
Could not perform any R&D
Could perform very little R&D
Could perform some R&D
Could perform almost all R&D
Could perform all expected R&D
Question B.
1. In FY 2020, were any R&D funds diverted from their originally-intended research focus as a result of the
COVID-19 pandemic?
• Think about all R&D funds including sponsored research, unrestricted gifts, and institutionally financed R&D.
Yes
No
Go to Question B2
Go to Question C
2. About what percentage of R&D funds did your institution divert for this purpose?
• Your best estimate is acceptable.
(Select one):
81 – 100%
61 – 80%
41– 60%
21 – 40%
1 – 20%
F-1
Question C.
1. In FY 2020, did your institution receive new funds for R&D as a result of the COVID-19 pandemic?
• This could include funding for R&D that is indirectly related to COVID-19 (e.g., effects of virtual learning on
student performance, economic impacts of community job loss) or supplemental funding for ongoing R&D (e.g.,
modifications to support extended timelines or purchases of additional safety equipment).
• Funds need not be spent in FY 2020.
Yes
No
Go to Question C2
Section complete
2. What were the external sources of funds?
• If needed, see Question 1 of the survey for more information about what is included in each source.
(Select all that apply)
Federal government
State or local government
Business
Nonprofit organization
Other sources, including foreign governments, other universities and gifts
F-2
APPENDIX G: FY 2020 HERD Survey Update Webinar Poll Results
All Respondents
Question A
All A Response
Row Labels
Very easy
44
7.8%
Easy
216
38.2%
Neither difficult nor easy
214
37.9%
Difficult
86
15.2%
Very difficult
5
0.9%
Grand Total
565
100.0%
Short Form Respondents
Standard For Respondents
Row Labels
Very easy
Easy
Neither difficult nor easy
Difficult
Very difficult
Grand Total
Short Form A Response
6
7.1%
39
45.9%
24
28.2%
15
17.6%
1
1.2%
85
100.0%
Row Labels
Very easy
Easy
Neither difficult nor easy
Difficult
Very difficult
Grand Total
standard form A Response
38
7.9%
177
36.9%
190
39.6%
71
14.8%
4
0.8%
480
100.0%
Question B1
Row Labels
Very easy
Easy
Neither difficult nor easy
Difficult
Very difficult
Grand Total
All B1 Response
33
6.1%
159
29.3%
147
27.1%
164
30.2%
40
7.4%
543
100.0%
Row Labels
Very easy
Easy
Neither difficult nor easy
Difficult
Very difficult
Grand Total
Short Form B1 Response
6
7.6%
30
38.0%
22
27.8%
19
24.1%
2
2.5%
79
100.0%
Row Labels
Very easy
Easy
Neither difficult nor easy
Difficult
Very difficult
Grand Total
Standard Form B1 Response
27
5.8%
129
27.8%
125
26.9%
145
31.3%
38
8.2%
464
100.0%
Question B2
Row Labels
Very easy
Easy
Neither difficult nor easy
Difficult
Very difficult
Grand Total
All B2 Response
14
2.6%
79
14.5%
139
25.6%
213
39.2%
98
18.0%
543
100.0%
Row Labels
Very easy
Easy
Neither difficult nor easy
Difficult
Very difficult
Grand Total
Short Form B2 Response
5
6.3%
16.5%
13
29
36.7%
23
29.1%
9
11.4%
79
100.0%
Row Labels
Very easy
Easy
Neither difficult nor easy
Difficult
Very difficult
Grand Total
Standard Form B2 Response
9
1.9%
66
14.2%
110
23.7%
190
40.9%
89
19.2%
464
100.0%
Question C1
Row Labels
Very easy
Easy
Neither difficult nor easy
Difficult
Very difficult
Grand Total
All C1 Response
69
12.5%
263
47.8%
125
22.7%
77
14.0%
16
2.9%
550
100.0%
Row Labels
Very easy
Easy
Neither difficult nor easy
Difficult
Very difficult
Grand Total
Short Form C1 Response
14
17.7%
34
43.0%
21
26.6%
8
10.1%
2
2.5%
79
100.0%
Row Labels
Very easy
Easy
Neither difficult nor easy
Difficult
Very difficult
Grand Total
Standard Form C1 Response
55
11.7%
229
48.6%
104
22.1%
69
14.6%
14
3.0%
471
100.0%
Question C2
Row Labels
Very easy
Easy
Neither difficult nor easy
Difficult
Very difficult
Grand Total
All C2 Response
54
9.8%
246
44.7%
161
29.3%
70
12.7%
19
3.5%
550
100.0%
Row Labels
Very easy
Easy
Neither difficult nor easy
Difficult
Very difficult
Grand Total
Short Form C2 Response
14
17.7%
31
39.2%
26
32.9%
8.9%
7
1
1.3%
79
100.0%
Row Labels
Very easy
Easy
Neither difficult nor easy
Difficult
Very difficult
Grand Total
Standard Form C2 Response
40
8.5%
215
45.6%
135
28.7%
63
13.4%
18
3.8%
471
100.0%
G-1
| File Type | application/pdf |
| Author | Gibbons, Michael |
| File Modified | 2020-12-08 |
| File Created | 2020-12-08 |