HERD Personnel Questions Cognitive Interview Report

Higher Education Research and Development (HERD)
Proposed Personnel Questions Testing

Report and Recommendations

July 23, 2020

prepared for
National Science Foundation
National Center for Science and Engineering Statistics
by

ICF • 530 GAITHER ROAD • ROCKVILLE, MD 20850

Table of Contents
Introduction .................................................................................................................................................. 4
Methodology................................................................................................................................................. 4
Question Development ............................................................................................................................. 4
Obtaining OMB Approval .......................................................................................................................... 5
Testing Timeline ........................................................................................................................................ 5
Sample....................................................................................................................................................... 5
Recruitment .............................................................................................................................................. 6
Participants ........................................................................................................................................... 7
Testing Format .......................................................................................................................................... 7
Detailed Findings........................................................................................................................................... 8
Test Question 1: Headcounts .................................................................................................................... 8
Function ................................................................................................................................................ 9
R&D Personnel Included ..................................................................................................................... 11
Demographic and Educational Categories .......................................................................................... 11
Changes to Test Question 1 Between Rounds .................................................................................... 12
Test Question 2: Full-time Equivalents ................................................................................................... 12
Methods of Calculating FTEs ............................................................................................................... 13
Changes to Test Question 2 Between Rounds: ................................................................................... 14
Estimated Time to Complete Questions ................................................................................................. 14
Recommendations ...................................................................................................................................... 15
Participant Questions and Other Changes Made ....................................................................................... 16
Participant Questions.............................................................................................................................. 16
Changes to Interview Protocol Throughout............................................................................................ 16
Appendices
A. Preliminary Cognitive Interview Communications
B. Preliminary Cognitive Interview Test Questions
C. Preliminary Cognitive Interview Protocol
D. Round 1 Initial Interview Communications
E. Round 1 Test Questions
F. Round 1 Initial Interview Protocol
G. Round 1 Debrief Interview Communications
H. Round 1 Debrief Interview Protocol
I. Round 2 Initial Interview Communications
ICF

2

J. Round 2 Test Questions
K. Round 2 Initial Interview Protocol
L. Round 2 Debrief Interview Communications
M. Round 2 Debrief Interview Protocol

ICF

3

Introduction
The National Center for Science and Engineering Statistics (NCSES), within the National Science
Foundation (NSF) conducts the Higher Education Research and Development (HERD) Survey annually.
The HERD survey collects information on the financial resources allocated to R&D by universities and
colleges and is one of several surveys at NCSES that collect comparable information on R&D from
different sectors of the economy (e.g., businesses, nonprofits, and government). However, it does not
currently collect as much information about the personnel carrying out R&D activities as other NCSES
surveys, making it less effective for measurements of the R&D workforce in the United States. The
Frascati Manual, 1 which is published by the Organisation for Economic Co-operation and Development
(OECD) and is the internationally recognized methodology for collecting and using R&D statistics, has a
full chapter on measuring R&D personnel. The United States is the only country included in OECD
statistics that does not report anything in the higher education sector in terms of full-time equivalents
(FTEs). NCSES intends to address this shortcoming by adding questions to the HERD survey regarding the
number of researchers, technicians, and support staff involved in R&D, and the number of FTEs for those
personnel. The addition of these variables will allow NCSES to provide information not available
elsewhere to data users interested in science policy, the nature of the science and engineering
workforce, and U.S. R&D competitiveness.
In preparation for adding those questions to the survey, ICF conducted interviews with several higher
education institutions to get feedback on proposed questions. ICF conducted three rounds of testing of
the proposed questions. ICF first conducted a preliminary round of interviews with four institutions to
test initial understanding of the questions. Recommendations from those interviews further informed
question development. In the subsequent rounds, ICF conducted initial interviews with institutions
before the institutions completed the test questions. Following question completion, ICF conducted
debrief interviews with each institution. Between rounds, ICF provided recommendations for changes to
the interview protocols, communications, and the test questions.
This report summarizes the findings and recommendations based on the testing.

Methodology
Question Development
The Frascati Manual has a full chapter on measuring R&D personnel and measuring FTEs, headcounts,
and costs. R&D personnel is comprised of three categories: researchers, technicians, and R&D support
personnel (integral to the effort but do not work in the lab). The manual also says that countries should
report the FTEs. NCSES has been using principal investigator (PI) headcounts as a proxy for
“researchers,” and there are not any FTE counts for R&D personnel for the higher education sector.
Using the Frascati manual guidelines, the questions were collaboratively designed by ICF and NCSES and
informed by interviews during site visits with HERD institutions in 2017 and 2018, as well as a HERD
respondent workshop in September 2019. The wording for the proposed new questions is similar to
questions currently found on other NCSES surveys of R&D performers. The question and instruction
OECD (2015), Frascati Manual 2015: Guidelines for Collecting and Reporting Data on Research and Experimental
Development, The Measurement of Scientific, Technological and Innovation Activities, OECD Publishing, Paris,
https://doi.org/10.1787/9789264239012-en.

1

ICF

4

format of the HERD survey further guided the design of the questions. The questions were then revised
based on feedback throughout the interview and testing process.

Obtaining OMB Approval
ICF staff prepared a variety of documents that were included in the generic clearance package for OMB.
These documents included the following:
•

•
•
•

Communications
o Invitation e-mail (signed by Michael Gibbons)
o Invitation reminder e-mail (signed by Michael Gibbons)
o Confirmation of initial interview e-mail
o Reminder of initial interview e-mail
o Reminder to complete test questions/schedule debrief interview e-mail
o Reminder of debrief interview e-mail
Test Questions to be added to the HERD Survey
Interview Protocol for Initial Interview
Interview Protocol for Debrief Interview

NCSES received generic clearance to proceed with the interviews on April 9, 2020.

Testing Timeline
Table 1 presents the timeline for each round of testing.
Table 1. Timeline for Testing
Task
Preliminary Cognitive Testing (4 institutions)
Round 1 Testing (8 institutions)
Invitations Sent
Reminder Invitations Sent
Initial Interviews Occur
Institutions Complete Test Questions
Debrief Interviews Occur
Round 2 Testing (11 institutions)
Invitations Sent
Reminder Invitations Sent
Initial Interviews Occur
Institutions Complete Test Questions
Debrief Interviews Occur

Timeframe
March 13–20, 2020
April 13, 2020
April 17, 2020
April 15–28, 2020
April 15–May 28, 2020
May 18–29, 2020
June 1, 2020*
June 4, 2020
June 4–16, 2020
June 4–July 8, 2020
June 30–July 10, 2020

* Three institutions from round 1 were re-sent invitations to participate in round 2 due to expressed interest but not being
selected to participate in round 1.

Sample
At the time the samples were drawn for all rounds of testing, the FY 2019 HERD survey was in the field.
The sample was limited to institutions that had submitted a completed survey and had their data
approved by the data review team. Before inclusion in the sample, an institution’s contact history and
data were reviewed for anything unusual (e.g., some institutions were not selected because the
respondent was already selected as part of another institution or attended the workshop in 2019). The
ICF

5

sample was then divided into four quartiles of total R&D personnel for FY 2019. Eligible institutions were
randomly selected from the four quartile groups. Since the sample for the preliminary round was drawn
early in the FY 2019 collection, the number of qualified institutions from the fourth quartile was limited.
The first quartile was not included in this preliminary round to ensure that we obtained information
from institutions that had several R&D personnel.
For rounds 1 and 2 of the interviews, ICF selected an initial sample of 100 institutions with the intention
of sending 50 invitations in each round.

Recruitment
Table 2 includes the number of sampled, invited, and interviewed institutions in each quartile for each
round of testing. In total, there were 20 institutions in the sample for the preliminary round, and 50
institutions each for rounds 1 and 2.
Not all institutions in the sample for each round were initially invited. In the preliminary round, we first
sent invitations to approximately half of the sample randomly selected from each quartile and then sent
more invitations in the quartiles where interviews were not yet scheduled. Out of the 14 institutions
invited, 4 were selected to interview.
In round 1, 50 institutions were sampled and sent invitations and 8 were selected for an interview. In
round 2, we first sent invitations to three institutions (in quartiles 3 and 4) that expressed interest
during round 1 but were not selected for an interview at that time. We then sent invitations to 41 of the
sampled institutions from round 2. After reminder invitations still did not generate enough interest, we
sent invitations to another 18 institutions from round 1 (in quartiles 1 and 2).
Table 2. Number of Institutions Sampled, Invited, and Interviewed in Each Round
Round
Quartile
Sampled
Invited for Each Round
Interviewed
1st
0
0
0
2nd
7
3
1
Preliminary
3rd
8
8
1
4th
5
3
2
Total
20
14
4
1st
12
12
1
2nd
13
13
2
Round 1
3rd
12
12
3
4th
13
13
2
Total
50
50
8
1st
12
21*
4
2nd
13
22*
3
Round 2
3rd
13
14*
2
4th
12
5
2
Total
50
62
11
Total
120
126
23

* Eighteen institutions sampled from quartiles 1 and 2 in round 1 were sent invitations to participate in round 2. Three
institutions from quartiles 3 and 4 that expressed interest during round 1 but were not selected for interviews were invited to
participate in round 2 before institutions sampled for round 2 were invited. This is why quartiles 1–3 have more invitations than
sampled and quartile 4 has fewer invitations than sampled.

ICF

6

Participants
Information on the participating institutions are presented in table 3.
Table 3. Information about Institutions That Participated in Interviews
Round

Institution

Type of Control

Quartile

Preliminary
Preliminary
Preliminary
Preliminary
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2

Central Washington University
Tulane University
University of Chicago
Howard University
Western Washington University
Medical University of South Carolina
George Washington University
Central State University
Loyola Marymount University
University of North Carolina, Wilmington
Niagara University
Loyola University, Chicago
Texas Tech University
The University of Texas, Tyler
Johnson C. Smith University*
Humboldt State University
Syracuse University
California State University, Channel Islands
Worcester Polytechnic Institute
Bradley University
Swarthmore University*
Long Island University*
Skidmore College

Public
Private
Private
Private
Public
Public
Private
Public
Private
Public
Private
Private
Public
Public
Private
Public
Private
Public
Private
Private
Private
Private
Private

2nd
4th
4th
3rd
3rd
4th
4th
1st
3rd
2nd
2nd
3rd
4th
2nd
1st
3rd
4th
1st
3rd
2nd
2nd
1st
1st

* These institutions were unable to participate in all parts of the research and only participated in the initial
interview.

FY 2019 R&D
Personnel
138
1,523
4,934
629
403
1,733
3,039
103
423
312
109
835
3,785
117
57
1,017
1,641
32
801
130
171
96
60

Testing Format
As previously mentioned, there were three rounds of testing and question refinement. During the
preliminary round, participants were shown the test questions and asked to participate in 30 minute
interviews but were not required to complete the questions and no debrief interviews were conducted.
During rounds 1 and 2, participants were asked to complete two 30 minute interviews and complete the
test questions.
For all rounds, the moderator sent the participants their institution’s completed FY 2019 HERD Survey
which included the two test questions inserted following current Question 15. This allowed the
participant to review the test questions within the context of the whole survey, including the
overarching survey instructions at the beginning of the questionnaire and to related questions about
R&D activities. This was sent to participants approximately 5 minutes before the interview began to
truly gain a participant’s initial reaction to the questions.

ICF

7

Participants in rounds 1 and 2 had 3–4 weeks to complete the test questions because they would likely
need to work with other offices such as Human Resources (HR) or Institutional Research, requiring more
time than if they could complete the questions themselves. ICF scheduled 30 minute debrief interviews
with participants when reminding participants of upcoming question due dates. Between each round,
changes to the questions and protocol were sent to NCSES for approval.
We offered a video conference option to participants and two chose this option in round 1 for their
initial interviews but did not use this option in debrief interviews. Therefore, nearly all interviews were
conducted via teleconference only.

Detailed Findings
All participants generally understood what was being asked in both questions. Participants expressed
more concern over the ability to respond to Test Question 2 (FTEs) than to Test Question 1
(headcounts). Nevertheless, most participants thought that responding to the questions was possible
but would be time consuming. At the debrief interview, two institutions were unable to provide some
data due to time constraints at their institution that coincided with testing, but in most cases thought
that they would be able to respond completely if the questions were included in the FY 2020 survey.
However, there were four participants who did not think the data would be available for future surveys.
All institutions were able to provide headcounts, but two institutions said that citizenship would not be
available. Another two institutions said that information about the highest degree of education would
not be available. In all cases, it was because this information was not tracked at their institution. Only
one institution said that FTE would not be available because of the complexity of the calculation across
units and employment statuses at the institution.
All of the participants said that they would need to ask HR and/or individual colleges for some of the
information requested in the questions. However, for this effort a few of the smaller institutions were
able to complete demographic information on their own since they know most of the research
personnel.
Below, we summarize the feedback provided by respondents for both test questions during initial
interviews and debriefs after completing the test questions.

Test Question 1: Headcounts
Once participants were given a few minutes to read the question, they were asked for their initial
reaction. Most participants focused on the demographic data asked for in the question and which
departments they would have to work with to get the information. Some of the participants only
focused on the function split, wondering out loud where they would place students or how they would
be able to determine which personnel fall under R&D technicians or support staff. This is often where
participants said that it would take a lot of time to pull the requested data. This was primarily because
either they would need to wait on HR to provide responses (noting this would not be a priority for them)
or they would need to talk to individual colleges to get the requested information. However, there were
a few participants who said that this question would not be difficult to complete because their
institution keeps this data and they would simply need to synthesize it.
During the first round of interviews, we asked participants how this test question compared to the
survey’s current Question 15, “How many principal investigators and other personnel (headcount) were
ICF

8

paid from the R&D salaries, wages, and fringe benefits you reported in Question 12, row a?” Participants
mostly said that the new question is adding demographic information or is now asking them to further
split out “all other personnel” requested in Question 15. In round 2, we did not ask this question during
the initial interviews as we felt it was leading participants to complete the question in a specific way,
and we only asked about it in debrief interviews. However, headcounts provided for Test Question 1 did
not always match totals for Question 15. Half of the participants provided total headcounts that
matched the total in Question 15, but of those, only four participants had matching PI counts (from
Question 15) and researcher counts (from Test Question 1). Of the participants whose totals did not
match:
•
•
•
•
•

Two participants provided total and researcher counts in Test Question 1 very close to the total
and PI counts in Question 15.
Three participants said that their totals should have matched Question 15 but did not because
they either provided a sample for this effort, updated reports to “do a deeper dive” into their
personnel, or they used FY 2020 data instead of FY 2019 data.
One said that Question 15 data was likely overstated and included some non-R&D personnel.
One said that this was an extension of Question 15 but never addressed why the counts were
much higher on Test Question 1.
One participant did not include PIs in Test Question 1 counts.

Of the four participants with matching total counts but non-matching researcher and PI counts, three
had higher researcher counts due to expanding it to include more than just PIs (e.g., all faculty, some
students, program directors).

Function
Participants understood that this question is asking for headcounts for staff who work in R&D functions.
Some participants said that they might use “category” instead, but participants said that they
understood the term “function” in the context of the question. During the preliminary interviews, the
question did not include definitions or clarification on which personnel to include in each function. This
was added before round 1 interviews and it provided some but not complete clarification.
There were some points of confusion or uncertainty, with the most common being how to classify types
of jobs or positions into the three research functions: researchers, R&D technicians and equivalent staff,
and R&D support staff. Many of the participants (16 of 23) wanted more clarification or instruction on
function classification. Participants noted wanting examples, guidance on titles, where to include
students, and some were unsure which R&D support staff would be included (e.g., should people in
accounting or HR be included here). Two participants noted that it might be helpful to have a clarifying
statement about R&D technician versus R&D support staff like there is for researcher versus R&D
technician.
Many participants said that their institution does not classify employees by function in the way the
question asks. There were a few participants who said that their institution does classify employees in
this way or they think using a job code would make it easy enough to determine function. However,
others were unsure initially if they would easily be able to determine personnel’s function. During the
debrief interviews, participants stated that they took various approaches to gathering this information.
Many started with a report they use annually to complete HERD’s Question 15 while others pulled
reports of employees who received R&D wages or were paid from R&D accounts. From here,
ICF

9

participants used a combination of job codes, profiles/descriptions of roles, asking colleagues, and using
their own knowledge of staff members (mostly done in small institutions) to place staff into the three
functions.
Most participants said that researchers include PIs, a research associate at a PI level that does not teach,
or faculty. However, three participants made comments in the initial interview about whether and/or
how a PI should be included in counts or not. One participant did not include PIs in the final count; note
that we think having Test Question 1 immediately following Question 15 might have added confusion
about who to include. One participant asked a question about whether the PI count was removed (but
ultimately included them in the count) while another asked what to do with a PI that is not doing the
research but is overseeing the research. Two participants asked for clarification on what “R&D
managers” means in the description.
Participants were less certain how to classify and separate R&D technicians and support staff from one
another. Several participants said if they looked at the job titles, they would assume anyone with the
term “tech” or “technician” in their title would go under R&D technician. In the debrief interviews,
participants confirmed that this was the case and that they categorized lab managers and anyone who
did not fit under researcher or support staff categories as a technician. Some participants included
students under the technician category. It is worth noting that “lab manager” is not an example
provided, and one participant wondered where someone with this title should be placed.
Support staff included clerical or administrative staff and grant specialists/administrators, and some
participants included students here.
As noted above, where participants would place students was mixed. During the preliminary round,
participants had different answers: one participant said students would be included under the
researcher or support function, one said under technician, one said under support, and one did not
know. Round 1 and 2 participants were asked where they would place students in the initial interviews
and then asked where students were included, if at all, in the debrief interview. During the initial
interviews round 1 and 2 participants again varied in their responses with the following responses:
•
•
•
•
•
•

Researcher – 1 institution
Technician – 5 institutions
Support – 2 institutions
Technician and support – 2 institutions
All functions – 1 institution
No answer/not sure – 8 institutions

During the debrief interviews, participants said they placed students in the following functions:
• Researcher – 1 institution
• Technician – 8 institutions
• Support – 3 institutions
• Technician and support – 3 institutions
• No answer/not sure – 1 institution
• Did not participate in debrief – 3 institutions
Four participants altered their response from the initial interview to the debrief interview regarding
placement of students. However, two participants narrowed down their placement of students once
ICF

10

they completed the questions (e.g., said students may be under technicians and support but placed
students under support staff when completing the question). One participant who said that students
would likely be placed under support staff placed students under technicians and support staff once
completing the question. Another participant said that students may be considered researchers during
the initial interview but placed them under technicians when completing the questions.
Two participants commented that having the categories as R&D faculty, staff, and students would be
easier and more interesting to their institution.

R&D Personnel Included
Another area of uncertainty in this question is whether participants included sponsored research and
institutionally funded research in their counts. During the initial interviews, several participants
wondered out loud if only paid personnel should be included and whether volunteers and staff paid
from indirect costs should be excluded. During the interviews, ICF and NCSES did not directly respond to
these questions but asked for more information on who was included during the debrief interviews.
Eleven of the participants included both types of research but others did not. Of those who did not
include institutionally funded research, two said that they could not pull this information easily and one
said that they could in the future. It was clear from some participants that this was not included in their
test question counts nor was it included in the survey’s current Question 15. It seemed as though
including this type of R&D did not occur to them, at least initially.

Demographic and Educational Categories
All participants agreed that once they determined where staff should be placed under the functions,
that entering the total (line a) and sex (line b) would not be a problem. There was some uncertainty
around the difficulty of obtaining information on citizenship (line c) and highest level of education
completed (line d).
Three participants did not provide citizenship data. One of the participants said that they may be able to
provide this in the future, but two said that they will not be able to because their institution does not
track this information or HR will not provide the information.
In round 1, we did not clearly specify that the highest level of education was only required for
researchers. Two participants provided degree counts for all personnel. In round 2, we clarified that this
information was only required for researchers, and all participants provided the information as
requested. In round 1, two institutions did not provide highest level of education at all and said that
they would not be able to provide it in the future either. One of these institutions provided all other
data requested while the other did not provide FTE data either.
All participants provided data for sex.
Sensitivity
Several participants expressed concern over how the demographic data, particularly citizenship and sex,
would be reported. Three smaller institutions worried that if this data were reported by institution, an
employee could be identified. Further, during the initial interviews, about half of the participants were
not sure if HR would provide the information. Participants mentioned citizenship as potentially being the
hardest demographic information to obtain, although some participants expressed concern over sex.
The concern over providing data for sex was more about whether we should ask for gender instead.
ICF

11

While this was mentioned during initial interviews, no institutions refused or were unable to provide
data for sex.
During the debrief, the participants that initially said that there may be pushback in getting demographic
information also said that they did not think their institution would have any concerns with publishing
the aggregate data by institution. Four participants who said that there may be concerns about
providing this data in the initial interview also said that there was no pushback or questions raised about
how this information would be used and/or reported. However, five participants said that they would
need approval elsewhere in the institution to know if it was acceptable to publish this data by
institution.

Changes to Test Question 1 Between Rounds
Following the preliminary interviews, a few changes were made to Test Question 1.
•

•

Descriptors of the functions were removed from Test Question 2 and added to Test Question
1. Because all of the participants asked for clarification on which staff would be included under
the functions, we removed some descriptors present under Question 2 (e.g., researchers
including R&D scientists, engineers, and their managers) and added a table with more of an
explanation of the types of staff to include under all functions in Question 1.
Added an Unknown category to citizenship section of Test Question 1. Although no
interviewee mentioned it as a shortcoming, it is a notable difference from the sections asking for
headcounts by sex and highest level of education, both of which have an unknown or not stated
option.

Following round 1 interviews, several changes were made to Test Question 1.
•

•

Replaced an instructional bullet regarding including paid personnel. We removed the bullet
“Include all personnel and students paid from R&D accounts regardless of how much they
received” and replaced it with “Include all personnel and students that worked in an R&D
function regardless of the amount of time spent in that role.” The original instruction came
directly from Question 15 on the HERD survey. That question was written so personnel
headcount was tied directly to expenditures reported on other questions. The test question was
purposefully written to be silent on the relationship between expenditures and headcount. One
participant during the first round of interviews noted that they did not report some people
because they were not “paid” for their time on specific research.
Clarified that only headcounts for researchers was desired for the highest level of education
completed. Several changes were made to clarify what information is desired because two
interviewees in round 1 mistakenly reported highest level or education for all R&D personnel
and one asked what personnel to include for this prior to completing the question.
o Added “Researchers only” to column a of Test Question 1.
o The instruction “Do not included highest level of education for R&D technicians and
R&D support staff.” was added in the blank space under columns b, c, and d of Test
Question 1 D.

Test Question 2: Full-time Equivalents
Initially, all participants said that they understood this question was asking for information on FTEs but
said they wanted more time to think about how they would calculate this for their institution. The
participants said that they would need to work with payroll or other departments to get the needed
information; however, they all said that they would personally need to make the calculations. This
ICF

12

mostly remained true once they spent time completing the questions. Two participants were unable to
provide FTE calculations. One said that timing was the main issue as he was trying to complete this
question during the institution’s year-end close and the staff he needed to help him had other priorities.
The other participant was not certain that this information would be available even for the FY 2020
survey. He explained that their institution has two different effort reporting systems and both 12-month
and 9-month appointed staff. He said it was difficult to work through the complications at this time, and
if it could not be worked through for the FY 2020 survey, it would not be reported as he would not want
to provide data that was not accurate.
Most of the participants said that the same individuals included in Test Question 1 were included in Test
Question 2. For those who said that this was not the case, it was primarily due to students who are paid
a stipend or are volunteers, and it is difficult to know their effort on a research project. Additionally,
most of the participants said that individuals were only counted under one function on Test Question 2,
but six participants said that some individuals were included or could be included in the future under
researcher and R&D technician based on their role on a research project.
In round 1, institutions reported FTEs as whole numbers and to 1 and 2 decimal places. Half of the
institutions reported whole numbers, three institutions rounded to two decimals, and one institution
rounded to one decimal in some places and two decimals in others. For round 2, we added an
instruction to round FTEs to one decimal place, and all institutions followed this instruction.

Methods of Calculating FTEs
Most participants were familiar with the concept of FTE, knew generally how it should be calculated,
and thought that the examples provided in the question made sense. Only one participant expressed
any confusion about what FTE was. A couple of participants said that their institution has a calculated
FTE in their records, but it was not for R&D specifically, so it was only minimally useful. All participants
reported doing the FTE calculation themselves or with the help of team members, even if the raw data
was provided by another office.
Participants used various ways to calculate FTEs, but there were two primary approaches: using effort
reporting or payroll accounts.
The majority of participants used effort reporting to calculate FTEs. Participants often obtained effort
reports either through HR or another department (e.g., research department). The level of detail and
ability to extract the detail varied from institution to institution. One participant said they used
personnel action forms that give hours and start and end dates for specific R&D projects. Another
participant used a similar approach by using their time and effort report, which provides the percentage
or time and time period of each staff person but noted that this was not available for part-time staff and
students and instead used the maximum number of hours these staff are allowed to work in a week.
Other participants that used time and effort reports noted that this was not necessarily available for all
R&D personnel, particularly students. In these cases, if they could provide it, they used timesheets or
tried to reach the PIs for information to determine estimates. One participant said that there is an FTE
field in the HR database that was useful in their calculation. When a job assignment is made, a
percentage of FTE time is noted, which the respondent then sums up for each individual’s various
projects.

ICF

13

Five participants used payroll accounts to calculate FTEs, dividing R&D salary by annual or annualized
salary. Most participants used payroll information and annual salary, while one participant said that they
used an individual’s FTE status and R&D pay to obtain the FTE. This allowed them to easily count 9month faculty as 1.0 FTE. A couple of participants said that they combined this approach with
calculations based on effort reporting when R&D personnel were paid hourly.
Internally funded effort was not always available or included even if it was. A couple of participants said
that they could not include internally funded effort in the calculation because the information was not
available at the time. However, these participants said that they could likely obtain this information for
the FY 2020 survey but needed more time than was allotted to complete the test questions.
Participants primarily used our example of a 40-hour work week and a 52-week year (2,080 hours) as 1.0
FTE despite our instruction to use the hours per week and weeks per year that typically represent a fulltime employee at their institution. While many staff are on the 2,080 hours, some faculty are appointed
differently, such as 9-month appointments and may do their research in the summer. Participants
expressed wanting to follow the 2,080 example we provided because it made it easier based on the
information they had to make their calculations regardless of whether they used effort reporting or
payroll accounts. We asked some participants in round 2 debrief interviews if an alternate approach was
considered and if examples for an alternate approach would be helpful, specifically FTE based on payroll.
While some participants said that a different approach was not considered, a couple of participants said
that it was considered and either a mix of approaches were used, or they ultimately did not use a
different approach. However, all of them said that examples of a different approach would be helpful.

Changes to Test Question 2 Between Rounds:
Following the preliminary interviews, one change was made to Test Question 2.
•

Descriptors of the functions were removed from Test Question 2 and added to Test Question
1. We also added “See Test Question 1 for descriptions of each function.” As the first
instructional bullet on this question.

Following round 1 interviews, several changes were made to Test Question 2.
•

Clarified what decimal place FTEs should be reported to. Participants reported FTEs at various
decimal places. Of the seven participants who provided FTEs, four participants reported whole
numbers, and three reported to the second decimal place. After discussion with NCSES, we
made several changes to specify rounding to one decimal place.
o Added the instruction to “(round to 1 decimal place)” was added under the FTEs column
header on Test Question 2.
o Added a decimal place two places over from the right margin of the data entry fields on
the question.
o Adjusted values in 3 of the examples of how to calculate FTE on Test Question 2 so that
the final amount could be easily rounded to 1 decimal place without confusion.

Estimated Time to Complete Questions
Prior to completing the questions, participants were asked to estimate the time to complete both
questions, including time spent by all persons gathering the data and the time it takes to talk to others.
Responses ranged from 2–3 hours to more than 100 hours. The participant who estimated 100 hours
was by far an outlier, and most responses ranged from 2–3 hours to 20 additional hours. Some
ICF

14

participants also gave responses in weeks versus hours and said it would take 2–4 weeks. Participants
said that it would be easier to estimate in weeks since they would need to go to HR or other
departments, and response time could be slow and require multiple reminder contacts. Four
participants were not sure how much time it would take and did not give an estimate initially.
Participants were asked the same time estimate question after they completed the two questions, and
responses ranged from 3–4 hours to 40 hours. Nearly all of the participants said that it would take less
time in the future to complete these questions; however, two smaller institutions said that it would take
the same amount of time.
Five of the participants said that it took approximately 10 hours, and three of those said that it would
take less time in the future. Four said that it took less than 10 hours, and 7 said that it took more than
10 hours (3 said 12–15 hours, 1 said 24 hours, and 3 said 30 hours or more). Based on this information,
we estimate that it could add 10–15 hours for most institutions to complete the new questions for the
first time but less time after the first year since many said that they would develop reports to help them
pull this information, and a lot of the time they spent was determining the approach to complete the
questions.

Recommendations
Following all interviews, ICF recommends changes to both questions.
•

Decide if only personnel paid from separately accounted for R&D (e.g., as in Question 15)
should be reported and revise questions or instructions on both proposed questions to clarify.
All institutions assumed that they should report a similar headcount as Question 15 on the HERD
survey. For many schools, that meant that they were including personnel working on internally
funded R&D projects, because they already had a process for tracking those expenditures.
Several others were not including headcounts for personnel working on internally funded R&D
and noted that this was because it was not stated in the question or did not occur to them. No
institution indicated that Test Question 1 was meant to be capturing all personnel doing any
R&D, or any personnel that would not be included in Question 15. The same assumptions were
made for Test Question 2; all schools reported that the same personnel would be included in
both questions. If NCSES desires headcounts for all R&D personnel, including those not paid
from “separately accounted for” R&D projects, which would almost exclusively be institutionally
financed research, this should be clearly stated in the question as done in other places on the
survey or in the instructions.

We recommend these additional changes to Test Question 1.
•

•

ICF

Add instruction that student headcounts can be included in multiple functions. Several
participants said that they wanted more guidance on where to include students despite making
determinations on where to include them in the current instructions and function descriptions.
We do not think we should prescribe where students should be included, but it would be
beneficial to state that they may be included in multiple functions if that is appropriate based on
student R&D roles.
Consider adding more examples or further explanation defining the difference between R&D
technician and support staff. It was difficult for some participants to tease out where some
staff, especially students, should be placed based on their roles. Some participants said that
more examples of technicians would be helpful. A few participants mentioned that the
15

•

•

additional information about the difference between a researcher and technician was helpful,
but two participants said it would be helpful to have something similar differentiating
technicians and support staff.
Determine whether data provided in Test Question 1 will be published in aggregate or by
institution and state this on the questionnaire. Some participants expressed concern or said
that they were uncertain about whether their institution would provide some demographic
information. In round 2, we asked participants in both interviews if their institution would be
comfortable if this data was published by institution. While most said that it would be fine in
aggregate, some participants were still uncertain and would need to clear it at a higher level.
Clarify in the definitions that personnel that would be considered indirect staff should not be
included with support staff. Although no one seemed to have included those staff when
completing the test questions, a few mentioned that personnel from the sponsored programs
office or research administration might be considered support staff. This change would be
particularly important if the question instructions were revised to clarify that all R&D personnel
should be reported.

We recommend these additional changes to Test Question 2.
•

•
•

Provide additional examples using payroll rather than hours for calculating FTEs. Many
participants used our example of using 40 hours week/52 weeks a year to create FTE
calculations based on effort reporting and labor hours. A few participants used a different
approach by basing their calculations on payroll accounts. Even a few participants who did not
use this approach said that examples with different approaches would be helpful.
Remove the “and equivalent staff” from the R&D technicians line. Only one participant
commented on this, and it was in a passing comment. But it may raise questions as to what
“equivalent staff” means and if it is different than what is described in Test Question 1.
Consider removing “(paid)” from the instruction, “FTE research personnel are calculated as the
total working (paid) effort spent on research during a specific period divided by the total
effort representing a full-time schedule within the same period.” Removing the word “paid”
from this instruction will it make more consistent with what is asked in Test Question 1 and the
rest of the survey. This is particularly the case if clarifications are made to include all R&D
personnel.

Participant Questions and Other Changes Made
Participant Questions
Several participants asked questions during the process that were not related to new question content.
These questions are presented below and to the extent possible should be addressed in future
communications about the addition of the new questions.
•
•

Why are the new questions being added to the HERD survey?
Will the information provided in the new questions be used for ranking purposes?

Changes to Interview Protocol Throughout
Several changes were made to the preliminary interview protocol between the preliminary interviews
and round 1. For more detail on the changes, see the summary report following the preliminary
interviews. The major changes were:
• Ask how the information requested in Test Question 1 would compare to current Question 15.
ICF

16

•
•
•
•

Consider rephrasing or reordering the question in the protocol that asks what “functions”
means in the context of the question.
Ask specifically about how citizenship and education information was collected.
Flip the order of the initial reaction question and the question about what information the
question is asking for.
Add a probe to ask participants if they noticed to use whatever hours equate to a full-time
employee in the example instructions.

Following round 1 interviews, changes were made to both the initial interview protocol and the debrief
interview protocol. The major changes are summarized below.
•

•

ICF

Initial interview protocol changes:
o Remove the question asking participants how Test Question 1 compares to what is
reported in Question 15. While we added this question in round 1, we stopped asking it
after a few interviews because it seemed to lead the participants to answer Test
Question 1 in a specific way. This line of questioning was determined to be better for
the debrief interviews.
o Add a question to gauge whether participants thought their institution would be
comfortable with demographic data published by institution. An institution brought up
the topic of personally identifying information, and NCSES and ICF deemed it important
to specifically ask this question.
Debrief interview protocol changes:
o Add more detailed probes about personnel included in headcounts and how they are
placed in each function. Probes were added to determine how personnel working in
R&D functions were identified; if counts include those working on R&D funded only by
internal sources; how it compares to data reported in current Question 15; and if (and
which) students are included.
o Add more detailed probes about personnel included and the calculations used to obtain
FTEs. Probes were added to specifically ask if the same personnel from Test Question 1
were included in Test Question 2; whether hours used were based on financial accounts,
effort reporting, or something else; calculations used for various personnel types (e.g.,
hourly versus salaried); and if (and which) students are included.

17

APPENDIX A: Preliminary Cognitive Interview Communications

HERD Preliminary Cognitive Interview Communications
Invitation
To: Primary contact
Cc: others normally included in survey contacts
From: NSF Higher Education R&D Survey
Re: Contribute to changes on NSF HERD Survey ([[inst_id]])
Dear [[firstname]] [[lastname]]:
Thank you, again, for completing the FY 2019 Higher Education Research and Development (HERD)
Survey. I appreciate your time and efforts.
I am writing today to request your assistance on a proposal for the next cycle of the survey. Before we
finalize the survey for FY 2020, we would like to get feedback from current survey respondents. The
National Center for Science and Engineering Statistics within the National Science Foundation is
considering adding two new questions about R&D personnel to the FY 2020 survey. The first would ask
for the number of personnel by demographic categories and would replace the current question 15. The
second would ask for full-time equivalents (FTEs) for personnel at your institution who perform R&D.
Would you be willing to participate in a brief phone interview? During this interview, we will share the
proposed questions with you and ask your feedback. The interview will be less than 30 minutes. We
would like to complete the interviews by March 20.
If you, or someone else from the team at [institution name] that completes the HERD survey, are
interested in participating, please contact Melinda Scott from our survey staff at ICF at
[email protected] or 866-936-9376. She will schedule a time for the interview.
Thank you for your help and for your continued participation in the NSF HERD Survey.
Sincerely,
Michael Gibbons
Project Officer, Higher Education R&D Survey
National Center for Science and Engineering Statistics
National Science Foundation

A-1

Confirmation of Interview
To: Primary contact email (or other person scheduled)
CC: other attendees, including relevant NCSES and ICF staff
From: NSF Higher Education R&D Survey
Re: Confirming interview ([[inst_id]])
Dear [[firstname]] [[lastname]]:
Thank you for agreeing to talk to us about two proposed questions that may be added to the FY 2020
Higher Education R&D Survey (HERD) related to head counts and full-time equivalents (FTEs) for R&D
personnel at your institution.
The phone call is scheduled for DayoftheWeek, Month XX, at X:XX AM/PM (EDT/CDT/MDT/WDT). At
that time please call our conference line at 1-855-423-6338, conference ID 450349. Sherri Mamon will
be leading the interview. I would be happy to send you a Microsoft Outlook invitation, if you prefer.
If you have further questions or need to reschedule, you may reach me by email at
[email protected] or call toll-free at (866) 936-9376.
Thank you very much for your help.
Sincerely,
Melinda Scott
Data Collection Specialist
Higher Education R&D Survey
ICF

A-2

Reminder of First Interview
This will be sent to institutions one business day before their scheduled interview, but only if they
received the first confirmation at least 4 days before the scheduled interview.

To: Primary contact email (or other person scheduled)
CC: other attendees, including relevant ICF staff
From: NSF Higher Education R&D Survey
Re: Interview Reminder ([[inst_id]])
Dear [[firstname]] [[lastname]]:
This is just a reminder about our phone call scheduled for DayoftheWeek, Month XX, at X:XX AM/PM
(EDT/CDT/MDT/WDT). At that time please call my conference line at 1-855-423-6338, conference ID
450349.
We will send you a copy of your completed FY 2019 survey with the new questions included prior to the
interview.
If you have any trouble accessing the attachment, have further questions or need to reschedule, you
may reach me by email at [email protected] or call toll-free at (866) 936-9376.
Thank you very much for your help.
Sincerely,
Sherri Mamon
Qualitative Researcher
Higher Education R&D Survey
ICF

A-3

APPENDIX B: Preliminary Cognitive Interview Test Questions

This is a proposed new question. None of the data collected during this study will be included in the
FY 2019 data maintained or published by NCSES.
Test Question 1. How many personnel (headcount) worked in the functions listed below in FY 2019 and how many
fell into the demographic and educational categories listed below?
•
•

Count each person only once.
Include all personnel and students paid from R&D accounts regardless of how much
they received.

Researchers

(b)
R&D
technicians

(c)
R&D support
staff

Total

_____

_____

_____

_____

1. Female

_____

_____

_____

_____

2. Male

_____

_____

_____

_____

3. Sex unknown or not stated

_____

_____

_____

_____

1. U.S. citizens and permanent residents (non-U.S.
citizens holding Green Cards)

_____

_____

_____

_____

2. Foreign nationals holding temporary visas

_____

_____

_____

_____

(a)

A. Total R&D personnel

(d)

B. Sex

C. Citizenship

D. Highest level of education completed
1. Doctorate (e.g., PhD, DSc, EdD)

_____

2. Professional degree (e.g., JD, LLB, MD, DDS, DVM)

_____

3. Master’s degree (e.g., MS, MA, MBA)

_____

4. Less than Master’s

_____

5. Education level unknown or not stated

_____

B-1

This is a proposed new question. None of the data collected during this study will be included in the
FY 2019 data maintained or published by NCSES.
Test Question 2.

How many full-time equivalents (FTEs) worked in the functions listed below in FY 2019?
•
•

An individual cannot be more than 1.0 FTE.
FTE research personnel are calculated as the total working (paid) effort spent on research
during a specific period divided by the total effort representing a full-time schedule within the
same period.

The following examples of FTE calculations assume a 40 hour work week and a 52 week year.
However, you should use the hours per week and weeks per year that typically represent a full-time
employee at your institution.
•
•
•
•
•

3 technicians that work on research full-time all year: (3 * 52)/52 = 3.0 FTE
2 support staff that work on research full-time for 32 weeks: (2 * 32)/52 =1.2 FTE
1 researcher that works on research 30% of the time for 20 weeks, 50% of the time for
another 20 weeks, and full-time for 12 weeks: ((20 * 0.3) + (20 * 0.5) + 12)/ 52 = .54 FTE
10 researchers that work on research 40% of the time for 20 weeks: (10 * (20 *0.4))/52 =
1.54 FTE
20 part-time employees that work as technicians for 10 hours a week for 40 weeks: (20 *
(10/40) * 40))/52 = 3.85 FTE

FTEs
a. Researchers (including R&D scientists, engineers, and their managers)

______

b. R&D technicians and equivalent staff

______

c. R&D support staff (clerical and other)

______

d. Total

______

B-2

APPENDIX C: Preliminary Cognitive Interview Protocol

HERD Preliminary Cognitive Interview Protocol
[Interviewer introduces herself.] Thank you very much for talking with me today. As you know from our
initial contact, we are considering adding some questions about head counts and full time equivalents
(FTEs) in the upcoming FY 2020 NSF Higher Education R&D Survey.
Before we begin our interview, I want to review some information with you.
• Your participation in this interview is voluntary.
• The interview should last no more than 30 minutes.
• There is another person on the telephone with me who is taking notes as well as Kathryn
Harper, the Project Director, listening in.
• If appropriate: , from NSF, is also listening to our discussion today. Is that okay? If
no, NSF staff member will hang up the phone.
• I would like to audio record this interview so that I get all of your comments. Is that okay with
you?
[Stop recording if consent if not given]
Just for the record, can you state your name, your title, and your role in the HERD survey?
Do you have any questions before we start?
We just sent you a copy of your completed FY 2019 survey. The proposed new questions are included in
that attachment. Did you receive the copy of the survey we sent?
[If they did not receive a copy of the revised survey, be prepared to e-mail it to them immediately.]
[Give them a few minutes to review before beginning interview]
(Head counts) Let’s start by going to page 29, and the proposed new question 1. I’ll give you a moment
to read through it. Please let me know when you’re finished.
1. What information is this question requesting?
2. What are your initial reactions to this question?
a. Can you tell me more about that?
3. This question refers to “functions” of personnel. What does “functions” mean to you, in the
context of this question?
4. What office(s) or department(s) at your institution would have this information?
5. Does your institution classify its own employees by function in the way the question asks?
a. What sort of employees would your institution consider to be “researchers”?
b. What about “R&D technicians”?
c. What about “R&D support staff?”
d. How would you classify student R&D personnel?
6. Are the instructions asking you to do something that you would normally do? Or are they asking
you to do something that seems unusual?

C-1

(FTEs) Now let’s look at the proposed new question 2, on page 30. I’ll give you a moment to read
through it. Please let me know when you’re finished.
7. What information is this question requesting?
8. What are your overall reactions to this question?
a. Can you tell me more about that?
9. This question refers to “full-time equivalents (FTEs).” Does your institution typically have this
information?
10. What office(s) or department(s) at your institution would have this information?
11. Are the instructions asking you to do something that you would normally do? Or are they asking
you to do something that seems unusual?
12. What are your reactions to the examples provided?
a. Are they too long, too short, or about right?
b. Are they helpful or not helpful?
13. Do you have any questions or comments about anything that we have not covered?
That’s all of our questions for this interview. Thank you so much for your time today. If we decide to put
the questions on the survey, we will send an email announcing the change as soon as we can.

C-2

APPENDIX D: Round 1 Initial Interview Communications

HERD Round 1 Initial Interview Communications
Invitation
To: Primary contact
Cc: others normally included in survey contacts
From: NSF Higher Education R&D Survey
Re: Contribute to changes on NSF-NCSES HERD Survey ([[inst_id]])
Dear [[firstname]] [[lastname]]:
Thank you, again, for completing the FY 2019 Higher Education Research and Development (HERD)
Survey. I appreciate your time and efforts.
I am writing today to request your assistance on a proposal for the next cycle of the survey. Before we
finalize the survey for FY 2020, we would like to get feedback from current survey respondents. The
National Center for Science and Engineering Statistics within the National Science Foundation is
considering adding two new questions about R&D personnel to the FY 2020 survey. The first would ask
for the number of personnel by demographic categories and would replace the current question 15. The
second would ask for full-time equivalents (FTEs) for personnel at your institution who perform R&D.
If you are willing to help, we would ask you to do three things:
1) Participate in an initial interview by phone to understand how you would go about
answering these questions. We would like to complete the interviews by [enter date]. The
interview will be less than 30 minutes.
2) Complete the proposed new questions. This step will likely involve working across offices
at your organization (e.g., Human Resources, Institutional Research). The responses you
provide will only be used for our internal question development. We would like you to
complete the test questions within four weeks of your initial interview.
3) Participate in a final interview by phone. The interview will be less than 30 minutes. We will
ask about your experience completing the proposed new questions. We would like your
final interview completed within one week of completing the test questions.
If you, or someone else from the team at [institution name] that completes the HERD survey, are
interested in participating, please contact Melinda Scott from our survey staff at ICF at
[email protected] or 866-936-9376. She will schedule a time for the interview. If we do not hear
from you by [date], we will contact you again.
Thank you for your help and for your continued participation in the NSF HERD Survey.
Sincerely,
Michael Gibbons
Project Officer, Higher Education R&D Survey
National Center for Science and Engineering Statistics
National Science Foundation
D-1

Invitation – Reminder
To: Primary contact
Cc: others normally included in survey contacts
From: NSF Higher Education R&D Survey
Re: Please reply: Contribute to changes on NSF-NCSES HERD Survey ([[inst_id]])
Dear [[firstname]] [[lastname]]:
Last week we sent you an e-mail asking for your assistance on some changes we are considering for the
Higher Education Research and Development (HERD) Survey. We want to be sure that the universities
understand the questions and can provide the newly requested data before we implement any changes.
This process is critical to the survey’s development. We are only inviting a few respondents to
participate. We would greatly appreciate your input on these new questions.
We are thinking about adding two questions about R&D personnel to the FY 2020 survey. The first
would ask for the number of personnel by demographic categories and would replace the current
question 15. The second would ask for full-time equivalents (FTEs) for personnel at your institution who
perform R&D.
Before we finalize the survey, we would like to get some feedback from current survey respondents. If
you are willing to help, we would ask you to do three things:
1) Participate in an initial interview by phone to understand how you would go about
answering these questions. We would like to complete the interviews by [enter date]. The
interview will be less than 30 minutes.
2) Complete the test questions. This step will likely involve working across offices at your
organization (e.g., Human Resources, Institutional Research). The responses you provide will
only be used for our internal question development. We would like you to complete the test
questions within three weeks of your initial interview.
3) Participate in a final interview. The interview will be less than 30 minutes. We will ask
about your experience completing the proposed new questions. We would like your final
interview completed within one week of completing the test questions.
If you, or someone else from the team at [institution name] that completes the HERD survey, are
interested in participating, please contact Melinda Scott from our survey staff at ICF at
[email protected] or 866-936-9376. She will schedule a time for the interview.
Thank you for your help and for your continued participation in the NSF HERD Survey.
Sincerely,
Michael Gibbons
Project Officer, Higher Education R&D Survey
National Center for Science and Engineering Statistics
National Science Foundation
D-2

Confirmation of First Interview
To: Primary contact email (or other person scheduled)
CC: other attendees, including relevant NCSES and ICF staff
From: NSF Higher Education R&D Survey
Re: Confirming interview ([[inst_id]])
Dear [[firstname]] [[lastname]]:
Thank you for agreeing to talk to us about two proposed new questions that may be added to the FY
2020 Higher Education R&D Survey (HERD) related to head counts and full-time equivalents (FTEs) for
R&D personnel at your institution.
The phone call is scheduled for DayoftheWeek, Month XX, at X:XX AM/PM (EDT/CDT/MDT/WDT). At
that time please call our conference line at 1-855-423-6338, conference ID [interviewer id]. [insert
interviewer name] will be leading the interview. You have the option of using video for this interview. If
you would like to use that option, please let us know.
Also, please note, following this interview, we ask you to complete the test questions. This step will
likely involve working across offices at your organization (e.g., Human Resources, Institutional Research).
We would like you to complete the test questions within four weeks of your initial interview [insert
date].
If you have further questions or need to reschedule, you may reach me by email at
[email protected] or call toll-free at (866) 936-9376.
Thank you very much for your help.
Sincerely,
Melinda Scott
Data Collection Specialist
Higher Education R&D Survey
ICF

D-3

Reminder of First Interview
This will be sent to institutions one business day before their scheduled interview, but only if they
received the first confirmation at least 4 days before the scheduled interview.

To: Primary contact email (or other person scheduled)
CC: other attendees, including relevant ICF staff
From: NSF Higher Education R&D Survey
Re: Interview Reminder ([[inst_id]])
Dear [[firstname]] [[lastname]]:
This is just a reminder about our phone call scheduled for DayoftheWeek, Month XX, at X:XX AM/PM
(EDT/CDT/MDT/WDT). At that time please call my conference line at 1-855-423-6338, conference ID
[interviewer id].
We will send you a copy of your completed FY 2019 survey with the new questions included prior to the
interview.
If you have any trouble accessing the attachment, have further questions or need to reschedule, you
may reach me by email at [email protected] or call toll-free at (866) 936-9376.
Thank you very much for your help.
Sincerely,
Sherri Mamon
Qualitative Researcher
Higher Education R&D Survey
ICF

D-4

APPENDIX E: Round 1 Test Questions

This is a proposed new question. None of the data collected during this study will be included in the
FY 2019 data maintained or published by NCSES.
Test Question 1. How many personnel (headcount) worked in the functions listed below in FY 2019 and how many
fell into the demographic and educational categories listed below?
•
•
•

Count each person only once.
Include all personnel and students paid from R&D accounts regardless of how much
they received.
Functions are defined primarily by the nature of the employee’s work, not the employee’s level of
education. See table below for more information.

Researchers

R&D technicians

R&D support staff

Professionals engaged in the
conception or creation of new
knowledge, products, processes,
methods and systems and also in the
management of the projects
concerned. Include R&D managers in
this category.

Persons whose main tasks require
technical knowledge and experience
in one or more fields of science or
engineering, but who contribute to
R&D by performing technical tasks
under the supervision of researchers.

Not directly involved with the conduct
of a research project, but support the
researchers and technicians. These
employees might include clerical staff,
report writers, safety trainers, and
other related employees.

Researcher versus R&D technician
Researchers contribute more to the creative aspects of R&D whereas technicians provide technical support. For
example, a researcher (scientist or engineer) would design an experiment and a technician would run the experiment
and assist in analyzing results.

Researchers

(b)
R&D
technicians

(c)
R&D support
staff

Total

_____

_____

_____

_____

_____
_____
_____

_____
_____
_____

_____
_____
_____

_____
_____
_____

_____
_____
_____

_____
_____
_____

_____
_____
_____

_____
_____
_____

(a)

A. Total R&D personnel

(d)

B. Sex
1. Female
2. Male
3. Sex unknown or not stated
C. Citizenship
1. U.S. citizens and permanent residents (non-U.S.
citizens holding Green Cards)
2. Foreign nationals holding temporary visas
3. Citizenship or residency status unknown or not stated
D. Highest level of education completed
1. Doctorate (e.g., PhD, DSc, EdD)
2. Professional degree (e.g., JD, LLB, MD, DDS, DVM)
3. Master’s degree (e.g., MS, MA, MBA)
4. Less than Master’s
5. Education level unknown or not stated

_____
_____
_____
_____
_____
E-1

This is a proposed new question. None of the data collected during this study will be included in the
FY 2019 data maintained or published by NCSES.
Test Question 2.

How many full-time equivalents (FTEs) worked in the functions listed below in FY 2019?
•
•
•

See Test Question 1 for descriptions of each function.
An individual cannot be more than 1.0 FTE.
FTE research personnel are calculated as the total working (paid) effort spent on research
during a specific period divided by the total effort representing a full-time schedule within the
same period.

The following examples of FTE calculations assume a 40 hour work week and a 52 week year.
However, you should use the hours per week and weeks per year that typically represent a full-time
employee at your institution.
•
•
•
•
•

3 technicians that work on research full-time all year: (3 * 52)/52 = 3.0 FTE
2 support staff that work on research full-time for 32 weeks: (2 * 32)/52 =1.2 FTE
1 researcher that works on research 30% of the time for 20 weeks, 50% of the time for
another 20 weeks, and full-time for 12 weeks: ((20 * 0.3) + (20 * 0.5) + 12)/ 52 = .54 FTE
10 researchers that work on research 40% of the time for 20 weeks: (10 * (20 *0.4))/52 =
1.54 FTE
20 part-time employees that work as technicians for 10 hours a week for 40 weeks: (20 *
(10/40) * 40))/52 = 3.85 FTE

FTEs
a. Researchers

______

b. R&D technicians and equivalent staff

______

c. R&D support staff

______

d. Total

______

E-2

APPENDIX F: Round 1 Initial Interview Protocol

HERD Round 1 Initial Interview Protocol
[Interviewer introduces herself.] Thank you very much for talking with me today. As you know from our
initial contact, we are considering adding some questions about head counts and full time equivalents
(FTEs) in the upcoming FY 2020 NSF-NCSES Higher Education R&D Survey.
Before we begin our interview, I want to review some information with you.
• Your participation in this interview is voluntary.
• The interview should last no more than 30 minutes.
• There is another person on the telephone with me who is taking notes.
• If appropriate: , from NSF-NCSES, is also listening to our discussion today. Is that
okay? If no, NCSES staff member will hang up the phone.
• I would like to audio record this interview so that I get all of your comments. Is that okay with
you?
• Before we continue with the interview, I need your consent to be interviewed. I’m going to read
you the consent form and then I’ll ask for your verbal consent.
The National Center for Science and Engineering Statistics (NCSES) within the National Science
Foundation designs, conducts, and sponsors surveys on science, engineering, and technology. In
order to produce the best information possible, NCSES routinely evaluates its surveys.
You have volunteered to take part in a study to evaluate one of the NCSES’s surveys. In order to have
a complete record of your comments, your interview session will be audio recorded. We plan to use
the recording to improve the survey. NCSES and ICF staff directly involved in this research project will
have access to the recording.
This study is authorized by law (42 U.S.C. 1862 Section 3.a.6). The OMB control number for this study
is 3145-0174.
I have volunteered to participate in this study, and I give permission for my recording to be used for
the purposes stated above.
If you consent to participate in this interview, please state your name and that you consent.
Do you have any questions before we start?
For the record can you tell me your role in the HERD Survey?
I just sent you a copy of your completed FY 2019 survey. The proposed new questions are included in
that attachment. Did you receive the copy of the survey I sent?
[If they did not receive a copy of the revised survey, be prepared to e-mail it to them immediately.]
[Give them a few minutes to review before beginning interview]
(Head counts) Let’s start by going to page 29, and the proposed new question 1. I’ll give you a moment
to read through the question, instructions, and definitions. Please let me know when you’re finished.
F-1

1. What are your initial reactions to this question?
a. Can you tell me more about that?
2. What information is this question requesting?
a. How does the information this question is requesting compare to what is currently
reported in Question 15?
i. Do you think the total headcount would be the same?
3. This question refers to “functions” of personnel. What does “functions” mean to you, in the
context of this question?
a. If you were just reading this question without being able to see the definitions for
column headers, what would you think “function” meant?
b. Knowing the information included on the question, what word would you use in the
question?
4. Is the information about what personnel to include in each function clear?
a. What sort of employees at your institution would be “researchers”?
b. What sort of employees at your institution would be “R&D technicians”?
c. What sort of employees at your institution would be “R&D support staff?”
d. Is the direction about functions above the table, “Functions are defined primarily by the
nature of the employee’s work, not level of education” clear?
5. Does your institution already classify its own employees by function similar to the way the
question asks?
a. If yes,
i. Tell me more about that.
ii. What sort of employees would your institution consider to be “researchers”?
iii. What sort of employees would your institution consider to be “R&D
technicians”?
iv. What about “R&D support staff?”
6. What office(s) or department(s) at your institution would have this information?
a. Do you think you will be able to obtain all of the demographic and educational
categories as requested?
i. Are there any that will be more difficult to obtain than others?
7. Are the instructions asking you to do something that you would normally do? Or are they asking
you to do something that seems unusual?
(FTEs) Now let’s look at the proposed new question 2, on page 30. I’ll give you a moment to read
through the question, instructions, and examples. Please let me know when you’re finished.
8. What are your initial reactions to this question?
a. Could you tell me more about that?
9. What information is this question requesting?
10. This question refers to “full-time equivalents (FTEs).” Does your institution typically have this
information?
11. What office(s) or department(s) at your institution would have this information?
12. What are your reactions to the examples provided?
a. Are they too long, too short, or about right?
b. Are they helpful or not helpful?
F-2

c. Did you notice the instruction to use the hours per week and weeks per year that
represent a full-time employee at your institution?
i. Does that help you determine how you will provide this information for your
institution?
13. Are the instructions asking you to do something that you would normally do? Or are they asking
you to do something that seems unusual?
Overall Thoughts for Proposed Questions
14. Do you think you would be able to answer these questions yourself, or would you have to
consult with others?
a. If others, who will you consult with? Is it different for each question?
b. If others, how will you go about getting information on this topic from that
person/those people? (Phone call? E-mail? Share the questionnaire or not?)
15. Approximately how long do you think it will take you, or others pulling this data together for
you, to answer these two questions, including the time it takes to gather information and talk to
others?
16. Do you have any questions or comments about anything that we have not covered?
Those are all of our questions for this interview today. The next step is for you to complete the two
proposed new questions we’ve just talked about. Just a reminder, the responses provided will not be
released publicly. They will only be used for research and evaluation purposes.
Once you’re finished with the new questions, we’d like to talk again. Will you be able to finish these
questions within 4 weeks or even sooner? Would you prefer to schedule that meeting now, or schedule
it later?
Thank you so much for your time today.

F-3

APPENDIX G: Round 1 Debrief Interview Communications

HERD Round 1 Debrief Interview Communications
Reminder to Complete Test Questions/Schedule Second Interview
To: Primary contact email (or other person scheduled)
CC: other attendees, including relevant NCSES and ICF staff
From: NSF Higher Education R&D Survey
Re: Confirming interview ([[inst_id]])
Dear [[firstname]] [[lastname]]:
Now that you have had time to work on completing the test questions for the HERD Survey, we’d like to
schedule your final interview. When you have completed the questions, please send them to us in
advance of our next discussion. As a reminder, we would like your completed questions returned by
[insert date].
We’d like to schedule your final 30 minute interview where we’ll ask about your experience completing
the survey questions. If you have availability during the days of [insert dates], please contact me at
[email protected] or 866-936-9376. I will schedule a time for the phone call.
Thank you very much for your help.
Sincerely,
Melinda Scott
Data Collection Specialist
Higher Education R&D Survey
ICF

G-1

Reminder of Second Interview
This will be sent to institutions one business day before their scheduled interview, but only if they
received the first confirmation at least 4 days before the scheduled interview.

To: Primary contact email (or other person scheduled)
CC: other attendees, including relevant ICF staff
From: NSF Higher Education R&D Survey
Re: Interview Reminder ([[inst_id]])
Dear [[firstname]] [[lastname]]:
This is just a reminder about our phone call scheduled for DayoftheWeek, Month XX, at X:XX AM/PM
(EDT/CDT/MDT/WDT). At that time please call my conference line at 1-855-423-6338, conference ID
[interviewer id].
Please be sure to send us your completed questions in advance of the call.
If you have any trouble accessing the attachment, have further questions or need to reschedule, you
may reach me by email at [email protected] or call toll-free at (866) 936-9376.
Thank you very much for your help.
Sincerely,
Sherri Mamon
Qualitative Researcher
Higher Education R&D Survey
ICF

G-2

APPENDIX H: Round 1 Debrief Interview Protocol

HERD Round 1 Debrief Interview Protocol
[Interviewer introduces herself.] Thank you very much for talking with me today and for taking the time
to complete the proposed questions for the upcoming FY 2020 NSF-NCSES Higher Education R&D
Survey.
Before we begin our interview, I want to remind you of some things.
• Your participation in this interview is voluntary.
• The interview should last no more than 30 minutes.
• There is another person on the telephone with me who is taking notes.
• If appropriate: , from NSF-NCSES, is also listening to our discussion today. Is that
okay? If no, NCSES staff member will hang up the phone.
• I would like to audio record this interview so that I get all of your comments. Is that okay with
you?
• Before we continue with the interview, I need your consent to be interviewed. I’m going to read
you the consent form and then I’ll ask for your verbal consent.
The National Center for Science and Engineering Statistics within then National Science Foundation
designs, conducts, and sponsors surveys on science, engineering, and technology. In order to
produce the best information possible, NCSES routinely evaluates its surveys.
You have volunteered to take part in a study to improve one of NCSES’s surveys. In order to have a
complete record of your comments, your interview session will be audio recorded. We plan to use
the recording to improve the survey. NCSES and ICF staff directly involved in this research project will
have access to the recording.
This study is authorized by law (42 U.S.C. 1862 Section 3.a.6). The OMB control number for this study
is 3145-0174.
I have volunteered to participate in this study, and I give permission for my recording to be used for
the purposes stated above.
If you consent to participate in this interview, please state your name and that you consent.
Do you have any questions before we start?
If you don’t have it in front of you already, please pull up your responses to the proposed new
questions.
(Head counts) Let’s start with your overall reactions to proposed new question 1.
1. Can you tell me how you went about gathering this information?
a. Were you able to do it yourself, or did you get help from other people?
i. If you got help from others, were those people in your own office or
department, or in other offices/departments? What other offices or
departments were they in? Did they have any feedback on the question?
H-1

2.

3.
4.

5.

6.

7.

ii. How did you go about determining which staff were included in each function?
iii. [If it is not mentioned, specifically ask about each demographic type to see if it
was gathered through different offices or in different ways]
b. What records did you use? Where are those records kept? (In what offices or
departments are they kept? In what system(s) are they kept?)
Did you have to make any assumptions about what the question was asking? If so, what were
they?
a. How does the information this question is requesting compare to what is currently
reported in Question 15?
i. Do you think the total headcount would be the same?
ii. If you made the assumption that the total for this question should be the same
as Question 15, then how would it be different if you had not made that
assumption?
Note to interviewer: if respondent asks specifically if it should match Q15,
respond with something like, “right now we’re seeing how different institutions
are interpreting this question. If they are the same as Q15, we want to know
what institutions can do with the additional breakouts. This will help us further
refine instructions before it is finalized.”
b. Some schools have told us that their system does not capture co-PIs, so on Question 15
co-PIs are included in all other personnel. Where are co-PIs at your institution reported
on Test Question 1?
c. Some schools have also told us that employees can only be categorized as PIs on
externally sponsored projects, so faculty whose R&D salary only comes from
institutional funds would be included in all other personnel on Question 15. Where are
faculty performing R&D reported on Test Question 1?
Did you or anyone that you consulted with have questions while working on this question?
a. If yes, what were your (or their) questions? Were you able to answer them?
How easy or difficult was it to gather this information?
a. If difficult, tell me more about that.
b. Were any of these things easier or harder to gather than others?
[If some of the survey question was not answered] I noticed that [line/column X] was blank. Is
there any particular reason for that?
a. Do you believe you will be able to provide those data points for the FY 2020 survey, or
would they also be blank?
i. If no, why not and do you think the information could be provided for a survey
beyond FY 2020?
Did you refer to the instructions as you were completing the question, or not?
a. If referred, what parts of the instructions were helpful? Which parts were not helpful?
Were there any parts of the instructions that you thought were confusing, or could be
confusing to another person?
Did you or any of your colleagues have concerns about reporting this data?

(FTEs) Now let’s look at proposed new question 2.
8. Can you tell me how you went about gathering this information?
H-2

a. Were you able to do it yourself, or did you get help from other people?
i. If you got help from others, were those people in your own office or
department, or in other offices/departments? What other offices or
departments were they in? Did they have any feedback on the question?
ii. Note, we want to know the hours were R&D if not specified.
b. Was anyone counted under both functions? For example, could someone be 0.2 FTE
under researcher and 0.2 under R&D technician and equivalent staff?
c. Do the numbers you reported represent estimates, or figures from records? If they were
estimates, how did you estimate?
d. What records did you use? Where are those records kept? (In what offices or
departments are they kept? In what system(s) are they kept?)
e. We noticed you reported in whole numbers, which we were vague about in our
instructions. Would specifying if we wanted decimal places or not make it easier to
complete or would it matter?
9. Did you have to make any assumptions about what the question was asking? If so, what were
they?
a. Did you see the instruction to use the hours per week and weeks per year that typically
represent a full-time employee at your institution? Did that influence how you
calculated the FTEs?
b. What calculation did you use to get to these numbers? (e.g., Did you base this on a 40
hour work week and 52 week year or something else?)
c. Was it different for full-time vs. part-time?
10. How easy or difficult was it to gather this information?
a. If difficult, tell me more about that?
b. Were any of these things easier or harder to gather than others?
11. Did you or anyone that you consulted with have questions while working on this question?
a. If yes, what were your (or their) questions? Were you able to answer them?
12. [If some of the survey question was not answered] I noticed that [line/column X] was blank. Is
there any particular reason for that?
a. Do you believe you will be able to provide those data points for the FY 2020 survey, or
would they also be blank?
b. If no, why not and do you think the information could be provided for a survey beyond
FY 2020?
13. Did you happen to refer to the instructions as you were completing the question, or not?
a. If referred, what parts of the instructions were helpful? Which parts were not helpful?
Were there any parts of the instructions that you thought were confusing, or could be
confusing to another person?
b. What did you think of the instruction that “an individual cannot be more than 1.0 FTE?”
14. Did you happen to refer to the examples for calculating the FTEs as you completed the
question?
a. Were they helpful or not helpful?
b. What did you think of the level of detail in the examples: was it too much, too little, or
about right?
Overall Thoughts for Proposed Questions
H-3

15. Approximately how many hours did it take you, and others who contributed, to answer these
two questions, including the time it takes to gather information, talk to anybody you’d need to
talk to, etc.?
a. Do you think it would take less time now that you’ve answered it the first time?
16. Do you have any questions or comments about anything that we have not covered?
Those are all our questions. Thank you so much for your time today. If we decide to put the question on
the survey, we will send an email announcing the change as soon as we can.

H-4

APPENDIX I: Round 2 Initial Interview Communications

Round 2 Initial Interview Communications
Invitation
To: Primary contact
Cc: others normally included in survey contacts
From: NSF Higher Education R&D Survey
Re: Contribute to changes on NSF-NCSES HERD Survey ([[inst_id]])
Dear [[firstname]] [[lastname]]:
Thank you, again, for completing the FY 2019 Higher Education Research and Development (HERD)
Survey. I appreciate your time and efforts.
I am writing today to request your assistance on a proposal for the next cycle of the survey. Before we
finalize the survey for FY 2020, we would like to get feedback from current survey respondents. The
National Center for Science and Engineering Statistics within the National Science Foundation is
considering adding two new questions about R&D personnel to the FY 2020 survey. The first would ask
for the number of personnel by demographic categories and would replace the current question 15. The
second would ask for full-time equivalents (FTEs) for personnel at your institution who perform R&D.
If you are willing to help, we would ask you to do three things:
1) Participate in an initial interview by phone to understand how you would go about
answering these questions. We would like to complete the interviews by June 10. The
interview will be less than 30 minutes.
2) Complete the proposed new questions. This step will likely involve working across offices
at your organization (e.g., Human Resources, Institutional Research). The responses you
provide will only be used for our internal question development. We would like you to
complete the test questions within three weeks of your initial interview.
3) Participate in a final interview by phone. The interview will be less than 30 minutes. We will
ask about your experience completing the proposed new questions. We would like your
final interview completed within one week of completing the test questions.
If you, or someone else from the team at [institution name] that completes the HERD survey, are
interested in participating, please contact Melinda Scott from our survey staff at ICF at
[email protected] or 866-936-9376. She will schedule a time for the interview. If we do not hear
from you by June 4, we will contact you again.
Thank you for your help and for your continued participation in the NSF HERD Survey.
Sincerely,
Michael Gibbons
Project Officer, Higher Education R&D Survey
National Center for Science and Engineering Statistics
National Science Foundation
I-1

Invitation – Reminder
To: Primary contact
Cc: others normally included in survey contacts
From: NSF Higher Education R&D Survey
Re: Please reply: Contribute to changes on NSF-NCSES HERD Survey ([[inst_id]])
Dear [[firstname]] [[lastname]]:
Earlier this week we sent you an e-mail asking for your assistance on some changes we are considering
for the Higher Education Research and Development (HERD) Survey. We want to be sure that the
universities understand the questions and can provide the newly requested data before we implement
any changes. This process is critical to the survey’s development. We are only inviting a few respondents
to participate. We would greatly appreciate your input on these new questions.
We are thinking about adding two questions about R&D personnel to the FY 2020 survey. The first
would ask for the number of personnel by demographic categories and would replace the current
question 15. The second would ask for full-time equivalents (FTEs) for personnel at your institution who
perform R&D.
Before we finalize the survey, we would like to get some feedback from current survey respondents. If
you are willing to help, we would ask you to do three things:
1) Participate in an initial interview by phone to understand how you would go about
answering these questions. We would like to complete the interviews by June 10. The
interview will be less than 30 minutes.
2) Complete the test questions. This step will likely involve working across offices at your
organization (e.g., Human Resources, Institutional Research). The responses you provide will
only be used for our internal question development. We would like you to complete the test
questions within three weeks of your initial interview.
3) Participate in a final interview. The interview will be less than 30 minutes. We will ask
about your experience completing the proposed new questions. We would like your final
interview completed within one week of completing the test questions.
If you, or someone else from the team at [institution name] that completes the HERD survey, are
interested in participating, please contact Melinda Scott from our survey staff at ICF at
[email protected] or 866-936-9376. She will schedule a time for the interview.
Thank you for your help and for your continued participation in the NSF HERD Survey.
Sincerely,
Michael Gibbons
Project Officer, Higher Education R&D Survey
National Center for Science and Engineering Statistics
National Science Foundation
I-2

Confirmation of First Interview
To: Primary contact email (or other person scheduled)
CC: other attendees, including relevant NCSES and ICF staff
From: NSF Higher Education R&D Survey
Re: Confirming interview ([[inst_id]])
Dear [[firstname]] [[lastname]]:
Thank you for agreeing to talk to us about two proposed new questions that may be added to the FY
2020 Higher Education R&D Survey (HERD) related to head counts and full-time equivalents (FTEs) for
R&D personnel at your institution.
The phone call is scheduled for DayoftheWeek, Month XX, at X:XX AM/PM (EDT/CDT/MDT/WDT). At
that time please call our conference line at 1-855-423-6338, conference ID [interviewer id]. [insert
interviewer name] will be leading the interview. You have the option of using video for this interview. If
you would like to use that option, please let us know.
Also, please note, following this interview, we ask you to complete the test questions. This step will
likely involve working across offices at your organization (e.g., Human Resources, Institutional Research).
We would like you to complete the test questions within three weeks of your initial interview [insert
date].
If you have further questions or need to reschedule, you may reach me by email at
[email protected] or call toll-free at (866) 936-9376.
Thank you very much for your help.
Sincerely,
Melinda Scott
Data Collection Specialist
Higher Education R&D Survey
ICF

I-3

Reminder of First Interview
This will be sent to institutions one business day before their scheduled interview, but only if they
received the first confirmation at least 4 days before the scheduled interview.

To: Primary contact email (or other person scheduled)
CC: other attendees, including relevant ICF staff
From: NSF Higher Education R&D Survey
Re: Interview Reminder ([[inst_id]])
Dear [[firstname]] [[lastname]]:
This is just a reminder about our phone call scheduled for DayoftheWeek, Month XX, at X:XX AM/PM
(EDT/CDT/MDT/WDT). At that time please call my conference line at 1-855-423-6338, conference ID
[interviewer id].
We will send you a copy of your completed FY 2019 survey with the new questions included prior to the
interview.
If you have any trouble accessing the attachment, have further questions or need to reschedule, you
may reach me by email at [email protected] or call toll-free at (866) 936-9376.
Thank you very much for your help.
Sincerely,
Sherri Mamon
Qualitative Researcher
Higher Education R&D Survey
ICF

I-4

APPENDIX J: Round 2 Test Questions

This is a proposed new question. None of the data collected during this study will be included in the
FY 2019 data maintained or published by NCSES.
Test Question 1. How many personnel (headcount) worked in the functions listed below in FY 2019 and how many
fell into the demographic and educational categories listed below?




Count each person only once.
Include all personnel and students that worked in an R&D function regardless of the amount of time
spent in that role.
Functions are defined primarily by the nature of the employee’s work, not the employee’s level of
education. See table below for more information.

Researchers
Professionals engaged in the
conception or creation of new
knowledge, products, processes,
methods and systems and also in the
management of the projects
concerned. Include R&D managers in
this category.

R&D technicians
Persons whose main tasks require
technical knowledge and experience
in one or more fields of science or
engineering, but who contribute to
R&D by performing technical tasks
under the supervision of researchers.

R&D support staff
Not directly involved with the conduct
of a research project, but support the
researchers and technicians. These
employees might include clerical staff,
report writers, safety trainers, and
other related employees.

Researcher versus R&D technician
Researchers contribute more to the creative aspects of R&D whereas technicians provide technical support. For
example, a researcher (scientist or engineer) would design an experiment and a technician would run the experiment
and assist in analyzing results.

Researchers

(b)
R&D
technicians

(c)
R&D support
staff

Total

_____

_____

_____

_____

_____
_____
_____

_____
_____
_____

_____
_____
_____

_____
_____
_____

_____
_____
_____

_____
_____
_____

_____
_____
_____

_____
_____
_____

(a)

A. Total R&D personnel

(d)

B. Sex
1. Female
2. Male
3. Sex unknown or not stated
C. Citizenship
1. U.S. citizens and permanent residents (non-U.S.
citizens holding Green Cards)
2. Foreign nationals holding temporary visas
3. Citizenship or residency status unknown or not stated
D. Highest level of education completed
1. Doctorate (e.g., PhD, DSc, EdD)
2. Professional degree (e.g., JD, LLB, MD, DDS, DVM)
3. Master’s degree (e.g., MS, MA, MBA)
4. Less than Master’s
5. Education level unknown or not stated

Researchers
only

_____
_____
_____
_____
_____

Do not include
highest level of education
for R&D technicians or
R&D support staff.

J-1

This is a proposed new question. None of the data collected during this study will be included in the
FY 2019 data maintained or published by NCSES.
Test Question 2.

How many full-time equivalents (FTEs) worked in the functions listed below in FY 2019?




See Test Question 1 for descriptions of each function.
An individual cannot be more than 1.0 FTE.
FTE research personnel are calculated as the total working (paid) effort spent on research
during a specific period divided by the total effort representing a full-time schedule within the
same period.

The following examples of FTE calculations assume a 40 hour work week and a 52 week year.
However, you should use the hours per week and weeks per year that typically represent a full-time
employee at your institution.






3 technicians that work on research full-time all year: (3 * 52)/52 = 3.0 FTE
2 support staff that work on research full-time for 32 weeks: (2 * 32)/52 =1.2 FTE
1 researcher that works on research 20% of the time for 20 weeks, 50% of the time for
another 20 weeks, and full-time for 12 weeks: ((20 * 0.2) + (20 * 0.5) + 12)/ 52 = 0.5 FTE
10 researchers that work on research 25% of the time for 20 weeks: (10 * (20 *0.25))/52 =
1.0 FTE
20 part-time employees that work as technicians for 15 hours a week for 40 weeks: (20 *
(15/40) * 40))/52 = 5.8 FTE

FTEs
(round to 1 decimal place)

a. Researchers

____.__

b. R&D technicians and equivalent staff

____.__

c. R&D support staff

____.__

d. Total

____.__

J-2

APPENDIX K: Round 2 Initial Interview Protocol

HERD Round 2 Initial Interview Protocol
[Interviewer introduces herself.] Thank you very much for talking with me today. As you know from our
initial contact, we are considering adding some questions about head counts and full time equivalents
(FTEs) in the upcoming FY 2020 NSF-NCSES Higher Education R&D Survey.
Before we begin our interview, I want to review some information with you.
• Your participation in this interview is voluntary.
• The interview should last no more than 30 minutes.
• There is another person on the telephone with me who is taking notes.
• If appropriate: , from NSF-NCSES, is also listening to our discussion today. Is that
okay? If no, NCSES staff member will hang up the phone.
• I would like to audio record this interview so that I get all of your comments. Is that okay with
you?
• Before we continue with the interview, I need your consent to be interviewed. I’m going to read
you the consent form and then I’ll ask for your verbal consent.
The National Center for Science and Engineering Statistics (NCSES) within the National Science
Foundation designs, conducts, and sponsors surveys on science, engineering, and technology. In
order to produce the best information possible, NCSES routinely evaluates its surveys.
You have volunteered to take part in a study to evaluate one of the NCSES’s surveys. In order to have
a complete record of your comments, your interview session will be audio recorded. We plan to use
the recording to improve the survey. NCSES and ICF staff directly involved in this research project will
have access to the recording.
This study is authorized by law (42 U.S.C. 1862 Section 3.a.6). The OMB control number for this study
is 3145-0174.
I have volunteered to participate in this study, and I give permission for my recording to be used for
the purposes stated above.
If you consent to participate in this interview, please state your name and that you consent.
Do you have any questions before we start?
For the record can you tell me your role in the HERD Survey?
I just sent you a copy of your completed FY 2019 survey. The proposed new questions are included in
that attachment. Did you receive the copy of the survey I sent?
[If they did not receive a copy of the revised survey, be prepared to e-mail it to them immediately.]
[Give them a few minutes to review before beginning interview]
(Head counts) Let’s start by going to page 29, and the proposed new question 1. I’ll give you a moment
to read through the question, instructions, and definitions. Please let me know when you’re finished.
K-1

1. What are your initial reactions to this question?
a. Can you tell me more about that?
2. Looking at the bulleted list of instructions below the question, are the instructions clear?
a. Do you think you will be able to include headcounts for faculty, staff and students?
b. Is the direction about functions above the table, “Functions are defined primarily by the
nature of the employee’s work, not level of education” clear?
3. Is the information about what individuals to include in each function clear?
a. Does your institution already classify its own employees by function similar to the way
the question asks?
b. What sort of employees at your institution would be “researchers”?
c. What sort of employees at your institution would be “R&D technicians”?
d. What sort of employees at your institution would be “R&D support staff?”
4. What office(s) or department(s) at your institution would have this information?
a. Do you think you will be able to obtain all of the demographic and educational
categories as requested?
i. Are there any that will be more difficult to obtain than others?
b. If this question is added to the HERD survey, would your institution be comfortable with
us publishing demographic data by institution?
5. Are the instructions asking you to do something that you would normally do? Or are they asking
you to do something that seems unusual?
(FTEs) Now let’s look at the proposed new question 2, on page 30. I’ll give you a moment to read
through the question, instructions, and examples. Please let me know when you’re finished.
6. What are your initial reactions to this question?
a. Could you tell me more about that?
7. This question refers to “full-time equivalents (FTEs).” Does your institution typically have this
information?
8. How does your institution define a full-time employee?
a. What criteria does your institution use when considering whether an employee is fulltime or part-time?
9. What are your reactions to the examples provided?
a. Are they too long, too short, or about right?
b. How likely are you to refer to them as you’re answering this question: very likely,
somewhat likely, not at all likely?
c. Are they helpful or not helpful?
10. How do you think you will go about getting the data you need to answer this question?
a. Will all of your institution’s research related employees be reported here, or only some
of them? Which ones will be included? Which will be excluded?
b. What office(s) or department(s) at your institution would have this information?
11. Are the instructions asking you to do something that you would normally do? Or are they asking
you to do something that seems unusual?
Overall Thoughts for Proposed Questions

K-2

12. Approximately how long do you think it will take you, or others pulling this data together for
you, to answer these two questions, including the time it takes to gather information and talk to
others?
13. Do you have any questions or comments about anything that we have not covered?
Those are all of our questions for this interview today. The next step is for you to complete the two
proposed new questions we’ve just talked about. Just a reminder, the responses provided will not be
released publicly. They will only be used for research and evaluation purposes.
Once you’re finished with the new questions, we’d like to talk again. Will you be able to finish these
questions [by the date we provided in an e-mail]? We’ll send a reminder e-mail as we get closer and
schedule a second interview at that time.
If someone from another office, such as HR or payroll, helps you complete the test questions, we
encourage you to invite them to the next interview. Sometimes they can provide some additional
valuable information about how the data was collected.
Thank you so much for your time today.

K-3

APPENDIX L: Round 2 Debrief Interview Communications

HERD Round 2 Debrief Interview Communications
Reminder to Complete Test Questions/Schedule Second Interview
This reminder will be sent 2 weeks after their interview/1 week before their completed questions are
due.
To: Primary contact email (or other person scheduled)
CC: other attendees, including relevant NCSES and ICF staff
From: NSF Higher Education R&D Survey
Re: Confirming interview ([[inst_id]])
Dear [[firstname]] [[lastname]]:
Now that you have had time to work on completing the test questions for the HERD Survey, we’d like to
schedule your final interview. When you have completed the questions, please send them to us in
advance of our next discussion. As a reminder, we would like your completed questions returned by
[insert date].
We’d also like to schedule your final 30 minute interview where we’ll ask about your experience
completing the survey questions. We encourage you to invite other people who contributed to
gathering this information to this interview. We would like to schedule your interview sometime
between [insert dates]. Please contact me at [email protected] or 866-936-9376 to schedule a
time for the phone call.
Thank you very much for your help.
Sincerely,
Melinda Scott
Data Collection Specialist
Higher Education R&D Survey
ICF

L-1

Reminder of Second Interview
This will be sent to institutions one business day before their scheduled interview.

To: Primary contact email (or other person scheduled)
CC: other attendees, including relevant ICF staff
From: NSF Higher Education R&D Survey
Re: Interview Reminder ([[inst_id]])
Dear [[firstname]] [[lastname]]:
This is just a reminder about our phone call scheduled for DayoftheWeek, Month XX, at X:XX AM/PM
(EDT/CDT/MDT/WDT). At that time please call my conference line at 1-855-423-6338, conference ID
[interviewer id]. You have the option of using video for this interview. If you would like to use that
option, please let us know. We encourage you to invite other people who contributed to gathering this
information to this interview.
Please be sure to send us your completed questions in advance of the call.
If you have any trouble accessing the attachment, have further questions or need to reschedule, you
may reach me by email at [email protected] or call toll-free at (866) 936-9376.
Thank you very much for your help.
Sincerely,
Sherri Mamon
Qualitative Researcher
Higher Education R&D Survey
ICF

L-2

APPENDIX M: Round 2 Debrief Interview Protocol

HERD Round 2 Debrief Interview Protocol
[Interviewer introduces herself.] Thank you very much for talking with me today and for taking the time
to complete the proposed questions for the upcoming FY 2020 NSF-NCSES Higher Education R&D
Survey.
Before we begin our interview, I want to remind you of some things.
• Your participation in this interview is voluntary.
• The interview should last no more than 30 minutes.
• There is another person on the telephone with me who is taking notes.
• If appropriate: , from NSF-NCSES, is also listening to our discussion today. Is that
okay? If no, NCSES staff member will hang up the phone.
• I would like to audio record this interview so that I get all of your comments. Is that okay with
you?
• Before we continue with the interview, I need your consent to be interviewed. I’m going to read
you the consent form and then I’ll ask for your verbal consent.
The National Center for Science and Engineering Statistics within then National Science Foundation
designs, conducts, and sponsors surveys on science, engineering, and technology. In order to
produce the best information possible, NCSES routinely evaluates its surveys.
You have volunteered to take part in a study to improve one of NCSES’s surveys. In order to have a
complete record of your comments, your interview session will be audio recorded. We plan to use
the recording to improve the survey. NCSES and ICF staff directly involved in this research project will
have access to the recording.
This study is authorized by law (42 U.S.C. 1862 Section 3.a.6). The OMB control number for this study
is 3145-0174.
I have volunteered to participate in this study, and I give permission for my recording to be used for
the purposes stated above.
If you consent to participate in this interview, please state your name and that you consent.
Do you have any questions before we start?
If you don’t have it in front of you already, please pull up your responses to the proposed new
questions.
(Head counts) Let’s start with your overall reactions to proposed new question 1.
1. Can you tell me how you went about gathering this information?
2. Did you have to make any assumptions about what the question was asking? If so, what were
they?
3. Were you able to do it yourself, or did you get help from other people?

M-1

4.

5.
6.

7.

8.

a. If you got help from others, were those people in your own office or department, or in
other offices/departments? What other offices or departments were they in? Did they
have any feedback on the question?
How did you identify faculty, staff or students that worked in an R&D function?
a. What records did you use? Where are those records kept? (In what offices or
departments are they kept? In what system(s) are they kept?)
[If it is not mentioned, specifically ask about each demographic type to see if it was
gathered through different offices or in different ways]
b. Does this headcount include faculty, staff, and students that worked on R&D funded
only by internal sources?
c. How does the information this question is requesting compare to what is currently
reported in Question 15?
i. Did you think the total headcount would be the same?
Note to interviewer: if respondent asks specifically if it should match Q15,
respond with something like, “right now we’re seeing how different institutions
are interpreting this question. If they are the same as Q15, we want to know
what institutions can do with the additional breakouts. This will help us further
refine instructions before it is finalized.”
d. If they suggest that PI status was used to determine is someone was a researcher,
i. Some schools have told us that their system does not capture co-PIs, so on
Question 15 co-PIs are included in all other personnel. Where are co-PIs at your
institution reported on Test Question 1?
ii. Some schools have also told us that employees can only be categorized as PIs
on externally sponsored projects, so faculty whose R&D salary only comes from
institutional funds would be included in all other personnel on Question 15.
Where are faculty performing R&D reported on Test Question 1?
e. Under which function did you place students?
i.
Did your institution include doctoral, masters and undergraduate students
working in R&D functions?
ii.
Were they all placed in one function or split across functions? How?
a. Were students categorized based on level (e.g., graduate versus
undergraduate), type of work performed, or some other criteria?
Did you or anyone that you consulted with have questions while working on this question?
a. If yes, what were your (or their) questions? Were you able to answer them?
In our initial interview you said your institution [would/would not] be comfortable with us
publishing demographic data by institution. Now that you’ve gathered this data and talked to
others, does this change your initial thoughts on publishing this data?
How easy or difficult was it to gather this information?
a. If difficult, tell me more about that.
b. Were any of these things easier or harder to gather than others?
[If some of the survey question was not answered] I noticed that [line/column X] was blank. Is
there any particular reason for that?
a. Do you believe you will be able to provide those data points for the FY 2020 survey, or
would they also be blank?
M-2

i. If no, why not and do you think the information could be provided for a survey
beyond FY 2020?
9. Did you refer to the instructions as you were completing the question, or not?
a. If referred, what parts of the instructions were helpful? Which parts were not helpful?
Were there any parts of the instructions that you thought were confusing, or could be
confusing to another person?
(FTEs) Now let’s look at proposed new question 2.
10. Can you tell me how you went about gathering this information?
a. Were you able to do it yourself, or did you get help from other people?
i. If you got help from others, were those people in your own office or
department, or in other offices/departments? What other offices or
departments were they in? Did they have any feedback on the question?
b. What records did you others use to complete this question? Where are those records
kept? (In what offices or departments are they kept? In what system(s) are they kept?)
i. Were the hours used to calculate FTE based on financial accounts, effort
reporting or another source?
c. Were the FTEs reported in this question the same individuals included in the headcount
in the previous question? If not, where were there differences?
d. Was anyone counted under multiple functions? For example, could someone be 0.2 FTE
under researcher and 0.2 under R&D technician and equivalent staff?
11. Did you have to make any assumptions about what the question was asking? If so, what were
they?
a. Did you see the instruction to use the hours per week and weeks per year that typically
represent a full-time employee at your institution? Did that influence how you
calculated the FTEs?
b. What calculation did you use to get to these numbers? (e.g., Did you base this on a 40
hour work week and 52 week year or something else?)
i. Was it different for full-time vs. part-time?
ii. Was it different for hourly vs. salaried employees?
iii. Was it different for students vs faculty and staff?
12. How easy or difficult was it to gather this information?
a. If difficult, tell me more about that?
b. Were any of these things easier or harder to gather than others?
13. Did you or anyone that you consulted with have questions while working on this question?
a. If yes, what were your (or their) questions? Were you able to answer them?
14. [If some of the survey question was not answered] I noticed that [line/column X] was blank. Is
there any particular reason for that?
a. Do you believe you will be able to provide those data points for the FY 2020 survey, or
would they also be blank?
b. If no, why not and do you think the information could be provided for a survey beyond
FY 2020?
15. Did you happen to refer to the instructions as you were completing the question, or not?
a. If referred, what parts of the instructions were helpful? Which parts were not helpful?
Were there any parts of the instructions that you thought were confusing?
M-3

16. Did you happen to refer to the examples for calculating the FTEs as you completed the
question?
a. Were they helpful or not helpful?
Overall Thoughts for Proposed Questions
17. Approximately how many hours did it take you, and others who contributed, to answer these
two questions, including the time it takes to gather information, talk to anybody you’d need to
talk to, etc.?
a. Do you think it would take less time now that you’ve answered it the first time?
18. If you worked with others to gather this information, how did you go about getting information
on the questions to that person/those people? (Phone call? E-mail? Share the question text?)
19. Do you have any questions or comments about anything that we have not covered?
Those are all our questions. Thank you so much for your time today. If we decide to put the question on
the survey, we will send an email announcing the change as soon as we can.

M-4