HERD 2020-2021 OMB SS Section B

No changes to universe or collection procedures are proposed for FY 2020-21 the HERD or FFRDC R&D surveys. Imputation or weighting methods for the proposed new questions described in supporting statement A will be developed after the collection of FY 2020 data.

Universe and Sample Description

The HERD and FFRDC R&D surveys are censuses of all eligible institutions (see attachments 2, 3 and 7). No sampling will be performed for either survey. The population and response rates for both surveys for FY 2017-2019, as well as the estimated population and response rates for FY 2020-21 are included in Table B.1.1 below.

The universe for the FFRDC R&D survey consists of all FFRDCs in operation during the fiscal year of measurement. The FFRDC population has remained steady at 42 for several years and is expected to remain at 42 for FY 2020-21. The total survey universe is identified through the NSF Master Government List of FFRDCs.¹ NSF is responsible for maintaining this list and queries all federal agencies annually to determine any changes to, additions to, or deletions from the list.

The HERD survey universe for FY 2020-21 will be the same as it has been since FY 2011. The population will include all public or private nonprofit bachelor’s and higher degree-granting U.S. institutions that spent at least $150,000 in separately accounted for R&D in their prior fiscal year. The population for FY 2020-21 is estimated to be 925, allowing for a modest increase from recent collection cycles.

Most institutions will be identified as part of the HERD population based on their HERD data from the previous survey cycle. Additions to the population will be identified through a screening of (1) institutions granting a bachelor’s degree or higher and reporting an amount greater than $0 for research on the most recently available Integrated Postsecondary Education Data System (IPEDS) Finance Survey, (2) all U.S. service institutions granting a bachelor’s degree or higher, not reporting to IPEDS, and not already being part of the HERD population, and (3) institutions granting graduate degrees, not reporting to IPEDS, and not already being part of the HERD population. All institutions reporting at least $150,000 but less than $1,000,000 in total R&D expenditures during the previous fiscal year will be included in the population and asked to complete the HERD short form. All institutions reporting $1,000,000 or more in total R&D expenditures during the previous fiscal year will be included in the population and asked to complete the HERD standard form. Based on recent collection cycles, and allowing for a modest increase, the FY 2020-21 populations for the HERD standard and short forms are estimated to be 650 and 275.

Table B.1.1. HERD and FFRDC Response Rates, FY 2017-2019

Survey and form	FY 2017		FY 2018		FY 2019		FY 2020-21 Estimates
Survey and form	Survey universe	Response rate (%)	Survey universe	Response rate (%)	Survey universe	Response rate (%)	Survey universe	Response rate (%)

HERD	903	96.2	915	96.9	916	96.7	925	96.0
Standard form	644	96.4	646	97.5	647	97.7	650	97.0
Short form	259	95.8	269	95.5	269	94.4	275	94.0

FFRDC	42	100.0	42	100.0	42	100.0	42	100.0

The AAPOR response rate 6 (RR6) is used to calculate the overall survey response rate (https://www.aapor.org/AAPOR_Main/media/publications/Standard-Definitions20169theditionfinal.pdf page 62). This response rate includes complete plus partial responses in the numerator divided by all known eligible institutions. The FFRDC R&D survey has consistently obtained 100% response each year and that rate is expected for FY 2020-21. Based on recent collections (see Table B.1.1), the estimated response rate for HERD for FY 2020-21 is at least 96%, with a 97% response rate for institutions completing the standard form and a 94% response rate for institution completing the short form.

Information Collection Procedures

B.2.1 Identifying Eligible Institutions

Prior to the survey launch, a population review will be conducted to identify newly eligible institutions for the surveys. For the FFRDC survey this involves a review of the current NCSES Master List of FFRDCs to update the population as needed. For the HERD survey, this involves sending a screener to public or private nonprofit universities and colleges that, based on degrees granted and research expenditures reported to IPEDS, may expend $150,000 or more annually for R&D activities (see attachments 5 and 6). Screeners will be sent to:

institutions granting a bachelor’s degree or higher and reporting an amount greater than $0 for research on the most recently available IPEDS Finance Survey,
all U.S. service institutions granting a bachelor’s degree or higher, not reporting to IPEDS, and not already being part of the HERD population, and
institutions granting graduate degrees, not reporting to IPEDS, and not already being part of the HERD population.

The number of academic institutions contacted is estimated to be approximately 125 per year. These institutions will receive an email with a pdf attachment of the Population Review questionnaire. This brief survey will ask whether the institution spent over $150,000 on R&D projects during either of the past two fiscal years, and if so, whether the expenditures were estimated to be between $150,000 to $999,999 or $1,000,000 or more. Institutions began being asked about two years of expenditures on the FY 2019 screener. Close to 74% of institutions receiving the screener is past years had been contacted repeatedly for multiple years. Asking about two years of R&D expenditures reduces burden to the population by limiting contacts to any specific institution to every two years.

All institutions reporting on the screener at least $150,000 but less than $1,000,000 in total R&D expenditures during the previous fiscal year will be included in the population and asked to complete the HERD short form. All institutions reporting $1,000,000 or more in total R&D expenditures during the previous fiscal year will be included in the population and asked to complete the HERD standard form.

B.2.2 Collection Procedures

After eligible institutions have been identified, an initial request to verify contact information will be sent to the primary contacts identified during the previous collection or on the population review screener. One week later the respondents will be sent an email with information about how to access the web survey including user ID and instructions to create a password. Each institution’s web survey includes data from the 2 preceding years for arithmetical and trend check purposes. Large increases or decreases in expenditures are compared with the preceding years and flagged, and respondents are asked to explain and reconcile these differences.

Telephone and email follow-up will begin approximately 2 weeks after survey transmittal to those respondents who have not verified they are the correct contact for their institution. Reminder mailings, emails and telephone calls will be made at several points throughout the survey collection period. All telephone contacts include an offer to assist the respondent in providing the data. NCSES and the contractor will monitor institution response status closely through a web-monitoring system during the data collection cycle, with periodic contact maintained via telephone and email until survey closeout to ensure that (a) a response is provided and (b) data errors are resolved. This level of interaction with the institutions is responsible for the high survey response rates that have been achieved for the past years (see Attachment 16 for the full set of contact procedures).

B.2.3 Estimation Procedures

Upon survey closeout, NCSES will develop estimates for that portion of the survey population that did not respond. Imputation is performed using prior years' figures derived from the data of respondent institutions with similar characteristics, such as highest degree granted and type of institutional control (public or private). This process has been used consistently since 1976. As a result of high overall response rates, especially from institutions that account for the largest share of the HERD total, imputed amounts account for a very small percentage of total R&D expenditures (0.1 percent of the total in FY 2019). Every year 100% of institutions ranked in the top 150 for total R&D expenditures in the previous year have responded to the survey. These 150 institutions account for 90% of total R&D expenditures at U.S. universities and colleges.

Statistical Accuracy of the Collection

B.3.1 Methods Used to Maximize Response Rates

All of the follow-up procedures described here and in the previous section have been instituted in part to maximize the survey’s response rate. While the official survey deadlines will be three months after the opening of the survey, the data collection window will be held open as long as possible to obtain responses from all FFRDCs and at least 95% of the HERD population (including all of the top 150 institutions in R&D spending in the prior FY). Institutions that do not respond by the deadline will be contacted every one to two weeks either by email or phone to remind them of the importance of completing the survey for their institution. For the HERD, if an institution has still not responded approximately six weeks after the survey deadline, a letter will be sent to their president or chancellor making one final request for their participation. These methods have delivered a response rate of 94% or higher approximately eight weeks after the initial survey deadline for the last several survey cycles. Additionally, because imputation of missing data will based on past reports, special efforts will be made to get participation from institutions that have not responded for multiple years or have never completed a HERD survey. During the FY 2019 collection this included tailored letters to the provosts of 10 institutions highlighting the importance of the survey and the inclusion of institution data in publicly used data.

B.3.2 Imputation Methods for Unit and Item Nonresponse

Unit and Item Nonresponse. As shown in section B.1, the response rate is expected to be over 95% each year for the HERD survey, and 100% for the FFRDC R&D survey. In FY 2019 thirty universities and colleges did not respond to the HERD survey, out of a total of 916 eligible institutions, for a nonresponse rate of 3.3%.

Item nonresponse rates for the HERD survey typically do not exceed 10% for any question and it is expected that this will continue for FY 2020-21. In FY 2019 item nonresponse varied from 0.0% for Question 1, total R&D expenditures by source of funds, and Question 9, federal expenditures by R&D field, to 7.9% for Question 6, R&D expenditures by type of R&D (basic research, applied research, and experimental development). No question had a nonresponse rate of 10.0% or higher.

Item nonresponse for the FFRDC survey is also expected to be low. Most FFRDCs have incorporated the data needed to complete the FFRDC R&D survey questions into their record-keeping systems. However, there are two question that typically experience some level of item nonresponse: 1) Question 5 which asks for expenditures by type of cost and 2) Question 6, which asks for the organization’s operating budget. In FY 2019 twelve FFRDCs chose not to complete Question 5 and one did not report its operating budget in Question 6.

Imputation Methods. NCSES will develop estimates for that portion of the HERD survey population that did not respond. All missing data within partial responding and total nonresponding institutions will be imputed, with the following exceptions:

where imputation of partial nonresponse will result in a change to a reported total (e.g. total R&D expenditures)
Question 10 which asks for the names of specific federal agencies funding R&D reported under other agencies in Question 9
Question 13 which asks for an institution’s capitalization thresholds for software and equipment

Imputation techniques for variables can be broadly divided into two steps:

Using inflator/deflator factors to impute key variables based on the previous year’s data for each nonresponding institution. Key variables are values identified as having high correlations across years and high correlations with other, smaller values within the current‑year survey responses.
Using the relative percentages that were last reported by that institution or by peer institutions in the current year as a reference for the distribution of the key variables across detail fields. Imputed amounts are based on a mean value or mean proportion of a value within a group of institutions with similar characteristics, referred to as an imputation class.

Imputation or weighting procedures used to derive a national estimate for R&D personnel (headcounts and FTEs) will be designed after the majority of FY 2020 data for these new questions has been submitted.

Details for these methods are documented in the FY 2018 HERD Methodology Report. The imputation section from that report is included in Attachment 17.

Historically, all units have responded to the FFRDC R&D survey and so unit imputation will not necessary. Some FFRDCs do not respond fully or in part to questions 1, 5, or 6, but imputation will not be used for item nonresponse. The small population size and variation in size and nature of research at FFRDCs, makes estimations of expenditures based averages unreliable.

B.3.3 Accuracy and Reliability of Data

Because the HERD and FFRDC R&D surveys are censuses with high response rates and include an extensive review process to check the consistency of each institution’s data within and across years, the accuracy of these data are very high. In addition, statistical imputation is conducted for nonresponse to the HERD survey, but imputed amounts account for a very small percentage of total R&D expenditures (0.1 percent of the total in FY 2019).

Tests of Procedures

No methodological research is planned for HERD or FFRDC R&D for FY 2020-2021. NCSES will submit plans informing OMB of its intention to use methodological research burden hours before any applicable studies are undertaken.

Contact Information

The individuals consulted on HERD and FFRDC R&D surveys technical and statistical issues are listed in Table B.5.1 along with project staff at ICF, the contractor that conducts the two surveys.

Table B.5.1. Individuals Consulted on HERD and FFRDC R&D Surveys Technical and Statistical Issues

Name	Affiliation	Telephone Number
Mr. Michael Gibbons, HERD and FFRDC Project Officer	National Science Foundation, NCSES, Alexandria, VA	703-292-4590
Mr. John E. Jankowski Program Director	National Science Foundation, NCSES, Alexandria, VA	703-292 7781
Mr. Jock Black Mathematical Statistician	National Science Foundation, NCSES, Alexandria, VA	703-292-7802
Ms. Rebecca L. Morrison Survey Methodologist	National Science Foundation, NCSES, Alexandria, VA	703-292-7794
Ms. Kathryn Harper HERD/FFRDC Project Director	ICF Rockville, MD	301-572-0202
Ms. Jennifer Greer Data Management Task Leader	ICF Rockville, MD	815-993-8012
Ms. Carolyn Bennett Data Collection Task Leader	ICF Rockville, MD	301-572-0322
Mr. David Greene Systems Development Task Leader	ICF Rockville, MD	301-572-0525
Mr. Adam Lee Survey Methodologist	ICF Rockville, MD	301-572-0814

1 See https://www.nsf.gov/statistics/ffrdclist/.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Gibbons, Michael
File Modified	0000-00-00
File Created	2021-01-13