3145-0100, Section B

3145-0100, Section B

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. The Universe and Response Rates

The Universe

This is a census of all eligible institutions. No sampling will be performed. The universe for the FY 2010 HERD Survey population will include all bachelor’s and higher degree-granting U.S. institutions that spent at least $150,000 in separately budgeted R&D in their academic FY 2010. Previously the population was determined by screening all institutions receiving at least $400,000 of federal funds over the previous four years as reported in the NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions. Because FY 2010 represents the first year that both S&E and non-S&E R&D spending are included in the threshold, the population for the FY 2010 survey will be identified through a screening of all bachelor’s and higher degree-granting institutions as identified by IPEDS (see Attachment 7 for a draft of the population screening survey). The screening survey will request total R&D expenditures by source of funding in FY 2010. All institutions reporting over $150,000 in total R&D expenditures will be automatically included in the FY 2010 survey population, using their screening survey responses in lieu of completion of the full HERD questionnaire for FY 2010.

The results of this screening will be analyzed against the IPEDS finance survey data on research expenses as well as NSF’s Federal S&E Support survey data to determine the degree of correlation. If a reasonable degree of correlation is found between the HERD screening survey data and the IPEDS and/or Federal Support survey data by institution, one or both of these data sources will be used to build the population frame in FY 2011 and beyond in order to minimize the burden and expense involved in annual screening of all bachelor’s and higher degree institutions. If it is possible to use the other data sources to construct the HERD survey frame, the screening in FY 2011 and later years will be reduced to those institutions identified through the other data sources as having some level of R&D activity (i.e. non-zero research expenses in IPEDS and/or federal obligations for S&E R&D.

Since FY 2001, the survey also collects information on R&D expenditures from each of the Nation’s 39 Federally Funded Research and Development Centers (FFRDCs), whether administered by universities, industrial firms, or nonprofit organizations. The NSF maintains information on each FFRDC’s mission statement, funding data, and type of R&D being performed, as directed under the Federal Acquisition Regulations Section 35-017-6, available at https://www.acquisition.gov/FAR.

Response Rate Calculations

The AAPOR response rate #6 is used to calculate the overall survey response rate. This response rate includes complete plus partial responses in the numerator divided by all known eligible institutions.

2. Description of Survey Methodology

The HERD population screening survey will be conducted via mail. A survey package will be mailed to the office of the institution president with a response requested within six weeks. A reminder letter will be sent after two weeks, and another copy of the survey will be mailed if no response is received by the deadline. Telephone and/or email follow-up will continue until the desired response rate is reached.

For the HERD and FFRDC surveys, the survey questionnaire and all materials will be provided electronically over the web. An initial contact verification email will be sent to all previous year respondents. One week later the respondents will be sent survey information and a user ID and password via email. Each institution’s web survey includes data from the 2 preceding years for arithmetical and trend check purposes. Large increases or decreases in expenditures are compared with the preceding years and flagged, and respondents are asked to explain and reconcile these differences.

Telephone and email follow-up will begin approximately 2 weeks after survey transmittal to those respondents who have not verified they are the correct contact for their institution. Reminder emails and telephone calls will be made at several points throughout the survey collection period. All telephone contacts include an offer to assist the respondent in providing the data. NSF and the contractor will monitor institution response status closely through a web-monitoring system during the data collection cycle, with periodic contact maintained via telephone and email until survey closeout to ensure that (a) a response is provided and (b) data errors are resolved. This level of interaction with the institutions is responsible for the high survey response rates that have been achieved for the past years.

Upon survey closeout, NSF will develop estimates for that portion of the survey population that did not respond. Imputation is performed using prior years' figures derived from the data of respondent institutions with similar characteristics, including highest degree granted and type of institutional control (public or private). This process has been used consistently since 1976. As a result of high overall response rates, especially from institutions that historically account for the largest share of the HERD total, imputed amounts account for a very small percentage of total R&D expenditures (only 0.1 percent of the total in FY 2008).

3. Methods Used to Maximize Response Rates

All of the follow-up procedures described here and in the previous section have been instituted in part to maximize the survey’s response rate. While the official survey deadline is three months after the opening of the survey, the data collection window is held open as long as possible to obtain responses from all of the previous year’s top 100 institutions in R&D expenditures as well as to achieve an overall response rate of at least 95 percent. Institutions that do not respond by the deadline are contacted every two weeks either by email or phone to remind them of the importance of completing the survey for their institution. If an institution has still not responded approximately six weeks after the survey deadline, a letter is sent to their president or chancellor asking for their participation. These methods have delivered a response rate of 94% or higher approximately eight weeks after the initial survey deadline for the last several survey cycles.

4. Tests of Procedures Used

Pilot Test of Redesigned Survey

In 2007, NSF began an intensive three-year effort to evaluate and redesign the survey. The goals of the redesign were (1) to update the survey instrument to reflect current accounting principles in order to obtain more valid and reliable measurements of the amount of academic research spending in the U.S., (2) to expand the current survey items to collect the additional detail most often requested by data users, and (3) to evaluate the feasibility of expanding the scope of data collected beyond that of R&D expenditures. As part of the redesign effort, NSF undertook the following activities with the assistance of Westat, a social science research firm:

• Literature review on uses of academic R&D data (May 2007)

• Data users’ workshop with approximately 30 academic R&D data users from government, universities, and nonprofit research organizations (June 2007)

• Expert panel meeting of 12 individuals with extensive experience in academic R&D policy, university accounting, and survey design (September 2007)

• Synthesis report summarizing university site visits and workshops over the previous 5 years to identify key cognitive issues on current survey (October 2007)

• Site visits to 16 universities to discuss current recordkeeping practices and feasibility of collecting additional survey items (January - March 2008)

• White paper on population and survey frame issues (March 2008)

• Email survey on “other fields” of research to help identify emerging disciplines (summer 2008)

• Pre-test of community college R&D screening survey (summer 2008)

• Second meeting of expert panel to discuss final recommendations based on results of previous work (August 2008)

• Development of revised questionnaire with additional items (October 2008)

• White paper on options for revising imputation methodology (November 2008)

• Site visits to 17 universities to cognitively test revised questionnaire (December 2008 – February 2009)

• Web usability testing of redesigned questionnaire (summer 2009).

Finally, SRS administered a pilot test of the redesigned survey to 40 institutions during the fall of 2009. These 40 institutions constituted a representative subset of the full survey population, and their response to the pilot test was considered a complete response to the regular FY 2009 survey.

A copy of the FY 2009 pilot survey questionnaire is included in Attachment 8. Debriefing interviews were held with each institution once their survey was completed in order to further revise the survey in preparation for full implementation in FY 2010. A copy of the debriefing report is included in Attachment 9. All changes made to the draft FY 2010 questionnaire as a result of the pilot debriefings are being cognitively tested with a subset of the pilot institutions this summer.

5. Names and Telephone Numbers of Individuals Consulted

The FY 2010 HERD Survey will be conducted by NSF's Division of Science Resources Statistics, under the general direction of Mr. John E. Jankowski, Program Director, Research and Development Statistics Program (703-292‑7781) and under the daily direction of Ms. Ronda Britt, Survey Manager for the HERD Survey (703-292-7765). Reports will be prepared by NSF staff. NSF is awaiting procurement of a new contract for the FY 2010-13 surveys.

Professional statisticians and survey methodologists on the NSF staff are also available for review and advice, such as Jock Black, Mathematical Statistician, Division of Science Resources Statistics, and Dr. Fran Featherston, who worked extensively on the survey redesign project.

Copies of cited documents are available from Ms. Ronda Britt, Survey Manager for the HERD Survey (703-292-7765).

ATTACHMENT 1

FY 2010 Higher Education R&D Survey questionnaire

ATTACHMENT 2

FY 2010 FFRDC R&D Survey questionnaire

ATTACHMENT 3

FY 2009 Higher Education R&D Survey pilot contact materials

ATTACHMENT 4

First Federal Register Notice for the FY 2010 HERD Survey

ATTACHMENT 5

Comment letter from the Council on Governmental Relations (COGR)

ATTACHMENT 6

Changes to FY 2010 Higher Education R&D Survey

ATTACHMENT 7

FY 2010 Higher Education R&D population screening questionnaire

ATTACHMENT 8

FY 2009 Higher Education R&D Survey questionnaire (pilot test)

ATTACHMENT 9

FY 2009 Higher Education R&D Survey pilot test debriefing report