Appendix B Quex and Protocols V2 (From the ZIP file)

3145-0100 Part B Revision

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. The Universe and Response Rates

The Universe

This is a census of approximately 750 eligible institutions. No sampling will be performed. The universe for the FY 2009 Academic R&D Survey population will be similar to the FY 2008 survey population. It includes bachelors and higher degree-granting U.S. institutions that annually perform at least $150,000 in separately budgeted S&E R&D. The frame for this survey is constructed through the screening of institutions identified as receiving federal funds by the NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions and from institutions identified in the NSF-National Institutes of Health Survey of Graduate Students and Postdoctorates in Science and Engineering as having master's or doctoral programs in S&E fields. All institutions on the compiled list that are not known to be within the scope of the survey are contacted by phone to determine whether they are in scope.

The reliability of the data and therefore the soundness of institutional, state, and national decisions based upon them are in large measure contingent upon receipt of complete and accurate data from all surveyed institutions. Since FY 2001, the survey also collects information on R&D expenditures from each of the Nation’s 37 Federally Funded Research and Development Centers (FFRDCs), whether administered by universities, industrial firms, or nonprofit organizations. (The NSF maintains information on each FFRDC’s mission statement, funding data, and type of R&D being performed, as directed under the Federal Acquisition Regulations recorded in the Federal Register, “35.017-6 Master list of FFRDC’s.”)

Response Rate Calculations

The response rate for the FY 2007 survey was 96.9 percent. The FY 2008 survey closed at 98.5 percent on April 22, 2009.

The AAPOR response rate #6 is used to calculate the overall survey response rate. This response rate includes complete plus partial responses in the numerator divided by all known eligible institutions.

2. Description of Survey Methodology

For the FY 2009 survey, the survey questionnaire and all materials will be provided electronically over the web. The regular survey will remain unchanged from the FY 2008 survey (see Attachment 3).

Telephone and email follow‑up will begin approximately 2 weeks after survey transmittal to those respondents who have not verified they are the correct contact for their institution. Reminder emails and telephone calls will be made at several points throughout the survey collection period. All telephone contacts include an offer to assist the respondent in providing the data. NSF and the contractor will monitor institution response status closely through a web-monitoring system during the data collection cycle, with periodic contact maintained via telephone and email until survey closeout to ensure that (a) a response is provided and (b) data errors are resolved. This level of interaction with the institutions is responsible for the high survey response rates that have been achieved for the past years.

NSF will continue to provide each respondent with a crosswalk between the Department of Education's National Center for Statistics' (NCES) Classification of Instructional Programs and the field of science and engineering disciplines defined by NSF on its Academic R&D Expenditures Survey. Electronically, this crosswalk is provided as a convenience to those institutions whose reporting systems were designed to accommodate the current NCES classification system.

Upon survey closeout, NSF will develop estimates for that portion of the survey population that did not respond. Imputation is performed using prior years' figures derived from the data of respondent institutions with similar characteristics, including highest degree granted and type of institutional control (public or private). This process has been used consistently since 1976. As a result of high overall response rates, especially from institutions that historically account for the largest share of the academic R&D total, imputed amounts account for a very small percentage of total R&D expenditures (only 0.1 percent of the total in FY 2007).

3. Methods Used to Maximize Response Rates

All of the follow-up procedures described here and in the previous section have been instituted in part to maximize the survey’s response rate. While the official survey deadline is two months after the opening of the survey, the data collection window is held open as long as possible to obtain responses from all of the previous year’s top 100 institutions in R&D expenditures as well as to achieve an overall response rate of at least 90 percent. Institutions that do not respond by the deadline are contacted every two weeks either by email or phone to remind them of the importance of completing the survey for their institution. If an institution has still not responded approximately six weeks after the survey deadline, a letter is sent to their president or chancellor asking for their participation. These methods have delivered a response rate of 94% or higher approximately eight weeks after the initial survey deadline for the last several survey cycles.

4. Tests of Procedures Used

Pilot Test of Redesigned Survey

In 2007, NSF began an intensive three-year effort to evaluate and redesign the survey. The goals of the redesign were (1) to update the survey instrument to reflect current accounting principles in order to obtain more valid and reliable measurements of the amount of academic research spending in the U.S., (2) to expand the current survey items to collect the additional detail most often requested by data users, and (3) to evaluate the feasibility of expanding the scope of data collected beyond that of R&D expenditures. As part of the redesign effort, NSF has undertaken the following activities with the assistance of Westat, a social science research firm:

• Literature review on uses of academic R&D data (May 2007)

• Data users’ workshop with approximately 30 academic R&D data users from government, universities, and nonprofit research organizations (June 2007)

• Expert panel meeting of 12 individuals with extensive experience in academic R&D policy, university accounting, and survey design (September 2007)

• Synthesis report summarizing university site visits and workshops over the previous 5 years to identify key cognitive issues on current survey (October 2007)

• Site visits to 16 universities to discuss current recordkeeping practices and feasibility of collecting additional survey items (January - March 2008)

• White paper on population and survey frame issues (March 2008)

• Email survey on “other fields” of research to help identify emerging disciplines (summer 2008)

• Pre-test of community college R&D screening survey (summer 2008)

• Second meeting of expert panel to discuss final recommendations based on results of previous work (August 2008)

• Development of revised questionnaire with additional items (October 2008)

• White paper on options for revising imputation methodology (November 2008)

• Site visits to 17 universities to cognitively test revised questionnaire (December 2008 – February 2009)

• Web usability testing of redesigned questionnaire planned for summer 2009 (OMB generic clearance will be requested for this activity).

SRS plans to administer a pilot test of the redesigned survey to 40 institutions during the fall of 2009. These 40 institutions will constitute a representative subset of the full survey population, and their response to the pilot test will be considered a complete response to the regular FY 2009 survey. The redesigned survey, which will be called the Higher Education R&D Survey, will continue to capture core information on R&D expenditures by sources of funding and field. However, the redesigned pilot survey will be expanded to include the following additional detail:

• Total R&D expenditures funded by nonprofit institutions (previously included under “Other sources”)

• Total amount of R&D expenditures funded from all types of foreign sources

• Total R&D expanded to include R&D expenditures in non-S&E fields as well as clinical trial expenditures

• Detail by field (both S&E and non-S&E) for R&D expenditures from each source of funding (federal, state/local, institution, industry, nonprofit, and other)

• Total R&D within interdisciplinary R&D centers

• Total R&D expenditures by direct cost categories (salaries, software, equipment, etc.)

• Counts of proposals submitted to funding organizations during fiscal year

• Counts and dollar amounts of R&D awards during fiscal year

• Headcount of R&D personnel (principal investigators and other staff)

• Test module on intellectual property and commercialization

A draft of the FY 2009 pilot survey is included in Attachment 4. Debriefing interviews will be held with each institution once their survey is completed in order to further revise the survey in preparation for full implementation in FY 2010. A revised OMB 83-I containing the final questionnaire and revised methodology will be submitted in advance of this implementation. The burden hours for the pilot test are included in this package with the 2009 burden estimates included in section A.12.

5. Names and Telephone Numbers of Individuals Consulted

The FY 2009 Academic R&D Survey and pilot test will be conducted by NSF's Division of Science Resources Statistics, under the general direction of Mr. John E. Jankowski, Program Director, Research and Development Statistics Program (703-292‑7781) and under the daily direction of Ms. Ronda Britt, Project Officer for the Academic R&D Expenditures Survey (703-292-7765). Reports will be prepared by NSF staff. Macro International, Bethesda, MD, is currently conducting the regular survey under the direction of NSF. The contact for Macro is Mr. Mark Morgan (telephone: 301-272-8542). Westat of Rockville, MD is conducting the redesign under the direction of NSF and will be responsible for administering the pilot test. The contact for Westat is Mary Hagedorn (telephone: 301-251-4273).

Professional statisticians and survey methodologists on the NSF staff are also available for review and advice, such as Jock Black, Mathematical Statistician, Division of Science Resources Statistics, and Dr. Fran Featherston, who has worked extensively on the survey redesign project.

Copies of cited documents are available from Ms. Ronda Britt, Survey Manager for the Academic R&D Expenditures Survey (703-292-7765).

5