Date: January 6, 2020
To: Margo Schwab, Desk Officer
Office of Management and Budget
From: Emilda B. Rivers, Division Director
National Center for Science and Engineering Statistics
Via: Suzanne Plimpton, Reports Clearance Officer
National Science Foundation
Subject: Notification of cognitive interview research under generic clearance
This memorandum is to inform you of the National Center for Science and Engineering Statistics’ plans to conduct cognitive research under the generic clearance for survey improvement projects (OMB number 3145-0174). This research will focus on testing and obtaining feedback regarding proposed new content for the Survey of State Government Research and Development.
Background
The National Center for Science and Engineering Statistics (NCSES), within the National Science Foundation (NSF) is broadly tasked with measuring the role of science and technology (S&T) in the United States’ economy. A component of this activity is its sponsorship of the Survey of State Government Research and Development (State R&D Survey), which has been conducted every one or two years since 2006, and annually since 2016. The State R&D Survey collects information on the sources and recipients of funds from state government agencies for research and development; approximately 700 state agencies are in the survey universe, although only about 350 have R&D expenses in a given fiscal year.
The State R&D Survey is one of several surveys at NCSES that collect comparable information on R&D from different sectors of the economy (e.g., businesses, colleges and universities, and government). However, unlike some of the other surveys, it does not currently collect information about the personnel carrying out R&D activities, making it less useful for measurements of the R&D workforce in the United States. The addition of these variables will allow NCSES to provide information not available elsewhere to data users interested in science policy, the nature of the science and engineering workforce, and U.S. R&D competitiveness. NCSES intends to address this shortcoming by adding questions regarding the number of researchers and technicians involved in R&D, and the number of full-time equivalents for those personnel. Before adding those questions to the survey, NCSES intends to conduct cognitive interviews with several state agencies to get their feedback on proposed questions. The wording proposed for the State R&D Survey mirrors questions currently found on other NCSES surveys of R&D performers.
Research Plan
There were approximately 180 state agencies which reported some internal R&D (that is, R&D performed by the state agency itself, as opposed to funding R&D performed by others) in FY 2018. Some states have many agencies that report internal R&D, while other states have only a few.
NCSES plans to conduct up to 20 in-person interviews with state agencies that reported internal R&D during the FY 2018 survey cycle; the 20 interviews would be spread across two rounds of testing. Agencies will be selected in states that can be clustered together for ease and efficiency of travel. Each of these states have sufficient numbers of agencies with internal R&D that would allow us to canvass approximately 10 to 15 percent of agencies reporting intramural R&D. Limiting the number of interviews would reduce our ability to assess question performance. Expanding the number of interviews would increase travel costs and redundancy. NCSES will select two of these four options:
Georgia, South Carolina, and North Carolina
New York, New Hampshire, Massachusetts, and Connecticut
Ohio, Indiana, Illinois, Kentucky
California, Oregon, Washington
To recruit respondents, NCSES will send e-mail invitations (Appendix A) to the FY 2018 survey respondents in selected agencies across a cluster of states. The email invitation will provide detailed information about the nature of the investigation, including the proposed revisions and the process for providing feedback. If the institution has not responded within a few days, the respondent will be contacted by telephone. An appointment will be requested, at their convenience, with the person most familiar with the content of the proposed revisions, which in most cases will be the agency’s FY 2018 survey respondent. After respondents agree to participate, NCSES will send an e-mail to thank them for their willingness to participate, offer to answer any questions, and confirm the appointment.
The cognitive interviews will be semi-structured to understand respondents’ record keeping of internal R&D performance and to obtain feedback about the proposed question changes for the FY 2020 survey cycle. The 2019 cycle of the survey will be in the field at the time of testing, so that version of the instrument (Appendix B) will be used to set the context for the proposed new content (Appendix C). The discussions will focus on the definitions of R&D, the wording of the questions, the ability of the respondent to answer the new questions, and the appropriate placement of the new questions within the existing questionnaire; the interview protocol is in Appendix D. If respondents have already completed the 2019 cycle, the protocol will be adjusted as necessary, to collect information retrospectively. The results of the cognitive research will be used internally by the NSF staff for questionnaire development, and will be shared with the U.S. Census Bureau, the data collection agent for the State R&D Survey.
To ensure accurate note-taking, discussions will be recorded with the consent of the participants (Appendix E). The participants will be informed of the intent of the discussion, privacy protections, and voluntary nature of the discussions. They will be asked to verbally confirm their approval to record the discussion. Participants can decline to be recorded, at which point the recorder will be turned off and note-taking will serve as the only means of documentation.
Burden Information
NCSES expects to conduct a total of 20 interviews of about an hour each with survey respondents to understand how they keep records on R&D performing personnel and if they understand the wording of the newly proposed question. NCSES estimates that, in the recruiting phase of the project, approximately 55 agencies will be contacted, for approximately 5 minutes per agency, for a recruiting burden of approximately 5 hours. Thus, the estimated total burden for this project is about 25 hours (5 recruiting burden hours + 20 cognitive interview hours).
Incentive Payments
NCSES will not be offering incentive payments.
Contact Information
The contact persons for questions regarding this research are:
Christopher Pece
Project Officer
Research and Development Statistics Program
National Center for Science and Engineering Statistics
National Science Foundation
703-292-7788
John E. Jankowski
Program Director
Research and Development Statistics Program
National Center for Science and Engineering Statistics
National Science Foundation
703-292-7781
Rebecca L. Morrison
Survey Methodologist
Statistics and Methods Program
National Center for Science and Engineering Statistics
National Science Foundation
703-292-7794
Attachments
Appendix A: Initial Contact Email
Appendix B: FY 2019 State Government R&D Survey questionnaire
Appendix C: Proposed Changes for Testing
Appendix D: Interview Protocol
Appendix E: Consent Form
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |