GSS NS Change request memo

Memorandum

Date: November 27, 2020

To: Margo Schwab, Desk Officer

Office of Management and Budget

From: Emilda B. Rivers, Director

National Center for Science and Engineering Statistics

National Science Foundation

Via: Suzanne Plimpton, Reports Clearance Officer

National Science Foundation

Subject: Request for approval of non-substantive changes in the 2020 NSF-NIH Survey of Graduate Students and Postdoctorates in Science and Engineering (3145-0062)

The National Center for Science and Engineering Statistics (NCSES) requests approval for non-substantive changes to the 2020 Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS). NCSES plans to add questionnaire items on the impact of Coronavirus Disease 2019 (COVID-19) to graduate students, postdoctorates, and GSS institutions. This request does not introduce changes to the estimated respondent burden or costs to the federal government presented in the recent OMB clearance request.

The recently approved information collection request (ICR) for the 2020-2022 GSS cycles included plans to cognitively test potential questionnaire items prior to the 2020 GSS data collection start. This memorandum summarizes the results of the cognitive testing effort and describes the questionnaire items proposed for inclusion on the 2020 GSS survey instrument. As discussed in the 2020-2022 GSS ICR, NCSES expects that the time to complete the new items will be approximately 7 minutes for each of the 911 school, student, and postdoc coordinators or 106 hours. The burden associated with these new items is reflected within the respondent burden estimate included in the 2020-2022 GSS ICR approved on August 26, 2020.

Background

The coronavirus pandemic is having a substantial impact on colleges, universities, and the nation’s workforce. In response, NCSES is adding short modules to some surveys to assess the pandemic’s disruption and impact. Each NCSES survey will assess the impact of the pandemic on a different subpopulation with the goal that the collective information obtained will provide insight on the impact of the pandemic on the nation’s science and engineering enterprise.

Many postsecondary institutions closed this spring and are still in flux this fall with a range of approaches to opening their institutions. Some research laboratories also closed, and although they may have reopened, capacity is limited due to social distancing concerns. The pandemic has also affected schools’ finances resulting in reductions in force and furloughs. These changes had only a small effect on the 2019 GSS response rates because the majority of data was submitted before the February 28 deadline, but it is possible that it could lead to major problems in the coming cycle, the 2020 GSS, and subsequent cycles. In addition, the pandemic could lead to noticeable changes in the GSS estimates of graduate student enrollment, postdoctoral appointments, and non-faculty researcher employment.

NCSES proposes adding a small number of questionnaire items to the upcoming 2020 GSS to assess the pandemic’s impact on the institutions that participate in the survey. These questionnaire items are designed to inform the following research questions.

• Will GSS coordinators be able to participate in the 2020 GSS?

Possible reductions in staff and data access issues will likely affect the ability of our respondents to provide data. NCSES would like to learn early in the process whether this might be an issue to enable the incorporation of data collection modifications to avoid a major reduction in survey participation.

• What changes are occurring within these institutions that may affect enrollments of master’s and doctoral students?

To address this question, proposals include items that ask about changes in school policies that may impact graduate student enrollment. This information will increase our understanding in two ways: 1) it will provide context for the GSS data quality checks performed after schools submit their data (a process that compares current data with past data to identify possible errors in reporting) and 2) it will provide insight on the broader data patterns related to the impact of the pandemic.

• What changes are occurring within these institutions that may affect postdocs and doctorate-level non-faculty researchers (NFRs)?

The GSS is the only federal survey that collects information on postdocs and NFRs, so it is important to gather information about how they may be affected by the pandemic. For example, are institutions rescinding or delaying offers to postdocs? Have the terms of postdoc appointments changed? Answers to these questions are designed to help inform our understanding of any changes in postdoc and NFR data in the 2020 GSS.

• Have research facilities or staffing at the institution been impacted?

Research laboratories closed this spring at many universities and have reopened with restricted access. In addition, many institutions have announced furloughs or other reductions in staff. Questionnaire items designed to inform this research question will provide insight on how facility changes have affected postdocs and NFRs.

Proposed GSS Changes

The GSS 2020 instrument is being modified to add new questions to collect information on the widespread impacts of the coronavirus pandemic. The 2020 GSS survey launched on November 17. The COVID items will be added to the Survey Menu of the 2020 GSS Survey instrument as soon as we receive approval from OMB. The web pages with the COVID questions will serve as the landing page for all coordinators, until the questions are answered and responses submitted. Coordinators will be able to navigate away from the COVID questions without providing answers, but this landing page approach will serve as a reminder and help to drive participation. The web pages with the COVID questions are separate from the main survey and have a unique drop-down menu that includes separate sections for (a) overall institutional questions; (b) questions specific to student data (if applicable); and (c) questions specific to postdoc and NFR data (if applicable). Coordinators save and submit their COVID question responses in the pages they are posed, which allows these questions to remain separate from GSS data reporting, submission, and survey participation.

Once the instrument has been modified, we will send an email to all coordinators that introduces the questions and announces their presence in the survey instrument. The announcement email and the survey instrument will instruct coordinators to respond by 1/29/21 though responses will be accepted through the duration of data collection. The email can be found in Attachment 1.

When considering both the COVID question rollout and the design of the questions themselves, it is important to note both the expertise of, and resources available to, the GSS survey coordinators. The GSS survey coordinators are typically from Institutional Research offices. Further, a vast majority of coordinators, 93.2% or 836 of the 897 invited to participate in the 2020 GSS have participated in the survey in prior years. All coordinators, whether veterans of the survey or not, are encouraged to attend a training webinar early in the survey cycle and have access to a survey help desk whereby they can communicate questions or concerns via email or phone. Employees from the survey contractor RTI operate the Help Desk phoneline and have an average of 3 years of experience. The GSS staff have established relationships with the coordinators and are prepared to address any questions regarding the COVID items.

The items we propose to include were tested in cognitive interviews with 20 individuals. Exhibit 1 presents the proposed new items. Attachment 2 includes the items tested and the interview protocol.

Recruitment of participants for cognitive interviews. A convenience sample based on GSS reporting history was generated from the universe of 893 GSS coordinators. The list included school, student, and postdoc coordinators that had a good rapport with the GSS Helpdesk and completed the survey annually. From 9 September 2020 through 25 September 2020 recruitment emails were sent to 79 coordinators inviting them to participate. Nonresponse follow ups were conducted via telephone. Potential participants were provided with a description of the study and asked to provide their interview scheduling availability. Twenty coordinators completed the cognitive interviews, representing thirteen school coordinators, 4 student coordinators, and 3 postdoc coordinators.

Cognitive interview procedures. Staff from NCSES worked collaboratively with the GSS survey team to develop items that would inform the research questions listed above. A survey methodologist conducted an expert review of the items and made suggestions that were used in the development of the final items to be tested in the cognitive interviews. All interviews were conducted using Zoom videoconferencing software. Participants were asked to log in to the web instrument where they were instructed to read the introduction by themselves for 1-2 minutes. After the participants finished reading, the interviewer confirmed that the participants agreed to participate and to be recorded. The interviews were conducted using a think-aloud protocol. Respondents were asked to answer the survey questions while talking to the interviewer, explaining how they decided to answer each question. The interviewer followed a guide (see Attachment 2 for the cognitive interview protocol) that contained a list of scripted probes. If the respondent did not provide sufficient detail, interviewers asked unscripted probes to inquire more deeply about issues raised by respondents (e.g., can you tell me more about that). After interviews were completed, all notes were compiled and analyzed to identify issues and suggest possible areas for improvement. The final questions proposed for inclusion in the 2020 GSS are shown in Exhibit 1.

Results of testing. Results of the cognitive tests suggest that participants largely understood the items, though slight revisions have been suggested to six of the questions. An overview of the types of edits suggested is provided here. Please see attachment 2 for further detail, and motivation, regarding the recommended edits. Recommendations include: the addition of clarifying text [A-2, C-2, C-3, C-6]; the addition of a “don’t know” response option [B-2, B-3, C-3]; and an adjustment in level-of-detail collected [B-1, C-1]. The revised recommended item wording is presented in Exhibit 1.

Note that not all respondents were asked all questions; the questions shown to coordinators depended on whether they reported master’s students, doctoral students, postdocs and/or doctorate-level nonfaculty researchers (NFRs) on the 2019 GSS. Specific findings about each item can be found in Attachment 2.

Data release. The data generated by these new question items will be of interest to NCSES, the National Science Foundation, the National Science Board, and other agencies and organizations. The Congressional Research Service (CRS) recently released a paper outlining concerns about disruptions to federal research and development activities, and specifically mentioned concerns about the impact of these disruptions on graduate students¹. The data will be used to identify potential issues facing GSS coordinators in reporting data so that the project team can be proactive in offering support that might facilitate continued participation.

The data provided by coordinators could also be used in a special topic InfoBrief to be published in 2021. It would provide a snapshot of the issues occurring on campuses that have graduate programs in science and engineering. These data will also provide context for analyzing increases or drops in enrollment that may occur, and changes in the numbers of postdocs by field, citizenship, and sex. This type of analysis will inform the data release InfoBrief that will provide information on trends in graduate enrollments and postdoc and NFR employment. A brief data analysis plan is included as Attachment 3.

Exhibit 1. COVID-19 module for GSS 2020

This year a new module has been added to the GSS.  The intent of this short module is to measure the impact of the novel coronavirus (COVID-19) pandemic on your ability to complete the GSS, as well as the expected impact on graduate student enrollment and the employment of postdoctorate and doctorate holding non-faculty researchers at your institution.

When responding to these questions, please consider only the graduate students, postdocs, and/or NFRs that you report to the GSS.

You may navigate through the pages of the survey by using the ‘Next’ and ‘Previous’ buttons at the bottom of each page. You may answer the questions in multiple sittings but remember to save your responses and press the ‘Submit’ button at the bottom of the last survey page.

 

We estimate that the module will take approximately 7 minutes to complete.  We ask that you complete these questions by 1/29/2021. The due date for GSS data submission is March 1, 2021

_____________________________________________________________________________________

Part A

The first two questions will help us understand the challenges you may face when completing the GSS this year. (All coordinators)

A-1. Do you anticipate that the novel coronavirus (COVID-19) pandemic might impact your ability to complete the GSS 2020 survey because of decreased staff resources?

__Yes __No __Don’t know

_____________________________________________________________________________________

A-2. Do you anticipate that COVID-19 might impact your ability to complete the GSS 2020 survey because of issues regarding data access? As you answer this question, please think only about accessing the data and not the quality of the available data.

__Yes __No __Don’t know

_____________________________________________________________________________________

Part B

The remaining questions ask about impacts your institution has experienced this fall due to the novel coronavirus (COVID-19) pandemic.

In response to the COVID-19 pandemic, have your institution’s graduate student policies, enrollments, or funding been impacted in any of the following ways? (School and Student coordinators)

B-1. Did any graduate programs intentionally reduce or suspend the number of newly admitted graduate students this fall due to the COVID-19 pandemic?

__Yes __No __ Don’t know

a. If yes, please describe the changes. _____________________________________________________________________________________

B-2. Did your institution change the definition of full-time or part-time graduate enrollment this fall due to the COVID-19 pandemic?

__Yes __No __Don’t know

a. If yes, please describe the changes.

_____________________________________________________________________________________

B-3. This fall, have any doctoral programs at your institution changed policies regarding…

a. time-to-degree extensions for doctoral students?

__Yes, change at the institution level

__ Yes, change at the program level

__No

__Don’t know

b. length of time doctoral students can receive financial support?

__Yes, change at the institution level

__ Yes, change at the program level

__No

__Don’t know

c. If yes to either, please describe the changes that were made.

_____________________________________________________________________________________

B-4. This fall, have enrollments increased, stayed about the same, or decreased for…

a. master’s students who are US citizens or permanent residents?

__Increased __Stayed about the same __Decreased __Don’t know

b. master’s students who are temporary visa holders?

__Increased __Stayed about the same __Decreased __ Don’t know

c. doctoral students who are US citizens or permanent residents?

__Increased __Stayed about the same __Decreased __ Don’t know

d. doctoral students who are temporary visa holders?

__Increased __Stayed about the same __Decreased __ Don’t know

_____________________________________________________________________________________

B-5. This fall, has financial support for graduate students increased, stayed about the same, or decreased for…

a. master’s students: __Increased __Stayed about the same __Decreased __Don’t know

b. doctoral students: __Increased __Stayed about the same __Decreased __Don’t know

_____________________________________________________________________________________

B-6. Are there other impacts ofCOVID-19 that may affect graduate student enrollment or financial support at your institution this fall?

__Yes __No

a. If yes, what are they? _____________________________________________________________________________________

Part C

The next set of questions concern impacts on postdocs and doctorate-level non-faculty researchers (NFRs) stemming from the novel coronavirus (COVID-19) pandemic this fall. (School and postdoc coordinators)

C-1. This fall, have the number of postdocs at your institution increased, stayed about the same, or decreased?

a. postdocs who are US citizens or permanent residents?

__Increased __Stayed about the same __Decreased __Don’t know

b. postdocs who are temporary visa holders?

__Increased __Stayed about the same __Decreased __ Don’t know

_____________________________________________________________________________________

C-2. This fall, have any programs at your institution extended the number of years postdocs can receive funding due to the COVID-19 pandemic?

__Yes __No __ Don’t know

_____________________________________________________________________________________

C-3. Most institutions have policies that limit the number of years someone can serve as a postdoc. This fall, has your institution extended appointments for postdocs due to the COVID-19 pandemic?

__Yes __No __Don’t know __Does not apply, length of postdoc appointments not limited.

_____________________________________________________________________________________

C-4. This fall, has your institution had to hold off making offers or rescind offers to any new postdocs due to the COVID-19 pandemic?

__Yes __No __Don’t know

_____________________________________________________________________________________

C-5. This fall, have any of the following been affected by hiring freezes, furloughs, involuntary reductions in time, or layoffs due to the COVID-19 pandemic?

a. Postdocs? __Yes __No __Don’t know

b. Doctorate-holding nonfaculty researchers (NFRs)? __Yes __No __Don’t know

_____________________________________________________________________________________

C-6. Are there other impacts ofCOVID-19 that may affect the quality of your institution’s postdoc or NFR data?

__Yes __No

a. If yes, what are they?

_____________________________________________________________________________________

Contact Person

Michael Yamaner

Project Officer

Survey of Graduate Students and Postdoctorates in Science and Engineering

Human Resources Statistics Program

National Center for Science and Engineering Statistics

National Science Foundation

[email protected]

703-292-7815

Attachments

Attachment 1: Email to Coordinators

Attachment 2: GSS COVID-19 Questions Cognitive Testing Report

Attachment 3: Analysis Plan for GSS COVID-19 Data Items

1 Morgan, Daniel and Sargent, John, Effects of COVID-19 on the Federal Research and Development Enterprise. Congressional Research Service Report #R46309. Washington DC. April 10, 2020. (See page 10 for specific concerns about impacts on graduate students and postdocs).

4