Request for approval of non-substantive changes in the 2021 Survey of Earned Doctorates (3145-0019)

Memorandum

Date: June 9, 2020

To: Margo Schwab, Desk Officer

Office of Management and Budget

From: Emilda B. Rivers, Director

National Center for Science and Engineering Statistics

National Science Foundation

Via: Suzanne Plimpton, Reports Clearance Officer

National Science Foundation

Subject: Request for approval of non-substantive changes in the 2021 Survey of Earned Doctorates (3145-0019)

The National Center for Science and Engineering Statistics (NCSES) requests approval for non-substantive changes to the 2021 Survey of Earned Doctorates (SED). NCSES plans to add questions on the impact of Coronavirus Disease 2019 (COVID-19) on doctorate students’ graduate training and postgraduation plans. A short item set is proposed for inclusion in the approved 2021 SED survey instrument. The number of respondents is not changing. This request does not introduce significant changes to the estimated respondent burden or the costs to the federal government. NCSES expects that the time to complete the new items will be offset by the reduced time needed for educational history items that were recently approved by OMB.

Background

The SED is an annual census survey of new recipients of research doctorates from U.S. institutions, sponsored by the National Center for Science and Engineering Statistics (NCSES) within the National Science Foundation, the National Institutes of Health, the Department of Education, and the National Endowment for the Humanities.

The SED collects educational history, education-related debt, financial support during graduate studies, information about postgraduation plans, and personal demographic information for each research doctorate recipient, primarily using a web survey mode with a small percentage of cases (approximately 2% of respondents) completing the survey by telephone.

The novel coronavirus pandemic has disrupted many aspects of higher education and the training of the next generation of the scientific and technical workforce. Doctoral graduates may encounter multiple challenges along the path to doctorate completion that are directly attributable to the effects of the pandemic.

Among the research questions these items are designed to address:

• Has the coronavirus pandemic affected time to degree for research doctoral graduates?

• Has the pandemic disrupted research activities of doctoral graduates?

• Have doctoral graduates experienced a loss of financial support related to budgetary impact of the pandemic?

• Have doctoral graduates changed their immediate postgraduate plans as a result of the pandemic?

• Did the pandemic cause doctoral graduates to alter their career plans or goals in terms of research specialization or employment sector?

• Did the pandemic have an effect on the geographic mobility of doctorates? Are doctoral graduates on temporary visas more or less likely to prefer to remain in the U.S.?

Additional avenues of inquiry for these questions address whether the impacts felt by research doctoral graduates are differentiated by field of study, sex, race/ethnicity, or citizenship.

Proposed Changes

The SED 2021 instrument is being modified to add new questions to collect information on the widespread impacts of COVID-19. The items were tested in cognitive interviews with 9 individuals. Attachment 1 includes the items tested and the interview protocol.

Recruitment of participants for cognitive interviews. The sample was recruited from a list of 182 interested participants who emailed their interest in testing the SED educational history test instrument earlier in the year. The interested participants provided information in January 2020 about their expected doctorate graduation date, field of study, and university name. From this list, we sent a recruitment email message in May 2020 to 32 students who said they were finishing their doctorate in 2020 or 2021. Potential participants were provided with a description of the study and directed to fill out a brief web screener that asked questions about the respondent’s expected graduation date, sex, and interview scheduling availability. Some participants had discrepancies in the expected graduation date they provided in January 2020 and May 2020. As this may be related to COVID-19, they were kept in the pool of eligible participants. The nine respondents who completed the cognitive interviews represented six states in the East, South, Midwest and West. The fields of doctorate study included 3 engineering fields, 3 biology fields, public policy, information science, and geography.

Cognitive interview procedures. Staff from NCSES worked collaboratively with the SED survey team to develop items that would inform the research questions listed above. A survey methodologist conducted an expert review of the items and made suggestions that were used in the development of the final items to be tested in the cognitive interviews. All interviews were conducted using Zoom videoconferencing software. Participants were asked to log in to the web instrument where they were instructed to read the introduction by themselves for 1-2 minutes. After the participants finished reading, the interviewer confirmed that the participants agreed to participate and to be recorded. They were then instructed to answer the survey questions as they would by themselves. Once they finished answering the yes/no questions and the corresponding follow-up questions, the interviewer followed a guide (see Attachment 1 for the cognitive interview protocol) that contained a list of scripted probes. When necessary, interviewers asked unscripted probes to inquire more deeply about issues raised by respondents (e.g., can you tell me more about that). After interviews were completed, all notes were compiled and analyzed to identify issues and suggest possible areas for improvement. The final questions proposed for inclusion in the 2021 SED are shown in Exhibit 1.

Exhibit 1. Proposed new 2021 SED questions

The next questions focus on how the Coronavirus Disease 2019 (COVID-19) may have affected your graduate experiences and career plans.

Did you experience any of the following as a result of the COVID-19 pandemic?

Select Yes or No for each

	Yes	No
1. The timeline for completing my doctoral degree changed.	1	2
2. My research was disrupted.	1	2
3. Funding for my doctoral studies was reduced or suspended.	1	2
4. My immediate postgraduate employment or education plans changed.	1	2
5. My longer term career plans or goals changed (e.g., type of employer, research focus).	1	2
6. My plans about where to live in the year after graduation were affected.	1	2
7. My graduate experience or career plans changed in other ways.	1	2

If respondent says yes to any item, corresponding open-ended items appear on next screens with text boxes.

Previously, you said: “The timeline for completing my doctoral degree changed.”

How did the timeline for completing your doctoral degree change?

Previously, you said: “My research was disrupted.”

How was your research disrupted?

Previously, you said: “Funding for my doctoral studies was reduced or suspended.”

How was your funding reduced or suspended?

Previously, you said: “My immediate postgraduate employment or education plans changed.”

How did your immediate employment or education plans change?

Previously, you said: “My longer term career plans or goals changed (e.g., type of employer, research focus).”

How did your longer term career plans or goals change?

Previously, you said: “My plans about where to live in the year after graduation were affected.”

How did it impact your plans about where to live in the year after graduation?

Previously, you said: “My graduate experience or career plans changed in other ways.”

In what other ways did your graduate experience or plans change?

Justification for Changes

Results of testing. Overall, the seven impact items and the follow-up open-ended questions worked as intended. Participants were able to provide details about their situation when they said yes to experiencing an impact of COVID-19 on their graduate experiences or career plans. When those with negative responses to a specific impact question were probed, we also found that they understood the meaning of the question and indeed meant to answer “no.” While the respondents’ comprehension of the proposed questions were consistent with the intended design, we noticed that the framing of two items could be improved to better assist respondents in answering the question. Thus, items #5 and #6 were reworded to address issues identified in testing. Specific findings about each item can be found in Attachment 2.

Data release. The data generated by these new question items will be of interest to NCSES, the National Science Foundation, the National Science Board, and other agencies and organizations. The Congressional Research Service (CRS) recently released a paper outlining concerns about disruptions to federal research and development activities, and specifically mentioned concerns about the impact of these disruptions on graduate students¹. The proposed SED data items address CRS concerns and can be used to explore impacts both now and in the future through the Survey of Doctorate Recipients, which follows a sample of these doctorate graduates from the SED.

Data based on these new data items will be presented in an InfoBrief focused on the impact of COVID-19 and included in tables that accompany the 2021 SED data release in November 2022. A brief data analysis plan is included as Attachment 3.

2021 SED Data Collection Schedule

Date	Activity/Deliverable
06/09/2020	Submission to OMB for approval of substantive change
06/23/2020	Receive OMB approval
06/25/2020	Finalize revised 2021 SED web survey instrument with new questions
06/28/2020	Launch new questions for AY2021 graduates (July 2020–June 2021)
12/30/2021	2021 survey closeout
06/01/2022	2021 final data file delivery
11/30/2022	2021 SED data release

Contact Person

Kelly Kang

Project Officer

Survey of Earned Doctorates

Human Resources Statistics Program

National Center for Science and Engineering Statistics

National Science Foundation

[email protected]

703-292-7796

Attachments

Attachment 1: SED Coronavirus 2019 Module Cognitive Interview Protocol

Attachment 2: Summary of SED COVID-19 Questions Cognitive Testing Results

Attachment 3: Analysis Plan for SED COVID-19 Data Items

1 Morgan, Daniel and Sargent, John, Effects of COVID-19 on the Federal Research and Development Enterprise. Congressional Research Service Report #R46309. Washington DC. April 10, 2020. (See page 10 for specific concerns about impacts on graduate students and postdocs).

3