Request for approval of non-substantive changes in the 2021 Survey of Earned Doctorates (3145-0019)

Memorandum

Date: April 24, 2020

To: Margo Schwab, Desk Officer

Office of Management and Budget

From: Emilda B. Rivers, Director

National Center for Science and Engineering Statistics

National Science Foundation

Via: Suzanne Plimpton, Reports Clearance Officer

National Science Foundation

Subject: Request for approval of non-substantive changes in the 2021 Survey of Earned Doctorates (3145-0019)

The National Center for Science and Engineering Statistics (NCSES) requests approval for non-substantive changes to the collection of education questions in the 2021 Survey of Earned Doctorates (SED). The changes described in this memo stem from a previously approved methodological study conducted under the NCSES generic clearance (OMB No. 3145-0174, memo dated October 30, 2019) in order to improve the quality of education data collected in the SED while also reducing respondent burden. The number of respondents is not changing. The methodological study results suggest that these non-substantive changes to the collection of SED education questions may reduce burden, though NCSES is not changing the total estimated burden at this time.

Background

The SED is an annual census survey of new recipients of research doctorates from U.S. institutions, sponsored by the National Center for Science and Engineering Statistics (NCSES) within the National Science Foundation, the National Institutes of Health, the Department of Education, and the National Endowment for the Humanities.

The SED collects educational history, education-related debt, financial support during graduate studies, information about postgraduation plans, and personal demographic information for each research doctorate recipient, primarily using the web survey mode with a small percentage of cases (approximately 2% of respondents) completing the survey by telephone.

The education section of the SED questionnaire is divided into Part A1, which collects respondents’ doctorate and dissertation research fields of study, and Part A2, which collects the field of study of respondents’ other postsecondary degrees. Proposed changes to the 2021 SED questions are summarized below with the justification for them in the next section.

Educational history data for research doctorate recipients is a rich and valuable source of information for federal agencies, policymakers, and researchers. However, collecting these data is a complex undertaking and requires considerable input from respondents. Innovation to reduce response burden and improve data quality is an important goal for NCSES. For this reason, NCSES undertook a methodological study to test (1) an alternative method for collecting respondents’ field of study (FOS) data and (2) different methods for collecting respondents’ complete educational history data.

The testing of the FOS data involved the comparison of two methods: Conditions A and B.

• Condition A, which was the current SED method of respondents typing their field in a searchable box and selecting one field from the dropdown list based on partial match of the typed field string.

• Condition B, a newly proposed method of respondents typing their field in a text box, which was matched against the exact field strings from an auto-coding list.

The testing of the educational history involved three methods of collecting respondents’ postsecondary degrees prior to earning their research doctorate: Conditions C, D, and E.

• Condition C was based on the current SED degree history questions, which asked the respondents about other postsecondary degrees they have in the following order: associate’s degree, bachelor’s degree, master’s degree, and professional doctorate. For each degree type, the survey first asked about the number of each type of degrees earned. If the respondent had multiple degrees of same type, then information about the most recent degree and the first degree were collected.

• Condition D was similar to Condition C, except the degrees were asked in reverse chronological order.

• Condition E allowed respondents to list all postsecondary degrees they have in any order they chose and then asked follow-up questions about each degree in the order of the degrees listed.

The summary of findings for the methodological study is included in Attachment 1.

Proposed Changes

Field of Study Questions

We do not propose making any changes to the FOS collection method because the methodological study showed that the current SED FOS coding method (Condition A) resulted in higher respondent self-coding rate than the Condition B. However, for 2021 SED and going forward, we propose to collect FOS data using the Classification of Instructional Programs (CIP) codes. Currently, SED allows the use of 334 FOS codes for degree fields, which are somewhat specialized and limited. Using the more detailed CIP code frame will improve the FOS data for the doctorate and dissertation research fields as well as other postsecondary degree fields. It also will help to completely align the SED FOS taxonomy with the NCSES’ Taxonomy of Disciplines.

Educational History Questions

The methodological study showed that the education history collection method used in Condition E resulted in better degree data with less response burden than Conditions C or D, which asked about each type of degree in sequence.

Therefore, we propose changing the educational history questions as tested in Condition E, which asked the respondents to list all the postsecondary degrees they have earned in any order they chose, and then asked follow-up questions about each degree in the order of the degrees listed. If the respondents’ degree list is missing any degree type (associate’s, bachelor’s, or master’s), they will be specifically asked to verify that they did not earn that degree type.

Justification for Changes

Using CIP codes for SED FOS questions

The FOS verbatim strings reported by the respondents for degree fields they were unable to self-code in the survey are manually reviewed by the SED expert coders to assign the appropriate SED FOS codes. In the 2019 SED, about 40% of the FOS verbatim strings were difficult even for the SED expert coders to code because the current list of 334 SED FOS codes did not provide sufficient detail fields to capture the respondents’ degree fields, particularly their associate’s or bachelor’s degrees. The CIP, which has more than 2,000 fields, provides a finer level of degree field detail for the respondents to select from.

To evaluate the impact of the FOS coding frame change from the SED fields of study to the 2020 CIP on the SED trend data, we plan to add the SED fields not included the 2020 CIP list that we plan use in the 2021 SED FOS collection and create a crosswalk between the SED and CIP codes. This will help to bridge the SED degree field trend data changes.

Using Degree Listing Method for Educational History

In the qualitative feedback from respondents after they completed the educational history questions, some of those who responded to Conditions C and D spontaneously offered suggestions for question improvement similar to the Condition E format. One respondent explained, “It would have been better if I could list all of the degrees and information on one page instead of having to click "next" several times.”

These quantitative and qualitative results suggest that Condition E is a better method than the other two conditions. Condition E collected the same amount of degree data from the respondents in significantly less time. As shown in Attachment 1, Condition D and E collected fewer associate’s degrees than Condition C. However, the manual review of the associate’s degree data in the current SED (which uses a format very similar to Condition C) showed that associate’s degrees are overreported due to respondents’ misunderstanding of what an associate’s degree is. Because this degree type is asked about first, some respondents report their first postsecondary degree, even if it is not an associate’s degree. Compared to condition C, conditions D and E collected more accurate associate’s degree data as they asked respondents to recall and report their degrees in reverse chronological order (D) or all at once (E).

To reduce the risk that any degrees are mis- or under-reported in the 2021 SED, we will include three follow up questions after the degree list to prompt respondents to report all of their degrees. When a respondent does not list any associate’s, bachelor’s, or master’s degree, a verification question about the missing degree type will be asked. These additional questions will help distinguish item nonresponse from the legitimate absence of a degree.

In summary, the proposed changes in the 2021 SED questions may result in changes to the trend data but they will improve the educational data quality by allowing respondents to choose from a more detailed field of study list, and enabling the collection of the respondents’ educational history data with reduced respondent burden. Although we expect the respondent burden to decrease with these changes, NCSES will keep the total burden estimates the same. The number of estimated respondents will also remain the same.

2021 SED Data Collection Schedule

Date	Activity/Deliverable
04/24/2020	OMB submission for approval of non-substantive changes
05/08/2020	OMB approval
05/13/2020	Finalize 2021 web survey instrument
05/15/2020	Launch 2021 survey collection for AY2021 graduates (July 2020–June 2021)
12/30/2021	2021 survey closeout
06/01/2021	2021 Doctorate Records File delivery
11/31/2021	2021 SED publication release

Contact Person

Kelly Kang

Project Officer

Survey of Earned Doctorates

Human Resources Statistics Program

National Center for Science and Engineering Statistics

National Science Foundation

[email protected]

703-292-7796

Attachments

Attachment 1: Summary of SED Education Questions Methodological Study Results

Attachment 2: Current 2020 SED Part A2 Educational History Questions

Attachment 3: Proposed 2021 SED Part A2 Educational History Questions

Attachment 1

Summary of SED Education Questions Methodological Study Results

A methodological study was conducted to determine the optimal way to collect field of study (FOS) information and educational histories from respondents. The study was conducted in two phases between December 2019 and January 2020 and included both quantitative analysis of missing and un-coded question items, and response time, along with an analysis of qualitative feedback from study participants.

The first phase test of the study consisted of a set of 1,146 participants with graduate degrees, recruited through Amazon’s MTurk, and the second phase consisted of a set of 56 doctoral students and doctorate holders recruited at different colleges and universities.

The MTurk participants completed the web survey questions followed by a series of debriefing questions. The 56 doctoral students or doctorate recipients participated in cognitive interviews conducted by the SED survey contractor’s research staff either in person or via web conference. All study participants held at least a master’s degree, with approximately 15 percent held a doctorate.

Each participant in the methodological testing was randomly assigned to one of two conditions to examine different methods of collecting FOS information.

• Condition A was the control and was similar to the current SED. The response option of the FOS question was an interactive textbox that recommended possible FOS codes as participants entered their FOS. Participants could select a satisfactory FOS code from the recommendations or click a checkbox to activate another open-ended textbox to enter their preferred description of their FOS.

• Condition B provided participants with an open-ended text box and participants entered in their field. The instrument then matched the verbatim response to a database that included all codes on the SED FOS code frame plus a set of common spelling errors of the FOS items.

Participants in Condition A produced significantly fewer uncoded FOS responses (Table 1) and completed the questions about their doctoral degree and dissertation research faster than their counterparts in Condition B (Table 2). Some respondents also expressed an expectation of an interactive FOS field search process within the survey instrument when they were asked about their experience. For these reasons, we plan to keep the current FOS coding method.

Table 1. Percent of Uncoded Doctorate and Dissertation Fields by Test Condition

	Condition A		Condition B
	N	% Uncoded	n	% Uncoded
Doctorate Field	178	25.8***	188	45.7
Dissertation Field	136	25.7***	144	58.3

*** P < .001 (t-test)

Note: Uncoded includes missing responses.

Table 2. Response Time for Doctorate and Dissertation Questions by Test Condition

	Condition A		Condition B
	n	Minutes	n	Minutes
Doctorate/Dissertation Questions	178	5.4*	188	5.6

*P < .05 (t-test)

To test the different methods for collecting educational history data, each study participant was randomly assigned to one of three conditions to examine different methods of collecting educational history data:

• Condition C was the control group, which asked the respondents to report their other postsecondary degrees as in the current SED in the following order: associate’s degree, bachelor’s degree, master’s degree, and professional doctorate. At the associate’s, bachelor’s, and master’s degree levels, the survey asked about the number of degrees earned, and for multiple degrees of same degree type, question about the most recent degree, followed by the first degree.

• Condition D asked respondents to report other postsecondary degrees in the reverse order: professional doctorate, master’s degree, bachelor’s degree, and associate’s degree. As in Condition C, if respondent reported multiple degrees of same type, the survey first asked about the most recent degree, followed by the first degree.

• Condition E asked respondent to list all postsecondary degrees they have earned in any order. The survey asked follow-up questions about each degree in the order the degrees were listed. This method is similar to that used by the Early Career Doctorates Survey (OMB control number 3145-0235). The only difference is that if the respondent did not list a bachelor’s or master’s degree, they were asked if they had earned one in order to distinguish between item nonresponse and legitimate absence of a degree.

Respondents in each of the conditions reported similar numbers of bachelor’s and master’s degrees, while those assigned to Condition C reported a slightly higher number of associate’s degrees (p < .05; Table 3). There was no statistical difference in the number of bachelor’s and master’s degrees reported across all conditions. The Condition E questionnaire included prompts for the respondents who did not report a bachelor’s or master’s degree to confirm they did not have the degree, but this prompt was not included for a missing associate’s degree. This may explain the slightly lower number of associate’s degrees collected in Condition E as compared to the other two test conditions. We will add in a prompt for the respondents who did not report an associate’s degree in the 2021 SED.

Table 3. Average Number of Reported Degrees by Degree Type and Test Condition

Degree Type	Condition C (n=387)	Condition D (n=389)	Condition E (n=370)
Associate's	0.25	0.16*	0.14*
Bachelor's	1.04	1.02	1.05
Master’s	1.02	1.03	1.06

*p < .05 (t-test)

Larger differences were observed among the conditions in response time. While there was no significant difference between the response time in Conditions C and D, the response time in Condition E was significantly less than Condition C and D (p < .001; Table 4). On average, respondents in conditions C and D spent almost 5 minutes, while those in Condition E spent only about 2.5 minutes to answer all educational history questions.

Table 4. Response Time in Educational History Section by Test Condition

	Condition C	Condition D	Condition E
N	387	389	370
Meana (minutes)	4.9	4.8	2.6***
Median (minutes)	4.0	3.7	2.1
SD	3.8	4.0	1.9

***p < .001 (t-test)

Attachment 2

Current 2020 SED Part A2 Educational History Questions

Attachment 3

Proposed 2021 SED Part A2 Educational History Questions

In what month and year did you first enter any graduate program, even if you did not earn a degree?

Graduate program first entered: MONTH YEAR

Please indicate the type of professional degree that was awarded in 2019.

I n addition to your doctorate, are you currently earning a professional doctoral degree, such as an MD, DDS, DVM, JD, or PsyD?