Testing of the 2021 SED Field of Study and Educational History Data Collection Methods
Educational history data for research doctorate recipients is a rich and valuable source of information for federal agencies, policymakers, and researchers. However, collecting these data accurately and reliably is a complex undertaking and requires considerable amount of input from respondents. To improve data quality and reduce response burden, NCSES conducted a methodological study to test alternative methods for collecting respondents’ field of study (FOS) and educational history data.
The study was conducted in two phases between December 2019 and January 2020, and included a quantitative analysis of missing and un-coded question items, response time, and an analysis of qualitative feedback from study participants. The study tested two versions of the field of study (FOS) questions and three versions of the educational history collection methods. The first phase test of the study consisted of 1,146 participants with graduate degrees recruited through Amazon’s MTurk to complete a web survey online. The second phase consisted of 56 doctoral students and doctorate holders recruited from different colleges and universities to participate in cognitive interviews. All study participants were randomly assigned to one of the six possible combinations of the FOS and education history questions.
Results indicated that the current SED version of the FOS data collection worked better than the proposed alternative, and the participants who listed their education degrees on a single page answered the questions more quickly while reporting just as many and detailed postsecondary degree information as those in other versions. The FOS data collection based on the Classification of Instructional Program (CIP) code frame, and educational history list method were implemented in the 2021 SED web instrument.
Testing of the 2021 SED COVID-19 Pandemic Question Module
The novel coronavirus pandemic that began in the U.S. around March 2020 disrupted many aspects of higher education and began to present multiple challenges to the doctoral students in their path to doctorate completion. NCSES quickly developed a new question module for the 2021 SED to measure the impacts of the pandemic on doctoral students’ graduate experience and post-graduation career plans. The module included seven filter questions about effects students may have experienced and follow up questions to collect more detailed information about their experience.
Cognitive interviews were conducted with 9 doctoral students in May 2020 to test whether the pandemic impact questions and response categories were understood by respondents and whether they captured the range of respondents’ experiences. Test participants were recruited from doctoral students who had expressed an interest in participating in the 2020 SED FOS and Educational History Modules (described above).
The cognitive interviews revealed that participants were able to understand the seven filter questions about the impacts of pandemic on graduate experience and career plans. The only exception was the questions about changes in their career plans or goals, where participants were unclear in reference to the time frame. Based on the test results, the question about changes in career plans was modified to include the phrase “longer term,” which clarified the reference period. These questions were included in the 2021 SED, which launched in June 2020.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Methodological Research Concerning the Survey of Earned Doctorates |
| Author | mason-admin |
| File Modified | 0000-00-00 |
| File Created | 2021-02-06 |