SDR 2021 OMB Supporting Statement Part A (5_24_2021) final

This request is for a three-year renewal of the previously approved OMB clearance for the Survey of Doctorate Recipients (SDR). The SDR has historically served as a valuable source of information on U.S.-trained science, engineering, and health doctorate recipients. The SDR was last conducted in 2019 under the following terms of clearance: Approved consistent with the understanding that the design continues to support biennial cross-sectional estimation (including collection for the 2019 reporting period) as well as establishing the baseline for a new longitudinal component. In the period between the 2019 data collection and the planned 2021 data collection, a follow up instrument will be designed for the longitudinal component. The OMB clearance for the 2019 SDR expires August 31, 2022 (OMB No 3145-0020).

The sample size of 125,938 in the 2021 cycle is slightly larger than the prior round of 120,000 cases. This larger sample size reflects a decision to retain all 115,937 eligible returning sample members from the prior 2019 cycle along with including 10,001 newly sampled graduates¹. The decision to retain all eligible returning sample cases from the prior cycle is in light of lower than expected response rates and a larger than expected number of ineligible sample members from the 2017 supplemental sample of 14,564 SEH doctoral degree holders.

In addition, with respect to the 2019 OMB terms of clearance and other considerations, NCSES reviewed the 2019 survey protocol and conducted methodological research studies in the fall of 2020 resulting in the following changes to the 2021 SDR operations:

Based on the results of a quantitative pilot study in the fall of 2020, NCSES will modify the electronic data collection instrument for web and CATI responses for sample members who had responded in prior SDR cycles since 2015. Specifically, in the 2021 SDR, NCSES will use a dependent interviewing approach that prefills sample members' employment information from their most recent cycle of participation among those returning eligible sample members who were working at that time. Newly eligible sample members who earned their doctorate degree since the last SDR and returning sample members who were not employed in their last cycle of participation will receive a data collection instrument without prefills similar to the 2019 SDR questionnaire. Using prefilled employment information facilitates longitudinal data when changes in employment characteristics occur at the individual level (see section B4 and Appendix G for detail).
In alignment with the National Survey of College Graduates, the 2021 SDR will include modifications to several questions’ response categories to collect data that speaks to the effect of the coronavirus pandemic on the sample members’ employment situation. (See section B4 and Appendix D1 for detail)
NCSES will include an incentive experiment in 2021 to assess the use and amount of early-stage monetary incentives among sampled cases with the goal of reducing the overall number and dollar amount of incentives distributed compared to 2019. In 2021, NCSES will also change from using checks as the monetary incentive to using prepaid debit cards which allow more accurate tracking of usage relative to distribution and survey participation. (See section A9 below and B3.4 for more detail)
Additionally, based on the results of a fall 2020 locating feasibility study that tested modified locating procedures, NCSES may initiate a between-cycle contact strategy after the 2021 cycle to update locating information in an effort to increase their participation in the 2023 SDR survey cycle. During the fall of 2022, NCSES may contact up to 30% of the sampled cases eligible for data collection in the 2023 SDR to request their voluntary cooperation in completing an online form that updates their contact information. The feasibility study tested this procedure on a sample of 1,000 cases, 300 of whom had not responded in the 2019 SDR, and 100 of whom were believed to be living outside of the U.S. Overall, 37% of the sample members responded to the request to update their contact information, including nearly 20% of the 2019 nonrespondents and almost 50% of the sample members living abroad. Based on that experience, any between-cycle contacts will focus on nonrespondents from the prior cycle with whom no response to any contact occurred, and thus the final disposition was ’unknown eligibility.’ If the typical batch tracing services that are conducted every cycle provide updated mailing information for any of these sample members, they will be dropped from this mailing.
A non-production bridge panel with a sample size of approximately 5,000 cases will be fielded in parallel to the production sample. This bridge panel provides a mechanism for NCSES to explore modifications to the SDR core survey questions and quantify the potential impact on key survey estimates (see details in Section B 4).

NECESSITY FOR INFORMATION COLLECTION

Established within the NSF by the America COMPETES Reauthorization Act of 2010² § 505, codified in the National Science Foundation Act of 1950, as amended, the National Center for Science and Engineering Statistics (NCSES) serves as a central Federal clearinghouse for the collection, interpretation, analysis, and dissemination of objective data on science, engineering, technology, and research and development for use by practitioners, researchers, policymakers, and the public. Information obtained through the SDR is critically important to NCSES’s ability to measure the education and employment outcomes of scientists and engineers. Furthermore, the SDR and NCSES’s National Survey of College Graduates (NSCG) are coordinated in both survey content and timing to form data collections that serve as the nation’s only source of comprehensive information about the size and characteristics of the science and engineering (S&E) workforce.³ These data are solicited under the authority of the NSF Act of 1950⁴, as amended, and are central to the analysis presented in a pair of congressionally mandated reports⁵^,⁶ published by NSF:

Science and Engineering Indicators
Women, Minorities, and Persons with Disabilities in Science and Engineering.

In addition, the Science and Engineering Equal Opportunities Act of 1980 directs NSF to provide to Congress and the Executive Branch an “accounting and comparison by sex, race, and ethnic group and by discipline, of the participation of women and men in scientific and engineering positions.”⁷ The SDR and NSCG provide much of the information to meet this mandate. The coordinated designs of these two surveys were developed throughout the past two decades and are based on recommendations from the National Research Council’s Committee on National Statistics (CNSTAT) report to NSF.⁸

SDR Background

The SDR provides education and employment-related information on scientists and engineers who were awarded a research doctoral degree from a U.S. institution in a science, engineering, or health (SEH) field. A research doctorate is a doctoral degree that (1) requires completion of an original intellectual contribution in the form of a dissertation or an equivalent culminating project (e.g., a published manuscript) and (2) is not primarily intended as a degree for the practice of a profession.

The 2021 SDR is comprised of two sample components:

A continuing panel of doctorate recipients from the 2019 survey cycle who remain eligible and are under 76 years of age.
A new cohort component that adds newly eligible doctorate recipients from academic years 2018 and 2019, also under 76 years of age.

The panel portion of the SDR sample provides information on the experienced stock of doctorate recipients. The new cohort sample from the two most recent doctorate award years provides important data on the early career experiences of new doctorate recipients with SEH degrees.

Since 2010, the SDR had included an international component of U.S.-trained doctorate recipients who received their degrees as of 2001 (i.e., 21^st century graduates only). The more recently redesigned 2015 SDR cycle used the cumulative Doctorate Records File, the ongoing SDR sampling frame, to refresh the 2015 SDR sample and significantly increase its size and target population representation. The redrawn sample expanded the international component to be representative of all academic years dating back to 1961. As with the three prior cycles, the 2021 SDR will field a sample of SEH doctorate holders predicted to reside either in or outside of the U.S. on the survey reference date of 1 February 2021. For example, based on the data from the 2018 and 2019 Survey of Earned Doctorates (SED), which is the frame for the 2021 SDR new cohort, 35% of the SEH doctorates were awarded to temporary visa holders and 22% of them planned to leave the U.S. upon graduation.⁹ Thus, the 2021 SDR will yield information about the educational, employment, and demographic characteristics of U.S.-trained SEH doctorate recipients living and working both in the U.S. and abroad.

In addition, the 2015 SDR sample size expansion from approximately 45,000 to 120,000 was designed to support estimation of employment outcomes for a larger number of fine fields of degree (FFOD) captured in the SED. This large sample size will continue to be maintained for the 2021 SDR cycle. A total of 115,937 of 120,000 sampled cases in the 2019 survey remain eligible for the 2021 cycle and will be increased by 10,001 new sample members from the 2018 and 2019 SED cohorts for a total sample size of 125,938 cases.

Originally, the SDR was designed to produce employment outcome estimates for various analytical domains defined by broad aggregated fields of degree and demographic characteristics. The objective of the 2015 sample expansion and the more recent 2019 sample design modification was to meet new FFOD estimation goals while maintaining the estimation goals for the traditional (historic) key analytic domains.

In 2019, NCSES modified its fine field of degree stratification variable based on results from 2015 and 2017 estimation capabilities. The modified detailed field of study sampling stratification variable for the SDR better aligns with the NCSES Taxonomy of Disciplines (TOD). The NCSES TOD more closely aligns with the National Center for Education Statistics (NCES) Classification of Instructional Programs (CIP). This revised stratification supports a more stable sampling design and more reliable estimation capability in subsequent cycles of the SDR. As with the 2019 SDR, the 2021 SDR stratifies the sample frame by 77 detailed fields rather than the more than 200 FFODs used in the 2015 and 2017 SDR cycles.

At the conclusion of the 2021 SDR survey cycle, all sampled respondents will be used to develop cross-sectional estimates describing the U.S.-trained SEH doctorate recipient population. In addition, the respondents from the 40,000-case sample identified in the 2019 SDR as the SDR longitudinal sample will be the basis for deriving longitudinal estimates from these data. The 40,000 longitudinal sample cases represent the 2015 SDR target population moving forward and will be weighted and maintained through the 2025 cycle of the SDR to provide longitudinal data for the 10-year period 2015 to 2025. The 2021 SDR longitudinal sample will provide longitudinal data over the 6-year period of 2015-2021.

USES OF INFORMATION

The data from the SDR has historically provided valuable cross-sectional trend data on careers, training, and educational development of the nation’s SEH doctoral-level personnel resources. These data enable government agencies to assess the scientific and engineering resources available in the U.S. to business, industry, and academia, and provide a basis for the formulation of the nation's S&E workforce policies. For example, educational institutions can use the SDR data in establishing and modifying scientific and technical curricula, while various industries can use the information to develop recruitment and remuneration policies.

Policymakers, researchers, and others use information from the SDR and its complimentary NSCG to answer questions about the number, employment, education, and characteristics of the S&E workforce. These surveys provide nationally representative data that policymakers and researchers use to address questions on topics such as: employment of foreign-born or foreign-degreed scientists and engineers, the transition from higher education to the workforce, the role and importance of postdocs as research personnel, diversity in both education and employment, the implications of an aging cohort of scientists and engineers as baby boomers reach retirement age, and information on long-term trends in the S&E workforce.

Findings from the 2021 SDR will enable NCSES to continue reporting employment patterns of recent SEH doctorate recipients, as well as more experienced doctorate recipients in the labor market. The expanded sample enables NCSES to produce reliable estimates of employment outcomes by at least 77 detailed fields of degrees listed in the NCSES TOD. The SDR data are made available through published reports, the Scientists and Engineers Statistical Data System (SESTAT), public use data files, and licenses for restricted-use data files. The online data tool, available at https://ncsesdata.nsf.gov/sestat/sestat.html, allows users to create customized data tabulations in a user-specified subject area using the public use file. In addition, the SDR public-use files are available for download through the NCSES data downloads web page at https://ncsesdata.nsf.gov/datadownload/.

In addition, the 2021 survey cycle will continue with the longitudinal component of the SDR target population and its sample first defined and selected in the 2019 data collection. The 40,000-case longitudinal sample provides the ability to derive longitudinal estimates from the 2015 SDR target population on a variety of policy-relevant topics. SDR longitudinal data can be used to address important issues such as changes in employer, occupation, field of research, international mobility, income over time and retirement. Within the academic sector, SDR longitudinal data can address issues such as length of post-doctoral training, achieving tenure, and changes in location especially for married SEH doctorate holding couples. As noted previously, the longitudinal sample will represent employment and other outcomes for the 2015 SDR population through the year 2025 based on 6 cycles of SDR data.

Uses for Policy Discussion

Data from NCSES’s surveys are used in policy discussions of the executive and legislative branches of Government, the National Science Board, NSF management, the National Academy of Sciences, Engineering, and Medicine, professional associations, and other private and public organizations. Some recent examples of the current use of the SDR data, and the combined SDR and NSCG data, are as follows:

The National Science Board (NSB) used combined SDR and NSCG data in its investigation to develop national policies for the S&E workforce¹⁰;
The Committee for Equal Opportunity in Science and Engineering (CEOSE), an advisory committee to NSF and other government agencies, established under 42 U.S.C. §1885c, has been charged by the U.S. Congress with advising NSF in assuring that all individuals are empowered and enabled to participate fully in science, mathematics, engineering and technology. Every two years CEOSE prepares a congressionally mandated report that makes extensive use of NSCG and SDR data to highlight key areas of concerns relating to students, educators, and technical professionals;
A chapter in the book The Science and Technology Labor Force: The Value of Doctorate Holders and Development of Professional Careers used SDR data to describe the S&E labor force in the U.S.¹¹
The importance of information on the S&E workforce to inform public policy can be seen in discussions of the NSB’s Task Group on Science, Technology, Engineering, and Math (STEM) Innovators. The task group used NSCG and SDR data to inform its deliberations about the S&E workforce and these data were part of the final report¹²;

Information from the SDR was presented at the Organisation for Economic Co-operation and Development (OECD) conference in December 2012, “Understanding and improving the contribution of doctoral graduates to innovation and the economy: Developing the statistical evidence”¹³; and
The Educational Testing Service (ETS) and Council of Graduate Schools (CGS) used SDR and NSCG data to examine national benchmarks for career outcomes of master’s and doctoral degree recipients by specific field.¹⁴

Uses by NSF and NCSES

The SDR data were used extensively in the latest versions of the congressionally mandated biennial reports: Science and Engineering Indicators, 2020 and Women, Minorities and Persons with Disabilities in Science and Engineering, 2021.

In addition, NCSES used the SDR data and the combined SDR and NSCG data in recent reports such as:

Science and Engineering State Profiles, Fall update, November 4, 2020.
Number of Women with U.S. Doctorates in Science, Engineering, or Health Employed in the United States More Than Doubles since 1997, NSF 19-307, February 12, 2019.
The 2015 Survey of Doctorate Recipients Expands Its Population Coverage and Reporting on Field of Study, NSF 17-319, June 28, 2017.
Data Tables: Survey of Doctorate Recipients, 2019, April 2021.
Data Tables: Survey of Doctorate Recipients, 2017, February 2019.

All NCSES publications can be accessed on the NCSES website at http://www.nsf.gov/statistics.

Uses by Researchers and Analysts

SDR and the combined SDR and NSCG data are presented at conferences and professional meetings by NCSES staff and survey contractor staff, such as the annual meeting of the Association for Institutional Research, the American Association for Public Opinion Research, the American Educational Research Association, and the Joint Statistical Meetings. Examples of these presentations are as follows:

A Novel Approach to Combine Survey and Bibliometric Data for Science Policy Research, BigSurv2020, November 2020.
Publications Output: Trends, Data, and Research, NIH Bibliometrics and Research Assessment Symposium 2020, October 2020.
Demographic Differences in the Publication Output of U.S. Doctorate Recipients, Atlanta Conference on Science and Innovation Policy, October 2019.
Demographic Differences in the Publication Output of U.S. Doctorate Recipients, International Conference on Scientometrics & Informetrics, September 2019.
Advancing Survey Data for Evidence-based Research and Evaluation, IRIS Summit, September 2019.
Managing Locating and Data Collection Interventions through Adaptive Survey Design, American Association for Public Opinion Research, May 2019.
Where in the world? How in the world? The Challenges of Collecting Data around the Globe, American Association for Public Opinion Research, May 2019.
Using Contacting Information to Derive Employer Name in the Survey of Doctorate Recipients, American Association for Public Opinion Research, May 2019.
Exploring Alternative Measures of Doctoral Underemployment, Society for Longitudinal & Lifecourse Studies International Conference, July 2018,
The problem of analytic error in secondary analysis of survey data: What we know, and what we need to do about it, Duke Initiative on Survey Methodology, June 2018,
Balancing cross-sectional and longitudinal design objectives for the Survey of Doctorate Recipients, Federal Committee on Statistical Methodology Research and Policy Conference, March 2018.

Since 2009, NCSES has distributed more than 6,500 copies of SDR public-use files (2003, 2006, 2008, 2010, 2013, 2015, and 2017 survey cycles), as well as over 6,945 copies of the combined SDR and NSCG data’s public-use files (1993-2013 survey cycles). There are currently 49 active SDR restricted-use licenses.

Selected recent citations by researchers using SDR data are as follows:

Jiang, X., W.Y. Chang, and B.A. Weinberg. (2021). “Man versus machine? Self-reports versus algorithmic measurement of publications.” National Bureau of Economic Research: Working Paper 28431 http://www.nber.org/papers/w28431
Eagly, A. H. (2020), “Do the social roles that women and men occupy in science allow equal access to publication?” Proc Natl Acad Sci U S A, DOI: 10.1073/pnas.2001684117

Khosla, P. (2018). "Wait time for permanent residency and the retention of immigrant doctoral recipients in the U.S." Economic Analysis and Policy 57: 33-43.
Kniffin, K. M. and A. S. Hanks (2018). "The trade-offs of teamwork among STEM doctoral graduates." American Psychologist 73(4): 420-432.
Webber, K. L. and M. González Canché (2018). "Is There a Gendered Path to Tenure? A Multi-State Approach to Examine the Academic Trajectories of U.S. Doctoral Recipients in the Sciences." Research in Higher Education, DOI: 10.1007/s11162-018-9492-4.
Tao, Y. (2018). "Earnings of Academic Scientists and Engineers: Intersectionality of Gender and Race/Ethnicity Effects." American Behavioral Scientist 62(5): 625-644.
Torche, F. (2018). "Intergenerational Mobility at the Top of the Educational Distribution." Sociology of Education, DOI: 10.1177/0038040718801812.
Perez-Silva, R., M. D. Partridge and W. E. Foster (2018). "Are foreign‑born researchers more innovative? Self‑selection and the production of knowledge among PhD recipients in the USA." Journal of Geographical Systems, DOI: 10.1007/s10109-018-0281-6.
Meyers, L. C., A. M. Brown, L. Moneta-Koehler and R. Chalkley (2018). "Survey of checkpoints along the pathway to diverse biomedical research faculty." PLoS One 13(1): e0190606.
Bucks, B. and M. P. Couper (2018). "The Fine Print: The Effect of Legal/Regulatory Language on Mail Survey Response." Survey Practice 11(2).
Cohen, W. M., H. Sauermann and P. Stephan (2018). “Academics' Motives, Opportunity Costs and Commercial Activities Across Fields.” Working Paper 24769. National Bureau of Economic Research.
Cummings, W. K. and O. Bain (2018). “US Doctoral Study to Early Career.” Doctoral Education for the Knowledge Society. J. C. Shin et al. editors, Springer International Publishing AG: 91-103.
Kahn, S. and M. MacGarvie (2018). “Immigration Policy and Stay Rates of STEM PhDs.” Proceedings of the 23rd International Conference on Science and Technology Indicators, Leiden, The Netherlands, September 12-14, 2018.
Kahn, S. and M. MacGarvie (2018). “The Impact of Permanent Residency Delays for STEM PhDS: Who Leaves and Why.” NBER Working Paper Series No. 25175, National Bureau of Economic Research.
Kawa, N. C., J. A. Clavijo Michelangeli, J. L. Clark, D. Ginsberg and C. McCarty (2018). "The Social Network of US Academic Anthropology and Its Inequalities." American Anthropologist, DOI: 10.1111/aman.13158.

CONSIDERATION OF USING IMPROVED TECHNOLOGY

As with earlier cycles, the 2021 SDR will collect data using three modes:

Self-administered online (or web) surveys (including access from mobile devices);
Paper self-administered questionnaires (mail); and
Computer-assisted telephone interviews (CATI).

The data collection effort will emphasize response by web, use a mobile optimized web instrument, incorporate adaptive design techniques, use optical scanning of paper questionnaires, and store contact history information in a case management system. These systems are described below.

Participation via the online response option has increased steadily since the 2006 cycle, from 47% in 2006 to 93% in 2019. In 2019, in addition to the online response option, 6% responded by paper questionnaire and 1% responded through CATI. Analysis indicates that the online web and CATI modes result in more complete and concise survey data and follow-up contacting information compared to the mail questionnaire mode. As with the 2019 survey protocol, for 2021, nearly all sample members initially will be asked to participate via the web form with exception for those known to have vision problems, no internet access, or other circumstances that warrant their participation by other modes. Mailed letters will include the SDR URL and the sample members Personal Identification Number (PIN). Email invitations will include a live link to the starting page of the 2021 SDR.

As in 2019, the 2021 online survey will be configured for use on mobile devices (e.g., smartphones and tablets) to ensure that the respondent experience is optimized regardless of the screen size or browser used to access the survey. The 2019 SDR obtained about 11% of the online responses via submission by a mobile device, up about three percentage points from 2017 (8%), and we expect a similar, if not higher, percentage of online submissions via a mobile device in 2021.

The 2021 SDR data collection effort will continue the use of a comprehensive case management system (CMS) to track data captured across the three modes (web, mail, CATI) with additional features added to the system. The additional features support an adaptive design data collection strategy for quickly prioritizing cases and real-time response rate calculations for Primary Analytic Domains (PADs) such as location of residency, employment sector and demographic characteristics of the sampled cases. Additionally, the CMS will include methods to monitor in near real time incentive use by response status, which will allow follow-up contacts tailored to incentive receipt and usage. Optical Character Recognition (OCR) and Optical Mark Recognition (OMR) technology was used to capture mail questionnaire data for the first time in 2019 and will continue in 2021. Scanned images will be stored in a secured database for archival purposes.

4. EFFORTS TO IDENTIFY DUPLICATION

Overlap exists between the SDR and the NSCG in their target population and in survey content. The NSCG and the SDR both capture estimates of the roughly 1 million U.S.-trained SEH doctorates residing in the U.S., though the NSCG also covers an estimated 165,000 employed foreign-trained SEH doctorates residing in the United States. However, unlike the NSCG, the SDR collects detailed information from U.S.-trained SEH doctorate recipients working in post-secondary academic institutions including their academic position, faculty rank, tenure status, and reasons for taking a postdoc if in a postdoctoral position. In contrast, and unlike the SDR, the NSCG collects information on all college graduates regarding veteran status, attainment of certifications and licenses, financial support for education, and community college enrollment history.

Based on the 2019 SDR sample, approximately 400 individuals may be selected for both the 2021 NSCG and the 2021 SDR. Due to the content differences between the surveys, the relatively small number of expected duplicates, and the operational challenges of the deduplication process, NCSES will not de-duplicate individuals selected for participation in both the SDR and NSCG.

EFFORTS TO MINIMIZE BURDEN ON SMALL BUSINESS

Not applicable. The SDR collects information from individuals only.

CONSEQUENCES OF LESS FREQUENT DATA COLLECTION

The SDR data are central to the analysis presented in a pair of congressionally mandated reports - Science and Engineering Indicators and Women, Minorities, and Persons with Disabilities in Science and Engineering. Since these reports are published on a biennial schedule, they rely on the availability of updated data on the S&E workforce every two years. Conducting the SDR on a less frequent basis would prohibit NCSES from meeting its congressional mandate to produce these reports that contain an accurate accounting and comparison, by sex, race, and ethnic group and by discipline, of the participation of women and men in scientific and engineering occupations. The impact of not being able to meet this congressional mandate is that government, business, industry, and universities would have less recent data to use as a basis for formulating the nation’s S&E policies.

Less frequent data collection would also impact SDR data quality and value. Follow-up surveys every two years enable both time-series and longitudinal analyses of changes in the S&E workforce especially with respect to employment in and out of S&E occupations and the growth of these occupations.

Because the SDR has been coordinated with the NSCG on a biennial basis since 1993 to ensure the availability of current national S&E workforce data, degradation of either component jeopardizes their integrity and value. These combined surveys provide comprehensive information on the S&E workforce. Conducting either survey less frequently would make it more difficult and costly to locate sampled persons given their mobility and the impact is likely to be higher attrition rate, higher potential for nonresponse bias, and less reliable cross-sectional and longitudinal estimates.

SPECIAL CIRCUMSTANCES

Not applicable. This data collection does not require any one of the reporting requirements listed.

8. FEDERAL REGISTER ANNOUNCEMENT AND CONSULTATIONS OUTSIDE THE AGENCY

Federal Register Announcement

In response to the Federal Register Notice for the SDR that appeared on February 4, 2021 [FR Doc. 2021–02450 Filed 2–4–21, Vol 86, Number 23], two public comments were received by the closing date of April 6, 2021. See Appendix C for the announcement.

On 5 February 2021, Dr. Andrew Reamer of George Washington University sent an email to NSF on behalf of the American Economic Association’s Committee on Economic Statistics. He requested the draft information collection request (ICR) materials for the 2021 SDR and asked whether any changes were proposed for the 2021 SDR compared to the 2019 SDR. NSF responded to Dr. Reamer on 2 February 2021, explaining that the 2021 SDR ICR materials were in the process of being prepared and that there were no substantive changes planned. He was directed to the 2019 SDR questionnaires on the NSF website, which would be updated to reflect the survey year. He was also informed that the same instrument, with updates to reflect survey year and modifications to accommodate the circumstances of the coronavirus pandemic, will be used for the 2021 cycle.

NCSES received a comment on 17 March 2021 from Dr. Jon Freeman representing the American Association for the Advancement of Science (AAAS) and the American Educational Research Association (AERA). The commenters requested that NCSES include measures of sexual orientation and gender identity on the SDR and on other NCSES surveys (specifically, the National Survey of College Graduates and the Survey of Earned Doctorates). NCSES informed the commenters that it continues to actively engage on the Federal Committee on Statistical Methodology’s (FCSM) Working Group on Measuring Sexual Orientation and Gender Identity and described its research efforts for development and fielding possible questionnaire additions to address the topic. NCSES further informed the commenters that though it is conducting research to evaluate these measures, NCSES does not intend to include them in the 2021 SDR.

Consultations Outside the Agency

NCSES routinely seeks the advice and guidance of survey methodologists, statisticians, demographers, researchers, data analysts, and policymakers to examine numerous issues related to further development and continuous quality improvement of the SDR. As noted in the 2017 and 2019 SDR OMB Information Collection Requests, NCSES previously convened a Human Resources Experts Panel (HREP) meeting in October 2016 to review the redesigned 2015 SDR sample expansion with a large number of data users and stakeholders, and to collect information on their research interests that could be addressed with SDR longitudinal data. This effort was developed and pursuant to the OMB approved ICR for the 2017 SDR which stated the following terms:

Approved consistent with the understanding that the primary goal of the design is to allow cross-sectional reporting for fine fields; a secondary goal is to build an efficient, effective, and sustainable longitudinal design.

In response to these terms and in alignment with NCSES’s mission, NCSES conducted research with the goal of developing a sustainable design that will maintain the integrity of the survey's cross-sectional estimates while producing reliable data to support longitudinal analysis.

As noted above (Section 1: Background of the SDR), the 2019 SDR survey sample was redesigned to produce detailed estimates on at least 77 fine fields of study according to demographic and employment characteristics. The redesigned sample stratification allowed for the continued cross-sectional reporting of historical SDR timeseries estimates of employment outcomes while also producing more detailed outcomes on fine fields of degree in subsequent cycle of data collection.

In addition, as noted in Appendix F, NCSES conducted research on fielding prefilled web and CATI forms to facilitate an efficient and effective longitudinal design.

Federal Committee on Statistical Methodology Research Conference presentation: (March 9, 2018)

In addition to the changes in sample size noted above, NCSES had promoted the development of the SDR longitudinal design by presenting at the 2018 FCSM Research Conference. The presentation focused on the complexities of balancing cross-sectional and longitudinal goals while building an effective, efficient, and sustainable longitudinal design into the newly expanded cross-sectional SDR sample. Specifically, the presentation noted that to enhance SDR’s utility and meet dual cross-sectional and longitudinal goals, longitudinal panels within the refreshed sample need to be established formally and maintained over time. The presentation also highlighted NCSES’s on-going research efforts described above. The presentation concluded by noting that the process of developing a longitudinal design to enhance overall survey utility is a good application of a major survey redesign effort and the approaches and lessons learned from the SDR experience could be informative for other researchers involved in survey design.

PAYMENT OR GIFTS TO RESPONDENTS

Incentives for the SDR began during the 2003 cycle and have continued for all subsequent cycles. The 2021 SDR incentive plan is modeled after the approach used in SDR survey cycles since 2013, using an early- and late-stage incentive protocol. However, NCSES will include an experiment in 2021 to assess the use and amount of early-stage monetary incentives with the goal of reducing the overall number and dollar amount of incentive offerings compared to 2019. In 2021, NCSES will also change from using checks as the monetary incentive to using prepaid debit cards which allow more accurate tracking of usage relative to distribution and survey participation.

The details of the plans to offer both early and late-stage incentives are described below.

All 2021 SDR sample cases will be eligible to receive an incentive, except for the following groups:

Non-U.S. residing sample members,
Sample members determined to work for the National Science Foundation, and
Sample members with no confirmed mailing address at the time of the incentive mailing.

Proposed Incentive Plan for the 2021 SDR

Early-Stage Incentive. In 2021, early-stage incentives will accompany the second mailing scheduled for the 5^th week of data collection (i.e., the beginning of the reminder phase). NCSES will conduct an experiment varying the dollar amount of the incentive offered. The early-stage incentive will target three types of sample members, as described below, who have not completed the SDR questionnaire during the first four weeks of data collection. An early incentive will not be offered to (a) those who participated in the 2019 SDR without an incentive, (b) 2019 respondents who received an incentive in 2019 but did not cash it, nor (c) those who were hard, hostile, or congressional refusals in 2019.

New cohort sample members (i.e., individuals earning their doctoral degree in academic year 2018 or 2019) who have not previously participated in the SDR: It is expected that at the start of week 5, about 70% of the 10,000 new sample members will not have responded. These non-responding cases will be selected into one of three groups:
1. 25% will receive a preloaded debit card with the traditional SDR incentive value of $30,
2. 25% will receive a $10 preloaded debit card, and
3. 50% will not receive an early incentive (i.e., $0 as noted in figure 1).
Previously incentivized continuing cohort sample members who participated in the prior SDR and cashed their 2019 incentive check: This group represents reluctant respondents in 2019 who eventually responded to the survey, but only after receiving an incentive. Their decision to cash the incentive suggests the incentive may have been a meaningful contributor to their decision to participate. This group (expected to number about 14,400) also will participate in an experiment varying the incentive amount offered, as follows:
1. 70% will receive a $30 preloaded debit card,
2. 20% will receive a $10 preloaded debit card, and
3. 10% will not receive an early incentive.
Continuing cohort members who we attempted to incentivize in 2019 but who did not participate and were not hard, hostile, or congressional refusals: With this group (expected to be about 14,800), we do not have information that indicates the incentive was or was not a consideration in their decision not to participate. About 25% of the sample members in this group had at least started their 2019 survey but did not complete enough critical items to be counted as a complete response. Another 5% contacted our contractor and provided a reason for not responding but were not angry or uninterested in the SDR (i.e., soft refusals). About 1% were not located and likely did not receive the mailed incentive check. The remaining sample cases in this group had no contact with the contractor or NSF and were closed out as having unknown eligibility. NCSES still considers this large group to have significant likelihood to respond - higher than prior round explicit refusals, but not as high as those who responded once they received an incentive. In 2021, NCSES will conduct an experiment varying the incentive amount offered, as follows:
1. 25% will receive a $30 preloaded debit card,
2. 25% will receive a $10 preloaded debit card, and
3. 50% will not receive an early incentive.

Late-Stage Incentive. Among those who were not offered an early incentive as noted above, the overall strategy for the use of a late-stage incentive is to ensure that all sample members who remain nonrespondents in the last quarter of the field period have a probability of receiving a monetary incentive. In the plan used for the 2008 through 2019 SDR, a higher probability of selection for the incentive was given to more challenging cases in key analytic domains with relatively lower response rates. This strategy was designed to improve the accuracy of survey estimates, and ideally, mitigate nonresponse bias. In 2019, the late-stage eligible cases offered an incentive achieved an unweighted survey response rate above 50% compared with an unweighted response rate of less than 30% for late-stage incentive eligible cases not selected for the incentive.

To effectively allocate limited resources for the monetary incentive to late-stage survey nonrespondents, there will be an analysis of the characteristics of the remaining nonrespondents using pre-set targets of key analytical domains. The analysis will be done toward the end of the additional mode phase (week 16) using multiple logistic regression models to derive cooperation propensity and determine which types of nonresponding sample members should receive additional inducement to mitigate response bias. The cases with lowest response propensity and/or who are in underperforming analytical domains will be selected for the late-stage incentive provided they reside in the U.S. The number of late-stage nonresponse domestic cases to be incentivized will be based on available funds. As shown in Figure 1, the late incentive stage covers weeks 17 through 24.

Also, during the late-stage data collection phase, all pending U.S.-residing sample cases, except for those who already received a $30 early incentive, will be eligible for a late-stage incentive, with the total amount of incentive not to exceed $30. Those who received a $10 early-stage incentive will be eligible for an additional $20 incentive rather than the traditional $30 incentive. In an effort to better balance sample representativeness, NCSES will offer a late-stage incentive to approximately 9,100 sample cases, and of those, roughly 7,800 will be newly selected for an incentive (i.e., not recipients of an early incentive). The late-stage incentive rates are detailed in Figure 1 and range from 20% to 50%.

Based on this dual-stage (early and late) incentive plan, NCSES expects to offer an incentive to approximately 31,600 sample members, including roughly 20,300 from the returning panel and 3,500 from the new cohort sample in the early stage and an additional 7,800 in the late stage. In past SDR cycles, a large number of incentivized sample members did not cash their prepaid incentive check though they participated in the survey. For example, in the 2017 SDR, 24,085 sample members were offered the incentive. Of these individuals, only 10,078 cashed the incentive check (41.8%) and yet 15,114 completed the survey (62.8%). The same general pattern occurred in 2019 with more of the incentivized sample members participating (56.2%) than the number who cashed their incentive (46.7%), though a higher proportion cashed the incentive in 2019 than in prior cycles and a smaller proportion of those incentivized responded to the survey. In 2021, we are assuming that cashing behavior will more closely resemble 2019, with approximately 45% cashing their incentive at each stage.

SAQ=self-administered paper questionnaire; CIO=critical item only [questionnaire]

Incentive Costs

According to this plan, a $10 or $30 prepaid incentive will be offered to some early-stage nonrespondents for the 2021 SDR. In the late stage, those who did not receive an early incentive, will be eligible for a $30 incentive. Those who received a $10 incentive in the early stage will be eligible for a $20 incentive in the late stage. Those receiving the early incentive will have a prepaid debit card with a 6-month expiration date, and those receiving the late incentive will have a 3-month expiration date. If the debit card has not been accessed by the sample member as of the expiration date, the debit card vendor will return the full amount of the unused card to the government. The total cost of incentives in the 2019 SDR was $620,000, approximately double the incentive cost from 2017. With the experimental design planned for the 2021 SDR, incentives are expected to cost approximately $363,000.

10. ASSURANCE OF CONFIDENTIALITY

NCSES and its contractors are fully committed to protecting the confidentiality of all survey respondents. SDR data will be collected under the authority of the National Science Foundation Act of 1950, as amended, the America COMPETES Reauthorization Act of 2010, and the Confidential Information Protection and Statistical Efficiency Act (CIPSEA) of 2002. With the Federal Cybersecurity Enhancement Act of 2015, all respondent data are protected from cybersecurity risks through screening of the Federal systems that transmit the data. Cover letters and survey questionnaires to each selected respondent will advise them that the information they provide is confidential (see Appendix D – Draft 2021 SDR Questionnaire that will be the same for 2021, and Appendix E – Draft 2021 SDR Survey Mailing Materials). The same notice of confidentiality will be used in the introduction to the CATI interview and will be displayed prior to the start of the survey in the online instrument. In addition, the CATI interviewers will inform the respondents of the voluntary nature of their response and that the data will be used for statistical purposes only.

Standard data collection procedures incorporate numerous safeguards for protecting the data and must conform to a detailed security plan approved by NCSES. While collecting SDR data, the information that could identify a sample member is separated from data about that person. Each sample member is assigned a unique identifier, and this identifier is used to store identifying information (such as name, address, etc.) in a separate, secure database apart from the survey response database. SDR contractors and NCSES staff receive annual CIPSEA training to reinforce their legal obligations to protect the privacy and confidentiality of the SDR data; staff must sign data use agreements annually to acknowledge this legal obligation.

Completed SDR hard copy questionnaires and other contact materials will be housed in a secure storage room at the contractor’s production facility. Only authorized staff – and only when necessary for data collection activities – will have access to hard copy materials from the SDR file room. The contractor’s electronic systems will be on a secure local area network (LAN), and all contractor systems for storage of electronic survey data will be secure by design and will be protected by passwords available only to authorized study staff.

The contractor will implement systems to make certain that data collected via the online questionnaire are secure. First, access to the online instrument will be allowed only with a valid Personal Identification Number (PIN). The PIN is rotated to a new PIN with every respondent contact for additional privacy measures. Second, data will be transmitted by the Transport Layer Security (TLS) protocol that employs powerful encryption during transmission through the Internet. If a respondent keeps an online survey open without any activity, the online server will close the connection after a short period of inactivity, both preserving the data up to the break-off point and preventing unauthorized persons from completing the questionnaire. The online survey system will place authentication information and response data on physically separate servers behind a firewall that is not directly accessible via internet protocols, a strategy that provides an extra layer of security to protect response data. Both development and production servers will be backed up nightly as required by the contractor’s disaster recovery plan.

NCSES and its contractors will analyze and make available SDR tabulations only in aggregate form and will take all measures necessary to assure that the identity of individuals or organizations will not be disclosed in either its statistical tabulations or in the SDR public-use micro-data files.

11. JUSTIFICATION FOR SENSITIVE QUESTIONS

No questions of a sensitive nature are asked in this data collection.

12. ESTIMATE OF RESPONDENT BURDEN

For the 2021 SDR, a statistical sample of 125,938 individuals with U.S. earned doctorates in science, engineering, or health will be contacted for participation in the survey. As with prior SDR data collection cycles, the sample consists of all eligible cases from the previous cycle (115,937), as well as a sample of 10,001 new PhD graduates. For 2021, the new graduate sample received their PhD between 1 July 2017 and 30 June 2019. Across the full production sample, approximately 111,760 individuals will reside in the U.S. and 14,240 will reside abroad. The amount of time to complete the questionnaire may vary depending on an individual’s circumstances; however, on average it will take approximately 21 minutes to complete the survey based on the average response time from 2019. The 2015, 2017 and 2019 SDR data collections each resulted in an overall response rate of just under 70%. Assuming a 70% response rate (88,157 respondents) in 2021, the total survey burden for the 2021 SDR production sample is estimated to be 30,855. In addition, NCSES will field a non-production bridge sample of 5,000 SEH doctorate holders residing in the U.S. Assuming a similar response rate among the 5,000 bridge sample members (3,500 respondents), this sample will require an additional 1,225 burden hours.

Starting in late 2022, NCSES will initiate a program for between cycle locating efforts to help reduce the effort and cost associated with locating sample members at the start of each cycle. Between cycles, NCSES will attempt to contact approximately 30% of the production sample (30% x 125,938 = 37,781) to update or confirm their mail, email, and phone information. Of the approximately 30% contacted, we anticipate a 30% response rate based on a feasibility study conducted in late 2020 (30% x 37,781 = 11,334). In that study, it took on average 3 minutes to complete the request, for an estimate burden of 567 hours.

The overall burden hours for these 3 component SDR samples is 32,647 hours. Annualized over the requested clearance period of three years, the annual burden of the production survey (30,855 hours), the bridge sample (1,225 hours), and between-cycle locating (567 hours) is estimated to be 10,882 hours.

The total cost to respondents for the 32,647 burden hours is estimated to be $1,867,502. This is based on an estimated median annual salary of $119,000 per full time employed SDR respondent from the 2019 data. Assuming a 40-hour workweek over 52-weeks of employment, this annual salary corresponds to an hourly rate of $57.21.

13. COST BURDEN TO RESPONDENTS

Not applicable. This survey will not require respondents to purchase equipment, software, or to contract out services.

14. COST BURDEN TO THE FEDERAL GOVERNMENT

The total estimated cost to the Government for the 2021 SDR is approximately $13.6 million, which includes survey cycle costs, and NCSES staff costs to provide oversight and coordination with the NSCG. The estimate for survey cycle costs is approximately $13.0 million, which is based on sample size; length of questionnaire; CATI and online data collection technology; administrative overhead, design, printing, mail and telephone data collection costs; incentive payments; critical items data retrieval; data capture and editing; data quality control; imputation for missing item responses; weighting and estimating sampling error; file preparation and delivery of both cross-sectional and longitudinal files; preparation of documentation and final reports; analysis, and tabulations. The NCSES staff costs are estimated at $562,500 (based on $150,000 annual salary of 1.5 FTE for 2.5 years).

15. REASON FOR CHANGE IN BURDEN

The decrease in burden from the 2019 SDR estimate of 37,500 hours to 32,647 hours expected in 2021 is a result of the higher proportion of sample members completing the online questionnaire rather than the CATI administered questionnaire or paper form questionnaire which may take slightly longer to complete. Additionally, NCSES expects the 2021 response rate to align with the final response rate observed in 2015, 2017 and 2019 of close to 70%, five percentage points lower than that used in prior burden estimates.

16. SCHEDULE FOR INFORMATION COLLECTION AND PUBLICATION

In general, SDR data are presented as cross-tabulations of the data in reports and other data releases and continue to support time series indicators. The time schedule for 2021 data collection and publication is currently estimated as follows:

Data Collection (Mail, CATI, online) July 2021 – January 2022

Coding and Data Editing August 2021 – February 2022

Final Edited/Weighted/Imputed Data File May 2022

SDR InfoBrief May 2022

SDR Detailed Statistical Tables July 2022

SDR Public Use File July 2022

17. DISPLAY OF OMB EXPIRATION DATE

The OMB Expiration Date will be displayed on the 2021 SDR questionnaire; in the online survey version, it will be included on the informed consent page of the online survey and available in the online FAQs; in the telephone interview, it will be read to sample members during the introductory informed consent.

18. EXCEPTION TO THE CERTIFICATION STATEMENT

Not Applicable. No exceptions to the certification statement are being sought.

1 Includes a new cohort member in prior round who was subsequently found to be ineligible for that survey administration because they had received their PhD after the reference date and is now in the 2021 new cohort.

2 Section 505, Pub. L, No. 111-358, See Appendix A

3 The S&E workforce includes individuals with degrees or occupations in computer and mathematical sciences, life sciences, physical sciences, social sciences, engineering, and health sciences.

4 See Appendix A.

5 42 U.S. Code § 1863(j)(1)

6 42 U.S. Code § 1885(a), 1885(d)

7 42 U.S. Code § 1885(d)

8 National Research Council, Committee on National Statistics. 1989. Surveying the Nation’s Scientists and Engineers: A Data System for the 1990s. Washington: National Academy Press.

9 The SED gathers information yearly from all new research doctorates awarded by U.S. institutions. Detailed information about the SED can be found at http://www.nsf.gov/statistics/srvydoctorates/.

10 https://www.nsf.gov/pubs/2018/nsb20187/nsb20187.pdf, http://nsf.gov/nsb/publications/2015/nsb201510.pdf and http://www.nsf.gov/nsb/documents/2003/nsb0369/nsb0369.pdf

11 L. Gokhberg et al. (eds.), The Science and Technology Labor Force, Science, Technology and Innovation Studies: Switzerland, Springer International Publishing, 2016, pp77-119.

12 http://www.nsf.gov/nsb/publications/2010/nsb1033.pdf

13 http://www.oecd.org/sti/inno/CDH%20final%20conference%20report.pdf

14 http://www.ets.org/c/19574/19089_PathwaysReptqp.pdf

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Foley, Daniel J
File Modified	0000-00-00
File Created	2021-06-03