SDR 2019 OMB Supporting Statement Part A (Final revision 08-28-2019)

This request is for a three-year renewal of the previously approved OMB clearance for the Survey of Doctorate Recipients (SDR). The SDR has historically served as a valuable source of information on U.S.-trained science, engineering, and health doctorate recipients. The SDR was last conducted in 2017 and the OMB clearance for the 2017 SDR expires June 30, 2020 (OMB No 3145-0020). Both the data collection instruments for the 2019 SDR and the sample size are largely unchanged from the prior round.

For the 2019 SDR, the following modifications are being implemented and described further in section B1.2:

New follow-up rule: Potential survey respondents introduced to the survey during the 2015 cycle that did not respond in 2015 and 2017 will not be included in the 2019 SDR sample. This decision was made in response to the cost inefficiencies of including these nonrespondents in subsequent SDR survey cycles. A weight adjustment will be incorporated to maintain population coverage of these dropped individuals.
Sample supplement: For the 2019 SDR survey cycle, to offset the loss of the nearly 15,000 potential respondents from the new follow-up rule, NCSES will supplement the 2019 SDR sample with previously unsampled individuals from the 2015 SDR frame.
Field of Study stratification variable revision: The 2019 SDR field of study stratification variable will be modified to a version that includes 77 detailed fields of study in order to create better alignment with the survey’s estimation capabilities and with NCSES’s taxonomy of disciplines (TOD). This modification continues our efforts that began in the 2015 SDR survey cycle to create better alignment between the stratification variables and our survey estimation goals. In the 2015 SDR survey cycle, we expanded our field of study stratification variable to over 200 levels to support estimation of employment outcomes by a larger number of fine fields of degree. After reviewing the results from the 2015 and 2017 survey cycles, this modification in the 2019 survey cycle is to align the field of study stratification variable with the number of detailed fields for which the survey supports reliable estimation. At the same time, we are creating alignment between this variable and the internal NCSES field of study taxonomy.
Precision requirements: The 2019 SDR sample design includes estimation precision requirements for the cross-sectional estimation of a large number of key analytic domains. Precision requirements cover over 40 key analytic domains and primarily are based on demographic characteristics cross-tabulated by aggregated fields of study. In contrast, precision requirements for longitudinal estimation will be developed after the 2019 SDR is concluded and the longitudinal sample subset is drawn from among them.
Longitudinal sample identification: Beginning with the sample design revisions in the 2015 SDR survey cycle, NCSES has set the goal of developing a sustainable SDR design that will maintain the integrity of the survey's cross-sectional estimates while producing reliable data to support longitudinal analysis. All 120,000 cases in the 2019 SDR sample will be used for cross-sectional estimation and a 40,000-case subset of these cases will be identified to serve as a longitudinal sample that will be the basis for deriving longitudinal estimates. The longitudinal sample will use the 2015 SDR responses as baseline information and will be weighted and maintained through the SDR 2025 cycle to support longitudinal estimation of doctorate holders during the 2015-2025 time period. The 2019 SDR longitudinal sample will provide outcome data over the 4-year period of 2015-2019.

NECESSITY FOR INFORMATION COLLECTION

In 2010, the America COMPETES Reauthorization Act of 2010¹ established the National Center for Science and Engineering Statistics (NCSES) within the National Science Foundation (NSF) and directed NCSES to “...collect, acquire, analyze, report, and disseminate statistical data related to the science and engineering enterprise in the United States and other nations that is relevant and useful to practitioners, researchers, policymakers, and the public...” Information obtained through the SDR is critically important to NCSES’s ability to measure the education and employment outcomes of scientists and engineers. Furthermore, the SDR and NCSES’s National Survey of College Graduates (NSCG) are coordinated in both survey content and timing to form data collections that serve as the nation’s only source of comprehensive information about the size and characteristics of the science and engineering (S&E) workforce.² These data are solicited under the authority of the NSF Act of 1950³, as amended, and are central to the analysis presented in a pair of congressionally-mandated reports⁴^,⁵ published by NSF:

Science and Engineering Indicators
Women, Minorities, and Persons with Disabilities in Science and Engineering.

In addition, the Science and Engineering Equal Opportunities Act of 1980 directs NSF to provide to Congress and the Executive Branch an “accounting and comparison by sex, race, and ethnic group and by discipline, of the participation of women and men in scientific and engineering positions.”⁶ The SDR and NSCG provide much of the information to meet this mandate. The coordinated designs of these two surveys were developed throughout the past two decades and are based on recommendations from the National Research Council’s Committee on National Statistics (CNSTAT) report to NSF.⁷

SDR Background

The SDR provides education and employment-related information on scientists and engineers who were awarded a research doctoral degree from a U.S. institution in a science, engineering or health (SEH) field. A research doctorate is a doctoral degree that (1) requires completion of an original intellectual contribution in the form of a dissertation or an equivalent culminating project (e.g., a published manuscript) and (2) is not primarily intended as a degree for the practice of a profession.

The 2019 SDR is comprised of three sample components:

An existing panel of doctorate recipients from the 2017 survey cycle who remain eligible and are under 76 years of age;
A new cohort component that adds eligible new doctorate recipients from academic years 2016 and 2017, also under 76 years of age; and
A supplement of previously unsampled SEH doctorates from the 2015 SDR frame who earned their degree during or before academic year 2013 and are under 76 years of age.

The panel portion and the supplemental portion of the SDR sample provides information on the experienced stock of doctorate recipients. The new cohort sample from the two most recent doctorate award years provides important data on the early career experiences of new doctorate recipients with SEH degrees.

Since 2010, the SDR has included an international component of U.S.-trained doctorate recipients who received their degrees as of 2001. The more recent redesigned 2015 SDR cycle used the DRF to refresh and significantly expand the SDR sample and this expanded sample allowed the international component to become representative of all academic years dating back to 1961. As in 2015 and 2017, the 2019 SDR will field a sample of members predicted to reside either in or outside of the U.S. on the survey reference date of 1 February 2019. For example, based on the data from the 2016 and 2017 Survey of Earned Doctorates (SED), which is the frame for the 2019 SDR new cohort, 35% of U.S. SEH doctorates were awarded to temporary visa holders and 21% of them planned to leave the U.S. upon graduation.⁸ Thus, the 2019 SDR will yield information about the educational, employment and demographic characteristics of U.S.-trained SEH doctorate recipients living and working both in the U.S. and abroad.

The 2015 SDR sample expansion from 47,000 to 120,000 was designed to support estimation of employment outcome by a larger number of fine fields of degree (FFOD) captured in the SED. This larger sample will be maintained in 2019 and will include 10,000 new sample members from the 2016 and 2017 SED cohorts. Originally, the SDR was designed to produce employment outcome estimates for various analytical domains defined by broad aggregated fields of degree and demographic characteristics. The objective of the 2015 sample expansion and the most recent 2017 sample design was to meet new FFOD estimation goals while maintaining the traditional (historic) analytic domain-level estimation goals.

Having reviewed the 2015 and 2017 SDR results to examine the field of study levels for which the survey can support reliable estimation, the 2019 SDR sample design will be modified to incorporate a field of study stratification variable that better aligns with our survey estimation capabilities. In making this stratification variable modification, NCSES created a new field of study sampling stratification variable for the SDR that aligns with the NCSES Taxonomy of Disciplines (TOD). The NCSES TOD more closely aligns with the National Center for Education Statistics (NCES) Classification of Instructional Programs (CIP). The redesigned sample approach for the 2019 SDR uses a field of study stratification variable that stratifies the sample frame by 77 detailed fields rather than the more than 200 FFODs used in the 2015 and 2017 SDR cycles. This revised stratification supports a more stable sampling design and more reliable estimation capability in subsequent cycles of the SDR. By creating a field of study stratification variable that aligns with the NCSES TOD, there will be minor changes in the way the detailed fields will be aggregated in 2019 compared to past cycles. For example, up to 9 of the 200 fine fields will change levels when aggregating to the 3 broad-field levels of science, engineering, and health; when aggregating to an 8-level major fields categorization, up to 11 of the 200 fine fields changed levels (See Appendix B). Further details of the 2019 SDR sample frame and sample design are reported in Section B.1.

At the conclusion of the 2019 SDR survey cycle, all respondents from the 120,000-case sample will be used to develop cross-sectional estimates describing the U.S.-trained SEH doctorate recipients. In addition, a 40,000-case subset of these cases have been identified to serve as a longitudinal sample that will be the basis for deriving longitudinal estimates. The 40,000 longitudinal sample cases will have participated in the 2015 and 2017 cycles of the SDR and represent the 2015 SDR target population moving forward into the 2021 SDR cycle of data collection. This longitudinal sample will be weighted and maintained through the 2025 cycle of the SDR to provide longitudinal data for the 10-year period 2015 to 2025. The 2019 SDR longitudinal sample will provide longitudinal data over the 4-year period of 2015-2019.

USES OF INFORMATION

The data from the SDR provide has historically provided valuable cross-sectional information on careers, training, and educational development of the nation’s SEH doctoral-level personnel resources. These data enable government agencies to assess the scientific and engineering resources available in the U.S. to business, industry, and academia, and provide a basis for the formulation of the nation's S&E workforce policies. For example, educational institutions can use the SDR data in establishing and modifying scientific and technical curricula, while various industries can use the information to develop recruitment and remuneration policies.

Policymakers, researchers, and other data users use information from the SDR and NSCG to answer questions about the number, employment, education, and characteristics of the S&E workforce. These surveys provide nationally representative data that policymakers and researchers use to address questions on topics such as: employment of foreign-born or foreign-degreed scientists and engineers, the transition from higher education to the workforce, the role and importance of postdocs as research personnel, diversity in both education and employment, the implications of an aging cohort of scientists and engineers as baby boomers reach retirement age, and information on long-term trends in the S&E workforce.

Findings from the 2019 SDR will enable NCSES to continue reporting employment patterns of recent SEH doctorate recipients, as well as more experienced doctorate recipients in the labor market. The expanded sample enables NCSES to produce reliable estimates of employment outcomes by fine fields of degrees used in the NCSES TOD. The SDR data are made available through published reports, the Scientists and Engineers Statistical Data System (SESTAT), public use data files, and licenses for restricted-use data files. The online data tool, available at https://ncsesdata.nsf.gov/sestat/sestat.html, allows users to create customized data tabulations with a user-specified subject area. The SDR public-use files are available for download through the NCSES data downloads web page at https://ncsesdata.nsf.gov/datadownload/.

The 2019 survey cycle marks the introduction of a longitudinal component to be identified within the current SDR sample design upon completion of the 2019 data collection. The 40,000-case longitudinal sample that will be drawn provides the ability to derive longitudinal estimates from the 2015 SDR target population on a variety of policy-relevant topics. SDR longitudinal data can be used to address important issues such as changes in employer, occupation, field of research, international mobility, income over time and retirement. Within the academic sector, SDR longitudinal data can address issues such as length of post-doctoral training, achieving tenure, and changes in location especially for married S&E couples. As noted previously, the longitudinal sample will represent the outcomes for the 2015 SDR population through the year 2025 based on 6 cycles of SDR data.

Uses for Policy Discussion

Data from NCSES’s surveys are used in policy discussions of the executive and legislative branches of Government, the National Science Board, NSF management, the National Academy of Sciences, Engineering, and Medicine, professional associations, and other private and public organizations. Some recent examples of the current use of the SDR data, and the combined SDR and NSCG data, are as follows:

The National Science Board (NSB) used combined SDR and NSCG data in its investigation to develop national policies for the S&E workforce⁹;
The Committee for Equal Opportunity in Science and Engineering (CEOSE), an advisory committee to NSF and other government agencies, established under 42 U.S.C. §1885c, has been charged by the U.S. Congress with advising NSF in assuring that all individuals are empowered and enabled to participate fully in science, mathematics, engineering and technology. Every two years CEOSE prepares a congressionally mandated report that makes extensive use of NSCG and SDR data to highlight key areas of concerns relating to students, educators, and technical professionals;
A chapter in the book The Science and Technology Labor Force: The Value of Doctorate Holders and Development of Professional Careers used SDR data to describe the S&E labor force in the U.S.¹⁰
The importance of information on the S&E workforce to inform public policy can be seen in discussions of the NSB’s Task Group on Science, Technology, Engineering, and Math (STEM) Innovators. The task group used NSCG and SDR data to inform its deliberations about the S&E workforce and these data were part of the final report¹¹;

Information from the SDR was presented at the Organisation for Economic Co-operation and Development (OECD) conference in December 2012, “Understanding and improving the contribution of doctoral graduates to innovation and the economy: Developing the statistical evidence”¹²; and
The Educational Testing Service (ETS) and Council of Graduate Schools (CGS) used SDR and NSCG data to examine national benchmarks for career outcomes of master’s and doctoral degree recipients by specific field.¹³

Uses by NSF and NCSES

The SDR data were used extensively in the latest versions of the congressionally mandated biennial reports: Science and Engineering Indicators, 2018 and Women, Minorities and Persons with Disabilities in Science and Engineering, 2019.

In addition, NCSES used the SDR data and the combined SDR and NSCG data in recent reports such as:

Science and Engineering State Profiles, July 2018
Number of Women with U.S. Doctorates in Science, Engineering, or Health Employed in the United States More Than Doubles since 1997, NSF 19-307, February 12, 2019.
The 2015 Survey of Doctorate Recipients Expands Its Population Coverage and Reporting on Field of Study, NSF 17-319, June 28, 2017.
Data Tables: Survey of Doctorate Recipients, 2017, February 2019.
Data Tables: Survey of Doctorate Recipients, 2015, March 2018.
Unemployment among Doctoral Scientists and Engineers Increased but Remained Below the National Average, April 2014.
Characteristics of Scientists and Engineers in the United States: 2008, June 2013.
Employment and Educational Characteristics of Scientists and Engineers, January 2013.
International Mobility and Employment Characteristics among Recent Recipients of U.S. Doctorates, October 2012.

All NCSES publications can be accessed on the NCSES website at http://www.nsf.gov/statistics.

Uses by Researchers and Analysts

SDR and the combined SDR and NSCG data are presented at conferences and professional meetings by NCSES staff and survey contractor staff, such as the annual meeting of the Association for Institutional Research, the American Association for Public Opinion Research, the American Educational Research Association, and the Joint Statistical Meetings. Examples of these presentations are as follows:

Managing Locating and Data Collection Interventions through Adaptive Survey Design, American Association for Public Opinion Research, May 2019.
Where in the world? How in the world? The Challenges of Collecting Data around the Globe, American Association for Public Opinion Research, May 2019.
Using Contacting Information to Derive Employer Name in the Survey of Doctorate Recipients, American Association for Public Opinion Research, May 2019.
Exploring Alternative Measures of Doctoral Underemployment, Society for Longitudinal & Lifecourse Studies International Conference, July 2018,
The problem of analytic error in secondary analysis of survey data: What we know, and what we need to do about it, Duke Initiative on Survey Methodology, June 2018,
Balancing cross-sectional and longitudinal design objectives for the Survey of Doctorate Recipients, Federal Committee on Statistical Methodology Research and Policy Conference, March 2018,
Evaluating how doctorate recipients make their career decisions, Decision Sciences Institute Annual Meeting, November 2017,
Challenges and Opportunities in Major Survey Redesigns: Experiences from the SIPP, NHIS, and SDR, Joint Statistical Meetings, July 2017,
NCSES, Pathways through Universities and Graduate Schools into Careers – NCSES S&E Workforce Surveys, American Educational Research Association, April 2016.

Since 2007, NCSES has distributed more than 3,500 copies of SDR public-use files (2003, 2006, 2008, 2010, 2013, 2015, and 2017 survey cycles), as well as over 5,100 copies of the combined SDR and NSCG data’s public-use files (1993-2013 survey cycles). There are currently 49 active SDR restricted-use licenses. Selected recent citations by researchers using SDR data are as follows:

Khosla, P. (2018). "Wait time for permanent residency and the retention of immigrant doctoral recipients in the U.S." Economic Analysis and Policy 57: 33-43.
Kniffin, K. M. and A. S. Hanks (2018). "The trade-offs of teamwork among STEM doctoral graduates." American Psychologist 73(4): 420-432.
Webber, K. L. and M. González Canché (2018). "Is There a Gendered Path to Tenure? A Multi-State Approach to Examine the Academic Trajectories of U.S. Doctoral Recipients in the Sciences." Research in Higher Education, DOI: 10.1007/s11162-018-9492-4.
Tao, Y. (2018). "Earnings of Academic Scientists and Engineers: Intersectionality of Gender and Race/Ethnicity Effects." American Behavioral Scientist 62(5): 625-644.
Torche, F. (2018). "Intergenerational Mobility at the Top of the Educational Distribution." Sociology of Education, DOI: 10.1177/0038040718801812.
Perez-Silva, R., M. D. Partridge and W. E. Foster (2018). "Are foreign‑born researchers more innovative? Self‑selection and the production of knowledge among PhD recipients in the USA." Journal of Geographical Systems, DOI: 10.1007/s10109-018-0281-6.
Meyers, L. C., A. M. Brown, L. Moneta-Koehler and R. Chalkley (2018). "Survey of checkpoints along the pathway to diverse biomedical research faculty." PLoS One 13(1): e0190606.
Bucks, B. and M. P. Couper (2018). "The Fine Print: The Effect of Legal/Regulatory Language on Mail Survey Response." Survey Practice 11(2).
Cohen, W. M., H. Sauermann and P. Stephan (2018). “Academics' Motives, Opportunity Costs and Commercial Activities Across Fields.” Working Paper 24769. National Bureau of Economic Research.
Cummings, W. K. and O. Bain (2018). “US Doctoral Study to Early Career.” Doctoral Education for the Knowledge Society. J. C. Shin et al. editors, Springer International Publishing AG: 91-103.
Kahn, S. and M. MacGarvie (2018). “Immigration Policy and Stay Rates of STEM PhDs.” Proceedings of the 23rd International Conference on Science and Technology Indicators, Leiden, The Netherlands, September 12-14, 2018.
Kahn, S. and M. MacGarvie (2018). “The Impact of Permanent Residency Delays for STEM PhDS: Who Leaves and Why.” NBER Working Paper Series No. 25175, National Bureau of Economic Research.
Kawa, N. C., J. A. Clavijo Michelangeli, J. L. Clark, D. Ginsberg and C. McCarty (2018). "The Social Network of US Academic Anthropology and Its Inequalities." American Anthropologist, DOI: 10.1111/aman.13158.
Lorenz, R. G., D. S. Karcher, M. D. Gautreaux, M. Limson and D. S. Zander (2018). "The Pathology Workforce and Clinical Licensure." Academic Pathology 5: 237428951877594.
Blau, D. M. and B. A. Weinberg (2017). "Why the US science and engineering workforce is aging rapidly." Proceedings of the National Academy of Sciences U S A 114(15): 3879-3884.

CONSIDERATION OF USING IMPROVED TECHNOLOGY

The 2019 SDR will collect data using three modes:

Self-administered online (or web) surveys (including access from mobile devices);
Paper self-administered questionnaires (mail); and
Computer-assisted telephone interviews (CATI).

The data collection effort will emphasize response by web, use a mobile optimized web instrument, incorporate adaptive design techniques, use optical scanning of paper questionnaires, and store contact history information in a case management system. These systems are described below.

Participation via the online response option has increased steadily since the 2006 cycle, from 47% in 2006 to 84% in 2017. In 2017, in addition to the 84% of participants that responded by the online response option, 12% responded by paper questionnaire and 4% by telephone. Of the respondents who answered the 2017 survey question regarding future mode preference, 83% indicated a preference for the online survey. Analysis indicates that the online mode results in more complete survey and follow-up contacting data than the mail questionnaire mode.

For returning sample members, the 2019 SDR will honor mode preferences reported in the 2017 SDR but also emphasize the efficiency of completing the survey via the web. Based on the 2017 SDR, we anticipate that approximately half of the new cohort will be started in the online survey mode. Like the 2017 survey cycle, about 84% of the 2019 survey responses are expected to be completed in the online mode.

As in 2017, the 2019 online survey will be configured for use on mobile devices (e.g., smartphones and tablets) to ensure that the respondent experience is optimized regardless of the screen size or browser used to access the survey. In addition, the 2017 online survey was substantially improved in its appearance and adaptation to mobile devices to further improve the user experience and this improvement will carry over into the 2019 SDR.

The 2019 SDR data collection effort will continue the use of a comprehensive case management system (CMS) to track data captured across the three modes (web, mail, CATI) with additional features added to the system. The additional features support an adaptive design data collection strategy for quickly prioritizing cases and real-time response rate calculations particularly by the location of the residency of sample members. Optical scanning will be used to capture mail questionnaires after keying and these images will be stored in a database for archival purposes.

4. EFFORTS TO IDENTIFY DUPLICATION

Some overlap exists between the SDR and NCSES’s Early Career Doctorates Survey (ECDS) (OMB Control # 3145-0235) in their target population and content. Regarding target population overlap, the ECDS has a 2-stage sample design and builds its frame by obtaining employee lists from U.S. academic institutions, Federally Funded Research and Development Centers, and the NIH Intramural Research Programs. These employee lists target individuals who received their first doctorate within the last ten years, in any field, in the U.S. or abroad. In contrast to the ECDS, the SDR excludes doctorate holders who only earned a professional doctorate, only earned a non-S&E doctorate, or earned their doctorate from an institution outside the United States. Overall, an estimated 140,000 doctorates in the 2017 SDR target population (~ 15%) were represented among the population of U.S.-trained early career doctorates who were in the ECDS target population. As a result, approximately 4% (~900) of the 2017 ECDS sample members are likely to also be sampled for the 2019 SDR (4% of the ECDS sample and 0.7% of the SDR sample).

Regarding content overlap, the ECDS collects substantially more information about respondent work experiences, especially about postdoc experiences, professional activities and achievements, funding, work-life balance, and future career plans. Thus, the ECDS is unique in providing a comprehensive survey and comparison of U.S.-trained and non-U.S. trained early career doctorates employed in U.S. academic and federal government research sectors. In contrast, the SDR is unique in representing employment, work activities, and years of experience among all U.S.-trained SEH doctorate recipients up to age 76 regardless of where they currently reside or work, including those working outside of academe and those residing outside of the U.S. The SDR represents U.S. doctorate recipients working full or part time in all employment sectors including those who are not working due to retirement or other reasons.

Because the ECDS collects substantially more information than the SDR, the 2017 ECDS contacted all sample members, even those identified as having been in the 2017 SDR. NCSES staff will work together with the ECDS and SDR survey contractors to identify sample members selected for both surveys. The 2019 SDR sample members identified as having been in the 2017 ECDS will receive a letter from NCSES prior to the start of SDR data collection. The letter will clarify the unique nature of each survey and emphasize the importance of participation in both.

Overlap also exists between the SDR and the NSCG in their target population and in survey content. The NSCG and the SDR both capture estimates of the roughly 1 million U.S.-trained SEH doctorates; however, the NSCG also covers an estimated 165,000 employed foreign-trained SEH doctorates residing in the United States. However, unlike the NSCG, the SDR collects detailed information from U.S.-trained SEH doctorate recipients working in post-secondary academic institutions including their academic position, faculty rank, tenure status, and reasons for taking a postdoc if in a postdoctoral position. In contrast, unlike the SDR, the NSCG collects information on veteran status, attainment of certifications and licenses, financial support for education, and community college enrollment history.

Based on the 2017 SDR sample, approximately 300 individuals may be selected for both the 2019 NSCG and the 2019 SDR. Due to the content differences between the surveys, the relatively small number of expected duplicates, and the operational challenges of the deduplication process, NCSES will not de-duplicate individuals selected for participation in both the SDR and NSCG.

EFFORTS TO MINIMIZE BURDEN ON SMALL BUSINESS

Not applicable. The SDR collects information from individuals only.

CONSEQUENCES OF LESS FREQUENT DATA COLLECTION

The SDR data are central to the analysis presented in a pair of congressionally mandated reports - Science and Engineering Indicators and Women, Minorities, and Persons with Disabilities in Science and Engineering. Since these reports are published on a biennial schedule, they rely on the availability of updated data on the S&E workforce every two years. Conducting the SDR on a less frequent basis would prohibit NCSES from meeting its congressional mandate to produce a report that contains an accurate accounting and comparison, by sex, race, and ethnic group and by discipline, of the participation of women and men in scientific and engineering occupations. The impact of not being able to meet this congressional mandate is that government, business, industry, and universities would have less recent data to use as a basis for formulating the nation’s S&E policies.

A less frequent data collection would also impact SDR data quality. Follow-up surveys every two years enable analyses of changes in the S&E workforce when individuals move in and out of S&E occupations over both business and life cycles. To ensure the availability of current national S&E workforce data, the SDR has been coordinated with the NSCG on a biennial basis since 1993. The degradation of either component jeopardizes the integrity and value of these combined surveys to provide comprehensive information on the S&E workforce.

Conducting the survey less frequently would make it more difficult and costly to locate sampled persons given the mobility of the doctorate population. The impact is likely to be a higher attrition rate, higher potential for nonresponse bias, and less reliable cross-sectional and longitudinal estimates.

SPECIAL CIRCUMSTANCES

Not applicable. This data collection does not require any one of the reporting requirements listed.

8. FEDERAL REGISTER ANNOUNCEMENT AND CONSULTATIONS OUTSIDE THE AGENCY

Federal Register Announcement

In response to the Federal Register Notice for the SDR that appeared on August 14, 2018, one public comment was received by the closing date of October 14, 2018. See Appendix C for both the announcement and the comment. The comment requested that NCSES include measures of sexual orientation and gender identity on the SDR and on other NCSES surveys (specifically, the NSCG and the SED).

NCSES informed the commenters that it shares their interest in improving federal data collections and providing reliable measures for important segments of the population. Furthermore, NCSES described its process for evaluating possible questionnaire additions, including the extensive experimentation involved and the time and resources required. Finally, NCSES informed the commenters that it is initiating research to evaluate these measures and does not intend to include them in the 2019 SDR.

Consultations Outside the Agency

NCSES routinely seeks the advice and guidance of survey methodologists, statisticians, demographers, researchers, data analysts, and policymakers to examine numerous issues related to the further development and continuous quality improvement of the SDR. As noted in the 2017 SDR OMB Information Collection Request, NCSES previously convened a Human Resources Experts Panel (HREP) meeting in October 2016 to review the redesigned 2015 SDR sample expansion with a large number of data users and stakeholders, and to collect information on their research interests that could be addressed with SDR longitudinal data.

On 15 June 2017, the Office of Management and Budget (OMB) approved the collection of the 2017 SDR under the following terms:

Approved consistent with the understanding that the primary goal of the design is to allow cross-sectional reporting for fine fields; a secondary goal is to build an efficient, effective and sustainable longitudinal design.

In response to these terms and in alignment with NCSES’s mission, NCSES has conducted research with the goal of developing a sustainable design that will maintain the integrity of the survey's cross-sectional estimates while producing reliable data to support longitudinal analysis. As part of this effort, NCSES has convened expert panels and held outreach meetings with key SDR stakeholders. The details of the expert panels and outreach meetings are described below. Summary notes from key expert panel meetings are included in Appendix G. Through this effort, NCSES has identified a portion of returning sample from the 2015 SDR that will be the basis for establishing a longitudinal panel representing the 2015 SDR target population. This panel will be weighted and maintained through the SDR 2025 cycle to provide longitudinal data for the 2015-2025 time period. The sample size of the 2015-2025 longitudinal panel will be comparable to that of the past SDR cycles prior to the 2015 expansion cycle, with approximately 40,000 members.

Meeting, Workshops, and Outreach related to the SDR 2019 Sample Design Activities

SDR stakeholders meeting (March 7, 2017)

On March 7, 2017, NCSES held a meeting with several academic researchers who were users of the 1973 to 2013 SDR data in conducting longitudinal studies of the academic SEH doctorate workforce. The purpose of the meeting was to reaffirm NCSES’s commitment to incorporate a more robust longitudinal design that would be sustainable over time. NCSES staff also discussed the limitations of the pre-2015 design in fully meeting their research needs, and various options for bridging the past and future SDR longitudinal components in meeting their research needs.

Participants:

Katherine Smith Evans, American Economic Association

Daniel Newlon, American Economic Association

Donna Ginther, Ph.D., Director, Institute for Policy & Social Research, University of Kansas

Bruce Weinberg, Ph.D., Professor of Economics, The Ohio State University

SDR Sample Design Expert Panel (May-September 2017)

The large-scale sample expansion and refreshment of the 2015 SDR provided an opportunity to formally establish longitudinal objectives for the survey. Therefore, in follow-up to the March 2017 stakeholders meeting noted above, NCSES convened a Sample Design Expert Panel that met five times on an approximate monthly basis from May through September 2017. The Panel was charged with providing guidance for NCSES’s effort to incorporate an efficient, effective, and sustainable longitudinal component into the SDR sample design. After reviewing the current SDR sample design and the survey’s cross-sectional and longitudinal estimation objectives, the panel developed outlines for several sample design options that could build on the current SDR sample design by adding a defined longitudinal component. For each design option considered, the sample design outline included: (i) sample design requirements, (ii) sample design descriptions, (iii) methods for sample allocation and sample selection, and (iv) cross-section and longitudinal estimation capabilities. In addition, after developing the sample design outlines, the Panel members compared the advantages and limitations of each sample design option and provided sample design recommendations for use in the 2019 and subsequent SDR survey cycles. Ultimately, the panel and NCSES staff agreed that a panel representative of the general SDR population rather than specified subpopulations would best meet the requirements for an efficient, effective, and sustainable longitudinal component.

Panel Members:

Jill DeMatteis, Ph.D., Associate Director and Senior Statistician, Westat

Don Jang, Ph.D., Vice President and Director, NORC at the University of Chicago

Jean Opsomer, Ph.D., Professor, Statistics, Colorado State University

Peter Siegel, Senior Statistician, RTI International

Human Resources Expert Panel (August 28-29, 2017)

The purpose of this invited panel of experts was to obtain input on SDR data uses, longitudinal design options, and SDR data product considerations. The goals for the meeting included the following:

Identify and discuss SDR longitudinal estimation objectives;
Review the longitudinal design options NCSES is considering and obtain feedback on the advantages and disadvantages of each option as they relate to the uses of SDR data; and
Identify and discuss SDR data products that address the survey’s cross-sectional and longitudinal objectives.

Panel Members:

Reba Bandyopadhyay, Ph.D., Science Policy Analyst, National Science Foundation

Jill DeMatteis, Ph.D., Associate Director and Senior Statistician, Westat

Kimberlee Eberle-Sudre, M.A., Senior Policy Analyst, Association of American Universities

Donna Ginther, Ph.D., Director, Institute for Policy & Social Research, University of Kansas

Kaye Husband Feeling, Ph.D., Chair, Professor of Economics, Georgia Institute of Technology

Don Jang, Ph.D., Vice President and Director, NORC at the University of Chicago

Hiro Okahana, Ph.D., Director, Council of Graduate Schools

Jean Opsomer, Ph.D., Professor, Statistics, Colorado State University

Earnestine Psalmonds, Ph.D., Division of Graduate Education, National Science Foundation

Peter Siegel, Senior Statistician, RTI International

Nicole Smith, Ph.D., Center on Education and the Workforce, Georgetown University

Ted Socha, Ph.D., National Center for Education Statistics

Karen Stamm, Ph.D., Senior Research Officer, American Psychological Association

Ruth Wasem, Ph.D., Clinical Professor of Public Policy Practice, The University of Texas at Austin

Bruce Weinberg, Ph.D., Professor of Economics, The Ohio State University

Matt Wilson, Ph.D., Policy Director/S&E Policy Analyst, National Science Foundation

John Czajka, Ph.D., Senior Fellow, Mathematica Policy Research

Federal Committee on Statistical Methodology Research Conference presentation: (March 9, 2018)

In addition to the meetings noted above, NCSES promoted the development of the SDR longitudinal design by presenting at the 2018 FCSM Research Conference. The presentation focused on the complexities of balancing cross-sectional and longitudinal goals while building an effective, efficient, and sustainable longitudinal design into the newly expanded cross-sectional SDR sample. Specifically, the presentation noted that to enhance SDR’s utility and meet dual cross-sectional and longitudinal goals, longitudinal panels within the refreshed sample need to be established formally and maintained over time. The presentation also highlighted NCSES’s on-going outreach efforts described above. The presentation concluded by noting that the process of developing a longitudinal design to enhance overall survey utility is a good application of a major survey redesign effort and the approaches and lessons learned from the SDR experience could be informative for other researchers involved in survey design.

Technical and analytical support for the SDR longitudinal design (April-December 2018)

Having explored a variety of longitudinal design options for the SDR and having received feedback from SDR stakeholders on cross-sectional and longitudinal objectives, the next phase of the SDR longitudinal design development involved identification of specific SDR design requirements and the determination of a longitudinal design to implement within the 2019 SDR survey cycle. In addition, one of the consultants, Professor Simon Woodcock, was tasked with providing a report on the development of longitudinal outcome measures and the connection of the longitudinal outcome measures to stakeholder data needs. Another expert, Professor Jean Opsomer, provided expert consultation in the development of an appropriate sample design detailing sample selection, allocation, and sample size maintenance requirements.

Toward the end of the consultation, NCSES convened another Expert Review Panel in November 2018 to evaluate the detailed proposed design and, if necessary, refine NCSES’s proposed longitudinal outcomes and longitudinal design decisions. The goal of the meeting was to review and vet several proposed changes to the 2019 SDR sample design including establishing a new longitudinal panel that will be maintained for at least 10 years starting from the 2015 survey cycle. Specifically, for the 2019 SDR cross-sectional design and sampling, NCSES proposed the following revisions:

Follow-up rule. NCSES proposed to drop people who have not responded in the first two waves. For the 2019 sample, NCSES proposes to drop sample cases that did not respond to the 2015 or 2017 wave. A weight adjustment will be incorporated to maintain coverage of the dropped cases.
Modified sample stratification. NCSES proposed two changes to the sampling stratification for future sample selection (new cohorts and supplemental samples): 1) change the number of fields of degree in the stratification from over 200 to 77 levels and 2) add two additional stratifying variables (sex and underrepresented minority (URM) indicator).
SDR Sample Supplement. With the follow-up rule implemented, NCSES proposed to supplement the current 2019 SDR sample using the modified sample stratification to reach the sample size goal of 120,000 sample cases.

The experts approved the revised cross-sectional design procedures, noting that they did not see any issues to the proposal nor did the experts propose any further modifications or improvements. For the longitudinal design and sampling, NCSES proposed implementing the design approach recommended by the 2017 SDR sample design expert panel with the following design details:

Stratification. NCSES proposed defining the longitudinal sample stratification using employment sector, age group, URM indicator, and sex (with some collapsing, as needed).
Implicit stratification. NCSES proposed sorting the longitudinal frame within strata by sex, residential location, race/ethnicity, disability indicator, career stage, U.S. citizenship status at the degree time, and minor field of degree (26 levels).
Allocation and sample selection. NCSES proposed the following allocation and sample selection approach for the longitudinal sample: (i) allocate the sample proportionally across all strata initially, then redistributing a portion of the large strata to meet the minimum stratum-level precision requirements, (ii) after determining the stratum-level allocation, sample using the probability proportional to size (PPS) systematic sampling approach.

This expert panel discussed a variety of facets of this approach and they did not find any major issues with the proposed approach. NCSES plans on continuing the discussion past this meeting and into a larger stakeholder meeting to be held in the winter of 2019.

Panel Members:

Jill DeMatteis, Ph.D., Associate Director and Senior Statistician, Westat

Stephanie Eckman, Ph.D., Survey Methodologist, RTI International

Don Jang, Ph.D., Vice President and Director, NORC at the University of Chicago

Jean Opsomer, Ph.D., Professor, Statistics, Colorado State University

Simon Woodcock, Ph.D., Research Fellow, Simon Fraser University

PAYMENT OR GIFTS TO RESPONDENTS

Incentives for the SDR began during the 2003 cycle and have continued for all subsequent cycles. The 2019 SDR incentive plan is modeled after the approach used in recent SDR survey cycle. Described below are the details of the NCSES plans to offer both early and late-stage incentives.

With the exception of two groups, all 2019 SDR sample cases will be eligible to receive an incentive. Non-U.S. residing sample members and sample members determined to work for the National Science Foundation will be excluded from the incentive offer.

Proposed Incentive Plan for the 2019 SDR

Early-Stage Incentive. The early-stage incentive will target two types of sample members: 1) those who have only responded after being incentivized in prior rounds and 2) new cohort sample members (i.e., individuals earning their degree in academic years 2016 or 2017).

Early-stage incentives will be offered to each sample component as described below:

Sample members who have historically only responded after receiving an incentive will be offered a monetary incentive in the first contact to encourage a faster response and to reduce the costs associated with follow-up contacts. The rationale for this approach is based on the 2013, 2015 and 2017 SDR. An examination of the 2013 response of sample members who consistently only participated after receiving an incentive in the past survey cycles shows 69.7% completed the 2013 survey after receiving a late-stage request for survey participation with an incentive offer, while only 37.6% completed at this stage without an incentive. In 2015, all sample members who only responded previously after receiving an incentive were sent an incentive with their first survey request. Of the cases from this group, 81.5% completed the 2015 survey. The protocol of sending an incentive offer upfront to individuals who only cooperate after receiving the monetary incentive was repeated in 2017 and 77.7% of this subgroup completed the survey after being sent a monetary incentive with the first request for survey participation.

Four subgroups of the 2017 panel will be eligible for an early incentive offer in the 2019 SDR:

Sample members who were in both the 2015 and 2017 rounds and who participated in 2015 with an incentive but did not participate in 2017 when they did not receive an incentive.
Sample members who were in both the 2015 and 2017 rounds and who participated in 2015 with an incentive but did not participate in 2017 until they received a late-stage incentive.
Sample members who were in both the 2015 and 2017 rounds and who participated in 2017, being assigned in 2017 to the group receiving the early-stage incentive (based on their participation and incentive history).
Sample members that were part of the 2015 expansion sample cohort and only participated in the 2015 and /or 2017 cycle after receiving an incentive.

For the new cohort sample members, incentive experiments conducted in the 2006 and 2008 SDR survey cycles indicate that offering a prepaid incentive in the second contact is a cost-effective way of encouraging survey response. Furthermore, post-survey analysis of the 2010 SDR data indicated the monetary incentive had a positive effect on response. In the 2017 cycle, new cohort sample members in the U.S. were offered an incentive with the second request for survey participation. This resulted in 57.0% of the new cohort sample receiving the incentive offer. Of these early incentivized new cohort cases, 85.8% completed the survey. Furthermore, preliminary review of incentive usage behavior shows that only half of those sent the incentive accepted it. Given the efficacy of the 2017 SDR new cohort incentive strategy, the proposal for the 2019 SDR is to offer a monetary incentive to the new cohort members in the second contact (i.e., those that do not respond to the initial contact).

Late-Stage Incentive. Among those who were not offered an early incentive as noted above, the overall strategy for the late-stage incentive among them is to ensure that all sample members who remain nonrespondents midway through the field period have a probability of receiving a monetary incentive. In the plan used for the 2008 through 2017 SDR, a higher probability of selection for the incentive was given to more challenging cases in key analytic domains with relatively lower response rates. This strategy was designed to improve the accuracy of survey estimates, and ideally, mitigate nonresponse bias.

The 2017 SDR results showed that late-stage eligible cases offered the incentive achieved a survey response rate of 56.5% versus 51.5% for late-stage incentive eligible cases not offered the incentive. Although this higher yield was not substantial, it achieved the goal of increasing the response rate among the more challenging cases in key analytic domains with relatively lower response rates in this late stage of data collection. Based on these results and findings from past cycles, we propose to continue this strategy for the 2019 cycle.

To most effectively allocate limited resources for the monetary incentive among late-stage survey nonrespondents who were not previously incentivized, NCSES will analyze characteristics of the remaining nonrespondents after the interim stage of data collection using a logistic regression model. This analysis will help to determine which types of sample members should receive a late-stage incentive to mitigate response bias of those residing in the U.S. Only those with a U.S.-based mailing address will be eligible to receive a late-stage incentive. The cases with lowest response propensity or those that contribute the most towards mitigating bias among sampled respondents will be selected for the incentive (See section B.5 for further details on the adaptive design goals and monitoring metrics). The volume of late-stage response cases to be incentivized will be determined based on the available budget.

Also, during the late-stage data collection phase, any nonrespondents selected for an early-stage incentive but were not sent their incentive because of locating or mailing address problems, will be issued or reoffered the incentive. Nonrespondents who were successfully sent the incentive during the early-stage phase will receive a non-incentivized late-stage treatment.

Incentive Costs

According to this plan, a $30 prepaid incentive would be offered for the 2019 SDR, as was done for the 2008 through 2017 SDR. The total cost of incentives in the 2017 SDR was $260,000. In 2019, it is expected to cost approximately $315,000. The complete incentive plan for 2019 is documented in section B 3.4 below.

10. ASSURANCE OF CONFIDENTIALITY

NCSES and its contractors are fully committed to protecting the confidentiality of all survey respondents. SDR data will be collected under the authority of the National Science Foundation Act of 1950, as amended, the America COMPETES Reauthorization Act of 2010. Through the Confidential Information Protection and Statistical Efficiency Act (CIPSEA) of 2002, and the Federal Cybersecurity Enhancement Act of 2015, all respondent data are protected from cybersecurity risks through screening of the Federal systems that transmit the data. Cover letters and survey questionnaires to each selected respondent will advise them that the information they provide is confidential (see Appendix D – Draft 2017 SDR Questionnaire that will be the same for 2019, and Appendix E – Draft 2019 SDR Survey Mailing Materials). The same notice of confidentiality will be used in the introduction to the CATI interview and will be displayed prior to the start of the survey in the online instrument. In addition, the CATI interviewers will inform the respondents of the voluntary nature of their response and that the data will be used for statistical purposes only.

Standard data collection procedures incorporate numerous safeguards for protecting the data and must conform to a detailed security plan approved by NCSES. While collecting SDR data, the information that could identify a sample member is separated from data about that person. Each sample member is assigned a unique identifier, and this identifier is used to store identifying information (such as name, address, etc.) in a separate, secure database apart from the survey response database. SDR contractors and NCSES staff receive annual CIPSEA training to reinforce their legal obligations to protect the privacy and confidentiality of the SDR data; staff must sign data use agreements annually to acknowledge this legal obligation.

Completed SDR hard copy questionnaires and other contact materials will be housed in a secure storage room at the contractor’s production facility. Only authorized staff – and only when necessary for data collection activities – will have access to hard copy materials from the SDR file room. The contractor’s electronic systems will be on a secure local area network (LAN), and all contractor systems for storage of electronic survey data will be secure by design and will be protected by passwords available only to authorized study staff.

The contractor will implement systems to make certain that data collected via the online questionnaire are secure. First, access to the online instrument will be allowed only with a valid Personal Identification Number (PIN) and password correctly entered in combination. Second, data will be transmitted by the Secure Sockets Layer (SSL) protocol that employs powerful encryption during transmission through the Internet. If a respondent keeps an online survey open without any activity, the online server will close the connection after a short period of inactivity, both preserving the data up to the break-off point and preventing unauthorized persons from completing the questionnaire. The online survey system will place authentication information and response data on physically separate servers, a strategy that provides an extra layer of security to protect response data. Both development and production servers will be backed up nightly as required by the contractor’s disaster recovery plan.

NCSES and its contractors will analyze and make available SDR tabulations only in aggregate form and will take all measures necessary to assure that the identity of individuals or organizations will not be disclosed in either its statistical tabulations or in the SDR public-use micro-data files.

11. JUSTIFICATION FOR SENSITIVE QUESTIONS

No questions of a sensitive nature are asked in this data collection.

12. ESTIMATE OF RESPONDENT BURDEN

A statistical sample of 120,000 persons, identified as having a doctorate in an SEH field from a U.S. academic institution will be selected for the 2019 SDR. This sample will include approximately 105,000 individuals residing in the U.S. and 15,000 residing abroad. The amount of time to complete the questionnaire may vary depending on an individual’s circumstances; however, on average it will take approximately 25 minutes to complete the survey. Assuming a 75% response rate (90,000 respondents), the total burden for the 2019 SDR is estimated to be 37,500 hours. Annualized over the requested clearance period of three years, the annual burden is estimated to be 12,5000 hours.

The total cost to respondents for the 37,500 burden hours is estimated to be $1,983,000. This is based on an estimated median annual salary of $110,000 per full-time employed SDR respondent from the 2017 SDR data. Assuming a 40-hour workweek over 52-weeks of employment, this annual salary corresponds to an hourly rate of $52.88.

13. COST BURDEN TO RESPONDENTS

Not applicable. This survey will not require respondents to purchase equipment, software, or to contract out services.

14. COST BURDEN TO THE FEDERAL GOVERNMENT

The total estimated cost to the Government for the 2019 SDR is approximately $12.6 million, which includes survey cycle costs, and NCSES staff costs to provide oversight and coordination with the NSCG. The estimate for survey cycle costs is approximately $12.0 million, which is based on sample size; length of questionnaire; CATI and online data collection technology; administrative, overhead, design, printing, mail and telephone data collection costs; incentive payments; critical items data retrieval; data keying and editing; data quality control; imputation for missing item responses; weighting and estimating sampling error; file preparation and delivery; preparation of documentation and final reports; analysis, and tabulations. The NCSES staff costs are estimated at $562,500 (based on $150,000 annual salary of 1.5 FTE for 2.5 years).

15. REASON FOR CHANGE IN BURDEN

The reduction in burden expected in 2019 is a result of the slight decrease in the number of sample members (decrease of 4,580). Questionnaire length and survey response rate are both expected to remain unchanged.

16. SCHEDULE FOR INFORMATION COLLECTION AND PUBLICATION

There are no plans to use any complex analytical techniques in NCSES publications using these data. Normally, SDR data are presented as cross-tabulations of the data in reports and other data releases. The time schedule for 2019 data collection and publication is currently estimated as follows:

Shape1

Data Collection (Mail, CATI, online) September 2019 – February 2020

Shape2

Coding and Data Editing February 2020 – June 2020

Shape3

Final Edited/Weighted/Imputed Data File June 2020

Shape4

SDR InfoBrief September 2020 Shape5

SDR Detailed Statistical Tables September 2020

Shape6

SDR Public Use File September 2020

Shape7

17. DISPLAY OF OMB EXPIRATION DATE

The OMB Expiration Date will be displayed on the 2019 SDR questionnaire; in the online survey version, it will be included on the informed consent page of the online survey and available in a help screen accessible at any point in the online survey; in the telephone interview, it will be read to sample members during the introductory informed consent.

18. EXCEPTION TO THE CERTIFICATION STATEMENT

Not Applicable.

1 Section 505, Pub. L. No. 111-358. See Appendix A.

2 The S&E workforce includes individuals with degrees or occupations in computer and mathematical sciences, life sciences, physical sciences, social sciences, engineering, and health sciences.

3 See Appendix A.

4 42 U.S. Code § 1863(j)(1)

5 42 U.S. Code § 1885(a), 1885(d)

6 42 U.S. Code § 1885(d)

7 National Research Council, Committee on National Statistics. 1989. Surveying the Nation’s Scientists and Engineers: A Data System for the 1990s. Washington: National Academy Press.

8 The SED gathers information yearly from all new research doctorates awarded by U.S. institutions. Detailed information about the SED can be found at http://www.nsf.gov/statistics/srvydoctorates/.

9 https://www.nsf.gov/pubs/2018/nsb20187/nsb20187.pdf, http://nsf.gov/nsb/publications/2015/nsb201510.pdf and http://www.nsf.gov/nsb/documents/2003/nsb0369/nsb0369.pdf

10 L. Gokhberg et al. (eds.), The Science and Technology Labor Force, Science, Technology and Innovation Studies: Switzerland, Springer International Publishing, 2016, pp77-119.

11 http://www.nsf.gov/nsb/publications/2010/nsb1033.pdf

12 http://www.oecd.org/sti/inno/CDH%20final%20conference%20report.pdf

13 http://www.ets.org/c/19574/19089_PathwaysReptqp.pdf

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Foley, Daniel J
File Modified	0000-00-00
File Created	2021-01-15