2025 NSCG SupportingStatement_A

Established within the NSF by the America COMPETES Reauthorization Act of 2010 § 505, codified in the National Science Foundation Act of 1950, as amended, the National Center for Science and Engineering Statistics (NCSES) serves as a central federal clearinghouse for the collection, interpretation, analysis, and dissemination of objective data on science, engineering, technology, and research and development for use by practitioners, researchers, policymakers, and the public.

This request is for a three-year renewal of the previously approved Office of Management and Budget (OMB) clearance for the National Survey of College Graduates (NSCG). The NSCG has historically served as a valuable source of information on the education and career paths of the nation’s college-educated population. The NSCG was last conducted in 2023. The current OMB clearance for the NSCG expires 30 April 2025 (OMB control number 3145-0141), which does not cover the complete survey cycle for the 2025 NSCG.

For the 2025 NSCG, the following modifications will be implemented:

The Computer Assisted Telephone Interviewing (CATI) operation is eliminated due to limited impact on the overall response rate and its associated cost.
In alignment with the revisions to OMB’s Statistical Policy Directive No. 15, the NSCG questionnaire will collect race and ethnicity information using one combined question that includes detailed race and ethnicity categories.
Two methodological experiments will be incorporated to investigate the effects of text messages and plain language letters on response (see Section B.4 and Appendices H and I).
There will not be a non-production bridge panel.

Prior terms of clearance:

A two year clearance is provided consistent with an understanding that NCSES will brief OMB

and FCSM colleagues on the results of the bridge panel embedded experiments for

measuring sexual orientation as soon as they are available so that any additional testing can

be done in advance of the 2025 NSCG.

The following list summarizes the efforts NCSES has undertaken since 2023 related to these terms of clearance:

NCSES established an internal working group that meets routinely to discuss on-going research efforts exploring the feasibility of collecting reliable sexual orientation and gender identity (SOGI) information in the NCSES surveys.
NCSES staff participate in interagency working groups to explore methodological and data quality issues related to measuring SOGI characteristics and to provide updates to federal statistical system colleagues on current NCSES SOGI research efforts.
NCSES created and periodically updates a page on the NCSES website summarizing its ongoing research regarding SOGI measurement and reporting.
NCSES fielded an experimental panel (referred to as the Bridge Panel) in the 2023 NSCG alongside the production survey to assess proposed question changes and potential new questionnaire content. The 2023 NSCG Bridge Panel examined potential modifications to the gender identify question wording and assessed the impact of the response option ordering for the sexual orientation question. The preliminary results of the 2023 NSCG Bridge Panel are summarized in Appendix J.
For the examination of potential modifications to the gender identity question wording, the 2023 NSCG Bridge Panel research did not uncover any reason to change the question wording used in the 2023 NSCG production data collection. As result, for the 2025 NSCG, NCSES will continue to use the two-step sex-at-birth/gender identity question from the 2023 NSCG survey cycle.
For the assessment of the response option ordering for the sexual orientation question, the 2023 NSCG Bridge Panel research found no significant difference in the response distribution due to the response option ordering. The Bridge Panel findings have been shared with the Census Bureau to help inform their on-going SOGI research efforts, including a proposed test of SOGI questions on the American Community Survey (ACS). The Census Bureau’s ACS is an authoritative source for demographic data used by federal and non-federal studies for benchmarking purposes to ensure consistency in survey estimates across different data collection programs. Given the critical role of the ACS in ensuring coordination and synchronization across the federal statistical system and its potential use as a benchmark for sexual orientation estimates, NCSES will await the findings from the Census Bureau’s ACS sexual orientation research to guide future data collection decisions. As a result, NCSES will not include a sexual orientation question on the 2025 NSCG and will use the Census Bureau’s ACS research, when completed, to inform our data collection plans for the 2027 NSCG.

1. WHY THE COLLECTION IS NECESSARY

The America COMPETES Reauthorization Act of 2010¹ established the National Center for Science and Engineering Statistics (NCSES) within the National Science Foundation (NSF) and directed NCSES to “...collect, acquire, analyze, report, and disseminate statistical data related to the science and engineering enterprise in the United States and other nations that is relevant and useful to practitioners, researchers, policymakers, and the public.” Information obtained through the NSCG is critically important to NCSES’s ability to measure the education and employment of scientists and engineers.

Furthermore, the NSCG data along with NCSES’s Survey of Doctorate Recipients (SDR)² data serve as the nation’s only source of comprehensive information about the size and characteristics of the science and engineering (S&E) workforce.³ These data are solicited under the authority of the NSF Act of 1950,⁴ as amended, and are central to the analysis presented in a pair of congressionally mandated reports published by NSF:

Science and Engineering Indicators⁵
Women, Minorities, and Persons with Disabilities in Science and Engineering⁶

In addition, the Science and Engineering Equal Opportunities Act of 1980 directs NSF to provide to Congress and the Executive Branch an “accounting and comparison by sex, race, and ethnic group and by discipline, of the participation of women and men in scientific and engineering positions.”⁷ The NSCG and SDR provide much of the information to meet this mandate.

PURPOSE AND USE OF THE INFORMATION

NSCG Background

The NSCG is a repeated cross-sectional survey conducted to provide data on the nation’s college graduates, particularly those in the S&E workforce. The NSCG samples individuals who are living in the United States, have at least a bachelor’s degree, and are less than 76 years of age. As of 2017, the NSCG fully implemented a four-panel rotating panel design, in which every new panel receives a baseline questionnaire, followed by three biennial follow-up questionnaires before rotating out of the survey. (See Section B.1 for further details about the panel design and sampling methods.)

The NSCG is a unique source for examining various characteristics of college-educated individuals, including occupation, work activities, salary, the relationship of degree field to occupation, and demographic information. This survey provides information on individuals residing in the United States with at least a bachelor’s degree, including those who received degrees only from foreign institutions. The SDR complements these data with more information on the population of U.S.-degreed doctoral level scientists and engineers. Collectively, the NSCG and SDR provide comprehensive information on the education and employment of the entire U.S. population of scientists and engineers with at least a bachelor’s degree. The NSCG and SDR are the only available sources of detailed information that support a broad range of policy and research topics on the dynamics of the S&E workforce over time.

The NSCG has a history of seeking survey improvements through methodological experiments, and the 2025 NSCG continues that trend with the inclusion of two studies to investigate the impacts of text messages and new letters developed following the principles of plain language on response. (See Section B.4 and Appendices H and I for more details about the studies.)

The NSCG enables government agencies to assess the scientific and engineering resources available in the United States to business, industry, and academia, and provide a basis for the formulation of the nation's S&E workforce policies. For example, educational institutions can use the NSCG data in establishing and modifying scientific and technical curricula, while various industries can use the information to develop recruitment and remuneration policies.

Policymakers, researchers, and other data users use information from the NSCG and SDR to answer questions about the number, employment, education, and characteristics of the S&E workforce. Because the NSCG and SDR provide up-to-date and nationally representative data, policymakers and researchers use these datasets to address questions on topics such as employment of foreign-born or foreign-degreed scientists and engineers, the transition from higher education to the workforce, diversity in both education and employment, the implications of an aging cohort of scientists and engineers as baby boomers reach retirement age, and long-term trends in the S&E workforce.

Uses for Policy Discussion

Data from NCSES’s surveys are used in policy discussions of the executive and legislative branches of government, the National Science Board, NSF management, the National Academy of Sciences, Engineering, and Medicine, professional associations, and other private and public organizations. Some recent, specific examples of the use of NSCG data are:

A study by Joni Hersch at Vanderbilt University used NSCG data to show that two groups of law graduates – Asian women and Black women – report low satisfaction with their jobs.⁸
A University of Kansas study used NSCG data to examine the pay gap between college -educated workers who worked in their field of study versus outside their field of study.⁹
Stuart Anderson, senior contributor to Forbes, furthered his research on the narrowing definition of an H-1B visa for “specialty occupations” and how this rule change might affect immigration to the US. ¹⁰
Researchers examined NSCG data to determine whether students who double-majored were better prepared to withstand negative earning shocks (pay cuts or job losses) compared to those who had a single college major. ¹¹
Jason Richwine, resident scholar at the Center for Immigration Studies, used data from NSCG to show that the value of foreign degrees are less valuable in the U.S. than U.S. degrees and vary substantially by the specific country or region where the degrees were earned.¹²
Weiss, D.M., Spitzer, M.L., Cronin, C. & Chin, N. used NSCG data to examine the returns to college majors and institutional selectivity.¹³
The Committee for Equal Opportunity in Science and Engineering (CEOSE), an advisory committee to NSF and other government agencies, established under 42 U.S.C. §1885c, has been charged by the U.S. Congress with advising NSF in assuring that all individuals are empowered and enabled to participate fully in science, mathematics, engineering, and technology. Every two years CEOSE prepares a congressionally mandated report that makes extensive use of NSCG and SDR data to highlight key areas of concerns relating to students, educators, and technical professionals.

Uses by NCSES

The NSCG data were used in the latest version of the congressionally mandated biennial report Diversity and STEM: Women, Minorities, and Persons with Disabilities (WMPD), 2023,¹⁴ and NSCG data will be used in the forthcoming WMPD 2025, set for release next year. Similarly, NSCG data were used in the congressionally mandated biennial report The State of U.S. Science and Engineering 2024,¹⁵ and The STEM Labor Force: Scientists, Engineers, and Skilled Technical Workers.¹⁶

NCSES used NSCG data in recent reports such as:

A Quarter of College Graduates and About Half of Doctorate Holders in Science and Engineering Occupations Were R&D Workers in 2019, September 2022
The Increasing Role of Community Colleges among Bachelor's Degree Recipients: Findings from the 2019 National Survey of College Graduates
Effects of the COVID-19 Pandemic on Employment, Earnings, and Professional Engagement: New Insights from the 2021 National Survey of College Graduates, January 2023

All NCSES publications can be accessed on the NCSES website at https://ncses.nsf.gov/browse-library.

Uses by Researchers and Analysts

NCSES makes the data from the NSCG available through published reports, online data tools, downloadable public-use files, restricted-use licenses, and the Federal Statistical Research Data Centers. The online data tools, available at https://ncsesdata.nsf.gov/home and https://ncsesdata.nsf.gov/sestat, allow users to create customized data tabulations using NSCG data. The NSCG public-use files are available for download through the NCSES data downloads web page at https://ncses.nsf.gov/explore-data/microdata/national-survey-college-graduates.

Since 2015, NCSES has distributed over 800 copies of the 1993 NSCG public-use files, over 800 copies of the 2003 NSCG public-use files, over 1000 copies of the 2010 NSCG public-use files, and over 2,200 copies of the 2013 NSCG public-use files to researchers in government, academia, and professional societies. The 2015 NSCG public-use files have been downloaded over 2,700 times since their release in January 2017, the 2017 NSCG public-use files have been downloaded nearly 3,000 times since their release in October 2018, the 2019 NSCG public-use files have been downloaded over 1,500 times since their release in January 2021, and the 2021 NSCG public-use files have been downloaded over 1,100 times since their release in January 2023. The 2023 NSCG public-use files will be available soon. The NSCG public-use files receive heavy use because they are the only data sets analysts can use to compare the S&E workforce to the general population of college degree holders in the United States.

In addition to users of the public-use files, there are currently 31 restricted-use licensees who specifically requested access to microdata files from the NSCG under a licensing agreement with NCSES. Plus, the U.S. Census Bureau has approved 11 research projects to use NSCG data in the Federal Statistical Research Data Centers (FSRDC).

Some of the recent research based on NSCG data resulted in papers such as the following:

Oerther, DB, Gautham, L and Folbre, N (2022) Environmental Engineering as Care for Human Welfare and Planetary Health. Journal of Environmental Engineering, 148 (6). ISSN 0733-9372 https://doi.org/10.1061/(ASCE)EE.1943-7870.0002013
Dirk Witteveen, The Effects of Undergraduate Financing on Advanced Degree Attainment, Social Forces, Volume 101, Issue 3, March 2023, Pages 1258–1287, https://doi.org/10.1093/sf/soac044

Rincon, R., & Williams, R. L., & Martínez, D. L. (2023, February), The Women of Color in Engineering Collaborative Paper presented at 2023 Collaborative Network for Computing and Engineering Diversity (CoNECD), New Orleans , Louisiana. 10.18260/1-2—44811
Paulsen, R.J. Student loan debt and the career choices of college graduates with majors in the arts. J Cult Econ 48, 95–115 (2024). https://doi.org/10.1007/s10824-023-09474-xTownsend, R. B. (2022).
What Becomes of Graduates after College?: A New Humanities Indicators Report Offers Clues. Bulletin of the American Academy of Arts and Sciences, 75(2), 14–17. https://www.jstor.org/stable/27204539

3. USE OF AUTOMATED, ELECTRONIC, MECHANICAL, OR OTHER TECHNOLOGICAL TECHNIQUES

The data for the 2025 NSCG will be collected by the U.S. Census Bureau under an interagency agreement between NCSES and the Census Bureau. The 2025 NSCG data collection will use a multi-mode approach that begins with a mailed invitation to sample persons asking them to complete the survey on the Internet. Nonrespondents will be followed up using a paper questionnaire mailing. The data will be collected and managed by the Census Bureau using multiple complementary systems, including Docuprint, Intelligent Mail Barcoding, Enterprise Internet Solutions, Adaptive Design and Intermittent Data Processing, and the Unified Tracking System. These systems are described below.

Docuprint and Intelligent Mail Barcoding

Invitation letters are produced through an in-house on-demand print process using a Docuprint system which allows personalization and the ability to tailor items to each specific respondent. The letters and questionnaire packets will be tracked using Intelligent Mail Barcoding (IMB). IMB requires separate outgoing and return barcodes to be placed on NSCG envelopes for tracking purposes. Using IMB has the potential to increase the overall efficiency of data collection by enabling the collection of detailed tracking information including:

When an outgoing questionnaire or other mail piece reaches a respondent's local post office;
When an outgoing mail piece leaves the post office with a postmaster for delivery;
If the outgoing mail piece is identified as undeliverable-as-addressed (UAA) and is being rerouted for return;
When a return questionnaire reaches a respondent’s local post office; and
When a return questionnaire reaches its destination.

This information will allow the NSCG to put cases on hold while the returned questionnaire is reviewed to determine whether it is a “good complete.” Placing cases on hold will reduce respondent burden by limiting unnecessary contacts. In addition, the IMB tracking will alert the NSCG staff to undeliverable mail pieces while they are still in circulation, allowing the Census Bureau to reduce the NSCG data collection costs by eliminating any future mailings to undeliverable addresses.

Enterprise Internet Solutions and Mobile Optimization

The Enterprise Internet Solutions (EIS) area of the Application Services Division (ASD) at the Census Bureau will host a web-based data collection instrument. Data will be transmitted and processed daily. The web instrument will be hosted on the fully certified and accredited Centurion system (infrastructure, security, and framework). The 2025 NSCG web instrument will be optimized for use on mobile devices, creating a better experience for mobile device users and, thereby, reducing survey breakoffs and the possibility of measurement errors.

Unified Tracking System

In 2025, the NSCG will continue its use of the Census Bureau’s Unified Tracking System (UTS) to assist in various aspects of survey management. Since 2013 the UTS has provided a full contact history report for the NSCG, giving survey managers a single place to view all contacts integrated from all three survey modes along with the outcomes of those contacts. This contact history reporting system enables the examination of contact strategies in various ways. For example, if respondents call in to check on the status of their response, NSCG staff are able to access the respondents’ contact history quickly and easily. In addition, this report provides an easily interpretable audit trail of all contacts, allowing survey managers to immediately verify if NSCG interviewers are following proper contact protocols, particularly when questions or complaints from respondents arise. In 2017, this contact report was enhanced by the integration of the previously mentioned IMB data.

For the 2025 NSCG, the UTS will continue to provide daily updates for R-indicators analysis at the cohort-level, so that survey management can understand how data collection operations affect representativeness. Additionally, the UTS will provide two reports to monitor IMB data. These reports will focus on the difference between the dates provided by the Census Bureau’s National Processing Center (NPC) and IMB-provided dates for survey monitoring purposes. For outgoing mailings, the report will show the lag between the scheduled mail date and when NSCG packages enter the mail stream. For incoming mailings, the report will provide the dates when UAAs or return questionnaires enter the IMB system versus when they are checked in at NPC. Both reports will have these data broken down by mailing geographies. These reports help us understand the relationship between when sample persons receive their mail and when they respond to survey requests, thus helping us anticipate response relative to mailout operations.

Finally, a UTS report that documents the interactions of the NSCG sample with the web instrument will be continued for the 2025 NSCG. This report provides information like the number of sample persons that have logged in and with what type of device, statistics about the time spent responding, and whether they logged out or submitted the survey. This report allows such valuable web paradata to be monitored throughout the data collection period.

4. EFFORTS TO IDENTIFY DUPLICATION

Duplication, in the form of a similar data collection, does not exist. No other data collection captures all components and characteristics of scientists and engineers in the United States. Data from the Current Population Survey provides occupational estimates but does not collect information on degree field for postsecondary degrees. The American Community Survey (ACS) collects the field of bachelor’s degrees but does not collect detailed information on education history, work activities, and employment characteristics as the NSCG does.

The NSCG and ACS both collect demographic information including sex, race, ethnicity, marital status, citizenship status, and veteran status. This survey content duplication between the ACS and NSCG is necessary because of the confidentiality restrictions placed on the public release of ACS data. Due to these restrictions, it is not possible for NCSES to link the demographic information from the ACS with the detailed education and employment information collected on the NSCG. Because linkage between demographic, education, and employment information is needed for the analyses used in NCSES’s congressionally mandated reports, all this information is collected on the NSCG.

Overlap does exist in the target populations for the NSCG and some of the other NCSES surveys, including the SDR and the National Training, Education, and Workforce Survey (NTEWS). As a result, it is expected there will be approximately 400 individuals selected for sample in both the 2025 NSCG and the subsequent SDR. Given the ACS sample design, in combination with the NSCG and NTEWS plans to use different ACS years for sampling purposes, it is expected there will be very few individuals selected for sample in both the 2025 NSCG and the subsequent NTEWS.

In the 2013 NSCG survey cycle, the NSCG and SDR survey contractors identified the individuals selected for both surveys, removed the individuals from the NSCG data collection effort, and, at the completion of the SDR data collection effort, used the SDR responses for these individuals to complete the individual’s record on the NSCG data file. This NSCG/SDR deduplication process required the SDR survey contractor to create numerous files containing all SDR sample cases for use by the NSCG survey contractor. Moreover, given file format and processing differences between contractors, the NSCG survey contractor needed to reformat and manually manipulate many of the SDR files to use them in combination with the NSCG files. The NSCG/SDR deduplication process added over a week of staff time to both the NSCG and SDR processing during the 2013 survey cycle.

Information collected on the NSCG questionnaire but not on the SDR includes attainment of certifications and licenses, financial support for education, community college enrollment, and veteran status. Information collected on the NSCG questionnaire but not on the NTEWS includes educational history, financial support for education, and community college enrollment. Because of the content differences, the small number of expected duplicates, and the operational challenges of the deduplication process, NCSES will not deduplicate individuals selected for sample in the NSCG with either the SDR or NTEWS during the 2025 survey cycle.

5. IMPACT ON SMALL ENTITIES

Not applicable. The NSCG collects information from individuals only.

6. CONSEQUENCES OF LESS FREQUENT COLLECTION

The NSCG data are central to the analysis presented in a pair of congressionally mandated reports published by NCSES – Science and Engineering Indicators and Women, Minorities, and Persons with Disabilities in Science and Engineering. Since these reports are published on a biennial schedule, they rely on the availability of updated data on the S&E workforce every two years. Conducting the NSCG on a less frequent basis would prohibit NCSES from meeting its congressional mandate to produce a report that contains an accurate accounting and comparison, by sex, race, and ethnic group and by discipline, of the participation of women and men in scientific and engineering positions. The impact of not being able to meet this congressional mandate is that government, business, industry, and universities would have less recent data to use as a basis for formulating the nation's science and engineering policies.

A less frequent data collection would also impact the quality of the NSCG data. Follow-up surveys every two to three years on the same sampled persons are necessary to track changes in the S&E workforce as there are large movements of individuals into and out of S&E occupations over both business and life cycles. To ensure the availability of current national S&E workforce data, the NSCG has been conducted and coordinated with the SDR on a biennial basis since 1993. The degradation of any component jeopardizes the integrity and value of these surveys to provide comprehensive information on the S&E workforce.

Finally, because the NSCG is a panel survey, conducting the survey less frequently would make it more difficult and costly to locate the sampled persons in follow-up cycles because of the mobility of the U.S. population. The likely impacts would be a higher attrition rate, higher potential for nonresponse bias, and less reliable estimates.

7. SPECIAL CIRCUMSTANCES

Not applicable. This data collection does not require any one of the reporting requirements listed.

CONSULTATIONS OUTSIDE THE AGENCY

Federal Register Announcement

The first Federal Register announcement for the 2025 NSCG appeared on 4 September 2024 (89 FR 71938, Document Number 2024-19850, pages 71938-71939, see Appendix C). NCSES received four public comments. A summary of the comments received are included below.

The American Association for the Advancement of Science (AAAS) and the American Educational Research Association (AERA) provided a joint comment discussing the necessity of the NSCG data to provide critical insights into opportunities and gaps in the STEM workforce. The commentors also encouraged the incorporation of a sexual orientation question into the 2025 NSCG survey cycle.
The Council of Professional Associations on Federal Statistics (COPAFS) provided a comment encouraging the incorporation of a sexual orientation question into the 2025 NSCG survey cycle.
The Federation of Associations in Behavioral & Brain Sciences (FABBS) provided a comment encouraging the incorporation of a sexual orientation question into the 2025 NSCG survey cycle.
An Associate Professor from Columbia University provided feedback on past SOGI research conducted by NCSES and the Census Bureau. The commentor also encouraged the incorporation of a sexual orientation question into the 2025 NSCG survey cycle.

In response, NCSES thanked the commentors for their support of the NSCG, referenced the NCSES website that summarizes past research findings, and summarized the necessity of using the Census Bureau’s ACS sexual orientation research findings, when completed, to inform NCSES’s future data collection plans.

Consultation Outside the Agency

NCSES has regularly sought the advice and guidance of survey methodologists, statisticians, demographers, researchers, data analysts, and policymakers to examine numerous issues related to the development of the NSCG. In the most recent two-year period, NCSES has partnered with the Census Bureau to explore research topics related to NSCG improvements and innovations.

Most recently, NCSES has partnered with staff at the Census Bureau’s Center for Economic Studies (CES) to continue efforts to explore supplementary sources for the NSCG’s sampling frame and to examine external data sources to inform NSCG measurement error issues. In addition, CES staff plan to conduct research into using administrative data to improve imputation and weighting procedures for the NSCG. Similarly, staff at the Census Bureau’s Demographic Statistical Methods Division (DSMD) systematically analyze paradata from the NSCG web survey instrument, using this information to improve the instrument each subsequent cycle. DSMD staff have also conducted multiple experiments to continue improving the effectiveness of the materials and strategies used to contact NSCG sample members.

The sections below summarize these efforts.

Evaluating Administrative Records as an NSCG Sampling Frame Source

At NCSES’s request, the staff at the Census Bureau’s Center for Economic Studies (CES) is continuing research examining the potential use of administrative records as a supplementary NSCG sampling frame source. The current frame, drawn from the ACS, covers the target population well but may omit some college-educated individuals. Additionally, the current use of the ACS as a frame makes the future of the NSCG dependent on future changes to the ACS, and thus identifying potential alternate frame data sources is a prudent course of action.

To date, CES has documented the strengths and weaknesses of possible frame sources and has conducted a detailed assessment of the quality and viability of two sources that show considerable promise: an extract of the National Student Clearinghouse (NSC) and the Census Bureau’s Master Address File Auxiliary Reference File (MAF-ARF).

This completed research on potential sample frame improvement has expanded into ongoing research to explore whether a new data source on college transcripts can be used to improve the NSCG sample frame. The Post-Secondary Education Outcomes (PSEO) data contains college transcript and diploma data from a sizable number of states and colleges across the country. As shown by Foote and Warren (2020),^¹⁷ who linked PSEO information on degrees obtained to the ACS, there may be non-trivial error in educational attainment information in the ACS due to imperfect imputation of this information in the event of nonresponse (or due to inaccurate reporting on the part of the household head). These errors affect the NSCG sampling frame and may result in a sample that does not fully represent the desired target population. CES staff have built on Foote and Warren (2020) to estimate numbers of individuals who appear to have been mistakenly included in or excluded from the NSCG’s sampling frame by gender and race. Future work is expected to explore how these coverage errors might change NSCG earnings estimates across these subgroups as well as degree and field types. CES anticipates this research will be conducted for 2025 NSCG data to explore whether rising fielding costs influenced the degree of measurement error in 2023 ACS data and thereby coverage error in 2025 NSCG data.

Finally, CES continues to explore new administrative and third-party sources that might improve the NSCG sample frame. Work is ongoing to acquire IRS Form 1098-T data. Additional work is ongoing to acquire a more comprehensive version of the NSC for the decennial census that CES could access for the completed NSCG sample frame improvement research. If successful, CES expects to incorporate these data into future NSCG research projects to evaluate whether these new data might serve as promising sources to improve the NSCG sample frame.

Evaluating Administrative Records to Inform Measurement Error Properties of NSCG Data

At NCSES’s request, CES is also continuing research using administrative records and third-party data sources to compare with NSCG data to inform measurement error issues for NSCG survey estimates. To date, CES has evaluated (1) earnings data and employment history data from the Census Bureau’s Longitudinal Employer-Household Dynamic (LEHD) program, IRS 1040s, IRS 1099s, and the Longitudinal Business Database (LBD) and (2) demographic data from a variety of data sources including ACS, 2010 Census, and the Census Bureau’s Numident File. CES is in the process of acquiring additional IRS data that are expected to present new opportunities for NSCG measurement error research. The results from this research will inform measurement error discussions and may provide guidance on the feasibility of using administrative records for NSCG survey replacement or supplementation. Additionally, pending resource and funding availability, longer term planning is investigating how administrative records and NSCG survey data might be combined to produce new blended data products, describing, for instance, how the income trajectories of college graduates in different STEM subfields evolve over time.

Imputing Earnings and Employment Data in the NSCG Using Administrative Data

Building upon the measurement error research described above, at NCSES’s request, CES has conducted preliminary research on using administrative data to better impute missing employment and earnings data in the NSCG. That research uses a technique called Sequential Regression Multiple Imputation (SRMI) for using the administrative data in model-based imputation. This technique does not directly replace any missing data with administrative data but does result in the new imputed data having a much better correlation with the administrative data as well as other respondent-provided earnings determinants, which should improve data utility. Such a technique has been used successfully to better impute missing data in the Survey of Income and Program Participation and Consumer Expenditure Survey, for example. Results to date will inform discussions of how the NSCG imputation model can be improved. Lastly, this project will examine how research findings could be integrated into operational processes.

Incorporating Administrative Data in Survey Weights for the NSCG

CES staff has been involved with research on incorporating administrative data in survey weights for other demographic surveys. In particular, CES staff was involved with the 2020 ACS 1-year experimental weights project,¹⁸ and remains involved in a working group to incorporate administrative data into ACS production weights which expects to complete work later this decade. Once a final, improved ACS methodology is known, with support from NCSES, CES plans to work in collaboration with other areas of Census or other NSCG consultants to conduct research on incorporating administrative data in survey weights for the NSCG to reflect improvements in the ACS methodology. Pending resource and funding availability, NCSES and CES anticipate this research will be conducted on 2025 NSCG data.

Paradata Analysis

At NCSES’s request, staff at the Census Bureau’s Demographic Statistical Methods Division (DSMD) began analyzing the NSCG web survey instrument paradata in 2013 and has continued in each subsequent NSCG cycle. The paradata includes every action the respondent takes while in the NSCG online questionnaire and the device they use. This information is used to calculate measures such as response time, changed answers, and breakoffs. This research has led to instrument changes to make the survey more intuitive and less burdensome for respondents. The changes have led to shortened completion times, reduced breakoffs, and fewer respondents moving backward in the instrument (i.e., clicking the “Previous” button). NCSES and DSMD will continue to evaluate the paradata from the 2025 NSCG to measure how device usage is changing over time and to identify and address potential usability issues with the instrument.

Contact Strategies Research

2023 NSCG Text Message Experiment

As part of the 2023 NSCG data collection effort, at the request of NCSES, the Census Bureau conducted a Text Message Experiment to measure the impact that text message reminders would have on response rate and follow-up workload. The experiment involved a control group that did not receive text messages, and a treatment group that received text messages. The experiment was designed to address the following questions:

What was the impact on response?

Were overall response rates higher when text messages were sent?
How did the timing of response and final mode distributions compare between the treatment and control groups?
Was the demographic makeup of respondents different between the treatment and control groups?

What were the operational follow-up workload impacts throughout the text message period?
1. After each text message, what proportion of sample cases opted out of receiving additional text messages?^¹⁹
2. Did the cases that opted out of receiving text messages ultimately respond to the NSCG?
3. What proportion of texts were undeliverable?
4. Did text messages lead to lower data collection costs, by reducing the number of follow-up CATI calls and mailings to nonrespondents?
5. After each text message, what proportion of cases logged into the web survey instrument that day?

The experiment was limited to returning sample members who opted in to receive text messages in 2021, did not respond by CATI (Computer Assisted Telephone Interviewing) in 2021, and did not report a CATI preference in 2021.

Analysis showed that the group receiving text messages did not respond at a higher rate and did not respond earlier than those who did not receive a text message. Additionally, the text message did not have an effect for any demographic groups, including younger respondents.

The texting operation saw a low rate of undeliverable text messages and a low opt-out rate throughout the operation. This low opt-out rate suggests that survey participants were not frustrated by the frequency of text messages, even after multiple texts. The fact that some cases responded to the survey after opting out of text messages suggests that while some respondents preferred to stop receiving text messages, it did not impact their survey participation. There was also a modest cost savings from fewer mailings of paper questionnaires. Complete results for the 2023 NSCG Text Message Experiment are included in Appendix K.

In response to experiment findings, the Census Bureau recommended additional research experimenting with the timing of text messages and the content of the message. Texts in the 2023 NSCG Text Message Experiment were sent after 5pm, and only a small percentage of sample cases logged into the web instrument the same day the text was sent. Future research should explore whether sample cases could be more inclined to log into the web instrument if they received a text earlier in the day. Additional research should examine the frequency and content of the text. In response to these findings and recommendations, NCSES is including a text message experiment in the 2025 NSCG data collection that is designed to evaluate the impact of sending text messages earlier in data collection, in an effort to reduce the number of cases receiving a paper questionnaire, as well as vary the time of day the text is sent (12pm vs 5pm). For more information on the 2025 NSCG Text Messaging Experiment, see Section B.4 and Appendices H and I.

2023 NSCG QR Code Experiment

The 2023 NSCG included a second experiment, the 2023 NSCG QR Code Experiment, to examine the impact that the inclusion of a quick response (QR) code in the NSCG mail materials would have on response rate and demographic representation. The control group was offered only a URL to access the survey instrument on their letters. The treatment group received both a QR code and a URL. To limit operational complexities, the experiment was conducted only on the new cohort sample. The goal of the experiment was to answer the following questions:

How does including a QR code affect the response rates?
Does including a QR code decrease the time to response?
Does including a QR code increase mobile device responses?
Does including a QR code increase response rates of younger people?
1. Does including a QR code affect response rates of other demographic groups?
Does including a QR code increase breakoffs?

Analysis showed that including QR codes did not increase response rate in a statistically significant manner. Nor did the data show that the treatment group completed the survey in fewer days compared to the control group. Additionally, the presence of the QR code did not alter the demographic composition of the respondents who completed the survey.

The analysis did show a statistically significant higher percentage of mobile device users in the treatment group. More users in the treatment group also spent a majority of their time in the instrument on a mobile device compared to the control group. Additionally, the treatment group had fewer breakoffs compared to the control group.

No significant effects were observed regarding other engagement activities or demographic differences. Specifically, the QR code intervention did not attract a younger demographic or expedite completion times. Complete results for the 2023 NSCG QR Code Experiment are included in Appendix L.

Related to the potential use of QR codes, some literature suggests that completing surveys on a mobile device can lead to a decrease in data quality. Concerns regarding response fatigue and the attention span of respondents were noted, as well as the possibility of issues with information clarity, reading time, and the ability of respondents to recall information. As such, NCSES will not continue the use of QR codes in the NSCG until more studies are conducted.

NCSES and the Census Bureau have disseminated information about the 2023 contact strategy research and other NSCG operations research in the following recent professional association conferences:

“The Use of QR Codes in a National, Multimode Survey” was presented at the August 2024 Joint Statistical Meetings
“Evaluation of Computer-Assisted Telephone Interviewing Outcome-Based Call Limits for the National Survey of College Graduates” was presented at the May 2024 American Association for Public Opinion Research meeting
“How can we reach you? An evaluation of text messages experiments in two surveys” was presented at the May 2024 American Association for Public Opinion Research meeting

9. PAYMENT OR GIFTS TO RESPONDENTS

The 2010 NSCG and 2013 NSCG included incentive experiments to examine the impact of offering incentives on response, data quality, and cost. The results from the incentive experiments²⁰^,²¹ provided NCSES and the Census Bureau with guidance and direction for using incentives beginning in the 2015 NSCG data collection effort. The incentive usage in the 2025 NSCG will largely follow the procedures used in all the cycles since 2015.

As was the case in the 2023 NSCG, we plan to offer a $30 prepaid debit card incentive to a subset of highly influential new sample cases at week 1 of the 2025 NSCG data collection effort. “Highly influential” refers to the cases with a large base weight and a low response/locating propensity. The highly influential cases will be identified by a model-based approach²² using a weighted response influence, which is the product of a sampled case’s base weight and predicted response propensity. The weighted response influence factor is calculated as follows:

, where .

The weighted response influence for a case, , is the product of the log of the base weight, , and the response influence, . The response influence is the inverse of the product of the locating propensity, , and the response propensity, . We will sort new sample cases in the 2025 NSCG production sample by their weighted response influence and randomly assign approximately 11,840 cases with the highest weighted response influence to receive an incentive.

In addition, using the findings from the 2013 NSCG incentive conditioning study and following our procedures from the 2015, 2017, 2019, 2021, and 2023 NSCG, we plan to offer a $30 prepaid debit card incentive to all past incentive recipients at week 1 of the 2025 NSCG data collection effort.

10. ASSURANCE OF CONFIDENTIALITY

NCSES and the Census Bureau are committed to protecting the confidentiality of all survey respondents. The NSCG data will be collected in conformance with the Privacy Act of 1974, the NSF Act of 1950, as amended, Title 13, Section 9 of the United States Code, and the Cybersecurity Enhancement Act of 2015. The Census Bureau is conducting the NSCG under the authority of Title 13, Section 8 of the United States Code.

The paper questionnaire cover will include the following confidentiality statement:

The information collected in this questionnaire is solicited under the authority of the National Science Foundation (NSF) Act of 1950, as amended. The U.S. Census Bureau is conducting this survey under the authority of Title 13, Section 8 of the United States Code. The Census Bureau is required by law to keep your information confidential and can use your responses for statistical purposes only. The Census Bureau is not permitted to publicly release your responses in a way that could identify you. Federal law protects your privacy and keeps your answers confidential (Title 13, United States Code, Section 9). Per the Federal Cybersecurity Enhancement Act of 2015, your data are protected from cybersecurity risks through screening of the systems that transmit your data. Your response is voluntary and failure to provide some or all of the requested information will not in any way adversely affect you. Actual time to complete the questionnaire may vary depending on your circumstances but on the average, it will take about 30 minutes. If you have any comments on the time required for this survey, please send them to the Reports Clearance Officer, Office of the General Counsel, National Science Foundation, 2415 Eisenhower Ave., Alexandria, VA 22314.

The cover letters will include additional statements in the Frequently Asked Questions section about the Census Bureau’s Title 13 as the data collection authority and assurances of confidentiality. The Census Bureau will include the same appropriate notices of confidentiality and the voluntary basis of the survey to respondents replying via the web instrument.

NCSES and the Census Bureau will operate within the guidelines established by the Privacy Act to protect respondents’ privacy and the confidentiality of the data collected. The Privacy Act states that “microdata files prepared for purposes of research and analysis are purged of personal identifiers and are subject to procedural safeguards to assure anonymity.”

The Census Bureau has demonstrated experience in handling sensitive data. Routine procedures will be in place to ensure data confidentiality, including the use of passwords and encrypted identifiers to prevent direct or indirect disclosures of information.

11. JUSTIFICATION FOR ASKING SENSITIVE QUESTIONS

No questions of a sensitive nature are asked in this data collection.

12. BURDEN ESTIMATE

NCSES estimates that it will contact approximately 161,000 sample persons by web or mail as part of the 2025 NSCG data collection. Based on questionnaire completion times from last cycle, it is estimated that the 2025 NSCG questionnaire will take on average, 23 minutes to complete. In fact, for returning sample members who complete by web, the average time will likely be closer to 20 minutes.

The amount of time to complete the questionnaire may vary depending on an individual’s educational history, employment status, past response to the NSCG, and selected survey mode. The questionnaire for new sample members is slightly longer than the two forms for returning sample members because it includes all the demographic questions and questions about one’s educational history. The time to complete the 2023 NSCG web survey ranged from 19.6 minutes for some returning sample members to 26.3 minutes for new sample members, and approximately 91% of respondents completed the web mode. Among respondents who completed the survey using Computer Assisted Telephone Interviewing (CATI) and Telephone Questionnaire Assistance (TQA) modes, in which sample members contacted Census’s call center and took the survey over the phone, their interview times ranged from 33.6 minutes for returning sample members to 44.3 minutes for new sample members, and less than 2% of respondents completed the NSCG questionnaire by using the telephone mode. It was estimated that all forms of the 2023 NSCG paper questionnaire took about 30 minutes to complete, and roughly 7% of respondents completed the paper form.

NCSES expects the response rate to be 60 to 65 percent. Based on an estimate of approximately 104,650 completed cases (161,000 x .65), the total burden hours for the 2025 NSCG data collection are 40,116 (104,650 x 23 minutes). The total cost to respondents for the 40,116 burden hours is estimated to be $1,639,541. This estimate is based on an estimated median annual salary of $85,000 per NSCG employed respondent.²³ Assuming a 40-hour workweek and a 52-week salary, this annual salary translates to an hourly salary of $40.87. Over the three-year OMB clearance period, the average annual cost to the public for the 2025 NSCG is estimated to be $546,513.

13. COSTS TO RESPONDENTS

Not applicable. The NSCG collection does not have a recordkeeping requirement that would impose additional costs (e.g., purchases equipment, software, or contract out services) to respondents.

14. COSTS TO THE FEDERAL GOVERNMENT

The total estimated cost to the Government for the 2025 NSCG is approximately $16.4 million, which includes survey cycle costs, and NCSES staff costs to provide oversight of the NSCG and coordination with the SDR. The estimate for survey cycle costs is approximately $15.9 million, which is based on sample size; length of questionnaire; administration; overhead; sample design; mailing; printing; sample person locating; web instrument development; incentive payments; data keying and editing; quality control; imputation for missing item responses; weighting and estimating sampling error; file preparation and delivery; preparation of documentation, data tables, and final reports. The NCSES staff costs are estimated at $562,500 (based on $150,000 annual salary of 1.5 FTE for 2.5 years). Over the three-year OMB clearance period, the average annual cost to the Government is estimated to be $5,480,900.

15. CHANGES FROM THE PRIOR CYCLE

The sample size is remaining the same in 2025 as it was in 2023 – 161,000.

PLANS FOR TABULATION OR PUBLICATION

NCSES does not plan to use any complex analytical techniques in publications using this data. Normally cross-tabulations of the data are presented in NCSES reports and other data releases.

The time schedule for 2025 data collection and publication is currently estimated as follows:

Data Collection	March 2025 – October 2025
Coding and Data Editing	June 2025 – December 2025
Final Edited/Weighted/Imputed Data File	March 2026
NSCG InfoBrief	Fall 2026
NSCG Public-Use Data File	Fall 2026

EXCEPTION TO DISPLAYING THE OMB EXPIRATION DATE

Not applicable. The OMB Control Number and expiration date will be displayed.

18. EXCEPTION TO THE CERTIFICATION STATEMENT

Not applicable. No exceptions to the certification statement are being sought.

1 Section 505, Pub. L. No. 111-358. See Appendix A.

2 The SDR is a repeated cross-sectional biennial survey that provides demographic and career history information about individuals with a research doctoral degree in a science, engineering, or health field from a U.S. academic institution. For more information, see http://www.nsf.gov/statistics/srvydoctoratework.

3 The S&E workforce includes individuals with degrees or occupations in computer and mathematical sciences, life sciences, physical sciences, social sciences, engineering, health sciences and related fields.

4 See Appendix B.

5 42 U.S. Code § 1863(j)(1)

6 42 U.S. Code § 1885(d)

7 42 U.S. Code § 1885(d)

8 https://law.vanderbilt.edu/job-satisfaction-lower-among-black-women-and-asian-women-law-graduates/

9 https://www.edweek.org/teaching-learning/students-pay-a-growing-price-for-landing-a-job-outside-their-college-major/2023/04

10 https://www.forbes.com/sites/stuartanderson/2024/06/18/h-1b-rule-expected-later-this-year-immigration-restrictions-possible/

11 https://www.bls.gov/opub/mlr/2024/beyond-bls/the-impact-of-double-majors-during-economic-downturns.htm

12 https://cis.org/Report/Value-Foreign-Degrees-Source-Country

13Weiss, D.M., Spitzer, M.L., Cronin, C. & Chin, N. (2024) Why college majors and selectivity matter: major groupings, occupation specificity, and job skills. Contemporary Economic Policy, 42(2), 278–304. Available from: https://doi.org/10.1111/coep.12634

14 https://ncses.nsf.gov/pubs/nsf23315

15 https://ncses.nsf.gov/pubs/nsb20243

16 https://ncses.nsf.gov/pubs/nsb20245

17 Foote, Andrew & Larry Warren. 2020. “It’s All a Matter of Degrees: Comparing Survey and Administrative Educational Attainment Data.” Available: https://appam.confex.com/appam/2020/mediafile/ExtendedAbstract/Paper35696/FooteWarren%20October2020%20ACS%20Merge%20Draft.pdf

18 See: Rothbaum, Jonathan, Jonathan Eggleston, Adam Bee, Mark Klee, & Brian Mendez-Smith. 2021. "Addressing Nonresponse Bias in the American Community Survey During the Pandemic Using Administrative Data." 2021 AMERICAN COMMUNITY SURVEY RESEARCH AND EVALUATION REPORT MEMORANDUM SERIES #ACS21-RER-05 and SEHSD Working Paper #2021-24.

19 Sample cases could opt out of receiving additional text messages by replying STOP to the text reminder.

20 Zotti, Allison, “Report for the 2013 National Survey of College Graduates Methodological Research Incentive Timing Experiment,” Census Bureau Memorandum from Reist to Finamore and Rivers, April 15, 2014, draft.

21 Thornton, Thomas, “2013 National Survey of College Graduates (NSCG) Incentive Conditioning Study,” Census Bureau Memorandum from Reist to Finamore and Rivers, April 15, 2014, draft.

22 This model-based approach enables the allocation of incentives at a rate that exceeds their population representation to critical populations of interest within the S&E workforce including underrepresented minorities and individuals with disabilities. This approach helps mitigate the potential for nonresponse bias resulting from the lower survey participation rates for these groups. Among all race/ethnicity groups included within the NSCG sample, approximately 32% of the incentives are offered to underrepresented minorities. Among all sample cases, approximately 10% of the incentives are offered to individuals with disabilities.

23Salary estimates were obtained using data from the 2023 NSCG.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Milan, Lynn M.
File Modified	0000-00-00
File Created	2025-01-16

2025 NSCG SupportingStatement_A