2015 NSCG Supporting Statement - Section A

SF-83-1 SUPPORTING STATEMENT
for the
2015
National Survey of College Graduates

CONTENTS
Section

Page

A.

JUSTIFICATION ..............................................................................................................1
1.
NECESSITY FOR INFORMATION COLLECTION ............................................1
2.
USES OF INFORMATION.....................................................................................3
3.
CONSIDERATION OF USING IMPROVED TECHNOLOGY............................6
4.
EFFORTS TO IDENTIFY DUPLICATION ...........................................................8
5.
EFFORTS TO MINIMIZE BURDEN ON SMALL BUSINESS............................9
6.
CONSEQUENCES OF LESS FREQUENT DATA COLLECTION .....................9
7.
SPECIAL CIRCUMSTANCES ...............................................................................9
8.
FEDERAL REGISTER ANNOUNCEMENT AND CONSULTATION
OUTSIDE THE AGENCY ....................................................................................10
9.
PAYMENT OR GIFTS TO RESPONDENTS ......................................................14
10.
ASSURANCE OF CONFIDENTIALITY.............................................................16
11.
JUSTIFICATION FOR SENSITIVE QUESTIONS .............................................16
12.
ESTIMATE OF RESPONDENT BURDEN .........................................................16
13.
COST BURDEN TO RESPONDENTS ................................................................17
14.
COST BURDEN TO FEDERAL GOVERNMENT..............................................17
15.
REASON FOR CHANGE IN BURDEN ..............................................................17
16.
SCHEDULE FOR INFORMATION COLLECTION AND
PUBLICATION .....................................................................................................18
17.
DISPLAY OF OMB EXPIRATION DATE ..........................................................18
18.
EXCEPTION TO THE CERTIFICATION STATEMENT ..................................18

B.

COLLECTION OF INFORMATION EMPLOYING STATISTICAL
METHODS .......................................................................................................................19
1.
RESPONDENT UNIVERSE AND SAMPLING METHODS .............................19
2.
SURVEY METHODOLOGY ...............................................................................20
3.
METHODS TO MAXIMIZE RESPONSE ...........................................................26
4.
TESTING OF PROCEDURES ..............................................................................28
5.
CONTACTS FOR STATISTICAL ASPECTS OF DATA COLLECTION .........37
APPENDIX A: America COMPETES Reauthorization Act of 2010
APPENDIX B: NSF Act of 1950
APPENDIX C: First Federal Register Announcement
APPENDIX D: 2015 NSCG Coefficient of Variation (CV) Target
APPENDIX E: NSCG Survey Mailing Materials
APPENDIX F: Draft 2015 NSCG Questionnaires
APPENDIX G: 2015 NSCG Default Data Collection Pathway
APPENDIX H: Adaptive Design Goals, Interventions, and Monitoring Metrics
APPENDIX I: 2015 NSCG Experiments – Minimum Detectible Differences
APPENDIX J: Recent Changes to the NSCG Design

2015 NATIONAL SURVEY OF COLLEGE GRADUATES
SUPPORTING STATEMENT
A.

JUSTIFICATION

This request is for a three-year renewal of the previously approved Office of Management and
Budget (OMB) clearance for the National Survey of College Graduates (NSCG). The NSCG has
historically served as a valuable source of information on the education and career paths of the
Nation’s college-educated population. The most recent NSCG was conducted in 2013 (OMB
approval number 3145-0141). The current OMB clearance for the NSCG expires November 30,
2015, which does not cover the complete survey cycle for the 2015 NSCG.
The NSCG introduced substantial design changes during the 2010 and 2013survey cycles, with
additional changes planned for the 2015 cycle. These NSCG design changes are described in
Appendix J of this document. As part of planning effort for the 2015 NSCG survey cycle, the
full set of NSCG questionnaire items underwent an evaluation that included an expert review and
cognitive interviews. As a result, a new questionnaire section on certifications and licenses is
being added as well as minor question wording revisions to numerous items throughout the
NSCG questionnaire. The specific revisions are discussed in section B.4. of this report.
1.

NECESSITY FOR INFORMATION COLLECTION

In 2010, the America COMPETES Reauthorization Act of 2010 1 established the National Center
for Science and Engineering Statistics (NCSES) at the National Science Foundation (NSF) and
directed NCSES to “...collect, acquire, analyze, report, and disseminate statistical data related to
the science and engineering enterprise in the United States and other nations that is relevant and
useful to practitioners, researchers, policymakers, and the public...” Information obtained
through the NSCG is critically important to NCSES’s ability to measure the education and
employment of scientists and engineers. Furthermore, the NSCG is the centerpiece of NCSES’s
statistical data system that produces the nation’s only source of comprehensive information about
the size and characteristics of the science and engineering (S&E) workforce 2. These data are
solicited under the authority of the NSF Act of 1950 3, as amended, and are central to the analysis
presented in a pair of congressionally mandated reports 4,5 published by NSF – Science and
Engineering Indicators and Women, Minorities, and Persons with Disabilities in Science and
Engineering.
In addition, the Science and Engineering Equal Opportunities Act of 1980 directs NSF to provide
to Congress and the Executive Branch an “accounting and comparison by sex, race, and ethnic
1

Section 505, Pub. L. No. 111-358. See Appendix A.
The S&E workforce includes the individuals with degrees or occupations in computer and mathematical
sciences, life sciences, physical sciences, social sciences, engineering, and health sciences.
2
3
4
5

See Appendix B.

42 U.S. Code § 1863(j)(1)
42 U.S. Code § 1885(a), 1885(d)

1

group and by discipline, of the participation of women and men in scientific and engineering
positions.” 6 NCSES’s Science and Engineering Statistical data system (SESTAT), of which the
NSCG comprises a large majority of records, provides much of the information to meet this
mandate. The SESTAT system of surveys, created for the 1993 survey cycle and developed
throughout the past two decades, is closely based on the recommendations of the National
Research Council’s Committee on National Statistics (CNSTAT) report to NSF. 7
NSCG Background
The NSCG provides data on the nation’s college graduates, with particular focus on those in the
S&E workforce. The NSCG samples individuals who are living in the United States, have at
least a bachelor’s degree, and are under the age of 76. This survey is a unique source for
examining various characteristics of college-educated individuals, including occupation, work
activities, salary, the relationship of degree field and occupation, and demographic information.
The NSCG and the Survey of Doctorate Recipients (SDR) are the two surveys that provide data
for the NCSES’s SESTAT data system. The purpose of SESTAT is to provide information on
the entire U.S. population of scientists and engineers with at least a bachelor’s degree.
Historically, the SESTAT surveys have been conducted every two to three years. The integrated
SESTAT data system provides longitudinal data on the education and employment of the
college-educated U.S. science and engineering workforce. SESTAT is the only available source
for detailed information to support a wide variety of policy and research analyses on the S&E
workforce and personnel.
The NSCG constitutes approximately 75% of the records in SESTAT and slightly over 97% of
the 2013 SESTAT population estimate. As the core of SESTAT, the NSCG provides
information on individuals educated or employed in S&E fields including individuals who
received degrees only from foreign institutions. The SDR supplements SESTAT with the stock
and inflow of U.S.-degreed doctoral level scientists and engineers. Through 2010, the National
Survey of Recent College Graduates (NSRCG) supplemented SESTAT with the inflow of U.S.degreed bachelor's and master's level scientists and engineers. Beginning in 2013, the NSCG
began capturing the bachelor’s and master’s level inflow population and eliminated the need for
the NSRCG. As a result, the NSRCG was discontinued after the 2010 survey.
The longitudinal data from the NSCG provides valuable information on careers, training, and
educational development of the nation’s college graduate population. These data enable
government agencies to assess the scientific and engineering resources available in the U.S. to
business, industry, and academia, and to provide a basis for the formulation of the nation's
science and engineering policies. Educational institutions use the NSCG data in establishing and
modifying scientific and technical curricula, while various industries use the information to
develop recruitment and remuneration policies.

6

42 U.S. Code § 1885(d)
National Research Council, Committee on National Statistics. 1989. Surveying the Nation’s Scientists
and Engineers: A Data System for the 1990s. Washington: National Academy Press.
7

2

2.

USES OF INFORMATION

Policymakers, researchers, and other data users use information from SESTAT to answer
questions about the number, employment, education, and characteristics of the S&E workforce.
Because it provides up-to-date and nationally representative data, policymakers and researchers
use the data system to address questions on topics such as the role of foreign-born or foreigndegreed scientists and engineers, the transition from higher education to the workforce, the role
and importance of postdocs, diversity in both education and employment, the implications of an
aging cohort of scientists and engineers as baby boomers reach retirement age, and information
on long-term trends in the S&E workforce.
Uses for Policy Discussion
Data from NCSES’s SESTAT component surveys are used in policy discussions of the executive
and legislative branches of Government, the National Science Board, NSF management, the
National Academy of Sciences, professional associations, and other private and public
organizations. Some recent specific examples of the use of the NSCG and SESTAT data are:
•

The Executive Office of the President used NSCG data to examine the contributions of
immigrants in S&E occupations 8;

•

The National Science Board (NSB) used SESTAT data in its investigation to develop
national policies for the S&E workforce 9;

•

The importance of information on the S&E workforce to inform public policy can be seen
in discussions of the NSB’s Task Group on Science, Technology, Engineering, and Math
(STEM) Innovators. The task group used SESTAT data to inform its deliberations about
the S&E workforce and SESTAT data were part of the final report 10;

•

The Committee for Equal Opportunity in Science and Engineering (CEOSE), an advisory
committee to NSF and other government agencies, established under 42 U.S.C. §1885c,
has been charged by the U.S. Congress with advising NSF in assuring that all individuals
are empowered and enabled to participate fully in science, mathematics, engineering and
technology. Every two years CEOSE prepares a congressionally mandated report that
makes extensive use of the SESTAT data to highlight key areas of concerns relating to
students, educators and technical professionals.

•

The Council of Graduate Schools (CGS) used NSCG data to estimate the potential
monetary cost and return on investment of pursuing advanced degrees 11, which is a key
element of CGS’s financial education website – www.gradsense.org; and

•

The Educational Testing Service (ETS) and CGS used SESTAT data to examine national
benchmarks for career outcomes of master’s and doctoral degree recipients by specific
field 12.

8

http://www.whitehouse.gov/sites/default/files/docs/report.pdf
http://www.nsf.gov/nsb/documents/2003/nsb0369/nsb0369.pdf
10
http://www.nsf.gov/nsb/publications/2010/nsb1033.pdf
11
http://www.gradsense.org/gradsense/methodology
12
http://www.ets.org/c/19574/19089_PathwaysReptqp.pdf
9

3

Uses by NSF
The SESTAT data were used extensively in the latest versions of the congressionally mandated
biennial reports Science and Engineering Indicators, 2014 and Women, Minorities and Persons
with Disabilities in Science and Engineering, 2015. In addition, Science and Engineering
Indicators, 2016, set for release in January 2016, will use SESTAT data.
NSF used the NSCG and SESTAT integrated data in recent reports such as:
•

Employment Decisions of U.S. and Foreign Doctoral Graduates: A Comparative Study,
December 2014

•

Unemployment among Doctoral Scientists and Engineers Remained Below the National
Average in 2013, September 2014

•

Employment and Educational Characteristics of Scientists and Engineers, January 2013

•

International Mobility and Employment Characteristics among Recent Recipients of U.S.
Doctorates, October 2012

•

International Collaboration of Scientists and Engineers in the United States, August
2012

•

Diversity in Science and Engineering Employment in Industry, March 2012

•

Racial and Ethnic Diversity among U.S.-Educated Science, Engineering, and Health
Doctorate Recipients: Methods of Reporting Diversity, January 2012

•

Community Colleges: Playing an Important Role in the Education of Science,
Engineering, and Health Graduates, July 2011

•

The End of Mandatory Retirement for Doctoral Scientists and Engineers in
Postsecondary Institutions: Retirement Patterns 10 Years Later, December 2010

•

Foreign Science and Engineering Students in the United States, July 2010

All NSF Publications can be accessed on the NCSES website at
http://www.nsf.gov/statistics/reports.cfm.
Uses by Researchers and Analysts
NCSES makes the data from the SESTAT surveys available through published reports, the
SESTAT online data tool, downloadable public use files, and restricted-use licenses. The
SESTAT online data tool, available at http://ncsesdata.nsf.gov/sestat/sestat.html, allows users to
create customized data tabulations with a user-specified subject area. The NSCG and SESTAT
public-use files are available for download through the NCSES data downloads web page at
http://ncesdata.nsf.gov/download.

4

Since 2005 13, NCSES has distributed over 600 copies of the more than decade-old 1993 NSCG
public-use data set and over 1,300 copies of the 2003 NSCG public-use files to researchers in
government, academia, and professional societies. And, since its release in January 2013, over
800 copies of the 2010 NSCG public-use files have been downloaded from the NCSES data
downloads page. The 2013 NSCG data are in the final stages of data review and will be
available later this year as a standalone public-use file. The NSCG public-use files receive heavy
use because they are the only data sets analysts can use to compare the S&E workforce to the
general population of college degree holders in the U.S.
The SESTAT public-use files have been downloaded from the NCSES data downloads page over
5,000 times since 2005. In addition to the users of the SESTAT public-use files, there are
currently 28 restricted-use licensees with access to the SESTAT integrated micro data files under
a licensing agreement with NCSES. As previously noted, the majority of the records in the
SESTAT file come from the NSCG.
Some of the research based on the public-use NSCG data, the public-use SESTAT data, and the
restricted-use SESTAT data resulted in papers such as:
•

Trends in Earnings Differentials across College Majors and the Changing Task
Composition of Jobs, Yale University, 2014

•

Are Asian American Women Advantaged? Labor Market Performance of College
Educated Female Workers, Kansas University, 2014

•

Opting Out among Women with Elite Education, Vanderbilt University, 2013

•

Startups by Recent University Graduates and their Faculty: Implications for University
Entrepreneurship Policy, Carnegie Mellon University, 2012

•

Findings from an Examination of the Labor Force Participation of College-Educated
Immigrants in the United States, Department of Education, 2012

•

Evolution of Gender Differences in Post-Secondary Human Capital Investments: College
Majors, New York University, 2011

•

Earning Trajectories of Highly Educated Immigrants: Does Place of Education Matter?,
Cornell University, 2011

•

Which Immigrants are Most Innovative and Entrepreneurial? Distinctions by Entry Visa,
National Bureau of Economic Research, 2011

•

Labor Market Penalties for Foreign Degrees Among College Educated Immigrants,
University of Minnesota, 2010

13

The America COMPETES Reauthorization Act of 2010 mandated the name and
responsibilities of NCSES. Prior to 2010, the organizational unit that would become NCSES
was referred to as the National Science Foundation’s (NSF) Division of Science Resource
Statistics (SRS). For simplicity, NCSES will be used throughout this report when referring to
work completed by SRS or NCSES.

5

•

Do Teachers have Education Degrees? Matching Fields of Study to Popular
Occupations of Bachelor’s Degree Graduates, Indiana University, 2010

•

Why Do Women Leave Science and Engineering?, National Bureau of Economic
Research, 2010

•

Functional Impairment and the Choice of College Major, University of South Florida,
2010

•

How Much Does Immigration Boost Innovation?, McGill University, 2010

•

Increasing Time to Baccalaureate Degree in the United States, National Bureau of
Economic Research, 2010

•

Higher Education and Disability: Education Needs a Coordinated Approach to Improve
Its Assistance to Schools in Supporting Students, GAO Report, 2009

•

Diversifying Science and Engineering Faculties: Intersections of Race, Ethnicity, and
Gender, Georgia Institute of Technology, 2010

3.

CONSIDERATION OF USING IMPROVED TECHNOLOGY

The data for the 2015 NSCG will be collected by the U.S. Census Bureau under an interagency
agreement between NCSES and the Census Bureau. The 2015 NSCG data collection will use a
multi-mode approach that begins with a web invitation letter mailed to sample persons asking
them to complete the survey on the Internet. Nonrespondents will be followed up using a paper
questionnaire mailing and computer assisted telephone interviews (CATI). The data will be
collected and managed by the Census Bureau using multiple complementary systems including:
Docuprint, Intelligent Mail Barcoding, Enterprise Internet Solutions, Adaptive Design and Daily
Processing, and the Unified Tracking System. These systems are described below.
Mail Materials
Web invitation letters are produced through an in-house on-demand print process using a
Docuprint system which allows personalization and the ability to tailor items to each specific
respondent. A new addition for the 2015 NSCG cycle is that letters and questionnaire packets
will be tracked using Intelligent Mail Barcoding (IMB). IMB requires separate outgoing and
return barcodes to be placed on NSCG envelopes for tracking purposes. Using IMB has the
potential to increase the overall efficiency of data collection enabling the collection of detailed
tracking information including:
•

When an outgoing questionnaire or other mail piece reached a respondent's local post
office;

•

When an outgoing mail piece left the post office with a postmaster for delivery;

•

If the outgoing mail piece was identified as undeliverable-as-addressed (UAA) and is
being rerouted for return;

•

When a return questionnaire reaches a respondent’s local post office; and

6

•

When a return questionnaire reaches its destination.

This information will allow the NSCG to put cases on hold while the returned questionnaire is
reviewed to determine whether it is a “good complete.” Placing cases on hold will reduce
respondent burden by limiting unnecessary contacts. In addition, the IMB tracking will alert the
NSCG staff to undeliverable mail pieces while they are still in circulation, allowing the Census
Bureau to reduce the NSCG data collection costs by eliminating any future mailings to
undeliverable addresses.
Archiving
Images will be scanned and archived on a secured server in case they are needed later. This
eliminates the need to save paper copies of the completed questionnaires.
Data from a Web Instrument
The Enterprise Internet Solutions (EIS) area of the Application Services Division (ASD) at the
Census Bureau will host a web-based data collection instrument. Data will be transmitted and
processed daily. The web application will be hosted on the fully certified and accredited
Centurion system (infrastructure, security, and framework). New to the 2015 NSCG, the
Telephone Questionnaire Assistance (TQA) and Email Questionnaire Assistance (EQA) systems
will utilize the Centurion system. This enhancement to the TQA and EQA systems will enable
faster data review and a seamless transition from telephone assistance to data collection for the
TQA interviewing staff.
Adaptive Design and Daily Processing
The 2015 NSCG will continue to expand the scope of adaptive design in an effort to attain highquality survey estimates in less time and at less cost than traditionally executed survey
operations. First, the Census Bureau will implement daily processing (editing, imputation,
weighting) of the response data throughout the data collection period. In addition to operational
efficiencies, daily processing will allow the NSCG survey team to monitor several quality
measures throughout data collection, including R-indicators, benchmarking, stability of
estimates, and response propensities by mode.
Second, the 2015 NSCG will include an adaptive design experiment that aims to document the
adaptive design goals most appropriate for NSCG, and in turn, identify appropriate data
collection interventions and the monitoring methods that can be used to drive those interventions.
More detail about the 2015 NSCG adaptive design experiment is provided in section B.4. of this
report. A larger sample size in the adaptive design experiment for the new sample cases will
provide the statistical power to make more definitive statements about statistical differences
between the treatment group and the control group on various measures, including response
rates, R-indicators, cost, and effect on key estimates. Including returning sample members as
part of the adaptive design experiment will provide insight into whether adaptive design is an
appropriate technique to use with returning sample cases in future NSCG rounds.
Unified Tracking System
In 2015, the NSCG will be expanding its use of the Census Bureau’s Unified Tracking System
(UTS) to assist in various aspects of survey management. As in 2013, the UTS will provide a

7

full contact history report for the NSCG, giving survey managers a single place to view all
contacts integrated from all three modes in the NSCG along with the outcomes of those
contacts. This report was useful in 2013 for following up on contact strategies in a number of
ways. As an example, if respondents called in to check on the status of their response, NSCG
staff were able to quickly and easily access the respondents’ contact history and outcomes to
provide the current status of their response. In addition, this report provides an easily accessible
and interpretable audit trail of all contacts, allowing survey managers to immediately verify if
NSCG interviewers are following proper contact protocols, particularly when questions or
complaints from respondents arise. For 2015, this contact report will also integrate the
previously mentioned IMB data.
The 2015 NSCG will utilize two additional survey management reports: a cases-on-hold report
and a TQA-completes report. The cases-on-hold report will provide information about which
cases are on hold in CATI for any given day as well as the reason they are on hold (e.g., due to a
paper or web questionnaire response in processing, or due to adaptive design data
monitoring). The TQA completion report provides up-to-date information about which
telephone interviewers are assisting respondents in completing the survey from an incoming
call. Both of these reports will help the telephone centers at the Census Bureau better understand
and manage the NSCG workload.
4.

EFFORTS TO IDENTIFY DUPLICATION

Duplication, in the sense of similar data collections, does not exist. No other data collection
captures all components of scientists and engineers in the United States. There is no similar
information available other than from this survey, conducted by the U.S. Census Bureau for NSF
since the 1960s. Data from the Current Population Survey provides occupational estimates but
does not collect information on degree field for higher education degrees. The American
Community Survey (ACS) collects the field of bachelor’s degrees but does not collect detailed
information on education history, work activities, and employment characteristics as the NSCG
does, nor is the ACS longitudinal in nature.
Overlap does exist in the target populations for the NSCG and the SDR. As a result, it is
expected there will be between 100-200 individuals selected for sample in both the 2015 NSCG
and the 2015 SDR.
In the 2013 NSCG survey cycle, the NSCG and SDR survey contractors identified the
individuals selected for both surveys, removed the individuals from the NSCG data collection
effort, and, at the completion of the SDR data collection effort, used the SDR responses for these
individuals to complete the individual’s record on the NSCG data file. This NSCG/SDR
deduplication process required the SDR survey contractor to create numerous files containing all
SDR sample cases for use by the NSCG survey contractor. Furthermore, given file format and
processing differences between contractors, the NSCG survey contractor needed to reformat and
manually manipulate many of the SDR files to use them in combination with the NSCG files.
The NSCG/SDR deduplication process adds over a week of staff time to both the NSCG and
SDR processing.

8

Given recent changes to the NSCG questionnaire content, there are noticeable differences in the
information collected on the NSCG and SDR. Examples of topics planned for collection on the
2015 NSCG, but not on the 2015 SDR include attainment of certifications and licenses, financial
support for education, and community college enrollment. Because of the content differences,
the small number of expected duplicates, and the operational challenges of the deduplication
process, NCSES will not deduplicate individuals selected for sample in both the NSCG and SDR
in the 2015 survey cycle.
5.

EFFORTS TO MINIMIZE BURDEN ON SMALL BUSINESS

Not applicable. The NSCG collects information from individuals only.
6.

CONSEQUENCES OF LESS FREQUENT DATA COLLECTION

The NSCG and SESTAT data are central to the analysis presented in a pair of congressionally
mandated reports published by NSF – Science and Engineering Indicators and Women,
Minorities, and Persons with Disabilities in Science and Engineering. Since these reports are
published on a biennial schedule, they rely on the availability of updated data on the S&E
workforce every two years. Conducting the NSCG on a less frequent basis would prohibit NSF
from meeting its congressional mandate to produce a report that contains an accurate accounting
and comparison, by sex, race, and ethnic group and by discipline, of the participation of women
and men in scientific and engineering positions. The impact of not being able to meet this
congressional mandate is that government, business, industry, and universities would have less
recent data to use as a basis for formulating the nation's science and engineering policies.
A less frequent data collection would also impact the quality of the NSCG data. Follow-up surveys
every two to three years on the same sampled persons are necessary to track changes in the S&E
workforce as there are large movements of individuals into and out of S&E occupations over
both business and life cycles. To ensure the availability of current national S&E workforce data,
the NSCG has been conducted and coordinated with the NSRCG and the SDR on a biennial basis
since 1993. The degradation of either component jeopardizes the integrity and value of the entire
SESTAT data system.
Finally, because the NSCG is a panel survey, conducting the survey less frequently would make
it more difficult and costly to locate the persons in the sample because of the mobility of the U.S.
population. The impact would be a higher attrition rate, higher potential for nonresponse bias,
and less reliable estimates.
7.

SPECIAL CIRCUMSTANCES

Not applicable. This data collection does not require any one of the reporting requirements
listed.

9

8.

FEDERAL REGISTER ANNOUNCEMENT AND CONSULTATION OUTSIDE
THE AGENCY

Federal Register Announcement
The Federal Register announcement for the NSCG appeared on July18, 2014 (See Appendix C).
NSF received no public comment in response to the announcement as of the close date of
September 16, 2014.
Consultation Outside the Agency
NCSES has sought the advice and guidance of survey methodologists, statisticians,
demographers, researchers, data analysts, and policymakers to examine numerous issues related
to the development of the NSCG and the SESTAT data system.
•

Survey Content
As part of the 2015 NSCG planning effort, NCSES conducted developmental work on new
questionnaire items to capture information on alternative credentials including industryrecognized certifications, occupational licenses, and educational certificates. As a starting
point for this developmental work, NCSES used the vast amount of research on this topic
conducted by the Interagency Working Group on Expanded Measures of Enrollment and
Attainment (GEMEnA). GEMEnA is a collaboration among federal statistical agencies
established by the OMB Office of Statistical and Science Policy, the Council of Economic
Advisors, and the Under Secretary of Education to improve federal data on the attainment of
non-degree credentials.
Survey Methodology Experts: GEMEnA Member Agency Representatives
Census Bureau
Bob Kominski
Stephanie Ewert

OMB Office of Statistical and Science Policy
Shelly Martinez
Department of Education
Jon O’Bergh

Bureau of Labor Statistics
Dori Allard
Harley Frazis

National Center for Education Statistics
Sharon Boivin
Lisa Hudson
Kashka Kubzdela
Sarah Crissey
Sarah Carroll
Andy Zukerberg

NCSES – National Science Foundation
Dan Foley
John Finamore
Council of Economic Advisors
Jordan Matsudaira

In order to maintain the currency of the SESTAT survey content and to obtain ongoing input
from the public and researchers, NCSES has convened four Human Resources Experts Panel
(HREP) workshops. At the HREP workshops, panel members are asked to contribute to the
continued success of the SESTAT surveys by 1) Suggesting methods to publicize and
promote the SESTAT data; 2) Providing advice on efforts to improve the timeliness and

10

accuracy of S&E workforce data; 3) Providing a mechanism for obtaining ongoing input
from both researchers and policy analysts interested in S&E personnel data; 4) Providing
perspectives on the data needs of policy makers; 5) Identifying issues and trends that are
important for maintaining the relevance of the data; 6) Identifying ways in which S&E
personnel data could be more useful and relevant for analyses; and 7) Proposing ways to
enhance the content of the NCSES human resources surveys.
Three HREP workshops were held over the past year to discuss survey content issues
relevant to the NSCG and SESTAT. The August 2013 panel discussed issues related to the
collection and value of data on education and career pathways. The January 2014 panel
continued the education and career pathways discussion, but also included time discussing
alternative credentials like certifications, licenses, and educational certificates. The June
2014 panel focused on the concepts of job mobility and occupational change.
HREP Members – August 2013 and January 2014 Workshops
Nathan Bell
Associate Director, Education Research & Policy
American Educational Research Association

Brian Hartz
Vice President of Client Services
TORQworks

Roman Czujko
Director, Statistical Research Center
American Institute of Physics

Beverly Karplus Hartline
Vice Chancellor for Research and Graduate Studies
Montana Tech

Ronni Denes
President and Executive Director
New Jersey SEEDS

Cheryl Leggon
Associate Professor, School of Public Policy
Georgia Institute of Technology

Catherine Didion
Senior Program Officer
National Academy of Engineering
Director, Committee on Women in S&E
National Academies

Sharon Levin
Professor of Economics
University of Missouri, St. Louis
Duncan McBride
Program Director, Division of Undergrad Ed.
National Science Foundation

Earnestine Psalmonds Easter
Program Director, Division of Graduate Education
National Science Foundation

Catherine Millett
Research Scientist
Educational Testing Service

Cary Funk
Senior Researcher
Pew Research Center

Cathee Johnson Phillips
Executive Director
National Postdoctoral Association

Donna Ginther
Professor of Economics
University of Kansas

George Wimberly
Director, Professional Development/Social Justice
American Educational Research Association

11

HREP Members – June 2014 Workshop

•

Jake Bartolone
Senior Research Scientist
National Opinion Research Center

Albert Sumell
Associate Professor of Economics
Youngstown State University

Kirk Doran
Assistant Professor of Economics
University of Notre Dame

Omari Swinton
Assistant Professor of Economics
Howard University

Donna Ginther
Professor of Economics
University of Kansas

John Bound
Professor of Economics
University of Michigan

Shulamit Kahn
Associate Professor of Public Policy & Law
Boston University

Charlie Brown
Professor of Economics
University of Michigan

Morris Kleiner
Professor of Public Affairs/Industrial Relations
University of Minnesota

Pamela Herd
Professor of Public Affairs and Sociology
University of Wisconsin-Madison

Iourii Manovskii
Associate Professor of Economics
University of Pennsylvania

Sheila Kirby
Senior Fellow
National Opinion Research Center

Erika McEntarfer
Supervisory Economist
U.S. Census Bureau

Cheryl Leggon
Associate Professor, School of Public Policy
Georgia Institute of Technology

Donna Rothstein
Research Economist
Bureau of Labor Statistics

Audrey Light
Professor of Economics
Ohio State University

Hal Salzman
Professor of Planning and Public Policy
Rutgers, The State University of New Jersey

Mike Pergamit
Senior Fellow
Urban Institute

Marc Scott
Associate Professor of Applied Statistics
New York University

Jeff Strohl
Senior Research Fellow
Georgetown University

John Skrentny
Professor of Sociology
University of California at San Diego

Josh Trapani
Director of Policy Analysis
Association of American Universities

Questionnaire Evaluation and Instrument Usability
The staff at the Census Bureau’s Center for Survey Measurement conducted an expert review
of the proposed 2015 NSCG questionnaire, usability testing of the proposed 2015 NSCG web
instrument, and two rounds of cognitive interviews of the NSCG questionnaire in all three
data collection response modes – web, mail questionnaire, and CATI. The findings from
these tasks influenced decisions regarding questionnaire content, questionnaire and

12

instrument format, and question wording. The specific revisions made to the NSCG
questionnaire since the 2013 survey cycle are discussed later in this document.
•

Adaptive Design
The 2013 NSCG Terms of Clearance stated that “OMB looks forward to NCSES
collaborating actively with the National Center for Education Statistics and the Census
Bureau on ways to experiment with and apply "responsive design" methods to the NSCG in
order to better measure and reduce bias and improve overall survey efficiency.” Over the
past two years, NCSES staff participated in multiple outreach and collaboration efforts with
the Census Bureau, NCES, and other agencies to take stock of the progress made in the field
of adaptive design, to identify the obstacles that currently exist, and to explore the adaptive
design possibilities for the future. Below are some examples of NCSES’s outreach and
collaboration efforts related to adaptive design.

•

-

NCSES, Census Bureau, and NCES held a meeting in 2013 to exchange ideas related to
adaptive design metrics.

-

NCSES and Census Bureau organized and participated in a topic-contributed session on
adaptive design at the 2013 American Associated for Public Opinion Research (AAPOR)
annual conference in May 2013.

-

The survey contractors for NCES surveys and NCSES surveys (Research Triangle
Institute, Inc. and the Census Bureau, respectively) attended an adaptive design workshop
in December 2013 sponsored by Statistics Netherlands. Workshop attendees included
employees from the U.S. federal government, academia, private sector, and foreign
governments.

-

NCSES staff organized and chaired an invited session at the 2014 Joint Statistical
Meetings in August 2014. Session presenters were from a U.S. federal government
agency (Census Bureau), an educational research institute (University of Michigan) and
foreign government agency (Statistics Netherlands). The session discussed adaptive
survey designs.

-

NCSES, Census Bureau, and NCES staff attended meetings of the recently-formed
Adaptive Design Interagency Working Group in 2014. This working group, established
by the OMB Office of Statistical and Science Policy, is a collaboration among federal
statistical agencies.

-

NCSES, Census Bureau, and NCES staff have organized and will participate in a topiccontributed panel at the 2015 AAPOR annual conference. The conference is scheduled
for May 2015. The panel topic is “Innovation in Federal Surveys – Opportunities,
Progress, and Challenges.”

Survey Design and Methodology
NCSES has sponsored and collaborated on multiple survey design and methodology research
projects in an effort to ensure that the NCSES surveys, including the NSCG, are
incorporating best practices for survey design and methodology. NCSES holds ongoing

13

discussions with staff from NCES and the Census Bureau to discuss survey design and
methodological issues of interest. In addition, NCSES funds research on survey design and
methodological issues. The following provides a listing for some of the ongoing research
funded by NCSES related to the NSCG and the SESTAT surveys:

9.

-

To ensure accurate variance estimation under the NSCG’s complex ACS-based sample
design, NCSES funded research to examine how to appropriately derive variance
estimates in a two-phase sample design setting. Jean Opsomer (Colorado State
University) was the principal investigator for this research.

-

For transparency and documentation purposes, and to create a more knowledgeable data
user community, NCSES funded research to examine and document the properties of the
successive difference replication methodology. The successive difference replication
methodology is used in the NSCG for variance estimation purposes. Jean Opsomer
(Colorado State University).

-

To produce more reliable survey estimates, NCSES funded research to examine and
mitigate extreme sample weight variation within the NSCG. Jean Opsomer and Jay
Breidt (Colorado State University) were the principal investigators for this research.

-

To examine the full potential of the ACS data as a source for data on the S&E workforce,
NCSES funded research to conduct an initial examination of whether ACS data can be
used along with ACS and NSCG model-based results to conduct off-year estimation 14 for
characteristics of the college-educated population. Michael Larsen (George Washington
University) was the principal investigator for this research.

-

To address the increasing nonresponse trends for governmental surveys, NCSES funded
research to examine contract strategies for the NCSES surveys. Jolene Smyth and
Kristen Olson (University of Nebraska – Lincoln) were the principal investigators for this
research.
PAYMENT OR GIFTS TO RESPONDENTS

The 2010 NSCG and 2013 NSCG included incentive experiments to examine the impact of
offering incentives on response, data quality, and cost. The results from the incentive
experiments 15,16 provided NCSES and the Census Bureau with guidance and direction for using
incentives in the 2015 NSCG data collection effort. Please see Appendix J for more information
on the results from the 2013 NSCG incentive experiments.
14

Off-year estimation would provide estimates for the college educated population, using only ACS data,
in the years where the NSCG is not in the field. For example, as the NSCG is conducted in 2013, 2015,
and 2017, off-year estimation would produce estimates for the college-educated population in 2014 and
2016.
15
Zotti, Allison, “Report for the 2013 National Survey of College Graduates Methodological Research
Incentive Timing Experiment,” Census Bureau Memorandum from Reist to Finamore and Rivers, April
15, 2014 draft.
16
Thornton, Thomas, “2013 National Survey of College Graduates (NSCG) Incentive Conditioning
Study,” Census Bureau Memorandum from Reist to Finamore and Rivers, April 15, 2014 draft.

14

Based on the results from the 2013 NSCG incentive timing study, we plan to offer a $30 prepaid
debit card incentive to a subset of highly influential new sample cases at week 1 of the 2015
NSCG data collection effort. “Highly influential” refers to the cases with a large base weight
and a low response/locating propensity. The highly influential cases will be identified by a
model-based approach using a weighted response influence, which is the product of a sampled
case’s base weight and predicted response propensity. We expect to offer $30 debit card
incentives to approximately 8,000 of the 42,000 new sample cases included in the 2015 NSCG.
The weighted response influence factor is calculated as follows:

 1  1

Wi = ωi * φˆi , where φˆi = 
 ρˆ Li  ρˆ Ri


 .


The weighted response influence for a case, Wi , is the product of the base weight, ωi , and the
response influence, φˆi . The response influence is the inverse of the product of the locating
propensity, ρˆ Li , and the response propensity, ρˆ Ri .

In addition, using the findings from the 2013 NSCG incentive conditioning study, we plan to
offer a $30 prepaid debit card incentive to past incentive recipients at week 1 of the 2015 NSCG
data collection effort. We expect to offer $30 debit card incentives to approximately 14,500 of
the 93,000 returning sample members.
The $30 incentive amount proposed for use in the 2015 NSCG was chosen based on findings
from the 2010 NSCG late-stage incentive experiment targeting hard to enumerate cases that had
not responded to the survey after multiple contacts. As part of the 2010 experiment, the hard to
enumerate cases were allocated to three treatment groups:
•

$30 debit card incentive

•

$20 debit card incentive

•

No incentive

Other than the use and amount of the debit card incentive, the three treatment groups in the 2010
NSCG late-stage incentive experiment received the same data collection contact strategy. At the
conclusion of the experimental period (approximately six weeks), the response rate for the three
treatment groups differed significantly. The $30 incentive treatment group had a response rate of
29.5%, the $20 incentive treatment group had a response rate of 24.1%, and the no incentive
group had a response rate of 6.4%.
In addition to the increase in the response rate for the hard to enumerate cases that were targeted
as part of this experiment, the use of the incentive also had a profound effect on the overall
representation of the responding sample. The incentive was successful in obtaining responses
from individuals who are demographically different than the set of respondents prior to the

15

incentive stage. This ability to increase the demographic diversity of our responding sample
helps decrease the potential for nonresponse bias in our estimates.
10.
ASSURANCE OF CONFIDENTIALITY
NCSES and the Census Bureau are committed to protecting the confidentiality of all survey
respondents. The NSCG data will be collected in conformance with the Privacy Act of 1974, the
NSF Act of 1950, as amended, Title 13, Section 9 of the United States Code, and the
Confidential Information Protection and Statistical Efficiency Act (CIPSEA) of 2002. The
Census Bureau is conducting the NSCG under the authority of Title 13, Section 8 of the United
States Code.
The statement on the questionnaire cover will cite the appropriate data collection authority as the
NSF Act and confidentiality assurances under the CIPSEA. The questionnaire cover statement
will also inform the respondents that the data will be used for statistical purposes only, and the
voluntary nature of their response. The cover letters will include additional statements in the
Frequently Asked Questions section about the Census Bureau’s Title 13 as the data collection
authority and assurances of confidentiality (see Appendix E). The Census Bureau will include
the same appropriate notices of confidentiality and the voluntary basis of the survey in the
introduction to respondents contacted during the web phase and CATI phase of the data
collection.
NCSES and the Census Bureau will operate within the guidelines established by the Privacy
Act to protect respondents’ privacy and the confidentiality of the data collected. The Privacy
Act states “microdata files prepared for purposes of research and analysis are purged of
personal identifiers and are subject to procedural safeguards to assure anonymity.”
The Census Bureau has demonstrated experience in handling sensitive data. Routine
procedures will be in place to ensure data confidentiality, including the use of passwords and
encrypted identifiers to prevent direct or indirect disclosures of information. Furthermore, the
Census Bureau’s management system is in full compliance with the government’s automatic
data processing systems requirements.
11.

JUSTIFICATION FOR SENSITIVE QUESTIONS

No questions of a sensitive nature are asked in this data collection.
12.

ESTIMATE OF RESPONDENT BURDEN

NCSES estimates that it will contact approximately 135,000 sample persons by web, mail or
computer-assisted telephone interviewing as part of the 2015 NSCG data collection. Based on
experience administering the NSCG interviews, the questionnaire takes an average of 30 minutes
to complete. An overall response rate of about 70 percent is estimated from the 42,000 new
sample cases, and an overall response rate of about 80 percent from the 93,000 returning sample
cases. Based on an estimate of approximately 103,800 completed cases, the total burden hours

16

for the 2015 NSCG data collection are 51,900. The total cost to respondents for the 51,900
burden hours is estimated to be $1,442,820. This estimate is based on an estimated median
annual salary of $58,000 per NSCG employed respondent. Assuming a 40-hour workweek and a
52-week salary, this annual salary translates to an hourly salary of $27.88. Salary estimates were
obtained using data from the 2013 NSCG.
13.

COST BURDEN TO RESPONDENTS

Not applicable. This survey does not require respondents to purchase equipment, software or
contract out services.
14.

COST BURDEN TO FEDERAL GOVERNMENT

The total estimated cost to the Government for the 2015 NSCG is approximately $14.3 million,
which includes survey cycle costs, and NCSES staff costs to provide oversight and coordination
with the other SESTAT survey. The estimate for survey cycle costs is approximately $13.7
million, which is based on sample size; length of questionnaire; administration; overhead;
sample design; mailing; printing; sample person locating, web instrument development;
telephone interviewing; incentive payments, critical items data retrieval, data keying and editing;
data quality control; imputation for missing item responses; weighting and estimating sampling
error; file preparation and delivery; and preparation of documentation and final reports. The
NCSES staff costs are estimated at $562,500 (based on $150,000 annual salary of 1.5 FTE for
2.5 years).
15.

REASON FOR CHANGE IN BURDEN

There were two main changes in the NSCG between the 2013 and 2015 survey cycles that
impact burden. First, the sample size for the 2015 NSCG (135,000 cases) is slightly less than the
2013 NSCG sample size (144,000 cases). The main explanation for this difference is a smaller
young graduate oversample will be selected in the 2015 NSCG compared to what was selected in
the 2013 NSCG. The second change that impacted burden was a change in the length of the
questionnaire. For the 2015 NSCG survey cycle, the NSCG questionnaire was revised to include
questions on certifications and licenses. The addition of the certifications and licenses questions
increased the estimated survey completion time from 25 to 30 minutes. In addition to these
changes, the inclusion of past nonrespondents into the eligible sample created the need to reduce
the expected response rate for returning sample cases. When these factors are considered, there
was a slight increase in the burden hours estimate between the 2013 and 2015 NSCG survey
cycles.

17

16.

SCHEDULE FOR INFORMATION COLLECTION AND PUBLICATION

NCSES does not plan to use any complex analytical techniques in publications using this data.
Normally cross tabulations of the data are presented in NCSES reports and other data releases.
The time schedule for 2015 data collection and publication is currently estimated as follows:
Data Collection

April 2015 – October 2015

Coding and Data Editing

April 2015 – February 2016

Final Edited/Weighted/Imputed Data File

March 2016

NSCG Info Brief

Summer 2016

NSCG Integrated Public Use Data File

Summer 2016

17.

DISPLAY OF OMB EXPIRATION DATE

The OMB expiration date will be displayed on the 2015 NSCG questionnaires, postal contacts,
and the web instrument introduction page.
18.

EXCEPTION TO THE CERTIFICATION STATEMENT

Not Applicable.

18