A_ICR Statement A NTEWS_09-24-2021

America COMPETES Act
NSF Act
NTEWS Methodological Experiments
NTEWS Contact Materials: FAQs and Letters (English version)
NTEWS Contact Materials: FAQs and Letters (Spanish version)
Summary of Existing Federal Survey Data to Measure the Skilled Technical Workforce
NTEWS Rotating Panel Design Chart
Federal Register Notification (FRN): Comments and Responses
Consultation Activities for the ATES Development
NCSES and NCES Inventory of Incentives
2022 NTEWS Paper Questionnaire (English version)
2022 NTEWS Paper Questionnaire (Spanish version)
Power Analysis: NTEWS Methodological Experiments and Seeded Sample

NTEWS Questionnaire Development through Cognitive Research
NTEWS Evaluating for Utility and Quality
NTEWS Utility-Quality Itemized Grid
2022 NTEWS Bridge Panel Web Questionnaire

2022 NATIONAL TRAINING, EDUCATION, AND WORKFORCE SURVEY (NTEWS)

SUPPORTING STATEMENT A. JUSTIFICATION

The National Center for Science and Engineering Statistics (NCSES) within the National Science Foundation (NSF) requests a three-year approval for a new collection referred to as the 2022 National Training, Education, and Workforce Survey (NTEWS). This initial cycle will be a voluntary data collection sponsored by NCSES and cosponsored by the National Center for Education Statistics (NCES) within the U.S. Department of Education. This collection serves to measure and understand two research concepts that are of national interest: (1) the education, training, and career pathways of skilled technical workers (defined below) and (2) the prevalence and interplay of education credentials (postsecondary degrees and certificates), work credentials (certifications and licenses), and work experience programs in fostering skill development among American workers.

As the initial year survey, NCSES will use the information collected from the 2022 NTEWS to guide subsequent full-scale collections. The 2022 NTEWS will survey a sample size of approximately 43,200 individuals compared to about 140,000 cases in the full-scale subsequent cycles (see Appendix G for information on the NTEWS Rotating Panel Design). To be fielded in parallel with the NTEWS production sample (national sample), NCSES will also survey a non-production, seeded sample of 1,000 cases and a non-production bridge panel of 5,000 cases. The seeded sample will further research on the underreporting of postsecondary certificates. The bridge panel will provide a mechanism for NCSES to explore modifications to the NTEWS survey questions and quantify the potential impact on key survey estimates. Information from the 2022 NTEWS production and non-production samples should identify areas for improvements in future NTEWS survey cycles that will enhance data quality and improve response rates while reducing costs (e.g., minimizing the use of incentives) and burden.

At the time of the submission of this information clearance request, NCSES is uncertain of the impact of the coronavirus pandemic on the initial NTEWS collection, which is expected to begin data collection in early spring of 2022. While the pandemic and its recovery may extend throughout the 2022 calendar year, NCSES will monitor and adjust, where possible, the data collection effort to mitigate for any pandemic impact on response patterns.

1. Why the collection is necessary

The NTEWS will address NCSES and NCES research and analytical needs. Established within NSF by the America COMPETES Reauthorization Act of 2010 § 505, codified in the NSF Act of 1950, as amended, NCSES serves as a central federal clearinghouse for the collection, interpretation, analysis, and dissemination of objective data on science, engineering, technology, and of research and development for use by practitioners, researchers, policymakers, and the public. NCSES also provides data to support the Science and Engineering Equal Opportunities Act of 1980, which directs NSF to provide to Congress and the Executive Branch an "accounting and comparison, by sex, race, and ethnic group and by discipline, of the participation of women and men in scientific and engineering positions." See Appendices A and B for copies of these legislations.

NCSES has historically met these legislative mandates through its suite of surveys, including the National Survey of College Graduates (NSCG) and the Survey of Doctorate Recipients (SDR), that measures the education, employment, and demographic characteristics of the nation's college-educated scientists and engineers with at least a bachelor's degree. The data from these surveys are central to the analyses presented in a pair of congressionally mandated reports¹^,² published by NCSES:

Science and Engineering Indicators (https://nsf.gov/statistics/seind/)
Women, Minorities, and Persons with Disabilities in Science and Engineering. (https://nsf.gov/statistics/women/)

These data sets and reports exclude a growing portion of the science and engineering (S&E) workforce—those who use science, technology, engineering, and mathematics (STEM) skills and knowledge in their jobs, but do not have a bachelor's degree. These workers comprise the skilled technical workforce (STW). With the pervasiveness of science and technology in society, including its central role in the economy, the nature of work has changed for individuals at all education levels, making skilled technical workers increasingly essential to U.S. economic competitiveness, national security, and scientific progress. While some data exist to quantify the number of skilled technical workers, limited information exists to examine how individuals enter, maintain relevance, or seek advancement in STW occupations. Through NTEWS, data can be made available to address the increasing policy interest in the STW and expand NCSES's clearinghouse of S&E data.

Simultaneously, the NTEWS will collect data to address NCES's interest in work credentials (certifications and licenses), postsecondary certificates, and work experience programs (WEP). WEP include internships, co-ops, practicums, clerkships, externships, residencies, clinical experiences, and apprenticeships. While these programs do not necessarily result in work or educational credentials, these are common educational and training pathways for U.S. adults to develop work skills. Prior to NTEWS, NCES administered the Adults Training and Education Survey (ATES, see https://nces.ed.gov/nhes/ates.asp for more information), which is an inactive federal survey that collected data on nondegree credentials and work experience programs. The NTEWS data collection continues NCES's research goals and purpose of the ATES by examining the interactions between work credentials and degree credentials to support workforce learning and the role of education institutions in helping individuals attain work credentials. The NTEWS is expected to have increased utility compared to the ATES because the NTEWS will provide more reliable STW estimates by allocating a larger sample to individuals in skilled technical occupations.

As a federally co-sponsored survey, the NTEWS will collect data from a nationally representative sample survey of adults ages 16 through 75 who are not enrolled in high school and residing in the United States, including the District of Columbia and Puerto Rico. The sample will be selected from respondents to the U.S. Census Bureau's 2018 American Community Survey (ACS), with oversamples of adults who do not have a bachelor's degree and adults in the STW. This sampling approach will allow the NTEWS to generalize to two populations: (1) all adults ages 16-75 living in households (the population of interest to NCES) and (2) all adults ages 16-75 in the skilled technical workforce (the population of interest to NCSES). The 2022 NTEWS will collect information on the following topics:

Employment characteristics
Educational background (enrollment and attainment)
Credential types
Work experience programs
Demographic characteristics

These topics allow for the examination of the education and career paths of the STW and the relationship between work credentials, certificates, and WEP with employment outcomes. Given these areas of mutual interest for NCSES and NCES, the NTEWS will reduce public burden by fielding one cosponsored survey that meets the information needs for both federal agencies.

2. Uses of the Information

NCSES and NCES intend to publish national estimates from the 2022 NTEWS, as well as use the results to inform the content, methodology, and survey design for subsequent NTEWS cycles. NCSES plans to use the NTEWS data for its two congressionally mandated biennial reports: Women, Minorities, and Persons with Disabilities in Science and Engineering and Science and Engineering Indicators. NCES plans to release a special-topic statistical report on the status of education and work credentials in the United States. Also, a public-release file of collected data, which includes safeguards to protect respondent confidentiality, will be made available online.

3. Use of Automated, Electronic, Mechanical, or Other Technological Techniques

The U.S. Census Bureau will collect and manage the data for the 2022 cycle of the NTEWS under an Interagency Agreement between NCSES and the Census Bureau. The NTEWS data collection will have three response modes, including an online instrument (web), a paper questionnaire, and a computer-assisted telephone interview (CATI). NTEWS will implement a contact-strategies experiment that will test six alternative sequences of these three response modes (see Supporting Statement B and Appendix C on the NTEWS Methodological Experiments). Given these complexities, the Census Bureau will employ multiple, complementary data collection and monitoring systems that efficiently coordinate a large-scale survey collection with multiple experiments. Many of these systems are used in another long-running, well-established NCSES survey collection, the NSCG. These systems are described below:

3a. Tracking Paper Questionnaires: Intelligent Mail Barcoding

The Census Bureau will mail out, collect, and process paper questionnaires. The NTEWS paper questionnaire will be formatted as a PDF file using Amgraf OneForm Designer Plus, which is a Windows-based software that allows for total automation of high-volume forms production. An outside contractor has been tasked with printing, trimming, and stitching work for the paper instrument, but the Census Bureau will print the survey invitation letters (see Appendices D and E for contact materials in English and Spanish, respectively). Survey invitation letters are produced by the Census Bureau through an in-house, on-demand print process using a Docuprint system, which allows tailored and personalized survey materials for each specific respondent.

The letters and questionnaire packets will be tracked using Intelligent Mail Barcoding (IMB). IMB will use separate outgoing and return barcodes on the 2022 NTEWS envelopes for tracking purposes. Using IMB has the potential to increase the overall efficiency of data collection, enabling the collection of detailed tracking information, including:

When an outgoing survey or other mailpiece reached a respondent's local post office;
When an outgoing mailpiece left the post office with a postmaster for delivery;
If the outgoing mailpiece was identified as undeliverable-as-addressed (UAA) and then rerouted for return;
When a returned survey reaches a respondent's local post office; and
When a returned survey reaches its destination.

When a survey is returned, the Census Bureau will put the individual on hold while determining the completeness of the questionnaire. Placing cases on hold will reduce respondent burden by limiting unnecessary follow-up contacts. The IMB tracking features also alert the Census Bureau of undeliverable mailpieces in circulation, information that can be helpful in immediately eliminating future mailings to the same undeliverable addresses.

3b. Processing Paper Questionnaire Data: The Integrated Computer Assisted Data Entry (iCADE) System

The iCADE is a data capture solution for paper-based data collection operations, which involve several steps. All completed paper questionnaires will be first preprocessed by iCADE. Each completed paper questionnaire will be converted into a document image format to correct any skewing at the time of scanning. The iCADE software creates a registration to align the individual questionnaire with a page template, resulting in an accurate scanned image. Then, the scanner cleans the image through a "despeckels" step, which removes unwanted pixels. Next, iCADE automatically captures questionnaire responses through an optical mark recognition (OMR). In this phase, iCADE reads the form image files, checks the presence of data, and digitizes alphanumeric data fields through OMR. The end result is the creation of a text image of handwritten fields (open-ended responses), which are sent to an operator for "keying from image" (KFI) and quality assurance.

All generated images will be scanned and archived to a magnetic storage located on a secure server in case they are needed later for verification purposes. This electronic storage eliminates the need to save paper copies of the completed questionnaires.

3.c Deploying an Online Questionnaire Instrument: Enterprise Internet Solutions and Mobile Optimization Systems

The Enterprise Internet Solutions area of the Application Services Division at the Census Bureau will host and secure the web-based NTEWS data collection instrument. Data will be transmitted and processed daily. The web instrument will be hosted on the Census Bureau's fully certified and accredited Centurion system, which provides a secure infrastructure for data exchange through the Internet. The system also provides extra functionality to help survey respondents in completing the questionnaires, such as special instructions, content-specific help, and contact information. The 2022 NTEWS web instrument will be optimized for use on mobile devices, creating a better experience for mobile device users and, thereby, reducing survey breaks or stoppages and the possibility of measurement errors. The 2022 NTEWS web-based instruments are designed to minimize respondent burden (and error) by eliminating the cumbersome skip patterns required in the paper questionnaire.

3.d Using Telephone Interviews: Computer Assisted Telephone Interviews

For the 2022 NTEWS, CATI will be implemented in several ways to ensure a high-quality data collection. A portion of the sample will be asked to complete NTEWS via CATI as part of the contact-strategy experiment (See Appendix C). Sample persons (SPs) with valid phones but not valid domestic addresses to mail the web invitation or paper questionnaire will also be redirected to CATI. Finally, SPs who did not meet the sample person verification (which determines the SP's eligibility for NTEWS survey participation) or failed to answer pre-determined critical items from the questionnaire are directed to complete the survey through CATI.

The CATI operation consists of trained interviewers calling SPs to ask them to complete an interview on the phone. The interviewers also have the capability to provide web logins, set appointments for later calls, log refusals, etc. Within CATI, a telephone questionnaire assistance (TQA) line will be in operation. The TQA is a help line for SPs to call in with any questions. TQA operators will also have the capability to conduct the interview.

3.e Monitoring the Data Collection: Unified Tracking System

Like the NSCG, the 2022 NTEWS will use the Census Bureau's Unified Tracking System (UTS) to assist in survey management. Since 2013 the UTS has provided a full contact history report for the NSCG, giving survey managers a central place to view all contacts integrated from all three survey modes along with the outcomes of those contacts. The Census Bureau will provide similar reporting for the NTEWS collection, and reports will be developed based on the following activities:

Reviewing UTS reports from other NCSES- and NCES-sponsored surveys to make additional updates to NTEWS-tailored reports
Validating UTS reports during system test
Documenting available reports and their uses for the NTEWS survey managers
Reviewing requirements with different areas of the collection operations to ensure necessary data are being transmitted
Designing, implementing, and modifying reports through an iterative process that meets the goals of the collection

The UTS contact history reporting system enables the monitoring and examination of the survey's responses and survey experiments in various ways. For example, if respondents call in to check on the status of their response, the NTEWS staff can access the respondents' contact history quickly and easily. Also, this report facilitates an audit trail of all contacts, allowing the survey team to immediately identify and verify if NTEWS interviewers are following proper contact protocols, particularly when questions or complaints from respondents arise. The UTS can provide daily updates for R-indicators analysis, which allows for the NTEWS survey managers to monitor the performance of the experiments. In future cycles of NTEWS, the UTS can also identify ways for the data collection operations to target representativeness and improve representativeness.

The Census Bureau has integrated the UTS contact report with the IMB data, which will enhance the coordinated operations of the 2022 NTEWS collection. For example, the UTS can generate two reports that allow for monitoring of differences between the outgoing dates provided by the Census Bureau's National Processing Center (NPC) and the incoming IMB-provided dates. For outgoing mailings, the report can show the lag time between the scheduled mail date and when NTEWS packages enter the mail stream. For incoming mailings, the report can provide the dates when UAAs or returned questionnaires enter the IMB system versus when they are checked in at NPC. For more tailored information about the NTEWS administration, both reports can disaggregate data by mailing geographies. These reports primarily function to help the NTEWS team understand the relationship between when respondents receive their invitation in the mail and when they respond to survey requests, thus helping the survey team anticipate response relative to mail-out operations and establish future mailout schedules.

Finally, the UTS report documents the interactions of the NTEWS sample with the web instrument. This report provides information such as the number of sampled respondents that have logged in and with what type of device, time spent responding, and whether respondents logged out or submitted the survey. The UTS report allows these web paradata (i.e., data about the process of collecting survey data via the online instrument) to be monitored throughout the data collection period. The UTS is a well-tested, cost-saving, and dependable system to monitor the web paradata throughout the data collection period, providing valuable information on the response patterns of a new subpopulation, the STW.

3.f Developing an Adaptive Survey Design for Future NTEWS Cycles

Adaptive survey deign (ASD) refers to tailored contact strategies implemented before and during a data collection to improve the quality of the survey outcomes, reduce survey costs, and minimize the potential for nonresponse bias. For the 2022 NTEWS, NCSES will not incorporate ASD techniques into the data collection effort. Instead, in preparation for the inclusion of ASD techniques in future NTEWS cycles, the NTEWS data collection contractor, the Census Bureau, will set up an ASD paradata collection system for 2022 NTEWS data collection monitoring purposes.

The Census Bureau has demonstrated through the NSCG data collections³ that an adaptive survey design can improve the "flow processing" approach to data processing (i.e., intermittent editing, imputation, and weighting of incoming response data). This flow processing approach has allowed NCSES to derive NSCG survey estimates throughout the data collection effort and use the estimates as part of the overall quality assurance plan. Given the population differences between the NSCG and NTEWS, NCSES and the Census Bureau agreed the NSCG ASD results may not be directly generalizable to the NTEWS. As a result, the planned ASD paradata collection and monitoring systems development will provide insight on an appropriate manner to incorporate ASD techniques to include in subsequent NTEWS cycles.

Paradata planned to be collected includes cumulative information about the state of the data collection, such as whether the case is on hold, whether the case has received an incentive, and if the case is currently considered a respondent. Census will also compile mail-out/mail-in paradata, Internet server-side paradata, Internet full paradata file, and CATI status and transaction files.

4. Efforts to Identify Duplication

Duplication, in the sense of a similar data collection, does not exist for the NTEWS. Appendix F provides a summary table that identifies existing federal data collection efforts to measure the skilled technical workforce. To NCSES's knowledge, there is no federal collection that captures the full set of education, training, and career components of the U.S. workforce, particularly at the level of occupational detail necessary to appropriately measure and understand the STW population. Data from the Current Population Survey provides occupational estimates but does not collect detailed training and credential information, nor does it collect information on the degree field for postsecondary degrees. The ACS collects occupational fields but does not collect the detailed information on work activities, training and credentials, and employment characteristics that is in the NTEWS. Finally, because NTEWS is cosponsored by NCSES and NCES, one survey serves both agencies’ statistical and research needs, thus removing the potential for duplicative efforts.

While there is no existing survey that provides the comprehensive education, training, and workforce information planned for the NTEWS, there may be some survey content duplication between the NTEWS and existing surveys like the ACS. As mentioned earlier, the 2022 NTEWS sample will be drawn from ACS respondents. As a result, NCSES will have ACS responses to develop the NTEWS sample frame including education level, occupational field, and demographic information, such as gender, race, ethnicity, marital status, citizenship status, and veteran status. However, due to federal confidentiality restrictions, NCSES is prohibited from linking or publicly releasing the demographic information from the ACS with the detailed education and employment information collected on the NTEWS. As a result, demographic information will be collected on the NTEWS. This duplication is necessary because linkage between demographic, education, and employment information is needed for the analyses used in NCSES's congressionally mandated reports.

While the proposed NTEWS and established NSCG use the ACS as a sampling frame, the sampling years are not the same. The NTEWS plans to use even-numbered years of the ACS (e.g., the 2022 NTEWS will use the 2018 ACS⁴). The NSCG uses odd-numbered years (e.g., the 2021 NSCG used 2019 ACS). Thus, the likelihood that the NTEWS would duplicate its sample with the same NSCG sampled respondents is extremely limited.

Nonetheless, overlap does exist in the target populations for the NTEWS and the two other NCSES surveys: the NSCG and Survey of Doctorate Recipients (SDR). As a result, NCSES estimates there could be approximately nine individuals selected for sample in the 2022 NTEWS and in either the 2021 NSCG and the 2021 SDR.

In a previous NSCG survey cycle, the NSCG and SDR survey contractors identified the individuals selected for both surveys, removed the individuals from the NSCG data collection effort, and, after the SDR data collection effort, used the SDR responses for these individuals to complete the individual's record on the NSCG data file. This NSCG/SDR deduplication process required the SDR survey contractor to create numerous files containing all SDR sample cases for use by the NSCG survey contractor. Furthermore, given file format and processing differences between contractors, the NSCG survey contractor needed to reformat and manually manipulate many of the SDR files to use them in combination with the NSCG files. Despite a small number of duplicates between the surveys, the NSCG/SDR deduplication process added over a week of staff time to both the NSCG and SDR processing during the survey cycle.

There are noticeable questionnaire content differences between the NTEWS and both the NSCG and SDR. Information collected on the NTEWS that is not collected on the NSCG and SDR includes the attainment of postsecondary certificates, professional certifications, and occupational licenses as well as participation in work experience programs. Because of the content differences, the small number of expected duplicates, and the operational challenges of the deduplication process, NCSES will not deduplicate individuals selected for the NTEWS sample with either the NSCG or SDR samples during the 2021 survey cycle.

5. Impact on Small Entities

Not applicable. The NTEWS collection does not collect information from small entities or businesses.

6. Consequences of Less Frequent Collection

NCSES reports on the S&E workforce in its pair of congressionally mandated reports—Science and Engineering Indicators and Women, Minorities, and Persons with Disabilities in Science and Engineering. These high-profile, biennial reports provide an accurate accounting and comparison (i.e., by sex, race, ethnic group, and educational discipline), of the participation of women and men in scientific and engineering positions and rely on the availability of updated data on the S&E workforce every two years. In the past, NCSES has primarily focused its reports on the S&E workforce with a bachelor's degree or above. The inclusion of the STW data in these NCSES reports will provide a more complete picture of the S&E workforce. Thus, NCSES needs the NTEWS to produce STW data to be of comparable quality to the NSCG data on the college-educated, and thus, NTEWS also needs to be on same 2-year periodicity as the NSCG. Conducting the NTEWS on a less frequent basis would prohibit NCSES from meeting its congressional mandate to produce comprehensive reports that contain an accurate, timely description of the S&E enterprise. Furthermore, other stakeholders including government, industry, training organizations, and educational institutions would have less recent data to use for formulating U.S. S&E policies and programs.

A less frequent data collection would also negatively affect the quality of the NTEWS data. As a rotating-panel survey, the NTEWS will follow up with the same respondents every two years for six years (see Appendix G). Follow-up surveys are necessary to monitor the movements of individuals into and out of S&E occupations (including STW occupations) throughout their careers. Because of the mobility of the U.S. population, NCSES has found conducting less frequent follow-ups make it difficult and costly to locate the respondents. Due to this increased inability to locate respondents, less frequent follow-ups would result in higher attrition rates, increased potential for nonresponse bias, and less reliable estimates. NCSES plans to conduct NTEWS every two years to minimize the data degradation that jeopardizes NCSES's ability to provide national estimates on the STW sector of the S&E workforce.

7. Special Circumstances

The initial cycle of the NTEWS is designed to produce valid and reliable results that will be generalized to the universe of the study (adults ages 16-75 not enrolled in high school). In addition, the NTEWS is implementing an experiment—the Seeded Sample Experiment—that would qualify as a special circumstance where NCSES does not plan to produce data that is generalizable.

A key estimate of the NTEWS is the prevalence of U.S. adults who possess a postsecondary certificate (i.e., a certificate awarded by a postsecondary institution). However, in past work on the ATES, NCES found respondents had difficulty reporting whether they held a postsecondary certificate. In the 2010 ATES pilot, NCES included a seeded sample of known certificate holders and found that ATES respondents often underreported their postsecondary certificates (i.e., reported they did not hold a certificate when postsecondary institution records indicated respondents had one). NCES continued to refine and revise the certificate items in subsequent ATES cycles (2014 and 2016), and yet, NCES continued to find high rates of underreporting. NCES conducted cognitive research on the certificate items post-2016 ATES and these revised items appear on the 2022 NTEWS. The NTEWS seeded sample experiment will evaluate the success of the last round of revisions at mitigating underreporting.

For the seeded sample experiment, the NTEWS will be administered to a sample drawn from postsecondary institutions' lists of certificate awardees (the seeded sample). The seeded sample will not be nationally representative but is instead a convenience sample of 1,000 adults known to hold postsecondary certificates. Therefore, respondents from the seeded sample will not be included in the publicly released NTEWS data products or in national estimates derived from the NTEWS production sample.

The seeded sample data are for NCSES and NCES research purposes only. NCSES and NCES plan to calculate the percentage of seeded sample respondents who do not report holding a postsecondary certificate. Because the seeded sample will be selected from a frame of known postsecondary certificate holders, this percentage can be interpreted as an underreporting rate. Based on the analysis, NCSES will determine (1) if the postsecondary certificate data (from the production sample) should be included in the NTEWS data products and (2) if further revisions to the postsecondary certificate survey items are needed for future NTEWS cycles. In anticipation of this analysis, NCES is currently supporting a literature review on under-reporting rates; this review will help evaluate the findings of the seeded sample experiment. Please see Appendix C for more details on the NTEWS seeded sample research.

8. Consultations Outside the Agency

The 60-day Federal Register announcement for the NSCG appeared on April 16, 2020 [Full citation: 85 FR 21271]. NCSES received five public comments in response to the announcement. See Appendix H for the announcement, comments, and responses.

NCSES has developed a multi-activity STW Initiative⁵ that serves as a framework for NCSES to understand and measure the STW population. One activity of the STW Initiative is to conduct stakeholder outreach to identify information and data needs on the S&E workforce and STW. These stakeholders include other federal agencies, policymakers, academics, researchers, data analysts, survey methodologists, statisticians, and nonprofit associations. In addition, the development of the NTEWS has benefited from extensive discussions, meetings, and planning related to the fielding of the ATES. Appendix I lists a high-level summary of NCES's activities for the ATES planning and development effort, conducted under the auspices of a federal interagency working group; their work included three expert panel meetings, focus groups, cognitive interviews, an initial pilot study, a response rate study, a feasibility study, and the 2016 national administration of the ATES.

The following is a summary of NCSES outreach activities (i.e., presentations, meetings, external consultation, and workshops) that were designed to introduce the STW Initiative to stakeholder groups and to gather the information to help in the development of the NTEWS survey content. While these activities preceded the submission of this information collection request, NCSES will continue to refine future NTEWS collections by continuing to hold informational presentations and meetings with stakeholders as we move forward.

8.a Invited Presentations

Finamore, J.M. (August 12, 2019) "The National Training, Education, and Workforce Survey (NTEWS)" presented at the Non-Degree Credentials Research Network (NCRN) Project Meeting. NCRN is a small, by-invitation-only group of leading researchers and key stakeholders (e.g., employers, policymakers and providers of employment, training, and certification). The purpose of NCRN is to identify and improve the current knowledge base on non-degree credentials (e.g., certificates, certifications, apprenticeships, licenses, badges, micro-credentials, etc.) and their place in the broader credentialing ecosystem; determine and support new directions for research; and identify research implications for policy and practice. Follow-up presentations to provide updates on the NTEWS development progress were provided at the January 27, 2020 and June 2, 2020 NCRN meetings.

Finamore, J.M. (October 8, 2019) "A Multidimensional Initiative to Measure and Understand the Skilled Technical Workforce" presented at the Council of Professional Associations on Federal Statistics (COPAFS) Quarterly Meeting. COPAFS is devoted to educational activities and to preserving the public good represented by federal statistical collections. Since 1980, COPAFS has provided an open dialog between those who use federal statistics in professional contexts and the Federal statistical agencies that produce those statistics for the public good. Supporting organizations include professional associations, businesses, research universities, and others that help to produce and/or use federal statistics.

Finamore, J.M. (October 25, 2019) "Sponsoring Agency Innovations: The National Training, Education, and Workforce Survey (NTEWS)" presented at the U.S. Census Bureau Household Surveys Workshop. The Census Bureau conducts more than 30 surveys of individuals in households. In addition to the decennial Census, the Census Bureau compiles information from household surveys to describe the characteristics and living conditions of the American population (i.e., age, education, housing, and income). This meeting convened representatives of federal agencies interested in household statistics to provide updates on procedures, operations, and innovations related to Census household surveys.

Jones, G.J. (May 15, 2020) "NCSES STW Initiative and the NTEWS: Measuring and Understanding the Skilled Technical Workforce" presented to U.S. Department of Agriculture's Research, Education & Economics (REE) group within the Office of the Chief Scientist (OCS). This meeting convened representatives from four USDA research and scientific centers who are working on developing data-driven, policy-relevant key performance indicators (KPIs) for measuring the USDA's impact on the agriculture workforce.

8.b Meetings

October 3, 2019: the office of U.S. Senator Jacky Rosen (NV). Senator Rosen has served on several committees, including the House Science, Space, and Technology Committee, where she passed bipartisan legislation through the House to improve early childhood STEM education.

October 6, 2019: Bureau of Labor Statistics Employment Projections program develops labor market projections for the nation.

November 4, 2019: Georgetown University Center for Security and Emerging Technology (CSET) on Artificial Intelligence is a research organization focused on studying the security impacts of emerging technologies, supporting academic work in security and technology studies, and delivering nonpartisan analysis to the policy community.

December 13, 2019: Credential Engine is a nonprofit organization that aims to foster transparency in the credential marketplace, increase credential literacy, and empower stakeholders to make more informed decisions about credentials and their value. Credential Engine created a centralized Credential Registry to house up-to-date information about all credentials, a common description language to enable credential comparability, and a platform to search and retrieve information about credentials.

December 13, 2019: Policy and programs offices from Commerce, Census Bureau, National Institute for Interdisciplinary Science and Technology (NIIST), and Economic Development Administration (E.D.A.) who are working together, along with other federal agencies, on the Advisory Board that advises the National Council for the American Worker.

December 16, 2019 and February 3, 2020: NSF Directorate for Education & Human Resources – Division of Graduate Education oversees several NSF programs such as graduate research fellowship program, graduate traineeships unit, CyberCorps Scholarships for service, and Project and Program Evaluation.

January 8, 2020: NSF Future of Work-Human Technology Frontier (FW-HTF) is one of the "10 Big Ideas for the Future" NSF Investments. The FW-HTF responds to challenges and opportunities for the future of jobs and workers in a landscape with rapid social and technological changes.

January 10, 2020: Team members from the Office of Personnel Management who are involved in classifying, developing, supporting, and analyzing data on the technical workforce for the federal government.

February 4, 2020: NSF Directorate for Education & Human Resources – Division of Undergraduate Education manages a variety of activities, including workforce development cluster administrative support center, scientific support center, and STEM-C partnerships.

February 25, 2020: The National Student Clearinghouse is a nonprofit education organization that captures student data from postsecondary institutions and provides degree verification, enrollment verification, and student outcomes research.

April 1, 2020: Coalition for Science published a 2020 report on "STEM and the American Workforce: An Inclusive Analysis of the Jobs, GDP and Output Powered by Science and Engineering."

April 8, 2020: a NORC team of researchers is investigating the workforce training that occurs outside traditional pathways. Starting the spring of 2019, the team designed, fielded, and analyzed a survey of a nationally representative sample of adult learners to understand the quality of non-accredited/non-portable credentials that are in the workforce.

May 11, 2020: NASA Academy of Program/Project & Engineering Leadership (APPEL) Knowledge Services brings together curriculum and career development tools to create a comprehensive, knowledge-dedicated resource for NASA. A key goal of the organization is to better meet the requirements for developing the NASA technical workforce while enhancing NASA's ability to manage and share the different types of knowledge needed for mission success.

8.c External Consultation

NCSES consulted with the following agencies or experts who have helped in the development of the NTEWS. NCSES intends to continue consulting with these external entities during and beyond the 2022 NTEWS data collection to help future NTEWS cycles. The following describes the purpose of each consultation.

NCSES met with representatives from the National Science Board (NSB) Skilled Technical Workforce Task Force. The NSB establishes the policies of the NSF and serves as an advisor to Congress and the President on policy matters related to S&E and education in S&E. Charged with identifying the opportunities and challenges facing students, workers, businesses, educators, and others involved with the STW, the NSB published a 2019 STW report recommending strategies to strengthen the STW. By meeting with the NSB STW task force, NCSES gained valuable STW information from the task force’s STW activities and created awareness among the NSB of the NCSES STW Initiative and the forthcoming NTEWS.

NCSES also met with representatives from the Office of Science Technology and Policy (OSTP). One of the OSTP directives is to serve as a source of scientific and technological analysis and judgment for the President concerning major policies, plans, and programs of the Federal government. Therefore, NCSES consulted with OSTP to discuss the federal needs for additional data on the STW and obtain information on potential partners and stakeholders to build coalitions that need STW data.

To aid in the NTEWS sample design development, NCSES met with Dr. Jean D. Opsomer, a faculty member in the Department of Statistics at Colorado State University. Dr. Opsomer is an expert in the statistical field and has extensive, historical knowledge of the NSCG design and sampling. Given his past consultation work on the NSCG sample design, NCSES consulted with Dr. Opsomer to develop the sample selection and sample allocation approach for the 2022 NTEWS.

8.d Workshops

NCSES held a three-part, Skilled Technical Workforce (STW) webinar workshop. Each webinar was interactive where attendees could provide input or ask questions through the interactive chat and Q&A features.

August 7, 2020: The Skilled Technical Workforce and Why it Matters. The focus of this webinar was to define the workforce, discuss the limited information on the STW, and highlighting the need for more targeted, in-depth data.

August 21, 2020: Administrative and Other Supplemental Data Sources for the STW. This webinar demonstrated NCSES's interests and explorations of administrative and other data to supplement what is currently known about the STW.

August 28, 2020: Surveying the STW to Answer Policy-Relevant STEM-Workforce Questions: The 2021 National Training, Education, and Workforce Survey (NTEWS). This webinar aimed to provide attendees an understanding of the NTEWS – why NCSES was fielding it, the national policy questions the survey was designed to inform, questionnaire item constructs, survey design and methodology, and the planned NTEWS timeline from present day to release of the data.

8.e Pre-testing Research

While the NTEWS might be perceived as a continuation of the ATES, a discontinued, OMB-approved survey conducted by NCES, the NTEWS is a new data collection. As such, NCSES conducted extensive cognitive interviews that addresses the NTEWS’ intended uses. Results from the cognitive interviews can be found in Appendix N. To meet OMB’s standards to collect high-quality and high-utility data, NCSES assess the NTEWS questionnaire content using a set of utility and quality guidelines (see Appendix O). NCSES’s application of these guidelines to the 2022 NTEWS questionnaire resulted in an itemized utility-quality grid (see Appendix P) for each questionnaire item to document our efforts of collecting and producing high-quality data that will have high utility for public use.

9. Payments or Gifts to Respondents

The 2022 NTEWS will provide incentives to individuals in the production and non-production (seeded and bridge panel) sample groups. The incentives are in the form of prepaid debit cards, which will have a six-month usage period after which the cards will expire. The funds from any unused prepaid debit cards will be returned to the Government.

For the production sample, NCSES is implementing a noncontingent-incentive experiment (See Appendix C). The incentive experiment will investigate the effects of using different amounts (including no incentive). The incentives in the form of prepaid debit cards are described as "noncontingent" because the receipt of an incentive will not depend on whether the respondent completes the survey (rather than contingent incentives, for which the receipt of the incentive requires a response).

Both NCSES and NCES have a history of conducting incentive experiments to maximize data quality, while minimizing survey response and cost (see Appendix J for the NCSES and NCES Inventory of Incentives). When the NCES administered the ATES as a module of NHES, NCES employed a cash-incentive experiment. NCSES has conducted several incentives studies in past NSCG survey cycles. Building off work on incentives from both agencies, NCSES designed a noncontingent-incentive experiment as part of the 2022 NTEWS. This experiment will provide the information NCSES needs to identify an optimal, tailored incentive strategy for future NTEWS cycles that will minimize the potential for nonresponse bias (by increasing sample representativeness among the responding cases) while also minimizing, to the extent possible, the burden on the public and cost to the federal government.

Because the NTEWS is a new federal survey, the noncontingent-incentive experiment will enable the collection of baseline data for evidence-based decision making on the use of incentives in future NTEWS cycles. These baseline data can also benefit other federal surveys by providing insight into the effectiveness of alternative incentive strategies at reducing nonresponse bias among the adult population. NCSES is uniquely positioned to provide such information because the NTEWS uses the ACS as a sampling frame; this frame provides a much richer set of data for use in nonresponse bias analysis than is typically available in a federal survey.

NCSES developed the following research questions to establish critical baseline information as part of the design for the NTEWS incentive structure:

Nonresponse bias for key estimates: Do key survey estimates vary across incentive levels? Can these differences be explained by sampling error or by differences in demographic representation, which can be mitigated by conventional weighting adjustments? If differences remain after accounting for sampling variability and demographic representation, is there an optimal level of incentive that leads to the highest level of reporting/least amount of bias for key estimates?
Nonresponse rates for key subpopulations: Which population groups of interest have lower-and higher-than-average response probabilities to the NTEWS? What are the impacts of incentive levels on these response propensities and corresponding weight adjustments related to estimates of uncertainty (standard errors)?
Sample representativeness: How does the representativeness of the responding sample differ across incentive levels, and how does this compare to the composition of the target population?
Predicting the likelihood of response using frame data: To what extent can sample members for which a higher incentive is effective (or ineffective) be reliably identified using the ACS frame data?

The NTEWS-specific experimentation will allow the ability to examine these research questions. Differences in the target populations and administration modes do not allow the generalization of the NHES-ATES and NSCG incentive experimentation results to the NTEWS.

The noncontingent-incentive experiment will test three alternative dollar values for a prepaid debit card, as well as a no-incentive control. The NTEWS sample will be randomly split into four equal-sized groups that will determine the value of the debit card sent with the first mailing: $0 (i.e., no debit card with the first mailing), $10, $20, and $30. The $0 group will be further split, with half receiving a $30 debit card with the final mailing (as a "late-stage incentive") and the other half receiving no incentive at any mailing.

Following on the research from the 2019 NSCG, NCSES plans to offer a $30 prepaid debit card incentive to subset of “highly influential” production SPs at week 1 of the 2022 NTEWS data collection effort. “Highly influential” refers to the cases with a large base weight and a low response/locating propensity. These highly influential cases will be identified by a model-based approach using a weighted response influence, which is a product of a sampled case’s case weight and predicted response propensity.

In addition to the production sample, the seeded sample and bridge panel sample will be incentivized during the first NTEWS mailing in a non-experimental manner. All members of the seeded sample will receive a $10 debit card (for more information on the seeded sample, see Appendix C) and approximately 20% of the bridge panel will receive a $30 debit card (for more information on the bridge panel, see Supporting Statement B).

10. Assurance of Confidentiality

The NCSES, NCES, and Census Bureau are committed to protecting the confidentiality of all survey respondents. The NTEWS data from the production sample and seeded sample are collected under the authority of the National Science Foundation Act of 1950, as amended, the America COMPETES Reauthorization Act of 2010, and the Education Sciences Reform Act of 2002. Also, data from both samples are protected from cybersecurity risks through the Federal Cybersecurity Enhancement Act of 2015.

The Census Bureau is conducting the NTEWS production sample under the authority of Title 13, Section 8 of the United States Code. The Census Bureau is required by law to keep the survey data from the production sample confidential and to use the data for statistical purposes only (Title 13, Section 9 of the United States Code).

The NTEWS seeded sample is a convenience sample and the data are not nationally representative. NCSES and NCES do not plan to release estimates from the seeded sample but instead plan to use the seeded sample data for research purposes. As such, the Census Bureau is conducting the NTEWS on behalf of NCSES and NCES, and the seeded sample data are not protected under Title 13, Section 9 of the United States Code. Instead, the seeded sample data are protected by a confidentiality law (20 U.S.C. §9573 and 6 U.S.C. §151) and may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law. The authorities to collect and protect respondents' information are written and provided to respondents through the printed and online versions of the Frequently Asked Questions (FAQs) (see Appendices D and E for English and Spanish contact materials/FAQs).

The NTEWS instrument for the production sample will include the following confidentiality statement on the web instrument, paper questionnaire, and (through interviewer reading) the CATI:

The information in this questionnaire is collected under the authority of the National Science Foundation (NSF) Act of 1950, as amended, the America COMPETES Reauthorization Act of 2010, and the Education Sciences Reform Act of 2002. The U.S. Census Bureau is conducting this survey under the authority of Title 13, Section 8 of the United States Code. The Census Bureau is required by law to keep your information confidential and can use your responses for statistical purposes only. The Census Bureau is not permitted to publicly release your responses in a way that could identify you. Federal law protects your privacy and keeps your answers confidential (Title 13, United States Code, Section 9). Per the Federal Cybersecurity Enhancement Act of 2015, your data are protected from cybersecurity risks through screening of the systems that transmit your data. Your response is voluntary, and failure to provide some or all of the requested information will not in any way adversely affect you. Actual time to complete the questionnaire may vary depending on your circumstances but on average, it will take about 15 minutes. If you have any comments on the time required for this survey, please send them to the Reports Clearance Officer, Office of the General Counsel, National Science Foundation, 2415 Eisenhower Ave., Alexandria, VA 22314.

The NTEWS instrument for the seeded sample will include the following confidentiality statement on the web instrument, paper questionnaire, and the CATI:

The information in this questionnaire is collected under the authority of the National Science Foundation (NSF) Act of 1950, as amended, the America COMPETES Reauthorization Act of 2010, and the Education Sciences Reform Act of 2002. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Per the Federal Cybersecurity Enhancement Act of 2015, your data are protected from cybersecurity risks through screening of the systems that transmit your data. Your response is voluntary, and failure to provide some or all of the requested information will not in any way adversely affect you. Actual time to complete the questionnaire may vary depending on your circumstances but on average, it will take about 15 minutes. If you have any comments on the time required for this survey, please send them to the Reports Clearance Officer, Office of the General Counsel, National Science Foundation, 2415 Eisenhower Ave., Alexandria, VA 22314.

NCSES, NCES, and the Census Bureau will operate within the guidelines established by the Privacy Act of 1974 to protect respondents' privacy and the confidentiality of the data collected. The Privacy Act states, "microdata files prepared for purposes of research and analysis are purged of personal identifiers and are subject to procedural safeguards to assure anonymity."

The Census Bureau, as the NTEWS data collection contractor on behalf of NCSES, has a long history of demonstrated experience in handling sensitive data. Established routine procedures will be in place to ensure data confidentiality, including the use of passwords and encrypted identifiers to prevent direct or indirect disclosures of information.

The bridge panel sample will not be collected by the Census Bureau but by a to-be-determined data collection contractor. NCSES will apply privacy protections and conduct a disclosure review before allowing any public dissemination of the bridge panel findings. Researchers working with the restricted-use data from the bridge panel will have to work within the NCSES data enclave, in which results are subject to disclosure review before export. All data from the bridge panel will be marked as probability based but not for the generation of official statistics.

11. Justification for Asking Sensitive Questions

The NTEWS does not ask questions of a sensitive nature.

12. Burden Estimate

Table 1 provides the estimated burden across all three samples. NCSES expects the NTEWS response rate across all three samples (production and non-production) to be 62% (30,565 total respondents). NCSES estimates 27,000 respondents of the 43,200 sampled adults (63% response rate) and 3,125 respondents out of the 5,000 from bridge panel (63% response rate). NCSES notes that the NSCG data collection expects a response rate between 65% to 75%, which is based only on sample cases with an education level at or above the bachelor's degree. As such, NCSES does not have a reliable source to estimate a response rate across all education levels when using an ACS-based sampling frame. As a result, NCSES estimates a response rate decrease for the NTEWS to account for the inclusion of subbaccalaureate cases.

NCSES estimates a 44% response rate for the NTEWS non-production 1,000 seeded sample based on the previous NCES seeded sample study. This rate results in 440 completed cases from the seeded sample.

SPs can respond to the 2022 NTEWS once. The amount of time to complete the survey may vary depending on an individual's circumstances and the collection mode (i.e., web, paper, or CATI). NCSES estimates an average of 15 minutes to review instructions, gather the data needed, and complete the NTEWS questionnaire (Appendices K and L, English and Spanish, respectively). The total burden hours for the 2022 NTEWS are 7,641 hours (30,565 completed cases x 15 minutes). The total cost to respondents for the 7,641 hours is estimated to be $298,763 (The average hourly wage $39.10⁶). Over the three-year OMB clearance period, the average annualized burden hours and cost to the public for the 2022 NTEWS are estimated to be 2,547 hours and $99,588, respectively.

Table 1. Estimated Burden Across 2022 NTEWS Samples

Sample Type	Sample Size	Estimated Number of Respondents	Estimated Response Rate	Estimated Burden Hours (15-minute Survey)	Average Annualized Burden Hours
Production	43,200	27,000	62.5%	6,750.0	2,250.0
Bridge	5,000	3,125	62.5%	781.3	260.4
Seeded	1,000	440	44.0%	110.0	36.7
Total	49,200	30,565	62.1%	7,641.3	2.547.1

13. Costs to Respondents

Not applicable. The NTEWS collection does not have a recordkeeping requirement that would impose additional costs to respondents (e.g., to purchase equipment, to purchase software, or to contract out services).

14. Costs to the Federal Government

The total estimated cost to the federal government for the 2022 NTEWS is approximately $11.6 million, including survey cycle costs and the NCSES and NCES staff costs to provide oversight of the NTEWS.

The estimate for survey cycle costs is approximately $10.7 million, which is based on sample size; length of questionnaire; administration; overhead; sample design; mailing; printing; sample person locating; web instrument development; telephone interviewing; incentive payments; data keying and editing; quality control; imputation for missing item responses; weighting and estimating sampling error; file preparation and delivery; and preparation of documentation and final reports.

The NCSES staff costs are estimated at $562,500 (based on a $150,000 annual salary of 1.5 FTE for 2.5 years). The NCES staff costs are estimated at $375,000 (based on $150,000 of 1 FTE for 2.5 years).

Over the three-year OMB clearance period, the average annualized cost to the Government is estimated to be $3,879,167.

15. Changes from the Prior Cycle

Not applicable. The 2022 NTEWS is the first collection cycle and does not have a prior cycle.

16. Plans for Tabulation or Publication

Table 2 presents the schedule of activities for the 2022 NTEWS. NCSES does not plan to use complex analytical techniques in publications. Typical cross-tabulations of the survey data are planned for inclusion in NCSES reports, NCES reports, and other data product releases. NCSES plans the following outcomes for the 2022 NTEWS:

Production Sample:

A fully documented public-use dataset that will be available for use within NCSES’s online data tools and for download from the NCSES website.
A fully documented restricted-use dataset that will be available for restricted-use data license holders only through the federal statistical research data centers.
A codebook with weighted and unweighted frequencies of all variables.
A methodology report.
NTEWS data products from NCSES and NCES that highlight key findings from the survey.

Bridge Panel Sample (non-production):

Data will be separated from the production survey data to ensure that bridge panel data cannot be used to generate official statistics.
A fully documented restricted-use dataset will be available for internal NCSES use and for approved restricted-use data license holders through the NCSES data enclave. Data use will be subject to NCSES’s disclosure review.
A codebook with weighted and unweighted frequencies of all restricted-use variables.
A methodology report.

Seeded Sample (non-production):

Data or products from the seeded sample will not be released. The seeded sample is collected for internal analysis by NCSES and NCES.

Table 2. 2022 NTEWS Data Collection: Tentative Schedule of Major Activities

Activity	Dates
(1) Survey Instrument Formatting and Printing	June 2021 – January 2022
(2) Data Collection	March 2022 – September 2022
(3) Data Processing and Quality Control Activities	October 2022 – Fall 2023
(4) NTEWS Data Products from NCSES and NCES	Winter 2023
(5) Public-use data files released	Winter 2023
(6) Restricted-use data files released	Winter 2023

17. Exception to Displaying the OMB Expiration Date

Not applicable. The OMB Control Number and expiration date will be displayed.

18. Exception to the Certification Statement

Not applicable. There are no exceptions to the certification statement.

1 42 U.S. Code § 1863(j)(1)

2 42 U.S. Code § 1885(a), 1885(d)

3 The 2013 NSCG survey cycle focused on developing operational capabilities for ASD, 2015 NSCG calibrated the development of ASD statistical and monitoring capabilities, 2017 NSCG aimed at increasing automation of existing capabilities and predicting the effects of data collection interventions, and 2019 NSCG incorporated adaptive design experiments that were similar to previous cycles, but with new experimental goals to: (1) reduce the root mean square error for key variables and (2) automate the identification and selection of cases for the interventions as well as the delivery of intervention file by collection mode.

4 The NTEWS plans to use a rotating four-panel design that will draw its sample from the ACS two years prior. Please note that the 2022 NTEWS will draw its sample cases from the 2018 ACS, rather than the 2020 ACS, because of concerns about reliability in that year’s collection as a result of the coronavirus pandemic.

5 Visit the NCSES’s Skilled Technical Workforce Initiative website for more information.

6 For the 2022 NTEWS, NCSES does not have an estimated median annual salary of a NTEWS employed respondent like the NSCG, which uses prior data collection to determine the total burden costs. As a reasonable alternative, NCSES will use BLS’s employer costs to employ a civilian worker as a proxy. At the time of this writing, the employer cost for a civilian worker was $39.01 per hour in March 2021 (total compensation which includes wages and salaries and benefits). Table 2. Civilian workers by occupational and industry group. Employer Costs for Employee Compensation for civilian workers by occupational and industry group available https://www.bls.gov/news.release/ecec.t02.htm [last modified date: June 17, 2021].

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	0000-00-00

A_ICR Statement A NTEWS_09-24-2021