B_ICR Statement B NTEWS_09-24-2021

The 2022 National Training, Education, and Workforce Survey (NTEWS) is a voluntary study, representing the first year of a biennial data collection by the National Center for Engineering and Science Statistics (NCSES) within the National Science Foundation (NSF). The National Center for Education Statistics (NCES) within the Department of Education is co-sponsoring the NTEWS, and the Census Bureau serves as the data collection contractor. This supporting statement section of the OMB package provides methodological descriptions and explanations for the 2022 NTEWS collection. Furthermore, as the 2022 NTEWS is the initial administration of a new survey effort, NCSES will use the information collected from these methods and experiments to guide and improve subsequent full-scale collections.

1. Universe and Sample Descriptions

The NTEWS is designed to represent the non-institutionalized American adult population ages 16 through 75 and not currently enrolled in high school. NCSES plans to use the 2018 American Community Survey (ACS) as the sampling frame for the 2022 NTEWS¹. The Census Bureau replaced the decennial census long-form with the ACS, which is conducted every month. The 2018 ACS was sent to a sample of about 3.5 million households in the 50 states, the District of Columbia, and Puerto Rico. Households are required to respond to the ACS by law or risk an imposed penalty (Title 13, USC §141 and §193). NCSES acknowledges that any missed housing units or missed individuals within sample households in the ACS would create undercoverage in the NTEWS. Additional undercoverage errors may exist because of self-reporting errors in the NTEWS sampling frame that led to incorrect classification of individuals, such as initially identifying respondents as being enrolled in high school when they were not at the time of the NTEWS interview. The sampling frame may also have an undercoverage of immigrants during the year ACS data were collected because some persons immigrated to the United States after the month when ACS would have sampled them. Despite these potential areas of undercoverage, NCSES will handle this undercoverage through a weighting adjustment. The NTEWS will not cover immigrants who arrive in the United States after 2018. As such, the ACS universe provides the NTEWS an enriched and robust sample to measure and understand work-related credentials attainment and the skilled technical workforce² (STW) at a national level.

In 2019, NCSES met with various federal governing boards to request the use of the ACS as the sampling frame for the initial survey cycle for the NTEWS. NCSES noted that using the ACS universe as the sampling frame would provide national estimates with reasonable precision within specific subgroups of the STW while minimizing the associated burden and cost to taxpaying respondents. The STW is a rare sub-population, comprising approximately 19 million people, or eight percent of the US adult population ages 16-75, according to the 2019 ACS.³ Further disaggregation by demographic characteristics implies that some estimation domains may make up less than one-half of one percent of the population. The ACS frame allows these rare subgroups to be identified and oversampled to achieve sufficient responses from such subgroups while keeping the total sample size relatively small. Through these meetings, NCSES received approval to sample from the 2018 ACS from the Census Bureau’s American Community Survey Office’s internal governing board on September 10, 2019, and from the Interagency Council on Statistical Policy’s Subcommittee on the ACS (ICSP-SACS) on October 21, 2019.

NCSES will use the 2022 NTEWS as the base year for a longitudinal, rotating panel survey design. This design is like that used in the National Survey of College Graduates (NSCG), also administered by NCSES (see Appendix G). There are several advantages to such a design. A rotating panel permits the following:

benchmarking of estimates to population totals derived from the sampling using the ACS;
a feasible sample size that allows for cross-sectional estimation for small populations;
an oversample of rare populations that provides for continued detailed estimation;
longitudinal analysis of the retained NTEWS individuals from the ACS-based sample; and
the examination of the feasibility and potential benefit of merging the NTEWS and the NSCG.

The 2022 NTEWS will include a production sample of 43,200 individuals selected from almost 3.3 million eligible 2018 ACS respondents, a bridge panel sample of 5,000 individuals, and a seeded sample of 1,000 individuals. The sample will consist of individuals who were: noninstitutionalized; resided in the United States, including the District of Columbia and Puerto Rico; aged 16 to 75 years old (inclusive) as of March 1, 2022; and were high school graduates, or otherwise not attending primary or secondary school. NCSES estimates an overall response rate of 62%, resulting in approximately 30,565 respondents of the 49,200 sampled individuals (See Table 1 from Supporting Statement A for complete burden estimate).

2. Information Collection Procedures

2.a Sample Design and Selection

Sample Allocation

The sample allocation for the 2022 NTEWS production sample will use a set of stratification variables, which are based upon the response information from the 2018 ACS. NCSES has defined these stratified cells by a crosstabulation of the following ACS variables: educational attainment (three levels), sex (two levels), underrepresented minority indicator (two levels), and science, technology, engineering, and mathematics (STEM) workforce status (two levels). The multiway cross-classification of these four stratification variables produces 24 non-empty sampling cells (3 x 2 x 2 x 2 = 24). The levels of stratification variables are as follows:

Educational Attainment

High school or less
Some college, but less than a bachelor’s degree
Bachelor’s degree or higher

Sex

Male
Female

Underrepresented Minority Indicator

Underrepresented minority⁴

Includes the following race/ethnicity groups:

Hispanic, any race
Non-Hispanic Black
Non-Hispanic American Indian/Alaskan Native

Not an underrepresented minority

Includes the following race/ethnicity groups:

Non-Hispanic White
Non-Hispanic Asian
Non-Hispanic Native Hawaiian/Pacific Islander
Non-Hispanic other race

STEM Workforce Indicator

In the STEM workforce
In the non-STEM workforce or not working

The sample size of 43,200 individuals will ensure that the 2022 NTEWS produces the data needed on demographics and occupational categories for the congressionally mandated report on Women, Minorities, and Persons with Disabilities in Science and Engineering (see 42 USC 1885d). To meet these data needs, NCSES has consulted with Jean Opsomer⁵, an expert on survey methodology, to inform the development of an appropriate sample selection approach to incorporate the NCSES precision requirements for NTEWS estimates. Through the consultation, NCSES has developed an approach that enables the implementation of a sample design with a coefficient of variation (CV) of 20% for estimates in the STEM workforce strata and 24% for estimates in the non-STEM workforce strata.

A minimum sample size is needed to meet the CV requirements for each of the three target variables by stratum:

Total number with a certification or license
Total number who have completed a work experience program
Total number with a postsecondary certificate

NCSES will randomly assign these characteristics to individuals on the 2018 ACS frame based on their corresponding frequency from the 2016 Adult Education and Training Survey (ATES) because the ACS does not collect information on these target variables. The following formula⁶ allows for a determination of the minimum sample size required for each target variable within each stratum.

The largest of these three minimum sample sizes will serve as the initial sample size (n_NTEWS) for that stratum, which should be sufficiently large enough to cover the precision requirements for all three target variables.

NCSES will incorporate a school enrollment criterion for sample persons from the 2018 ACS to identify those individuals enrolled in school as of the 2022 NTEWS data period. Thus, adjustments to the initial sample sizes will be made to account for the estimated number of ineligible sample individuals because the individuals were enrolled in primary or secondary school at the time of the NTEWS data collection. Therefore, these individuals will be ineligible for the survey.

In addition, adjustments to the initial sample sizes will be made for estimated nonresponse. As noted in section A.12, NCSES estimates an overall response rate of 62% to estimate the burden on the public. However, to ensure that the number of NTEWS respondents meets the NCSES precision requirements for NTEWS estimates, NCSES will use a conservative estimated response rate in the sample size calculation. Table 1 provides a summary of the stratum-level response rates used in the 2022 NTEWS sample size calculations. Without past NTEWS response information on which to base these conservative estimated response rates, NCSES used the 2019 NSCG response rates for the college-educated strata (Bachelor’s degree or higher). Given that the NSCG does not include coverage of the subbaccalaureate population, NCSES estimated the “some college, but less than Bachelor’s degree” strata responses rates for NTEWS by subtracting five percentage points from the NSCG response rate and estimated the “high school or less” strata by subtracting ten percentage points. These five and ten percentage point estimates were based on the NSCG’s pattern of decreasing response rate by education attainment level.⁷

The sample size will be the smaller of this nonresponse-adjusted sample size and the number of individuals available on the frame, rounded up to the next highest integer. For example, if NCSES wanted to sample 100 individuals to meet the precision requirements, but only 95 are available on the frame, then NCSES could only select a sample of 95 individuals. With 24 strata, NCSES does not expect to exhaust the entire available ACS frame within any stratum. As a result, the anticipated final production sample size is approximately 43,200 individuals.

Table 1. 2022 NTEWS Production Sample: Frame Size, Sample Size, Estimated Response Rate, and Respondents

2022 NTEWS	2018 ACS Frame Size	NTEWS Sample Size	Estimated Response Rate¹	Estimated Number of Respondents

Production Sample	3,289,000	43,200	62.5%	27,000

Production Sample Strata
Educational Attainment
High school or less	1,292,000	24,000	41.8%	10,022
Some college, but no bachelor’s degree	979,000	6,200	50.0%	3,102
Bachelor’s degree or higher	1,018,000	13,000	51.9%	6,748

Sex
Female	1,594,000	26,200	45.9%	12,028
Male	1,695,000	17,000	46.1%	7,844

Underrepresented Minority Indicator
Underrepresented minority²	755,000	26,200	41.3%	10,817
Non-underrepresented minority	2,534,000	17,000	53.3%	9,055

STEM Workforce
In the STEM workforce	588,000	13,700	50.1%	6,866
In the non-STEM workforce or unemployed	2,701,000	29,500	44.1%	13,006

Notes:

¹The production sample estimated response rate (62.5%) reflects the targeted overall survey response. The stratum-level estimated response rates are conservative estimates for use in the stratum-level sample size calculation. These conservative response rate estimates are used to ensure that the number of respondents meets the NCSES precision requirements for NTEWS estimates.

²This category comprises three racial or ethnic minority groups (Hispanics, any race; Non-Hispanic Blacks, and Non-Hispanic American Indians/Alaskan Natives).

Sample Selection

The 2022 NTEWS has three separate samples—production, seeded (non-production), and bridge panel (non-production). The 2022 NTEWS production sample of 43,200 individuals will be used to produce national estimates. It will be drawn from the final ACS sampling frame using a stratified systematic sample with probability proportional to size (PPS). Using this process, the likelihood of a person being selected into the sample varies with their measure of size. For 2022 NTEWS, NCSES will use the person’s Name and Address (NAD)-adjusted weight⁸ as their measured size. Thus, a person with a higher NAD-adjusted weight will have a higher probability of being selected into the sample.

Before systematic sampling is applied to the 24 sampling strata described above within the 2018 ACS frame to create the 2022 NTEWS production sample, NCSES will sort each stratum by the following characteristics: disability status, veteran status, race/ethnicity, occupation group, occupation science and engineering status, age group, education level, and Census division. This procedure will help ensure the NTEWS production sample reflects approximately the same distribution for these characteristics as the ACS frame, within each stratum. Without sorting, the production sample could exclude certain groups (e.g., a sample with no disabled persons).

The seeded sample of the 2022 NTEWS is a sample of approximately 1,000 individuals known to hold a postsecondary certificate. This sample will be used internally to evaluate measurement error in the survey’s certificate item; no data will be publicly released from this sample. NCES has obtained a list of certificate holders from four known credential bodies (three public community/technical college systems and one community college) from different geographic regions. NCSES and NCES are interested in the proportion of postsecondary certificate-holders who report on the NTEWS that they do not hold a postsecondary certificate, which provides an estimate of the certificate item’s underreporting rate. The seeded sample will inform decisions on (1) the inclusion of postsecondary certificate data in the NTEWS data products and (2) the need for further revisions to the postsecondary certificate questionnaire items for subsequent NTEWS survey cycles. Thus, the seeded sample will be a nonprobability convenience sample and is not expected to be nationally representative. Therefore, respondents from the seeded sample will not be weighted and will not be included in any national estimates. NCSES estimates the seeded sample response rate to be at or above 44%, which was the response rate from the 2016 ATES seeded sample. See Appendix C for more details on the seeded sample.

The expected response rate for the non-production bridge panel sample is expected to be like the production sample of 63%, which yields 3,125 respondents. Therefore, the sample plan for the bridge sample follows the sample plan for the production sample (Table 2). However, the bridge panel sample does not include the cells for the STEM Workforce or the “some college” group for two reasons. First, the bridge panel has a smaller sample size, and eliminating these small groups helps to provide a sufficient sample size for the remaining sample cells. Also, the bridge panel will use a commercial frame instead of the ACS. NCSES, in consultation with the selected data collection contractor, will determine the appropriate frame(s). The sample will be probability based, covering the 50 states, including the District of Columbia. NCSES plans to start with a complete address-based frame (e.g., the Postal Service’s Delivery Sequence File) and append commercial data to form the strata and facilitate alternate modes of data collection like CATI. Thus, the variables in Table 2 will be easily identified from a commercial frame. For more information on the NTEWS bridge panel experiment, see Section B.4 of this statement.

Table 2. 2022 NTEWS Bridge Sample: Sample Size, Estimated Response Rate, and Respondents

2022 NTEWS Bridge Panel	NTEWS Sample Size	Estimated Response Rate	Estimated Number of Respondents

Bridge Panel Sample	5,000	62.5%	3,125

Stratum
Educational Attainment
High school or less	2,813	57.7%	1,623
Some college or higher	2,187	67.2%	1,470

Sex
Female	2,963	63.5%	1,882
Male	2,037	61.1%	1,245

Underrepresented Minority Indicator
Underrepresented minority	2,963	58.8%	1,742
Non-underrepresented minority	2,037	67.9%	1,383

Weighting

The 2022 NTEWS weighting plan will be similar to the approach used in the NSCG, which is based on standard weighting procedures. NCSES will provide a final weight to reflect the 2022 NTEWS population. To produce the final weight, NCSES will start with a base weight for each case defined as the probability of selection into the 2022 NTEWS production sample from the 2018 ACS sampling frame. This base weight reflects the differential sampling across strata. Base weights will then be adjusted to account for unit nonresponse.

Weighting Adjustment for Survey Nonresponse

NCSES will use propensity modeling to account and adjust for unit nonresponse. When characteristics are available for all sample individuals (e.g., 2018 ACS responses and 2022 NTEWS data collection paradata), propensity modeling through logistic regressions can be used to predict response. The main advantage of this approach over the cell-collapsing approach, which was used for the NSCG during the 1990s and 2000s, is the potential to reallocate weight more accurately from nonrespondents to similar respondents, thereby attempting to reduce the potential for nonresponse bias. Propensity modeling also has the advantage of avoiding the creation of complex cell-collapsing rules for non-interviews.

To create a propensity model for NTEWS, NCSES will predict response using the existing variables from the sampling frame for both respondents and nonrespondents. A logistic regression model will use “a response” as the dependent variable. The propensities output from the model will be used to categorize individuals into cells of approximately equal size, with similar response propensities in each cell. The non-interview weighting adjustment factors will be calculated within each of the cells. The weight of the nonrespondents will be redistributed to the respondents and ineligibles within the cells of the 2022 NTEWS sample formed by the propensity model. After the non-interview adjustment, weights will be controlled to ACS population totals through a post-stratification procedure that ensures the population totals are upheld.

Weighting Adjustment for Extreme Weights

Extreme weights can significantly increase the variance of survey estimates. Should NCSES find relatively large weights following the weighting steps outlined above, NCSES will adjust for extreme weights within the sampling data strata or collapsed strata. These “trimming” adjustments can introduce bias into the final survey estimates. However, by trimming the extreme weights, the assumption is that the decrease in variance will offset the associated increase in bias, resulting in final survey estimates with a smaller mean square error.

Weighting Adjustment to Reallocate Weights

Due to trimming, the weighted population totals within each of the analytical domains may be less than the pre-trimmed weighted population totals. This issue leads to biased estimates of totals. NCSES will reallocate the trimmed weights through iterative raking to pre-trimmed weights totals within key domains to reduce bias.

Weighting Adjustment for Second Post-Stratification

Finally, NCSES will perform an additional execution of the post-stratification procedure to control the ACS population totals within pre-defined adjustment cells (defined by the cross of key stratification variables).

Standard Errors

Because the NTEWS uses a stratified sample rather than a simple random sample, standard procedures for calculating standard errors will produce inaccurate error estimates. To allow users to produce correct standard errors, the NTEWS data will include one set of replicate weights, which will be developed using the successive difference replication method. The variance of a survey estimate based on any probability sample may be estimated using the replication method. This method requires that the estimation procedures be independently carried through (replicated) several times. The dispersion of the resulting replicated estimates can then be used to measure the variance of the full sample.

2.b Non-Sampling Error Evaluation

To understand the potential impact of non-sampling errors on 2022 NTEWS survey estimates, NCSES, with the assistance of the Census Bureau, will develop a non-sampling error report. Two areas of non-sampling error can be informative: (1) nonresponse error and (2) error as a result of the inconsistency between the 2018 ACS and 2022 NTEWS responses (measurement error).

Nonresponse Error

Several metrics will be computed to detect nonresponse error, such as unit response rates and estimates of key domains. Unit response rates refer to the percentage of the sample that responded to the survey. Individuals must respond to a few survey items identified as critical items⁹ to be considered a complete survey. Some variation in response among respondent subgroups (e.g., males versus females) is expected due to random variation; however, large variations in response behavior can cause concern with the potential to introduce nonresponse bias. For example, suppose a subgroup is under-represented in the respondent sample, and that subgroup also is different from other subgroups on an outcome measure. In that case, the under-representation (i.e., the lower response rate) can result in a biased outcome estimate.

Examining the estimates of key domains provides evidence of the extent to which bias due to nonresponse error adversely affects the survey estimates. Nonresponse weighting adjustments are made to the respondent population to account for nonresponse, ensuring the respondent population represents the target population in size. The nonresponse weight adjustment accounts for differential response rates by subgroup (i.e., lower responding groups being underrepresented and higher responding groups being overrepresented). It, therefore, brings the weighted respondent distributions closer to the sample distributions.

NCSES will compare key survey estimates obtained using three sets of weights: (1) base weights for the sample, (2) base weights for the respondents, and (3) nonresponse adjusted weights for respondents. This procedure gives an indication of the potential for nonresponse bias and shows how much nonresponse bias is reduced through an appropriate weighting adjustment. This examination will provide insight into whether the survey’s weighting adjustments appropriately meet the NTEWS survey estimation goals.

For the subsequent NTEWS cycles (when more data are made available and other ACS sample frames are used), NCSES plans to use compound response rates and R-indicators to evaluate nonresponse error. The compound response rate looks at response rates over time and considers how attrition can affect the respondent population. Attrition is important when considering the effect of nonresponse in rotating panel surveys like the NTEWS. Also, R-indicators and corresponding standard errors will be examined on the originating ACS sources of sample for the NTEWS (e.g., 2018 ACS, 2022 ACS, 2024 ACS, and 2026 ACS). R-indicators are useful, in addition to response rate and domain estimates, for assessing the potential for nonresponse bias. R-indicators are based on response propensities calculated using a predetermined balancing model (“balancing propensities”), which provide information on how different the respondent population is compared to the full sample population. Also, an evaluation of R-indicators can identify which variables in the predetermined model drive the variation in nonresponse.

Error Resulting from ACS and NTEWS Response Inconsistency

Information from the 2018 ACS responses is used to determine NTEWS eligibility and to develop the NTEWS sampling strata. To develop a sample design that will efficiently and effectively meet NCSES estimation goals, the NTEWS uses different sampling rates in each of the strata. Examining the consistency between ACS responses and NTEWS responses on the questionnaire items used to develop the NTEWS stratification variables provides insight into the efficiency of the sample design and the impact on the variance of the survey estimates.

3. Statistical Accuracy of the Collection

To maximize the overall survey response rate, NCSES will implement procedures such as conducting extensive locating efforts and collecting the survey data using three different modes (mail, web, and CATI).

Respondent Locating Techniques

As the NTEWS data collection contractor, the Census Bureau will refine and use a combination of locating and contact methods to maximize the survey response rate. The Census Bureau will utilize all available locating tools and resources to make the first contact with the sample person. The Census Bureau will use the U.S. Postal Service’s (USPS) automated National Change of Address (NCOA) database to update addresses for the sample. The NCOA incorporates all change of name/address orders submitted to the USPS nationwide and is updated, at minimum, on a biweekly basis.

Before mailing the survey invitation letters to the sample members, the Census Bureau will engage in locating efforts to find good addresses for problem individuals (“upfront locating”). The mailings will use the “Return Service Requested” option to ensure that the postal service will provide a forwarding address for any undeliverable mail. In most cases, the initial mailing to the NTEWS sample members will be a letter introducing the survey and inviting them to complete the survey by the web, paper, or telephone mode (see Appendix C for more information on the NTEWS methodological experiments).

Data Collection Methodology

The 2022 NTEWS will use a 27-week data collection period to collect data from its sample of approximately 43,200 individuals. The initial cycle will gather baseline data to improve the data quality and maximize response rates for subsequent full-scale NTEWS survey cycles (NCSES anticipates the full-scale rotating panel design planned for the NTEWS will sample about 140,000 individuals). As discussed in the Supportive Statement A (A.9 Payments or Gifts to Respondents) and Appendix C, NCSES plans to implement two experiments on the production sample:

A contact-strategies experiment to test six sequences of contact modes, utilizing various combinations of three response modes (web, paper, and telephone).
A noncontingent-incentive experiment to test three alternative dollar values ($10, $20, $30) for a prepaid debit card, including a late-stage offering of an incentive, as well as a no-incentive control group.

A power analysis was developed for the methodological experiments on the production sample (see Appendix M for the NTEWS power analysis). The purpose of a power analysis is to evaluate the sufficiency of the 2022 NTEWS sample for detecting differences among treatment groups. Through the power analysis, NCSES confirmed that the planned comparisons could be conducted with acceptable power for minimum detectable differences between treatment groups.

Table C.2 of Appendix C provides an overview of the NTEWS data collection methodology by week across the different contact-strategies groups. At the start of data collection, all individuals will be mailed a letter inviting them to respond to the NTEWS, with the letters tailored for each of the six contact-strategies treatments. Across the contact-strategies experiments, the individuals will receive either no-incentive or one of three incentive dollar amounts as prepaid debit cards. In addition, some individuals will receive an incentive in week 23 as a late-stage strategy. Thus, for 2022 NTEWS, approximately 77% of the sample will receive some incentive treatment.¹⁰ Treatment groups will be randomly assigned in a cross-factorial design (see Table 3 for a breakdown of NTEWS production sample by contact-strategy and incentive treatments).

These experiments will help NCSES understand the independent effects of alternative contact strategies and noncontingent incentives. NCSES plans to examine any potential interaction effects between the contact and noncontingent incentives treatments. In particular, NCSES seeks to encourage the participation of “highly influential” individuals, which have a large sampling weight but a low response/locating propensity (i.e., hard-to-reach). Through these 2022 NTEWS experiments, NCSES can identify an optimal data collection strategy for future NTEWS cycles that maximizes response rates while minimizing nonresponse bias, and as much as possible, cost to the federal government and burden on the public. It is thus expected that future NTEWS administrations will include less extensive, more targeted incentive use.

Table 3. Expected number of sampled individuals, and expected number receiving any incentive, by experimental treatment group
Noncontingent-incentive treatment	Contact-strategy treatment
Noncontingent-incentive treatment	NSCG-style	NHES-style	Choice	Choice-plus	Paper-first	CATI-first
Expected number of sampled individuals (~43,200)
No noncontingent incentive	1,181	1,181	591	591	1,181	525
$10 with week 1 mailing	2,363	2,363	1,181	1,181	2,363	1,050
$20 with week 1 mailing	2,363	2,363	1,181	1,181	2,363	1,050
$30 with week 1 mailing	2,363	2,363	1,181	1,181	2,363	1,050
$30 with week 23 mailing	1,181	1,181	591	591	1,181	525
Expected number receiving any incentive (~32,440)
No noncontingent incentive	0	0	0	207	0	0
$10 with week 1 mailing	2,363	2,363	1,181	1,181	2,363	1,050
$20 with week 1 mailing	2,363	2,363	1,181	1,181	2,363	1,050
$30 with week 1 mailing	2,363	2,363	1,181	1,181	2,363	1,050
$30 with week 23 mailing	118	118	59	266	118	53

NOTE: Due to rounding, details may not sum to totals. Detailed information about each experimental treatment is provided in Experiment Appendix C.

Along with these experiments, the following steps will be taken to maximize the response rates and minimize nonresponse within the NTEWS data collection:

Providing “user-friendly” survey materials that are simple to understand and use (See Appendices D (English version) and E (Spanish version) for contact materials – FAQs and letters – for the 2022 NTEWS)
Sending attractive, personalized material, making a reasonable request of the respondent’s time, and making it easy for the respondent to comply
Using priority mail and post-card reminders for targeted mailings to improve the chances of reaching respondents and convincing them that the survey is important
Devoting significant time to interviewer training on how to deal with problems related to nonresponse and ensuring that interviewers are appropriately supervised and monitored
Implementing refusal-conversion strategies that specifically address the reason why a potential respondent has initially refused, and then training conversion specialists in convincing counterarguments

Taken together, NCSES confidently believes these strategies will maximize response rates and will yield accurate, unbiased, and reliable 2022 NTEWS data and estimates. Furthermore, NCSES expects the information collected through the methodological experiments and the knowledge gained through the 2022 NTEWS data collection operations will introduce improvements and efficiencies into the subsequent NTEWS survey cycles.

4. Testing Procedures

4.a Cognitive Research Studies

In accordance with OMB’s Statistical Policy Directive No. 2 and its addendum on cognitive interview, NCSES conducted several rounds of cognitive testing to inform the questionnaire development for NTEWS. Following the completion of the 2016 Adult Training and Education Survey (ATES), which was administered by NCES as part of its National Household Education Surveys (NHES) program, NCES conducted cognitive research in preparation for future administrations of the ATES. After the cognitive research was completed, a change in NCES’s funding priorities led to the discontinuation of the ATES program. However, with the NTEWS building on the research goals of the ATES, NCSES used the findings from the ATES cognitive research to directly inform the content development for the 2022 NTEWS data collection instrument.

As noted in Statement A, the NTEWS is expected to have increased utility compared to the ATES because the NTEWS will provide more reliable skilled technical workforce (STW) estimates through the allocation of a larger sample to individuals in skilled technical occupations. Despite the similarities between the NTEWS and ATES survey content, NCSES felt the ATES cognitive testing was insufficient for NTEWS purposes. In response, NCSES conducted additional cognitive testing in preparation for the 2022 NTEWS and its STW emphasis. See Appendix N for information on the cognitive research conducted as part of the 2022 NTEWS questionnaire development. See Appendices O and P on NCSES’s efforts to ensure the 2022 NTEWS questionnaire meets OMB’s standards of quality and utility. See Appendices K and L for the NTEWS questionnaire in English and Spanish.

4.b Non-production Bridge Panel

The nature of educational attainment, the labor market, and workforce-related training is changing. In response to this changing environment, and further motivated by a recommendation from the National Academies of Science, Engineering, and Medicine’s Committee on National Statistics,¹¹ NCSES would like to explore modifications to the NTEWS survey content. Standard cognitive testing of question wording modifications provides a qualitative measure of quality and usability but does not assess or quantify the potential impact on survey estimates. Given the importance of establishing the NTEWS and extending the NSCG’s trend data, NCSES plans to include a small, representative, non-production sample (referred to as a bridge panel) to quantify the potential impact of question wording modifications on key survey estimates.

The bridge panel will allow NCSES to compare the 2022 NTEWS survey estimates (using responses from the NTEWS production sample) with estimates resulting from the modified questions (using responses from the bridge panel). Thus, the bridge panel would serve as a bridge to NCSES’ current questions and could aid in improvements to future cycles of the NTEWS through concurrent question-wording testing while NTEWS is in full production. For example, the 2022 NTEWS bridge panel will test the inclusion of sexual orientation and gender identity (SOGI) questions in the demographic section. In future cycles, the bridge panel would provide NCSES the opportunity to assess and quantify the impact to survey estimates of potential methodological changes.

Sample Design and Selection

The 2022 NTEWS non-production bridge panel will include a sample of 5,000 individuals selected from a high-quality commercial vendor source, which has successfully been used for other federal surveys, such as NHES. The target population of the 2022 NTEWS bridge panel is similar to the production sample designated for NTEWS, with some minor changes to accommodate the smaller sample size. The bridge panel sample will target adults ages 16-75 living in the US, including the District of Columbia but excluding Puerto Rico.

As part of the 2022 NTEWS sample selection effort, the 5,000-case non-production bridge panel will be sampled separately from the 43,200-case production sample. The sample selection for the bridge panel will use stratification variables like those used for the new sampled individuals, as discussed in Section 2 of this Statement. These stratification variables will be formed using information on the commercial frame. After determining the sample allocation per sampling cell, individuals will be selected using systematic probability proportional to size sampling. The use of aggregated versions of the new sample stratification variables will enable comparison of key estimates between the bridge panel and the new sample.

Weighting Procedures, Replicate Weights, and Standard Errors

Estimates from the 2022 NTEWS bridge panel will be based on standard weighting procedures. As was the case with sample selection, the weighting adjustments will be done separately for the bridge panel and production sample individuals. The goal of the separate weighting processes is to produce final weights for the bridge panel. To produce the final weights, the bridge panel will follow the weighting methodology outlined in Section B.2 of this Statement. In addition, sets of replicate weights, using the successive difference method, will be constructed to allow for separate variance estimation for the bridge panel.

Questionnaire and Survey Content

The 2022 NTEWS bridge panel questionnaire will include content similar to the new sample questionnaire included in Appendix Q (Bridge Panel Web Questionnaire) but with modifications to allow for the exploration of challenges that were seen in testing. The new questions or modifications address the following:

The educational history section will first ask respondents to provide a roster of each non-degree credential earned. Then, two credentials will be chosen from this roster to solicit their degree history and confirm correct classification of the credential.
To understand how respondents define their “main job”, they will be asked to roster all jobs and answer follow-up questions as to which is their main job and why.

In addition, the bridge panel questionnaire collects data on several new topics that either could not be included in the production questionnaire, were requested in a comment to the first FRN, or are questions that NCSES would like to develop and improve their surveys in general. These changes are:

A new question asks if the credential was received outside of the US.
A new question asks if any of the credential requirements were completed online.
A couple of new questions determine if the respondent was willing to move after receiving the credential and how far. NCSES will ask the data collection agency to program this as a map of the United States that allows the respondent to designate which states they were considering.
Two questions ask if the respondent’s employer covered some or all of the costs to get the credential.
Within the demographics section,
1. a new 3 question series asks respondents about their sex at birth, current gender identity, and sexual orientation (SOGI).
2. the marriage question was altered to remove the confound between romantic status and living situation.
3. several new disability questions were added to the disability series to represent additional limitations specific to this population.
4. a 3-question series explores position of the respondent within the household and the household’s reliance on the respondent’s income.
5. the questions on race and ethnicity were altered into one question that is all inclusive and may allow respondents to report their ethnic origins more accurately.

In addition, for the questionnaire items that were modified for 2022 to include coronavirus pandemic response options (i.e., employment status, part-time employment, job benefits, earnings, and conference attendance), the question wording without the coronavirus pandemic response options will be used to compare the impact of the pandemic.

Data Collection Methodology

The bridge panel will use a single-mode, web-based data collection protocol. Data collection will be administered by a survey contractor with the flexibility to program the questionnaire, given the auto-fills and map innovation. Given the smaller number of individuals, the bridge panel individuals will likely require a shorter field period than NTEWS, targeting 12 weeks. The bridge will follow the “web first pathway” with two modifications given the single web mode: (1) bridge panel individuals will not receive a paper questionnaire and (2) no outgoing telephone calls will be made to the bridge panel individuals. The bridge panel will receive English and Spanish survey mailing materials similar to those planned for the production sample individuals (see Appendices D and E – Contact Materials) and the web survey will allow for completion in English or Spanish.

NCSES plans to offer a $30 prepaid debit card to a subset of highly influential bridge panel individuals at week 1 of the 2022 NTEWS data collection effort. “Highly influential” refers to individuals that have large sampling weights and a low response/locating propensity. NCSES expects to offer the debit card incentives to approximately 1,000 of the 5,000 bridge panel individuals at the first contact. In addition, at week 6, 50% of the remaining individuals would be offered nonresponse follow-up incentives, considering which are most influential for balancing the sample. These debit cards will have a six-month usage period at which time the cards will expire, and the unused funds will be returned to the Government.

Evaluation of Bridge Panel Question Modifications

As noted earlier, the bridge panel is designed to provide NCSES with an opportunity to compare current NTEWS survey estimates (using responses from the NTEWS production sample) with estimates resulting from the modified questions (using responses from the bridge panel sample). This comparison is designed to improve the NTEWS for future cycles with possible question-wording modifications.

To determine if the modified questionnaire items should be included in subsequent NTEWS survey cycles, NCSES will conduct an evaluation of the bridge panel question modifications at the completion of the 2022 NTEWS data collection effort. The evaluation between the production and bridge panel samples will include these three components:

Comparison of survey estimates for the modified questionnaire items. This comparison will be at both the STW and credential levels and the level of the bridge panel stratification variables.
Comparison of overall nonresponse rates and item-level nonresponse rates.
Comparison of web instrument paradata (e.g., breakoff rates, changed answer rates, etc.) to assess the user experience associated with the modified questionnaire items.

NCSES will also address the different frames and data collection methodologies as part of the overall evaluation because direct comparisons may not be possible. This evaluation approach is designed to provide insight on both the estimation impact of these question modifications as well as any non-sampling error issues that would be introduced through the inclusion of these questions.

5. Contact Information

The following NCSES persons can be contacted on the statistical aspects of the design and data collection:

John Finamore, Chief Statistician

National Center for Science and Engineering Statistics

National Science Foundation

(703) 292-2258

Matthew Williams, Mathematical Statistician

National Center for Science and Engineering Statistics

National Science Foundation

(703) 292-8238

The following contacts from the US Census Bureau will be responsible for collecting data for the 2022 NTEWS through an Interagency Agreement:

Stephen Simoncini, NTEWS Survey Director

Associate Director for Demographic Programs (ADDP)

Census Bureau

(301) 763-4816

Chief consultant on data collection and methodological issues.

Aaron Gilary, Lead Scientist

Demographic Statistical Methods Division (DSMD)

Census Bureau

(301)763-9660.

DSMD manages NTEWS sample selection operations and statistical issues.

1 The NTEWS will draw its sample from the American Community Survey (ACS) two years prior, with one exception: the 2022 NTEWS drew from the 2018 ACS instead of the 2020 ACS because of concerns about reliability in that year’s collection as a result of the coronavirus pandemic.

2 The STW includes individuals who use science, technology, engineering, and mathematics (STEM) skills and knowledge in their jobs but do not have a bachelor’s degree.

3 See figure LBR-2 from National Science Board (2021). The STEM labor force of today: Scientists, engineers, and the skilled technical workers. Science & Engineering Indicators. https://ncses.nsf.gov/pubs/nsb20212.

4 This category comprises these three racial or ethnic minority groups whose representation in science and engineering education or employment is smaller than their representation in the U.S. population.

5 Opsomer, J. (2019a). Consultation Request #1 – Identify minimum information necessary to determine sample size needs for the pilot data collection effort. Provide expert consultation in the development of an appropriate sampling frame and sample design to measure the skilled technical workforce. February 11, 2019, memorandum for the National Center for Science and Engineering Statistics that examines the feasibility of collecting data on the skilled technical Workforce (STW) through an adult training and education survey.

Opsomer, J. (2019b). Consultation Request #2 – Determine minimum sample size under alternative stratification scenarios. Provide expert consultation in the development of an appropriate sampling frame and sample design to measure the skilled technical workforce. February 11, 2019 memorandum for the National Center for Science and Engineering Statistics that examines the feasibility of collecting data on the skilled technical Workforce (STW) through an adult training and education survey.

6 In the formula, n_NTEWS represents the NTEWS sample size, n_ACS represents the ACS frame size, w represents the ACS person-level weight, V_NTEWS represents the estimated variance of the target population estimate within the NTEWS sample, and V_ACS represents the variance of the target population estimates within the ACS frame stratum.

7 The NSCG pattern of response rate is as follows: individuals with a bachelor’s degree have a lower response rate than individuals with a master’s degree, while individuals with a master’s degree have a lower response rate than individuals with a doctorate. Without data available to estimate the response rate for the subbaccalaureate population, NCSES extrapolated the decreasing response rate to the “some college” and “high school or less” strata and suggested the five and ten percentage point adjustments, respectively. For subsequent NTEWS survey cycles, the previous cycle’s response rates will be used in the sample size determination calculation.

8 NAD-adjusted weights are the final ACS simplified person-level weights adjusted for the removal of individuals with bad names or incomplete addresses from the preliminary sampling frame.

9 Individuals must respond to the following NTEWS survey items (aka critical items): educational attainment, current high school enrollment, living in US, working for pay or profit, looking for work, last worked (month/year), job title and description, number of hours worked, work activities, and birth date (month/year).

10 The 77% applies to the 43,200 production sample. Separately, all 1,000 individuals in the seeded sample will receive a $10 noncontingent incentive in week 1.

11 At NCSES’s request, CNSTAT convened an expert panel to review, assess, and provide guidance on NCSES’s efforts to measure the S&E workforce population in the United States. Recommendation 5.2 of the panel’s consensus study report noted that NCSES “should continue to monitor, and formally evaluate as needed, the content of its survey questionnaires to ensure that the concepts and terminology are up to date and familiar to respondents. Changes should be implemented with careful consideration of their impact on trend data.”

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Gigi Jones
File Modified	0000-00-00
File Created	2021-10-03

B_ICR Statement B NTEWS_09-24-2021