NTEWS Evaluating for Utility and Quality

APPENDIX O

NTEWS Evaluating for

Utility and Quality

National Training, Education, and Workforce Survey (NTEWS): Evaluating for Utility and Quality

This appendix summarizes the National Center for Science and Engineering Statistic’s (NCSES) effort to develop a high-quality survey that produces data with high utility value for public use. The effort involved documenting priority research questions from the two NTEWS sponsoring agencies (NCSES and the National Center for Education Statistics) and using these priority research questions as a guide to evaluate the quality and utility for each NTEWS questionnaire item. The results of this effort can be found in the Utility-Quality Itemized Grid (see Appendix P).

Motivation and Purpose

As is the case with all federal surveys, the NTEWS is designed to align with OMB’s standards and guidelines for statistical surveys. While the NTEWS could be perceived as a continuation of the Adult Education and Training Survey (ATES), a discontinued survey conducted by NCES, the NTEWS is a new survey with the following characteristics that differ from the ATES:

• NTEWS is funded by a new funding source (NCSES).

• NTEWS has a special focus on the skilled technical workforce (STW).

• NTEWS will employ a new sampling technique by using the American Community Survey (ACS) as the sampling frame.

As noted in OMB’s standards and guidelines, agencies initiating a new survey or major revisions of an existing survey must provide justification for the survey, including its goals and objectives, intended uses, and the purpose and quality of the proposed data. The standards specifically stress the importance for the sponsoring agencies to provide evidence that the proposed survey will (1) collect high quality data and (2) collect data that is of high utility value in that it will address the survey’s intended uses. To justify the quality and utility of the data being proposed in the NTEWS, NCSES evaluated the 2022 NTEWS questionnaire and produced an itemized record of the survey’s utility and quality (see Appendix P).

NTEWS Foundational Priorities

The following foundational priorities guided the development of the NTEWS questionnaire content.

1. The initial cycle of the NTEWS is designed as a production survey that will provide nationally representative data. The data will be released to the public and will serve as a baseline for subsequent NTEWS cycles. The first cycle of NTEWS should not be considered a pilot study.

2. Both of the NTEWS sponsoring agencies have specific research uses intended to inform national policy discussions on the STW and the credentialling of working U.S. adults.

3. The NTEWS questionnaire items must meet both quality and utility standards. Items that do not meet the quality or utility standards for the initial cycle of the NTEWS will be considered for subsequent cycles if the quality/utility deficiencies are addressed (e.g., through question-wording revisions and additional testing).

4. During NTEWS planning discussions, OMB suggested NCSES and NCES take steps to document the quality and utility for each item included on the final NTEWS questionnaire.

The Review Process

In the fall of 2019, NCSES develop a draft NTEWS questionnaire that drew from several federal surveys (ATES, the National Survey of College Graduates, and the ACS). To ensure that the questionnaire content met OMB’s quality and utility standards, NCSES assessed the alignment of the NTEWS content and the following research questions that each agency is seeking to inform:

NCSES Priority Research Questions

1. What are the demographic characteristics of the STW?

2. What is the prevalence and role of education and/or credentials for the STW?

3. Do education and/or credentials influence an individual’s ability to enter, maintain relevance, and seek advancement in the STW?

4. What are the employment characteristics of the STW?

5. What are the employment outcomes of the STW?

6. How is education and/or credential attainment related to employment outcomes?

NCES Priority Research Questions

1. How many adults, and which adults, have an education and/or work credential, or some combination of credentials?

2. How do work and education credentials interact (e.g., to what extent do adults at each educational attainment level have a work credential)? To what extent do work credentials supplement versus supplant higher education?

3. To what extent is the formal education system involved in providing training for work credentials, and what level of education is involved?

4. How is the attainment of a work credential related to key employment outcomes (e.g., unemployment, under-employment, earnings, job satisfaction?

5. How many adults, and which adults, are in the pipeline to earn a license or certification for the first time?

6. What role does credential attainment play in adults’ entry, advancement, and mobility in the labor market?

7. To what extent do adults work in fields related to their credentials?

To evaluate if a questionnaire item informs a priority research question, NCSES identified and operationalized the following “utility” and “quality” definitions:

Utility refers to the usefulness of the information that is disseminated to its intended users.

For the review, NCSES evaluated the utility of the NTEWS questionnaire by attempting to map each questionnaire item to a research question. If the questionnaire item did not align with a research question, it would not be useful information for the federal government to collect and, thus, did not meet the utility standard.

Quality refers to assuring: (1) that the questionnaire items ask what the sponsoring agencies intended, (2) that the questionnaire items are consistently understood as intended, (3) that respondents are willing and able to consistently answer the items as intended, and (4) that providing the full response is not overly burdensome.

Within the review, NCSES evaluated the quality of the NTEWS questionnaire by examining if each questionnaire item will result in precise, rigorous data through the testing and assessment of the question wording. An overarching goal from the quality perspective is for the questionnaire items to work as intended, and there is no misinterpretation or confusion by the respondent.

For each NTEWS questionnaire item, NCSES evaluated if it met either the utility or quality standard. Questionnaire items that met both the utility and quality standards were considered core survey items for the final 2022 NTEWS questionnaire. Questionnaire items that did not meet either the utility or quality standard were not included on the final questionnaire. Further revision and testing of these items are needed to meet the required utility and quality standards for possible inclusion in future NTEWS cycles. The result of this review process for the final 2022 NTEWS questionnaire is documented in Appendix P.