Federal Register Notification (FRN): Comments and Responses

APPENDIX H

60-day Federal Register Notification,
Comments, and Responses
https://www.federalregister.gov/documents/2020/04/16/2020‐08067/agency‐
information‐collection‐activities‐comment‐request 

1 of 2

https://www.fdms.gov/fdms/getcontent?objectId=09000064846f1708&f...

As of: 6/22/20 1:30 PM
Received: June 15, 2020
Status: Pending_Post
Tracking No. 1k4-9ha0-skha
Comments Due: June 15, 2020
Submission Type: API
Docket: NSF_FRDOC_0001
Recently Posted NSF Rules and Notices.
Comment On: NSF_FRDOC_0001-2485
Agency Information Collection Activities; Proposals, Submissions, and Approvals
Document: NSF_FRDOC_0001-DRAFT-0425
Comment on FR Doc # 2020-08067

Name: Jeff Grann
Address:
4163 COLUMBINE CT.
VADNAIS HEIGHTS, MN, 55127-6143
Email: [email protected]
Phone: 6512085748
Organization: Credential Engine

Credential Engines comments regarding document number: 2020-08067, publication date: 4/16/2020
June 15, 2020
Credential Engine supports the proposed development of the National Training, Education, and
Workforce Survey (NTEWS).
We offer a few comments here suggesting to expand the survey topics, to improve the interpretation of
findings by coordinating credential references with the CTDL schema, and to build upon the
supplemental sources of data provided via Credential Engines Registry.
Survey topics: Credential Engine has documented over 730,000 credentials being offered throughout
America. The breadth of these offerings makes it challenging for the public to evaluate these options
and make well informed enrollment decisions. As such, we recommend that NTEWS consider
expanding its data collection topics to also include Americans credential information seeking behavior
in order to help target career counseling and related outreach efforts. We believe this expansion is
especially important in order to increase social mobility and promote more equitable outcomes. For
example, Strada Education Networks Public Viewpoint: COVID-19 Work and Education Survey Updated June 10, 2020 found that, compared to other racial and ethnic groups, black Americans rank
advice about education and training from colleges and universities as less valuable than advice from

6/22/2020, 1:31 PM

2 of 2

https://www.fdms.gov/fdms/getcontent?objectId=09000064846f1708&f...

other sources. By including questions concerning this topic, NTEWS could provide consistent
information about how the skilled technical workforce learns about credential offerings and how those
offerings are evaluated relative to an enrollment decision. Credential Engine is particularly interested in
using such data to ensure credential providers are providing the most helpful information about their
offerings via the most effective channels.
Interpretation of findings: A founding purpose of Credential Engine has been to develop, maintain, and
expand the openly-licensed Credential Transparency Description Language (CTDL) schema, which
currently defines over 500 terms used to describe many aspects of credentials. Currently, CTDL is
being used by government agencies across seventeen states to communicate credential offerings in their
jurisdiction to the public. We recommend coordinating all credential terms used in the future NTEWS
with the CTDL schema in order to help the research community better interpret survey findings and to
facilitate analysis with the broad credential offering marketplace. For example, NTEWS respondents
may fail to consistently interpret certifications and therefore over report instances of obtaining various
training certificates used for compliance or regulatory purposes. We welcome additional discussion on
how the CTDL schema could be used to improve the NTEWS design and also facilitate the
interpretation of findings relative to the credential marketplace.
Supplemental sources of data: Credential Engine maintains a Registry of linked open data on all
credential offerings, which could serve as a valuable supplemental source of data. Verified accounts can
access data in the Registry via an API, an html widget, or via a bulk download. These data are also
made available to the public via the demonstration website https://credentialfinder.org/. You can
register via an account using the following linked quick start guide https://credreg.net/quickstart/search.
We would welcome the opportunity to discuss how the hundreds of thousands of data points already
available in the Registry concerning credentials of all types could supplement this research and
potentially reduce respondents reporting burden.

6/22/2020, 1:31 PM

1 

(FRN COMMENT #1)
September 24, 2021
Jeff Grann
Credential Engine
4163 Columbine Court
Vadnais Heights, MN 55127-6143
[email protected]
RE: NCSES Responses to 60-day Public Comment Period
Dear Jeff,
Thank you for responding to the National Center for Science and Engineering Statistics
(NCSES) within the National Science Foundation call for comments published in the Federal
Register on April 16, 2020 (FR Doc. 2020-08067), which announced our plan to ask the Office
of Management and Budget for authority to conduct the National Training, Education, and
Workforce Survey (NTEWS). The Paperwork Reduction Act (PRA) provides an opportunity for
an open and public comment period where comments on collections can be made. We are
grateful for this process and your comment. Please see our responses to your comments:
We agree with you that gathering information on how individuals obtain and weigh information
to make a decision on seeking a credential could be useful data for the federal government and
its public users of the data. We will take into consideration your suggestion to expand the
NTEWS to include information-seeking behavior for future cycles of NTEWS as long as such
questions stay within the scope of NTEWS’s goals and purpose. For the first cycle of NTEWS,
we are primarily focused on obtaining a baseline of data on individuals that have a work-related
credential and work in the skilled technical workforce. NCSES will seek to coordinate with other
federal statistical agencies, such as the National Center for Education Statistics (NCES) within
the Department of Education or the Bureau of Labor Statistics (BLS) within the Department of
Labor to see if such questions would be better suited in their longitudinal studies. We appreciate
that you recommended that there be more coordination on the credential terms and their
definitions.
As you know, the U.S. credentialing system is vast and complicated. We understand your
concern that misreporting may occur due to misunderstanding or misinterpretation of credential
terms. We will work within the federal government as well as stakeholders such as the Credential
NATIONAL SCIENCE FOUNDATION 
NATIONAL CENTER FOR SCIENCE AND ENGINEERING STATISTICS 
2415 Eisenhower Avenue 
Alexandria, Virginia 22314 

2 

Engine to ensure that we aim for accurate reporting. We believe that this goal is a process and
that it will take some time to calibrate our measurements. We welcome that Credential Engine is
willing to share CTDL schema and discussions to improve the interpretation of findings.
Finally, as federal statistical agencies are looking for other data sources to supplement survey
data, we appreciate you sharing Credential Engine’s Registry of open data on all credential
offerings. We will explore it as a resource and follow up on your offer to discuss any questions
that will help the NCSES better measure and understand the skilled technical workforce.
Thank you again for your comments and support for the new NTEWS data collection. As a new
survey, NCSES believes that NTEWS benefits from feedback from stakeholders, researchers,
and policy makers. If you would like clarification on any of these responses or would like to ask
additional questions about NTEWS, please do not hesitate to contact me at your convenience. I
can be reached by email at [email protected] or by phone at (703) 292-2258.
Sincerely,
John Finamore
Chief Statistician
National Center for Science and Engineering Statistics

NATIONAL SCIENCE FOUNDATION 
NATIONAL CENTER FOR SCIENCE AND ENGINEERING STATISTICS 
2415 Eisenhower Avenue 
Alexandria, Virginia 22314 

1 of 1

https://www.fdms.gov/fdms/getcontent?objectId=09000064846f1960&f...

As of: 6/22/20 1:28 PM
Received: June 15, 2020
Status: Pending_Post
Tracking No. 1k4-9h9y-pzu6
Comments Due: June 15, 2020
Submission Type: API
Docket: NSF_FRDOC_0001
Recently Posted NSF Rules and Notices.
Comment On: NSF_FRDOC_0001-2485
Agency Information Collection Activities; Proposals, Submissions, and Approvals
Document: NSF_FRDOC_0001-DRAFT-0424
Comment on FR Doc # 2020-08067

Name: Wendy Sedlak
Address:
30 S Meridian Street Suite 700
Indianapolis, IN, 46204
Email: [email protected]
Organization: Lumina Foundation

Would it be possible to add a question about industry-embedded certifications. These are programs that
allow students to obtain marketable industry and educational credentials simultaneously. Such
programs are often implemented in partnership with the relevant certification body, which may provide
industry-validated skill standards and curricula that are aligned with the certification requirements. We
would be interested in understating where these certifications are embedded (AD, for-credit certificate,
BA, non-credit CTE, etc.) as well as who is responsible for covering the cost for the industry
certification exam (employer, institution, student, etc.)

6/22/2020, 1:29 PM

1 

(FRN COMMENT #2)
September 24, 2021
Wendy Sedlak
Lumina Foundation
30 S. Meridian Street Suite 700
Indianapolis, IN 46204
[email protected]
RE: NCSES Responses to 60-day Public Comment Period
Dear Wendy,
Thank you for responding to the National Center for Science and Engineering Statistics
(NCSES) within the National Science Foundation call for comments published in the Federal
Register on April 16, 2020 (FR Doc. 2020-08067), which announced our plan to ask the Office
of Management and Budget for authority to conduct the National Training, Education, and
Workforce Survey (NTEWS). The Paperwork Reduction Act (PRA) provides an opportunity for
an open and public comment period where comments on collections can be made. We are
grateful for this process and your comment. Please see our response to your specific comment:
Adding questions that investigate the prevalence and financial support of industry-embedded
credentials would be appropriate for the NTEWS. Any new survey content must be thoroughly
researched to develop an adequate measure of the concept. Part of that research includes
cognitively testing the new survey content with a sample before going out to the field. There has
been extensive cognitive testing on the NTEWS survey content, and to date, there has been no
participant that has mentioned the term “industry embedded certification.” NCSES would be
interested in knowing more on how it is defined and recognized by the industry, and how it is
different from other certifications because it tends to focus on occupational skills. Would a
respondent be able to make this distinction in a survey? As such, NCSES needs time to test new
items to measure concepts that can be operationalized in a study on adults. This process usually
takes half a year or more before NCSES has a survey question. As the initial NTEWS cycle is
primarily concerned with obtaining baseline credential information and identifying those in the
skilled technical workforce, this NTEWS cycle is unable to accommodate new survey topics.
However, we will take into consideration your suggestion when we begin testing for the next
NTEWS cycle.

NATIONAL SCIENCE FOUNDATION 
NATIONAL CENTER FOR SCIENCE AND ENGINEERING STATISTICS 
2415 Eisenhower Avenue 
Alexandria, Virginia 22314 

2 

Thank you again for your comments. As a new survey, NCSES believes that NTEWS benefits
from feedback from stakeholders, researchers, and policy makers. If you would like clarification
on any of these responses or would like to ask additional questions about NTEWS, please do
not hesitate to contact me at your convenience. I can be reached by email at [email protected]
or by phone at (703) 292-2258.
Sincerely,
John Finamore
Chief Statistician
National Center for Science and Engineering Statistics

NATIONAL SCIENCE FOUNDATION 
NATIONAL CENTER FOR SCIENCE AND ENGINEERING STATISTICS 
2415 Eisenhower Avenue 
Alexandria, Virginia 22314 

Suzanne H. Plimpton, Reports Clearance Officer
National Science Foundation
2415 Eisenhower Avenue, Suite W18200
Alexandria, Virginia 22314
June 15, 2020
Dear Ms. Plimpton and Colleagues:
The undersigned members of the Non-Degree Credentials Research Network (NCRN), a project of the
George Washington Institute of Public Policy (GWIPP) at George Washington University, wish to convey
our strong endorsement of the planned information collection request (ICR) for the National Education,
Training, and Workforce Survey (NTEWS), OMB control number 3145-NEW, proposed by the National
Center for Science and Engineering Statistics (NCSES) of the National Science Foundation (per 85 FR
21271).
We believe that the NTEWS promises to be an essential source of data on phenomena of serious
interest to our research community, including the prevalence of credentials not captured in federal data
sources related to college degrees, inequality in who attains such credentials, and the relationship
between credential attainment and labor market outcomes such as salary and job satisfaction. Its
longitudinal nature will also permit us to track whether attainment is increasing or declining over time,
which will help policymakers understand the changing value of non-degree credentials in the overall
labor market.
In addition, we think that the NTEWS also represents a promising transition on the part of the National
Science Foundation away from focusing on the population of individuals who hold a baccalaureate
degree in favor of recognizing the contributions to the American scientific and engineering enterprise
made by skilled workers without a bachelor’s degree. By doing so, NCSES is collecting data that will be
needed to plan and evaluate potential new public policies that promise to increase the attainment of
sub-baccalaureate credentials, including the Department of Labor’s efforts to introduce IndustryRecognized Apprenticeship Programs and the expansion of Pell grants for short-term training programs.
With this in mind, we encourage NCSES to continue to explore opportunities to improve our
understanding of who is earning non-degree credentials and characteristics of those credentials. In
particular, we encourage NCSES to:
•

•

consider ways to add further depth to items about the attainment of credentials and other
forms of training, including the extent of costs associated with earning credentials (be they outof-pocket or covered by employers);
expand the survey as much as possible to include items from the Adult Training and Education
Survey not reflected in the current draft ICR (which might include items that ask individuals to
rate the perceived value of their credentials) and the National Survey of College Graduates (for
example, about individuals’ job responsibilities and dimensions of job satisfaction); and

•

investigate the compatibility of language used in survey items, especially concerning the
definitions of different types of credentials, with those used by its international counterparts
such as Eurostat so that the NTEWS can facilitate cross-national comparative research.

We commend NCSES for efforts that it has made already to engage in outreach with our research
community and encourage continued collaboration with the many scholars and researchers who hope
to use NTEWS data. To that end, we encourage NCSES to consider how it may engage in targeted
outreach and training in the use of NTEWS public-use datasets. We suggest that the online data user
community, e-mail list, conferences, and series of in-person trainings funded by the US Department of
Education for researchers interested in the International Program for the Assessment of Adult
Competencies (PIAAC) may be a model for this sort of outreach. We would be pleased to consult with
NCSES about how it could engage in such an outreach strategy.
Thank you for the opportunity to comment on the draft NTEWS ICR. Again, we strongly support NCSES’
development of the NTEWS and hope that NCSES finds our suggestions to be of interest and value.
On behalf of the undersigned members of the Non-Degree Credentials Research Network,
Kyle Albert
GW Institute of Public Policy, George Washington University
Stephen Crawford
GW Institute of Public Policy, George Washington University
Andrew Reamer
GW Institute of Public Policy, George Washington University
Tingting Zhang
Merrimack College
Jeff Strohl
Georgetown University
David Bills
University of Iowa
Deborah M. Seymour
Higher Education Innovation Consulting, LLC

Also co-signed by the following NCRN organizational participants:
Credential Engine
New America Education Policy Program
Education and Employment Research Center, Rutgers University
Concentric Sky, Inc.
Center for the Future of Higher Education & Talent Strategy, Northeastern University
Association for Career and Technical Education
Workcred, an affiliate of the American National Standards Institute

1 

(FRN COMMENT #3)
September 24, 2021
Non-Degree Credentials Research Network (NCRN)
A project of the George Washington Institute of Public Policy (GWIPP)
George Washington University
Dear NCRN,
Thank you for responding to the National Center for Science and Engineering Statistics
(NCSES) within the National Science Foundation call for comments published in the Federal
Register on April 16, 2020 (FR Doc. 2020-08067), which announced our plan to ask the Office
of Management and Budget for authority to conduct the National Training, Education, and
Workforce Survey (NTEWS). The Paperwork Reduction Act (PRA) provides an opportunity for
an open and public comment period where comments on collections can be made. We are
grateful for this process and your comment. Please see our responses to your comments:
We appreciate that the recommendations are from a membership that researches the prevalence
and impact of U.S. non-degree credentials. As the first cycle of data collection, we have many
goals for the NTEWS. The two most important is to understand and measure the prevalence of
credentials and identify the skilled technical workforce – those working in occupations with
scientific and technical skills but do not require a bachelor’s degree. As such, please consider the
first cycle as a baseline of information. For NTEWS collections that follow, we will research
and test additional items that should address your comment for in-depth information on the
attainment of credentials and other ways of training. As NCSES considers the addition of new
items for future cycles, we also have to consider the optimal survey length that minimizes the
survey burden on the respondent, resulting in a high response rate.
You noted that the draft of the NTEWS that you based your comments was on missing a cost
question. We successfully tested a cost question during our cognitive research, and as a result of
that research, we added the question to understand the costs associated with earning credentials.

NATIONAL SCIENCE FOUNDATION 
NATIONAL CENTER FOR SCIENCE AND ENGINEERING STATISTICS 
2415 Eisenhower Avenue 
Alexandria, Virginia 22314 

2 

We will explore this topic more if information from studies of college costs (i.e., the National
Postsecondary Study on Student Aid) or administrative data would be better data sources on
understanding the costs of a credential. We believe that information on costs is an important but
complicated topic that requires continual investigation.
For the first cycle of the NTEWS, which is a voluntary survey, we have included as much Adult
Training and Education Survey (ATES) and National Survey on College Graduates (NCSG)
items as possible. The ones that were in the draft you reviewed were deemed to be reliable
measures and performed well within the additional cognitive testing we conducted. For those
ATES and NSCG items that did not transfer over to the NTEWS, the primary reasons are either
we needed to ensure that the survey instrument was not burdensome, resulting in non-response,
or the item requires further testing. We have plans to continue to refine and testing omitted
ATES and NSCG items, and these may appear in future NTEWS cycles.
We agree with NCRN’s comment on establishing consistency of credential terminology used in
the NTEWS. Upholding a level of consistency will serve the purpose of accurate reporting by
respondents as well as improve data usability when compared with other data sources on
credentials, such as Eurostat. We will investigate the Eurostat framework to ensure we are
consistent wherever possible in future administrations; however, this goal will need to be
weighed against the goals of interpretability for an U.S. population and continuity over time
within NTEWS. NCSES believes that the first cycle will allow us to gain baseline measurements
on credentials and that future cycles will be refined through data users’ and policy makers’
feedback.
Finally, NCSES will investigate various outreach and training deliveries to data users. NCRN's
suggestion that NCSES look at the Department of Education's PIAAC outreach and training as a
program to model is a helpful start. NCSES is always looking for areas to improve stakeholders'
experience and understanding of NCSES data.
Thank you again for your comments and a strong endorsement for the new NTEWS data
collection. As a new survey, NCSES believes that NTEWS benefits from feedback from
stakeholders, researchers, and policy makers. If you would like clarification on any of these
responses or would like to ask additional questions about NTEWS, please do not hesitate to
contact me at your convenience. I can be reached by email at [email protected] or by phone at
(703) 292-2258.
Sincerely,
John Finamore
Chief Statistician
National Center for Science and Engineering Statistics
NATIONAL SCIENCE FOUNDATION 
NATIONAL CENTER FOR SCIENCE AND ENGINEERING STATISTICS 
2415 Eisenhower Avenue 
Alexandria, Virginia 22314 

Suzanne H. Plimpton, Reports Clearance Officer
National Science Foundation
2415 Eisenhower Avenue, Suite W18200
Alexandria, Virginia 22314
June 15, 2020
Comment on the National Training, Education, and Workforce Survey (OMB control no. 3145-NEW)
Dear Ms. Plimpton and Colleagues:
First of all, I want to emphasize that I strongly recommend that OMB allow the National Science
Foundation to proceed with this information collection request. Even in its current form, the data it will
generate will be of incredible value to the research community and provide insights necessary for
policymakers to understand the extent and value of all types of credentials in the American workforce.
The below suggestions are intended to help the National Center for Science and Engineering Statistics
improve an incredibly valuable ICR as it prepares to finalize the survey instrument.
In many respects, the research community is looking to the NTEWS to carry on the work of the
Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA). Among
the many accomplishments of GEMEnA was the creation and adoption of a common language across
federal survey research programs for describing non-degree credentials (often referred to as
“alternative credentials” in GEMEnA documents). I believe that it is vital for the NTEWS to carry forward
as many of the indicators of credential attainment, quality and value that were adopted for the GEMEnA
surveys as possible going forward to ensure that researchers will be able to identify longitudinal trends
in the attainment and value of credentials. Therefore, the most urgent point that I would like to raise is
that the utility of the NTEWS could be maximized for the research community by importing more
survey items from the Department of Education’s Adult Training and Education Survey (ATES) and
NCSES’s National Survey of College Graduates (NSCG).
Specifically, I encourage NCSES to explore the feasibility of adding the following items from the ATES and
NSCG to the draft NTEWS survey instrument:
•

•

•

ATES Item 15, parts A through D, Item 24, parts A through D, and Item 38, A through C. These
items, which ask respondents to identify the perceived labor market value of their two most
important certifications or licenses, provide essential measures of the utility of non-degree
credentials. Including these items on the NTEWS would give researchers essential data points
about whether the value of non-degree credentials changed for workers between 2016 and
2021, which may also help us to understand the relative value of credentials in a strong labor
market relative to times of economic distress should the United States remain in a recession
when the NTEWS is in the field.
ATES Item 51, regarding labor union membership. Understanding the relationship between
union membership and credential attainment in the skilled technical workforce could be
extremely helpful to the Department of Labor as it explores options for upskilling the noncollege workforce.
NSCG (2019) Item A46, regarding the reasons that one pursued a particular certification or
license. This should be repeated for each certification and license reported on the NTEWS.
Asking this will provide useful insights regarding any differences in the motivations of the sub-

•

•

baccalaureate workforce for attaining non-degree credentials relative to peers who hold a
college degree.
NSCG (2019) Item A48, regarding whether an employer contributed to the cost of credential
attainment. This should be asked for each certification and license reported on the NTEWS, and
ideally would be modified to ask respondents to indicate the percentage of direct costs that
were covered by their employer.
NSCG (2019) Section C, all questions. These items extend beyond the attainment of non-degree
credentials, but are vital for understanding the work-related context in which individuals choose
(or do not choose) to pursue credentialing opportunities. While the receipt of employersponsored, non-credentialed training is an obvious alternative to earning a certification or
license in some situations, it is also important to know how attendance at meetings and
association membership is related to certification attainment.
o I will also note that knowing more about association membership and meeting
attendance in the sub-baccalaureate workforce is important for understanding patterns
of occupational engagement and professional development within the skilled technical
workforce. Having estimates from the sub-baccalaureate portion of the NTEWS sample
to compare to the NSCG would provide data points on trends in association membership
and meeting attendance that would be very valuable to professional societies in STEM
and allied disciplines.

I also encourage the NCSES to reevaluate the response choices provided for Item 47, “Do you use this
certification for your MAIN job?” Asking respondents to choose yes or no may not give researchers a full
and accurate picture of the level of variation that exists in the use of non-degree credentials on-the job.
I suggest offering intermediate choices, which might include allowing individuals to describe how often
they use knowledge from a credential: for example, responses could include “Every day,” “At least once
a week, but not every day,” “Less than once a month,” and “Never.”
I also suggest asking whether each credential reported was obtained through distance learning or a
hybrid online/in-person format. Collecting additional data points about the prevalence of distance
learning within the skilled technical workforce would add to the evidence base used to justify the
expansion of such programs, especially for workers from socioeconomically disadvantaged backgrounds.
Moreover, researchers would benefit from the addition of items about the extent to which all
credentials respondents are asked to list on the NTEWS – degree or non-degree – were supported by
employer tuition aid programs. Realistically, this could be asked as two part question – the first part
asking if the respondent had access to tuition assistance benefits at work, and the second part asking if
the respondent has used employer tuition assistance for any type of credential since starting at their
current employer.
Finally, I also want to endorse the points made in the “consensus” letter signed by myself and members
of the Non-Degree Credentials Research Network. Specifically, I want to encourage NCSES to plan for
more aggressive dissemination and outreach activities with the research community than it has done in
the past with its flagship surveys, such as the NSCG. NCSES also should be mindful of opportunities to
gauge the competiveness of the US skilled technical workforce relative to the workforces of other
countries in its survey designs, which may include consulting with foreign statistical agencies to work
toward common definitions of non-degree attainment. The 2022 Eurostat Adult Education Survey is
probably among the most important surveys to review in this regard, though NCSES may wish to
commission research examining how other countries are accommodating the rise of non-degree
attainment in their programs of survey research.

To conclude, I would like to reiterate that the above recommendations – perhaps more of a wish list
from a research perspective - are intended to stimulate efforts to improve the quality of the NTEWS
survey instrument in the limited time remaining before the survey goes into the field. Regardless of
NCSES’s ability to implement these recommendations, I strongly recommend the approval of this
information collection request.
Best regards,
Kyle W. Albert

1 

(FRN COMMENT #4)
September 24, 2021
Dear Kyle Albert,
Thank you for responding to the National Center for Science and Engineering Statistics
(NCSES) within the National Science Foundation call for comments published in the Federal
Register on April 16, 2020 (FR Doc. 2020-08067), which announced our plan to ask the Office
of Management and Budget for authority to conduct the National Training, Education, and
Workforce Survey (NTEWS). The Paperwork Reduction Act (PRA) provides an opportunity for
an open and public comment period where comments on collections can be made. We are
grateful for this process and your comment.
We appreciate your feedback on the value of the Interagency Working Group on Expanded
Measures of Enrollment and Attainment (GEMEnA) work to federal statistical agencies seeking
to describe and measure non-degree credentials better. We hear the overall theme of your
comments that, as much as possible, NCSES and NCES should optimally aim to incorporate
more survey items that target non-degree credentials from NCES’ Adult Training and Education
Survey (ATES) and NCSES’s National Survey of College Graduates (NSCG). For the first cycle
of the NTEWS, both cosponsoring agencies believe that we have accomplished this objective
for 2021 NTEWS. Survey items from either the ATES or NSCG that do not appear in the
NTEWS have not measured the intended concept well, or do not address either agency’s
research questions. The research questions that both agencies are trying to answer with NTEWS
data collection are:

1

NCSES Research Questions
What are the demographic characteristics of the
STW?

2

What is the prevalence and role of education and/or
credentials for the STW?

3

Do education and/or credentials influence an
individual’s ability to enter, maintain relevance, and
seek advancement in the STW?

NCES Research Questions
How many adults, and which adults, have an
education and/or work credential, or some
combination of credentials?
How do work and education credentials interact
(e.g., to what extent do adults at each educational
attainment level have a work credential)? To what
extent do work credentials supplement versus
supplant higher education?
To what extent is the formal education system
involved in providing training for work credentials,
and what level of education is involved?

NATIONAL SCIENCE FOUNDATION 
NATIONAL CENTER FOR SCIENCE AND ENGINEERING STATISTICS 
2415 Eisenhower Avenue 
Alexandria, Virginia 22314 

2 
4

What are the employment characteristics of the
STW?

5

What are the employment outcomes of the STW?

6

How is education and/or credential attainment
related to employment outcomes?

7

[Intentionally Left Blank]

How is the attainment of a work credential related to
key employment outcomes (e.g., unemployment,
under-employment, earnings, job satisfaction?
How many adults, and which adults, are in the
pipeline to earn a license or certification for the first
time?
What role does credential attainment play in adults’
entry, advancement, and mobility in the labor
market?
To what extent do adults work in fields related to
their credential?

Also, we judiciously have to weigh the burden of the survey on the respondents against
researchers’ desire for extensive information. With that caveat in mind, as the NTEWS collection
moves forward, we will consider your specific suggestions for additional questionnaire items
from the ATES and NSCG. Please see our responses to your specific comments:
1. We excluded the ATES 15 (certification or license), ATES 24 (second certification or
license), and ATES 38 (certificate) questions because these questions were too long and
increased the burden on the respondent.
2. In the 6/2/2020 version of the survey you reviewed, we decided not to include the ATES
51 (union member) question for the initial NTEWS survey cycle for the reasons of
shortening the survey length, and the question ranked in lower when compared to other
survey items that answered the agencies’ research questions. However, we will consider
adding it back as this question tested well in previous cognitive interviews.
3. The 6/2/2020 version of the survey that you reviewed did not include NSCG A46, but we
conducted cognitive testing to develop a survey item similar to A46. Based on our
cognitive research, we have added onto the NTEWS a question for each credential
addressing why they obtained that credential.
4. We believe that information on costs is an important but complicated topic that requires
more investigation. The 6/2/2020 version of the survey that you reviewed did not include
a cost question on the extent of the costs associated with earning a credentials (i.e.,
understanding the employer’s contribution to finance the respondent’s training, education,
and credential fee). Since then, we have cognitively tested a new cost question to
understand the various financial sources that help individuals finance various credentials.
We added this question for each credential type to the initial NTEWS survey cycle.

NATIONAL SCIENCE FOUNDATION 
NATIONAL CENTER FOR SCIENCE AND ENGINEERING STATISTICS 
2415 Eisenhower Avenue 
Alexandria, Virginia 22314 

3 

5. Adding questions similar to NSCG Part C – Other Work-Related Experiences is on hold
until NCSES and NCES have established a baseline survey of data that measures
credentials of more immediate national attention, which are licenses, certifications,
certifications, and work experience programs. We understand that there is a vast network
of credentials offered by a diverse array of providers. It is essential for our agencies to first
establish a baseline level of information before exploring other work-related experiences.
6. We see your point about understanding the association membership of those with have a
sub-baccalaureate education. Associations can help individuals’ network and support
members’ professional goals. However, we think that this question should be considered
for future cycles of the NTEWS, while minimizing burden on respondents.
7. Question 47 certification (along with Q34 license and Q68 certificate), do you use this
credential for your MAIN job is a placeholder for the NSCG A46 question was missing
from the draft because we had plans to cognitively test it. From our cognitive testings, we
developed a variety of response options of reasons for obtaining the work-related
credential instead of a yes or no response. This new question was added to the initial
NTEWS cycle.
8. We will consider developing a survey item for future NTEWS survey cycles that
investigates if the credential’s training or education was provided through distance
learning for future NTEWS cycles. Increasing offering and enrollment of distance learning
in its various forms has garnered the attention of policy makers and researchers. The
impact of the COVID-19 pandemic has only driven more individuals to learn and obtain a
credential via distance education methods.
9. Thank you for the suggestion to investigate the employer benefit of tuition-aided programs
as well as offering a possible solution to measure this benefit in two parts: 1) if the benefit
is provided to the employee and 2) if the employee uses the benefit for their work-related
credential(s). The initial cycle NTEWS has a general question on benefits, which is in the
Current Employment section. We agree that investigating employer support is important
and will consider the best options for assessing this concept as part of future cognitive
testing.
10. We have received several comments that also support your suggestion to look into
Eurostat’s Adult Education survey. Thank you for your reference to a specific Eurostat
survey to examine. We will review that instrument as we prepare for future NTEWS
survey cycles. We are excited that there could be potential international comparisons.
11. 11. We appreciate your suggestion for NCSES to be more aggressive in its data
dissemination and outreach activities. NCSES is always exploring for areas to improve
stakeholders’ experience and understanding of NCSES data.

NATIONAL SCIENCE FOUNDATION 
NATIONAL CENTER FOR SCIENCE AND ENGINEERING STATISTICS 
2415 Eisenhower Avenue 
Alexandria, Virginia 22314 

4 

Thank you again for your comments to help improve NTEWS data and its usefulness to the
research community. We also appreciate your strong recommendation for the NTEWS data
collection. As a new survey, NCSES believes that NTEWS benefits from feedback from
stakeholders, researchers, and policy makers. If you would like clarification on any of these
responses or would like to ask additional questions about NTEWS, please do not hesitate to
contact me at your convenience. I can be reached by email at [email protected] or by phone
at (703) 292-2258.
Sincerely,
John Finamore
Chief Statistician
National Center for Science and Engineering Statistics

NATIONAL SCIENCE FOUNDATION 
NATIONAL CENTER FOR SCIENCE AND ENGINEERING STATISTICS 
2415 Eisenhower Avenue 
Alexandria, Virginia 22314 

Suzanne H. Plimpton, Reports Clearance Officer
National Science Foundation
2415 Eisenhower Avenue, Suite W18200
Alexandria, Virginia 22314
June 15, 2020
Dear Ms. Plimpton and Colleagues:
Workcred, a non-profit affiliate of the American National Standards Institute, with a mission of to
strengthen workforce quality by improving the credentialing system, ensuring its ongoing relevance, and
preparing employers, workers, educators, and governments to use it effectively, would like to provide
the following comments of the planned information collection request (ICR) for the National Education,
Training, and Workforce Survey (NTEWS), OMB control number 3145-NEW, proposed by the National
Center for Science and Engineering Statistics (NCSES) of the National Science Foundation (per 85 FR
21271).
We believe that the NTEWS promises to be an essential source of data on phenomena of interest to
Workcred, including the prevalence of credentials not captured in federal data sources related to college
degrees, inequality in who attains such credentials, and the relationship between credential attainment
and labor market outcomes such as salary and job satisfaction. Its longitudinal nature will also permit us
to track whether attainment is increasing or declining over time, which will help policymakers
understand the changing value of non-degree credentials in the overall labor market.
In addition, we think that the NTEWS also represents a promising transition on the part of the National
Science Foundation away from focusing on the population of individuals who hold a baccalaureate
degree in favor of recognizing the contributions to the American scientific and engineering enterprise
made by skilled workers without a bachelor’s degree. By doing so, NCSES is collecting data that will be
needed to plan and evaluate potential new public policies that promise to increase the attainment of
sub-baccalaureate credentials, including the Department of Labor’s efforts to introduce IndustryRecognized Apprenticeship Programs and the expansion of Pell grants for short-term training programs.
With this in mind, we offer comments to NCSES to clarify the questions and terminology used in the
survey. In particular, we encourage NCSES to consider the following comments on specific survey
questions:
Question #13
The question states “…what are your most important duties or activities.” We suggest making the
language in the question consistent with the header located about the answer box. Either use “activities

and duties” or “activities or duties.” Also, NCES might consider adding a drop-down menu of example
activities and/or duties to help the survey respondent appropriately complete the survey.
Question #14
NCES might consider adding “social and cultural working environment” as an additional answer choice.
For many people, it is the social and cultural working environment that cause dissatisfaction with their
job.
Question #18
NCES might consider updating the examples used in question #18 to make them more aligned with what
the question is asking. Examples of business or industry could include: Education, IT services, Residential
Construction.
Question #20
This question is the same as question #18. We suggest that it be deleted.
Question #22
Workcred suggests that the answers grouped together in the “other category” be separated. The revised
answers could be: “Manufacturing, Wholesale Trade, Retail Trade, Agriculture, Construction, Service,
Government, and Other.” The “other” category is too broad and the answer will not provide useful
information.
Question #23
Workcred suggests that NCES revise question #23 to the following: “Do you have a CURRENTLY ACTIVE
local, state, or federal license?” Most licenses are issued by local, state, and federal government entities.
Industries do not issue licenses. NCES might also consider adding driver’s licenses for personal vehicles
to the list of licenses that should not be included when answering this question.
Question #25
Workcred suggests that NCES consider eliminating this question. The reason that individuals obtain a
license is because it is required to practice a specific occupation. There is a cost associated with
obtaining and maintaining a license.
Question #29
Workcred suggests that NCES either eliminate this question or restructure it to focus on whether the
individual is using their license for their primary job or another job. As stated above, occupations that
have licenses mean that an individual cannot practice that occupation without obtaining the license.
There is a legal ramification to this type of credential.
Question #30
Question #30 asks about the entity that issues the license. Workcred suggest eliminating “professional
and trade organization” as one of the answers. Professional and trade organizations do not issue

licenses. The examples given, the American Culinary Federation and CompTIA issue certifications not
licenses. In addition, Workcred suggests eliminating “business or company” as an answer choice.
Businesses and companies do not issue licenses for people. Instead, they may issue licensing
agreements for products and services.
Question #34
This question is similar to question #29. NCES should consider using either question #29 or #34 to ask
whether a license is used for the respondent’s main job.
Question #43
For answer #2 “business or company,” Workcred suggests adding examples of companies that issue
personnel certifications such as Microsoft, Cisco, or CrossFit)
Question #48
Is the purpose of this question to only address learning that occurred in schools? If not, there are other
ways for individuals to obtain the skills and knowledge that prepare an individual to sit for a
certification. These include apprenticeships, boot camps, professional societies, trade associations, and
study materials provided by certification bodies. NCES might consider broadening the opportunities
where individuals can learn the knowledge and skills that prepare them to sit for a certification.
Question #57
Workcred suggests that NCES consider removing the word “formal” from the question. It is unclear what
is meant by “formal training from a coworker or mentor.”
Question #63
Vocational certificates are offered by a variety of entities outside of the formal education system. For
example, vocational certificates may be offered by professional or industry associations. Workcred
suggests that the options outlined in the question be expanded to include professional and industry
associations.
Question #87
There are several other answer options that may be added to better understand the career paths of
individuals with different types of limitations. Workcred suggests adding depth perception, sensory
deficits, and dexterity, since these capabilities are considerations in occupational choice. Suggested
language includes:
• FINE DEXTERITY to perform tasks that require coordination of small muscle movements to
accomplish detailed tasks
• DEPTH PERCEPTION to be able to judge spatial relations of objects and move one’s body in the
work environment
• SENSORY DEFICITS to feel or touch objects and manipulate them without the use of vision and
to prevent bodily injury

Thank you for the opportunity to comment on the draft NTEWS ICR. Again, we strongly support NCSES’
development of NTEWS and hope that NCSES finds our suggestions of interest and value.
Best regards,

Roy Swift, Executive Director
Workcred

1 

(FRN COMMENT #5)
September 24, 2021
Roy Swift
Executive Director
Workcred, a non-profit affiliate of the American National Standards Institute 
Dear Roy,
Thank you for responding to the National Center for Science and Engineering Statistics
(NCSES) within the National Science Foundation call for comments published in the Federal
Register on April 16, 2020 (FR Doc. 2020-08067), which announced our plan to ask the Office
of Management and Budget for authority to conduct the National Training, Education, and
Workforce Survey (NTEWS). The Paperwork Reduction Act (PRA) provides an opportunity for
an open and public comment period where comments on collections can be made. We are
grateful for this process and your comment. Please see our responses to your comments for the
following survey questions:
Question 13. We will make the recommended change to “activities or duties” for consistency as
well as align with what is used in the American Community Survey (ACS). A drop-down menu
would be applicable only for the online version of NTEWS. We will consult with our expert
survey methodologists and web developers if a drop-down list of examples should be used,
which will be based on best practices for online federal surveys.
Question 14. We will consider this possible job characteristic (social and cultural working
environment) for future NTEWS cycles. Considering the current pandemic, we are uncertain
how this measure would perform, and we would need to test it.
Question 18. We will consider testing these examples to describe the MAIN BUSINESS OR
INDUSTRY for future cycles of NTEWS. This exact language comes from the ACS. NCSES is
interested in keeping it intact to remain consistent with the ACS, which serves as the sampling
frame for NTEWS. Also, in the summer of 2018, we tested this survey item, and it performs well
with these examples.
Question 20. You are correct that this question is the same as question 18. However, the draft
version was incorrect. Question 18 is supposed to be for self-employed respondents, and the stem
of the question is, “What is your MAIN BUSINESS OR INDUSTRY…”. Our apologies for the
confusion.
NATIONAL SCIENCE FOUNDATION 
NATIONAL CENTER FOR SCIENCE AND ENGINEERING STATISTICS 
2415 Eisenhower Avenue 
Alexandria, Virginia 22314 

2 

Question 22. This question is the exact question used in the ACS. This question will help with
back-end coding and validating ACS responses—specifically, to help clarify a respondent’s
answer. For example, if a respondent states that her business or industry is “shoes,” this question
helps to determine if that response refers to manufacturing shoes, the wholesale trade of shoes, or
the retail trade of shoes. We appreciate your suggestion to improve the question, but we decided
to use the question as used on the ACS.
Question 23. We based this question on ATES question 6. We have revised this question stem
wording slightly and will evaluate its measurement quality in the 2021 NTEWS. We had many
conversations about the possible misinterpretation of respondents considering personal driver’s
license and marriage license a work-related license. We will consider investigating what
“industry licensing” means to the respondent as well as your suggested wording for future testing
and cycles should the current wording produce poor data.
Question 25. We appreciate your insights on this recommendation, and we have cognitively
tested this question, which performed well. We made a slight wording adjustment, but the
question remains intact for the initial NTEWS cycle because we need the individual to confirm
that they obtained the credential for work-related reasons.
Question 29. We have revised this question to a yes or no response instead of using the three
response options that you reviewed in the 6/2/2020 draft. We agree that there are legal
consequences if one does not hold a license to perform the work. Question 34 serves to
understand if the license is used for the MAIN job.
Question 30. This question is a variation of the NSCG question A45, noting that the only
difference is NTEWS expanded the levels of government. We will evaluate if either or both
“professional or trade association” or “business or company” should be deleted from the first
cycle or future cycles.
Question 34. This question will be revised to have a variety of response options that describes
the reasons for holding the work-related credential. The survey item you reviewed in the
provided 6/2/2020 draft was not intended to be final but a placeholder for a version that will
provide more helpful information. After cognitive testing, we have finalized the response options
that performed well with individuals.
Question 43. We had deliberatively reduced the examples for space and survey exhaustion
reasons. This question is a slight variation from NSCG question A45, and as a survey item that
has been in use for several collection cycles, we will use the same survey item for NTEWS.
However, we will consider your suggestion for future collections of NTEWS.
Question 48. Yes, the purpose is to understand if the skills and knowledge were obtained
through schools.  
Question 57. The word “formal” in “formal training from a coworker or mentor” while in a
work experience program was intentionally included to distinguish from informal or unstructured
occurrences of mentoring. This question had tested well throughout our cognitive testing. We
prefer to keep the word in the stem of the question and let the respondent determine if applicable.
Question 63. We designed this question, in combination with Question 64, to identify the
respondents who have a certificate from a postsecondary institution.
NATIONAL SCIENCE FOUNDATION 
NATIONAL CENTER FOR SCIENCE AND ENGINEERING STATISTICS 
2415 Eisenhower Avenue 
Alexandria, Virginia 22314 

3 

Question 87. To maintain comparability, this survey question, which is used in other NCSES
surveys, cannot be changed. NCSES uses this survey question to produce trend and snapshot
information for the NCSES congressional report Women, Minorities, and People with
Disabilities.
Thank you again for your comments and your strong support for the new NTEWS data
collection. As a new survey, NCSES believes that NTEWS benefits from feedback from
stakeholders, researchers, and policy makers. If you would like clarification on any of these
responses or would like to ask additional questions about NTEWS, please do not hesitate to
contact me at your convenience. I can be reached by email at [email protected] or by phone
at (703) 292-2258.
Sincerely,
John Finamore
Chief Statistician
National Center for Science and Engineering Statistics

NATIONAL SCIENCE FOUNDATION 
NATIONAL CENTER FOR SCIENCE AND ENGINEERING STATISTICS 
2415 Eisenhower Avenue 
Alexandria, Virginia 22314