NTEWS Questionnaire Development through Cognitive Research

This appendix includes details on the NTEWS questionnaire development efforts conducted by NCSES and NCES. Both statistical agencies carried out two sets of cognitive research. The final reports from the American Institutions for Research’s (AIR) 2018 cognitive research and Mathematica’s 2021 cognitive research are provided in this appendix.

Background

AIR’s past cognitive research was conducted on the Adult Training and Education Survey (ATES), which was administered by the National Center for Education Statistics (NCES) as part of its National Household Education Surveys (NHES) program. This cognitive research was conducted for the originally planned 2019 and 2021 administrations of the ATES. However, when the ATES was terminated before its 2019 administration, the National Center for Science and Engineering Statistics (NCSES) used the findings from the ATES cognitive research to directly inform the contact development for the 2022 NTEWS questionnaire. Thus, the NTEWS builds upon the research goals of ATES by collecting data on work-related credentials.

However, the NTEWS is a new survey because NCSES will provide more reliable skilled technical workforce (STW) estimates by allocating a larger sample of individuals in skilled technical occupations. According to OMB’s standards and guidelines for statistical surveys, federal agencies initiating a new survey or significant revisions of an existing survey must provide justifications, including its goals and objectives, intended uses, and the purpose and quality of the proposed data. Despite the similarities between the NTEWS and ATES survey content, NCSES felt the ATES cognitive testing was insufficient to meet NCSES data focus on the STW. In response, NCSES conducted additional cognitive testing in preparation for the 2022 NTEWS and its STW emphasis.

AIR Cognitive Interview Findings for the Adult Training and Education Survey (ATES) 2019: Final Report (March 2018)

(OMB# 1850-0803 v.195 & v.205)

Cognitive Interview Findings for the Adult Training and Education Survey (ATES) 2019: Final Report

Prepared for the National Center for Education Statistics (NCES)

March 2018

Samantha Lilly & Maria Payri

Introduction

The American Institutes for Research (AIR) is working with the National Center for Education Statistics (NCES) to develop a strategy for measuring the prevalence of non-degree credentials—in particular, industry-recognized certifications, licenses, and educational certificates—and work-related training and education in the United States. This multiphase research effort began in 2009 and includes the development and testing of questions that will be included in the Adult Training and Education Survey (ATES). The ATES has undergone extensive development work over the past nine years. To develop the ATES instrument, NCES has: conducted focus groups in 2013 on characteristics of educational certificates and participation in work-related training; conducted cognitive interviews in 2013 and 2014 to test English versions of the ATES instrument; administered a response rate pilot study in early 2013; administered a feasibility study in 2014 to test processes for incorporating ATES into the NHES program; and conducted a series of concept interviews and cognitive testing in 2015 to develop the 2016 ATES instrument. In January 2016, the National Household Education Survey (NHES) fielded the first full-scale administration of the ATES. After that administration, a series of focus groups (in Spanish only), concept interviews, cognitive testing in both English and Spanish, and usability interviews were conducted to revise the instrument for a 2017 pilot test of an NHES web administration. These efforts led to the development of new English and Spanish versions of the ATES instrument, which underwent cognitive interview testing. This report documents the key findings from the English and Spanish language cognitive interviews conducted in 2017 on the ATES to be used in a future administration (originally planned for 2019).

Methodology

Each cognitive interview lasted approximately one hour and was conducted using a “think-aloud” approach, with concurrent probing as needed, based on a structured protocol. For English cognitive interviews, there were three rounds of testing, with instrument revisions made between each round. Spanish cognitive interview were conducted in two rounds, with no revisions made to the instrument between rounds. The methodology was developed by a senior researcher at AIR in consultation with NCES and drew on best practices and methods from cognitive science (see Appendix A for an English version of the protocol used during the third and final round of testing, Appendix B for a Spanish version of the protocol used during the both rounds of testing, and Appendix C for the instrument used during the third round of English cognitive interviews). These interviews were designed to identify problems of ambiguity or misunderstanding in item wording. The interviews were conducted by interviewers who received training from a senior researcher at AIR in the cognitive interview technique and the interview protocol. Researchers from AIR and staff from NCES listened to a subset of interviews and provided guidance and feedback throughout the study.

During each interview, the study participant was welcomed by the interviewer and led to an interview room. The participant was then told that the purpose of the interview was to help answer questions about how people respond to items about adult education and training experiences. All participants were assured that their participation was voluntary and that their answers would be used only for research purposes and would not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C §9573). Interviewers explained the think-aloud process (see below) and modeled a think-aloud practice with a sample question. Participants were then given either an English or a Spanish paper copy of the questionnaire to fill out, respective to the language for which they were recruited.

The interviews were conducted using “concurrent probing,” a technique in which interviewers asked respondents to “think aloud” as they answer the survey items. The interviews also included a verbal probing component, conducted after completion of the think-aloud portion for a given section of the questionnaire. The verbal probes included a combination of pre-planned, item-specific questions identified before the session as important and ad hoc questions that the interviewer identified as important from observations during the interview, such as clarifications or expansions on points raised by the participant. Participants in the English ATES interviews were probed in English, and participants in the Spanish ATES interviews were probed in Spanish.

Sample

The English ATES cognitive interview participants were recruited in Washington, DC, Los Angeles and San Mateo, CA, and Jacksonville, FL. The Spanish ATES participants were recruited in Washington, DC, Los Angeles, CA, and Austin, TX. All interviews were conducted in person at these locations. Respondents received $40 for their participation.

Respondents were required to have at least one professional certification or license and either 1) currently working on another professional certification or license or 2) have at least one other professional certification or license. Those with multiple professional certifications or licenses either had more than one in the same state or more than one in different states. AIR aimed to recruit individuals with the same license in two different states (for example, a real estate license in Maryland and Virginia).

AIR conducted a total of 46 English cognitive interviews between May and September 2017, and a total of 25 Spanish interviews between September and December 2017. The interviews were conducted in multiple rounds to facilitate revisions and re-testing. The final survey version from the three English interview rounds was used for the first round of Spanish interviews.

Key Findings

Findings are presented by survey section, and changes made to the survey to address response issues are described below. Unless otherwise indicated, findings are from the English ATES testing.

Items on Education

The education section of the survey consisted of two items asking the respondents’ highest level of education and field of study for the highest level of education. Most respondents did not have any difficulty indicating the field of study for highest level of education. However, some respondents marked more than one field of study, noting that they either did not notice the instructions to mark one field only, or indicated that they had two highest levels of education in different fields and had difficulty considering one as most important.

Some Spanish respondents marked more than one option on both the highest level of education item and the field of study item. For level of education, these respondents thought they had to choose all the options that applied to their level of education (e.g., from high school diploma to master’s degree). For the field of study, some Spanish respondents interpreted this as all the classes they had taken while obtaining their degree. Additionally, some Spanish respondents noted challenging vocabulary such as “associate’s degree”, “GED”, which are not used in their country of origin. Due to the low occurrence of these issues, AIR did not recommend any changes to these options, however, we will continue to monitor the issue during the next series of Spanish interviews.

Due to extensive testing on these questions in previous cognitive interview efforts for the 2016 ATES administration, AIR did not recommend any changes to the items:

“Which ONE of the following best describes the field of study for the highest level of school you have completed? Mark [X] ONE only. If there was more than one, please choose the one you consider most important.”

Items on Certifications and Licenses

A primary area of interest during cognitive testing was exploring approaches to better distinguish certifications or licenses respondents possessed that were the same but from different states, to better standardize the number of professional certification or licenses reported by respondents.

The item that indicates how many professional certifications or licenses a respondent has was difficult to answer for some respondents. Instructions were revised throughout cognitive testing to determine the best approach. Respondents in the first and second rounds were asked how many currently active professional certifications or licenses they have, and instructed to count each license separately, as utilized in the 2016 ATES administration. Two respondents misreported, counting the licenses held in the same field or the same state as one. During these rounds, an additional item was included which asked respondents if their second/third most important license was the same as their most/second-most important license but in a different state. In both rounds most respondents encountered some difficulty and confusion when answering this question. Some respondents were confused as to how a certification or license could be the same in another state and some interpreted the question to mean a different field or different status level rather than geographical state. Certification holders, with national recognition of their credential, also struggled answering this question. A couple respondents with the same certification in multiple states did not report their second or third-most important certification, as they considered it as one credential, despite instructions to count each separately.

As such, instructions were revised in the third round to request that respondents count the same certification or license in multiple states as one. This round resulted in four misreports, in which these respondents counted each certification or license separately. In revising instructions to count each certification or license separately, an additional item was included which asked in how many states the respondents’ most/second-most/third-most important are currently active. Response options included all states, 1 state, or 2 or more states but not all states in an effort to provide a response option for certification holders, credential holders with licenses valid in one state, and credential holders with licenses valid in two or more states (i.e., the same certification in multiple states), respectively. However, respondents still encountered difficulty answering this item, with some continuing to count the same license in multiple states separately. Based on these interview findings, we determined that there is not enough evidence to support any changes to this item for future ATES administrations. Therefore, the item will continue to instruct respondents to count each certification and license separately using the following text:

If yes, how many CURRENTLY ACTIVE certifications and licenses do you have? If you had to get a certification in order to get a license, count each certification and license separately.

Spanish testing used the survey version noted above based on the findings from the three English rounds. Some respondents had issues reporting the number of active certifications or licenses because they were either thinking of their primary certification or they were unsure if one of the other certifications was still active. One respondent also misreported their master’s degree as a certification or license. When responding to whether the credential was valid in one or more states, some Spanish respondents were unsure of how to accurately determine this. Most respondents knew that their credentials were valid in one or more than one state, but did not know exactly which states accepted their credentials other than the ones where they usually worked.

It is worth noting that while talking about their credentials, some Spanish respondents were thinking of certifications or licenses obtained outside of the U.S. (usually from their country of origin) and were not valid in the U.S. While these respondents would be ineligible for participation in the cognitive interviews, it was not until after much probing that the interviewers realized the origin of their credential. AIR recommends in future administrations that instructions be added to note that respondents should only include professional certifications or licenses obtained in the U.S or that have undergone a validation process in order to be recognized in the U.S.

The item that serves as the primary indicator of the credential being a license rather than a certification was difficult for only a few English respondents who were not sure if a government entity issued their credential. For some, the parenthetical text caused respondents to believe that only credentials required by a state board warranted a “yes” response. However, despite some confusion, all but two respondents responded correctly based on their specific credential. As such, no further revisions were made to the following item:

Is your most important certification or license required by a federal, state, or local government agency (such as a state board) in order to do that kind of work?

Spanish respondents had more difficulty with this item. Some Spanish respondents interpreted the item as asking if the credential is required by their job in order to do the work, rather than specifically by the state. One respondent saw the university where the credential was required as an extension of the government. Another respondent interpreted the item to be asking if the credential was free, since anything obtained through the government should be free. It is worth noting that the one respondent who had obtained their credential through a professional board was able to distinguish that from a state board and marked “no”.

Spanish respondents also had difficulty with the item that asks if their credential is for their current job, as many of the participants recruited had different positions for which they did not use the reported credentials consistently. This also caused some confusion when reporting about respondents’ jobs, which will be explained in the employment section later in this report.

For these interviews, we merged two items in the 2016 ATES which asked respondents about the types of classes and certificates, internships, and/or exams completed in the process of getting a certification or license. This item allowed respondents to indicate various activities completed in the process of getting their certification or license, yet caused confusion for some respondents who were unsure whether to include only activities that were required. During interviews, some respondents were uncertain if activities completed in undergraduate and/or graduate schooling should be included. For clarification, we included the text “(undergraduate or graduate)” after response options that included classes taken from a college. This item was also included in the Preparing for New Certifications and Licenses section for respondents who indicate that they are getting an additional professional certification or licenses.

Spanish respondents had several issues with this item. Some confusion arose from respondents’ uncertainty as to whether the classes taken during high school or college related to their credential should be included, even if they were not part of getting the credential. Some respondents only marked “yes” for the activities that applied to them, leaving the ones that did not apply to them blank. Additionally, Spanish respondents had issues with the option that asks whether they had participated in an internship, practicum, clerkship, externship, or apprenticeship. These terms do not have a straightforward translation that is equivalent in all regional variation of Spanish, and often only individuals who have gone through the specific program are familiar with its meaning. AIR did not recommend making any changes to this item as there were not any misreports during cognitive testing.

Items on Preparing for New Certifications and Licenses

Two items during this testing ask respondents if they are currently working on renewing a professional certification or license and currently working on getting a new professional certification. While respondents understood the meaning of renewing a certification or license, some struggled with correctly interpreting the meaning of a “new” professional certification or license. To aid in understanding, a slight revision to the text was made from “new” to “additional”, as seen below.

Are you currently working on getting an ADDITIONAL professional certification or license?

Spanish respondents generally could distinguish between renewing and getting a new certification or license. However, some respondents, Spanish and English, misreported credentials they were thinking about getting but not currently in the process of getting. Upon probing, it was determined that respondents considered the planning (e.g. thinking about going back to school for business management) as part of getting a new credential. For future administrations, a potential modification to help minimize misreporting could be to provide additional instructions such as “Please mark ‘No’ if you are thinking of getting an additional certification or license but are not currently in the process of getting it yet”.

For consistency across the survey, we also included the merged item from the Certifications and Licenses section regarding activities completed during the process of getting a certification or license in the Preparing for New Certifications and Licenses section for respondents who indicated that they are working on getting an additional professional certification or license.

Items on Employment

The main employment item which asks respondents how many jobs they have proved difficult for some with more complex employment circumstances. Some respondents were unsure whether to include full-time, part-time, and/or consulting work. Additionally, those who were self-employed were uncertain whether this was considered employed for pay at a job or business. In order to provide clarity, we added “self-employed” to the instructions:

Are you currently employed for pay at a job or business? If you are self-employed, or if you are temporarily absent from a job or business (on vacation, temporarily ill, on maternity leave, etc.), answer “Yes.”

Some Spanish respondents who were self-employed had issues when answering this item. These respondents noted that their working hours were irregular and that they were not sure if their self-employment should be considered as being employed in this context. Despite the confusion, most of these respondents were able to correctly respond, indicating that the additional instructions were helpful.

These interviews tested a variation of response organization for the item which asks respondents to describe their current employment. The interviews tested restructured response options, which included headers to separate various types of employment (e.g., private sector, government, self-employed). However, the reorganization yielded inaccurate responses from some respondents who were distracted by the headers and/or confused by the length of the item and response options. As such, we determined the more condense item version present in the 2016 ATES is best for eliciting the most accurate responses describing respondent employment. However, this item was challenging for some Spanish respondents who had several jobs and could not decide which was more important. Therefore, some Spanish respondents marked more than one option. A potential suggestion to minimize this error is for future administrations, add text reiterating instructions to consider only their most important job for each follow-up question.

Additionally, some respondents noted being uncomfortable and unwilling providing the name of their employer within the survey. As such, we included a “prefer not to answer” option in addition to the write-in box for the item for those respondents who prefer not to disclose. This issue was also prevalent among some Spanish respondents, who selected “prefer not to answer”. One Spanish respondent did not notice this response option and reported the name of their employer while verbalizing that they would prefer not to disclose.

General Issues

We encountered two additional issues across the survey for the English version of ATES. The first being a negative reaction to the language “most important”, “second-most important”, and “third-most important” when referring to different certifications or licenses. Some respondents indicated that they felt each certification or license they held was equally important, and that choosing a most important certification or license seemed to devalue or undermine the other certifications or licenses. Spanish respondents did not have a negative reaction towards the “most important” language, but did encounter some difficult in determining a most important credential and/or job throughout the survey. Despite the concerns, previous testing confirmed this approach to be valid, and that most respondents can easily identify what they consider to be their most important certification or license.

The second issue was that some respondents expressed frustration with the redundancy of items for their most, second-most, and third-most important. In three instances, including two English interviews and one Spanish interview, these respondents misreported having additional certifications and licenses in order to skip answering the repetitive items. Based on previous testing and the small frequency of these occurrences, no revisions were made. However, it is important to note that the individuals who participate in the cognitive interviews are usually inclined to complete the survey in its entirety and to the best of their ability. This is likely due to the receiving of an incentive and presence of the interviewer, as this adds a level of social desirability.

Items on NCES Childcare

The Department of Health and Human Services (DHHS) requested that two questions about the child care workforce be added to ATES, as a cost-savings measure to avoid administration of a separate national survey of adults. These items were modified versions of items asked on the 2012 household survey component of the National Survey of Early Care and Education (NSECE). The items asked respondents if they spend 5 or more hours each week in their home looking after any children under age 13 who are not their own, and if they are paid to do so. The items were placed at the end of the survey and while many respondents noted that these items appeared out of place, both English and Spanish respondents agreed that there was not a more appropriate location for them within the survey. Some respondents shared that they watch family members (e.g., grandchildren, nieces, nephews), including one Spanish respondent, or babysit other children, but are not childcare providers or watch the children in another’s home. It was also mentioned that the term “paid” is vague, as one could be compensated in various ways such as monetarily, through services, or food and/or housing.

Spanish-specific Issues

As in previous iterations of the Spanish cognitive interviews, some respondents had issues with survey vocabulary. Two respondents had difficulty understanding “sin animo de lucro” (non-profit), specifically the word “lucro”. The team encountered this issue in one of the first interviews and searched alternatives, but determined that the current language is the most accurate and common translation for “non-profit”. Therefore, no changes were made. One respondent interpreted the word “estado” (state) to mean “country”. While most Spanish speakers living in the U.S. are familiar with the word “estado” to mean “state”, “country” is a more common meaning for this word in Spanish-speaking countries. Due to the infrequency of this issue, no further changes were suggested. Finally, one respondent noted that the word “licencia” (license) can be confused with “licenciatura” (equivalent to a bachelor’s degree in some Spanish-speaking countries). Given the isolated incident and that there is regional idiosyncrasy regarding the term “licenciatura”, AIR did not recommend any revision to this term.

Summary

AIR conducted cognitive interviews to test both English and Spanish versions of ATES questions for their potential to accurately capture individuals who hold multiple certifications or licenses, particularly those with the same licenses in multiple states, collect better information on individuals preparing for new certifications and licenses, and to test DHHS’ questions about the child care workforce. During the cognitive testing, the AIR team found that respondents had problems both reporting multiple certifications and licenses separately and reporting the same license in different states one, regardless of instructions. Despite attempts to address this problem by modifying instructions and/or including additional follow up items, there was not enough evidence to recommend any revisions from the 2016 ATES for future ATES administrations. In addition, we tested a new item which merged two previous items from the 2016 ATES. This approach was valid and allows for more condensed sections of the survey. Finally, the inclusion of DHHS childcare workforce questions were confusing to some respondents, but were not revised due to DHHS protocol.

Appendix A. Cognitive Interview Protocol (English)

Interview ID:		Date of Interview:
First Name:		Interviewer

Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is suggested content with which the interviewer should be thoroughly familiar in advance. Text in brackets [ ] presents instructions for the interviewer.

Hello, my name is <your name > and I work for American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today.

[Create small talk to build rapport with the respondent]

You are here today to help us with a survey that we are working on for the U.S. Department of Education. This survey asks about adults’ education and training experiences. I will ask you to complete the survey as if you were taking a real survey. There are no right or wrong answers, and it’s ok if you do not know some of the answers. As you are answering the questions I will ask you about your answers. This will help us make the survey better and clearer for other adults like you.

To make sure we learn everything we can from this session, I am going to be taking notes while you are answering the survey and while we are talking [INSERT IF THERE IS AN OBSERVER: and my colleague will also be observing our session today behind the glass/by video].

Also, this interview will be audio recorded so researchers can review the recording later. Any information that refers to you personally, like your name or employer, will not be included in our report, and your responses will be used for research purposes only. Is all of this OK?

Remember, you are not being tested; all we want is to learn how well the survey works.

If at any time you want to stop, just let me know.

[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.]

Before we continue, do you have any questions? [Answer any questions they may ask.]

[Provide respondent with a copy of the informed consent form. Ask the respondent to read the form (or read it to him/her, depending on respondent preference), answer any questions, and have the respondent sign the form. Leave a separate copy of the form with the respondent.

If the respondent has consented to recording, start the recorder.]

STEP 2: THINK-ALOUD PRACTICE

While you fill out the survey I want you to talk about how you are answering the questions. Hearing you talk about how you decide on your answers will help me understand how to make the questions better. I am more interested in how you arrive at your answers than in the answers you give. To help you get started on thinking aloud, we are going to do a practice activity. [Show an example by asking yourself the question and answering it using ‘think-aloud’]

I’m going to give you a piece of paper with a question on it. I’m going to ask you to read out loud the question and think aloud as you decide on your answer.

[Give respondent handout with think aloud practice.]

How many glasses of water did you drink yesterday?

Please tell me what you are thinking while you are figuring out your answer.

Probe as needed for detail: How did you arrive at that answer? What were you visualizing when you were deciding on that answer? Can you tell me more about what you are including in your count?

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about your answers to the questions in the survey.

[Hand respondent a copy of the questionnaire and a pen or pencil.]

STEP 3: COMPLETION OF THE QUESTIONNAIRE

Now I would like you to complete the questionnaire. I’d like you to think aloud while you decide on your answer. Please read aloud anything you would read to yourself if I were not here.

Remember that I am interested in hearing you talk about what you are thinking as you figure out your answers to the questions.

[REMEMBER TO TELL THEM THROUGHOUT THE QUESTIONNAIRE THAT THEY ARE DOING A GREAT JOB THINKING ALOUD, AND THEIR ANSWERS TO YOUR QUESTIONS ARE REALLY HELPFUL.]

Useful Probes—Use only as needed when response is unclear or R appears confused or unsure

Can you tell me in your own words what that question was asking?

What does the word [term] in this question mean to you?

How easy or difficult is it to remember [topic]?

How easy or hard was it to choose an answer?

Tell me what you are thinking.

How did you come up with that answer?

You said [answer]. Can you tell me more about that?

[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?

[If you pick up on a visual cue like a furrowed brow or a puzzled look]: Tell me what you are thinking.

[If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old question before answering this question]: Tell me what just happened. Tell me about what you were just doing.

Education

STOP RESPONDENT AFTER Q2 IF NEEDED – Probe on Q1-2

ASK QUESTIONS AND USE PROBES ONLY AS NECESSARY

Instructions: It is not necessary to probe on the instructions. Note if the respondent has any problems if they read them.

Q1-Q2: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Certifications and Licenses

STOP RESPONDENT AFTER Q12 – Probe on Q3-12

Q3-6: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q7: Is your most important certification or license required by a federal, state, or local government agency (such as a state board) in order to do that kind of work?

Who issued your certification or license?

Did you interpret this item to mean that you should answer yes ONLY IF the certification or license is required by a state board? Why do you think you interpreted it that way?

Q8: In how many states is your most important certification or license currently active?

In your own words, what does this question mean to you?

How did you come to select your answer?

How easy or difficult was it to answer this question?

Q9: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q10: To get your most important certification or license, did you…

How easy or difficult was it to answer this question?

Are any of these options confusing?

Are there any other activities you participated in that are not listed here?

IF R FIRST SAYS THEY TOOK CLASSES (“Yes” to 9a-c):

How easy or difficult was it to remember the classes, trainings, or other activities you participated in?

Q11: Did you prepare for your most important certification or license through a FREE government program?

How easy or difficult was it to answer this question?

If YES, tell us more about the program. Who offered it and who paid the program costs?

Q12: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q3 through 12

If the R did not explain from the think aloud and probes above probe further: Tell me a little bit more about your most important certification or license—in your own words, what was the process you went through in order to get it?

STOP RESPONDENT AFTER Q18 – Probe on Q13-18

Q13-15: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q16: Is your second-most-important certification or license required by a federal, state, or local government agency (such as a state board) in order to do that kind of work?

Who issued your certification or license?

(If R seems confused) How easy or hard was it to answer this question?

Q17: In how many states is your second-most important certification or license currently active?

How did you come to select your answer?

Q18: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q13 through 18

If the R did not explain from the think aloud and probes above probe further: Tell me a little bit more about your second most important certification or license—in your own words, what was the process you went through in order to get it?

STOP RESPONDENT AFTER Q24 – Probe on Q19-24

Q19-22: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q22: Is your third-most-important certification or license required by a federal, state, or local government agency (such as a state board) in order to do that kind of work?

Who issued your certification or license?

(If R seems confused) How easy or hard was it to answer this question?

Q23: In how many states is your third-most important certification or license currently active?

How did you come to select your answer?

Q24: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Preparation for New Certifications and Licenses

STOP RESPONDENT AFTER Q30 – Probe on Q25-30

Q25: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q26: Are you currently working on getting an ADDITIONAL professional certification or license?

In your own words, how does this question differ from the previous question?

(If R happens to mention currently being enrolled in school during talk-aloud portion of the Education section, try to probe on whether they are in a program that leads to certification or licensure (e.g., teaching, nursing, cosmetology, computer networking, electrician) and if so, whether they consider their current studies to be a part of preparing for certification or licensing in their field.)

Q27-28: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q29: Is this certification or license required by a federal, state, or local government agency (such as a state board) in order to do that kind of work?

Who issued your certification or license?

(If R seems confused) How easy or hard was it to answer this question?

Q30: In preparing for your MOST IMPORTANT new certification or license, are you currently participating in or have participated in any of the following activities?

Tell me a bit more about the process that you have to go through to get this certification or license.

Is there anything else important that you are doing or have done to get this certification or license that is not listed here?

Are you completing any of these activities online? If yes, did you include them when making your selections? If no, would you include any activities completed online (for example, taking online classes from a college) in marking your selections?

IF RESPONDENT CHECKS THE BOX FOR CERTIFICATE OR DEGREE: Can you tell me a bit more about the certificate or degree that you are required to complete to get this certification or license?

Employment

STOP RESPONDENT AFTER Q38 – Probe on Q31-38

Q31-36: It is not necessary to probe on this section. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q37: Do you spend 5 or more hours each week IN YOUR HOME looking after any children under age 13 who are not your own?

Can you tell me in your own words what this question was asking?

How easy or difficult was it to answer this question?

If yes, tell me a bit more about this. How many children to you look after? How many hours each week do you look after them? And these are NOT your children?

Q38: Are you paid for watching those children?

How easy or difficult was it to answer this question?

Looking at questions 36 and 37 does their placement make sense or do they seem out of place?

FINAL QUESTION

Is there anything else you would like to tell me about this questionnaire that you haven’t had a chance to mention?

(IF APPLICABLE) REMEMBER TO ASK OBSERVER FOR INPUT/ADDITIONAL QUESTIONS

STEP 4. THANK YOU FOR PARTICIPATING

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for improving this survey.

Interviewer Instructions: Collect materials and escort respondent out of the interviewing room and to the lobby. Please give the respondent incentive envelope [$40], and thank them for their time and effort.

Appendix B. Cognitive Interview Protocol (Spanish)

Interview ID:		Date of Interview:
First Name:		Interviewer

PASO 1: BIENVENIDA E INTRODUCCION DE LA ENTREVISTA

Instrucciones para el encuestador: El siguiente protocolo NO debe ser leído textualmente. El texto en letra cursiva solo es contenido con el cual el encuestador debe familiarizarse antes de la entrevista. El texto entre paréntesis [] contiene instrucciones para el encuestador.

Hola, mi nombre es <su nombre> y trabajo para [COMPAÑIA]. Mucho gusto de conocerle. Gracias por venir a ayudarnos el día de hoy.

[Platique de algún tema casual para establecer una buena relación con el participante.]:

Usted está aquí hoy para ayudarnos con un nuevo estudio de investigación que estamos realizando para el Departamento de Educación de los Estados Unidos. Este estudio pregunta acerca de las experiencias educativas y de entrenamiento para el trabajo de los adultos. Como parte de este estudio, le voy a pedir que complete una encuesta como que si estuviera completando una encuesta real en su casa. No existen respuestas correctas o incorrectas y está bien si usted no sabe algunas de las respuestas. A medida que usted vaya respondiendo a las preguntas, yo le voy a hacer algunas preguntas acerca de sus respuestas. Esto nos ayudará a que las preguntas en la encuesta sean mejores y más claras para otras personas como usted.

Debido a que la información que usted nos dará es muy importante, voy a estar tomando notas mientras estamos hablando [INSERTE SI HAY UN OBSERVADOR: y mi colega también va a observar nuestra sesión de hoy a través del cristal/por vídeo].

Además, esta entrevista será grabada en audio para que los investigadores del estudio puedan revisar la grabación más tarde. Cualquier información que se refiera a usted personalmente, tal como su nombre o empleador, no será incluida en nuestro informe y sus respuestas serán utilizadas únicamente para fines de investigación. ¿Está usted de acuerdo con lo que he mencionado?

Recuerde que esto no es un examen. Queremos que nos diga lo que usted piensa acerca de las preguntas de la encuesta.

Si en algún momento usted desea parar, solo déjemelo saber.

[Si por algún motivo el participante ya no tiene interés en participar, agradézcale por su tiempo y termine la entrevista.]

Antes de que continuemos, ¿tiene alguna pregunta sobre lo que acabo de decir? [Responda a cualquier pregunta que pueda hacer el participante.] Remember, you are not being tested; all we want is to learn how well the survey works.

DELE AL ENCUESTADO UNA COPIA DEL FORMULARIO DE CONSENTIMIENTO. PEDIRLE QUE LEA LA FORMA (O LEASELA, DEPENDINEDO DE LA PREFERENCIA DEL ENCUESTADO) RESPONDA CUALQUIER PREGUNTA Y PEDIRLE QUE FIRME LA FORMA. DELE UNA COPIA DEL FORMULARIO DE CONSENTIMIENTO AL ENCUENTADO

SI EL ENCUESTADO HA DADO SU CONSENTIMIENTO EMPIECE LA GRABACION

PASO 2: PRACTICA DE PENSAR VOZ ALTA

Mientras esté completando la encuesta, quiero que hable acerca de cómo está contestando las preguntas. Quiero escucharla(o) hablar acerca de cómo usted llega a sus respuestas para así entender cómo mejorar estos materiales. Me interesa más saber cómo llega a sus respuestas que en las respuestas que usted me dé. Para ayudarle a empezar a pensar en voz alta vamos a hacer un ejercicio de práctica.

Primero, le voy a dar una demostración de cómo pensar en voz alta. Después le voy a dar una oportunidad para que practique.

[Model the think aloud process for the respondent, including reading the question out loud. Elaborate on counting all of the glasses of water as needed you can make up pieces of information to emphasize the think aloud process e.g., “I don’t know if I should count the coffee I drank in the morning, I drink three cups of coffee, I am going to count those since that is the only liquids I have in the morning...” When you are modeling, write down the number of glasses of water as you are thinking aloud.]

[Once finished the model practice think aloud]

Voy a darle un papel con una pregunta. Le voy a pedir que lea en voz alta la pregunta y que piense en voz alta mientras decide su respuesta.

[Give respondent handout with think aloud practice.]

¿Cuántos vasos de agua tomó usted ayer?

Por favor dígame lo que está pensando mientras me da su respuesta.

Pregunte según sea necesario para obtener información detallada: ¿Cómo llegó a esa respuesta? ¿Qué estaba visualizando mientras estaba decidiendo qué responder? ¿Puede decirme más acerca de lo que usted está incluyendo en su cuenta?

Cuando haya completado el ejercicio: Gracias. Ese es el tipo de detalle que busco cuando usted me hable acerca de sus respuestas a las preguntas de la encuesta.

[ENTREGUE AL ENCUESTADO UNA COPIA DEL CUESTIONARIO Y UNA PLUMA O UN LAPIZ.]

PASO 3: RESPUESTAS A LA ENCUESTA

Ahora voy a darle la encuesta. Me gustaría que piense en voz alta mientras decide su respuesta. Por favor lea en voz alta cualquier cosa que usted hubiera leído para sí misma(o), como si yo no estuviera aquí.

Recuerde que estoy muy interesada(o) en escucharle hablar acerca de lo que está pensando mientras decide sus respuestas a las preguntas.

[RECUERDE DECIRLE AL ENCUESTADO QUE ESTÁ DANDO INFORMACIÓN MUY ÚTIL CUANDO PIENSA EN VOZ ALTA Y QUE SUS RESPUESTAS A LAS PREGUNTAS SON DE GRAN UTILIDAD.]

Preguntas útiles—Utilicelas solo cuando las respuestas no son claras o el encuestado parece confundido

En sus propias palabras, ¿qué dice esta pregunta?

¿Qué significa para usted la palabra [término] en esta pregunta?

¿Qué tan fácil o difícil es recordar [tema]?

¿Qué tan fácil o difícil fue escoger una respuesta?

Dígame lo que está pensando.

¿Cómo llegó a esa respuesta?

Usted dijo [respuesta]. ¿Puede decirme un poco más acerca de cómo llego a esa respuesta?

[En respuesta a las acciones o comentarios del encuestado]: Por favor, dígame que sucedió. Dígame que estaba usted haciendo.

[Si nota alguna señal visual, como cejas fruncidas o una expresión de confusión]: Por favor, dígame que sucedió. Dígame que estaba usted haciendo. Dígame lo que está pensando

[Si nota alguna señal visual que indica confusión, como pasar páginas adelante y atrás, volver a preguntas anteriores]: Por favor, dígame que sucedió. Dígame que estaba usted haciendo. Dígame lo que está pensando

Educación

ARE AL ENCUESTADO DESPUÉS DE Q2 SI ES NECESARIO – Pregunte sobre Q1-2

HAGA PREGUNTAS SEGÚN SEA NECESARIO

Instrucciones: No es necesario preguntar sobre las instrucciones, nada más anote si el encuestado tiene problemas con ellas.

Q1-Q2: No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Certificaciones y Licencias

PARE AL ENCUESTADO DESPUÉS DE Q12 – Pregunte sobre Q3-12

Q3-6: No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q7: ¿Es su certificación o licencia más importante requisito de un gobierno federal, estatal o local (por ejemplo una junta estatal) para poder realizar ese tipo de trabajo?

¿Quién emitió esta certificación o licencia?

¿Interpretó usted esta pregunta de manera que SOLO contestaría sí SI la certificación o licencia es requerida por una junta estatal? ¿Por qué cree que lo interpretó de esta manera?

Q8: ¿En cuántos estados es su certificación o licencia válida en la actualidad?

En sus propias palabras, ¿qué dice esta pregunta?

¿Me podría decir un poco más acerca de cómo seleccionó su respuesta?

¿Qué tan fácil o difícil le resultó responder a esta pregunta?

Q9: No es necesario indagar sobre esta pregunta. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q10: Para obtener su certificación o licencia más importante, ¿tomó algunos de estos tipos de clases?

¿Qué tan fácil o difícil le resultó responder a esta pregunta?

¿Le resultan confusas alguna de las opciones de respuesta?

¿Hay alguna otra actividad en la que participó que no está incluida en esta lista?

SI EL ENCUESTADO DICE QUE TOMÓ CLASES (Sí a 9a-c):

¿Qué tan fácil o difícil le resultó recordar las clases, entrenamientos, u otras actividades en las que participó?

Q11: ¿Se preparó para su certificación o licencia más importante a través de un programa del gobierno GRATIS?

¿Qué tan fácil o difícil le resultó responder a esta pregunta?

[SI contestó afirmativamente] Díganos un poco más sobre el programa. ¿Quién lo ofrecía y quién pagó los costos del programa?

Q12: No es necesario indagar sobre esta pregunta. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q3 through 12

Si el encuestado no dio suficiente información al pensar en voz alta o con las preguntas del encuestador, haga más preguntas: Dígame un poco más sobre su certificación o licencia más importante—en sus propias palabras, ¿qué tuvo que hacer para conseguirla?

PARE AL ENCUESTADO DESPUÉS DE Q18 – Pregunte sobre Q13-18

Q13-15: No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q16: ¿Es su certificación o licencia segunda en importancia requisito de una agencia del gobierno federal, estatal o local (por ejemplo una junta estatal) para poder realizar ese tipo de trabajo?

¿Quién emitió esta certificación o licencia?

(Si el encuestado parece confundido) ¿Qué tan fácil o difícil le resultó responder a esta pregunta?

Q17: ¿En cuántos estados es su certificación o licencia válida en la actualidad?

¿Me podría decir un poco más acerca de cómo seleccionó su respuesta?

Q18: No es necesario indagar sobre esta pregunta. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q13 through 18

Si el encuestado no dio suficiente información al pensar en voz alta o con las preguntas del encuestador, haga más preguntas: Dígame un poco más sobre su certificación o licencia segunda en importancia—en sus propias palabras, ¿qué tuvo que hacer para conseguirla?

PARE AL ENCUESTADO DESPUÉS DE Q24 – Pregunte sobre Q19-24

Q19-22: No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q22: ¿Es su certificación o licencia tercera en importancia requisito de una agencia del gobierno federal, estatal o local (por ejemplo una junta estatal) para poder realizar ese tipo de trabajo?

¿Quién emitió esta certificación o licencia?

(Si el encuestado parece confundido) ¿Qué tan fácil o difícil le resultó responder a esta pregunta?

Q23: ¿En cuántos estados es su certificación o licencia válida en la actualidad?

¿Me podría decir un poco más acerca de cómo seleccionó su respuesta?

Q24: No es necesario indagar sobre esta pregunta. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Preparación para Nuevas Certificaciones y Licencias

PARE AL ENCUESTADO DESPUÉS DE Q30 – Pregunte sobre Q25-30

Q25: No es necesario indagar sobre esta pregunta. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q26: ¿Está tratando actualmente de obtener una NUEVA certificación o licencia profesional adicional

En sus propias palabras, ¿en qué se diferencia esta pregunta de la anterior?

(Si el encuestado menciona estar inscrito en la escuela durante la sección de Educación, intenta indagar si se trata de un programa que lleva a una certificación o licencia (ej. educación, enfermería, cosmetología, electricista) y en tal caso, si consideran sus estudios actuales como parte de prepararse para una certificación o licencia en ese campo.)

Q27-28: No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q29: ¿Es esta certificación o licencia requisito de una agencia del gobierno federal, estatal o local (por ejemplo una junta estatal) para poder realizar ese tipo de trabajo?

¿Quién emitió esta certificación o licencia?

(Si el encuestado parece confundido) ¿Qué tan fácil o difícil le resultó responder a esta pregunta?

Q30: Para preparar su nueva certificación o licencia MÁS IMPORTANTE está usted participando o ha participado en alguna de las siguientes actividades?

Cuénteme un poco más sobre el proceso para conseguir esta certificación o licencia.

¿Hay algo importante que usted hizo o está haciendo para conseguir esta certificación o licencia que no está en esta lista?

¿Está usted completando alguna de estas actividades online? En caso afirmativo, ¿las incluyó al seleccionar las opciones? Si es que no, ¿incluiría actividades completadas online (por ejemplo, clases online de una universidad) al hacer su selección?

SI EL ENCUESTADO SELECCIONA LA CASILLA DE CERTIFICADO O TÍTULO: ¿Podría decirme un poco más sobre ese certificado o título que es requerido para completar esta certificación o licencia?

Empleo

PARE AL ENCUESTADO DESPUÉS DE Q38 – Pregunte sobre Q31-38

Q31-36: No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q37: ¿Pasa usted 5 o más horas cada semana EN SU HOGAR cuidando a algún niño(a) menos de 13 años que no son sean sus hijos?

En sus propias palabras, ¿qué dice esta pregunta?

¿Qué tan fácil o difícil le resultó responder a esta pregunta?

En caso afirmativo, cuénteme un poco más. ¿A cuántos niños cuida? ¿Cuántas horas a la semana pasa cuidándolos? Y estos no son sus hijos, ¿correcto?

Q38: ¿Le pagan por cuidar de esos niños?

¿Qué tan fácil o difícil le resultó responder a esta pregunta?

Cuando ve las preguntas 36 y 37, ¿le parece que tiene sentido donde están situadas, o le parece que están fuera de lugar?

PREGUNTA FINAL

¿Hay algo más sobre lo que le gustaría hablar acerca de estos materiales? ¿Hay algo acerca de lo que no ha tenido oportunidad de hablar?

(SI APLICA) RECUERDE PREGUNTAR AL OBSERVADOR SI TIENE ALGO QUE AÑADIR O PREGUNTAS ADICIONALES

PASO 4. GRACIAS POR PARTICIPAR

Hemos terminado. Gracias por ayudarnos. La información que nos ha dado será muy útil para mejorar las cartas y la encuesta.

Instrucciones para el entrevistador: Recoja los materiales y acompañe a los encuestados al vestíbulo. Por favor dele a los encuestados el sobre con el incentivo, y agradézcales por su tiempo y su ayuda.

Appendix C. English Cognitive Interview Instrument – Round 3

AIR 2021 ATES Cognitive Interviews – Round 1 (May 2018)

(OMB# 1850-0803 v.227 & v.228)

Date:	May 16, 2018
To:	Lisa Hudson (NCES)
CC:	Sarah Grady (NCES)
From	Samantha Lilly (AIR); Soner Dumani (AIR); Stacey Bielick (AIR)
Re:	2021 ATES Cognitive Interviews, Round 1

This memo presents common issues found during the first round of cognitive interviews for the Adult Training and Education Survey (ATES), held from April 30 through May 11, 2018. This memo also includes suggestions from AIR in response to these issues.

Fourteen interviews were conducted in Washington, DC by AIR interviewers using an abbreviated ATES instrument that included four sections: Education; Employment; Work Licenses and Certifications; and Work Experience Programs. All interviews lasted approximately an hour and were conducted in English. All but one interview was audio recorded. Seven respondents reported being currently enrolled in or taking courses from a college, university, technical or trade school, or other school after high school; two respondents reported having a certificate; ten respondents reported having at least one professional certification or license; nine respondents reported having completed a work experience program; and two respondents who work in a job that uses science, medicine, engineering, mathematics, or computer science¹.

Education

Education Level (A1-A4)

Question A4 asks respondents to describe why they first got their highest degree. Four respondents offered suggestions for additional response options to be included. These respondents reported an “other” category could be incorporated to address reasons such as “parental pressure”, “to meet social expectations”, “to help others”, “seemed interesting.” These responses seemed to speak to those pursuing degrees immediately after high school, who may not have a specific career path in mind.

AIR recommendation

Due to extensive testing of these questions in the past, AIR does not recommend any changes at this time. However, should NCES prefer, an additional “other” response option may be added to A4. The “other” response could also provide a write in option to allow NCES and AIR insight as to whether specific response options could be added later.

Certificates (B1-B6)

Six respondents missed the skip pattern or intentionally continued to the certificate section. These respondents quickly determined the section was not applicable to them and proceeded with the following section except for one respondent who erroneously continued through the certificate section despite not having a certificate.

Two respondents completed this section. One respondent reported earning a vocational certificate in video production, the other respondent reported earning a Microsoft Office certificate through a community or technical college. Both respondents did not have any difficulty answering questions within this section and did not report any response options to be confusing.

AIR recommendation

AIR does not suggest any revisions to this section.

Enrollment (C1-C7)

Most of the respondents did not have a problem with this section. In response to C7 which asks respondents if their current schooling is intended to prepare them to earn a professional license or certification, three respondents expressed concerns. One respondent who is enrolled in continuing education courses noted that the question does not consider those who already have a license and are enrolled in courses to maintain their license. One respondent hesitated to provide an answer because she expressed that some of her courses (e.g., finance) would prepare her to earn a professional license or certification but others would not. The third respondent reported that she was having a hard time between choosing Yes or No because she is undecided with whether she will use the courses she is currently enrolled in to earn a professional credential.

AIR recommendation

There is potential to change this question to capture more response possibilities than the current yes/no format, which we will discuss with NCES during the debrief meeting. For example, we could revise the question to “Is any of your current coursework intended to prepare you to earn…” and potentially open the question to capture maintaining a credential.

Employment

Employment Status (D1-D4)

Five participants expressed that the second part of D1 “If you are temporarily absent from a job (on vacation, temporary illness, maternity leave, etc.), answer “Yes”) either confused them or kept them from quickly responding to the question. Three of these participants read the question several times and indicated that they answered only to the first part of the question.

AIR recommendation

This may easily be resolved by presenting the instructional information differently. We recommend presenting this like the examples in D4, where the second part is not bolded.

Current Employment (E1-E10)

Respondents indicated confusion with the instruction “Count the same work with multiple clients as one job” in E1. One respondent indicated that freelance work with multiple clients introduces complexity. This respondent indicated that he could report one job, considering different work within the same field as one job, or multiple jobs, considering different types of work as separate jobs. The respondent, an audio engineer, elaborated, stating that recording music in the studio, going to a venue to do live sound, and remotely mixing recordings for different clients could be considered three jobs. Similarly, another respondent, an independent consultant, indicated that “same work” was restrictive. This respondent noted that operational aspects of business with one client and proposal development for another client could be considered separate jobs. The third respondent indicated that these instructions only applied to self-employed respondents. The fourth respondent initially read this instruction the opposite way and was not sure why this instruction was necessary.

Some respondents noted issues with E4, which asked respondents if they work at a permanent job or if all jobs are temporary. Those who experienced confusion with this question were consultants or freelance employees who work mostly with temporary contracts.

Several respondents experienced confusion with the response options provided in E6, which asks about respondents’ employer. Respondents experienced issues with the formatting of responses (uncertain of response “headings”) and uncertainty with selection (two respondents worked for a private university, three respondents selected self-employed or business owner rather than the sub-categories).

AIR recommendation

For EI, we recommend dropping the word “same” from the second sentence to read “Count work with multiple clients…” For E6, AIR recommends condensing response options, if possible. Language from previous survey administrations may be adapted to fit the needs and interest of NCES.

An employee of a private company, business, or individual, for wages, salary, or commission

A local (city, county, etc.), state, or federal government employee

Self-employed in own business, professional practice, or farm

Working without pay for family business or farm

However, if the response options are from the ACS or NSCG and need to be kept, we recommend replacing the capitalization with underline.

Credentials

Work Licenses and Certifications (F1-H5)

Several respondents exhibited difficulty in distinguishing licenses and certifications. See comments in “Other Comments and Recommendations” section below for more information and recommendations.

Some respondents attempted to report any or all licenses and certification, regardless of whether they were for the respondent’s main job. This resulted in misreports of credentials not used for respondents’ main job and confusion for those with multiple credentials who would either report on different credentials throughout the sections or feel the “Other Licenses and Certification” section was redundant.

Three respondents read F8 several times before answering and were confused about how certification was related to the license. One of these respondents expressed that she eventually concluded that the question was asking if someone needs to be certified first to get licensed but she wasn’t clear about the distinctions.

Four respondents reported having difficulty in understanding what is being asked in G4. Two of these respondents expressed that the question was asking whether the license or the certification was most important to them.

For G5, three respondents reported additional reasons to why they got their most important certifications. These reasons included “to stay current in the field”, “to be recognized”, “because of being a lifelong learner”, and “because the job paid for it”.

When responding to H1, asking about licenses or certifications required by any other jobs, some respondents noted uncertainty if the item meant the respondent should report credentials that are/have been used for another job or a credential that could be used for any other job, regardless of whether they have another job or have used it for a past job.

AIR recommendation

To ensure respondents answer these questions about licenses and certifications for their main job, AIR recommends including specific instructions and stressing specific words in the questions:

Insert a small instruction prior to the question F1, “The following questions (F1 through G11) are about licenses and certifications about the MAIN JOB you identified in E4”.
Replace “to work in this job” in F1 with “to work in your main job” and capitalize the words “MAIN JOB”.
Revise G1 to capitalize the words “MAIN JOB” and “NOT REQUIRED BY A GOVERNMENT AGENCY” and align the sentence structure with F1, such as “Do you have a professional certification that is NOT REQUIRED BY A GOVERNMENT AGENCY to work in your MAIN JOB?”

AIR recommends stressing the most important certification in G4 to increase clarity, such as “If you also reported a license in Section F, did you get your MOST IMPORTANT CERTIFICATION you reported in G3 in order to get that license?”. However, this question is difficult in a paper survey. We recommend NCES consider dropping it for the cognitive interviews. It may work better in a web survey where you can refer to the license and certificate names.

Should NCES prefer, AIR recommends adding an additional “other” response option to G5. The “other” response could also provide a write in option to allow NCES and AIR insight as to whether specific response options could be added later.

AIR recommends clarifying the language in question H1 to align with NCES underlying intent of this question. Below AIR proposes two options, dependent upon NCES needs:

If NCES wishes to capture any other licenses or certifications the respondent has, regardless of whether it is/has been used for another job, the question language could be revised to state, “Do you have ANY OTHER license or certification that is REQUIRED by a government agency?”
If NCES wishes to capture only licenses or certifications that have been used for other the respondent has held, language could be revised to state, “Do you have a license or certification that is REQUIRED by a government agency FOR ANY OTHER JOBS YOU HAVE OR HAVE HAD PREVIOUSLY? Include any other licenses or certifications, even if you don’t have that job now.”

Work Experience Programs (I1-I16)

Some respondents indicated issues with the work experience section. Specifically, some respondents reported on work experience programs that related to their current job rather than the program last completed or responded to questions considering multiple work experience programs. Additionally, three respondents found question I7 difficult to answer. Two respondents believed that none of the response options fit their situation. Among these two respondents, one respondent indicated that while the work experience program was completed while she obtained a bachelor’s degree, it was an optional program and not required of everyone. The second respondent indicated that her work experience program was completed to obtain a nursing assistant certification and did not feel any of the response options reflected this. The third respondent was confused about which option to choose. She chose her last work experience program being a part of school program because she completed it at a college.

Some respondents expressed a desire to explain certain situations in response to question I13, which asked if the respondent was hired by the company or business who sponsored the program. Respondents would have liked to report that they were offered a job but declined, or that the program was not for hiring purposes (such as student internships).

When probed about whether respondents’ last work experience program was a good launching pad for their career, some respondents reported that the response options did not apply. For example, some respondents reported that the work experience program was positive and useful, but they decided to switch fields or it was not a program purposed for entry-level positions.

AIR recommendation

AIR suggests providing language after question I1 to instruct respondents to refer only to the most recently completed work experience program for questions in this section. Instructions could read:

“We will refer to internships, apprenticeships, and similar programs as "work experience programs". Please answer about your MOST RECENTLY COMPLETED work experience program.”

AIR recommends keeping language to specify “last completed” or “most recently completed” work experience program within each question for the remainder of the section as well.

Other Comments and Recommendations

Many respondents indicated confusion in terms of the differences between license, certification, and certificate. Two respondents used terminology interchangeable at times, but reported credentials appropriately throughout the survey. Another respondent interpreted section F and G as the same, overlooking the distinction within the question regarding requirement by a government agency.

Additionally, many interviewers encountered challenges with completing the interview within the hour timeframe. Many respondents had multiple credentials which required completion of several or all sections. This resulted in less time to follow up with probing questions or inability to complete the survey in its entirety.

AIR recommendation

Although difficulty distinguishing various credentials is a common challenge across survey administration years, AIR recommends providing respondents with a reference to help define these credentials prior to beginning the survey. For example, a potential solution could be providing respondents with a short paragraph prior to beginning the survey that reads as such:

“This survey will ask you about your professional credentials, such as a professional license, certification or certificate. Please use the information provided to help distinguish which credential you may have.”

A definition of each credential, how they are typically obtained, and examples of each would then be provided for respondents’ reference. Due to the web format of the future ATES administration, this information could be presented as a pop-up message or a tab for which respondents could reference at any time.

To ensure respondents have sufficient time to complete the survey, AIR has taken steps to ensure all respondents arrive on time and allow for one hour of interviewing, such as revising communication with respondents to encourage all participants to arrive 10-15 minutes prior to the scheduled interview time. To the extent possible, AIR and NCES should also consider conciseness when revising the survey for future interview rounds.

AIR 2021 ATES Cognitive Interviews – Round 2 (June 2018)

(OMB# 1850-0803 v.227 & v.228)

Date:	June 13, 2018
To:	Lisa Hudson (NCES)
CC:	Sarah Grady (NCES)
From	Samantha Lilly (AIR); Soner Dumani (AIR); Stacey Bielick (AIR)
Re:	2021 ATES Cognitive Interviews, Round 2

This memo presents common issues found during the second round of cognitive interviews for the Adult Training and Education Survey (ATES), held from May 21 through June 8, 2018. This memo also includes suggestions from AIR in response to these issues.

During this round, five interviews were conducted in Washington, DC by AIR interviewers using an abbreviated ATES instrument that included four sections: Education; Employment; Work Licenses and Certifications; and Work Experience Programs. All interviews lasted approximately an hour and were conducted in English. All interviews during this round were audio recorded. One respondent reported being currently enrolled in or taking courses from a college, university, technical or trade school, or other school after high school; two respondents reported having a certificate; five respondents reported having at least one professional certification or license; four respondents reported having completed a work experience program; and one respondent works in a job that uses science, medicine, engineering, mathematics, or computer science².

Education

Education Level (A1-A2)

Question A2 asks respondents to describe why they first got their highest degree. Most respondents did not indicate any difficulty with this item. One respondent indicated some difficulty in that he went to college because his family and friends all went, implying a sense of obligation. Another respondent noted difficulty selecting the best response option, indicating that while some response options did apply, such as “To enter the workforce”, the actual reason was to pursue her passion and go to obtain the degree “for herself”.

AIR recommendation

Due to extensive testing of these questions in the past, AIR does not recommend any changes at this time.

Certificates (B1-B5)

One respondent overlooked the instructions to skip this section if respondent has a degree and completed the certificate section. This responded misreported a certificate by responding in to her EKG certification. This respondent reported this credential in both the certificate and certification sections.

For B1, one respondent indicated confusion when reading the instructions “Do not count a teaching certification or other certification”, as this respondent felt that this example would be applicable to this section, suggesting the respondent’s uncertainty in the difference between credentials. This respondent went on to report her Food Service Management certificate as a certificate and a license, and answered some questions in the certification section referring to this certificate.

The two respondents who completed this section did not have any difficulty answering questions within this section and did not report any response options to be confusing.

AIR recommendation

AIR does not suggest any revisions to this section. To address issues with misreported credentials, please see Other Comments and Recommendations Section below.

Enrollment (C1-C6)

Only one respondent indicated being currently enrolled in or taking courses from a college, university, technical or trade school, or other school after high school. This respondent did not have any difficulty or confusion when completing this section. All respondents stated that they found the phrase “other school after high school” clear.

AIR recommendation

AIR does not suggest any revisions to this section at this time.

Employment

Employment Status (D1-D4)

All five participants indicated that they are currently working for pay or profit and correctly skipped through questions D2-D4. Only one respondent indicated that the instructional information for D1 (“If you are temporarily absent from a job because you are on vacation, out sick, on maternity leave, etc., please answer ‘Yes’”) was confusing.

AIR recommendation

AIR does not suggest any revisions to this section.

Current Employment (E1-E9)

Two respondents indicated some uncertainty in how to respond to question E1, which asks respondents how many jobs they have. One respondent indicated she was unsure if freelance work is considered a self-employed job, while another respondent was uncertain if participation in research studies and clinical trials should be included. The respondent inquiring about freelance work counted these types of jobs in her total number, but the other respondent did not count participation in studies and trials in the total number of jobs.

Three respondents experienced confusion with the response options provided in E6, which asks about respondents’ employer. Two respondents interpreted the formatting of the responses to indicate that the respondent should select one response under each heading rather than selecting one, most appropriate response. Another respondent responded as though the question was a “select all that apply”.

Respondents identified a wide range of factors when reporting satisfaction of benefits in E9, including flow of income, 401K, insurance, pension, paid time off, and health benefits. Similarly, respondents also identified several considerations for physical working conditions, such as the safety of the environment, physical tasks, and physical aspects of the workplace.

AIR recommendation

In previous discussions with NCES, it was decided that the current language of E6 should remain as is to best fit the needs and interest of NCES. AIR recommends that, if condensing response options is not feasible, item text should capitalize and underline important instruction text as such:

We have some questions about your main job—that is, the job at which you work the MOST HOURS. First, which ONE of the following best describes your employment at this job?”

Credentials: Work Licenses and Certification (F1-H7)

Licenses

Four of the five respondents struggled answering question F1, which asks if the respondent has a license required by a government agency, because they were uncertain if their credential was required by a government agency. Three respondents reported having a license. One respondent reported having a Private Security Services License obtained for work as a security officer. Another respondent reported a Sales and Use license, which, through further probing and reflection, AIR believes to have been a business license. One respondent misreported a Food Service Management certificate, the same credential the respondent reported in the previous certificate section.

Two respondents indicated difficulty identifying which government agency requires the license or certification, as asked in question F5. It is important to note that both respondents were potential misreports (one a business license and one a certificate), which could have contributed to the uncertainty.

Two respondents misreported in question F12, which asks if respondents prepared to get licensed through a free state or city program. Both respondents indicated that these classes were through city entities, but were not free.

Certifications

One respondent, despite indicating “No” in G1 and “0” in G2, erroneously continued answering questions within this section. During interviewing, it was unclear whether the respondent was referencing a Microsoft skills certificate or the same Food Service management certificate previously reported in the certificate and license section prior. Two other respondents appropriately reported professional certifications in this section.

In responding to question G7, which asks about classes taken to get the knowledge and skills needed for respondents’ most important certification, two respondents included classes that were not mandatory for the certification, but rather contributed to gaining the skills necessary for certification later.

Other Licenses and Certifications

No issues to report in this section.

AIR recommendation

To address respondents’ confusion in terms of reporting the appropriate credential in the appropriate section, including who issued the credential, please see AIR recommendations under the Other Comments and Recommendations Section below.

Work Experience Programs (I1-I16)

Respondents did not appear to have any issues with this section. One respondent did indicate confusion responding to question I2, which asks if the last completed work experience program was for the respondent’s current main job. The respondent was unsure which response to select, as the internship was not for the same job, but was for the same type of work at a different place of employment.

AIR recommendation

AIR does not recommend any revisions to this section.

Other Comments and Recommendations

Multiple respondents indicated confusion in terms of the differences between license, certification, and certificate. As noted above, one respondent reported a business license and two reported a credential in multiple sections, including at least one incorrect section for their credential. This also was an issue experience by many respondents in the previous round.

AIR and NCES have exchanged ideas regarding concerns from the previous round that were not able to be addressed prior to Round 2. These issues and corresponding recommendations are presented below.

AIR recommendation

AIR and NCES have discussed potential resolutions to combat respondents’ difficulty distinguishing various credentials, a common challenge across survey administration years. Based upon previous discussions of Round 1 findings, AIR and NCES have determined that Round 3 testing will include a definitions page prior to the first page of survey items. This reference will inform respondents that the survey will ask about specific credentials and will outline important information such as the typical awarding body, process for obtainment, and renewal requirements for certificate, certifications, and licenses. The inclusion of this resource aims to guide respondents in identifying their most appropriate type of credential and reporting accurately within the survey.

To address previous concerns from Round 1 regarding respondents’ confusion from mentioning both license and certification in section F items, AIR and NCES have previously discussed removing “or certification” from questions F1-F4. This would allow a clear distinction between section F, which would only mention licenses, and section G, which would only ask about certifications.

Additionally, to avoid confusion brought about by the language “required by a government agency” and “not required by a government agency” in questions F1 and G1, respectively, AIR and NCES have discussed removing such language and instead inserting a question in each section to ask respondents who issued the license/certification.

AIR 2021 ATES Cognitive Interviews – Round 3 (August 2018)

(OMB# 1850-0803 v.227 & v.228)

Date:	August 06, 2018
To:	Lisa Hudson (NCES)
CC:	Sarah Grady (NCES)
From	Samantha Lilly (AIR); Soner Dumani (AIR); Stacey Bielick (AIR)
Re:	2021 ATES Cognitive Interviews, Round 3

This memo presents common issues found during the second half of the third round of cognitive interviews for the Adult Training and Education Survey (ATES), held from July 17 through August 3, 2018. The first portion of the third round, which yielded eight interviews, was conducted June 25 through July 13, 2018. The findings from the first portion of the third round of interviews were discussed with the client during an in-depth meeting and resulted in several changes to the instrument to be tested during the second portion of the third round. This memo only presents issues from the second portion of the third round and suggestions from AIR in response to these issues.

During this round, four interviews were conducted in Washington, DC by AIR interviewers using an abbreviated ATES instrument that included four sections: Education; Employment; Work Licenses and Certifications; and Work Experience Programs. All interviews lasted approximately an hour and were conducted in English. All interviews during this round were audio recorded. Two respondents reported being currently enrolled in or taking courses from a college, university, technical or trade school, or other school after high school; one respondents reported having a certificate; three respondents reported having at least one professional certification or license; three respondents reported having completed a work experience program; and one respondent works in a job that uses science, medicine, engineering, mathematics, or computer science³.

Education

Education Level (A1-A2)

Questions A1 and A2 ask respondents to indicate their highest degree or level of school completed and to describe why they first got their highest degree, respectively. Most respondents did not indicate any difficulty with this item. One respondent answered the questions as though they were a “select all that apply”, selecting each applicable option rather than only the best response.

AIR recommendation

Due to extensive testing of these questions in the past, AIR does not recommend any changes.

Certificates (B1-B4)

One respondent indicated having multiple certificates, albeit expired, and correctly skipped out this section due to the skip pattern instruction for those with a degree. Another respondent reported an office assistant certificate but indicated that it was earned in high school (rather than a school after high school as denoted in the definitions box). It is uncertain whether this was a misreport of credential, or whether the office assistant certificate can be obtained through a high school program in some instances.

AIR recommendation

AIR does not suggest any revisions to this section.

Enrollment (C1-C6)

One respondent initially indicated that she was currently enrolled in or taking courses from a college, university, technical or trade school, or other school after high school, referring to classes she takes to earn continuing education units (CEUs) to maintain her license. Upon reading the subsequent questions she revised her answer, as she did not consider herself enrolled for these classes. One other respondents reported being currently enrolled. This respondent struggled selecting an answer to question C6, which asks respondents to describe why they are in school. The respondent noted that she would have liked an option “to advance in a new line of work.” However, the respondent selected “to switch to a new line of work”. Upon probing, the respondent was not in the line of work previously or presently and therefore, made the appropriate selection.

AIR recommendation

AIR does not suggest any revisions to this section.

Employment

Employment Status (D1-D6)

Three participants indicated that they are currently working for pay or profit and correctly skipped through questions D2-D4. One of these respondents encountered some confusion when answering question D1 (“Are you currently working for pay or profit?”), noting that she believed the question was asking respondents to choose between working for pay or working for profit, rather than considering them together. Upon reading the response options (yes or no), she realized the intent of the question and answering accordingly.

One participant indicated that she was not currently working. This respondent had no issues answering the questions in this section, including two new items in the instrument (D5 & D6) added to prevent potential confusion from wording regarding respondents’ main job in questions H1 & H2.

AIR recommendation

Although respondents have indicated some confusion in the past regarding “pay or profit” in question D1, ultimately all respondents appear to answer appropriately upon reading further. Therefore, AIR does not suggest any revisions to this section.

Current Employment (E1-E11)

One respondents misreported when respondent to question E1, which asks respondents how many jobs they have. This respondent initially indicated that she had one job, but later began discussing a second job. When probed, the respondent noted that she works at one job during the week, the job which she counted in E1, and then another job on the weekend.

Question E4 asks respondents “Do you work at a permanent job (that is, a job with no set end date)?” One respondent said that she interpreted a “permanent job” to mean a stable and reliable full-time job. This respondent answered no, stating that she had a part-time instead. One respondent appeared confused by the parenthetical (“that is, a job with no set end date”), but ultimately selected Yes appropriately. Another respondent selected No because her job as a teacher is through 1-year contracts, meaning technically there is a set end date. R stated that if the question did not include the parenthetical she would have selected Yes. Therefore, this respondent also indicated her teaching job as temporary in question E9.

One respondent had difficulty with the wording of question E6, which asks respondents about their “main job—that is, the job at which you work the MOST HOURS.” This respondent interpreted the question to ask about all jobs, past and present. Therefore, the respondent interpreted “main job” to be her current job, but “the job at which you work the most hours” to be a job she previously held for a long period of time. The respondent ultimately responded incorrectly, referencing a past job.

One respondent experienced confusion with the response options provided in E6, which asks about respondents’ employer. This respondent appeared uncertain as to the meaning of response options, interpreting “for profit” to mean a job in which you get paid, and “not-for-profit” to be a volunteer job that does not require you to pay taxes. The respondent selected “for profit” since she gets paid for her work.

AIR recommendation

AIR suggests the consideration of potentially revising question E4, dependent upon NCES’ desired interpretation of the term “permanent job.” One suggested revision would be to ask respondents if they work at a temporary job in question E4 and E9, as most respondents appear more confident in the understanding of a “temporary” than “permanent job.” However, the current language provides explanatory text (“no set end date”), which supports uniform responses. Another consideration may be to include additional text in E4 stating that both full-time and part-time jobs can be permanent.

Credentials

Licenses (F1-F13)

One respondent answered questions in this section referencing two different licenses throughout the section. Therefore, without probing, it was not possible to decipher which license responses corresponded to. Another respondent answered questions in this section referring to a license which is not currently for her main job.

Two respondents indicated some difficulty in determining who issued their license. One respondent noted that the example of “state board” helped determine the appropriate response.

One respondent incorrectly responded to question F8, which asks about how respondents maintain their license. This respondent appeared confused, answering the item as though it was asking about how the respondent first obtained her license, rather than the renewal process. Another respondent indicated that she had to earn CEUs and also take classes, but later noted that she did not believe the classes were required to maintain her license and should probably have not selected that response option.

Question F12, which asks respondents to select the response option which best describes the classes taken to prepare to get licensed, resulted in some difficulty for two respondents. One respondent, despite indicating taking multiple classes to get the skills and knowledge needed for her license (question F11), selected “I did not take any classes to get licensed.” Upon probing, this respondent appeared to have been considering only the examination step in the process required to get licensed, rather than any preparation. This respondent repeated this reporting behavior when responding to these items in the certification section (question G9 and G10). The other respondent was uncertain whether to select classes as part of a bachelor’s program or Master’s program to get her teaching license, as both were integral parts of the preparation process. This respondent ultimately selected the Master’s program, as this was specifically focused on teaching and was required to earn her teaching license, but noted that she felt best prepared through her bachelor-level classes.

Certifications (G1-G15)

One respondent referred to “certification” as “certificate” throughout the section. This respondent did not read the definition box provided above question G1 and selected Yes to G1 because “she has a certificate.” However, this respondent did appropriately respond to questions referring to a Home Health Aide certification, despite verbally calling it a certificate.

Other Licenses and Certifications (H1-H8)

One respondent reported the same license in H1 that she reported previously in section F. The license was appropriately reported in section H because it was not for her main job. Another respondent indicated that she was working to get a new certification in H3 because she is currently in school and once school is completed she will be eligible to take the state board of nursing exam. Therefore, this respondent interpreted being enrolled in school as “working to get a new license or certification”.

AIR recommendation

Misreporting has been a challenge for many previous survey administrations. Many respondents struggle to report only one credential and/or job at a time, resulting inaccurate or unreliable data. To address these concerns, AIR proposes utilizing an automation process via web administration to auto-fill portions of questions to specify which specific job or credential the respondent should reference in answering the question. This solution has been proposed by NCES and will be discussed with DOL to determine feasibility and logistics of this approach.

Work Experience Programs (I1-I15)

One respondent completed the work experience program section referring to a two-day medication training completed as part of her job at the time.

One respondent struggled with question I6, which asks about wage earned during the work experience program. The respondent, who was referencing a teaching internship, explained that she did not receive pay for the internship, but through the internship could be offered substitute teaching opportunities that did pay. She stated that the pay did not increase, but the opportunities that paid did increase. When she substituted for classrooms, the pay was the same as a regular substitute teacher, so she ultimately selected “The same wage as a fully qualified worker.” When asked what a set wage meant to this respondent, she stated she thought of it as a defined pay. For example, earning $12 an hour as an intern substitute teaching, and earning $15 hour as a regular substitute teacher.

AIR recommendation

AIR does not recommend any revisions to this section.

Other Comments and Recommendations

The version of the instrument utilized in the last half of the third round of interviews incorporated text boxes of definitions and relevant information about each credential prior to the first question in the corresponding section. These were used to help respondents determine the appropriate credential and decipher amongst credentials. Although one respondent admittedly did not read the information in the boxes while completing the survey, she later stated that reading the boxes first would have helped her understand the definitions of credentials and make responding to the questions easier. The three other respondents noted that all information presented in the text boxes was clear and helpful in determining or confirming the appropriate credential. All three respondents specifically stated that the use of bold key terms for each credential was helpful.

AIR recommendation

AIR recommends continued use of the text box definitions and relevant information to help respondents in accurately report credentials within the appropriate section. The informative text box may be able to be displayed as a sidebar throughout corresponding sections in web survey administrations as a constant reference for respondents.

Mathematica 2022 National Training, Education, and Workforce Survey (NTEWS) Cognitive Research (April 2021)

(OMB# 3145-0174)

I. Introduction

The National Center for Science and Engineering Statistics (NCSES) has a legislative mandate to collect, analyze, interpret, and disseminate data on the U.S. science and engineering workforce, disaggregated by detailed demographic subgroups such as sex, race/ethnicity, and disability status. The National Center for Education Statistics (NCES) also has a legislative mandate to collect, collate, analyze, and report complete statistics on the condition of American education. Under these mandates, a new survey, the National Training, Education and Workforce Survey (NTEWS), aims to measure and understand two research concepts that are of national interest: (1) the effects of a diverse credentialing system (certifications, licenses, postsecondary certificates, and work experience programs) on career opportunities and (2) the education, training, and career paths of skilled technical workers to meet America’s workforce demands and stay globally competitive in an increasingly scientific and technological era.

The NTEWS is a voluntary, nationally representative survey geared toward individuals ages 16 to 75 who are not enrolled in high school. This specific population allows for coverage of employment-eligible adults in the United States. The NTEWS instrument draws content from three surveys: the National Survey of College Graduates (NSCG), the Adult Training and Education Survey (ATES), and the American Community Survey (ACS). Each of these surveys has previously undergone cognitive testing, a survey development method that seeks to ensure that respondents understand the questions and the questions are capturing data accurately.

NCSES and NCES closely evaluated each question from the three source surveys for quality and utility, according to federal survey guidance when drafting the NTEWS instrument. Although tested, the populations targeted in testing were not always directly comparable to the target population for the NTEWS. For example, questions on the NSCG were tested and fielded on individuals who have a bachelor’s degree or higher. In addition, some questions were modified to address known difficulties and new questions were considered. Considering the COVID-19 pandemic, there were additional changes to the NTEWS questionnaire to ensure relevancy of response categories. Cognitive testing focused on ensuring that all questions were appropriate, meaningful, and understood consistently among participants.

This report documents findings from 58 cognitive interviews designed to test the NTEWS instrument. Mathematica conducted the interviews as a subcontractor to the NCSES. The report is divided into two sections. The first section describes the cognitive interview methodology and provides characteristics of the sample. The second section provides findings by survey section. This includes an overall summary as well as detailed findings and revisions made through the course of testing. Appendix A provides a detailed summary of all changes made to the survey throughout the cognitive testing process. This report aims to provide context for changes made as a result of instrument testing and provide guidance for researchers who will use NTEWS data in the future about the respondents’ understanding of the questions.

II. Testing Methodology and Characteristics of the Sample

From September 2020 to January 2021, Mathematica conducted four rounds of virtual cognitive interviews. This section describes the questionnaire, participant recruitment, and the cognitive testing process.

A. Instrument

Although the full NTEWS instrument was developed before the cognitive testing, the study team did not include all questions in all rounds of testing. Early rounds of testing concentrated on the survey sections related to licenses, certificates, and vocational certificates. Later testing rounds included the sections on education and enrollment and the full series of background questions. Table 1 indicates the sections tested during each round of testing.

Table 1. NTEWS instrument sections cognitively tested, by round

NTEWS instrument section	Round 1	Round 2	Round 3	Round 4	Total rounds tested
Employment Status	X	X	X	X	4
Current Employment	X	X	X	X	4
Licenses and Certifications^a	X	X	X	X	4
Work Experience Programs	X	X	X	X	4
Educational Background			X	X	2
Vocational Certificates	X	X	X	X	4
Enrollment				X	1
Background^b	X	X	X	X	4

^a In Round 1, Licenses for Work and Certifications for Work were separate sections. In all later rounds, these were combined into one Licenses and Certifications section.

^b In Rounds 1 through 3, only a subset of the questions in the Background section were tested. All items in the Background section were tested in Round 4.

Between each round of testing, Mathematica, in consultation with NCSES, revised the survey instrument to address issues identified during testing or to add questions that had not previously been tested. Appendices B through E contain the four tested versions of the instrument.

B. Sample and recruitment

Mathematica conducted 61 interviews across four rounds of cognitive testing, 58 of which were used in this analysis⁴. Cognitive interview participants were primarily recruited through two market research firm’s national panels. Several participants also were recruited from the study team’s personal network.

As noted, the first three rounds of testing focused on the licenses, certifications, and vocational certificates sections of the questionnaire. In these rounds, recruitment included participants who self-identified as having an active license, certification, or vocational certificate. The study team conducted the final round of testing with a general population sample, designed to mirror the ultimate NTEWS population. The first round included participants from the Northeast, Southeast, and Midwest regions of the United States. For the second and third rounds, the study team requested that the recruitment firm focus on recruiting participants from the West Coast to balance the heavier East Coast sample from the previous round. For the last round, the study team requested that the firm recruit participants across the United States.

Mathematica also worked with its recruitment partners to ensure diversity in terms of gender, race/ethnicity, and employment status in each round of cognitive testing. Participant demographic characteristics are presented by round in Table 2. Table 3 contains participants’ credentials by educational attainment in each round.

Table 2. Demographic characteristics of NTEWS interview participants, by round

		Round 1 (n = 18)	Round 2 (n = 11)	Round 3 (n = 12)	Round 4 (n = 17)	Total (n = 58)
Gender
	Female	8	5	6	7	26
	Male	10	6	6	10	32
Race/ethnicity
	African American/Black	9	5	1	1	16
	White	8	4	10	15	37
	Other	1	2	1	1	5
Educational attainment
	Bachelor’s degree or higher	5	6	4	9	24
	Less than bachelor’s degree	13	5	8	8	34
Current employment status
	Employed	14	10	9	15	48
	Unemployed	4	1	3	2	10
Credentials
	Licenses or certifications	3	3	5	4	15
	Vocational certificates	8	5	4	4	18
	Both credentials	7	3	3	2	14
	Neither credential	0	0	0	7	11

Table 3. Credentials of NTEWS interview participants, by educational attainment and round

		Had license or certification	Had vocational certificate
Round 1	Bachelor’s degree or higher	4	4
Round 1	Less than bachelor’s degree	5	12
Round 2	Bachelor’s degree or higher	4	3
Round 2	Less than bachelor’s degree	2	5
Round 3	Bachelor’s degree or higher	4	1
Round 3	Less than bachelor’s degree	4	6
Round 4	Bachelor’s degree or higher	3	1
Round 4	Less than bachelor’s degree	2	1
Total	Bachelor’s degree or higher	15	9
Total	Less than bachelor’s degree	11	24

C. Conducting cognitive interviews

Mathematica conducted the cognitive interviews using the Zoom video conferencing app, with two to four people on each call. At minimum, the respondent and a lead interviewer were on each call, with a notetaker and an observer on most calls. Both respondents and interviewers were on video during each call. This both facilitated rapport between interviewer and respondent and allowed the interviewer to track responses and monitor respondents’ nonverbal cues. Interviews were 45 to 90 minutes long, depending on responses to questions. The interviews were recorded (both audio and video) to facilitate analysis and allow for other members of the team to conduct post-interview observations.

Respondents completed the survey using a web-based, editable PDF document, simulating the experience of completing a paper-and-pencil survey. This format allowed respondents to see multiple questions per page, easily scroll back and forth to look at upcoming and previous questions, click radio buttons or checkboxes to indicate their responses, and type directly into the PDF to provide responses to “other–specify” or open-ended questions. At the beginning of each interview, the interviewer obtained verbal consent from the respondent and explained the structure of the interview process. The interviewer then asked the respondent to open a link to the editable PDF and share their screen.

With the respondent sharing their screen, the interviewer, notetaker, and observer (if applicable) could directly observe how the respondent interacted with the survey instrument (in terms of mouse movement, scrolling, changing responses, and so on) in addition to seeing the respondent’s video to observe facial cues and gestures. When respondents were unsure of how to answer a question and asked for assistance, the interviewer instructed them to answer as if they were completing the survey on their own. Respondents completed the survey one or two sections at a time, after which interviewers would stop respondents and retrospectively probe on target questions in that section or questions where respondents made an error (such as missing skip logic) or seemingly struggled with a question.

III. Summary of Findings, Revisions Between Rounds, and Remaining Issues, by Survey Section

A. Employment Status

Summary of findings for Employment Status questions

Respondents had little difficulty with the Employment Status section of the survey. Some respondents struggled with the “for pay or profit” phrasing in the question “Are you currently working for pay or profit?” However, this confusion did not affect their ability to correctly answer the question. Several respondents also missed the skip instructions following this question, which directed respondents to the next question if they were not working or to the next section of the survey if they were. The missed skip is likely an artifact of the way the respondents interact with an editable PDF on screen rather than having a physical paper and pencil survey. The study team does not expect this to be an issue when the survey is administered as a true pen-and-paper personal interview and is not in web format. The only change made in this section was to include additional COVID-19–related response options for the question “What is your main reason for not working?” This change was made to address issues for respondents who indicated their lack of employment was due to the COVID-19 pandemic.

1. Findings from Round 1

Overall, respondents did not have significant problems with this section. However, two questions presented challenges for some respondents.

Q1: Working for pay or profit

Some respondents struggled with the phrasing “for pay or profit” in the question “Are you currently working for pay or profit?” Though some found this distinction confusing, respondents were still able to accurately answer the question. Several respondents missed the skip after “Are you currently working for pay or profit?” (If yes “Go to Question 6”). The study team did not make any revisions but did monitor these questions in subsequent rounds to determine whether they caused respondent error.

Q3: Reason for not working

Although it was not an issue in this round of testing, response options were added to the question “What is your MAIN reason for not working?” to reflect that some respondents may not be working due to the COVID-19 pandemic.

All other questions in this section performed as intended in this round and no changes were needed.

2. Findings from Round 2

There were no major problems with this section in Round 2. However, the study team still observed problems with respondents not noticing the skip instruction.

Q1: Working for pay or profit

Unlike in Round 1, no respondents expressed confusion by the “pay or profit” wording in Round 2, but respondents continued to struggle with the skip instructions in the question “Are you currently working for pay or profit?” (If yes “Go to Question 6”). Roughly half of respondents in Round 2 missed the skip instructions and incorrectly proceeded to the next question. The study team suspected that problems with noticing the skip instructions was most likely an artifact of testing a paper-and-pencil questionnaire on the web, therefore, the study team did not make any revisions before Round 3.

All other questions in this section performed as intended in this round and no changes were needed.

3. Findings from Round 3

The study team continued to observe the same issues that were observed in Rounds 1 and 2.

Q1: Working for pay or profit

As in Round 2, no respondents expressed confusion by the “pay or profit” wording in Round 3. One-third of Round 3 respondents missed the skip instructions and incorrectly proceeded to the next question.

Q3: Reasons for not working

Some respondents who were not working struggled with choosing the right response option. “On layoff” was not accurate, especially if they were self-employed or furloughed during the pandemic. These respondents selected the “other” response option. For example, one respondent was self-employed and has not had clients since the pandemic. He was not “laid off” because he does not work for someone else, but the pandemic was still the cause of his lack of work. Another respondent explained that she has been consistently looking for work since she was laid off at the start of the pandemic. Not working due to the layoff, which was now many months ago, felt inaccurate to her because “I’ve been looking for jobs…I’m still in pursuit of working.” Changing the response options would create redundancy with the remaining options, so ultimately no changes were made. This question may require additional backcoding during the initial data collection year, but the pandemic-related response options most likely will be removed after the initial collection year, eliminating the unique issues that the pandemic has created.

All other questions in this section performed as intended in this round and no changes were needed. As previously stated, although there was continued confusion from Question 3, the study team did not ultimately make any changes to this section.

4. Findings from Round 4

Q1: Working for pay or profit

Respondents continued to struggle in Round 4 with the skip instructions in the question “Are you currently working for pay or profit?” (If yes “Go to Question 6”). Some respondents, albeit a smaller proportion than in Round 3, missed the skip instructions and incorrectly proceeded to the next question.

Again, the missed skip is likely an artifact of the way the respondents interact with an editable PDF on screen rather than having a paper-and-pencil survey. The study team does not expect this to be an issue when the survey is administered as a true pen-and-paper personal interview and is not in web format. No additional changes were made to the instrument.

All other questions in this section performed as intended.

B. Current Employment

Summary of findings for Current Employment questions

Respondents had different understandings of the term “main” job, which had consequences for the entire section. Respondents expressed some confusion in how they thought about their “main” job, particularly for those who work in the gig economy (for example, driving for Uber Eats or Lyft)⁵. Because gig work is a relatively new concept, it is unclear how respondents should treat this work and how they should report it, especially because the work can be ad hoc and infrequent. Respondents who worked in the gig economy provided inconsistent answers about number of jobs they have, due to some respondents thinking about a gig job as separate work “on the side” and not as a job on its own. These issues prompted significant changes to the instrument between the second and third rounds that is described in detail below. Other issues in this section included respondents struggling to provide their earnings and several respondents’ inability to provide their work address accurately from memory.

1. Findings from Round 1

There were some minor issues in Round 1 with respondents not following the skip instructions and with respondents reporting their earnings before deductions.

Q11: Earnings before deductions

Some respondents had difficulty providing the exact value of their earnings before deductions. The respondents either used a calculator or provided an estimate instead. The study team also noted that some respondents were hesitant to answer this question. Income questions can be sensitive. Some respondents asked if they had to answer this question, and others skipped it without saying anything.

Q16-18: Self-employment

A few respondents overlooked the skip instruction at Question 16 if the they were not self-employed, leading them complete nonapplicable questions (Questions 17 and 18).

Q22: Name and address of employer

Some respondents either needed to look up the ZIP codes for their workplaces or left the ZIP code blank. While the study team noted the issues with these questions, it is necessary to maintain this question as is to enable post-production data coding of employers. These questions were monitored in subsequent rounds.

All other questions in this section performed as intended.

2. Findings from Round 2

Most of the Round 2 findings were similar to the Round 1 findings. One new issue that emerged in this round was that some respondents had difficulty estimating how many hours they worked each week.

Q10: Hours worked each week

Two respondents indicated that they frequently work more hours than they are salaried for and were therefore unsure how to answer this question. Both respondents were teachers, and one explained, “Like most teachers, I could work every single day and still have work to do…Being a teacher, I’m contracted to work 7.5 hours a day, but I work substantially more than that. Which is why I was asking if you want me to list on-paper hours, or the hours I spend planning and researching and other things that I put into my job. I still think I work well over 60.” These respondents interpreted this question as the number of hours for which they are paid, rather than the number of hours they actually worked. It is unclear whether other respondents were also misreporting.

Q11: Earnings before deductions

Similar to Round 1, one respondent reported difficulty providing the exact value of his earnings and only knew his hourly wage and after-tax earnings.

Q22: Name and address of employer

Several respondents did not know their employer’s full address from memory and had to look up this information.

Due to the relatively few respondents that faced challenges in this section, no revisions were made after Round 2. These questions were monitored during the next round of testing.

All other questions in this section performed as intended.

3. Findings from Round 3

As in prior rounds, respondents struggled to provide the exact value of their earnings and employer address. However, the main issue that emerged in this round was additional questions on how to report their work, either as a result of change in work during the pandemic or working in the gig economy.

Q12: Main job

In at least two cases, it appeared that respondents considered both their current job and a prior job when reporting about their “main” job. One respondent noted, “I got laid off, so based on how this is worded, the one that I worked the most hours for is the one I got laid off from. But I have not worked there since the end of October. I’m now self-employed, which is how I answered the previous questions.” This respondent switched from answering about her current employment to reporting on her past employment. Respondents working in the gig economy also expressed some confusion about whether they should count gig jobs and how to do so. These two issues affected how respondents answered Questions 13 through 22, as they all referenced the respondent’s “main” job.

After discussion with stakeholders to pinpoint the desired construct to be measured, NCSES and Mathematica made significant revisions to this section of the survey before the final round of testing to address response errors related to what they consider as their “main” job and for those working in the gig economy. The revisions included:

Changing the questions “Do you work at a full-time job (a job where you usually work 35 or more hours per week)?” to “During the PAST 12 MONTHS, how many hours did you usually work each WEEK across ALL jobs?”
Moving “What is the name of your MAIN job?” earlier in the section
Adding an instruction to the MAIN job question so that respondents focused on their main job rather than the one they worked the most hours at in the past 12 months
Adding the questions “During the PAST 12 MONTHS (52 weeks), did you work EVERY week at the MAIN job you wrote in Question 8?” and “(if No) During the PAST 12 MONTHS (52 weeks), how many WEEKS did you work at the MAIN job you wrote in Question 8?”
Adding a reference to the MAIN job in this question: “During the PAST 12 MONTHS (52 weeks), how many WEEKS did you work at the MAIN job you wrote in Question 8?”
Adding street address when capturing employer address, which was a new addition to allow for more detailed post-production coding

4. Findings from Round 4

Despite revisions to the instrument between rounds, respondents, namely those working in the gig economy, still had challenges when completing this section. For example, one respondent worked for three gig companies and did not count any of these in her total number of jobs. She was not thinking about the “gigs” when counting her jobs.

No additional revisions were made to these questions after the final round, as more testing is needed to determine how these questions could or should be revised. However, future rounds of data collection should further consider how the gig economy fits into the current set of employment questions on the NTEWS questionnaire, and whether these questions accurately measure the extent to which people work these types of jobs. Specifically considering the following may be useful: (1) Does gig work count as one job?; (2) Does all gig work count as one job (for example, if someone works for Uber and Lyft, is that a single job?); and (3) How should this type of work be counted if it is based on when a person wants to perform it, and is intermittent or infrequent?

Q8: Main job

Respondents continued to interpret their “main” job differently. Some were unsure whether it should be based on income, time, or interest level. One respondent said, “The one I spend more of my time on and more of my income. I interpreted ‘you most identify with’ is the one I spend the most time or income on, not just about what I’m passionate about.” The variation in the interpretation of what a “main job” is may yield different responses from different respondents and raise reliability concerns. No additional changes were made to this question. However, future iterations of this question could include a clear definition of what is meant by “main” job or by asking the questions two ways – one focused on the job with the most income or hours and one focused on the job with which respondents most identify.

Q22: Name and address of employer

Respondents continued to struggle with entering their employer’s address. As in prior rounds, some respondents did not know their employer’s full address from memory, including the street number. They either left it blank or looked it up during the interview. Some respondents entered address information on incorrect lines (for example, street address on the department), although the study team does not expect this type of error to be an issue in a web survey. Another respondent explained she did not know the address because she had never gone to her employer’s physical location due to the COVID-19 pandemic. For self-employed workers or contractors, the study team observed respondents providing their home address or the contractor address. It was unclear how self-employed respondents should report their address. Because this question will be used for post-production coding and needs to be consistent with questions on other surveys, the study team did not make any additional revisions to this question. However, researchers should note that the data from this question, in particular the “Street Address” field, may contain a high amount of missing or inaccurate data. When the survey is fielded, using auto-filling ZIP codes based on city and state could reduce missing data and respondent burden on the web version of the survey.

All other questions in this section performed as intended. The issues that emerged in this section were fairly consistent across rounds.

C. Licenses and Certifications

1. Findings from Round 1

Summary of findings for Licenses and Certifications questions

These sections posed the most challenges for respondents throughout testing, with respondents largely unable to differentiate between the concepts of licenses and certifications. At the outset of testing, licenses and certifications were separate sections, but the team later combined them because respondents were frequently confused about how to classify their credentials, leading to misreporting, both underreporting and double counting their credential. To better understand the focus of the respondent’s license or certification, multiple versions of open-ended questions were tested (for example, asking about field of study, area of specialization, primary subject area). This allowed the study team to determine which one worked best with respondents and yielded the most useful data. In addition to respondents who had either a license or certification being unable to differentiate between these credentials, many respondents of the final round of testing (with the general population) also were unable to differentiate between them. This points to broader issues with the concepts of licenses and certification. Future iterations of the survey should include more exploratory research to determine how these credentials are and are not interrelated.

In Round 1, questions on licenses and certifications were in separate sections of the questionnaire, named “Licenses for Work” and “Certifications for Work”, respectively. Before discussing the issues, the study team observed with these credential questions, it is important to note that in this initial round, and subsequent rounds, the majority of the questions in these sections worked well. These two sections took a different approach to identifying credentials, using a series of questions about the attributes of the credential that would help identify any misreporting, rather than asking a single question to identify the credential. The response errors the study team observed were overwhelmingly due to respondents being unable to map their credentials onto the overall concepts of licenses, certifications, and vocational certificates, and not any confusion about the attributes of their credential.

The study team also observed that credentials are not consistently classified across occupational fields. For example, by definition, at the outset of testing, a credential should be identified as a license if it is legally required to work in a specific occupation and a government agency issues it. However, for some occupations, workers have certifications that are legally required in order to work in the field, and a professional organization issues these certifications, making the legal requirement and issuer not useful for differentiating credentials. During this initial round of testing, it became clear that different terms were used to refer to these types of credentials, and the methods in which people obtained them differed greatly. In general, respondents understood what they had obtained and how they got it, they just did not understand how to initially identify that credential on the survey. Fortunately, the series of questions that ask about the credential attributes worked well, allowing the study team to reclassify credentials in post-collection editing and determine some of the response errors, which are discussed below.

Q37: Professional certifications

This question posed some potential response errors, which was sometimes due to the different situations and requirements of different industries. For example, one respondent initially said that he does have a professional certification, but when probed later, he said, “When I think about it, it’s a certification but it’s not a professional certification. There’s a thin line between a certification where you take a course and you got a certificate or certification for attending the class. But then you have a certification because you need to be licensed by a board or government body to perform the job.” When asked if he would change his response, he said that he would change his answer because his certifications would not count in this category. It was unclear how his credential should be counted.

Other respondents double counted their credential, which may not have been a response error, given the requirements of their industry. For example, one respondent had reported that she had both a certification for work and a vocational certificate. When asked to describe her vocational certificate, she said, “It’s basically the certificate so that I can take the MRI boards from the training program. I said yes because I got a certificate which allowed me to take the certifying boards.” The respondent received a vocational certificate for completing a training program, and that certificate allowed her to take a certifying exam that was required to work in nuclear medicine. Another respondent reported the same credential for as a license and a certification and asked if respondents can have a license and certification. In this case, she double counted the credential because the terms that the issuing body used were changing. She said, “They were in the works of changing from calling it a certification to a license, so that’s why I get confused.”

Additionally, respondents were underreporting the number of credentials they had. For example, two respondents in Round 1 testing did not report certifications that should have been included in the certifications section. One respondent indicated that he did not report his workplace-safety certifications because he did not need them prior to being hired and they were not required by law to work at his job. Another respondent said that although his employer required that he obtain multiple certifications in order to work at his job, he did not report them because he felt they were not the kind of certifications that were relevant to this survey.

Q38: Number of professional certifications

The study team also observed confusion among respondents who held the same license in multiple states when they answered the question about how many currently active professional certificates they held. The question instruction indicated that the same license in multiple states should only count as one license. However, that was not how respondents thought of their licenses. For example, one respondent, who had the same license in four states, asked, “Should I count each state license that I have? I’m gonna put that. Because I can’t use my Georgia license in South Carolina.”

Q41: Primary subject area or field of study

Respondents also struggled with Question 41. The phrase “subject area or field of study” did not resonate with respondents. As a result, they were providing generic information, such as their job industry, rather than a specific field work.

All other questions in this section performed as intended.

After the first round of testing, the study team made several revisions to questionnaire to address the response errors were observed. The study team combined the Licenses for Work section and Certifications for Work section, borrowing wording for a modified screener question from the NSCG that asked about licenses and certifications in a single question, eliminating the need for respondents to identify which type of credential they held. Additionally, the questions were modified to ask about the most important credential, with the follow-up questions focusing on that particular credential. Respondents with more than one credential also were asked to identify and answer questions about an additional credential. In addition, the following changes were made:

Added a new question at the end of the “most important” credential series to determine if respondents had more than one credential and to direct them to the appropriate skip path.
Added an instruction to the question on number of licenses: “If you have the same license in more than one state, count each license separately.”
Added an open-ended question after the Licenses and Certifications section to capture the reasons why people pursue a credential: “Thinking back to when you first decided to pursue any of your licenses, professional certifications, or vocational certificates, why did you feel this was important for your career goals?”
Changed the question on field of study to read, “What is the area of specialization of this license of professional certification?” This revision to the question aimed to prompt respondents to provide additional or new information, rather than a generic job title.

2. Findings from Round 2

Combining the Licenses for Work section and Certifications for Work section into one section reduced respondent confusion and the misreporting of credentials. However, some respondents struggled to provide an accurate count of the number of licenses and certifications they had earned, which may be due to what the study team observed in Round 1, where some respondents were unsure of what would be relevant for this survey.

Q39: Area of specialization

The study team still observed issues with the question intended to collect the area of specialty for the credential (reasons for this change are discussed in the Findings from Round 1 section). Although respondents struggled less with this version of the question compared to the question on field of study from Round ,1 the responses still did not provide much useful information. For example, when asked what “area of specialization” means to them, respondents provided the following answers:

“The area that I am working in the most or that I’m certifying that I’m knowledgeable in”
“Special expertise or training in that field”
“Maybe you’re in business but your specialization is marketing. Or you’re a doctor but you’re an anesthesiologist.”

Q48: Financial support

The study team monitored this question to understand what respondents were thinking of when they reported having received loans. Although most respondents understood the question, one respondent in Round 2 chose the “loans” response option to indicate receiving loans from a friend, rather than choosing the “money from spouse, partner, or family member” response option.

All other questions in this section performed as intended.

The team made a few revisions to this section following Round 2 testing.

The question “What is the area of specialization of this license of professional certification?” was replaced with a new version aimed to get at the focus of the credential: “What kind of work does this license or certification allow you to do?”
To ensure that respondents did not include loans from friends or family in the “loans” response option, the study team revised the “Money from a spouse, partner, or family member” response option to read “Money from spouse, partner, family member, or friend (including loans).” The study team also revised the response option “loans” to say “Loans from the Federal government or private lenders” to further distinguish between the two response options.

3. Findings from Round 3

In general, few new issues arose in this section during Round 3 testing. However, there continued to be confusion about how to report credentials among respondents.

Q36: Having a License or professional certification

Although many respondents understood this question, some respondents were unsure of how to answer. For example, one respondent answered that he did not have a license or professional certification but during probing revealed that he did have a state license. When probed about his response, he said “State or industry licenses was a little tricky because I do have a license to do what I do, but I don’t know if it would fall in this category. I read the verbiage underneath and I didn’t qualify for any of those examples.”

Q63: Decision to pursue credential

In this round, the study team added this question to try to better understand people’s initial motivations for pursuing their credential in general. Respondents tended to provide two types of responses: (1) that it was required for their work or (2) a narrative for how they decided to work in a given field. For example, one respondent said, “I wasn’t thinking of anything besides that it’s required.” Another respondent said, “It wasn’t that I wanted a career as a teacher. I wanted something to provide for my family.” Responses were usually repetitive information that respondents had shared for previous questions. Because of the repetition of information, this question was dropped in the subsequent round.

All other questions in this section performed as intended.

The study team did not make any major revisions to this section of the instrument between Rounds 3 and 4 of testing. The question “Why did you choose to get your license or certification in this particular specialty?” was revised to read “Why did you choose to get your license or certification in this particular field of study?” Response options were drawn from the responses to the probing and survey questions gathered in this question during Rounds 2 and 3, making the new question closed-ended.

4. Findings from Round 4

No major or new issues emerged in the Licenses and Certifications section during Round 4 testing. Like other sections of the survey, in this round, these questions were relevant to only a few respondents, as most people had neither licenses nor certifications. Only 7 of the 17 respondents in Round 4 had a currently active license or certification.

Q42: Having a license or professional certification

Similar to probing in the Vocational Certificates section, respondents were asked to explain the difference between licenses and certification, regardless of if they held either credential. Many respondents struggled to differentiate between licenses and certifications. For example, one respondent said, “For professional certificate, now that I’m thinking of it, it’s similar but not similar. For a certificate you would have to earn it. For a license you would still have to earn it but in a different way, if that makes sense.” Another respondent said a license is when “you’re cleared, and you’ve been trained and approved. Kind of the same thing with a certification. I think they mean the same thing.”

Although target population respondents are largely able to categorize their credential correctly with this final revised survey questionnaire, some additional research would be helpful to determine how the general population understands these credentials to help understand the survey data.

All other questions in this section performed as intended.

D. Work Experience Programs

Summary of findings for Work Experience Programs questions

In general, respondents understood and were able to correctly answer whether they had completed a work experience program (WEP). Many respondents cited the terms in the question language as helpful for determining if they had one of these programs. For respondents who reported having participated in a WEP, some had difficulty responding to the questions that asked where they completed their WEP. Some respondents also had difficulty interpreting the term “formal training” or understanding the difference between the roles of coworkers and mentors in that formal training. Overall, limited changes were made to this section given stakeholder interest in maintaining the comparability of these questions to other existing surveys.

1. Findings from Round 1

Because the first round of testing would include the most probing, the study team sought to reduce the number of questions that would be tested at one time. In Round 1, there were two versions of the questionnaire. One version contained the Vocational Certificates section, but not the Work Experience Programs section. If respondents reported having a vocational certificate during eligibility screening, they received the Vocational Certificates version of the questionnaire. For Round 1, only one respondent received the WEP questionnaire, who had no issues with these questions. No changes were made between rounds.

2. Findings from Round 2

In this round, there was some confusion among respondents about what is considered a WEP.

Q59: Participating in a work experience program

Respondents generally understood concept of a “work experience program” concept, and most respondents were able to correctly identify whether they had completed a WEP. However, some confusion about what would be considered a WEP emerged. For example, a few teachers were not sure if their experience as a student teacher qualified because student teaching was not included in examples in the question. Alternatively, some respondents easily identified that they completed a WEP because the question text included their program name (for example, “internship”).

Q65: Work experience program description

Respondents struggled with this question. The response options included “It was part of a high school program,” “It was part of a program after high school,” and “It was not part of a formal education program.” One respondent said, “I don’t know. It seems like my work experience doesn’t relate to high school. I use basic skills you learn in high school, but I don’t really think about it like that.” Respondents who worked in education tended to misreport that their WEP was “part of a high school program” because they worked in a high school setting. One respondent struggled with the phrase “formal education program,” understanding it as “a university or college program, as opposed to a vocational program.”

Q67: Formal training from a coworker or mentor

Some respondents did not differentiate between coworkers and mentors, seeing them as the same because their coworkers also acted as mentors. One respondent said, “If the intent is to know if you worked closely with people and learned things, I would just say mentor. Coworkers could be mentors if they’re senior to you.” Another respondent echoed this thought, saying, “I was thinking about a coworker. I didn’t consider the people I worked with at the program as coworkers, because I wasn’t employed. But I considered them mentors.” For some respondents, the phrase “formal training” implied something more structured and from more-senior colleagues, resulting in the underreporting of training.

All other questions performed as intended.

Following testing for this round, the team made one minor revision. For the question “Have you ever completed an internship, apprenticeship, clerkship, externship, residency, clinical experience, or similar program?,” the study team added “student teaching” as an example and modified the last phrase to say “similar work experience program.”

3. Findings from Round 3

Few issues arose with the WEP section in Round 3. Most respondents understood the series of questions and answered them without difficulty.

Q70: Work experience program description

There was still confusion with this question. One respondent noted, “I was a little confused. One was standing out to me, so I could only do one. And best described. It wasn’t part of high school. It was not part of a high school program after high school, so I checked number 3. That it was not part of a formal education program, meaning K–12.” This respondent had completed the WEP during college but interpreted “formal education program” as meaning a program through the K-12 education system. It was unclear how respondents should classify a WEP completed during college, because formal education program is not clearly defined.

All other questions in this section performed as intended.

4. Findings from Round 4

Similar to the previous round of testing, most respondents were able to identify whether they had completed a WEP.

Q69: Work experience program

Some respondents were unsure of how to interpret this question. They either reported that they had not completed a WEP, despite probing revealing that they had, or possibly misreported a job as a WEP when it was not. For example, one respondent completed a teaching induction program but said, “I wasn’t sure if an induction program fit into there. I never had to do any technical internships or student teaching.” In this program, the respondent was able to teach with a degree and enrollment in the induction program, despite not yet having a license. It is likely the respondent incorrectly responded no and that this experience should be counted as a WEP. Another respondent also likely misunderstood the intent of the question by counting a job as a WEP because it was paid through a federal work study and she considered it a “huge learning opportunity.”

Q75: Work experience program description

Four respondents reported being unsure whether to include the WEP they completed during college. One respondent said that her WEP was not required in order to meet the requirements for her college degree, so she did not consider it as part of a “formal education program.” Three respondents said that they looked at the subsequent questions and saw that the next question referenced college. With this insight, they decided that their program was “part of a school program after high school,” rather than “not part of a formal education program.” One respondent said, “I looked at the other questions below to see if I was choosing correctly.”

Q77: Formal training from a coworker or mentor

Respondents again varied in how they interpreted the phrase “formal training.” Interpretations included a structured program with objectives and a time frame, a “reusable training…from an outside person,” a program where a mentor approves of your qualifications at the end of the program, or a set time frame where people show you how tasks and the company work.

All other questions in this section performed as intended. The study team did not make any additional changes to the WEP section given stakeholder interest in maintaining the comparability of these questions to other existing surveys; researchers should be aware of possible confusion around the questions highlighted in this section.

E. Educational Background

Summary of findings for Educational Background questions

The Educational Background section was only included in Rounds 3 and 4 of testing. Overall, respondents found the questions straightforward. However, some respondents felt the questions did not capture their full educational experience and answered “incorrectly” to give themselves credit for education not yet completed or, in the case of vocational certificate holders, a missing response option.

1. Findings from Round 3

Interviewers observed a few instances of respondents wanting to share more information about their education, beyond their highest degrees.

Q23: Highest degree or level of school

One respondent felt that he needed more detail than this question captured to accurately convey his educational background, noting that the term “highest degree” was confusing. He “didn’t know if that meant your most important degree or opportunity to get highest salary or years put in.” Another respondent held two separate bachelor’s degrees that he obtained at different times, different universities, and in different fields. In his case, answering questions about his “highest degree” was challenging because he had two of the same degree type. Between Rounds 3 and 4, an option for “vocational certificate or diploma” was added to the question to discourage vocational certificate holders from selecting “some college credit, no degree” or “associate’s degree.” In addition, an open-ended question “Why did you choose to get this degree?” was included in testing in Round 4 to determine possible closed-ended response options for the final survey.

All other questions in this section performed as intended.

2. Findings from Round 4

In Round 4, a similar issue arose for the question about highest level of school that was observed in Round 3.

Q28: Highest degree or level of school

Several respondents who were currently enrolled in college and pursuing a degree responded inconsistently, raising potential reliability concerns. Two of these respondents selected “some college credit, no degree,” while another respondent selected “high school diploma.” This respondent explained his choice by saying “I just thought about what I’ve completed education-wise, and I’m in college, but I haven’t completed, I haven’t gotten my AA or graduated from a two-year, and high school is what I have under my belt. I’m not finished with college yet. It just sounds better because I’m not done, and I have high school under my belt.” These respondents appeared to have interpreted the word “completed” differently. Although no changes were made to this question, researchers should be aware of this inconsistency when analyzing these data. Researchers may want to couple these data with data from the Enrollment section to resolve inconsistencies when cleaning the data. For example, those with a high school diploma and currently enrolled in college could be coded as “some college credit, no degree.”

All other questions in this section performed as intended.

F. Vocational Certificates

Summary of findings for Vocational Certificates questions

In general, respondents understood what vocational certificates were, and, in most cases, could identify whether they had one. Some respondents were unsure if their associate’s degree would be classified as a vocational certificate and may have misreported this degree. The main challenge with the Vocational Certificates section was its placement in the survey, rather than problems with the question wording. As a result, the section was moved between rounds. A few other minor issues also appeared during testing of this section, as described below.

1. Findings from Round 1

In the initial round of testing, questions on licenses and certifications (detailed below) came before questions on vocational certificates, which led to confusion over where to count credentials, and in some cases, resulted in misreporting of credentials.

Q51: Vocational Certificates

Many respondents obtained their certifications as a direct result of, or after attending, a vocational training program through which they received a certificate. As a result, they reported the same experiences in both the Licenses and Certifications and the Vocational Certificates sections. Some respondents most likely double counted a credential by reporting it both in the Certifications for Work and Vocational Certificates sections. For example, one respondent reported the same credential in both sections and during probing on the Vocational Certificates section said “This whole section is either redundant or a little vague as compared to the previous section. When you go to a [vocational technical school], they give you a paper or PDF certificate… The piece of paper you get for the course versus an actual earned certification is different and means different things.” For the same question, one respondent with a diploma from a community college incorrectly answered no to this item, as she felt the term “vocational certificate” was too narrowly specific to apply to her diploma.

Q53: Hours of instruction

Some respondents struggled to choose a response when asked “How many hours of instruction did you complete in order to earn your MOST RECENT vocational certificate?” The response options focused on actual hours rather than calendar time, which was included in parentheses. Respondents felt that hours were not an intuitive way to think of their time in a program, unless the program was explicitly described as being a certain number of hours, which was true for a few respondents. One respondent also noted that the response options did not seem sufficient for those who completed their program part time, as each response option referenced “full-time” school weeks or academic years, and many programs can be completed part time over a longer period.

All other questions in this section performed as intended.

The study team made three revisions to this section after Round 1:

This section was moved to come before a newly combined Licenses and Certifications section to keep the three credential sections together and provide clarity for respondents who were unsure of how to report their credentials.
The question “Why did you choose to get a vocational certificate in this particular specialty?” was added to elicit open-ended responses that the study team could use to formulate response options for the subsequent rounds of testing.
The response options for the question about hours of instruction were modified to have the calendar time come first and hours of instruction in parentheses.

2. Findings from Round 2

Round 2 findings were similar to those from Round 1, with most respondents able to correctly identify whether they received a vocational certificate. However, like in Round 1, some respondents were confused as to whether their associate’s degree or time at a community college counted as a vocational certificate.

Q50: Having a Vocational Certificate

When probed on this question, one respondent said, “Well, it’s asking me about the training or certificate, but I got an associate’s degree, is that the same thing?” In a few cases, respondents reported their associate’s degrees. One respondent, who reported an associate’s degree in audio recording, said “I think in general it was asking if I have any sort of degree or anything from a community or technical college.” Several people said that community college stuck out to them, as they had attended a community college. This led some people to report their credential from a community college as a vocational certificate. In some instances, it was unclear whether a credential should have been counted as a vocational certificate, license, certification, or a combination of those credentials.

All other questions in this section performed as intended and no additional revisions were required.

3. Findings from Round 3

Similar issues occurred in Round 3, with respondents double counting a license or certification in this section when they had already included it in the Licenses and Certifications section.

Q28: Having a Vocational Certificate

In a few cases, respondents who counted their credential both in the Vocational Certificate section and the Licenses and Certifications section. For example, one respondent had several professional certifications for his job that he considered as one credential. He reported them in both sections. When probed about his response to this question, he gave the same description of the credential in both sections. When asked how he came up with his answer, the respondent said “I probably shouldn’t have done it as yes. I didn’t go to some place of a school to get it. I wasn’t sure how to answer it.”

At this point, in testing, it was clear that respondents were struggling to distinguish these credentials, and some respondents struggled to map their experiences onto the questions. The question was monitored in the subsequent round.

All other questions in this section performed as intended.

4. Findings from Round 4

The final round of testing focused on the general population rather than the skilled technical workforce. Consequently, this section was not extensively tested during Round 4 because many respondents screened out of these questions. However, the interviewers probed all respondents, regardless of whether they had a vocational certificate, on their understanding of the term “vocational certificate” to assess respondents’ understanding of the concept. As in prior rounds, multiple respondents in Round 4 expressed confusion about what a vocational certificate is and which credentials apply to this section. Responses to probes asking respondents to define a vocational certificate further highlighted both the misunderstanding and misreporting the study team observed in the prior rounds of testing.

Q34: Having a Vocational Certificate

In several cases, respondents were confused or expressed uncertainty on whether to consider their credential a vocational certificate. Respondents who held associate’s degrees either counted, or wondered if they should count, their degree as a vocational certificate. At least one respondent incorrectly assumed his associate’s degree would count as a vocational certificate. In three cases, it was unclear if the respondents’ credentials counted as vocational certificates. For example, one respondent indicated that he had a vocational certificate and explained that it was an associate’s degree combined with a certificate. Another respondent answered no when asked, “Have you earned a vocational certificate or diploma for completing a training program from a community college, technical or trade school, or some other school?” but then also noted that he completed a technical education certificate from a state university. He did not include this because it was not from a community college or technical school, but it is possible this should have counted as a vocational certificate.

Many respondents were unsure of how to define or explain the term “vocational certificate.” In response to the probe “We use ‘vocational certificate’ in this question, what does that mean to you?” respondents provided a wide range of interpretations. Some were unsure, saying “I’m not sure,” “Good question. I have no idea,” or “Truthfully, I’m not sure without looking it up. My guess would be some kind of trade certificate or something.” Others demonstrated a basic understanding of the term, providing the responses “Vocational, to me, means if you’re getting a certificate or a degree, some sort of training for the job you’ll get,” “To me it makes me think of hands-on work like mechanical work, construction work, anything that requires physical energy and hands on,” and “I think of a trade where you have to get a certification or a license, something to allow you to do something like cosmetology.”

Changes were made to the survey to reduce measurement error. However, as illustrated, because these concepts are unclear and difficult to distinguish, the misreporting the study team observed during testing most likely will persist in production.

All other questions in this section performed as intended.

G. Enrollment

Summary of findings for Enrollment questions

The Enrollment section was only included in the final round of testing. Prior to testing, this section was placed toward the end of the survey. However, because respondents were misreporting in-progress degrees in the Educational Background section in earlier rounds, the Enrollment section was placed before that section to reduce potential errors. Respondents did not encounter any issues with this section with the revised ordering.

H. Background

Summary of findings for Background questions

Most of the questions in the Background section had been previously tested in other surveys; therefore, the study team included a limited number of questions in the first three rounds of cognitive testing. The full series of background questions was included in the last round of testing. The primary testing focus for this section was on questions asking how well the respondent needed to speak English and Spanish for their jobs. Overall, no major issues emerged in this section.

1. Findings from Round 1

In Round 1, only two questions were included for testing - “Are you a native English speaker?” and “How well do you need to speak English/Spanish for your main job?” There were minor issues that arose for both questions.

Q60: Native English speaker

Most respondents understood this question to mean if English is their first or primary language or if they grew up speaking English. One respondent was confused by the term “native language,” saying it reminded her of “tribal” or “Indians.”

Q61/62: Necessity of speaking English/Spanish for your main job

Many respondents took their time reading through the response options before selecting a response to make sure they knew what the question was asking, and then selected the first response option (“I need to be able to carry on a conversation”). Respondents who were not currently working also tended to respond to the question about their prior job before finding the last response option, “I am not currently working” option.

As stated above, no other questions were tested in this round. Two revisions were made in this section following Round 1 testing. For the question “How well do you need to speak English/Spanish for your main job?,” the team moved the response option “I am not currently working” from the last response option to the first response option, so that people not currently working would not answer for a job they are not currently working. The second revision was to the question “Are you a native English speaker?” The study team changed this item to “Was English your first language?” based on respondent feedback described above.

2. Findings from Round 2

No major issues arose in the Background section of the survey.

3. Findings from Round 3

No major issues arose in the Background section of the survey.

4. Findings from Round 4

No major issues arose in the Background section of the survey.

This page has been left blank for double-sided copying.

APPENDIX A
NTEWS Survey Revisions by Testing Round

Table A.1. Employment Status questions by testing round

R1 Q#	R2 Q#	R3 Q#	R4 Q#	R1 wording and ordering	R2 wording and ordering	R3 wording and ordering	R4 wording and ordering
1	1	1	1	Are you currently working for pay or profit? If you are temporarily absent from a job because you are on vacation, out sick, on maternity leave, etc., please answer “Yes.” Yes No	Are you currently working for pay or profit? If you are temporarily absent from a job because you are on vacation, out sick, on maternity leave, etc., please answer “Yes.” Yes No	Are you currently working for pay or profit? If you are temporarily absent from a job because you are on vacation, out sick, on maternity leave, etc., please answer “Yes.” Yes No	Are you currently working for pay or profit? If you are temporarily absent from a job because you are on vacation, out sick, on maternity leave, etc., please answer “Yes.” Yes No
2	2	2	2	If No Did you look for work during the last 4 weeks? Yes No	If No Did you look for work during the last 4 weeks? Yes No	If No Did you look for work during the last 4 weeks? Yes No	If No Did you look for work during the last 4 weeks? Yes No
3	3	3	3	What is your MAIN reason for not working? Retired On layoff from job Student Family responsibilities Chronic illness or permanent disability Suitable job not available Do not need or want to work Other—specify	What is your MAIN reason for not working? Retired On layoff from job due to the coronavirus pandemic On layoff from job for reasons unrelated to the coronavirus pandemic Student Family responsibilities due to the coronavirus pandemic (e.g., childcare, eldercare) Family responsibilities unrelated to the coronavirus pandemic Chronic illness or permanent disability Suitable job not available Do not need or want to work Other—specify	What is your MAIN reason for not working? Retired On layoff from job due to the coronavirus pandemic On layoff from job for reasons unrelated to the coronavirus pandemic Student Family responsibilities due to the coronavirus pandemic (e.g., childcare, eldercare) Family responsibilities unrelated to the coronavirus pandemic Chronic illness or permanent disability Suitable job not available Do not need or want to work Other—specify	What is your MAIN reason for not working? Retired On layoff from job due to the coronavirus pandemic On layoff from job for reasons unrelated to the coronavirus pandemic Student Family responsibilities due to the coronavirus pandemic (e.g., childcare, eldercare) Family responsibilities unrelated to the coronavirus pandemic Chronic illness or permanent disability Suitable job not available Do not need or want to work Other—specify
4	4	4	4	Have you ever worked for pay or profit? Yes No	Have you ever worked for pay or profit? Yes No	Have you ever worked for pay or profit? Yes No	Have you ever worked for pay or profit? Yes No
5	5	5	5	If Yes When did you last work for pay or profit? MONTH YEAR	If Yes When did you last work for pay or profit? MONTH YEAR	If Yes When did you last work for pay or profit? MONTH YEAR	If Yes When did you last work for pay or profit? MONTH YEAR

Text in blue indicates text updates between rounds.

Q = question.

Table A.2. Current Employment questions by testing round

R1 Q#	R2 Q#	R3 Q#	R4 Q#	R1 wording and ordering	R2 wording and ordering	R3 wording and ordering	R4 wording and ordering
6	6	6	6	How many jobs do you have? If you are self-employed, count work with multiple customers or clients as one job. NUMBER OF JOBS	How many jobs do you have? If you are self-employed, count work with multiple customers or clients as one job. NUMBER OF JOBS	How many jobs do you have? If you are self-employed, count work with multiple customers or clients as one job. NUMBER OF JOBS	How many jobs do you have? If you are self-employed, count work with multiple customers or clients as one job. NUMBER OF JOBS
7	7	7	N/A	Do you work at a full-time job (a job where you usually work 35 or more hours per week)? Yes No	Do you work at a full-time job (a job where you usually work 35 or more hours per week)? Yes No	Do you work at a full-time job (a job where you usually work 35 or more hours per week)? Yes No
			7 NEW				During the PAST 12 MONTHS, how many hours did you usually work each WEEK across ALL jobs? NUMBER OF HOURS WORKED
8	8	8	8	During the PAST 12 MONTHS (52 weeks), did you work EVERY week? Count paid vacation, paid sick leave, and military service as work. Yes No	During the PAST 12 MONTHS (52 weeks), did you work EVERY week? Count paid vacation, paid sick leave, and military service as work. Yes No	During the PAST 12 MONTHS (52 weeks), did you work EVERY week? Count paid vacation, paid sick leave, and military service as work. Yes No	What is the name of your MAIN job? If you have more than one job, answer for the job that you most identify with. For example, 4th grade teacher, entry level plumber, web developer. NAME OF MAIN JOB
							The next few questions ask about the MAIN job you wrote in Question 8. Please answer for just that job.
9	9	9	9	If No During the PAST 12 MONTHS (52 weeks), how many WEEKS did you work? Include paid time off and weeks when you worked for only a few hours. NUMBER OF WEEKS	If No During the PAST 12 MONTHS (52 weeks), how many WEEKS did you work? Include paid time off and weeks when you worked for only a few hours. NUMBER OF WEEKS	If No During the PAST 12 MONTHS (52 weeks), how many WEEKS did you work? Include paid time off and weeks when you worked for only a few hours. NUMBER OF WEEKS	For the MAIN job you wrote in Question 8, what are your most important activities or duties? For example: Instruct and evaluate students and create lesson plans; assemble and install pipe sections and review building plans for work details; design, create, and modify websites, etc. ACTIVITIES OR DUTIES
10	10	10	10	During the PAST 12 MONTHS, in the WEEKS WORKED, how many hours did you usually work each WEEK? NUMBER OF HOURS WORKED	During the PAST 12 MONTHS, in the WEEKS WORKED, how many hours did you usually work each WEEK? NUMBER OF HOURS WORKED	During the PAST 12 MONTHS, in the WEEKS WORKED, how many hours did you usually work each WEEK? NUMBER OF HOURS WORKED	During the PAST 12 MONTHS (52 weeks), did you work EVERY week at the MAIN job you wrote in Question 8? Count paid vacation, paid sick leave, and military service as work. Yes No
11	11	11	11	What were your earnings from wages, salary, commissions, bonuses, or tips, from all jobs during the PAST 12 MONTHS? Report the amount before deductions for taxes, bonds, dues, or other items. EARNINGS	What were your earnings from wages, salary, commissions, bonuses, or tips, from all jobs during the PAST 12 MONTHS? Report the amount before deductions for taxes, bonds, dues, or other items. EARNINGS	What were your earnings from wages, salary, commissions, bonuses, or tips, from all jobs during the PAST 12 MONTHS? Report the amount before deductions for taxes, bonds, dues, or other items. EARNINGS	If No During the PAST 12 MONTHS (52 weeks), how many WEEKS did you work at the MAIN job you wrote in Question 8? Include paid time off and weeks when you worked for only a few hours. NUMBER OF WEEKS WORKED
12	12	12	12	What is the name of your MAIN job? If you have more than one job, answer for the job that you worked the most hours during the past 12 months. For example, 4th grade teacher, entry level plumber, web developer. NAME OF MAIN JOB	What is the name of your MAIN job? If you have more than one job, answer for the job that you worked the most hours during the past 12 months. For example, 4th grade teacher, entry level plumber, web developer. NAME OF MAIN JOB	What is the name of your MAIN job? If you have more than one job, answer for the job that you worked the most hours during the past 12 months. For example, 4th grade teacher, entry level plumber, web developer. NAME OF MAIN JOB	During the PAST 12 MONTHS, in the WEEKS WORKED, how many hours did you usually work each WEEK at the MAIN job you wrote in Question 8? NUMBER OF HOURS WORKED
				The next few questions ask about the MAIN job you wrote in Question 12. Please answer for just that job.	The next few questions ask about the MAIN job you wrote in Question 12. Please answer for just that job.	The next few questions ask about the MAIN job you wrote in Question 12. Please answer for just that job.
13	13	13	13	For the job you wrote in Question 12, what are your most important activities or duties? For example: instruct and evaluate students and create lesson plans; assemble and install pipe sections and review building plans for work details; design, create, and modify websites, etc. ACTIVITIES OR DUTIES	For the MAIN job you wrote in Question 12, what are your most important activities or duties? For example: instruct and evaluate students and create lesson plans; assemble and install pipe sections and review building plans for work details; design, create, and modify websites, etc. ACTIVITIES OR DUTIES	For the MAIN job you wrote in Question 12, what are your most important activities or duties? For example: instruct and evaluate students and create lesson plans; assemble and install pipe sections and review building plans for work details; design, create, and modify websites, etc. ACTIVITIES OR DUTIES	During the PAST 12 MONTHS, what were your earnings from wages, salary, commissions, bonuses, or tips, from all jobs during the PAST 12 MONTHS for the job you wrote in Question 8? Report the amount before deductions for taxes, bonds, dues, or other items. EARNINGS
14	14	14	14	For the job you wrote in question 12, please rate your satisfaction with this job's… Wages or salary Benefits Job security Opportunities for advancement Physical working conditions	For the MAIN job you wrote in question 12, please rate your satisfaction with this job's… Wages or salary Benefits Job security Opportunities for advancement Physical working conditions	For the MAIN job you wrote in question 12, please rate your satisfaction with this job's… Wages or salary Benefits Job security Opportunities for advancement Physical working conditions	For the MAIN job you wrote in Question 8, please rate your satisfaction with this job's… Wages or salary Benefits Job security Opportunities for advancement Physical working conditions
15	15	15	15	How would your rate your overall satisfaction with this job? Very satisfied Somewhat satisfied Somewhat dissatisfied Very dissatisfied	How would your rate your overall satisfaction with this job? Very satisfied Somewhat satisfied Somewhat dissatisfied Very dissatisfied	How would your rate your overall satisfaction with this job? Very satisfied Somewhat satisfied Somewhat dissatisfied Very dissatisfied	How would your rate your overall satisfaction with this job? Very satisfied Somewhat satisfied Somewhat dissatisfied Very dissatisfied
16	16	16	16	For the job you wrote in Question 12, are you self-employed? Yes No	For the MAIN job you wrote in Question 12, are you self-employed? Yes No	For the MAIN job you wrote in Question 12, are you self-employed? Yes No	For the MAIN job you wrote in Question 8, are you self-employed? Yes No
17	17	17	17	If Yes Which one of the following best describes your self employment? Self-employed in a NON-INCORPORATED business, professional practice, or farm Self-employed in a INCORPORATED business, professional practice, or farm	If Yes Which one of the following best describes your self employment? Self-employed in a NON-INCORPORATED business, professional practice, or farm Self-employed in a INCORPORATED business, professional practice, or farm	If Yes Which one of the following best describes your self employment? Self-employed in a NON-INCORPORATED business, professional practice, or farm Self-employed in a INCORPORATED business, professional practice, or farm	If Yes Which one of the following best describes your self employment? Self-employed in a NON-INCORPORATED business, professional practice, or farm Self-employed in a INCORPORATED business, professional practice, or farm
18	18	18	18	What is your MAIN BUSINESS OR INDUSTRY—that is, what does this employer make or do? Include the main activity, product, or services you provide. For example: Web design, management consulting, childcare, landscaping, realtor, etc. BUSINESS OR INDUSTRY	What is your MAIN BUSINESS OR INDUSTRY—that is, what does your business make or do? Include the main activity, product, or services you provide. For example: Web design, management consulting, childcare, landscaping, realtor, etc. BUSINESS OR INDUSTRY	What is your MAIN BUSINESS OR INDUSTRY—that is, what does your business make or do? Include the main activity, product, or services you provide. For example: Web design, management consulting, childcare, landscaping, realtor, etc. BUSINESS OR INDUSTRY	What is your MAIN BUSINESS OR INDUSTRY—that is, what does your business make or do? Include the main activity, product, or services you provide. For example: Web design, management consulting, childcare, landscaping, etc. BUSINESS OR INDUSTRY
19	19	19	19	Which one of the following best describes your employer at this job? FOR-PROFIT company, organization, or family business NON-PROFIT organization (including tax-exempt and charitable organizations) LOCAL GOVERNMENT (e.g., city, county, school district STATE GOVERNMENT (including state colleges and universities) US MILITARY SERVICE, ACTIVE DUTY OR COMISSIONED CORPS (e.g., USPHS, NOAA) FEDERAL GOVERNMENT (e.g., civilian employee) Other—specify	For the MAIN job you wrote in Question 12, which one of the following best describes your employer at this job? FOR-PROFIT company or organization NON-PROFIT organization (including tax-exempt and charitable organizations) LOCAL GOVERNMENT (e.g., city, county, school district STATE GOVERNMENT (including state colleges and universities) US MILITARY SERVICE, ACTIVE DUTY OR COMISSIONED CORPS (e.g., USPHS, NOAA) FEDERAL GOVERNMENT (e.g., civilian employee) Other—specify	For the MAIN job you wrote in Question 12, which one of the following best describes your employer at this job? FOR-PROFIT company or organization NON-PROFIT organization (including tax-exempt and charitable organizations) LOCAL GOVERNMENT (e.g., city, county, school district STATE GOVERNMENT (including state colleges and universities) US MILITARY SERVICE, ACTIVE DUTY OR COMISSIONED CORPS (e.g., USPHS, NOAA) FEDERAL GOVERNMENT (e.g., civilian employee) Other—specify	For the MAIN job you wrote in Question 8, which one of the following best describes your employer at this job? FOR-PROFIT company or organization NON-PROFIT organization (including tax-exempt and charitable organizations) LOCAL GOVERNMENT (e.g., city, county, school district STATE GOVERNMENT (including state colleges and universities) US MILITARY SERVICE, ACTIVE DUTY OR COMISSIONED CORPS (e.g., USPHS, NOAA) FEDERAL GOVERNMENT (e.g., civilian employee) Other—specify
20	20	20	20	What is your employer's MAIN BUSINESS OR INDUSTRY—that is, what does this employer make or do? Include the main activity, product, or service provided at the location where you work. For example: Elementary school, residential construction, microprocessor chip production. BUSINESS OR INDUSTRY	What is your employer's MAIN BUSINESS OR INDUSTRY—that is, what does this employer make or do? Include the main activity, product, or service provided at the location where you work. For example: Elementary school, residential construction, microprocessor chip production. BUSINESS OR INDUSTRY	What is your employer's MAIN BUSINESS OR INDUSTRY—that is, what does this employer make or do? Include the main activity, product, or service provided at the location where you work. For example: Elementary school, residential construction, microprocessor chip production. BUSINESS OR INDUSTRY	What is your employer's MAIN BUSINESS OR INDUSTRY—that is, what does this employer make or do? Include the main activity, product, or service provided at the location where you work. For example: Elementary school, residential construction, microprocessor chip production. BUSINESS OR INDUSTRY
21	21	21	21	Is this mainly: Manufacturing? Wholesale trade? Retail Trade? Other (agriculture, construction, service, government, etc.)?	Is this mainly: Manufacturing? Wholesale trade? Retail Trade? Other (agriculture, construction, service, government, etc.)?	Is this mainly: Manufacturing? Wholesale trade? Retail Trade? Other (agriculture, construction, service, government, etc.)?	Is this mainly: Manufacturing? Wholesale trade? Retail Trade? Other (agriculture, construction, service, government, etc.)?
22	22	22	22	What is the name and address of the business or employer for the job you wrote in Question 12? Name Department/Division City/Town State/Territory Zip Code	What is the name and address of the business or employer for the job you wrote in Question 12? Name Department/Division City/Town State/Territory Zip Code	What is the name and address of the business or employer for the job you wrote in Question 12? Name Department/Division City/Town State/Territory Zip Code	What is the name and address of the business or employer for the job you wrote in Question 8? Name Department/Division Street Address City/Town State/Territory Zip Code

Text in blue indicates text updates between rounds.

Q = question.

Table A.3. Licenses and Certifications questions by testing round

R1 Q#	R2 Q#	R3 Q#	R4 Q#	R1 wording and ordering	R2 wording and ordering	R3 wording and ordering	R4 wording and ordering
23	N/A	N/A	N/A	Do you have a CURRENTLY ACTIVE state or industry license? For example, teaching license, realtor license, land surveyor license, registered nurse license, certified public accountant license, etc. Do not include vendor's licenses or other licenses to operate a business. Yes No
N/A	23 NEW	36	42		Do you have any CURRENTLY ACTIVE state or industry licenses or professional certifications? For example, teaching license, land surveyor license, a nurse midwife certification, ASE master technician certification, Cisco Networking Professional certification, etc. Do not include vendor's licenses or other licenses to operate a business. Yes No	Do you have any CURRENTLY ACTIVE state or industry licenses or professional certifications? For example, teaching license, land surveyor license, a nurse midwife certification, ASE master technician certification, Cisco Networking Professional certification, etc. Do not include vendor's licenses or other licenses to operate a business. Yes No	Do you have any CURRENTLY ACTIVE state or industry licenses or professional certifications? For example, teaching license, land surveyor license, a nurse midwife certification, ASE master technician certification, Cisco Networking Professional certification, etc. Do not include vendor's licenses or other licenses to operate a business. Yes No
24	24	37	43	If Yes How many CURRENTLY ACTIVE licenses do you have? If you earned a certification in order to get a license, count only the license. NUMBER OF LICENSES	If Yes How many CURRENTLY ACTIVE licenses or professional certifications do you have? If you have the same license in more than one state, count each license separately. If you earned a certification in order to get a license, count the license and certification separately. NUMBER OF LICENSES AND PROFESSIONAL CERTIFICATIONS	If Yes How many CURRENTLY ACTIVE licenses or professional certifications do you have? If you have the same license in more than one state, count each license separately. If you earned a certification in order to get a license, count the license and certification separately. NUMBER OF LICENSES AND PROFESSIONAL CERTIFICATIONS	If Yes How many CURRENTLY ACTIVE licenses or professional certifications do you have? If you have the same license in more than one state, count each license separately. If you earned a certification in order to get a license, count the license and certification separately. NUMBER OF LICENSES AND PROFESSIONAL CERTIFICATIONS
25	N/A	N/A	N/A	Thinking of all the CURRENTLY ACTIVE licenses you have, did you get any of them for work-related reasons? Yes No
26	N/A	N/A	N/A	If Yes What is the name of your MOST RECENT work-related license? Please write out the full name; do not use abbreviations. NAME OF LICENSE
N/A	25 NEW	38	44		If Yes What do you consider to be your MOST IMPORTANT license or professional certification? Please write out the full name. Do not use abbreviations. NAME OF LICENSE OR PROFESSIONAL CERTIFICATION	If Yes What do you consider to be your MOST IMPORTANT license or professional certification? Please write out the full name. Do not use abbreviations. NAME OF LICENSE OR PROFESSIONAL CERTIFICATION	If Yes What do you consider to be your MOST IMPORTANT license or professional certification? Please write out the full name. Do not use abbreviations. NAME OF LICENSE OR PROFESSIONAL CERTIFICATION
27	26	N/A	N/A	What is the primary subject area or field of study for this license? SUBJECT AREA	What is the area of specialization for this license or professional certification? SPECIALIZATION
N/A	N/A	39 NEW	45			What kind of work does this license or professional certification allow you to do?	What kind of work does this license or professional certification allow you to do?
N/A	27 NEW	40	46		Why did you choose to get a license or professional certification in this particular specialty? (open-ended)	Why did you choose to get a license or professional certification in this particular specialty? (open-ended)	Why did you choose to get a license or professional certification in this particular specialty? It was required for the job I was already doing It was required to get a job I wanted It allowed me to do more in the job I was already doing It allowed me to earn more money It was a new or emerging area in my field I was pursuing my passion I was exploring potential interest in a new job or field Someone recommended this field or job to me It was a free or inexpensive opportunity Some other reason—specify
28	N/A	N/A	N/A	If No What is the name of your MOST RECENT license? Please write out the full name; do not use abbreviations. NAME OF LICENSE
N/A	28 NEW	41	47		Did you get your MOST IMPORTANT license or professional certification for work-related reasons? Yes No	Did you get your MOST IMPORTANT license or professional certification for work-related reasons? Yes No	Did you get your MOST IMPORTANT license or professional certification for work-related reasons? Yes No
				The next few questions ask about the license you wrote in Question 26.
29	29	42	48	Were you legally required to have this license in order to work? Yes No	Were you legally required to have your MOST IMPORTANT license or professional certification in order to work? Yes No	Were you legally required to have your MOST IMPORTANT license or professional certification in order to work? Yes No	Were you legally required to have your MOST IMPORTANT license or professional certification in order to work? Yes No
30	30	43	49	Who issued this license? City or county government agency (e.g., County Electrical Licensing Board, County Department of Land Use) State government agency (e.g., State Board of Education or other state board) Federal government agency (e.g., OSHA, FAA) Professional or trade association (e.g., American Culinary Federation, CompTIA) Business or company (e.g., Xerox, 3M Company) Other—specify	Who issued your MOST IMPORTANT license or professional certification? City or county government agency (e.g., County Electrical Licensing Board, County Department of Land Use) State government agency (e.g., State Board of Education or other state board) Federal government agency (e.g., OSHA, FAA) Professional or trade association (e.g., American Culinary Federation, CompTIA) Business or company (e.g., Xerox, 3M Company) Other—specify	Who issued your MOST IMPORTANT license or professional certification? City or county government agency (e.g., County Electrical Licensing Board, County Department of Land Use) State government agency (e.g., State Board of Education or other state board) Federal government agency (e.g., OSHA, FAA) Professional or trade association (e.g., American Culinary Federation, CompTIA) Business or company (e.g., Xerox, 3M Company) Other—specify	Who issued your MOST IMPORTANT license or professional certification? City or county government agency (e.g., County Electrical Licensing Board, County Department of Land Use) State government agency (e.g., State Board of Education or other state board) Federal government agency (e.g., OSHA, FAA) Professional or trade association (e.g., American Culinary Federation, CompTIA) Business or company (e.g., Xerox, 3M Company) Other—specify
31	31	44	50	Does this license need to be renewed? Yes No	Does this license or professional certification need to be renewed? Yes No	Does this license or professional certification need to be renewed? Yes No	Does this license or professional certification need to be renewed? Yes No
32	32	45	51	If Yes How often does it need to be renewed? Yearly Every other year Other—specify	If Yes How often does it need to be renewed? Yearly Every other year Other—specify	If Yes How often does it need to be renewed? Yearly Every other year Other—specify	If Yes How often does it need to be renewed? Yearly Every other year Other—specify
33	33	46	52	In what year did you first get this license? YEAR	In what year did you first get this license or professional certification? YEAR	In what year did you first get this license or professional certification? YEAR	In what year did you first get this license or professional certification? YEAR
34	34	47	53	Do you use this license for your MAIN job you wrote in Question 12? Yes No Does not apply—I am not currently working	Do you use this license or professional certification for your MAIN job you wrote in Question 12? I am not currently working Yes No	Do you use this license or professional certification for your MAIN job you wrote in Question 12? I am not currently working Yes No	Do you use this license or professional certification for your MAIN job you wrote in Question 8? I am not currently working Yes No
35	35	48	54	To pay for this license, did you have any of the following support? My own savings or earnings Employer support or reimbursement Loans, grants, or scholarships from someplace other my employer Support from a professional association Some other support—specify	To obtain your MOST IMPORTANT license or professional certification, did you have any of the following support? My own money Money from a spouse, partner, or family member Financial support or reimbursement from an employer Loans, grants, or scholarships from someplace other my employer Support from a professional association Some other support—specify	To obtain your MOST IMPORTANT license or professional certification, did you have any of the following support? My own money Money from a spouse, partner, or family member (including loans) Employer support or reimbursement Loans (from the Federal government or private lenders) Grants or scholarships from someplace other my employer Support from a professional association Some other support—specify	To obtain your MOST IMPORTANT license or professional certification, did you have any of the following support? My own money Loans from a spouse, partner, or family member Money from a spouse, partner, or family member that does not need to be repaid Financial support or reimbursement from an employer Loans from the government or private lenders Grants or scholarships from someplace other my employer Support from a professional association Some other support—specify
36	36	49	55	To get the skills and knowledge needed for this license, did you take classes from a college, university, technical or trade school, or other school after high school? Yes No	To get the skills and knowledge needed for this license or professional certification, did you take classes from a college, university, technical or trade school, or other school after high school? Yes No	To get the skills and knowledge needed for this license or professional certification, did you take classes from a college, university, technical or trade school, or other school after high school? Yes No	To get the skills and knowledge needed for this license or professional certification, did you take classes from a college, university, technical or trade school, or other school after high school? Yes No
N/A	37 NEW	50	56		Did you report having TWO OR MORE licenses or professional certifications in Question 24? Yes No	Did you report having TWO OR MORE licenses or professional certifications in Question 37? Yes No	Did you report having TWO OR MORE licenses or professional certifications in Question 43? Yes No
					You indicated that you have two or more licenses or professional certification. The next few questions will be about another one of those licenses and professional certifications.	You indicated that you have two or more licenses or professional certification. The next few questions will be about another one of those licenses and professional certifications.	You indicated that you have two or more licenses or professional certification. The next few questions will be about another one of those licenses and professional certifications.
37	N/A	N/A	N/A	Do you have a CURRENTLY ACTIVE professional certification? For example, a nurse midwife certification, ASE master technician certification, or Cisco Networking Professional certification. Yes No
38	N/A	N/A	N/A	If Yes How many CURRENTLY ACTIVE professional certifications do you have? If you earned a certification in order to get a license, count only the certification. NUMBER OF CERTIFICATIONS
39	N/A	N/A	N/A	Thinking of all the CURRENTLY ACTIVE certifications you have, did you get any of them for work-related reasons? Yes No
40	N/A	N/A	N/A	If Yes What is the name of your MOST RECENT work-related certification? Please write out the full name; do not use abbreviations. NAME OF CERTIFICATION
N/A	38 NEW	51	57		If Yes Thinking about all of your CURRENTLY ACTIVE licenses and professional certifications you reported in Question 24, which one to do consider to be your SECOND MOST IMPORTANT? NAME OF LICENSE OR PROFESSIONAL CERTIFICATION	If Yes Thinking about all of your CURRENTLY ACTIVE licenses and professional certifications you reported in Question 37, which one to do consider to be your SECOND MOST IMPORTANT? NAME OF LICENSE OR PROFESSIONAL CERTIFICATION	If Yes Thinking about all of your CURRENTLY ACTIVE licenses and professional certifications you reported in Question 43, which one to do consider to be your SECOND MOST IMPORTANT? NAME OF LICENSE OR PROFESSIONAL CERTIFICATION
41	39	N/A	N/A	What is the primary subject area or field of study for this certification? SUBJECT AREA	What is the area of specialization for this license or professional certification? SPECIALIZATION
N/A	N/A	52 NEW	58			What kind of work does this license or professional certification allow you to do?	What kind of work does this license or professional certification allow you to do?
N/A	40 NEW	53	59		Why did you choose to get a license or professional certification in this particular specialty? (open-ended)	Why did you choose to get a license or professional certification in this particular specialty? (open-ended)	Why did you choose to get a license or professional certification in this particular specialty? It was required for the job I was already doing It was required to get a job I wanted It allowed me to do more in the job I was already doing It allowed me to earn more money It was a new or emerging area in my field I was pursuing my passion I was exploring potential interest in a new job or field Someone recommended this field or job to me It was a free or inexpensive opportunity Some other reason—specify
42	N/A	N/A	N/A	If Yes What is the name of your MOST RECENT certification? Please write out the full name; do not use abbreviations. NAME OF CERTIFICATION
N/A	41 NEW	54	60		Did you get your SECOND MOST IMPORTANT license or professional certification for work-related reasons? Yes No	Did you get your SECOND MOST IMPORTANT license or professional certification for work-related reasons? Yes No	Did you get your SECOND MOST IMPORTANT license or professional certification for work-related reasons? Yes No
				The next few questions ask about the certification you wrote in Question 26.
43	42	55	61	Were you legally required to have this certification in order to work? Yes No	Were you legally required to have your SECOND MOST IMPORTANT license or professional certification in order to work? Yes No	Were you legally required to have your SECOND MOST IMPORTANT license or professional certification in order to work? Yes No	Were you legally required to have your SECOND MOST IMPORTANT license or professional certification in order to work? Yes No
44	43	56	62	Who issued this certification? Professional or trade association (e.g., American Culinary Federation, CompTIA) Business or company (e.g., Xerox, 3M Company) City or county government agency (e.g., County Electrical Licensing Board, County Department of Land Use) State government agency (e.g., State Board of Education or other state board) Federal government agency (e.g., OSHA, FAA) Other—specify	Who issued your SECOND MOST IMPORTANT license or professional certification? City or county government agency (e.g., County Electrical Licensing Board, County Department of Land Use) State government agency (e.g., State Board of Education or other state board) Federal government agency (e.g., OSHA, FAA) Professional or trade association (e.g., American Culinary Federation, CompTIA) Business or company (e.g., Xerox, 3M Company) Other—specify	Who issued your SECOND MOST IMPORTANT license or professional certification? City or county government agency (e.g., County Electrical Licensing Board, County Department of Land Use) State government agency (e.g., State Board of Education or other state board) Federal government agency (e.g., OSHA, FAA) Professional or trade association (e.g., American Culinary Federation, CompTIA) Business or company (e.g., Xerox, 3M Company) Other—specify	Who issued your SECOND MOST IMPORTANT license or professional certification? City or county government agency (e.g., County Electrical Licensing Board, County Department of Land Use) State government agency (e.g., State Board of Education or other state board) Federal government agency (e.g., OSHA, FAA) Professional or trade association (e.g., American Culinary Federation, CompTIA) Business or company (e.g., Xerox, 3M Company) Other—specify
45	44	57	63	Does this certification need to be renewed? Yes No	Does this license or professional certification need to be renewed? Yes No	Does this license or professional certification need to be renewed? Yes No	Does this license or professional certification need to be renewed? Yes No
46	45	58	64	If Yes How often does it need to be renewed? Yearly Every other year Other—specify	If Yes How often does it need to be renewed? Yearly Every two years Other—specify	If Yes How often does it need to be renewed? Yearly Every two years Other—specify	If Yes How often does it need to be renewed? Yearly Every two years Other—specify
47	46	59	65	In what year did you first get this certification? YEAR	In what year did you first get this license or professional certification? YEAR	In what year did you first get this license or professional certification? YEAR	In what year did you first get this license or professional certification? YEAR
48	47	60	66	Do you use this certification for your MAIN job you wrote in Question 12? Yes No Does not apply—I am not currently working	Do you use this license or professional certification for your MAIN job you wrote in Question 12? I am not currently working Yes No	Do you use this license or professional certification for your MAIN job you wrote in Question 12? I am not currently working Yes No	Do you use this license or professional certification for your MAIN job you wrote in Question 8? I am not currently working Yes No
49	48	61	67	To pay for this certification, did you have any of the following support? My own savings or earnings Employer support or reimbursement Loans, grants, or scholarships from someplace other my employer Support from a professional association Some other support—specify	To obtain your SECOND MOST IMPORTANT license or professional certification, did you have any of the following support? My own money Money from a spouse, partner, or family member Employer support or reimbursement Loans, grants, or scholarships from someplace other my employer Support from a professional association Some other support—specify	To obtain your SECOND MOST IMPORTANT license or professional certification, did you have any of the following support? My own money Money from a spouse, partner, or family member (including loans) Employer support or reimbursement Loans (from the Federal government or private lenders) Grants or scholarships from someplace other my employer Support from a professional association Some other support—specify	To obtain your SECOND MOST IMPORTANT license or professional certification, did you have any of the following support? My own money Loans from a spouse, partner, or family member Money from a spouse, partner, or family member that does not need to be repaid Financial support or reimbursement from an employer Loans from the government or private lenders Grants or scholarships from someplace other my employer Support from a professional association Some other support—specify
50	49	62	68	To get the skills and knowledge needed for this certification, did you take classes from a college, university, technical or trade school, or other school after high school? Yes No	To get the skills and knowledge needed for this license or professional certification, did you take classes from a college, university, technical or trade school, or other school after high school? Yes No	To get the skills and knowledge needed for this license or professional certification, did you take classes from a college, university, technical or trade school, or other school after high school? Yes No	To get the skills and knowledge needed for this license or professional certification, did you take classes from a college, university, technical or trade school, or other school after high school? Yes No
N/A	58 NEW	63	N/A		Thinking back to when you FIRST DECIDED to pursue any of your licenses, professional certifications, or vocational certificates, why did you feel this was important for your career goals? (open-ended)	Thinking back to when you FIRST DECIDED to pursue any of your vocational certificates, licenses, or professional certifications, why did you feel this was important for your career goals? (open-ended)

Text in blue indicates text updates between rounds.

Q = question.

Table A.4. Work Experience Programs questions by testing round

R1 Q#	R2 Q#	R3 Q#	R4 Q#	R1 wording and ordering	R2 wording and ordering	R3 wording and ordering	R4 wording and ordering
51	59	64	69	Have you ever COMPLETED an internship, apprenticeship, clerkship, externship, residency, clinical experience, or similar program? Yes, I have completed this type of program No, but I am in one now No, and I am not in one now	Have you ever COMPLETED an internship, apprenticeship, clerkship, externship, residency, clinical experience, or similar program? Yes, I have completed this type of program No, but I am in one now No, and I am not in one now	Have you ever COMPLETED an internship, apprenticeship, clerkship, externship, residency, clinical experience, student teaching, or similar program? Yes, I have completed this type of program No, but I am in one now No, and I am not in one now	Have you ever COMPLETED an internship, apprenticeship, clerkship, externship, residency, clinical experience, student teaching, or similar program? Yes, I have completed this type of program No, but I am in one now No, and I am not in one now
				In the next series of questions, we refer to internships, apprenticeships, and similar programs as “work experience programs.”	In the next series of questions, we refer to internships, apprenticeships, and similar programs as “work experience programs.”	In the next series of questions, we refer to internships, apprenticeships, and similar programs as “work experience programs.”	In the next series of questions, we refer to internships, apprenticeships, and similar programs as “work experience programs.”
52	60	65	70	What type of work was your last completed work experience program for? For example: Administrative assistant, entry-level plumber, etc. TYPE OF WORK	What type of work was your last completed work experience program for? For example: Administrative assistant, entry-level plumber, etc. TYPE OF WORK	What type of work was your last completed work experience program for? For example: Administrative assistant, entry-level plumber, etc. TYPE OF WORK	What type of work was your last completed work experience program for? For example: Administrative assistant, entry-level plumber, etc. TYPE OF WORK
53	61	66	71	How related is your last completed work experience program to your MAIN job you wrote in Question 12? Very related Somewhat related Not related Does not apply—I am not currently working	How related is your last completed work experience program to your MAIN job you wrote in Question 12? I am not currently working Very related Somewhat related Not related	How related is your last completed work experience program to your MAIN job you wrote in Question 12? I am not currently working Very related Somewhat related Not related	How related is your last completed work experience program to your MAIN job you wrote in Question 8? I am not currently working Very related Somewhat related Not related
54	62	67	72	How often do you use the skills or knowledge that you gained during your last work experience program for your MAIN job you wrote in Question 12? Almost all of the time Sometimes Almost never Never	How often do you use the skills or knowledge that you gained during your last work experience program for your MAIN job you wrote in Question 12? Almost all of the time Sometimes Almost never Never	How often do you use the skills or knowledge that you gained during your last work experience program for your MAIN job you wrote in Question 12? Almost all of the time Sometimes Almost never Never	How often do you use the skills or knowledge that you gained during your last work experience program for your MAIN job you wrote in Question 8? Almost all of the time Sometimes Almost never Never
55	63	68	73	In what year did you complete your last work experience program? YEAR	In what year did you complete your last work experience program? YEAR	In what year did you complete your last work experience program? YEAR	In what year did you complete your last work experience program? YEAR
56	64	69	74	What wage did you earn during your last completed work experience program? No wage—I was not paid A wage that increased as I gained skills and experience A set wage that was lower than the wage of a fully qualified worker The same wage as a fully qualified worker	What wage did you earn during your last completed work experience program? I was not paid A wage that increased as I gained skills and experience A set wage that was lower than the wage of a fully qualified worker The same wage as a fully qualified worker	What wage did you earn during your last completed work experience program? I was not paid A wage that increased as I gained skills and experience A set wage that was lower than the wage of a fully qualified worker The same wage as a fully qualified worker	What wage did you earn during your last completed work experience program? I was not paid A wage that increased as I gained skills and experience A set wage that was lower than the wage of a fully qualified worker The same wage as a fully qualified worker
57	65	70	75	Which ONE of the following best describes your last work experience program? It was part of a high school program It was part of a school program after high school It was not part of a formal education program	Which ONE of the following best describes your last work experience program? It was part of a high school program It was part of a school program after high school It was not part of a formal education program	Which ONE of the following best describes your last work experience program? It was part of a high school program It was part of a school program after high school It was not part of a formal education program	Which ONE of the following best describes your last work experience program? It was part of a high school program It was part of a school program after high school It was not part of a formal education program
58	66	71	76	Did your last completed work experience program require that you take classes from… A college, university, technical or trade school, or other school after high school? Yes No A company, association, union, or other provider? Yes No	Did your last completed work experience program require that you take classes from… A college, university, technical or trade school, or other school after high school? Yes No A company, association, union, or other provider? Yes No	Did your last completed work experience program require that you take classes from… A college, university, technical or trade school, or other school after high school? Yes No A company, association, union, or other provider? Yes No	Did your last completed work experience program require that you take classes from… A college, university, technical or trade school, or other school after high school? Yes No A company, association, union, or other provider? Yes No
59	67	72	77	As part of your last completed work experience program, did you get formal training from a coworker or mentor? Yes No	As part of your last completed work experience program, did you get formal training from a coworker or mentor? Yes No	As part of your last completed work experience program, did you get formal training from a coworker or mentor? Yes No	As part of your last completed work experience program, did you get formal training from a coworker or mentor? Yes No

Text in blue indicates text updates between rounds.

Q = question.

Table A.5. Educational Background questions by testing round

R1 Q#	R2 Q#	R3 Q#	R4 Q#	R1 wording and ordering	R2 wording and ordering	R3 wording and ordering	R4 wording and ordering
N/A	N/A	23	28			What is the highest degree or level of school that you have COMPLETED? Elementary, middle, or high school, but no high school diploma or alternative high school credential (e.g., GED) High school diploma Alternative high school credential (e.g., GED) Vocational certificate or diploma Some college credit, no degree Associate's degree (e.g., AA, AS) Bachelor's degree (e.g., BA, BS) Master's degree (e.g., MA, MS, MEng, MEd, MSW, MBA) Professional degree beyond a Bachelor's degree (e.g., MD, DDS, DVM, LLB, JD) Doctorate degree (e.g., PhD, EdD)	What is the highest degree or level of school that you have COMPLETED? Elementary, middle, or high school, but no high school diploma or alternative high school credential (e.g., GED) High school diploma Alternative high school credential (e.g., GED) Vocational certificate or diploma Some college credit, no degree Associate's degree (e.g., AA, AS) Bachelor's degree (e.g., BA, BS) Master's degree (e.g., MA, MS, MEng, MEd, MSW, MBA) Professional degree beyond a Bachelor's degree (e.g., MD, DDS, DVM, LLB, JD) Doctorate degree (e.g., PhD, EdD)
N/A	N/A	24	29			If elementary or high school, but no high school diploma or alternative high school credential Are you currently enrolled in high school? Do not include any programs to earn a GED or alternative high school credential. Yes No	If elementary or high school, but no high school diploma or alternative high school credential Are you currently enrolled in high school? Do not include any programs to earn a GED or alternative high school credential. Yes No
N/A	N/A	25	30			In what year did you get your highest degree? YEAR	In what year did you get your highest degree? YEAR
N/A	N/A	26	31			What was the field of study for your highest degree? FIELD OF STUDY	What was the field of study for your highest degree? FIELD OF STUDY
N/A	N/A	27	32			Do you use this highest degree for your MAIN job you wrote in Question 12? I am not currently working Yes No	Do you use this highest degree for your MAIN job you wrote in Question 12? I am not currently working Yes No
N/A	N/A	N/A	33 NEW				Why did you choose to get this degree?

Text in blue indicates text updates between rounds.

Q = question.

Table A.6. Vocational Certificates questions by testing round

R1 Q#	R2 Q#	R3 Q#	R4 Q#	R1 wording and ordering	R2 wording and ordering	R3 wording and ordering	R4 wording and ordering
51	50	28	34	Have you earned a vocational certificate or diploma for completing a training program from a community college, technical or trade school, or some other school? Yes No	Have you earned a vocational certificate or diploma for completing a training program from a community college, technical or trade school, or some other school? Yes No	Have you earned a vocational certificate or diploma for completing a training program from a community college, technical or trade school, or some other school? Yes No	Have you earned a vocational certificate or diploma for completing a training program from a community college, technical or trade school, or some other school? Yes No
52	51	29	35	Who awarded your most recent vocational certificate? If you have more than one, answer for your most recent one. A high school A community or technical college A vocational, trade, or business school Another college or university Someplace else—specify	Who awarded your MOST RECENT vocational certificate? If you have more than one, answer for your most recent one. A high school A community or technical college A vocational, trade, or business school Another college or university Someplace else—specify	Who awarded your MOST RECENT vocational certificate? If you have more than one, answer for your most recent one. A high school A community or technical college A vocational, trade, or business school Another college or university Someplace else—specify	Who awarded your MOST RECENT vocational certificate? If you have more than one, answer for your most recent one. A high school A community or technical college A vocational, trade, or business school Another college or university Someplace else—specify
53	52	30	36	How many hours of instruction did you complete in order to earn your most recent vocational certificate? 50 hours of instruction or less (2 full-time school weeks or less) 51 to 299 hours of instruction (more than 2 weeks but less than 3 months of full-time schooling) 300 to 899 hours of instruction (3 full-time school months, but less than a full-time academic year) 900 or more hours of instruction (1 full-time academic year or more)	How many hours of instruction did you complete in order to earn your MOST RECENT vocational certificate? 2 full-time school weeks or less (50 hours of instruction or less) More than 2 weeks but less than 3 months of full-time schooling (51 to 299 hours of instruction) 3 full-time school months, but less than a full-time academic year (300 to 899 hours of instruction) 1 full-time academic year or more (900 or more hours of instruction)	How many hours of instruction did you complete in order to earn your MOST RECENT vocational certificate? 2 full-time school weeks or less (50 hours of instruction or less) More than 2 weeks but less than 3 months of full-time schooling (51 to 299 hours of instruction) 3 full-time school months, but less than a full-time academic year (300 to 899 hours of instruction) 1 full-time academic year or more (900 or more hours of instruction)	How many hours of instruction did you complete in order to earn your MOST RECENT vocational certificate? 2 full-time school weeks or less (50 hours of instruction or less) More than 2 weeks but less than 3 months of full-time schooling (51 to 299 hours of instruction) 3 full-time school months, but less than a full-time academic year (300 to 899 hours of instruction) 1 full-time academic year or more (900 or more hours of instruction)
54	53	31	37	In what year did you get your most recent vocational certificate? YEAR	In what year did you get your MOST RECENT vocational certificate? YEAR	In what year did you get your MOST RECENT vocational certificate? YEAR	In what year did you get your MOST RECENT vocational certificate? YEAR
55	54	32	38	What was the field of study for your most recent vocational certificate? FIELD OF STUDY	What was the field of study for your MOST RECENT vocational certificate? FIELD OF STUDY	What was the field of study for your MOST RECENT vocational certificate? FIELD OF STUDY	What was the field of study for your MOST RECENT vocational certificate? FIELD OF STUDY
N/A	55 NEW	33	39		Why did you choose to get a vocational certificate in this particular specialty? (open-ended)	Why did you choose to get a vocational certificate in this particular specialty? (open-ended)	Why did you choose to get a vocational certificate in this particular specialty? It was required for the job I was already doing It was required to get a job I wanted It allowed me to do more in the job I was already doing It allowed me to earn more money It was a new or emerging area in my field I was pursuing my passion I was exploring potential interest in a new job or field Someone recommended this field or job to me It was a free or inexpensive opportunity Some other reason—specify
56	56	34	40	Do you use this vocational certificate for your MAIN job you wrote in Question 12? Yes No Does not apply—I am not currently working	Do you use this vocational certificate for your MAIN job you wrote in Question 12? I am not currently working Yes No	Do you use this vocational certificate for your MAIN job you wrote in Question 12? I am not currently working Yes No	Do you use this vocational certificate for your MAIN job you wrote in Question 12? I am not currently working Yes No
57	57	35	41	To pay for this certificate, did you have any of the following support? My own savings or earnings Employer support or reimbursement Loans, grants, or scholarships from someplace other my employer Support from a professional association Some other support—specify	To obtain your MOST RECENT vocational certificate, did you have any of the following support? My own money Money from a spouse, partner, or family member Employer support or reimbursement Loans, grants, or scholarships from someplace other my employer Support from a professional association Some other support—specify	To obtain your MOST RECENT vocational certificate, did you have any of the following support? My own money Money from a spouse, partner, or family member (including loans) Employer support or reimbursement Loans (from the Federal government or private lenders) Grants or scholarships from someplace other my employer Support from a professional association Some other support—specify	To obtain your MOST RECENT vocational certificate, did you have any of the following support? My own money Loans from a spouse, partner, or family member Money from a spouse, partner, or family member that does not need to be repaid Financial support or reimbursement from an employer Loans from the government or private lenders Grants or scholarships from someplace other my employer Support from a professional association Some other support—specify

Text in blue indicates text updates between rounds.

Q = question.

Table A.7. Enrollment questions by testing round

R1 Q#	R2 Q#	R3 Q#	R4 Q#	R1 wording and ordering	R2 wording and ordering	R3 wording and ordering	R4 wording and ordering
N/A	N/A	N/A	23				Are you currently enrolled in or taking courses from a college, university, technical or trade school, or other school after high school? Yes No
N/A	N/A	N/A	24				Are you currently enrolled full time or part time? Full time Part time
N/A	N/A	N/A	25				Are you currently enrolled in a degree program? Do not count certificate or diploma programs. Yes No, but I plan to transfer to a degree program No, and I do not plan to transfer to a degree program
N/A	N/A	N/A	26				What degree are you working toward? Associate's degree (e.g., AA, AS) Bachelor's degree (e.g., BA, BS) Master's degree (e.g., MA, MS, MEng, MEd, MSW, MBA) Professional degree beyond a Bachelor's degree (e.g., MD, DDS, DVM, LLB, JD) Doctorate degree (e.g., PhD, EdD) Other—specify
N/A	N/A	N/A	27				What is your primary field of study? FIELD OF STUDY

Text in blue indicates text updates between rounds.

Q = question.

Table A.8. Background questions by testing round

R1 Q#	R2 Q#	R3 Q#	R4 Q#	R1 wording and ordering	R2 wording and ordering	R3 wording and ordering	R4 wording and ordering
N/A	N/A	N/A	78				Are you… Male? Female?
N/A	N/A	N/A	79				What is your birthdate? MM DD YYYY
N/A	N/A	N/A	80				Are you of Hispanic, Latino, or Spanish origin? Yes No
N/A	N/A	N/A	81				What is your race? American Indian or Alaska Native Specify tribal affiliation(s): Native Hawaiian or other Pacific Islander Asian Black or African American White
N/A	N/A	N/A	82				What is your current marital status? Married Living in a marriage-like relationship Widowed Divorced Separated Never married
58	68	73	83	Are you a native English speaker? Yes No	Was English your first language? Yes No	Was English your first language? Yes No	Was English your first language? Yes No
59	69	74	84	How well do you need to speak English for your MAIN job you wrote in Question 12? I need to be able to easily carry on a conversation I need to be able to reply to or ask simple questions I need to be able to use single words or short phrases I don’t need to speak English for this job Does not apply—I am not currently working	How well do you need to speak English for your MAIN job you wrote in Question 12? I am not currently working I need to be able to easily carry on a conversation I need to be able to reply to or ask simple questions I need to be able to use single words or short phrases I don’t need to speak English for this job	How well do you need to speak English for your MAIN job you wrote in Question 12? I am not currently working I need to be able to easily carry on a conversation I need to be able to reply to or ask simple questions I need to be able to use single words or short phrases I don’t need to speak English for this job	How well do you need to speak English for your MAIN job you wrote in Question 8? I am not currently working I need to be able to easily carry on a conversation I need to be able to reply to or ask simple questions I need to be able to use single words or short phrases I don’t need to speak English for this job
60	70	75	85	How well do you need to speak Spanish for your MAIN job you wrote in Question 12? I need to be able to easily carry on a conversation I need to be able to reply to or ask simple questions I need to be able to use single words or short phrases I don’t need to speak Spanish for this job Does not apply—I am not currently working	How well do you need to speak Spanish for your MAIN job you wrote in Question 12? I am not currently working I need to be able to easily carry on a conversation I need to be able to reply to or ask simple questions I need to be able to use single words or short phrases I don’t need to speak Spanish for this job	How well do you need to speak Spanish for your MAIN job you wrote in Question 12? I am not currently working I need to be able to easily carry on a conversation I need to be able to reply to or ask simple questions I need to be able to use single words or short phrases I don’t need to speak Spanish for this job	How well do you need to speak Spanish for your MAIN job you wrote in Question 8? I am not currently working I need to be able to easily carry on a conversation I need to be able to reply to or ask simple questions I need to be able to use single words or short phrases I don’t need to speak Spanish for this job
N/A	N/A	N/A	86				Have you ever served on active duty in the U.S. Armed Forces, Reserves, or National Guard? Never served in the military Only on active duty for training in the Reserves or National Guard Now on active duty On active duty in the past, but not now
N/A	N/A	N/A	87				Are you currently living in the United States or Puerto Rico, another U.S. territory, or are you living in another country? United States or Puerto Rico Another U.S. territory Another country
N/A	N/A	N/A	88				In what U.S. state, U.S. territory, or foreign country were you born? U.S. STATE, TERRITORY, OR FOREIGN COUNTRY
N/A	N/A	N/A	89				Are you a U.S. Citizen? Yes No
N/A	N/A	N/A	90				If non-U.S. citizen Are you currently a non-U.S. citizen with a... Permanent U.S. Resident Visa (Green Card)? Temporary U.S. Resident Visa?

Text in blue indicates text updates between rounds.

Q = question.

This page has been left blank for double-sided copying.

APPENDIX B
NTEWS Round 1: Licenses and Certifications

Shape1 NATIONAL TRAINING, EDUCATION, AND

WORKFORCE SURVEY

INSTRUCTIONS

People can participate in different types of education and training to get the skills and knowledge needed for a job. Participation in education and training may result in a credential. High school diplomas and college de- grees are types of credentials. Other types of credentials include voca- tional certificates, professional certifications, and licenses. In this survey, we will be asking you questions about your education, your work, and if you have any of these credentials.

Directions for filling it out are provided with each question.

Follow all appropriate skip instructions after marking a box. If no skip instruction is provided, you should continue to the next question.

If you have any questions or concerns, please email us at [email protected] or call us toll-free at 1-888-xxx-xxxx.

Thank you for your help. We really appreciate it.

The information collected in this questionnaire is solicited under the authority of the National Science Foundation (NSF) Act of 1950, as amended. The information collected in this question- naire is solicited under the authority of the National Science Foundation (NSF) Act of 1950, as amended. The U.S. Census Bureau is conducting this survey under the authority of Title 13, Sec- tion 8 of the United States Code. The Census Bureau is required by law to keep your information confidential and can use your responses for statistical purposes only. The Census Bureau is not permitted to publicly release your responses in a way that could identify you. Federal law pro- tects your privacy and keeps your answers confidential (Title 13, United States Code, Section 9). Per the Federal Cybersecurity Enhancement Act of 2015, your data are protected from cyberse- curity risks through screening of the systems that transmit your data. Your response is voluntary, and failure to provide some or all of the requested information will not in any way adversely affect you. Actual time to complete the questionnaire may vary depending on your circumstances but on average, it will take about 15 minutes. lf you have any comments on the time required for this survey, please send them to the Reports Clearance Officer, Office of the General Counsel, National Science Foundation, 2415 Eisenhower Ave., Alexandria, VA 22314.

Shape2 Employment Status

1. Are you currently working for pay or profit?

If you are temporarily absent from a job because you are on vacation, out sick, on maternity leave, etc., please answer "Yes".

1 Yes Go to Question 6

2 No

(If No) Did you look for work during the last 4 weeks?

1 Yes

2 No

What is your MAIN reason for not working?

Mark one answer.

1 Retired

2 On layoff from job

Student

4 Family responsibilities

5 Chronic illness or permanent disability

6 Suitable job not available

7 Do not need or want to work

8 Other – Specify

Have you ever worked for pay or profit?

1 Yes

2 No Go to Question 23

(If Yes) When did you last work for pay or profit?

LAST WORKED

MONTH YEAR

Go to Question 23

Current Employment

How many jobs do you have?

If you are self-employed, count work with multiple customers or clients as one job.

NUMBER OF JOBS

11. What were your earnings from wages, salary, commissions, bonuses, or tips, from all jobs during the PAST 12 MONTHS?

Report the amount before deductions for taxes, bonds, dues, or other items.

, ,

EARNINGS _$

Do you work at a full-time job (a job where you usually work 35 or more hours per week)?

1 Yes

2 No

During the PAST 12 MONTHS (52 weeks), did you work EVERY week?

Count paid vacation, paid sick leave, and military service as work.

1 Yes Go to Question 10

2 No

12. What is the name of your MAIN job?

If you have more than one job, answer for the job that you worked the most hours during the past 12 months.

For example: 4th grade teacher, entry-level plumber, web developer.

NAME OF MAIN JOB

The next few questions ask about the MAIN job you wrote in Question 12. Please answer for just that job.

9. (If No) During the PAST 2 MONTHS (52 weeks), how many WEEKS did you work?

Include paid time off and include weeks when you worked for only a few hours.

NUMBER OF WEEKS

10. During the PAST 12 MONTHS, in the WEEKS WORKED, how many hours did you usually work each WEEK?

NUMBER OF HOURS WORKED

13. For the job you wrote in Question 12, what are your most important activities or duties?

For example: instruct and evaluate students and create lesson plans; assemble and install pipe sections and review building plans for work details; design, create, and modify websites.

ACTIVITIES OR DUTIES

14. For the job you wrote in Question 12, please rate your satisfaction with this job's.

Mark one answer for each item.

Shape4 Shape5 Very Somewhat Somewhat Very Does not satisfied satisfied dissatisfied dissatisfied apply

a Wage or salary...1	2	4	5
b Benefits 1	2	4	5
c Job security 1	²	4	5
d Opportunities for Advancement 1	2	4	5
e Physical working Conditions 1	²	4	5

Shape7 Shape8 15. How would your rate your overall satisfaction with this job?

Mark one answer.

1 Very satisfied

2 Somewhat satisfied

Somewhat dissatisfied

4 Very dissatisfied

7. (If Yes) Which one of the following best describes your self-employment?

Mark one answer.

1 Self-employed in a NON-INCORPORATED business, professional practice, or farm

2 Self-employed in an INCORPORATED business, professional practice, or farm

8. What is your MAIN BUSINESS OR INDUSTRY—

that is, what does this employer make or do?

Include the main activity, product, or services you provide.

For example: Web design, Management Consulting, Childcare, Landscaping, Realtor, etc.

BUSINESS OR INDUSTRY

Shape9 16. For the job you wrote in Question 2, are you self-employed?

1 Yes

2 No Go to Question 19

Go to Question 21

19. For the job you wrote in Question 12, which one of the following best describes your employer at this job?

Mark one answer.

1 FOR-PROFIT company, organization, or family business

2 NON-PROFIT organization (including tax- exempt and charitable organizations)

LOCAL GOVERNMENT (e.g., city, county, school district)

4 STATE GOVERNMENT (including state colleges and universities)

5 US MILITARY SERVICE, ACTIVE DUTY OR COMISSIONED CORPS (e.g., USPHS, NOAA)

6 FEDERAL GOVERNMENT (e.g., civilian employee)

7 Other – Specify

21. Is this mainly

Mark one answer.

1 Manufacturing?

2 Wholesale trade?

Retail trade?

4 Other (agriculture, construction, service, government, etc.)?

What is the name and address of the business or employer for the job you wrote in Question 12?

If you have more than one job, report your MAIN job.

If your employer has more than one location, report the location where you work.

If you work for a contracting or consulting company, report the name of that company, not the client organization.

Shape10

Name

Department/Division

City/Town

State/Territory

Zip Code

20. What is your employer's MAIN BUSINESS OR INDUSTRY—that is, what does this employer make or do?

Include the main activity, product, or service provided at the location where you work.

For example: Elementary school, residential construction, microprocessor chip production.

BUSINESS OR INDUSTRY

Licenses for Work

Do you have a CURRENTLY ACTIVE state or industry license?

For example, teaching license, realtor license, land surveyor license, registered nurse license, certified public accountant license, etc. Do not include vendor's licenses or other licenses to operate a business.

1 Yes

2 No Go to Question 37

(If Yes) How many CURRENTLY ACTIVE licenses do you have?

If you earned a certification in order to get a license, count only the license.

NUMBER OF LICENSES

What is the primary subject area or field of study for this license?

SUBJECT AREA

Go to Question 29

(If No) What is the name of your MOST RECENT license?

Please write out the full name of the license. Do not use abbreviations.

NAME OF LICENSE

Thinking of all the CURRENTLY ACTIVE licenses you have, did you get any of them for work-related reasons?

Shape11 1 Yes

2 No Go to Question 28

(If Yes) What is the name of your MOST RECENT work-related license?

Please write out the full name of the license. Do not use abbreviations.

NAME OF LICENSE

Go to Question 37

The next few questions ask about the license you wrote in Question 26.

2 . Were you legally required to have this license in order to work?

1 Yes

2 No

30. Who issued this license?

Mark one answer.

1 City or county government agency

(e.g., County Electrical Licensing Board, County Department of Land Use)

2 State government agency

(e.g., State Board of Education or other state board)

Federal government agency (e.g., OSHA, FAA)

4 Professional or trade association

(e.g., American Culinary Federation, CompTIA)

5 Business or company (e.g., Xerox, M Company)

6 Other – Specify

In what year did you first get this license?

YEAR

Do you use this license for your MAIN job you wrote in Question 12?

Mark one answer.

1 Yes

2 No

Does not apply—I am not currently working

31. Does this license need to be renewed?

1 Yes

2 No Go to Question

32. (If Yes) How often does it need to be renewed?

Mark one answer.

1 Yearly

2 Every other year

Other – Specify

To pay for this license, did you have any of the following support?

Mark one answer for each item.

Yes No

My own savings or earnings ...... 2

Employer support or reimbursement..... 2

Loans, grants, or scholarships from

Someplace other my employer .... 2

Support from a professional

association ............. 2

Some other support – Specify ....... 2

Shape12

To get the skills and knowledge needed for this license, did you take classes from a college, university, technical or trade school, or other school after high school?

1 Yes

2 No

Certifications for Work

Do you have a CURRENTLY ACTIVE professional certification?

For example, a nurse midwife certification, ASE master technician certification, or Cisco Networking Professional certification.

1 Yes

2 No Go to Question 51

(If Yes) How many CURRENTLY ACTIVE professional certifications do you have?

If you earned a certification in order to get a license, count only the certification.

NUMBER OF CERTIFICATIONS

41. What is the primary subject area or field of study for this certification?

SUBEJCT AREA

Go to Question 43

42. (If No) What is the name of your MOST RECENT certification?

Please write out the full name. Do not use abbreviations.

NAME OF CERTIFICATION

39. Thinking of all the CURRENTLY ACTIVE certifications you have, did you get any of them for work-related reasons?

1 Yes

2 No Go to Question 42

Go to Question 51

The next few questions ask about the certification you wrote in Question 40.
(If Yes) What is the name of your MOST RECENT work-related certification?

Please write out the full name. Do not use abbreviations.

Shape13 NAME OF CERTIFICATION

Were you legally required to have this certification in order to work?

1 Yes

2 No

Who issued this certification?

Mark one answer.

1 Professional or trade association

(e.g., American Culinary Federation, CompTIA)

2 Business or company (e.g., Xerox, M Company)

City or county government agency

(e.g., County Electrical Licensing Board, County Department of Land Use)

4 State government agency (e.g., State Board of Education or other state board)

5 Federal government agency (e.g., OSHA, FAA)

6 Other – Specify

In what year did you first get this certification?

YEAR

Do you use this certification for your MAIN job you wrote in Question 12?

Mark one answer.

1 Yes

2 No

Does not apply—I am not currently working

Does this certification need to be renewed?

1 Yes

2 No Go to Question 47

49. To pay for this certification, did you have any of the following support?

Mark one answer for each item.

Yes No

My own savings or earnings ...... 2

Employer support or reimbursement..... 2

Loans, grants, or scholarships from

Someplace other my employer .... 2

Support from a professional

association ............. 2

Some other support – Specify ....... 2

(If Yes) How often does it need to be renewed?

Mark one answer.

Shape14

1 Yearly

2 Every other year

Other – Specify

50. To get the skills and knowledge needed for this certification, did you take classes from a college, university, technical or trade school, or other school after high school?

1 Yes

2 No

Work Experience Programs

51. Have you ever COMPLETED an internship, apprenticeship, clerkship, externship, residency, clinical experience, or similar program?

1 Yes, I have completed this type of program

How often do you use the skills or knowledge that you gained during your last work experience program for your MAIN job you wrote in Question 12?

Mark one answer.

1 All or most of the time

2 Sometimes

2 No, but I am in one now

No, and I am not in one now

Go to Question 60

Almost never

4 Never

In the next series of questions, we refer to internships, apprenticeships, and similar programs as "work experience programs."
In what year did you complete your last work experience program?

YEAR

What type of work was your last completed work experience program for?

For example: administrative assistant, entry-level plumber, etc.

TYPE OF WORK

How related is your last completed work experience program to your MAIN job you wrote in Question 2?

Mark one answer.

1 Very related

2 Somewhat related

What wage did you earn during your last completed work experience program?

Mark one answer.

1 No wage—I was not paid

2 A wage that increased as I gained skills and experience

A set wage that was lower than the wage of a fully qualified worker

4 The same wage as a fully qualified worker

Which ONE of the following best describes your last work experience program?

Mark one answer.

1 It was part of a high

Shape15 Not related

4 Does not apply—I am not

currently working Go to Question 55

school program

2 It was part of a school program after high school

It was not part of a formal education program

Go to Question 60

Did your last completed work experience program require that you take classes from.

Mark one answer for each item.

Background

For the last few questions, we are interested in the languages people may speak for their jobs.

A college, university, technical or trade school, or other school

Yes No

Shape16 after high school?......... 2

A company, association, union,

or other provider?........................... 2

59. As part of your last completed work experience

program, did you get formal training from a coworker or mentor?

1 Yes

2 No

60. Are you a native English speaker?

1 Yes

2 No

61. How well do you need to speak English for your MAIN job you wrote in Question 12?

Mark one answer.

1 I need to be able to easily carry on a conversation

2 I need to be able to reply to or ask simple questions

I need to be able to use single words or short phrases

4 I do not need to speak English for this job

5 Does not apply—I am not currently working

62. How well do you need to speak Spanish for your MAIN job you wrote in Question 12?

Mark one answer.

1 I need to be able to easily carry on a conversation

2 I need to be able to reply to or ask simple questions

I need to be able to use single words or short phrases

4 I do not need to speak Spanish for this job

5 Does not apply—I am not currently working

Reset Form

Shape17

Thank you for your participation in the National Training,

Education, and Workforce Survey!

Please return this questionnaire in the postage-paid envelope provided. If you have lost the envelope, mail the completed questionnaire to

U.S. Census Bureau

1201 E. 10^th St.

Jeffersonville, IN 47132-0001

If you have any questions or need assistance, please email us at xxxx@xxxx.gov or call us toll-free at 1-888-xxx-xxxx.

Results of the National Training, Education, and Workforce Survey will be released next year and can be found on the National Science Foundation's website at

http //www.nsf.gov/statistics.

You are not required to respond to any information collection unless it displays a valid approval number from the Office of Management and Budget. The approval number for this survey is 3145-XXXX.

APPENDIX C
NTEWS Round 1: Vocational Certificates

Shape18 NATIONAL TRAINING, EDUCATION, AND

WORKFORCE SURVEY

INSTRUCTIONS

Directions for filling it out are provided with each question.

Follow all appropriate skip instructions after marking a box. If no skip instruction is provided, you should continue to the next question.

If you have any questions or concerns, please email us at [email protected] or call us toll-free at 1-888-xxx-xxxx.

Thank you for your help. We really appreciate it.

Shape19 Employment Status

1. Are you currently working for pay or profit?

If you are temporarily absent from a job because you are on vacation, out sick, on maternity leave, etc., please answer "Yes".

1 Yes Go to Question 6

2 No

(If No) Did you look for work during the last 4 weeks?

1 Yes

2 No

What is your MAIN reason for not working?

Mark one answer.

1 Retired

2 On layoff from job

Student

4 Family responsibilities

5 Chronic illness or permanent disability

6 Suitable job not available

7 Do not need or want to work

8 Other – Specify

Have you ever worked for pay or profit?

1 Yes

2 No Go to Question 23

(If Yes) When did you last work for pay or profit?

LAST WORKED

MONTH YEAR

Go to Question 23

Current Employment

How many jobs do you have?

If you are self-employed, count work with multiple customers or clients as one job.

NUMBER OF JOBS

11. What were your earnings from wages, salary, commissions, bonuses, or tips, from all jobs during the PAST 12 MONTHS?

Report the amount before deductions for taxes, bonds, dues, or other items.

, ,

EARNINGS _$

Do you work at a full-time job (a job where you usually work 35 or more hours per week)?

1 Yes

2 No

During the PAST 12 MONTHS (52 weeks), did you work EVERY week?

Count paid vacation, paid sick leave, and military service as work.

1 Yes Go to Question 10

2 No

12. What is the name of your MAIN job?

If you have more than one job, answer for the job that you worked the most hours during the past 12 months.

For example: 4th grade teacher, entry-level plumber, web developer.

NAME OF MAIN JOB

The next few questions ask about the MAIN job you wrote in Question 12. Please answer for just that job.

9. (If No) During the PAST 12 MONTHS (52 weeks), how many WEEKS did you work?

Include paid time off and include weeks when you worked for only a few hours.

NUMBER OF WEEKS

10. During the PAST 12 MONTHS, in the WEEKS WORKED, how many hours did you usually work each WEEK?

NUMBER OF HOURS WORKED

13. For the job you wrote in Question12 , what are your most important activities or duties?

For example: instruct and evaluate students and create lesson plans; assemble and install pipe sections and review building plans for work details; design, create, and modify websites.

ACTIVITIES OR DUTIES

14. For the job you wrote in Question 12, please rate your satisfaction with this job's.

Mark one answer for each item.

Shape21 Shape22 Very Somewhat Somewhat Very Does not satisfied satisfied dissatisfied dissatisfied apply

a Wage or salary...1	2	4	5
b Benefits 1	2	4	5
c Job security 1	²	4	5
d Opportunities for Advancement 1	2	4	5
e Physical working Conditions 1	²	4	5

Shape24 Shape25 15. How would your rate your overall satisfaction with this job?

Mark one answer.

1 Very satisfied

2 Somewhat satisfied

Somewhat dissatisfied

4 Very dissatisfied

17. (If Yes) Which one of the following best describes your self-employment?

Mark one answer.

1 Self-employed in a NON-INCORPORATED business, professional practice, or farm

2 Self-employed in an INCORPORATED business, professional practice, or farm

18. What is your MAIN BUSINESS OR INDUSTRY—

that is, what does this employer make or do?

Include the main activity, product, or services you provide.

For example: Web design, Management Consulting, Childcare, Landscaping, Realtor, etc.

BUSINESS OR INDUSTRY

Shape26 16. For the job you wrote in Question 12, are you self- employed?

1 Yes

2 No Go to Question 19

Go to Question 21

19. For the job you wrote in Question 12, which one of the following best describes your employer at this job?

Mark one answer.

1 FOR-PROFIT company, organization, or family business

2 NON-PROFIT organization (including tax- exempt and charitable organizations)

LOCAL GOVERNMENT (e.g., city, county, school district)

4 STATE GOVERNMENT (including state colleges and universities)

5 US MILITARY SERVICE, ACTIVE DUTY OR COMISSIONED CORPS (e.g., USPHS, NOAA)

6 FEDERAL GOVERNMENT (e.g., civilian employee)

7 Other – Specify

21. Is this mainly

Mark one answer.

1 Manufacturing?

2 Wholesale trade?

Retail trade?

4 Other (agriculture, construction, service, government, etc.)?

What is the name and address of the business or employer for the job you wrote in Question 12?

If you have more than one job, report your MAIN job.

If your employer has more than one location, report the location where you work.

If you work for a contracting or consulting company, report the name of that company, not the client organization.

Shape27

Name

Department/Division

City/Town

State/Territory

Zip Code

20. What is your employer's MAIN BUSINESS OR INDUSTRY—that is, what does this employer make or do?

Include the main activity, product, or service provided at the location where you work.

For example: Elementary school, residential construction, microprocessor chip production.

BUSINESS OR INDUSTRY

Licenses for Work

Do you have a CURRENTLY ACTIVE state or industry license?

Yes

2 No Go to Question 37

(If Yes) How many CURRENTLY ACTIVE licenses do you have?

If you earned a certification in order to get a license, count only the license.

NUMBER OF LICENSES

What is the primary subject area or field of study for this license?

SUBJECT AREA

Go to Question 29

(If No) What is the name of your MOST RECENT license?

Please write out the full name of the license. Do not use abbreviations.

NAME OF LICENSE

Thinking of all the CURRENTLY ACTIVE licenses you have, did you get any of them for work-related reasons?

Shape28 1 Yes

2 No Go to Question 28

(If Yes) What is the name of your MOST RECENT work-related license?

Please write out the full name of the license. Do not use abbreviations.

NAME OF LICENSE

Go to Question 37

The next few questions ask about the license you wrote in Question 26.

29. Were you legally required to have this license in order to work?

1 Yes

2 No

30. Who issued this license?

Mark one answer.

1 City or county government agency

(e.g., County Electrical Licensing Board, County Department of Land Use)

2 State government agency

(e.g., State Board of Education or other state board)

Federal government agency (e.g., OSHA, FAA)

4 Professional or trade association

(e.g., American Culinary Federation, CompTIA)

5 Business or company (e.g., Xerox, M Company)

6 Other – Specify

In what year did you first get this license?

YEAR

Do you use this license for your MAIN job you wrote in Question 2?

Mark one answer.

1 Yes

2 No

Does not apply—I am not currently working

Shape29

31. Does this license need to be renewed?

1 Yes

2 No Go to Question

32. (If Yes) How often does it need to be renewed?

Mark one answer.

1 Yearly

2 Every other year

Other – Specify

To pay for this license, did you have any of the following support?

Mark one answer for each item.

Yes No

My own savings or earnings ...... 2

Employer support or reimbursement..... 2

Loans, grants, or scholarships from

Someplace other my employer .... 2

Support from a professional

association ............. 2

Some other support – Specify ....... 2

To get the skills and knowledge needed for this license, did you take classes from a college, university, technical or trade school, or other school after high school?

1 Yes

2 No

Certifications for Work

Do you have a CURRENTLY ACTIVE professional certification?

For example, a nurse midwife certification, ASE master technician certification, or Cisco Networking Professional certification.

1 Yes

2 No Go to Question 51

(If Yes) How many CURRENTLY ACTIVE professional certifications do you have?

If you earned a certification in order to get a license, count only the certification.

NUMBER OF CERTIFICATIONS

4 . What is the primary subject area or field of study for this certification?

SUBEJCT AREA

Go to Question 43

42. (If No) What is the name of your MOST RECENT certification?

Please write out the full name. Do not use abbreviations.

NAME OF CERTIFICATION

39. Thinking of all the CURRENTLY ACTIVE certifications you have, did you get any of them for work-related reasons?

1 Yes

2 No Go to Question 42

Go to Question 51

The next few questions ask about the certification you wrote in Question 40.
(If Yes) What is the name of your MOST RECENT work-related certification?

Please write out the full name. Do not use abbreviations.

Shape30 NAME OF CERTIFICATION

Were you legally required to have this certification in order to work?

1 Yes

2 No

Reset Page

Page 10

Who issued this certification?

Mark one answer.

1 Professional or trade association

(e.g., American Culinary Federation, CompTIA)

2 Business or company (e.g., Xerox, M Company)

City or county government agency

(e.g., County Electrical Licensing Board, County Department of Land Use)

4 State government agency (e.g., State Board of Education or other state board)

5 Federal government agency (e.g., OSHA, FAA)

6 Other – Specify

In what year did you first get this certification?

YEAR

Do you use this certification for your MAIN job you wrote in Question 12?

Mark one answer.

1 Yes

2 No

Does not apply—I am not currently working

49. To pay for this certification, did you have any of the following support?

Mark one answer for each item.

Does this certification need to be renewed?

1 Yes

2 No Go to Question 47

Yes No

My own savings or earnings ...... 2

Employer support or reimbursement..... 2

Loans, grants, or scholarships from

Someplace other my employer .... 2

Support from a professional

association ............. 2

Some other support – Specify ....... 2

(If Yes) How often does it need to be renewed?

Mark one answer.

Shape31

1 Yearly

2 Every other year

Other – Specify

50. To get the skills and knowledge needed for this certification, did you take classes from a college, university, technical or trade school, or other school after high school?

1 Yes

2 No

Vocational Certificates

51. Have you earned a vocational certificate or diploma for completing a training program from a community college, technical or trade school, or some other school?

1 Yes

2 No Question 58

52. Where did you earn your most recent vocational certificate?

If you have more than one, answer for your most recent one.

Mark one answer.

1 A high school

2 A community or technical college

A vocational, trade, or business school

4 Another college or university

4 Someplace else – Specify

In what year did you get your most recent vocational certificate?

YEAR

What was the field of study for your most recent vocational certificate?

FIELD OF STUDY

Do you use this vocational certificate for your MAIN job you wrote in Question 12?

Mark one answer.

1 Yes

2 No

Does not apply—I am not currently working

Shape32

53. How many hours of instruction did you complete in order to earn your most recent vocational certificate?

Mark one answer.

1 50 hours of instruction or less (2 full-time school weeks or less)

2 51 to 299 hours of instruction (more than 2 weeks but less than months of full-time schooling)

300 to 899 hours of instruction (full-time school months, but less than a full-time academic year)

900 or more hours of instruction (1 full-time academic year or more)

To pay for this certificate, did you have any of the following support?

Mark one answer for each item.

Yes No

My own savings or earnings ...... 2

Employer support or reimbursement..... 2

Loans, grants, or scholarships from

Someplace other my employer .... 2

Support from a professional

association ............. 2

Some other support – Specify ....... 2

Shape33

Background

For the last few questions, we are interested in the languages that people may speak for their jobs.

Are you a native English speaker?

1 Yes

2 No

59. How well do you need to speak English for your MAIN job you wrote in Question 12?

Mark one answer.

1 I need to be able to easily carry on a conversation

2 I need to be able to reply to or ask simple questions

I need to be able to use single words or short phrases

4 I do not need to speak English for this job

5 Does not apply—I am not currently working

How well do you need to speak Spanish for your MAIN job you wrote in Question 12?

Mark one answer.

1 I need to be able to easily carry on a conversation

2 I need to be able to reply to or ask simple questions

I need to be able to use single words or short phrases

4 I do not need to speak Spanish for this job

5 Does not apply—I am not currently working

Reset Form

Shape34

Thank you for your participation in the National Training,

Education, and Workforce Survey!

Please return this questionnaire in the postage-paid envelope provided. If you have lost the envelope, mail the completed questionnaire to

U.S. Census Bureau

1201 E. 10^th St.

Jeffersonville, IN 47132-0001

If you have any questions or need assistance, please email us at xxxx@xxxx.gov or call us toll-free at 1-888-xxx-xxxx.

Results of the National Training, Education, and Workforce Survey will be released next year and can be found on the National Science Foundation's website at

http //www.nsf.gov/statistics.

APPENDIX D
NTEWS Round 2

Shape35

NATIONAL TRAINING, EDUCATION,

AND WORKFORCE SURVEY

INSTRUCTIONS

People can participate in different types of education and training. Sometimes education and training programs result in a credential. High school diplomas and college degrees are types of credentials. Other types of credentials include vocational certificates, professional certifications, and licenses. In this survey, we will be asking you questions about your education and if you have any of these credentials. Although we will be asking about your work, we want to hear from everyone about their education and credentials even if you are not currently working.

Directions for filling it out are provided with each question.

Follow all appropriate skip instructions after marking a box. If no skip instruction is provided, you should continue to the next question.

If you have any questions or concerns, please email us at [email protected] or call us toll-free at 1-888-xxx-xxxx.

Thank you for your help. We really appreciate it.

The information collected in this questionnaire is solicited under the authority of the National Science Foundation (NSF) Act of 1950, as amended. The information collected in this question- naire is solicited under the authority of the National Science Foundation (NSF) Act of 1950, as amended. The U.S. Census Bureau is conducting this survey under the authority of Title 13, Sec- tion 8 of the United States Code. The Census Bureau is required by law to keep your information confidential and can use your responses for statistical purposes only. The Census Bureau is not permitted to publicly release your responses in a way that could identify you. Federal law pro- tects your privacy and keeps your answers confidential (Title 13, United States Code, Section 9). Per the Federal Cybersecurity Enhancement Act of 2015, your data are protected from cyberse- curity risks through screening of the systems that transmit your data. Your response is voluntary, and failure to provide some or all of the requested information will not in any way adversely affect you. Actual time to complete the questionnaire may vary depending on your circumstances but on average, it will take about 15 minutes. lf you have any comments on the time required for this survey, please send them to the Reports Clearance Officer, Office of the General Counsel, National Science Foundation, 2415 Eisenhower Ave., Alexandria, VA 22314.

Employment Status

Are you currently working for pay or profit?

If you are temporarily absent from a job because you are on vacation, out sick, on maternity leave, etc., please answer "Yes".

Have you ever worked for pay or profit?

□

1 Yes

□

2 No Go to Question 23

(If Yes) When did you last work for pay or profit?

□

1 Yes Go to Question 6

□

2 No

(If No) Did you look for work during the last 4 weeks?

□

1 Yes

□

2 No

What is your MAIN reason for not working?

□

Mark one answer.

1 Retired

□

2 On layoff from job due to the coronavirus pandemic

□

3 On layoff from job for reasons unrelated to the coronavirus pandemic

4 Student

□

5 Family responsibilities due to the coronavirus pandemic (e.g., childcare, eldercare)

□

6 Family responsibilities unrelated to the coronavirus pandemic

7 Chronic illness or permanent disability

□

8 Suitable job not available

□

9 Do not need or want to work

□

10 Other - specify

MONTH YEAR

Go to Question 23

□

Current Employment

What were your earnings from wages, salary, commissions, bonuses, or tips, from all jobs during the PAST 12 MONTHS?

Report the amount before deductions for taxes, bonds, dues, or other items.

EARNINGS $ ^{, ,}

What is the name of your MAIN job?

If you have more than one job, answer for the job that you worked the most hours during the past 12 months.

For example: 4th grade teacher, entry-level plumber, web developer.

NAME OF MAIN JOB

How many jobs do you have?

If you are self-employed, count work with multiple customers or clients as one job.

NUMBER OF JOBS

Do you work at a full-time job (a job where you usually work 35 or more hours per week)?

1 Yes

2 No

During the PAST 12 MONTHS (52 weeks), did you work EVERY week?

Count paid vacation, paid sick leave, and military service as work.

1 Yes Go to Question 10

2 No

(If No) During the PAST 12 MONTHS (52 weeks), how many WEEKS did you work?

Include paid time off and weeks when you worked for only a few hours.

NUMBER OF WEEKS

During the PAST 12 MONTHS, in the WEEKS WORKED, how many hours did you usually work each WEEK?

NUMBER OF HOURS WORKED

The next few questions ask about the MAIN job you wrote in Question 12. Please answer for just that job.

13. For the MAIN job you wrote in Question 12, what are your most important activities or duties?

ACTIVITIES OR DUTIES

14. For the MAIN job you wrote in Question 12, please rate your satisfaction with this job's…

Mark one answer for each item.

Does

What is your MAIN BUSINESS OR INDUSTRY— that is, what does your business make or do?

Include the main activity, product, or services you provide.

Wage or salary

Benefits

Very satisfied

Somewhat satisfied

Somewhat dissatisfied

Very dissatisfied

not apply

For example: Web design, Management Consulting, Childcare, Landscaping, Realtor, etc.

BUSINESS OR INDUSTRY

□

Job security

□

Opportunities for

advancement

□

Physical working

conditions

How would you rate your overall satisfaction with this job?

Mark one answer.

□

1 Very satisfied

□

2 Somewhat satisfied

□

3 Somewhat dissatisfied

□

4 Very dissatisfied

For the MAIN job you wrote in Question 12, are you self-employed?

□

1 Yes

□

2 No Go to Question 19

(If Yes) Which one of the following best describes your self-employment?

Mark one answer.

□

1 Self-employed in a NON-INCORPORATED business, professional practice, or farm

2 Self-employed in an INCORPORATED business, professional practice, or farm

Go to Question 21

For the MAIN job you wrote in Question 12, which one of the following best describes your employer at this job?

Mark one answer.

□

1 FOR-PROFIT company or organization

□

2 NON-PROFIT organization (including tax- exempt and charitable organizations)

□

3 LOCAL GOVERNMENT (e.g., city, county, school district)

□

4 STATE GOVERNMENT (including state colleges and universities)

□

5 US MILITARY SERVICE, ACTIVE DUTY, or COMISSIONED CORP (e.g., USPHS, NOAA)

□

6 FEDERAL GOVERNMENT (e.g., civilian employee)

7 Other - specify

Shape39

What is your employer's MAIN BUSINESS OR INDUSTRY—that is, what does this employer make or do?

Include the main activity, product, or service provided at the location where you work.

For example: Elementary school, residential construction, microprocessor chip production.

BUSINESS OR INDUSTRY

Is this mainly:

Mark one answer.

□

1 Manufacturing?

□

2 Wholesale trade?

□

3 Retail trade?

□

4 Other (agriculture, construction, service, government, etc.)?

What is the name and address of the business or employer for the MAIN job you wrote in Question 12?

If your employer has more than one location, report the location where you work.

If you work for a contracting or consulting company, report the name of that company, not the client organization.

Name

Department/Division

City/Town

State/Territory

Zip Code

ations

Licenses and Certific
Do you have any CURRENTLY ACTIVE state or industry licenses or professional certifications?

For example, teaching license, land surveyor license, a nurse midwife certification, ASE master technician certification, Cisco Networking Professional certification, etc.

Do not include vendor's licenses or other licenses to operate a business.

□

1 Yes

□

2 No Go to Question 50

(If Yes) How many CURRENTLY ACTIVE licenses and professional certifications do you have?

If you have the same license in more than one state, count each license separately.

If you earned a certification in order to get a license, count the license and certification separately.

NUMBER OF LICENSES AND PROFESSIONAL CERTIFICATIONS

Thinking about all of your CURRENTLY ACTIVE licenses and professional certifications, which one do you consider to be your MOST IMPORTANT?

Please write out the full name. Do not use abbreviations.

NAME OF LICENSE OR PROFESSIONAL CERTIFICATION

What is the area of specialization of this license or professional certification?

SPECIALIZATION

Shape40

Why did you choose to get a license or professional certification in this particular specialty?

Did you get your MOST IMPORTANT license or professional certification for work-related reasons?

□

1 Yes

□

2 No

Were you legally required to have your MOST IMPORTANT license or professional certification in order to work?

□

1 Yes

□

2 No

Who issued your MOST IMPORTANT license or professional certification?

Mark one answer.

□

1 City or county government agency (e.g., County Electrical Licensing Board, County Department of Land Use)

□

2 State government agency (e.g., State Board of Education or other state board)

□

3 Federal government agency (e.g., OSHA, FAA)

□

4 Professional or trade association (e.g., American Culinary Federation, CompTIA)

□

5 Business or company (e.g., Xerox, 3M Company)

6 Other - specify

Does this license or professional certification need to be renewed?

□

1 Yes

□

2 No Go to Question 33

(If Yes) How often does it need to be renewed?

Mark one answer.

□

1 Yearly

□

2 Every two years

□

3 Other - specify

In what year did you first get this license or professional certification?

YEAR

Do you use this license or professional certification for your MAIN job you wrote in Question 12?

□

0 I am not currently working

□

1 Yes

□

2 No

Shape41

To obtain your MOST IMPORTANT license or professional certification, did you have any of the following financial support?

□

Mark one answer for each item.

Yes

My own money..................................

Money from a spouse, partner, or family member ..................................

Financial support or reimbursement from an employer..............................

Loans ................................................

Grants or scholarships from someplace other than my employer..

Financial support from a professional association........................................

Some other financial support –

specify..........................................

□

□

□

□

□

□
To get the skills and knowledge needed for this license or professional certification, did you take classes from a college, university, technical or trade school, or other school after high school?

□

1 Yes

□

2 No

Did you report having TWO OR MORE CURRENTLY ACTIVE licenses or professional certifications in Question 24?

□

1 Yes Go to Question 38

□

2 No Go to Question 50

You indicated that you have two or more licenses or professional certifications. The next few questions will be about another one of those licenses and professional certifications.

Thinking about all of your CURRENTLY ACTIVE licenses and professional certifications you reported in Question 24, which one do you consider to be your SECOND MOST IMPORTANT?

Please write out the full name. Do not use abbreviations.

NAME OF LICENSE OR PROFESSIONAL CERTIFICATION

What is the area of specialization of this license or professional certification?

SPECIALIZATION

Why did you choose to get a license or professional certification in this particular specialty?

Did you get your SECOND MOST IMPORTANT license or professional certification for work- related reasons?

□

1 Yes

□

2 No

Shape42

Were you legally required to have your SECOND MOST IMPORTANT license or professional certification in order to work?

□

1 Yes

□

2 No

Who issued your SECOND MOST IMPORTANT license or professional certification?

Mark one answer.

□

1 City or county government agency (e.g., County Electrical Licensing Board, County Department of Land Use)

□

2 State government agency (e.g., State Board of Education or other state board)

□

3 Federal government agency (e.g., OSHA, FAA)

□

4 Professional or trade association (e.g., American Culinary Federation, CompTIA)

□

5 Business or company (e.g., Xerox, 3M Company)

6 Other - specify

Does this license or professional certification need to be renewed?

□

1 Yes

□

2 No Go to Question 46

(If Yes) How often does it need to be renewed?

Mark one answer.

□

1 Yearly

□

2 Every two years

□

3 Other - specify

In what year did you first get your SECOND MOST IMPORTANT license or professional certification?

YEAR

Do you use this license or professional certification for your MAIN job you wrote in Question 12?

□

0 I am not currently working

□

1 Yes

□

2 No

To obtain your SECOND MOST IMPORTANT license or professional certification, did you have any of the following financial support?

□

Mark one answer for each item.

	Yes	No
My own money .................................	1	2
Money from a spouse, partner, or family member..................................	1	2
Financial support or reimbursement from an employer .............................	1	2
Loans................................................	1	2
Grants or scholarships from someplace other than my employer .	1	2
Financial support from a professional association ...................	1	2
Some other financial support –
specify ..............................................	1	2

□

□

□

□

□

□
To get the skills and knowledge needed for this license or professional certification, did you take classes from a college, university, technical or trade school, or other school after high school?

□

1 Yes

□

2 No

Vocational Certificates

Have you earned a vocational certificate or diploma for completing a training program from a community college, technical or trade school, or some other school?

□

1 Yes

□

2 No Go to Question 58

What was the field of study for your MOST RECENT vocational certificate?

FIELD OF STUDY

Why did you choose to get a vocational certificate in this particular specialty?

(If Yes) Who awarded your MOST RECENT vocational certificate?

If you have more than one vocational certificate, answer for your most recent one.

Mark one answer.

□

1 A high school

□

2 A vocational, trade, or business school

□

3 A community or technical college

□

4 Another college or university

□

5 Other - specify

How many hours of instruction did you complete in order to earn your MOST RECENT vocational certificate?

Mark one answer.

Do you use this vocational certificate for your MAIN job you wrote in Question 12?

□

0 I am not currently working

□

1 Yes

□

2 No

To obtain your MOST RECENT vocational certificate, did you have any of the following financial support?

Mark one answer for each item.

Yes No

□

My own money.................................. ₁₂

□

1 2 full-time school weeks or less (50 hours of instruction or less)

□

2 More than 2 weeks but less than 3 months of full-time schooling (51 to 299 hours of instruction)

□

3 3 full-time school months, but less than a full-time academic year (300 to 899 hours of instruction)

4 1 full-time academic year or more (900 or more hours of instructions)

53. In what year did you get your most recent vocational certificate?

YEAR

Money from a spouse, partner, or

□

family member .................................. ^{1 2}

□

Financial support or reimbursement

from an employer .............................. ^{1 2}

□

Loans ................................................ ₁₂

□

Grants or scholarships from

someplace other than my employer.. ^{1 2}

□

Financial support from a professional association........................................ ¹²

□

Some other financial support –

specify.......................................... ^{1 2}

We would like to know more about why people choose their careers paths.

Thinking back to when you FIRST DECIDED to pursue any of your licenses, professional certifications, or vocational certificates, why did you feel this was important for your career goals?

Work Experience Programs

Have you ever COMPLETED an internship, apprenticeship, clerkship, externship, residency, clinical experience, or similar program?

□

1 Yes, I have completed this type of program

□

2 No, but I am in one

now

□

3 No, and I am not in one now

Go to Question 68

In the next series of questions, we will refer to internships, apprenticeships, and similar programs as "work experience programs."

What type of work was your last completed work experience program for?

For example: administrative assistant, entry-level plumber, etc.

TYPE OF WORK

How related is your last completed work experience program to your MAIN job you wrote in Question 12?

Mark one answer.

□

1 I am not currently working

□

2 Very related

□

3 Somewhat related

□

4 Not related

Go to Question 63

Shape44

How often do you use the skills or knowledge that you gained during your last work experience program for your MAIN job you wrote in Question 12?

Mark one answer.

□

1 All or most of the time

□

2 Sometimes

□

3 Almost never

□

4 Never

In what year did you complete your last work experience program?

YEAR

What wage did you earn during your last completed work experience program?

Mark one answer.

□

1 I was not paid

□

2 A wage that increased as I gained skills and experience

□

3 A set wage that was lower than the wage of a fully qualified worker

4 The same wage as a fully qualified worker

Which ONE of the following best describes your last work experience program?

Mark one answer.

□

1 It was part of a high school program

Was English your first language?

□

1 Yes

□

2 No

How well do you need to speak English for your MAIN job you wrote in Question 12?

Mark one answer.

□

1 I am not currently working

□

2 I need to be able to easily carry on a conversation

□

3 I need to be able to reply to or ask simple questions

□

4 I need to be able to use single words or short phrases

5 I don’t need to speak English for this job

How well do you need to speak Spanish for your

□

2 It was part of a school program after high school

Go to

Question 67

MAIN job you wrote in Question 12?

Mark one answer.

□

1 I am not currently working

□

3 It was not part of a formal education program

Did your last completed work experience program require that you take classes from…

□

Mark one answer for each item.

A college, university, technical or trade school or other school

Yes

after high school?...................

A company, association, union, or other provider?...................

□

□
As part of your last completed work experience program, did you get formal training from a coworker or mentor?

□

1 Yes

□

2 No

2 I need to be able to easily carry on a conversation

□

3 I need to be able to reply to or ask simple questions

□

4 I need to be able to use single words or short phrases

5 I don’t need to speak Spanish for this job

APPENDIX E
NTEWS Round 3

Shape46

NATIONAL TRAINING, EDUCATION, AND

WORKFORCE SURVEY

INSTRUCTIONS

Directions for filling it out are provided with each question.

Follow all appropriate skip instructions after marking a box. If no skip instruction is provided, you should continue to the next question.

If you have any questions or concerns, please email us at [email protected] or call us toll-free at 1-888-xxx-xxxx.

Thank you for your help. We really appreciate it.

The information collected in this questionnaire is solicited under the authority of the National Science Foundation (NSF) Act of 1950, as amended. The information collected in this question- naire is solicited under the authority of the National Science Foundation (NSF) Act of 1950, as amended. The U.S. Census Bureau is conducting this survey under the authority of Title 13, Sec- tion 8 of the United States Code. The Census Bureau is required by law to keep your information confidential and can use your responses for statistical purposes only. The Census Bureau is not permitted to publicly release your responses in a way that could identify you. Federal law pro- tects your privacy and keeps your answers confidential (Title 13, United States Code, Section 9). Per the Federal Cybersecurity Enhancement Act of 2015, your data are protected from cyberse- curity risks through screening of the systems that transmit your data. Your response is voluntary, and failure to provide some or all of the requested information will not in any way adversely affect you. Actual time to complete the questionnaire may vary depending on your circumstances but on average, it will take about 15 minutes. lf you have any comments on the time required for this survey, please send them to the Reports Clearance Officer, Office of the General Counsel, National Science Foundation, 2415 Eisenhower Ave., Alexandria, VA 22314.

Employment Status

1. Are you currently working for pay or profit?

If you are temporarily absent from a job because you are on vacation, out sick, on maternity leave, etc., please answer "Yes".

Have you ever worked for pay or profit?

□

1 Yes

□

2 No Go to Question 23

(If Yes) When did you last work for pay or profit?

□

1 Yes Go to Question 6

□

2 No

(If No) Did you look for work during the last 4 weeks?

□

1 Yes

□

2 No

What is your MAIN reason for not working?

□

Mark one answer.

1 Retired

□

2 On layoff from job due to the coronavirus pandemic

□

3 On layoff from job for reasons unrelated to the coronavirus pandemic

4 Student

□

5 Family responsibilities due to the coronavirus pandemic (e.g., childcare, eldercare)

□

6 Family responsibilities unrelated to the coronavirus pandemic

7 Chronic illness or permanent disability

□

8 Suitable job not available

□

9 Do not need or want to work

□

10 Other - specify

MONTH YEAR

Go to Question 23

□

Current Employment

What were your earnings from wages, salary, commissions, bonuses, or tips, from all jobs during the PAST 12 MONTHS?

Report the amount before deductions for taxes, bonds, dues, or other items.

^EARNINGS^$, ,

What is the name of your MAIN job?

If you have more than one job, answer for the job that you worked the most hours during the past 12 months.

For example: 4th grade teacher, entry-level plumber, web developer.

NAME OF MAIN JOB

How many jobs do you have?

If you are self-employed, count work with multiple customers or clients as one job.

NUMBER OF JOBS

Do you work at a full-time job (a job where you usually work 35 or more hours per week)?

1 Yes

2 No

During the PAST 12 MONTHS (52 weeks), did you work EVERY week?

Count paid vacation, paid sick leave, and military service as work.

1 Yes Go to Question 10

2 No

(If No) During the PAST 12 MONTHS (52 weeks), how many WEEKS did you work?

Include paid time off and weeks when you worked for only a few hours.

NUMBER OF WEEKS

During the PAST 12 MONTHS, in the WEEKS WORKED, how many hours did you usually work each WEEK?

NUMBER OF HOURS WORKED

The next few questions ask about the MAIN job you wrote in Question 12. Please answer for just that job.

13. For the MAIN job you wrote in Question 12, what are your most important activities or duties?

ACTIVITIES OR DUTIES

For the MAIN job you wrote in Question 12, please rate your satisfaction with this job's…

□

Mark one answer for each item.

Very satisfied

Somewhat satisfied

Somewhat dissatisfied

Very dissatisfied

Does not apply

Wage or salary

Benefits

Job security

Opportunities for advancement

Physical working

conditions

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□
How would you rate your overall satisfaction with this job?

Mark one answer.

□

1 Very satisfied

□

2 Somewhat satisfied

□

3 Somewhat dissatisfied

□

4 Very dissatisfied

For the MAIN job you wrote in Question 12, are you self-employed?

□

1 Yes

□

2 No Go to Question 19

(If Yes) Which one of the following best describes your self-employment?

Mark one answer.

□

1 Self-employed in a NON-INCORPORATED business, professional practice, or farm

2 Self-employed in an INCORPORATED business, professional practice, or farm

What is your MAIN BUSINESS OR INDUSTRY— that is, what does your business make or do?

Include the main activity, product, or services you provide.

For example: Web design, Management Consulting, Childcare, Landscaping, Realtor, etc.

BUSINESS OR INDUSTRY

Go to Question 21

For the MAIN job you wrote in Question 12, which one of the following best describes your employer at this job?

Mark one answer.

□

1 FOR-PROFIT company or organization

□

2 NON-PROFIT organization (including tax- exempt and charitable organizations)

□

3 LOCAL GOVERNMENT (e.g., city, county, school district)

□

4 STATE GOVERNMENT (including state colleges and universities)

□

5 US MILITARY SERVICE, ACTIVE DUTY, or COMISSIONED CORP (e.g., USPHS, NOAA)

□

6 FEDERAL GOVERNMENT (e.g., civilian employee)

7 Other - specify

Education Level

What is your employer's MAIN BUSINESS OR INDUSTRY—that is, what does this employer make or do?

Include the main activity, product, or service provided at the location where you work.

For example: Elementary school, residential construction, microprocessor chip production.

BUSINESS OR INDUSTRY

23. What is the highest degree or level of school that you have COMPLETED?

Mark one answer.

□

1 Elementary, middle, or high school,

but no high school diploma or alternative high school credential (e.g., GED)

□

2 High school diploma

□

3 Alternative high school

Is this mainly:

Mark one answer.

□

1 Manufacturing?

□

2 Wholesale trade?

□

3 Retail trade?

□

4 Other (agriculture, construction, service, government, etc.)?

What is the name and address of the business or employer for the MAIN job you wrote in Question 12?

If your employer has more than one location, report the location where you work.

If you work for a contracting or consulting company, report the name of that company, not the client organization.

credential (e.g. GED)

□

4 Some college credit, no degree

□

5 Associate’s degree (for example: AA, AS)

□

6 Bachelor’s degree (for example: BA, BS)

7 Master’s degree

(for example: MA, MS, Meng, Med, MSW, MBA)

□

8 Professional degree beyond a Bachelor’s degree (for example, MD, DDS, DVM, LLB, JD)

□

9 Doctorate degree

(for example: PhD, EdD)

Go to Question 28

Go to Question 25

Shape55

Name

Department/Division

City/Town

State/Territory

Zip Code

(If elementary, middle, or high school, but no high school diploma or alternative high school credential)

Are you currently enrolled in high school?

Do not include programs to earn an alternative high school credential.

□

1 Yes Go to Question 73

□

2 No Go to Question 73

Vocational Certificates

In what year did you earn your highest degree?

YEAR

What was the field of study for your highest degree?

FIELD OF STUDY

Did you use this highest degree for your MAIN job in Question 12?

Mark one answer.

□

0 I am not currently working

□

1 Yes

□

2 No

Have you earned a vocational certificate or diploma for completing a training program from a community college, technical or trade school, or some other school?

□

1 Yes

□

2 No Go to Question 36

(If Yes) Who awarded your MOST RECENT vocational certificate?

If you have more than vocational certificate, answer for your most recent one.

Mark one answer.

□

1 A high school

□

2 A vocational, trade, or business school

□

3 A community or technical college

□

4 Another college or university

□

5 Other - specify

Shape56

How many hours of instruction did you complete in order to earn your MOST RECENT vocational certificate?

Mark one answer.

□

1 2 full-time school weeks or less (50 hours of instruction or less)

□

2 More than 2 weeks but less than 3 months of full-time schooling (51 to 299 hours of instruction)

□

3 3 full-time school months, but less than a full-time academic year (300 to 899 hours of instruction)

4 1 full-time academic year or more (900 or more hours of instructions)

In what year did you get your most recent vocational certificate?

YEAR

Licenses and Certifications

What was the field of study for your MOST RECENT vocational certificate?

FIELD OF STUDY

Why did you choose to get a vocational certificate in this particular specialty?

Do you use this vocational certificate for your MAIN job you wrote in Question 12?

□

0 I am not currently working

□

1 Yes

□

2 No

To obtain your MOST RECENT vocational certificate, did you have any of the following financial support?

□

Yes

My own money..................................

Money from a spouse, partner, family member, or friend (including loans) ................................................

Financial support or reimbursement from an employer..............................

Loans (from the Federal government or private lenders) .............................

Grants or scholarships from someplace other than my employer..

Financial support from a professional association ...................

Some other financial support –

specify...............................................

Mark one answer for each item.

ations

Do you have any CURRENTLY ACTIVE state or industry licenses or professional certifications?

For example, teaching license, land surveyor license, a nurse midwife certification, ASE master technician certification, Cisco Networking Professional certification, etc.

Do not include vendor's licenses or other licenses to operate a business.

□

1 Yes

□

2 No Go to Question 63

(If Yes) How many CURRENTLY ACTIVE licenses and professional certifications do you have?

If you have the same license in more than one state, count each license separately.

If you earned a certification in order to get a license, count the license and certification separately.

NUMBER OF LICENSES AND PROFESSIONAL CERTIFICATIONS

□
Thinking about all of your CURRENTLY ACTIVE licenses and professional certifications, which one do you consider to be your MOST IMPORTANT?

Please write out the full name. Do not use abbreviations.

□

NAME OF LICENSE OR PROFESSIONAL CERTIFICATION

□

□

□

□

□

□
What kind of work does this license or professional certification allow you to do?

Shape57

Why did you choose to get a license or professional certification in this particular specialty?

Did you get your MOST IMPORTANT license or professional certification for work-related reasons?

□

1 Yes

□

2 No

Were you legally required to have your MOST IMPORTANT license or professional certification in order to work?

□

1 Yes

□

2 No

Who issued your MOST IMPORTANT license or professional certification?

Mark one answer.

□

1 City or county government agency (e.g., County Electrical Licensing Board, County Department of Land Use)

□

2 State government agency (e.g., State Board of Education or other state board)

□

3 Federal government agency (e.g., OSHA, FAA)

□

4 Professional or trade association (e.g., American Culinary Federation, CompTIA)

□

5 Business or company (e.g., Xerox, 3M Company)

6 Other - specify

Does this license or professional certification need to be renewed?

□

1 Yes

□

2 No Go to Question 46

(If Yes) How often does it need to be renewed?

Mark one answer.

□

1 Yearly

□

2 Every two years

□

3 Other - specify

In what year did you first get this license or professional certification?

YEAR

Do you use this license or professional certification for your MAIN job you wrote in question 12?

□

0 I am not currently working

□

1 Yes

□

2 No

To obtain your MOST IMPORTANT license or professional certification, did you have any of the following financial support?

□

Mark one answer for each item.

Yes

My own money..................................

Money from a spouse, partner, family member, or friend (including loans) ..

Financial support or reimbursement from an employer..............................

Loans (from the Federal government or private lenders) .............................

Grants or scholarships from someplace other than my employer..

Financial support from a professional association........................................

Some other financial support –

specify..........................................

□

□

□

□

□

□
To get the skills and knowledge needed for this license or professional certification, did you take classes from a college, university, technical or trade school, or other school after high school?

□

1 Yes

□

2 No

Did you report having TWO OR MORE CURRENTLY ACTIVE licenses or professional certifications in Question 37?

□

1 Yes Go to Question 51

□

2 No Go to Question 63

You indicated that you have two or more licenses or professional certifications. The next few questions will be about another one of those licenses and professional certifications.

Thinking about all of your CURRENTLY ACTIVE licenses and professional certifications you reported in Question 37, which one to do consider to be your SECOND MOST IMPORTANT?

Please write out the full name. Do not use abbreviations.

NAME OF LICENSE OR PROFESSIONAL CERTIFICATION

What kind of work does this license or professional certification allow you to do?

Why did you choose to get a license or professional certification in this particular specialty?

Did you get your SECOND MOST IMPORTANT license or professional certification for work- related reasons?

□

1 Yes

□

2 No

Were you legally required to have your SECOND MOST IMPORTANT license or professional certification in order to work?

□

1 Yes

□

2 No

Who issued your SECOND MOST IMPORTANT license or professional certification?

Mark one answer.

□

1 City or county government agency (e.g., County Electrical Licensing Board, County Department of Land Use)

□

2 State government agency (e.g., State Board of Education or other state board)

□

3 Federal government agency (e.g., OSHA, FAA)

□

4 Professional or trade association (e.g., American Culinary Federation, CompTIA)

□

5 Business or company (e.g., Xerox, 3M Company)

7 Other - specify

Does this license or professional certification need to be renewed?

□

1 Yes

□

2 No Go to Question 59

(If Yes) How often does it need to be renewed?

Mark one answer.

□

1 Yearly

□

2 Every two years

□

3 Other - specify

In what year did you first get your SECOND MOST IMPORTANT license or professional certification?

YEAR

Do you use this license or professional certification for your MAIN job you wrote in question 12?

□

0 I am not currently working

□

1 Yes

□

2 No

To obtain your SECOND MOST IMPORTANT license or professional certification, did you have any of the following financial support?

□

Mark one answer for each item.

Yes

My own money .................................

Money from a spouse, partner, family member, or friend (including loans)................................................

Financial support or reimbursement from an employer .............................

Loans (from the Federal government or private lenders) ............................

Grants or scholarships from someplace other than my employer .

Financial support from a professional association ...................

Some other financial support –

specify ..............................................

□

□

□

□

□

□
To get the skills and knowledge needed for this license or professional certification, did you take classes from a college, university, technical or trade school, or other school after high school?

□

1 Yes

□

2 No

We would like to know more about why people choose their careers paths.

Thinking back to when you FIRST DECIDED to pursue any of your vocational certificates, licenses, and professional certifications, why did you feel this was important for your career goals?

Work Experience Programs

Have you ever COMPLETED an internship, apprenticeship, clerkship, externship, residency, clinical experience, student teaching, or similar program?

□

1 Yes, I have completed this type of program

□

2 No, but I am in one

now

□

3 No, and I am not in one now

Go to Question 73

In the next series of questions, we will refer to internships, apprenticeships, and similar programs as "work experience programs."

What type of work was your last completed work experience program for?

For example: administrative assistant, entry-level plumber, etc.

TYPE OF WORK

How related is your last completed work experience program to your MAIN job you wrote in Question 12?

Mark one answer.

1 □ I am not currently working

2 □ Very related

3 □ Somewhat related

4 □ Not related

Go to Question 68

Shape61

How often do you use the skills or knowledge that you gained during your last work experience program for your MAIN job you wrote in Question 12?

Mark one answer.

□

1 All or most of the time

□

2 Sometimes

□

3 Almost never

□

4 Never

In what year did you complete your last work experience program?

YEAR

What wage did you earn during your last completed work experience program?

Mark one answer.

□

1 I was not paid

□

2 A wage that increased as I gained skills and experience

□

3 A set wage that was lower than the wage of a fully qualified worker

4 The same wage as a fully qualified worker

Which ONE of the following best describes your last work experience program?

Mark one answer.

□

1 It was part of a high school program

Background
Was English your first language?

□

1 Yes

□

2 No

How well do you need to speak English for your MAIN job you wrote in Question 12?

Mark one answer.

□

1 I am not currently working

□

2 I need to be able to easily carry on a conversation

□

3 I need to be able to reply to or ask simple questions

□

4 I need to be able to use single words or short phrases

5 I don’t need to speak English for this job

How well do you need to speak Spanish for your

□

2 It was part of a school program after high school

Go to

Question 72

MAIN the job you wrote in Question 12?

Mark one answer.

□

1 I am not currently working

□

3 It was not part of a formal education program

Did your last completed work experience program require that you take classes from…

□

Mark one answer for each item.

Yes

A college, university, technical or trade school or other school after high school?.................................

A company, association, union, or other provider?.............................

□

□
As part of your last completed work experience program, did you get formal training from a coworker or mentor?

□

1 Yes

□

2 No

2 I need to be able to easily carry on a conversation

□

3 I need to be able to reply to or ask simple questions

□

4 I need to be able to use single words or short phrases

5 I don’t need to speak Spanish for this job

APPENDIX F
NTEWS Round 4

Shape63

NATIONAL TRAINING, EDUCATION, AND

WORKFORCE SURVEY

INSTRUCTIONS

Directions for filling it out are provided with each question.

Follow all appropriate skip instructions after marking a box. If no skip instruction is provided, you should continue to the next question.

If you have any questions or concerns, please email us at [email protected] or call us toll-free at 1-888-xxx-xxxx.

Thank you for your help. We really appreciate it.

The information collected in this questionnaire is solicited under the authority of the National Science Foundation (NSF) Act of 1950, as amended. The information collected in this question- naire is solicited under the authority of the National Science Foundation (NSF) Act of 1950, as amended. The U.S. Census Bureau is conducting this survey under the authority of Title 13, Sec- tion 8 of the United States Code. The Census Bureau is required by law to keep your information confidential and can use your responses for statistical purposes only. The Census Bureau is not permitted to publicly release your responses in a way that could identify you. Federal law pro- tects your privacy and keeps your answers confidential (Title 13, United States Code, Section 9). Per the Federal Cybersecurity Enhancement Act of 2015, your data are protected from cyberse- curity risks through screening of the systems that transmit your data. Your response is voluntary, and failure to provide some or all of the requested information will not in any way adversely affect you. Actual time to complete the questionnaire may vary depending on your circumstances but on average, it will take about 15 minutes. lf you have any comments on the time required for this survey, please send them to the Reports Clearance Officer, Office of the General Counsel, National Science Foundation, 2415 Eisenhower Ave., Alexandria, VA 22314.

Employment Status

1. Are you currently working for pay or profit?

If you are temporarily absent from a job because you are on vacation, out sick, on maternity leave, etc., please answer "Yes".

Have you ever worked for pay or profit?

□

1 Yes

□

2 No Go to Question 23

(If Yes) When did you last work for pay or profit?

□

1 Yes Go to Question 6

□

2 No

Shape64

(If No) Did you look for work during the last 4 weeks?

□

1 Yes

□

2 No

What is your MAIN reason for not working?

□

Mark one answer.

1 Retired

□

2 On layoff from job due to the coronavirus pandemic

□

3 On layoff from job for reasons unrelated to the coronavirus pandemic

4 Student

□

5 Family responsibilities due to the coronavirus pandemic (e.g., childcare, eldercare)

□

6 Family responsibilities unrelated to the coronavirus pandemic

7 Chronic illness or permanent disability

□

8 Suitable job not available

□

9 Do not need or want to work

□

10 Other - specify

MONTH YEAR

Go to Question 23

Current Employment

How many jobs do you have?

If you are self-employed, count work with multiple customers or clients as one job.

NUMBER OF JOBS

During the PAST 12 MONTHS, how many hours did you usually work each WEEK across ALL jobs?

NUMBER OF HOURS WORKED

What is the name of your MAIN job?

If you have more than one job, answer for the job that you most identify with.

For example: 4th grade teacher, entry-level plumber, web developer, etc.

NAME OF MAIN JOB

For the MAIN job you wrote in Question 8, what are your most important activities or duties?

ACTIVITIES OR DUTIES

During the PAST 12 MONTHS (52 weeks), did you work EVERY week at the MAIN job you wrote in Question 8?

Count paid vacation, paid sick leave, and military service as work.

□

1 Yes Go to Question 12

□

The next few questions ask about the MAIN job you wrote in Question 8. Please answer for just that job.

2 No

(If No) During the PAST 12 MONTHS (52 weeks), how many WEEKS did you work at the MAIN job you wrote in Question 8?

Include paid time off and weeks when you worked for only a few hours.

NUMBER OF WEEKS

Shape65

During the PAST 12 MONTHS, in the WEEKS WORKED, how many hours did you usually work each WEEK at the MAIN job you wrote in Question 8?

NUMBER OF HOURS WORKED

During the PAST 12 MONTHS, what were your earnings from wages, salary, commissions, bonuses, or tips, from the MAIN job you wrote in Question 8?

Report the amount before deductions for taxes, bonds, dues, or other items.

EARNINGS $

For the MAIN job you wrote in Question 8, please rate your satisfaction with this job's…

Mark one answer for each item.

Does

(If Yes) Which one of the following best describes your self-employment?

Mark one answer.

□

1 Self-employed in a NON-INCORPORATED business, professional practice, or farm

2 Self-employed in an INCORPORATED business, professional practice, or farm

What is your MAIN BUSINESS OR INDUSTRY— that is, what does your business make or do?

Include the main activity, product, or services you provide.

Very

Somewhat

Very

not

For example: Web design, Management

satisfied

dissatisfied dissatisfied

apply

Consulting, Childcare, Landscaping, etc.

Shape66

□

Wage or salary

□

Benefits

□

Job security

Opportunities

□

for advancement

□

Physical working

conditions ^{1 2 3 4 5}

How would you rate your overall satisfaction with this job?

Mark one answer.

□

1 Very satisfied

□

2 Somewhat satisfied

□

3 Somewhat dissatisfied

□

4 Very dissatisfied

□
For the MAIN job you wrote in Question 8, are you self-employed?

□

1 Yes

2 No Go to Question 19

BUSINESS OR INDUSTRY

Go to Question 21

For the MAIN job you wrote in Question 8, which one of the following best describes your employer at this job?

Mark one answer.

□

1 FOR-PROFIT company or organization

□

2 NON-PROFIT organization (including tax- exempt and charitable organizations)

□

3 LOCAL GOVERNMENT (e.g., city, county, school district)

□

4 STATE GOVERNMENT (including state colleges and universities)

□

5 US MILITARY SERVICE, ACTIVE DUTY, or COMISSIONED CORP (e.g., USPHS, NOAA)

□

6 FEDERAL GOVERNMENT (e.g., civilian employee)

7 Other - specify

What is your employer's MAIN BUSINESS OR INDUSTRY—that is, what does this employer make or do?

Include the main activity, product, or service provided at the location where you work.

For example: Elementary school, residential construction, microprocessor chip production, etc.

BUSINESS OR INDUSTRY

Is this mainly:

Mark one answer.

□

1 Manufacturing?

□

2 Wholesale trade?

□

3 Retail trade?

□

4 Other (agriculture, construction, service, government, etc.)?

Enrollment
Are you currently enrolled in or taking courses from a college, university, technical or trade school, or other school after high school?

□

1 Yes

□

2 No Go to Question 28

(If Yes) Are you currently enrolled full time or part time?

Mark one answer.

□

1 Full time

□

2 Part time

Are you currently enrolled in a degree program?

□

1 Yes

□

2 No, but I plan to transfer to a degree program

22. What is the name and address of the business or employer for the MAIN job you wrote in Question

3 No, and I do not plan to transfer

□

to a degree program

Go to Question 28

Shape67 8?

If your employer has more than one location, report the location where you work.

Name

Department/Division

Street Address

City/Town

State/Territory

Zip Code

If you work for a contracting or consulting company, report the name of that company, not the client organization.

What degree are you working toward?

Mark one answer.

□

1 Associate’s degree (e.g., AA, AS)

□

2 Bachelor’s degree (e.g., BA, BS)

□

3 Master’s degree (e.g., MA, MS, MEng, MEd, MSW, MBA)

□

4 Professional degree beyond a Bachelor’s degree (e.g., MD, DDS, DVM, LLB, JD)

5 Doctorate degree (e.g., PhD, EdD)

□

6 Other – specify:

What is your primary field of study?

FIELD OF STUDY

Educational Background

What is the highest degree or level of school that you have COMPLETED?

Mark one answer.

□

1 Elementary, middle, or high school, but no high school diploma or alternative high school credential (e.g., GED)

2 High school diploma

In what year did you earn your highest degree?

YEAR

What was the field of study for your highest degree?

FIELD OF STUDY

□

3 Alternative high school credential (e.g., GED)

4 Some college credit, no degree

□

5 Vocational certificate or diploma

□

6 Associate’s degree (e.g., AA, AS)

7 Bachelor’s degree (e.g., BA, BS)

□

8 Master’s degree (e.g., MA, MS, MEng, MEd, MSW, MBA)

□

9 Professional degree beyond a Bachelor’s degree (e.g., MD, DDS, DVM, LLB, JD)

10 Doctorate degree (e.g., PhD, EdD)

Go to Question 34

Go to Question 30

Do you use this degree for your MAIN job you wrote in Question 8?

Mark one answer.

□

0 I am not currently working

□

1 Yes

□

2 No

Why did you choose to get this degree?

Shape68

29. (If elementary, middle, or high school, but no high school diploma or alternative high school credential)

Are you currently enrolled in high school?

Do not include programs to earn an alternative high school credential (e.g., GED).

□

1 Yes Go to Question 78

□

2 No Go to Question 34

Vocational Certificates

Have you earned a vocational certificate or diploma for completing a training program from a community college, technical or trade school, or some other school?

□

1 Yes

□

2 No Go to Question 42

(If Yes) Who awarded your MOST RECENT vocational certificate?

If you have more than one vocational certificate, answer for your most recent one.

Mark one answer.

□

1 A high school

□

2 A vocational, trade, or business school

□

3 A community or technical college

□

4 Another college or university

□

5 Other - specify

What was the field of study for your MOST RECENT vocational certificate?

FIELD OF STUDY

Why did you choose to get a vocational certificate in this particular field of study?

□

Yes

It was required for the job I was already doing.....................................

It was required to get a job I wanted .

It allowed me to do more in the job I was already doing .............................

It allowed me to earn more money ....

It was a new or emerging area in my field ...................................................

I was pursuing my passion ................

I was exploring potential interest in a new job or field ..................................

Someone recommended this field or job to me ...........................................

It was a free or inexpensive opportunity ........................................

Some other reason – specify.............

Mark one answer for each item.

□

□

□

□

□

□
How long did it take to complete your MOST RECENT vocational certificate?

□

Mark one answer.

□

1 2 full-time school weeks or less (50 hours of instruction or less)

□

2 More than 2 weeks but less than 3 months of full-time schooling (51 to 299 hours of instruction)

□

3 3 full-time school months, but less than a full- time academic year (300 to 899 hours of instruction)

4 1 full-time academic year or more (900 or more hours of instructions)

In what year did you get your MOST RECENT vocational certificate?

YEAR

Shape70

□

□
Do you use this vocational certificate for your MAIN job you wrote in Question 8?

□

0 I am not currently working

□

1 Yes

□

2 No

Licenses and Certifications

To obtain your MOST RECENT vocational certificate, did you have any of the following financial support?

□

Mark one answer for each item.

Yes

My own money ..................................

Loans from a spouse, partner, or family member ...................................

Money from a spouse, partner, or family member that does not need to be repaid ...........................................

Financial support or reimbursement from an employer...............................

Loans from the government or private lenders ...................................

Grants or scholarships from someplace other than my employer...

Financial support from a professional association.........................................

Some other financial support –

specify ...............................................

Do you have any CURRENTLY ACTIVE state or industry licenses or professional certifications?

For example, teaching license, land surveyor license, a nurse midwife certification, ASE master technician certification, Cisco Certified Network Associate (CCNA), etc.

□

Do not include vendor's licenses or other licenses to operate a business.

□

1 Yes

□

2 No Go to Question 69

□

□

□

□
(If Yes) How many CURRENTLY ACTIVE licenses and professional certifications do you have?

□

If you have the same license in more than one state, count each license, separately.

□

If you earned a certification in order to get a license, count the license and certification, separately.

NUMBER OF LICENSES AND PROFESSIONAL CERTIFICATIONS

Thinking about all of your CURRENTLY ACTIVE licenses and professional certifications, which one do you consider to be your MOST IMPORTANT?

Please write out the full name. Do not use abbreviations.

NAME OF LICENSE OR PROFESSIONAL CERTIFICATION

What kind of work does this license or professional certification allow you to do?

Did you get your MOST IMPORTANT license or professional certification for work-related reasons?

□

1 Yes

□

2 No

Why did you choose to get a license or professional certification in this particular specialty?

□

Mark one answer for each item.

	Yes	No
It was required for the job I was already doing ....................................	1	2
It was required to get a job I wanted.	1	2
It allowed me to do more in the job I was already doing.............................	1	2
It allowed me to earn more money ...	1	2
It was a new or emerging area in my field ...................................................	1	2
I was pursuing my passion ...............	1	2
I was exploring potential interest in a new job or field .................................	1	2
Someone recommended this field or job to me...........................................	1	2
It was a free or inexpensive opportunity........................................	1	2
Some other reason – specify ............	1	2

□

□

□

□

□

□

□

□
Were you legally required to have your MOST IMPORTANT license or professional certification in order to work?

□

1 Yes

□

2 No

Who issued your MOST IMPORTANT license or professional certification?

Mark one answer.

□

1 City or county government agency (e.g., County Electrical Licensing Board, County Department of Land Use)

□

2 State government agency (e.g., State Board of Education or other state board)

3 Federal government agency (e.g., OSHA, FAA)

□

4 Professional or trade association (e.g., American Culinary Federation, CompTIA)

□

5 Business or company (e.g., Xerox, 3M Company)

6 Other - specify

Does this license or professional certification need to be renewed?

□

1 Yes

□

2 No Go to Question 52

(If Yes) How often does it need to be renewed?

Mark one answer.

□

1 Every year

□

2 Every two years

□

3 Other - specify

In what year did you first get this license or professional certification?

YEAR

Do you use this license or professional certification for your MAIN job you wrote in Question 8?

□

0 I am not currently working

□

1 Yes

□

2 No

To obtain your MOST IMPORTANT license or professional certification, did you have any of the following financial support?

□

Mark one answer for each item.

Yes

My own money ..................................

Loans from a spouse, partner, or family member ...................................

Money from a spouse, partner, or family member that does not need to be repaid ...........................................

Financial support or reimbursement from an employer...............................

Loans from the government or private lenders ...................................

Grants or scholarships from someplace other than my employer...

Financial support from a professional association.........................................

Some other financial support –

specify ...............................................

Thinking about all of your CURRENTLY ACTIVE licenses and professional certifications you reported in Question 43, which one do you consider to be your SECOND MOST IMPORTANT?

Please write out the full name. Do not use abbreviations.

NAME OF LICENSE OR PROFESSIONAL CERTIFICATION

□

□

□

□

□

□

□

□
What kind of work does this license or professional certification allow you to do?

□

□

□
Did you get your SECOND MOST IMPORTANT license or professional certification for work- related reasons?

□

1 Yes

□

2 No

Shape74

□
To get the skills and knowledge needed for this license or professional certification, did you take classes from a college, university, technical or trade school, or other school after high school?

□

1 Yes

□

2 No

Did you report having TWO OR MORE CURRENTLY ACTIVE licenses or professional certifications in Question 43?

□

1 Yes Go to Question 57

□

2 No Go to Question 69

Why did you choose to get a license or professional certification in this particular specialty?

□

Mark one answer for each item.

Yes

It was required for the job I was already doing ....................................

It was required to get a job I wanted.

It allowed me to do more in the job I was already doing.............................

It allowed me to earn more money ...

It was a new or emerging area in my field ...................................................

I was pursuing my passion ...............

I was exploring potential interest in a new job or field .................................

Someone recommended this field or job to me...........................................

It was a free or inexpensive opportunity........................................

Some other reason – specify ............

□

□

□

□

□

□

□

□
Were you legally required to have your SECOND MOST IMPORTANT license or professional certification in order to work?

□

1 Yes

□

2 No

Who issued your SECOND MOST IMPORTANT license or professional certification?

Mark one answer.

□

1 City or county government agency (e.g., County Electrical Licensing Board, County Department of Land Use)

□

2 State government agency (e.g., State Board of Education or other state board)

3 Federal government agency (e.g., OSHA, FAA)

□

4 Professional or trade association (e.g., American Culinary Federation, CompTIA)

□

5 Business or company (e.g., Xerox, 3M Company)

6 Other - specify

Does this license or professional certification need to be renewed?

□

1 Yes

□

2 No Go to Question 65

(If Yes) How often does it need to be renewed?

Mark one answer.

□

1 Every year

□

2 Every two years

□

3 Other - specify

Shape75

In what year did you first get this license or professional certification?

YEAR

Do you use this license or professional certification for your MAIN job you wrote in Question 8?

□

0 I am not currently working

□

1 Yes

□

2 No

grams

Work Experience Pro

69. Have you ever COMPLETED an internship, apprenticeship, clerkship, externship, residency, clinical experience, student teaching, or similar program?

□

1 Yes, I have completed this type of program

To obtain your SECOND MOST IMPORTANT license or professional certification, did you have any of the following financial support?

Mark one answer for each item.

2 No, but I am in one now

□

In the next series of questions, we will refer to internships, apprenticeships, and similar programs as "work experience programs."

3 No, and I am not in one now

Go to Question 78

Yes

□

My own money ..................................	1	2
Loans from a spouse, partner, or
family member ...................................	1	2
Money from a spouse, partner, or
family member that does not need to be repaid ...........................................	1	2
Financial support or reimbursement
from an employer...............................	1	2
Loans from the government or
private lenders ...................................	1	2
Grants or scholarships from
someplace other than my employer...	1	2
Financial support from a professional
association.........................................	1	2
Some other financial support –
specify ...............................................	1	2

70. What type of work was your last completed work experience program for?

For example: administrative assistant, entry-level plumber, etc.

□

TYPE OF WORK

71. How related is your last completed work

□

experience program to your MAIN job you wrote in Question 8?

Mark one answer.

To get the skills and knowledge needed for this license or professional certification, did you take classes from a college, university, technical or

1 I am not currently working

□

2 Very related

□

3 Somewhat related

□

4 Not related

Go to Question 73

Shape76 trade school, or other school after high school?

□

1 Yes

□

2 No

How often do you use the skills or knowledge that you gained during your last work experience program for your MAIN job you wrote in Question 8?

Mark one answer.

□

1 All or most of the time

□

2 Sometimes

□

3 Almost never

□

4 Never

Background
In what year did you complete your last work experience program?

YEAR

What wage did you earn during your last completed work experience program?

Mark one answer.

□

1 I was not paid

□

2 A wage that increased as I gained skills and experience

□

3 A set wage that was lower than the wage of a fully qualified worker

4 The same wage as a fully qualified worker

Which ONE of the following best describes your last completed work experience program?

□

Mark one answer.

1 It was part of a high school

Are you male or female?

□

1 Male

□

2 Female

What is your birthdate?

MM/DD/YYYY

Are you of Hispanic, Latino, or Spanish origin?

□

1 Yes

□

2 No

What is your race?

Mark one or more.

□

1 American Indian or Alaska Native

Specify tribal affiliation(s):

program

□

2 It was part of a school program after high school

3 It was not part of a formal education program

Go to Question 77

□

2 Native Hawaiian or other Pacific Islander

□

3 Asian

□

4 Black or African American

Did your last completed work experience program require that you take classes from…

□

Mark one answer for each item.

	Yes	No
A college, university, technical or trade school or other school after high school? .............................................	¹	²
A company, association, union, or other provider? .................................	1	2

□

□
As part of your last completed work experience program, did you get formal training from a coworker or mentor?

□

1 Yes

□

2 No

5 White

□
What is your current marital status?

Mark one answer.

□

1 Married

□

2 Living in a marriage-like relationship

□

3 Widowed

□

4 Divorced

□

5 Separated

□

6 Never married

Was English your first language?

□

1 Yes

□

2 No

How well do you need to speak English for your MAIN job you wrote in Question 8?

Mark one answer.

□

1 I am not currently working

□

2 I need to be able to easily carry on a conversation

□

3 I need to be able to reply to or ask simple questions

□

4 I need to be able to use single words or short phrases

5 I don’t need to speak English for this job

How well do you need to speak Spanish for your MAIN job you wrote in Question 8?

Mark one answer.

□

1 I am not currently working

□

2 I need to be able to easily carry on a conversation

□

3 I need to be able to reply to or ask simple questions

□

4 I need to be able to use single words or short phrases

5 I don’t need to speak Spanish for this job

Have you ever served on active duty in the U.S. Armed Forces, Reserves, or National Guard?

Mark one answer.

□

1 Never served in the military

□

2 Only on active duty for training in the Reserves or National Guard

3 Now on active duty

□

4 On active duty in the past, but not now

Are you currently living in the United States or Puerto Rico, another U.S. territory, or are you living in another country?

Mark one answer.

□

1 United States or Puerto Rico

□

2 Another U.S. territory

□

3 Another country

In what U.S. state, U.S. territory, or foreign country were you born?

U.S. STATE OR TERRITORY

FOREIGN COUNTRY

Are you a U.S. citizen?

□

1 Yes END

□

2 No Go to Question 90

(If non-U.S. citizen) Are you currently a non-U.S. citizen with a…

Mark one answer.

□

1 Permanent U.S. Resident Visa (Green Card)?

□

2 Temporary U.S. Resident Visa?

1 Note that some respondents fit multiple recruitment categories, which is why the number of respondents here adds to more than 14. Only 14 interviews were conducted.

2 Note that some respondents fit multiple recruitment categories, which is why the number of respondents here adds to more than 5. Only 5 interviews were conducted.

3 Note that some respondents fit multiple recruitment categories, which is why the number of respondents here adds to more than four. Only four interviews were conducted.

4 In addition to the 58 interviews conducted for this analysis, Mathematica scheduled 7 interviews where the respondent was a ‘no-show’ and conducted 3 interviews that were not included in this analysis. Two of these excluded interviews were from Round 3 and one interview was from Round 4. One interview was excluded because the respondent was noticeably uncomfortable during the cognitive interview and distressed by having multiple people watch him answer questions, which led to several skip logic and response errors. One respondent provided inconsistent responses to the questions that did not seem to be based on his actual experiences. For example, the respondent initially reported that he was not working in Question 1, then reported that he had seven jobs in Question 6, but that he only works 12 hours a week in Question 10. The last interview was dropped for technical and methodological reasons – the respondent was not able to open the e-PDF. To try to resolve the technical problems, the study team shared their screen with the survey questions and talked through the questions with the respondent, rather than seeing the respondent select answers and probing afterwards.

5 According to the U.S. Bureau of Labor Statistics (2016), a gig can be understood as “a single project or task for which a worker is hired, often through a digital marketplace, to work on demand” (https://www.bls.gov/careeroutlook/2016/article/what-is-the-gig-economy.htm).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	0000-00-00

NTEWS Questionnaire Development through Cognitive Research

14_NTEWS APPENDIX N_Questionnaire Development.docx

2022 National Training, Education, and Workforce Survey (NTEWS) (NCSES)