2017 National Household Education Survey (NHES)
Web Data Collection Test
National Center for Education Statistics (NCES)
August 4, 2016
Appendix C. ATES Cognitive interview results memos
Round 1 Memo 2
Round 2 Memo 11
Round 3 Memo 19
|
Round 1 Memo |
Date: |
March 22, 2016 |
To: |
Sharon Boivin (NCES); Lisa Hudson (NCES) |
CC: |
Sarah Grady (NCES) |
From |
Kirsten Manville (AIR); Meghan McQuiggan (AIR); Maria Payri Alvarez (AIR); Stephanie Cronen (AIR); Danielle Battle (AIR); Rebecca Medway (AIR); Cameron McPhee (AIR); Stacey Bielick (AIR) |
Re: |
NHES: 2016 ATES Cognitive Interviews, Round 1 |
This memo presents the issues found during the first round of cognitive interviewing of the Adult Training and Education Survey (ATES) from March 7th through March 16th, 2016. This memo also includes suggestions from AIR in response to these issues, as well as comments from respondents who proposed potential solutions.
Ten interviews were carried out by AIR interviewers using an abbreviated ATES instrument that included four sections: Education; Certifications and Licenses; Preparation for New Certifications and Licenses; Certifications and Licenses You Have Considered Getting. Interviews were conducted in Washington, DC. All interviews lasted about an hour, were conducted in English, and were tape recorded. Seven respondents reported having a currently active certification or license, seven respondents reported that they are currently working on getting a new certification or license, and four respondents reported that they considered getting a new professional certification or license in the last five years that they did not end up getting.
Question 2 asks respondents about the field of study for the highest level of school they have completed. A few of the respondents reported the field of study for something other than the highest level of education they reported having completed. For example, one respondent answered based on the field of study for her multiple certifications. Because they were in a variety of fields, she marked “general studies” (which just happened to be what she should have marked for her high school diploma). Another respondent answered the question based on a degree she is currently working on but has not yet completed. A third respondent student mapped his field of study onto the incorrect response option; he marked “science or mathematics” despite later stating that he received his BS in computer science.
To determine the extensiveness of this issue, we may want to probe in round 2 about (1) which degree/credential the respondent is reporting on, and (2) the respondent’s exact field of study. If reporting on the wrong level of education proves to be a widespread issue, NCES might consider underlining “completed” to mirror the presentation of question 1 or adding an instruction to report on the level of education reported in question 1. If selecting the wrong response option proves to be a widespread issue, NCES might consider further research into the response options used in similar items in other federal surveys or conducting qualitative research to better understand how respondents classify different fields of study.
LH: For online ATES can we use CIP coder? Can we afford write-ins on PAPI ATES?
Result: Added “or major” to Q2. Will continue testing and observing this question.
As found in previous cognitive testing, some respondents reported on different types of credentials than what was requested in each of the sections that asked about certifications and licenses. One respondent filled out the “Certifications and Licenses” section for a certification she had not yet completed. She later reported the same certification (that she was currently pursing) in the “Preparation for New Certifications and Licenses” section. Another respondent reported her business degree in finance as a certification. A third respondent flipped back and forth between the license she currently has (bartending) and the one she is currently working on (audio/music engineering) while completing the “Certifications and Licenses” section. She explained that she did this because she saw the words “most important” and started answering about the audio/music certification she is currently pursing because she considers that to be her most important. A few respondents reported on credentials in the “Preparation for Certifications and Licenses” section that seemed like they could be certificates1.
It is possible that reporting of in-progress certifications is a function of recruitment focusing on and targeting individuals who are currently working on certifications; this may prime these individuals to come to the interviews ready to discuss those in-progress certifications (however, this type of recruitment is necessary to get respondents on which to test the “Preparation” section). We should continue to probe carefully to try to identify such situations; it may be worthwhile to add probes to the protocol that attempt to understand why these respondents are reporting on in-progress certifications in a section that asks about currently active certifications. If this issue continues, NCES might consider adding instructions to the Certifications and Licenses section that ask respondents to exclude any certifications that are currently in-progress.
LH: (AIR might have done this already) Fix typo in Q8—should be “certification or license” instead of “certification and license”
Result – AIR Updated this typo.
LH: Rs clearly want to talk about what they are doing NOW. So how about if we reverse the order of these sections and ask first about CNs they are working on, and then about others they have? If we do this, we should move the CN definition from the current Q6 (“Do you have a currently active professional certification…”) to the current Q30 (“Are you currently working on any of the following? Maintaining or renewing a professional certification or license…”). To accommodate the definition, the current Q30 would also need to be reworded, to something like “Are you currently working on either getting or renewing a professional certification or license?” [add defn] (a) Working on renewing a professional certification or license N/Y; (b) Working on getting a new professional certification or license N/Y.
Result – We will monitor this in upcoming rounds but will make no change due to concerns with order effects on estimates. We need to make sure we know what they are reporting on to make sure it is not what they are currently working on. We will continue our probes if not sure what they are talking about.
Question 10 asks respondents if their certification or license is required by a federal, state, or local government agency. Responses to this item were mixed; some respondents had no trouble, marked “yes”, and were able to easily identify who issued their credential. Others had more difficulty; several said they were receiving their certifications from vocational schools and marked “don’t know”. One respondent marked “yes,” and explained that he/she was receiving the credential from a college. Another respondent interpreted the question as asking if the vocational school itself (where he/she was getting the credential) was certified. The respondent marked “yes,” explaining that the program was just as qualified as other universities in the area. Overall, it seemed that respondents were confusing the school where they were taking classes with the actual credentialing body. These comments also apply to questions 21, 27, and 33.
Given that this is one of the more “established” items on ATES, we suggest continuing to monitor respondents’ reactions in round 2 before making extensive changes.
LH: Q10: delete “(such as a state board)” here and in similar Qs throughout
Result – Test what is currently in the metadata to see if this works (removed types of government).
Question 11 (Would you describe this certification or license as…) caused some respondents to misreport information. This stemmed from confusion with the sub-items. One respondent defined “field” more narrowly than intended. She said “no” to 11a (your first certification or license in this field?) because she considered the field of her certification to be a specific branch of healthcare, not just healthcare. However, the respondent had previously mentioned having another certification in the healthcare field. Another respondent failed to read the entire question for 11a; she stopped after the first line (“your first certification or license”) and prematurely marked no. She previously mentioned that this certification is in a new field for her, so she should have marked “yes” to this question. A third respondent was very confused by 11a, so much that she wrote a “?” and skipped the question. When probed she said that it would have helped her if 11a had stated “your first certification or license in this field or another field,” because the way it was written now didn’t allow her to report or take into consideration her nursing aide certification that is no longer active.
Sub-item 11b (a certification or license in a field in which you already have another certification or license?) in this series of questions also proved to be challenging for one respondent. This respondent marked yes to b because his license was valid in multiple states (VA, DC, and MD).
For questions 11c and 11d, some respondents had difficulty distinguishing between “specialized” and “advanced-level.” When probed to give a definition, most respondents defined “specialized” as something that is industry specific, a sub-component within an industry, or as a niche. “Advanced-level” was generally described as something that acknowledges a higher-level of proficiency in a field, or an upgrade to a certification or license you already have, though some respondents thought “advanced” was related to the amount of education necessary to get the credential. One example given by a respondent was a “special police” certification, which he described as an advanced-level certification of his basic unarmed security officer certification. While they were able to provide accurate definitions when probed, there seemed to be a lot of confusion when they answered the items. A couple of respondents misreported information when responding to these questions. One respondent (who had a lead abatement license) marked “no” to both sub-items (c & d). When probed, he explained that he thought advanced-level meant that you needed a higher degree of education to attain it, and said that he would consider it specialized (despite marking “no” to this item). These comments also apply to questions 22, 28, and 34.
To reduce the cognitive burden associated with this item, NCES might want to consider splitting this into four separate items that each read as a full sentence (e.g., Is this your first certification or license in this field? Do you have another certification or license in the same field from a different state? Is this a specialized certification or license that focuses on a specific skill or subpart of a field in which you already have a certification or license? Is this an advanced-level certification or license that builds on a certification or license you already have in the same field?) and have the yes/no response options listed below each question stem. The revised items also would attempt to provide more explanation of what is meant by “advanced-level” and “specialized”.
Ideally, a skip pattern would not be used here (following a “no” response to question 11a) due to respondents’ difficulty following them on the paper instrument. NCES might instead consider moving question 11a to be the last item in this series; it is anticipated that most respondents will respond “no” to questions 11b-d, which allows for question 11a to be asked subsequently without it seeming like an illogical follow-up.
LH: Q11a should be its own Q (see metadata version). I suggest combining “specialized” and “advanced level” into Q11c. Then have Q11b and Q11c each be their own Q, as AIR suggests. But I do think we need to keep Q11a first, with a skip, because if someone is reporting their first CN in a field, Q11b and Q11c do not apply to them. Skipping won’t be an issue in the 2017 web test (although we do need to consider what happens in 2019…)
Result: New question 11 (updated language to match Q10 – for that kind of work); also updated language to say “most important..”; New Q12 (Q11b-d) changed stem to “Is your most important CN…” Updated Q12a (old Q11b).
Finally, in round 2, we may want to add probes that ask respondents how they define “field”. If their definitions do not line up with NCES’s intended meaning, then NCES might want to consider adding examples in parentheses at the end of item 11a. We should continue to probe on respondents’ understanding of “specialized” and “advanced-level” and their ability to map those terms to their personal experiences.
LH: Yeah, let’s probe. I’m worried about this.
Result: We will continue to probe on these terms.
Respondents reported various problems when answering questions, such as question 11, that use the “Mark [X] ONE box for EACH ITEM below” structure. One respondent marked “no” for some items she reported aloud as “yes”, and vice versa. The interviewer thought the respondent lost track of which box was “yes” and which box was “no”, especially when there was a long list of sub-items. Another respondent said that for question 11 it was tricky to connect the main question text with each of the sub-items, so he had to remember to go back and re-read the first part of the question each time for the sub-item, and thus, the whole question, to make sense. Two respondents seemed to think this question followed a “mark all that apply” structure, marking items only when they applied to them and leaving anything that would be a “no” blank. However, this type of response pattern would be corrected in the data processing stage; when respondents mark “yes” to at least one item and leave all others blank, the blank items will be recoded to “no”. This comment also applies to questions 22, 28, and 34.
As noted above, to reduce cognitive burden, NCES might want to break question 11 into several items that each comprise a full sentence. In future rounds, we should continue to monitor whether respondents have trouble responding to the other items that are presented in this format.
LH: Agreed. I noticed at least one R would answer each item in the list until she got a “yes”, then she skipped the rest of the items. She seemed to think she just needed to find the one response that applied to her. Sigh.
Question 12 asks respondents if their certification or license can be revoked. One respondent understood “any reason” as “an unspecified” reason, so she answered “no” because the reasons that her license can be revoked are very specific. However, the interviewer thought she should have said “yes” because she listed reasons why it could be revoked. This comment also applies to questions 23 and 29.
Since this confusion was only mentioned by one respondent, we might want start by adding a probe for any respondents who report “no” in round 2. If this issue turns out to be more widespread, NCES might want to consider dropping “for any reason” from the question.
LH: I am in favor of dropping “for any reason”. Sharon suggested replacing it with “for due cause”. That language is technically correct but I worry it might confuse some people.
Result – dropped “for any reason.”
Question 15 asks respondents about the sources that help pay for their certification or license. While respondents were often able to answer this question with little difficulty, some were unsure how to classify the provider of their loan/scholarship. For example, one respondent wasn’t sure if her loan from Navy Federal Credit Union would be considered “private.” Another respondent reported receiving a scholarship from a school; however, upon probing, the interviewer found out that Goodwill and the community college worked in tandem on the program and that the funding was technically from Goodwill, not the community college. This comment also applies to question 44.
We should continue to monitor respondent difficulty identifying the source of loan, scholarship, and grant funding. If this appears to be a common issue, then NCES might want to consider removing references to who exactly provided these types of funding (government, school, etc.).
LH: We have shortened and simplified these response options (see metadata version).
Result: updated to the more streamlined question and added a response option for “Your own resources”
Three respondents had problems with the skips on question 18 (Do you have another currently active certification or license?) and 24 (Do you have another currently active certification or license?). Two respondents marked “no” to question 18 and should have skipped to question 30 (“Preparation for Certifications and Licenses” section). However, both respondents continued on to read question 19. One of them correctly skipped questions 19 and 20, but then answered questions 21-23 instead of skipping to question 30. The other respondent read questions 19-21 and started scanning the page for an indication of what to do. After reading question 18 for the third time, the respondent saw the skip and continued with the survey. A third respondent had the same problem, but in this case, with question 24. There were a few other instances of not noticing skip instructions and incorrectly continuing onto the next question; however none of these led to the same level of confusion as the previously described mistakes.
Respondents’ continued difficultly with skip patterns supports the importance of minimizing their use on the ATES paper form.
LH: So the “If yes” in the front of the Q stem does not seem to help. Rats. Should we try adding non-bolded, parenthetical text at the end of the stem saying something like “(If you do not have a second certification or license, go to question 30)”? (This would only apply to the 2019 PAPI instrument, but we could test it, or at least start testing it, now)
The respondents were rather consistent in their definitions of “currently working on” for question 30 (Are you currently working on any of the following?). Most respondents defined it as what they were doing right now, at the present moment.
We may want to continue to probe on this in future rounds to confirm that respondents are not interpreting the term more narrowly than intended.
LH: I think we could stop probing on this and just probe if Rs seem to hesitate or seem confused when answering the question.
Result – update protocol to remove the probe on all.
Respondents generally said that a “new” certification or license either meant something they never had before or something that is in a different field than what they are working in now. One respondent explained that she has never done anything with coaching before, so the leadership coaching certification is something she would consider to be “new.”
Respondents seem to correctly understand this term, so it may be possible to remove this probe from future interviews.
LH: I agree that we can stop probing on this.
Result – update protocol to remove the probe on all.
The skip on question 30 seemed to be particularly problematic. Three respondents answered “yes” to question 30b (Getting a new professional certification or license) and incorrectly skipped to question 47 (In the past five years, have you seriously considered working on a new professional certification or license that you did not end up getting?) as if they had answered “no” to question 30b. On all three occasions the interviewers had to stop the participants and ask them to answer the rest of the questions in the section.
As noted above, respondents’ continued difficultly with skip patterns supports the importance of minimizing their use on the ATES paper form. We also will review the presentation of the skip instructions for this item and attempt to clarify the instructions prior to round 2.
LH: It’s weird that here we get OVER-skipping, when we usually get UNDER-skipping. I suppose researchers have tried using an arrow to a box that says “Continue” on the “yes” responses (in addition to the “no” arrow and “go to” box), but that doesn’t work either? Too confusing perhaps? I’m stumped.
Result – Possibly move the arrow to the right of the “yes” options but would not work if the “yes/no” order is changed then it would have to stay on the left.
Respondents were asked how they identified the “most important” new certification or license on which they were working. Many respondents remarked that they were only working on one, so it was defaulted to be their “most important”; however, they all had a similar definition when probed. Respondents explained that the “most important” certification or license would be the one that is their top priority, or most in line with their interests or passions. One respondent defined it as the one that “yields the best results professionally.”
Since respondents seem to have a consistent and accurate understanding of this phase, we may be able to remove this probe from future rounds of testing.
LH: I agree that we can stop probing on this.
Result – update protocol to remove the probe on all.
Question 35 asks respondents when they started working on the certification or license. Most respondents defined the date for “starting to work on” their certification or license as either when they enrolled in classes or actually attended their first class.
As noted above, most respondents based their response on when they signed up for or started classes. However, one respondent reported the date that she passed the first of several required exams, though one of the requirements for her certification or license was having a certain number of college credits, which she had to have started prior to taking that first test (based on the reported date). Another respondent reported the date that she was hired by the employer that wanted her to get the certification or license.
Due to the relatively small number of potential issues, we should continue to probe on this item but no changes are suggested at this time.
Question 36 asks respondents when they expect to get the certification or license they are working on. One respondent reported that she expected to get the credential when her course ended. However, she later referenced also having to take a test to get the credential, which she did not seem to have taken into account when deciding on her expected completion date. A few other respondents also seemed to use the end of their courses as their expected completion date. Two respondents expressed uncertainty over the month that they would get the certification or license.
We may want to probe more about how respondents are determining the expected end date. Are they taking into account things like having to pass a test, which would likely occur after having completed coursework? We should also continue to monitor whether respondents are able to report an expected month of completion – and how confident they feel in that report.
Result - continue probing about when they get their CN.
Question 37 asks respondents about the requirements for getting a certification or license. Of the available options, the most commonly reported requirements for getting the certification or license were taking classes and passing a written exam. In the cases where the certification or license was issued by a federal, state, or local agency, respondents mentioned also having to pass an exam in front of the board. Respondents generally seemed to understand what was being asked of them, with a few exceptions. In response to question 37f (submitting a portfolio of work), one respondent reported she was required to submit a portfolio of work to show documentation of having completed each of the other requirements; though she described the intended meaning of the question correctly when discussing the concept of a portfolio, she reported incorrectly about herself. Another respondent reported that she is required to work 85 hours in the field before she can receive her CNA, but explained that she marked “no” to question 37c (working a minimum number of years in the field) because the question specifically asks about years.
NCES might want to consider changing question 37c to “working a minimum amount of time in the field” in response to this issue.
LH: Good idea. But the new version of this question (see metadata version) does not include this option so we don’t need to worry about it.
Result – Item has been removed.
Question 38 asks respondents about their things they may have done as part of working on their certification or license. Even though one respondent mentioned she was required to have a certain number of college credits to get her certification or license (credits which she had gotten as part of her bachelor’s), she said “no” she did not enroll in a degree program as part of working on the certification or license (question 38d). A few respondents expressed concern over the fact that the question did not ask about things that they were in the in the process of working on, as opposed to just asking about things they have already completed. One respondent suggested that it might be helpful to ask how far along they are in the process for each sub-item; there are some things that she is in the process of doing, but that she hasn’t completed yet. Another felt there was really a question for people in the “intermediate stage” or who were in the process. A third respondent felt that it would be more logical to flip questions 37 and 38 to match their chronological order. She viewed question 38 as being what you were currently working on (what she had already done), compared to question 37 that asked about what she would have to eventually do in the future to get the license or certification.
We should continue to monitor respondents’ reaction to not being able to report that they are in the process of completing requirements. If this compromises their ability to respond to the question, then NCES may want to consider revisions to the response options for this item. However, if respondents are still able to answer the item as-is and NCES is not interested in things respondents are in the process of completing, then the response options could remain as they are now. The respondent’s incorrect response to question 38d suggests that NCES might want to consider revising this text to something like “participated in…”
LH: This question should be dropped (see metadata version).
Result – Item has been removed.
Questions 39 to 42 ask respondents if they had taken classes or trainings as part of working on their certification or license. Most respondents did not have trouble responding to these questions. One respondent expressed some uncertainty as to whether she should count a seven-part course as one class or as seven classes. Another respondent questioned whether she should include college courses taken as part of obtaining a college degree or limit her response to classes taken after she had received her degree.
Due to the relatively small number of issues with these items we should continue to probe on them, but no changes are suggested at this time.
LH: In Q39 and Q40, should we delete “or trainings”? Also, note that one R reported 1class, but that class lasted for a year. Another R in the same situation reported 7 classes cuz the class had 7 sections. So I’m not sure what these “number of…:” questions are going to give us—I think these data might be more misleading than helpful. But we can keep testing for now.
Result: Deleting these question but adding in a level of effort question in the CN section.
Question 46 asks why respondents are working on a certification or license. Respondents did not have any difficulty articulating the difference between “a job in the same field” and “a job in a new field”. Respondents explained that something in the “same field” would be within the same industry, but with a different employer perhaps. Respondents also saw a distinction between employer and field requirements; they explained that while there might be general requirements to work in an industry, individual employers might have their own specific requirements. Respondents also understood the meaning of “marketable to employers and clients”. In their own words, they described this as making you “more attractive to employers” and “being more competitive, having more credentials, and seeming more knowledgeable.” Another explained that “a client or employer would choose her over a similarly qualified individual because she has the certification and they do not.”
As respondents seem to be consistently and correctly understanding these terms, it may be possible to remove some of these probes from the protocol.
LH: New version of Q (see metadata version) has shorter list, so we definitely will need fewer probes. However, I would like to keep probing on the difference between satisfying requirements in the field and meeting employer requirements.
Result: (Q45 – change “Did or will” to “Does”); update Q46 with the shorter list and will continue to probe.
One respondent did struggle with the term “field” because she wasn’t sure how narrow it was supposed to be. For example, would accounting be her field, or business in general? She stated that this would impact her answer. If she were to consider business as her field, then yes she wants a different job in the same field, but if it’s accounting, then no, she does not, since she is planning to work in management consulting.
As noted above, it may be helpful to include probes in future interviews that ask respondents how they define and understand “field”.
LH: I think the only remaining use of “field” in the new version will be OK. But I’m OK if you want to probe on it.
When asked if any reasons were missing, one respondent said the economy, as he/she was preparing to earn a new a credential in case of an economic downturn. Another said that he/she simply wanted to learn the technology taught in the program.
We should continue to offer respondents the opportunity to report additional reasons in future interviews. Economy/economic downturn seems as though it could technically fall under some of the existing response options, so we suggest waiting to see if other respondents also mention it as a separate reason.
LH: We have shortened the list of reasons and deleted the “other” response but I agree that it would be useful to ask Rs if we left out any reasons. The interviews did include a few people who were switching jobs and our first response option (“to satisfy the requirements for working in my field”) doesn’t seem quite right for them—should it be “to satisfy the requirements for working in [the][that] field”?
LH: This section is being dropped (see metadata version).
Result: Removed this section.
When probed, most respondents defined “seriously considered” to mean something they have taken action on. One respondent said it meant “really doing something about it”, such as talking to banks about loans, looking at grants and scholarships, looking up schools, etc. Another respondent said that it meant that you started the process but didn’t finish it. For example she said enrolling in school or putting money down would be considered “serious” whereas just doing the research online and thinking about it would not. The exception was a respondent who counted a certification she had found online the previous week but had not taken any action toward getting.
Since this is a new item, we should continue to probe on this term to ensure respondents are understanding it as intended.
A couple of respondents did have trouble understanding the meaning of the question. One said that combining “considered” with “did not end up getting” could really throw people off, because everyone is always considering things. This respondent suggested phrasing it as “pursued but did not end up getting.” The other respondent said that he thought this question was asking if you had tried to get a certification and failed to get it for whatever reason. Though the respondent eventually marked “yes”, he struggled with his response because he didn’t “fail” to get it, he just never started the process.
We suggest continuing to monitor how respondents react to this item. If confusion seems to be widespread, NCES could consider simplifying the item so that it only asks if the respondent has seriously considered working on a new certification or license in the past five years. The response options could then become: no; yes, and I ended up getting it; yes, but I did not end up getting it. Only respondents who selected the final response option would be asked the follow-up items.
Question 48 asks respondents about barriers they faced when they considered getting a new certification or license. Respondents defined “barriers” as something that holds you back from getting the certification or license. One respondent said that when she thinks of barriers, cost and time come to her mind.
Given the small number of respondents who provided feedback on this item, we many want to continue to probe here.
Due to either time constraints or the question not being applicable to them, only three respondents provided feedback on this item. The main barriers identified were “time” and “work responsibilities”. “Cost”, “family responsibilities”, and “personal problems” were each mentioned on one occasion. One respondent added “age” as another barrier to getting a new certification or license.
To get additional feedback on this item (if we continue to find that few respondents say “yes” to question 47), we may consider asking any respondents who report having a certification or license (or being in the process of getting one) about their reactions to the barriers listed here. We may want to specifically probe respondents who mention “time” to see if anything specific is making them feel like they do not have enough time and to see if they can map their “lack of time” onto some of the existing options that are intended to represent lack of time.
At the end of the interview, respondents were asked if they had any additional feedback about the survey that was not already covered. One respondent suggested adding space for more credentials as some people have more than three. Another respondent commented that the survey was long. He/she also said that the survey seemed to be intended for those who had completed a certification or license, not those currently in the process. Another respondent felt the same way, saying that there wasn’t really a question for those people in the “intermediate-stage” or still in the process.
|
Round 2 Memo |
Date: |
April 12, 2016 |
To: |
Sharon Boivin (NCES); Lisa Hudson (NCES) |
CC: |
Sarah Grady (NCES) |
From |
Kirsten Manville (AIR); Meghan McQuiggan (AIR); Stephanie Cronen (AIR); Danielle Battle (AIR); Rebecca Medway (AIR); Cameron McPhee (AIR); Stacey Bielick (AIR) |
Re: |
NHES: 2016 ATES Cognitive Interviews, Round 2 |
This memo presents the issues found during the second round of cognitive interviewing of the Adult Training and Education Survey (ATES) from March 28th through April 6th, 2016. This memo also includes suggestions from AIR in response to these issues, as well as comments from respondents who proposed potential solutions.
Eighteen interviews were carried out by AIR interviewers using an abbreviated ATES instrument that included three sections: Education; Certifications and Licenses; and Preparation for New Certifications and Licenses. However, the data quality of one interview was questionable (there were concerns that the respondent may have been fabricating information), so it was excluded from the analysis. Therefore, the memo reports findings for the remaining 17 interviews.
Interviews were conducted in Washington, DC. All interviews lasted about an hour, were conducted in English, and were audio recorded. Eleven respondents reported having a currently active certification or license and 11 respondents reported that they are currently working on getting a new certification or license.
Question 2 asks respondents about the field of study for the highest level of school they have completed. A few of the respondents reported the field of study for something other than the highest level of education they reported having completed. For example, one respondent answered for the schooling he/she was currently pursuing but had not yet completed. When answering question 1 he/she said he/she was looking for “some college” but did not see it in the response options, so he/she reported what he/she was currently pursuing (his Associate’s). Another respondent marked two response options (“general studies, no major or undeclared major” and law and “legal studies”). The highest level of school he/she had completed was high school, and the respondent explained that he/she thought of those classes as “general.” He/she marked “law or legal studies” because that is the field he/she currently works in. Overall, two respondents reported fields that they work in, but don’t yet have a certification or license in. Four respondents marked fields of their certification or licenses instead of their highest level of education. Two respondents marked the incorrect level of education. In addition, two respondents marked two fields despite instructions to mark just one.
Several of these errors are likely due to the salience of their certifications or licenses from recruitment materials. However, we may want to continue probing in round 3 about (1) which degree/credential the respondent is reporting on, and (2) the respondent’s exact field of study. If reporting on the wrong level of education proves to be a widespread issue, NCES might consider underlining “highest level of school” or “completed” or adding an instruction to report on the level of education reported in question 1. If selecting the wrong response option proves to be a widespread issue in our future testing for 2019, NCES might consider further research into the response options used in similar items in other federal surveys or conducting qualitative research to better understand how respondents classify different fields of study. Also, note that on the web, this item can be programmed to read differently depending on what the person selected in Q1. E.g., refer to HS, AA, BA, etc.
This item asks respondents if they are currently enrolled in a college, technical, or trade school. In general, respondents did not have difficulty with the item. One respondent was initially confused as he/she was not taking classes, but was still working on the training hours needed to qualify for the barbers’ exam. At first he/she was unsure how to answer, but eventually selected “no,” which was the correct response given his/her situation. Another respondent, who did not seem to be taking classes as he/she recently received a certification and was not working on another, marked “yes, as a part-time student”. He/she explained that he/she missed the graduation ceremony last semester, so he/she believed he/she was still enrolled; if he/she wanted to go back and take more classes, he/she would not have to “re-test” with the school.
If respondents reported being enrolled in school (either part-time or full-time), interviewers asked about their major or field of study. Some of the fields reported included hospitality, customer service, computer science, intellectual property, dance therapy, and nursing.
Since the majority of respondents had no issues with this item and it is well-tested, we do not recommend any changes.
Two respondents exhibited some confusion on this item, asking if they had a currently active certification or license. One respondent thought that a certification was a degree, so he/she mentioned his/her associate’s degree. However, after reading the question a couple times, he/she correctly marked “no”. Another respondent, who had reported having a construction certification in the recruitment screener, marked “no” for this item. When probed, the respondent said he/she incorrectly marked “no”; he/she was focused on the new certification he/she was working on and had forgotten about the construction certification. Another respondent was hesitant because he/she received a teaching credential that was supposed to be a “life license”. However, the state changed their law to require continuing education, thus it was no longer technically active. The respondent correctly marked “no”.
Two respondents missed the skips off of this item and continued onto Q7 after reporting they had no active certifications or licenses (other respondents made the same mistake on Q21 and Q28). Interviewers re-directed respondents to the skip logic so they did not have to answer items that did not pertain to them.
Since the majority of respondents had no issues with this item and it is well-tested, we do not recommend any changes.
Question 10/24/31/38 asks respondents if their certification or license is required by a government agency. One respondent appeared confused when answering this question. He/she stated that he/she wasn’t sure what a “state board” was, so he/she marked “don’t know” for his/her answer. When probed he/she was able to explain that the certification was not required for his/her job. A few other respondents hesitated when answering this question, but most respondents did not experience difficulty when answering this question.
The majority of respondents had no issues with this item; however, it does not appear to be an improvement over the 2016 version of the item. Therefore, we suggest either continued testing or reverting to the 2016 version for continuity.
Question 11/25/32/39 asks respondents if their most important certification or license is their first one for that kind of work. Interviewers also asked respondents what this question was asking in their own words. Most respondents had no problems, saying that it was asking if it was your first one for that “field” or “line” of work, regardless of whether they had other certifications. One respondent, who had the same certification from multiple companies, incorrectly marked “yes” for this item, interpreting the question as asking about the first certification from his/her current employer. Another respondent had the same license in multiple states, but marked “yes” here as the credentials had slightly different names. Another respondent explained that this question could be interpreted in multiple ways. First, he/she described the intended meaning, if it was your first “in this space.” However, he/she said people might also think it’s asking if it’s your first “attempt” at the certification; i.e., if you did not pass the credentialing exam the first time and are preparing to take it again.
Two respondents who marked “yes” to this item missed the skip pattern and proceeded to answer the next question, which was only supposed to be answered by those who had multiple certifications or licenses in that field.
Since a few respondents are interpreting this question in multiple ways, we can either continue testing or try “Is this certification or license your only one for that kind of work?” However, this language will make it more difficult to understand “stacked” credentials.
Question 12/26/33/40 asks respondents to describe their certification or license. When probed to give a definition of the terms “specialized” and “advanced-level”, most respondents defined “specialized” as something that is industry or field specific, a sub-component within an industry. Two respondents described “specialized” as a certification that indicates that you are an expert in the field. “Advanced-level” was uniformly described as something that acknowledges a higher-level of proficiency in a field, or an upgrade to a certification or license you already have. One respondent defined “advanced-level” as meaning something that required a higher-degree or more education. One respondent said no to all, but said that he/she thought most advanced-level certifications were specialized.
A couple of respondents had trouble with the question format. Rather than marking either “yes” or “no” for each sub-item, two respondents marked only “yes” to sub-items that applied to them and left sub-items they would have marked “no” to blank. However, this type of response pattern would be corrected during data editing.
One respondent said that the question was tricky because he/she had to remember to “add to the question” to each option (he/she was referring to connecting the stem and the sub-items). He/she said that asking if your certification or license was a specialized certification or license (when read all together), was like “double talk”.
After seeing the language about “different state”, one respondent suggested a question be added asking if the credential was valid across the country or even internationally.
We suggest deleting the “specialized” option for round 3. In regards to the structural issue, making each item a separate question would likely solve this problem; we could test this in the 2019 interviews (it is not necessary for the web test). The “different state” option could be revised to “a different state or nationally” if NCES would like to include the “nationally” language. However, that is more certification-oriented and may cause more confusion than clarity.
This question asks about the types of classes respondents may have taken in preparing for their certification or license. Most respondents had no difficulty and described taking classes through the DC government, trade schools, a security company, a financial company, and an online college, among others. Respondents described programs in a variety of fields, with lengths ranging from one week to nine months.
Two respondents had difficulty with this item. One respondent who had a CPR and first aid certification marked “yes” to all of the sub-options. However, after probing it was evident that he/she was not reporting about the certification, but rather about the requirements for his/her job. The respondent described classes he/she took in high school and college, prior to preparing for the certification. When asked what he/she had to do specifically for the CPR/first aid certification, the respondent said it took 5-6 months to get. The classes, which were on a variety of topics, lasted one hour and were three days a week.
Another respondent, who recently received a customer service certification, initially marked “no” to “college, technical, or trade school”, but “yes” to “company, association or union”. When first answering, it seemed like he/she was focusing on the “trade school” language, which is why he/she marked “no” (the respondent went to a university). In marking “yes” for “company, association or union”, the respondent added that most job settings teach customer service, which he/she saw as a company. When probed on these items, the respondent was unsure why he/she put these answers. The respondent said that after reading the first option about “classes during high school,” he/she thought the question was asking only about the past; i.e., if you took classes in college related to this field, not recent classes at a college. The respondent then explained that he/she took one customer service class at a university in DC in order to get the certification.
One respondent suggested that this question have an option to mark studying using online resources.
Since the majority of respondents had no issues with this item and it is well-tested, we do not recommend any changes.
This item asks respondents the number of hours they spent in classroom training in preparation for the certification. The response option selected most often was 40-159 hours, although all of the other options were selected by at least one respondent (except for 960 hours or more). Most respondents were able to estimate based on the frequency and duration of their classes. However, not all respondents included the same hours in their estimates; one reported all of the hours he/she attended the college (which were not all for the certification). Another respondent, who took both classes from a company and an online school, only included the hours provided by the company. Finally, another respondent said that they were reporting the number of credit hours needed to graduate from the program. However, after probing, they said they would not change their response as credit hours were based on hours spent in the classroom.
Since the majority of respondents had no major issues with this item and it is reasonably well-tested, we do not recommend any changes. We do, however, recognize that there will be more reporting error with this item than with some of the other items.
Question 17 asks respondents about the process for getting their most important certification or license. A few respondents had trouble understanding sub-item a. “enrolled in a certificate program from a college, technical school, or trade school.” One respondent indicated that he/she wasn’t sure if this was referencing a program separate from his/her certification/licensing class and ended up incorrectly marking yes to report the licensing class he/she took at a trade school. Another respondent misreported by marking “yes” in reference to enrolling at a university to get his/her law degree. A third respondent thought this was asking if you attempted to start another credential program while you were in the process of getting the first one (i.e., getting two at one time). He/she ultimately correctly marked “no,” however, it was only because he/she wanted to do an Excel program at the same time, but was unable to register for it. Two respondents were a little confused about what “submitted a portfolio of work” meant. One of these respondents marked yes, because he/she had to submit a resume, while the other ended up marking “no” and used the “other (specify)” response option to write in that he/she had to provide documentation of work experience.
Two respondents struggled with the structure of this item. Both respondents marked “yes” for one of the sub-items and left the remaining sub-items blank. Both respondents indicated that they thought they only needed to mark “yes” for sub-items that applied to them, however after being probed, one of the respondents indicated that he/she would have marked “yes” to the rest of the sub-items.
If NCES is not comfortable having all unanswered yes/no items recoded to “no,” we recommend splitting the checklist items up into separate items for the 2019 testing; this has been an ongoing issue, and although not all respondents make this mistake, the pattern has been consistent. As for the wording of sub-item 17a, we suggest changing to “postsecondary certificates and degrees” (if that is the focus of this option) and consider whether it would be enough to ask about “classes” from the same providers or whether the goal is really to measure the role of postsecondary certificates in preparing adults for certifications and licenses.
Question 18/34 asks respondents about the funding sources used to help pay for their certification or license. One respondent had significant difficulties with this question. This respondent earned his/her CPR/First Aid certification while she was enrolled at a college for seven months. Thus, when answering this question he/she was reporting sources for his/her entire college tuition, not specifically for his/her certification (as the costs were covered in the tuition). The respondent reported receiving a grant and loan to help pay for the tuition, as well as reported employer contributions. When probed about the employer contributions, the respondent reported that he/she was referencing his/her time spent working at a fast-food restaurant during his/her enrollment to help pay off the loan. A second respondent initially misreported on this question, by marking “no” to all sub-items. This respondent said he/she read the sub-items fairly quickly and missed “your own resources;” he/she said he/she should have marked “yes” to this.
One respondent viewed this question as a “mark all that apply” question and only marked “yes” to “your own resources,” as she paid for the certification entirely out of pocket, and left the rest of the sub-items blank. This issue would be fixed in the data editing process, as the respondent’s blank answers would be marked as “no”.
Since the majority of respondents had no issues with this item, we do not recommend any changes.
This question was not specifically probed on, however one respondent suggested that an item be added that asks the usefulness of the credential in helping you to switch fields.
Since the majority of respondents had no issues with this item and it is well-tested, we do not recommend any changes.
When reading this item (“do you have another currently active…”) one respondent was confused because he/she wasn’t sure if they were referencing back to the first license (do you have another currently active…) or asking about a different one. Once he/she read Q29 that mentioned explicitly “third” he/she realized that they were asking about an additional license and was able to correctly respond to Q28 and skip to the following section.
Since the majority of respondents had no issues with this item and it is well-tested, we do not recommend any changes.
Question 35 asks respondents if they are currently working on any certifications or licenses. One respondent was very confused by this question. When probed, he/she said he/she thought sub-item b. ”Getting a new professional certification or license” was asking if he/she had to get a new certification for one he/she already had; since he/she is working towards his first one, he/she marked “no.” Another respondent misreported information on this question. He/she marked “yes” for maintaining/renewing a certification or license, because he/she wants to “renew” his/her New York security license so he/she can use it in DC.
One respondent missed the skip on this question. The respondent marked “no” to sub-item b, but missed the skip instructing him/her to go to the end of the survey. He/she continued reading (but not answering) the next few questions, before the interviewer stopped him/her and instructed him/her to end the survey.
Since the majority of respondents had no issues with this item, we do not recommend any changes.
This question asks respondents about their most important new certification or license. No issues were reported for this question.
Since no respondents had no issues with this item, we do not recommend any changes.
Question 41 asks respondents when they started working on their certification or license. Only two respondents had difficulty marking a date for this question. One respondent, who got the certification in 2008, could not remember the month when the classes started so he/she put “06” (June) since it was in the middle of the year. Another respondent asked what was meant by “start working on a certification or license.” He/she said that he/she was confused because there were two separate dates; the date when he/she started school and the date when he/she initiated the process of getting the license. He/she described these as different steps, as some individuals complete the school portion without getting the license. The respondent reported starting classes in 2013, but wrote 01/2016.
Interviewers asked respondents what they considered as starting to work on the credential. Respondents had a variety of responses to this probe. One said they reported when they registered for the certification, a couple said they indicated when they started classes, one said it was when they were accepted to the program, and a couple said it was when they inquired into the program/school. Another respondent, who completed the first of two phases of a certification, wrote the start date for the second phase as opposed to when he/she started the first portion.
It may not be possible to get all respondents to the same starting point, but we suggest providing some additional guidance, such as “please report the date you started any activity intended to prepare you for getting your certification or license.”
The next question asked respondents when they expect to complete the credential. Respondents had fewer challenges with this item than the prior. Some respondents reported when they planned to take (and hopefully pass) their certification or licensing exams. Others said that they were in programs of specific lengths, so they knew when it would be over. Others, who were not on specific timelines, were able to estimate their anticipated completion date.
Since the majority of respondents had no issues with this item, we do not recommend any changes.
This item asks if respondents received paid release time to work on their certification or license. Most respondents either marked “no” or replied that they had not been employed while working on the credential. Only one respondent marked “yes”. Interviewers probed on the meaning of “paid release time”. One respondent said it would be if your employer allowed you to attend a training during work hours without taking vacation or sick days. Another said it was like vacation time; you are paid but do not have to be at work. Another said it is when your employer gives you time to go to school. Yet another respondent said it was like a sabbatical leave. One respondent, who was self-employed, said the question was tricky as it did not really apply to his/her situation. The respondent who marked “yes” had an interesting interpretation: he/she saw the two sentences in the question as different from one another. He/she said that “paid release time” is like paid time off, where you are paid to take time off to prepare for the certification. He/she said his/her employer would not allow this. However, he/she said that the second sentence, which clarifies that the employer lets you work on the credential as part of your work hours, would apply to his/her situation as he/she is allowed to work on it at work if he/she has spare time.
Respondents seemed to have difficulty with the wording of this item, in particular the “that is” language. No respondent specifically mentioned it, but multiple respondents had to read it a couple times. The second sentence seemed slightly awkward when read aloud.
Given the issues with this item, we suggest dropping the first sentence and the “that is” transition and only asking the second question. The revised question would read: “Does your employer let you work on this certification or license as part of your paid work hours?”
In this question, respondents are asked to select why they are pursuing their credential. For option a, “satisfy the requirements for working in your field,” one respondent was confused; he/she was unsure if it was referring to the field he/she currently works in or is hoping to work in. The respondent reported “no” about the field he/she was currently in, but said he/she would have marked “yes” if it was about the new field they wanted to work in. Another respondent had a similar reaction; although he/she was working towards and IT certification, he/she marked “no” to option a as he/she was not yet working in that field.
When asked if respondents saw options a and d (“employer requirements and expectations”) differently, all but one said they did (and many responded differently to these two sub-items). Many respondents explained that requirements for the field were things you had to do in order to have the type of job (i.e., a lawyer cannot practice without passing the bar exam), whereas employer expectations were things a particular company or employer might require in addition. One respondent described employer expectations as “a certain level beyond the minimum”. Another said that employers might require certain things if you want to get promoted. One respondent suggested making “employer” plural in option d. Another respondent, who was self-employed, felt that the options did not apply to him/her.
Option b says “to make you more marketable to employers or clients”. Two respondents struggled to read the word “marketable”. Interviewers probed on what other words respondents would use to describe this reason. Some suggested terms included: “more hirable”, “sellable”, “branding oneself”, “additional advantage”, “more appealing”, and “open up to more opportunities”.
Finally, respondents were asked if any reasons for pursuing their credential were missing from the list. Some other suggested reasons included: “to pursue a different line of work”, “leisure or passion”, and “increase your skills”.
Since few respondents have had difficulty with option b (“marketable to employers or clients”), as well as distinguishing between a and d, we recommend discontinuing probing on these items. Rather, interviewers should probe on the revised versions of a and new b. These two items would read: “to satisfy the requirements for working in your current field” and “to satisfy the requirements for working in a different field”.
At the end of the interview, respondents are asked if they have any additional feedback on the questionnaire. Overall, respondents thought the questions were clear and straightforward. One respondent said he/she did not care for the “most-important” language throughout the survey. He/she had two certifications and did not really see one as more important than the other; he/she explained that some certifications are helpful, but only if you have other certain skills (he/she saw his/her two certifications as complementary to one another). He/she said he/she would have preferred if the survey asked for most recent credential first, followed by others, similar to how resumes generally list your most recent job first. One respondent said he/she was surprised to not see any demographic questions on the survey (which are in the background section of the full ATES survey, just not administered during cognitive testing).
No additional changes are recommended.
|
Round 3 Memo |
Date: |
June 6th, 2016 |
To: |
Sharon Boivin (NCES); Lisa Hudson (NCES) |
CC: |
Sarah Grady (NCES) |
From |
Meghan McQuiggan (AIR); Stephanie Cronen (AIR); Danielle Battle (AIR); Rebecca Medway (AIR); Cameron McPhee (AIR); Stacey Bielick (AIR) |
Re: |
NHES: 2016 ATES Cognitive Interviews, Round 3 |
This memo presents the issues found during the third round of cognitive interviewing of the Adult Training and Education Survey (ATES), which took place from April 18th through May 3rd, 2016. This memo also includes revisions that were made to the instrument for NHES:2017, as well as considerations for the 2019 administration.
Nineteen interviews were carried out by AIR interviewers using an abbreviated ATES instrument that included three sections: Education; Certifications and Licenses; and Preparation for New Certifications and Licenses. Interviews were conducted in Washington, DC. All interviews lasted about an hour, were conducted in English, and were audio recorded. Fifteen respondents reported having a currently active certification or license and nine respondents reported that they are currently working on getting a new certification or license.
Question 2 asks respondents about the field of study for the highest level of school they have completed. One respondent reported the field of study for something other than his/her highest level of education; he/she had completed a high school diploma, but wrote in “child care” as this was the field of his/her credential. Four respondents marked multiple response options. One of these respondents marked four options, saying that he/she started studying psychology but switched his/her major several times, and did not complete the Bachelor’s degree. Another respondent who had two Bachelor’s degrees marked two options, missing the instruction to pick only one. Another respondent explained that he/she didn’t select a major, but took courses in social services; thus he/she marked both “General studies, no major, or undeclared major” and “Social or human services or public administration”. Another respondent, who had a credential in massage therapy, marked “healthcare” and also wrote in “massage therapy”.
No changes were made to this item for the NHES:2017 web pilot.
Review the response distributions in the NHES:2016 and 2017 data and consider autocoding on the web. Consider additional testing before the 2019 administration.
This item asked respondents if they were currently enrolled in school. Almost all respondents said “no” and a few reported being enrolled part-time. No respondents had difficulty with this item.
No changes were made to this item for the NHES:2017 web pilot.
If the respondent reports that he/she is currently enrolled in school, consider asking for his/her major; this may be relevant to any certification he/she is working towards.
Question 10/26/34/42 asks respondents if their certification or license is required by a government agency. Several respondents had difficulty with this item in round 3. Two respondents marked “don’t know”. When probed, two additional respondents didn’t actually know who would be issuing their credential (one had marked “yes” and one had marked “no” for this item). Another, who correctly marked “yes” for this item, said that he/she didn’t know what a state licensing board was. One respondent incorrectly marked “yes”, explaining that the credential was issued by a local college.
This item will be administered as part of a split panel experiment for the NHES:2017 web pilot. One version will be the 2016 version: “…required by a federal, state, or local government agency (such as a state board)…”. The other will be the version tested in this round: “…required by a government agency (such as a state licensing board)…”
See how the split panel versions perform before deciding what wording to use for the 2019 administration.
Question 11/27/35/43 asks respondents if their certification or license is their only one for that kind of work. Interviewers asked respondents what this question was asking in their own words. Several respondents had difficulty with this item in round 3. Two respondents marked “yes”, even though they had two certifications for related work; one was a certified nursing assistant and the other had a lifeguarding certification (they both also had CPR certifications). However, they were thinking specifically about being a nursing assistant and lifeguard, respectively, and did not consider CPR as part of “that kind of work”. Another respondent, who had multiple education certifications, marked “yes”, explaining that it was his/her only credential for that specific area of education (teaching exceptional students). One respondent marked “no” (since he/she had had multiple credentials), but when probed, said he/she probably should have chosen “yes” (since the credentials were not in the same field). One respondent chose “no” despite previously mentioning it would be the first in that field for him/her. Another respondent marked “no” although he/she didn’t report any currently active credentials: he/she explained that in order to get a contractor’s license, you also needed a business license, which he/she saw as related. One respondent chose the correct answer, but said that he/she wasn’t sure if “that kind of work” meant his/her job field in general (environmental consulting) or the specific purpose of the certification (reading emissions).
One respondent noted that the skip pattern here was confusing; he/she found it odd that the skip was off of “no” in Q6, but off of “yes” in this item (Q11). Three respondents missed the skip off of this item; they should have skipped the following item, but answered it anyway.
Given respondents’ difficulty with this item, it was deleted for the NHES:2017 web pilot.
Consider more development and testing to improve the data quality associated with this item; figure out a way to capture the concept of a “portfolio of certifications,” if possible, but this may be a concept that we cannot measure in a self-report survey.
This item asks respondents if they have the same certification or license in another state. Two respondents who had CPR certifications were confused by this item; one said “no” but emphasized that it was confusing since the credential is valid everywhere; he/she described this as a “trick question”. The other said “yes” since it is nationally-recognized and can be used in all 50 states, but he/she wrote a question mark next to the item. One respondent was very confused by this item, uncertain why someone would have a certification in two states; this respondent later asked the interviewer why this question was in the survey. Another respondent incorrectly marked “yes”, explaining that they are planning to work on a certification in a different state than his/her current one. One respondent thought this item was the same as the previous question, which asks if the credential is your only one for that kind of work.
One respondent suggested that the survey should ask which states the credential is valid in.
Given respondents’ difficulty with this item, it was deleted for the NHES:2017 web pilot.
Consider more development and testing to improve the data quality associated with this item, although it may not be possible to capture this information in a self-report survey.
Question 13/29/37/45 asks respondents to describe their certification or license. Respondents were also probed on their interpretations of “entry-level”, “advanced”, and “specialized”. Many respondents were able to understand and articulate the intended meaning of the question. However, three respondents understood “entry level” as referring to a job (e.g. “low paid job,” “basic computer work”) and/or requiring low levels of education. One respondent said he/she didn’t know what “add-on or specialty” and “advanced” meant. Three respondents thought “specialty” and “advanced” were very similar or the same. Three respondents associated “advanced” with an advanced degree (e.g., MA or PhD). One respondent, who had a teaching certification, chose “entry-level” but would have preferred to call it a “professional certification”. Another respondent, who had a cosmetology certification, chose “advanced-level” as he/she reported that there is no entry-level credential in cosmetology.
Given respondents’ varied interpretations of this item, it was deleted for the NHES:2017 web pilot.
Consider more development and testing to improve the data quality associated with this item, although it may not be possible to capture this information in a self-report survey. We may instead want to consider a coding approach to determine “entry level,” “specialized” and “advanced” based on the name and field of the credential.
This question asks about the types of classes respondents may have taken in preparing for their certification or license. Most respondents had no difficulty and described taking classes through a community college, trade schools, and outside organizations, among others. Respondents described programs in a variety of fields, with lengths ranging from one week to seven months (not including respondents who reported their college classes). However, a few respondents had trouble with this item. One respondent initially marked “no” to all, despite having described taking a class; when probed, he/she said that the first option about high school confused him/her, making him/her think the questions were asking about classes taken in the past, not those in order to get the credential. This respondent also noted that he/she didn’t identify with “classes” as he/she saw it as more of a “training”. After probing, the respondent marked yes to “classes or training from a company, association, or other organization”. Another respondent reported all of the classes in this item that he/she had taken for multiple credentials, rather than just the most important one. Similarly, another respondent switched back and forth between reporting about his/her most important and another credential. Finally, one respondent said he/she took classes related to the childcare certification in high school, yet he/she did not mark “yes” for this option.
Two respondents only marked “yes” to item b, but left a and c blank.
To maintain consistency with other items, the question marks on each sub-item were deleted. Additionally, a new option c was added (“classes or training from the military”). The current sub-item c was moved to d.
See how this item performs in 2017 and consider additional testing before the 2019 administration.
This item asks respondents the number of hours they spent in classroom training in preparation for the certification. The response option selected most often in this round was “1 or more full time school years”, although all of the other options were selected by at least one respondent. Most respondents were able to estimate their response based on the frequency and duration of their classes. However, some respondents were confused. One respondent had trouble remembering the number of hours. Another was unsure if he/she should include college classes, which were helpful, but not required for getting a teaching certification. One respondent thought the language “to prepare for” was confusing; it made him/her think of things he/she did before starting the certification process. When probed on “clock hours”, most respondents understood the meaning. However, one respondent was confused, saying he/she thought it was referring to military time.
No changes were made to this item for the NHES:2017 web pilot. These response options will also be used for the same question in the “Certificates” section.
One outstanding issue is that some respondents are unsure whether or not to include everything they did to obtain the knowledge and skills to complete the certification (i.e. college classes), or just what they did to apply and study for the test. Therefore, an additional instruction to “include college classes” may be helpful. This was not added for the 2017 web pilot, but could be reconsidered for 2019.
Question 19 asks respondents about the process for getting their most important certification or license. Most respondents answered this question with no difficulty. One respondent marked “yes” to “demonstrated skills while on the job” but when probed, simply said that he/she was currently working in the field related to the credential. Two respondents also marked “yes” to “internship” and “demonstrating skills on the job” for the same activity. Finally, when probed about other requirements, one respondent said a background check. Another said volunteer hours and a research project.
One respondents marked “yes” for one of the sub-items and left the remaining sub-items blank.
Given the redundancy with sub-items c (“internship…”) and e (“demonstrated skills while on the job”), sub-item e was deleted for the 2017 web pilot.
See how this item performs in 2017 before deciding on item wording for the 2019 administration.
Question 20 asks respondents if they received their credential through a free city or state program. Most respondents demonstrated an understanding of the question and responded “no”. Three respondents said “yes” and reported that the programs were provided through TANF, a community college, and a local college.
No changes were made to this item for the NHES:2017 web pilot.
See how this item performs in 2017 before deciding on item wording for the 2019 administration.
Question 21 asks respondents about the funding sources used to help pay for their certification or license.
There were very few issues encountered with this item; only one respondent marked “yes” to scholarship after saying it was a free city or state program. When probed, he/she said there was no application process and it was free to everyone who qualified.
No changes were made to this item for the NHES:2017 web pilot.
See how this item performs in 2017 before deciding on item wording for the 2019 administration. Consider that some students are inclined to report their employer as a funding source if the company provided tuition support.
Some respondents were not necessarily referring to the correct credential when responding to the follow-up items in this section. One respondent switched between multiple existing licenses while responding to “most important”. Another switched between “currently active” license and “currently working on” while responding to the “currently active” section. One respondent got confused by “do not include business licenses…” and interpreted it as “only include business licenses…”; as a result, he/she did not report his/her license as a “currently active” credential. In addition, some respondents provided inconsistent or questionable responses related to the number of currently active certifications or licenses they had. Three respondents did not provide the correct number of certifications/licenses based on descriptions of the credentials they possessed. One respondent indicated having two certifications, but did not fill out the “second-most important” section; when probed, he/she said he/she didn’t see it as important to his/her work.
Question 39 asks respondents if they are currently working on any certifications or licenses. Almost no respondents had difficulty with this item. Only one respondent was confused because he/she thought “maintaining” and “getting a new certification or license” were the same.
One respondent was unsure if the skip arrow was only on 39b or if it referred to the entire item.
One respondent marked “yes” to 39b, but left 39a blank.
No changes were made to this item for the NHES:2017 web pilot.
See how this item performs in 2017 before deciding on item wording for the 2019 administration. A possible revision that has been discussed for consideration in 2019 is shown below. However, this version adds an extra skip pattern to the questionnaire. If analysis of the NHES:2016 data shows that skip patterns continue to cause confusion for respondents to the paper questionnaire, then we may want to look for options for 2019 that do not result in added skip patterns.
Are you currently working on getting a new certification or license?
Yes (continue)
No (skip section)
Which one of the following best describes what you are currently working on?
I am working on getting this certification or license for the first time (continue)
I already have this certification or license and am working on getting it in a new state (skip section)
I am working on renewing or reinstating a certification or license that I already have (skip section)
This item asks respondents to provide the name of the new credential they are working on. No respondents had difficulty with this item; they were all able to provide the name of the credential they were working on.
To shorten this item, the first sentence was deleted (“The next few questions ask about the new certification or license you are working on that you consider to be the most important”). The question now just asks (“What is the name of the most important new certification or license that you are working on?”).
See how this item performs in 2017 before deciding on item wording for the 2019 administration.
This item asked respondents if their credential could be revoked or suspended. All respondents said “yes” except for one respondent who said “don’t know”. However, this respondent was believed to be reporting about a certificate rather than a certification. Overall, the item worked well and respondents had no issues.
No changes were made to this item for the NHES:2017 web pilot.
See how this item performs in 2017 before deciding on item wording for the 2019 administration.
The next question asked respondents when they expect to complete the credential. Only two respondents had difficulty with this item; one wrote “2016” for the year, but said he/she didn’t know the month because he/she will receive the exam date from DC. Another respondent put dashes through the dates, saying there was no set time frame.
No changes were made to this item for the NHES:2017 web pilot.
See how this item performs in 2017 before deciding on item wording for the 2019 administration. In particular, examine the frequency of blank responses.
The next question asks respondents if their employer let them work on the credential as part of their paid work hours. In general, respondents had no problems with this item, but interpretations were varied. One respondent marked “yes”, explaining that he/she would be working as a nanny and would receive a stipend and tuition reimbursement from his/her employer. Two others marked “yes” and said that their jobs were providing the work hours needed to get the credential. Another respondent marked “does not apply” as the certification was not applicable to his/her current job.
For 2017, the order of the response options will be changed to “Yes”, “No”, “I am either self-employed or not working” (the “not applicable” text was deleted).
See how this item performs in 2017 before deciding on item wording for the 2019 administration.
In this question, respondents are asked to select why they are pursuing their credential. Respondents also had minimal difficulties with this item. One respondent had difficulty reading “marketable” and another defined it as “making friends and being nice to employers and clients”.
One respondent didn’t see the “mark one for each option” instruction and asked the interviewer if he/she could only mark one reason.
When asked if there were other reasons respondents were completing their credentials, they offered reasons related to personal growth, benefits, stability, or prestige, rather than employer-related reasons.
No changes were made to this item for the NHES:2017 web pilot.
See how this item performs in 2017 before deciding on item wording for the 2019 administration.
At the end of the interview, respondents are asked if they have any additional feedback on the questionnaire. One respondent mentioned the grid items, and said it took him/her a while to realize you are supposed to mark either “yes” or “no” for each one. He/she contrasted this with the “mark one only” items, noting the difference in format and saying it was a little inconsistent. One respondent commented that he/she didn’t like that the picture on the cover page was black and white and that it looked very dark. Another respondent reported that Q22 (“How useful has your most important certification or license been for each of the following?”) seemed repetitive and did not really apply to him/her. He/she mentioned he/she didn’t think the question was really helpful and did not see a difference between any of the sub-items.
1 Though these respondents may be reporting a certificate rather than a certification all comments have been included in analyses.
C-
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | swilkinson-flicker |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |