ED Response to Public Comment

Docket: ED-2014-ICCD-0071
2015 National Household Education Survey (NHES 2015) Full Scale Data Collection

Comment On: ED-2014-ICCD-0071-0005
Document: ED-2014-ICCD-0071-0008
Comment on FR Doc # 2014-15935

COMMENT

I'm happy to see that there is a second chance for those of us who hoped to comment on the first docket due in July but let it fall through on our summer to-do lists! I hope I'm not the only academic sitting on the sidelines eagerly waiting for data from this survey, because this is a truly important project that will fill a massive gap in our understanding of the extent and effectiveness of this country's educational and credentialing system If successful, it will be a gold mine for sociologists and other social scientists studying a range of topics, from membership associations to educational achievement to inequality.

However, I worry that the proposed CWS might be a missed opportunity to learn more about the characteristics of certifications themselves. The draft instrument is a very good start, but we need all the data we can get. NHES is not a great survey for learning about the demographic and labor market correlates of certification (and licensure) because it lacks the extensive and well-developed core items found in such surveys as the SIPP and ELS. Perhaps, then, it makes sense to emphasize NHES as a means of getting a broad picture of the certification landscape and characteristics of certifications across occupations. It is difficult to prioritize in such a space-limited environment as the NHES, but I would like to suggest a few points that would greatly enhance our understanding of this phenomenon that the NHES would be a uniquely suitable place to inquire about:

• Why did you choose this credential over others in this occupation?

• How many hours did you spend preparing for this credential?

• How much does this certification cost, in dollars?

• Is it a generalist certification covering most aspects of your occupation, or more specialized?

• How would you rate the difficulty of earning this credential?

• How closely related was the curriculum of the certification program related to your day-to-day work?

• Is this certification issued by a non-profit association?

• If so, are you a member of that association?

Some further inspiration for items about certification characteristics might be found in the 2002 follow up to the 1992 Baccalaureate and Beyond, which seems to be the only major federal survey that asked about professional and trade association certifications before the current working group was established.

I hope that GEMEnA and/or DoEd will consider expanding on some of these themes in future revisions, but even in its current form this survey will be tremendously helpful for researchers and policymakers alike. I eagerly await the results!

Respectively submitted,

Kyle Albert

Ph.D. Candidate

Department of Sociology, Cornell University

[email protected]

NCES RESPONSE

First, we are happy to hear that the proposed data collection will be of use to researchers. We look forward to seeing what researchers will learn from the survey and how this information might be used by policymakers, credentialing bodies, and others interested in education and skill development.

Second, we agree with the commenter that the NHES, while useful in many ways, is limited in the amount of corollary information it can collect on adult’s labor market background, educational background, and other characteristics. For this reason, GEMEnA is working with both the Census Bureau and the Bureau of Labor Statistics (BLS) to add core certification and licensure questions to their key surveys. At this time, these core questions have been added to the Census Bureau’s Survey of Income and Program Participation, and are in the process of being added to BLS’s Current Population Survey. Those surveys will provide more detailed background on the characteristics of credential-holders than NHES can, while NHES will provide more detail on the credentials obtained (getting extensive details on all of these topics would be prohibitively expensive and time-consuming in a sample survey, but might eventually be possible in a unit-record data collection system, which some states are starting to build).

Thank you for suggesting a number of interesting additional questions for inclusion in the NHES data collection. As explained below, we had considered some of these questions in our development process, but decided to drop them in an effort to minimize respondent burden and maximize data quality. The proposed questions, our plans for future testing, and reasons for not including them in the current instrument are listed below.

Why did you choose this credential over others in this occupation?

This is a very interesting question, but “why” questions are notoriously difficult in survey data collections. People are often unclear of their motivations, and social desirability effects tend to be strong when people are asked to explain their motivations (i.e., people tend to give the answer they think is most socially appropriate). Because of this, we do not believe it would be prudent to add this question without first cognitively testing it. As 2015 will be the first administration of the Adult Training and Education Survey, we expect that some revisions might be needed after 2015, and we thus hope to conduct a set of cognitive interviews in 2016. We will test this question during those interviews.

How many hours did you spend preparing for this credential? How much does this certification cost, in dollars?

We agree that these are useful questions, and we originally tried to ask them. Unfortunately, our cognitive testing showed that these were difficult questions for people to answer, as what to count in the time and cost calculations was often unclear. For example, if an educational certificate or degree provided the training necessary to earn the credential, should that time and cost be counted? What about the cost of equipment (e.g., cosmetology supplies, mechanics tools)? Or the time spent in an internship? Recall was also an issue, as many people earned their credential many years ago. Even asking about renewal costs was difficult, as renewals differ in the time they cover (one year, two years, etc.) and in who pays for the renewal (e.g., the adult or the employer). So we dropped these questions from our initial instrument. But we do plan to keep thinking about how we might best get at these concepts, as the “investment costs” to credentialing are an important issue.

Is it a generalist certification covering most aspects of your occupation, or more specialized?

This is another interesting question. We did see both types of certifications during our cognitive interviews, with adults often first getting a generalist certification and then adding specializations. But it is not clear how this question should be asked in a survey. Based on our work to-date, we are not sure that the terms “generalist” and “specialized” will work for all respondents. So, again, we would need to do cognitive testing before adding this to the instrument. We would also need to weigh the value of this information against other information in the survey, given burden issues. Meanwhile, we encourage researchers to investigate whether the write-in responses on the question asking for the name of the certification can reliably provide this information. This might be one way to get this information.

How closely related was the curriculum of the certification program to your day-to-day work?

Although this is an interesting question, it is subjective in nature and probably not as useful for research and policy purposes as the other questions currently on the survey (and the other questions raised for consideration here). Also, some adults will be many years out of their certification program and are likely to have recall issues, as well as issues related to the relevancy of programs taken 10, 15, or 20 years ago. We will mention this as a question to consider by our NCES colleagues who do longitudinal studies of postsecondary students, where the question would be more appropriate.

Is this certification issued by a non-profit association? If so, are you a member of that association?

We originally tried to capture information on the type of organization that issued the certification, but we found that adults had trouble answering this question; they often were not clear on who the issuing agency was. We did not drop this issue entirely, because it is important to distinguish licenses (issued by the government) from certifications (issued by non-governmental organizations). So we do ask if the issuing agency was the government. Fortunately, adults do seem better at identifying the government as an issuing agency than at identifying other issuers—although even here, there will inevitably be some measurement error. We hope that eventually all the states will have unit-record data collection systems that include certification and licensure records, which would be the most reliable source for this information (admittedly, that solution is likely in a more distant future).

With regards to the 2002 Baccalaureate and Beyond (B&B) survey instrument, it included a number of questions focusing on the relationship between education and licensure/certification, but these questions are more relevant for B&B (which samples recent graduates) than for NHES (which samples all adults). The more recent B&B included fewer questions on certification and licensure, but we are working with our NCES colleagues to re-visit this section of the survey in order to increase the information collected on the certification and licensure topics.

Thank you very much for your interest in our surveys.

With regards,

Sharon A. Boivin, Ph.D.
Chair, Interagency Committee on Expanded Measures of Enrollment and Attainment (GEMEnA)

National Center for Education Statistics