FCSM Paper

Attachment D

Measurement Strategies for Identifying

Holders of Certificates and Certifications

Sharon A. Boivin and Isaiah L. O’Rear

National Center for Education Statistics

1990 K Street NW

Washington, DC 20006

[email protected]

isaiah.l.o’[email protected]

Introduction

In February of 2009, President Obama in his State of the Nation address called for the U.S. to become once again “first in the world” in postsecondary educational attainment by the year 2020. As a means to this goal, he asked each American “…to commit to at least one year or more of higher education or career training. This can be a community college or a four-year school, vocational training or an apprenticeship. But whatever the training may be, every American will need to get more than a high school diploma.” Concurrently, the deepening recession was leading to an increased research and policy interest in the role of industry-conferred and -recognized credentials in preparing adults for specific, living-wage jobs.

In this context, the Council of Economic Advisors and the Office of Management and Budget partnered with the Office of the Under Secretary in the U.S. Department of Education to convene senior staff from the Census Bureau, the Bureau of Labor Statistics (BLS), and the National Center for Education Statistics (NCES) to discuss ways to improve the data on educational certificates and industry-recognized certifications in the federal statistical system in order to meet research and policy data needs. After a series of initial meetings outlining the goals and resources of the project, the Interagency Working Group on Certificates and Certifications (IWG) began meeting monthly in December of 2009.¹

This paper describes a development process that sought to identify a short set of survey items that would count the number of educational certificates and industry-recognized certifications in the U.S. adult population. It includes a summary analysis of a pilot study designed to evaluate item validity by comparing its results against population estimates from other sources and by assessing key characteristics of known credentials. It discusses the tradeoffs between using more specificity to define the construct “certification/license” possibly resulting in underreporting of the prevalence of these credentials and using less specificity possibly resulting in overreporting. Finally it presents recommendations from the IWG for using the new items and next steps in development and deployment in the federal statistical system. These recommendations and their associated measurement discussions may help survey developers decide how best to test and adapt or adopt these items in their studies of education, training, occupational credentials, and outcomes of out-of-school youth and adults.

The IWG guided the item development work, providing detailed feedback at each stage, reviewing table results, discussing findings, and developing recommendations. The members of the IWG have also worked within their respective agencies to find opportunities to further develop, test, and use the questionnaire items found valid for their purpose in this project. NCES managed, staffed, and funded the work described in this paper and is responsible for its contents.

The full results of this development effort, including reports from the focus groups and cognitive interviews and detailed tables from the pilot study results, will be available from NCES 2012-190, “The Adult Training and Education Survey (ATES) Pilot Study” technical report forthcoming in early Spring 2012. This paper extracts key findings and lessons from that report and includes some additional reflections on the tradeoffs involved in how narrowly or broadly federal surveys operationally define certifications and licenses.

Background

The measurement of educational attainment has been a staple in social and economic research for decades, and federal data collections have long collected reliable, valid data on the attainment of postsecondary academic degrees, including associate’s, bachelor’s, and graduate and professional degrees. This ability to measure degree-based educational credentials has played a significant role in helping researchers learn about the relationship between educational attainment and employment outcomes and in shaping public policy aimed at increasing Americans’ access to education. In recent years, the role of credentials other than academic degrees in helping adults obtain jobs and advance in careers has become a more prominent part of policy discussions.

However, there is no federal data collection that measures the prevalence of industry-recognized certifications, state and local government issued licenses, and subbaccalaureate educational certificates among the US adult population. The prevalence and utility of these credentials cannot be assessed through institution-based data collection efforts, such as the National Center for Education Statistics’ (NCES) Integrated Postsecondary Education Data System (IPEDS). IPEDS collects information on certificates awarded, but is limited to counting awards based on credit-bearing instruction in institutions of higher education that participate in federal student financial aid programs. Students receive educational certificates from institutions that do not participate in federal student financial aid programs, and they receive certifications and licenses from a wide variety of institutional bodies other than postsecondary institutions such as trade associations, private companies, state governments, and employers. As a result, no institutional sampling frame is available for enumerating these credentials at the national level.

Additionally, information about certifications, licenses, and certificates is not adequately covered in federal data collections that survey adults directly. Without the ability to count certifications, licenses, and certificates, it is difficult to accurately assess their effects on social and economic outcomes and difficult to evaluate public policies designed to help Americans obtain and benefit from these credentials.

After its formation in late 2009, the IWG undertook the development of a short set of survey items to measure the prevalence of certifications, licenses, and educational certificates. This development effort culminated in the Adult Training and Education Survey (ATES) Pilot Study, a national household survey of noninstitutionalized adults ages 18 and over. The pilot was conducted from September 2010 to January 2011, and the results are presented in this summary.

Item Development Process

The first step for the IWG was to review the recent research literature on education and training credentials and develop a list of existing survey items on certifications, licenses, and educational certificates. Based on this background research and discussions with experts, certifications and licenses were found to be distinct from educational certificates in that they are a job or occupational requirement meant to demonstrate that the holder has the authority or has met certain qualifications necessary to perform specific job duties; educational certificates are awarded to show completion of a program of study—which may help in the performance of a specific job, but is not necessarily required or considered proof of qualification for work. This review led to the following working definitions of these credentials for the purposes of pilot testing:

• Certification: A credential awarded by a certification body based on an individual demonstrating, through an examination process, that he or she has acquired the designated knowledge, skills, and abilities to perform a specific job. The examination can be written, oral, or performance-based. Certification is a time-limited credential that is renewed through a recertification process.

• License: A credential awarded by a licensing agency based on pre-determined criteria. The criteria may include some combination of degree attainment, certifications, certificates, assessment, apprenticeship programs, or work experience. Licenses are time limited and must be renewed periodically.

• Educational certificate: A credential awarded by a training provider or educational institution based on completion of all requirements for a program of study, including coursework and tests or other performance evaluations.

Certificates, as an academic award, are not time limited and do not need to be renewed. Most educational certificates are awarded at the subbaccalaureate level, but a small number are awarded after the completion of a postsecondary degree. Certificates of attendance or participation are not in the definitional scope for educational certificates.

As the definitions illustrate, although these credentials have specific characteristics and definitions, they are not mutually exclusive. Individuals may enroll in a certificate program to gain the knowledge needed to attain a certification which may be required prior to applying for a license. The interrelated nature of these credentials was important to consider when developing and evaluating potential survey measures.

The next step for the IWG was to engage in a series of focus groups and cognitive interviews designed to yield a pilot study questionnaire to test for validity a short series of measures designed to enumerate adults with certifications, licenses, or certificates.

Four major findings from the focus groups and cognitive interviews influenced key decisions during the design of the pilot study questionnaire. First, the focus groups showed that the terms “certification” and “license” are not always distinguishable to respondents. As noted above, certification is often the process by which someone gains the prerequisite qualifications needed to be awarded a license, and several focus group participants did not make a distinction between the process and the outcome. Consequently, the decision was made to ask about certification and licensure jointly.

Second, focus groups confirmed that certification and license holders do not view their certifications or licenses as educational credentials. When asked where they would place a certification on their resume, respondents indicated it would be placed under a header such as Professional Qualifications or Professional Certifications. Consequently, certifications/licenses were not asked about within the questionnaire section on educational attainment.

Third, the cognitive interviews showed that respondents also had trouble distinguishing between the terms “certification” and “certificate.” These two terms sound similar and, in fact, are shown as synonyms in the Merriam-Webster dictionary. The cognitive interviews revealed that some respondents who have a certification will answer that they have a “certificate” if the questionnaire asked about certificates before certifications. Therefore, the questionnaire was designed to specifically ask about certifications and licenses first and certificates second.

Fourth, respondents’ interpretation of the term “certificate” was varied and broad when given little or no definitional context. Accordingly, examples of appropriate certificate providers were included in the lead item asking whether or not a respondent has a certificate.

Based on these findings the following items were used as gatekeepers into a set of questions on each type of credential:

Main Certification/License Item (CN1)

Now I’d like to ask you about professional certification and licensure. Do you have a professional certification or a state or industry license?

 

Probe: A professional certification or license shows you are qualified to perform a specific job and includes things like Licensed Realtor, Certified Medical Assistant, Certified Construction Manager, a Project Management Professional or PMP certification, or an IT certification.

Main Certificate Item (CT1)

Some people decide to enroll at a college, university, community college, or trade school to earn a certificate rather than a degree. Have you ever earned this type of certificate?

 

Probe: An educational certificate is typically earned by completing a program of study offered by a college or university, a community college, or a trade school, but it does not lead to an associate’s, bachelor’s, or graduate degree. Sometimes these are also called vocational diplomas, for example, a cosmetology or mechanics diploma, which differs from a high school diploma.

The focus groups and cognitive interviews helped to identify key characteristics associated with certifications, licenses, and certificates. These key characteristics formed the operational constructs for the remainder of the questionnaire items. They included the level of effort to obtain the credential, examination and performance assessment requirements, type of provider or awarding body, industry recognition, occupational specificity, requirement for employment, suspension or revocation potential, continuing education and recertification requirements, and perceived labor-market value. Survey items measuring these key characteristics were developed for the pilot study questionnaire and asked of respondents for their most recent certification/license and certificate.

The pilot study questionnaire also collected information about respondents’ demographic characteristics, such as age, gender, employment status, race/ethnicity, and income. These questions were derived from the NCES National Household Education Survey (NHES) and the Current Population Survey (CPS).

ATES Pilot Study Design

The ATES Pilot Study was administered from September 2010 through January 2011. It is important to note that the pilot study was not conducted for the purpose of collecting data to make population estimates (although survey weights were developed to examine the reasonableness of population estimates generated from the survey measures). The primary purpose of the pilot study was to examine the empirical properties of potential survey measures.

The ATES Pilot Study was conducted by mail and telephone using an address-based sample covering the 50 states and the District of Columbia. The study was sponsored by NCES and designed and implemented by the American Institutes for Research (AIR) and Westat. The ATES Pilot Study data file contains a nationally-representative sample of 3,730 adults, of whom 1,230 reported certifications/licenses and 530 reported certificates (220 reported both).² In addition to the nationally-representative sample, the ATES Pilot Study data file contains a convenience, or “seeded”, sample of 340 adults who were voluntarily reported by three community colleges to have been awarded a certificate and by five credentialing bodies to have received a certification or license.³ The seeded sample contains state license holders in public health, electrical, pharmaceutical, plumbing, real estate, and energy-related fields; certifications/licenses in lab technology, radiation, and other health-related fields; and community college certificates covering a variety of programs. The seeded sample is not a representative sample and was used primarily to assess underreporting of credentials. Although care was taken to find current and reliable administrative records for the seeded sample by requesting records directly from awarding bodies, these records have unknown measurement error, and therefore, at least some portion of the reporting error discussed in the findings of this paper may be attributed to errors in the seeded sample frame.

Finally, in a subsample of approximately 1,000 households with two or more adults, the responding adult was also asked to report on certifications, licenses, and certificates held by another adult in the household. This was done in order to examine measurement error for responses reported by proxy because federal surveys that may be candidates for the inclusion of these new items may use proxy respondents when they are unable to interview the sampled adult in the household.

The first-stage mail survey response rate was 52 percent. The second-stage telephone interview response rate was 44 percent (unweighted) and 42 percent (weighted). It should be noted that limited refusal conversion was undertaken at the telephone stage in order to maintain timeliness of the data collection. Timeliness and sample yield were a higher priority for the pilot study than were response rates.

The data file used for analysis underwent limited data editing. Imputation was performed on variables needed for weighting and on the overall certification/license and certificate items. Weights and variables to calculate standard errors were developed and used to examine the reasonableness of population estimates generated from the survey items.

ATES Pilot Study Results: Weighted Estimates and Comparisons to Other Nationally-Representative Data

Responses from the ATES sample were weighted to represent all U.S. adults ages 18 and older. As shown in table 1, an estimated 38 percent of adults, or approximately 82 million adults, reported having at least one of the credentials of interest. A certification or license was more prevalent than a certificate; 30 percent of adults reported having a certification or license, compared to 14 percent who reported having a certificate.⁴

To determine whether the ATES Pilot Study estimates were consistent with past research, the ATES estimates were compared to estimates derived from the Princeton Data Improvement Initiative (PDII) for certifications/licenses and to the Survey of Income and Program Participation (SIPP) for certificates.

The PDII survey was conducted in 2008 by Westat on behalf of Princeton University and covered the topics of worker perceptions of the “offshoreability” of their jobs, occupational licensing, and lifetime work experience.⁵ Participants were adults in the labor force, currently employed or unemployed and looking for work. After restricting the ATES and PDII samples to adults who were currently employed, the percentage of adults reporting a certification/license was 38 percent based on the ATES Pilot Study data and 34 percent based on the PDII data (table 2). Although the ATES estimate was measurably different from the PDII estimate, the difference is small enough to be considered reasonable, given dissimilarities in the data sets in terms of timing, item wording, and item order. When broken out by educational attainment level, the rate of certifications/licenses was not measurably different for adults whose highest level of education was below a bachelor’s degree, but was measurably higher in ATES than in the PDII for those at the bachelor’s degree level and above (by around 7 to 8 percentage points).

To examine the ATES estimates on certificates, the ATES data were reanalyzed and compared to 2008 data from the U.S. Census Bureau’s SIPP, using adults ages 23–64 who reported an income or reported working for earnings in the past year.⁶ The overall rate of certificates based on ATES was 15 percent, compared to 18 percent for SIPP. Again, the difference was statistically significant, but reasonable given the dissimilarities between the two surveys (e.g. timing, mode of interview). Significant differences also existed between the two surveys for adults with educational attainment levels at or below high school level, and at the graduate or professional degree level (differences in the range of 2 to 6 percentage points). There were no measurable differences in the estimates for adults with some college, an associate’s degree, or a bachelor’s degree.

ATES Pilot Study Results: Unweighted Analysis of Underreporting of Credentials in the ATES Seeded Sample

Reverse record checks, like the one employed in the ATES Pilot Study, are designed to measure underreporting of known characteristics. The seeded sample in ATES can be used to identify characteristics of respondents and/or credentials that could help explain underreporting (false negatives). This analysis examined differential underreporting rates by type of credential, age, and employment industry. Some portion of underreports could be due to errors in the administrative records or data collection, such as reaching the wrong individual.

One possible explanation for underreporting is that respondents may misidentify a certification/license as a certificate (or vice versa). Differential underreporting rates can be used to evaluate this by determining whether a certificate was also reported by the respondent (and whether that certificate was from someplace other than a college or other educational program, as this would be an indication that the credential was more likely a certification/license than a certificate). Similarly, the evaluation of seeded certificates looked for differential underreporting rates by whether a certification/license was also reported by the respondent and whether those certifications/licenses were part of a degree or other educational program. Finally, the evaluation of seeded certifications/licenses assessed differential underreporting rates by credential type (certification or license). These additional analyses (reported in the full NCES technical report) can provide evidence of credential misidentification at an aggregate level.

A record was classified (at the person level) as a certification/license underreport if a respondent who was sampled from the seeded certification/license frame answered “no,” “don’t know,” or “refused” to the main certification/license survey question (item CN1: Do you have a professional certification or a state or industry license?). Similarly, a record was classified (at the person level) as a certificate underreport if a respondent who was sampled from the certificate frame answered “no,” “don’t know,” or “refused” to the main certificate survey question (item CT1: Some people decide to enroll at a college, university, community college, or trade school to earn a certificate rather than a degree. Have you ever earned this type of certificate?). Proxy responses were not included in this analysis to avoid confounding the possibility of limited proxy knowledge with actual underreports.

The overall underreporting rate for seeded certifications/licenses was 15 percent, and the overall underreporting rate for seeded certificates was 50 percent (table 3). Few of the differences in underreporting rates by subgroup were statistically significant. However, the results for age and employment status suggest that saliency may be a factor. Those in the oldest age group (65 years and over) and those not employed were most likely to underreport these credentials.

Respondents selected from one of the community colleges in the certificate sample frame were removed after finding that they had an unusually high certificate underreporting rate (85 percent); further investigation revealed that there was a problem with the sample frame itself. The school had encoded students who achieved a transfer sequence of 18 to 60 units as “certificates.” These students represented about 78 percent of the records in the sampling frame and 86 percent of the survey respondents for this school. The sampled respondents with problematic records from this school had an underreporting rate of 93 percent, while the sampled respondents with valid records had an underreporting rate of 57 percent. However, while the rate for the latter group is more in line with the other certificate samples, the sample size was too small to be included in the analyses (n <10).

The identified inaccuracies in the certificate seeded sampling frame may reflect similar (though less extensive) issues in data supplied by the other community colleges. Therefore, the high underreporting rate found for certificates may be partially due to characteristics of the frame. This possibility will be investigated by NCES in upcoming work on certificate item development and testing.

ATES Pilot Study Results: Unweighted Analysis of Key Characteristics of Certifications/Licenses

Based on the focus groups and cognitive interviews, several factors were expected to covary with holding a certification/license or certificate. First, certification/license holders were expected to report that their credentials were in some way job relevant; for example, respondents might report that their certifications/licenses were required for their current job. Respondents were also expected to report taking courses or training to earn the certification/license and that they had been required to demonstrate their knowledge or skills in order to be certified/licensed. Finally, respondents were expected to report being certified or licensed by a state, industry, company, or professional association and that their certification/license could be revoked or suspended. Factors expected to be related to certificates reflected the fact that this type of credential is obtained through an educational program. Therefore, certificate holders were expected to report that their credentials were awarded by an educational institution (e.g., a community college).

Respondents were asked some items, such as the name and year the credential was obtained, for all certifications/licenses or certificates held (up to five of each), whereas the majority of credential-specific items were only asked for the most recent certificate earned and the most recent work-related certification/license held.⁷ Because information was collected on potentially multiple credentials per respondent, items were analyzed at the credential level rather than the respondent level and are reported as the percentage of certifications/licenses or certificates held by respondents. In addition, these analyses are not weighted because, unlike comparisons of estimates, they are methodological analyses of sample characteristics and are not intended to show population characteristics.

In fact, the national estimates from the ATES Pilot Study did find relationships between holding a certification/license or certificate and other factors as expected, based on ATES developmental work and the definitions of the credentials. As shown in the NCES technical report, most reported certifications/licenses were pursued for mainly work-related reasons (90 percent), and most work-related certifications/licenses required the respondent to pass a test or exam (91 percent) and take continuing education units to maintain them (64 percent). The majority of the most recent certifications/licenses could be revoked or suspended (76 percent). Similarly, the most of the most recent certificates were conferred by an educational institution (57 to 73 percent) and required 21 or more credit hours to complete (57 percent for those reporting in credit hours). Instances where these expected relationships were not found might indicate misreporting by respondents.

ATES Pilot Study Results: Unweighted Analysis of Overreporting of Certifications/Licenses in the ATES National Sample

Although the seeded sample is not representative of all credential holders, it does allow us to examine the measurement properties of the survey items in two important ways. First, we can compare information reported by the sampled adult (self-reported survey data) to the frame information provided from the administrative records of the credentialing body to examine reporting accuracy. This analysis is referred to as “matching.” Matched cases are considered true credentials because they are a report of a credential that is confirmed both in the administrative records data and by the survey data. Second, the characteristics of true credentials can then provide a rule-of-thumb for identifying reported credentials in the nationally-representative sample that do not reflect the characteristics of a true credential. These cases are classified as credential overreports.

Because of the high underreporting rate and the small number of respondents with known certificates in the seeded sample, the analysis of overreporting was limited to matched certifications/licenses.

Characteristics of these matched credentials in the seeded sample can serve as checks on the accuracy of reporting in the national sample. The pilot study identified three key characteristics most often associated with having a known certification/license:

• Was the credential earned for mainly work related reasons? (Item CN6)

• Does the credential require a demonstration of skills while on the job or passing a test or exam? (Item CN10a/b)

• Can the credential be used to get a job with any employer in that field? (Item CN15b)

Similar to the results reported for the national sample above, a high percentage of certifications and licenses that matched the seeded sample frame were obtained for mainly work-related reasons—95 percent of certifications and 98 percent of licenses (details in the NCES technical report); 100 percent of the most recent matching certifications and 80 percent of the most recent licenses required respondents to pass a test or exam; 71 percent of certifications and 62 percent of licenses required a demonstration of skills while on the job; and 96 percent of the certifications and 100 percent of the licenses could be used to get a job with any employer in the field. Nearly all certifications and licenses had at least one of these requirements.

This analysis considers whether these key characteristics identified using the seeded sample can be used to spot possible instances of overreporting of credentials in the national sample (false positives), by determining the extent to which respondents report a certification/license that does not have the characteristics of typical known credentials. Using this approach, uncommon responses (e.g., reporting a certification/license, but not reporting that the certification/license was work-related) are used to flag possible instances of overreporting.

Table 4 shows the percentage of matching seeded sample certifications/licenses that would be miscategorized as overreports in the seeded sample—and possibly in the national sample—based on responses to various combinations of the three related factors measured by items CN6 (credential earned for work related reasons), CN10a/b (test or demonstration of skills required), and CN15b (credential can be used to get a job with any employer in the field).

The analysis defined a “true” certification/license as one with all three of these characteristics (based on the analysis of the seeded sample). Applying this definition to the national sample, a respondent who reports having a certification/license and all three characteristics would be considered a true positive. If the credential does not have all three key characteristics, it could possibly be a false positive, or overreport. As shown in the second column of Table 4, different combinations of these three characteristics result in different percentages of overreports. If the certification/license has only two of the key characteristics, the overreporting rate ranges from 1.8% to 5.6% depending on the combination of items. If the certification/license has only one of the key characteristics, the overreporting rate ranges from 6.9% to 9.1%. Finally, if the respondent reports that they have a certification or license in the main certification item (CN1), this analysis projects a 13.3% overreporting rate in the national sample.

If surveys define “true” certifications/licenses using the main certification item and all three key characteristics, therefore, overreporting is minimized. But, as the first column in Table 4 shows, using such a restrictive definition underestimates the percentage of matched true certifications/licenses in the seeded sample. Applying this most restrictive definition using the main item and all three key characteristics to the seeded sample resulted in 6.8% of known, matched certifications/licenses being misclassified as overreports. Using fewer key characteristics results in fewer miscategorized overreports.

Figure 1 illustrates the tradeoffs between having more or less restrictive definitions on possible over- and underreporting. Using all three key characteristics reduces overreporting to zero (by definition) but it results in possible underreporting among known credential holders because they failed to answer one of the three items in the expected way. Using none of the three key characteristics results in relatively high overreporting of credentials, but minimizes underreporting by not eliminating known credential holders because they failed to answer one of the three items in the expected way.

The results (discussed in more depth in the technical report) imply that all combinations perform fairly well at identifying true positives (while minimizing overreporting) and that any one of the key characteristics could be used independently to eliminate overreporting. Federal surveys that elect to use these items will need to weigh the tradeoffs between overreporting and underreporting taking into consideration the analytic needs of their customers and the amount of survey space they can devote to this topic. An in-depth set of items on certifications/licenses would ideally include the main item and all three key characteristics, enabling analysts to apply their own definition to the construct.

IWG Conclusions and Recommendations

The ATES Pilot Study findings have implications for determining a parsimonious set of items that could be included in other federal household surveys of adults—either in the near future for those items requiring no further development or after additional testing has been conducted to address the measurement issues identified in this report. It is important to note that the items developed for the pilot test were limited in their scope. They were designed specifically to enumerate adults with alternative credentials, and their utility as survey measures outside the scope of this Pilot Study has not been tested. Therefore, although results of the pilot are used in this section to recommend items for researchers with various goals; the properties of the items may differ in other survey administration contexts.

With that caveat, this section summarizes the recommendations of the IWG for items to use in other household surveys trying to achieve the following research goals: counting adults with a certification or license; counting the certifications or licenses held, and distinguishing between certifications and licenses.

To count the number of adults in the United States who have a certification or license, the IWG recommends the main certification/license item CN1. This recommended item is an aggregate, broad measurement item that could be administered alone or in conjunction with other credential-specific items to provide counts of adults with work-relevant credentials. To reduce measurement error, survey sponsors may consider adding to this item one to three of the items that measure key defining characteristics of certifications/licenses. These items include whether or not the certification/license is work-related (CN6); if the individual had to pass a test or exam (CN10b), and; if the certification/license is transferable (CN15b).

The main certificate item was found to have a high underreporting rate among the seeded sample respondents in the ATES Pilot Study and is not recommended for use at this time. The main certificate item and the series of items tested in the certificate questionnaire section require additional testing before they should be used.

The results of the pilot have several implications for measurement. First, the items in the ATES were tested in a short survey focused on certifications, licenses, and certificates. It is possible that the items may perform differently in different survey contexts, particularly in surveys where other topics are covered in more depth. Second, cognitive testing for the ATES showed an order effect when asking about certifications and certificates. If certification holders were asked about certificates before certifications, they would report a certification as a certificate. Also, cognitive testing confirmed that respondents do not think of certifications and licenses as educational attainment, leading the ATES to ask about them in separate sections. Researchers should carefully consider the placement of the items within a survey.

The results of the pilot also demonstrated that the salience of the credential (as characterized by recency and work relevance) affects the accuracy of reporting. If respondents are reporting their most salient credentials, results may be biased toward the credentials held and used by those currently employed. It is also important to note that respondents who were not employed had greater reporting error compared to employed respondents.

The IWG member agencies are planning several data collection efforts to provide counts of adults with certifications or licenses. The IWG is also planning to conduct additional developmental work to better measure educational certificates. These plans are summarized below.

Upcoming work on certification/license measurement

IWG members have looked within their own agencies for opportunities to incorporate the tested items into ongoing or planned studies of out-of-school youth and adults. In addition, they have sought ways to test these items further to explore their performance in different survey contexts. These plans are described below.

NCES. NCES is incorporating a set of questions on certifications/licenses into its upcoming Baccalaureate and Beyond (B&B) longitudinal study of 2008 Bachelor’s degree recipients. Currently-active secondary longitudinal studies (the High School Longitudinal Study of 2009 freshmen and the Education Longitudinal Study of 2002 sophomores) will incorporate questions on certification for out-of-school youth and young adults. There are short-term plans to develop an adult education and training household study that will feature items on certifications and licenses and long-term plans to collect background data that would permit linkages between credentials and literacy.

Census. The Census Bureau has recognized an opportunity for further testing of certification items in the field test of the redesigned Survey of Income and Program Participation (SIPP). The inclusion of these items in a different survey context will test the robustness of the item across platforms. The Bureau is also planning to collect weighted national estimates of the number of people with certifications and will field a topical module with the main certification item as the gateway question for the final existing SIPP panel (2008 panel).

Upcoming work on certificate measurement

Due to the inconclusive results of the Pilot Study analysis of data on certificates, the IWG has recommended additional developmental work on the survey items related to certificate attainment. In addition, NCES plans to respond to increased interest in subbaccalaureate educational certificates by improving the ability of current data collections to measure these credentials. The four projects described below will improve and expand the ability of NCES studies to answer important policy questions about the prevalence and economic impact of certificate education.

Developing a Valid Survey Item to Enumerate Certificates in Federal Surveys

In April of 2012, the IWG will begin development and testing of a new survey item designed to measure the prevalence of educational certificates in the US adult population. This effort will be designed to improve on the measurement of certificates that was attempted in ATES, including a sharper focus on these measures, and use of a larger and cleaner sample of known certificate holders. The study will begin with focus groups to determine how certificate holders talk and think about their credential. Next, questionnaire items will be developed and tested in cognitive interviews. A new pilot study will feature a larger sample of known certificate holders that is more representative both geographically and by field of study. Paired with a new national sample, this new seeded sample will provide a good test of the validity of the new certificate items.

Improving Institutional Data Collection on Certificates

NCES currently collects annual counts of the number of certificates awarded through its Integrated Postsecondary Education Data System (IPEDS). IPEDS gathers information from every college, university, and technical and vocational institution that participates in federal student aid programs—over 7,000 postsecondary institutions in all. The IPEDS data collection is guided by the National Postsecondary Education Cooperative (NPEC), which oversees IPEDS research and development.

NPEC has recently created a working group to look at how institutions report certificate completions to IPEDS in order to improve the quality, comparability, and usefulness of these data. To support this work, a study of certificate reporting practices is being carried out. The working group began meeting in October 2011; a final report is scheduled for September 2012.

Oversampling Certificate Holders in BPS:12

Starting in 1996, NCES has conducted a periodic longitudinal study of students who have recently matriculated into postsecondary education, the Beginning Postsecondary Students Study (BPS). The BPS is conducted about every eight years, with the next study scheduled to start in 2012. BPS is the largest federal survey that regularly examines the economic outcomes of students in subbaccalaureate programs. However, previous rounds of BPS have lacked the capacity to conduct in-depth analyses of the economic returns to educational certificates in specific fields of study. NCES plans to drastically increase the sample size for beginning students who enter postsecondary education with the goal of attaining an educational certificate. This oversample will allow finer-grained analysis of the reasons students select educational programs leading to a certificate, the characteristics of students in such programs, their persistence and attainment, and their occupational and economic outcomes.

Fielding a Household Survey on Adult Education and Training

NCES is planning a household study to document adults’ acquisition of education and training that is oriented towards work. This new study will collect in-depth information on the education required to attain various levels of qualifications, the relationship between industry-recognized certifications and employment, the prevalence of industry-recognized certifications among adults with formal education credentials, the career pathways of adults seeking certificates and certifications, and the supports and barriers faced by adults seeking such credentials.

NCES previously had a survey of adult education, the National Household Education Study (NHES) Adult Education Survey, but it was not focused on industry-recognized credentials. Moreover, that survey was halted in 2007 because of low response rates. NCES has redesigned the NHES to collect data on early childhood education. The redesigned NHES will use a mail administration process rather than random digit dialing as had been done in the past—a change that yielded improved response rates in field testing. To see if response rates would be similarly improved with the redesigned procedures when asking adults about themselves, rather than their children, NCES will conduct a response rate test for a survey on adult education and training in 2012. The response rate test will consist of a nationally-representative sample of households and adults to evaluate response rates for adults with key characteristics related to age, educational attainment, and employment status. NCES is also planning an in-depth nonresponse bias study with personal interviews of a subset of nonrespondents as part of this project. If response rates are sufficiently high or if nonresponse bias is deemed to be sufficiently negligible, NCES could field a full scale adult education and training survey as soon as funding becomes available.

Appendix A. Members of the Interagency Working Group (IWG)

Beginning in December 2009, the federal interagency working group (IWG) convened monthly to develop new survey measures of certifications, licenses, and certificates. The IWG represents a broad range of federal agencies committed to improving the measurement of work-related credentials. Current members of the IWG (as of March 2012), as well as past IWG members from each participating agency include:

Bureau of Labor Statistics: Harley Frazis

Tom Nardone

Census Bureau: Stephanie Ewert

Bob Kominski

Council of Economic Advisors: Elizabeth Ananat (past)

Sarena Goodman (past)

Jessie Rothstein (past)

Chuck Pierret (past)

Department of Education, Office of the Undersecretary: Jon O’Bergh

National Center for Education Statistics: Sharon Boivin

Sarah Crissey (originally at Census Bureau)

Lisa Hudson

Kashka Kubzdela

Isaiah O’Rear

Matthew Soldner

Tom Weko

National Science Foundation: John Finamore

Daniel Foley

Office of Management and Budget: Shelly Martinez

1 See Appendix A for a list of IWG members.

2 The pilot study data are not available for secondary analysis due to their preliminary nature, their methodological rather than reporting purpose, and disclosure concerns.

3 It is not possible to identify the specific credentialing bodies for disclosure reasons.

4Although the primary interest of the ATES Pilot Study was on subbaccalaureate certificates, the survey instrument did not distinguish the level of the certificate, so respondents may have reported postbaccalaureate certificates.

5 The PDII data are available at http://irs.princeton.edu/Conferences/PDII/PDII_RDD_Survey.dta.

6 The SIPP 2008 data are available at http://www.bls.census.gov/sipp_ftp.html.

7 If respondents did not report any work-related certifications/licenses, they were asked follow-up questions about the most recent credential they obtained for personal interest.

12