Volume I:
Adult Training and Education Survey (ATES) 2010 Pilot Test
New Items to Measure Professional Certifications and Education Certificates among Adults in the United States
Request for Clearance
OMB# 1850-0803 v.34
July 16, 2010
Purposes and Uses of the Data 4
Use of Improved Information Technology 4
Efforts to Identify Duplication 5
Consultations Outside the Agency 6
Assurance of Confidentiality 6
Cost to the Federal Government 9
Description of Statistical Methodology 9
Methods for Maximizing Response Rates 16
Attaining a postsecondary credential has become increasingly important for securing opportunities to get high-return jobs in the United States in the 21st century. However, postsecondary certificates and degrees awarded through credit-bearing instruction in traditional institutions of higher education comprise only a portion of subbaccalaureate education and training for American adults looking to find jobs. Data limitations have precluded any enumeration or analysis of the economic returns of alternative credentials such as industry-recognized certifications.
At the request of the Council of Economic Advisors and the Office of Management and Budget, the National Center for Education Statistics (NCES) has taken the lead on data development to improve federal data collection on the education and training that youth and adults need to prepare for jobs and to contribute to the growth of the economy. Specifically, NCES is developing a short series of measures to enumerate education certificates and industry-recognized certifications through existing federal household data collections such as the Current Population Survey (CPS), as well as conceptualizing a longer-term study to examine a wider range of credentials in greater depth and probe the education, social, and economic correlates of attainment.
This request is for clearance of an Adult Training and Education Survey (ATES) pilot test of the short series of measures. The ATES Pilot Test will provide an opportunity to examine the accuracy and validity of measures developed to enumerate education certificates and industry-recognized certifications. The ATES Pilot Test will be conducted in the fall of 2010 and will involve the screening of approximately 20,000 households by mail to obtain contact information for telephone follow-up with sampled adults.
Clearance is requested by August 18, 2010, in order to complete the final formatting of questionnaires and final instrument programming for telephone follow-up.
In his February 2009 State of the Union address, President Obama stated an ambitious goal: by 2020 America will once again have the highest proportion of college graduates in the world. He then asked every American
“…to commit to at least one year or more of higher education or career training. This can be a community college or a four-year school, vocational training, or an apprenticeship. But whatever the training may be, every American will need to get more than a high school diploma.”
The American Graduation Initiative was introduced by the President in July 2009 to directly support this goal by helping “an additional 5 million Americans earn degrees and certificates in the next decade.”
Attaining a postsecondary credential has become increasingly important for securing opportunities to get high-return jobs in the United States in the 21st century. However, NCES has traditionally only collected data on postsecondary certificates and degrees awarded through credit-bearing instruction in traditional institutions of higher education that participate in Title IV federal student aid programs. These comprise only a portion of subbaccalaureate education and training American adults seek and complete to learn the skills they need to find and keep good-paying jobs. In fact, a 2008 study using student unit record data from the state of Florida found that, in many cases, industry-recognized certifications have a greater economic value than associate degrees.
Items developed from the ATES Pilot will provide a means to investigate education issues that cannot be adequately studied through the Center’s institution-based data collection efforts. For example, adults receive professional certifications from many institutional bodies, such as trade associations, private companies, and employers. As a result, no institutional sample frame is available for enumerating certifications at the national level. Additionally, certifications and certificates are not adequately covered in other federal statistical data collections. For example, the American Community Survey (ACS) only includes “vocational certificate” as a volunteered answer category in its education attainment item.
While the ATES Pilot Test is not being conducted to make survey estimates, the data gathered will allow for examination of the empirical properties of potential survey measures.
The data collected in the ATES Pilot Test will be used to evaluate the quality of data received using the new certification and certificate measures. It will not be used to generate published estimates. Information gathered from this pilot will be used to make recommendations for a core set of items that can be used by NCES and other federal statistical agencies to measure the prevalence of professional certifications and education certificates in the United States.
The ATES Pilot Test will be conducted using two complementary survey systems that will improve the efficiency and accuracy of the data collection process.
The self-administered questionnaires will be implemented in Teleform, which offers a wide range of capabilities to projects. The following features are important for the ATES Pilot Test:
Forms Design. Questionnaires can be created using the Designer module. Form templates are used to classify each data field as a text entry, choice, signature or image zone. Completed hardcopy forms can be processed by TeleForm to capture responses without manual data entry.
Image Preprocessing. TeleForm applies image preprocessing to the forms in their image format in order to correct any skewing that may have occurred during scanning or faxing, and to remove other unwanted marks from the form according to project specifications.
Data Capture. TeleForm reads the form image files and extracts data according to rules established for each questionnaire template. TeleForm can recognize handwritten (ICR), printed (OCR), check box, and ‘bubble’ (OMR) data types.
Verification. Extracted data are subject to field validation according to project specifications. If a data value violates validation rules, the data may be flagged for review by verifiers who interactively review the images and the corresponding extracted data, and resolve validation errors.
Telephone follow-up interviews will be conducted using a computer-assisted telephone interviewing (CATI) system. The most important features of CATI for ATES are the following:
Scheduling: The CATI scheduler will be used to route telephone numbers to interviewers, maintain a schedule of callback appointments, and reschedule unsuccessful contact attempts to an appropriate day and time.
Skip Patterns: The CATI system will automatically guide interviewers through the skip patterns in the questionnaire, reducing the potential for interviewer error and shortening the questionnaire administration time.
Receipt Control: The CATI system will provide for automatic receipt control in a flexible manner that will be used to produce status reports that allow ongoing monitoring of the survey’s progress.
Senior policy officials in the Departments of Education and Labor, foundations including the Gates Foundation and Lumina, and research organizations such as the Georgetown Center for Education and the Workforce have recognized there is a lack of valid statistical information on prevalence of industry-recognized certifications and education certificates and called for the development of new data sources. A series of meetings during the fall of 2009 launched a broad effort to begin to define and enumerate these credentials. NCES also conducted a review of research literature and data collections since the work of a previous Interagency Committee in 2000 and developed a bank of existing survey items on certifications (completed 11/2009) and education certificates (completed 1/22/2010). This research found no survey measures that adequately capture the prevalence of industry-recognized certifications and education certificates.
NCES convened a federal interagency working group (IWG) to review and develop survey measures. The IWG has met monthly since October 2009 and consists of staff from the Department of Education, the Census Bureau, the Bureau of Labor Statistics, the Council of Economic Advisors, and the Office of Management and Budget:
Council of Economic Advisors
Elizabeth Ananat
Sarena Goodman
Jessie Rothstein (no longer at CEA)
Census Bureau
Sarah Crissey
Bob Kominski
Bureau of Labor Statistics
Harley Frazis
Tom Nardone
Office of Management and Budget
Shelly Martinez
Department of Education – Office of the Under Secretary
Jon O’Bergh
National Center for Education Statistics
Sharon Boivin
Lisa Hudson
Stuart Keraschky
Kashka Kubzdela
Matthew Soldner
Tom Weko
An advance cash incentive of $2 will be sent with the first screener mailing. A promised incentive will be used for some telephone follow-up respondents. As an experiment of the effects of incentives on response rates, 20 percent of respondents completing the telephone follow-up will receive no incentive, 40 percent will receive an incentive of $10, and 40 percent will receive $20. This allocation is expected to allow for detection of a 5 percentage point difference in extended survey response rates between the no incentive and $10 incentive groups, and a 4.6 percentage point difference between the $10 and $20 incentive groups (with an overall Type I error rate of 5 percent and 80 percent power.) The topical incentive will be paid by check to the respondent.
The following confidentiality pledge accompanies the first-stage mail survey in this pilot:
Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) Public Law 107-279, Section 183]. The information you provide will be combined with information from other participants to produce statistical summaries and reports.
Respondents at the telephone follow-up will be reminded that the survey is voluntary and that their answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) Public Law 107-279, Section 183].
All contracting staff with access to the ATES data are required to sign the NCES affidavit of nondisclosure:
NCES Affidavit of Nondisclosure
___________________________ ___________________________
(Job
Title) (Date Assigned to work with NCES Data)
___________________________
(Organization,
State or Local Agency)
___________________________ ___________________________
(Organization
or Agency Address) (NCES Database or File Containing
Individually Identifiable Information*)
I, _________________________, do solemnly swear (or affirm) that when given access to the subject NCES database or file, I will not -
(i) use or reveal any individually identifiable information furnished, acquired, retrieved or assembled by me or others, under the provisions of Section 183 of the Education Sciences Reform Act of 2002 (P.L. 107-279) and Title V, subtitle A of the E-Government Act of 2002 (PL 107-347) for any purpose other than statistical purposes specified in the NCES survey, project or contract;
(ii) make any disclosure or publication whereby a sample unit or survey respondent (including students and schools) could be identified or the data furnished by or related to any particular person or school under these sections could be identified; or
(iii) permit anyone other than the individuals authorized by the Commissioner of the National Center for Education Statistics to examine the individual reports.
_____________________________
(Signature)
[The penalty for unlawful disclosure is a fine of not more than $250,000 (under 18 U.S.C. 3571) or imprisonment for not more than five years (under 18 U.S.C. 3559), or both. The word "swear" should be stricken out when a person elects to affirm the affidavit rather than to swear to it.]
City/County
of _______________ Commonwealth/State of ___________
Sworn
to and subscribed before me this _______________ day of
_______________, 20________. Witness my hand and official Seal.
_________________________ ____________________
(Notary
Public/Seal) My commission expires
The ATES Pilot Test is a voluntary survey, and no persons are required to respond to the interviews. In addition, respondents may decline to answer any question in the survey.
A few items in the surveys may be considered sensitive by some respondents. These include:
Personal and household income;
Address confirmation
Phone number
Measures of income are important because education attainment is statistically associated with income and the empirical properties of the survey measures may differ for people with different income levels. It is necessary to confirm respondents’ addresses to ensure we have reached the appropriate household and to verify the address to which the promised incentive should be mailed. It is also necessary to collect a phone number in the household mail screener in order to administer the extended topical survey by phone.
The response burden per instrument and the total response burden for the ATES Pilot Test are shown in Table 1. The administration times for the Screener and extended topical survey are based on practice administrations. It is expected that respondents with credentials will take about 25 minutes and respondents without credentials will take about 10 minutes to complete the extended topical phone survey. On average, the survey will take about 15 minutes.
The cost to respondents for the total hour burden is estimated to be $32,225, that is, $19.88 per hour for 1,621 burden hours. The hourly rate is based on the average for all civilian workers from the 2007 National Compensation Survey (http://www.bls.gov/ncs/ocs/sp/nctb0298.pdf). There are no other costs to respondents and no recordkeeping requirements associated with the ATES Pilot Test.
Table 1. Estimated response burden for the ATES Pilot
Instrument |
Avg. Completion time |
Sample size |
Expected number of completed questionnaires |
Expected response rate (%) |
Total burden hours (completes * avg. completion time) |
Mail Screener (ATES Mail Screener.pdf) |
3 |
18,000* |
9,900 |
55 |
495 |
Extended Topical Survey (ATES Extended Topical Survey.doc) |
15 |
6,930 |
4,505 |
65 |
1,126 |
Total |
NA |
NA |
14,405 |
NA |
1,621 |
The total cost of ATES Pilot Test data collection to the government is approximately $820,000. This includes all direct and indirect costs of the data collection.
August 19, 2010: Print materials
September 6: Labor Day holiday
September 7: Assemble mailing packages
September 15: Initial mailing by USPS (First Class)
September 22: Mail reminder postcard
October 2–3: Telephone interviewer training
October 6: Nonresponse follow-up by USPS (First Class)
October 6: Begin telephone interviews
October 20: Nonresponse follow-up by FedEx
January 15, 2011: End data collection
The ATES Pilot Test is an address-based sample covering the 50 states and the District of Columbia. The ATES Pilot Test will be conducted from September through January 2010. Households will be randomly sampled as described below, and a screening interview will be administered by mail to an adult household respondent. In order to limit respondent burden and improve ease of survey estimation, regardless of the number of eligible adults, no more than one adult per household will be contacted for the telephone follow-up.
For the ATES Pilot Test, a nationally representative sample of 18,750 addresses will be used. This nationally representative sample of addresses will be drawn in a single stage from a file of residential addresses maintained by a vendor, based on the United States Postal Service (USPS) Computerized Delivery Sequence File (CDSF). To accommodate the use of telephone to administer the extended topical survey as described in section 3 below, the sample of addresses will be reverse-matched to landline telephone directories; it is expected that a telephone number will be obtained for about 60 percent of addresses through this reverse matching. Additionally, in the screening questionnaire, the respondent will be asked to provide a telephone number. Taking into account both the vendor-matched phone numbers and the respondent-provided phone numbers, it is expected that a telephone number will be available for about 70 percent of households with completed Screeners. For a given household, if both a vendor-matched telephone number and a respondent-provided telephone number are available, the extended topical survey will be attempted first on the respondent-provided telephone number.
In addition to the nationally representative sample, a seeded sample of 1,250 addresses containing adults known to have certificates or certifications will be included. Thus, the overall sample size is 20,000 addresses. The sample size of 1,250 for the seeded sample is expected to be of sufficient size to detect a false negative rate (i.e., the proportion of persons with certificates/certifications who fail to report the certificates/certifications) of 2 percent. Data collection procedures for the seeded sample will be comparable to those for the nationally representative sample.
An initial Screener1 will be mailed via First Class mail to each sampled address. One week after the initial Screener mailing, a thank you/reminder postcard will be mailed to each address. A follow-up Screener will be mailed via First Class mail to each address that did not respond to the first Screener mailing. A second follow-up Screener will be sent via FedEx 2Day2 to each address that did not respond to the initial or first follow-up Screener mailings.
Eligible adults within households that have a completed Screener will be sampled for the ATES Pilot Test. To be eligible, an adult must be age 18 or older. For this sample design plan, the assumption is that within a given household, one adult is sampled with equal probability. As completed Screeners are returned, they will be scanned for data capture. For each household with a completed Screener, one adult will be randomly selected from among all eligible adults in the household. The sampling will be done using a pre-loaded random number that is attached to each address. That random (Uniform(0,1)) number will be multiplied by the count of eligible adults to determine which adult to select.
Sampling one adult per household has the advantage of minimizing intrahousehold burden. Additionally, with this approach in contrast to an approach that selects more than one adult in some households, any intrahousehold correlations (either in certificate credentials or certification participation or in measurement error in the responses to the certificate/certification questions) will not adversely affect the precision of estimates.
As described above, Screeners will be sent to each of the 20,000 sampled addresses. It is expected, based on experience from the 2009 Pilot Test of the National Household Education Surveys Program (NHES), that about 10 percent of addresses (or a total of 2,000 addresses) will be nondeliverable. An expected Screener unit response rate of 55 percent is assumed. This unit response rate would yield 9,900 completed Screeners.
It is further expected that a telephone number will be available (either from the vendor match or from a response to the telephone number question in the Screener) for 70 percent of households with a completed Screener, or an expected 6,930 households. As a result, the extended topical survey will be attempted with an expected 6,930 adults.
For the extended topical survey, a unit response rate of 65 percent is assumed; this would yield an expected 4,505 completed extended topical interviews. Table 2 summarizes the expected numbers of completed interviews for the ATES Pilot Test.
Table 2. Expected numbers sampled and expected numbers of completed Screeners and extended topical surveys in the ATES Pilot Test
Survey |
Expected number sampled |
Expected number of completed surveys |
Household Screeners |
20,000* |
9,900 |
Extended topical Surveys |
6,930 |
4,505 |
The key objective of the ATES Pilot Test is to assess the measurement properties of a new series of items about certificates and certifications (C&C). The sample requirements should enable the generation of cross-sectional estimates of C&C holders in the population, allowing for factor analysis of a series of 10-15 items for some key subgroup analysis.
Table 3 shows the expected numbers of completed extended topical surveys for subgroups defined by credential status, education attainment, age, and race/ethnicity. MacCallum et al. (1999) cite various recommendations in the literature for minimum sample sizes for factor analysis. Although these sample size recommendations vary, there appears to be a general consensus amongst the source papers cited that a sample size of 500 is generally adequate, and some authors cited indicate that sample sizes in the range of 100-250 may be sufficient.
Table 3. Estimated population distribution and expected numbers of completed extended topical surveys for key subgroups
Characteristic |
Estimated percent of adult population |
Expected number of completed interviews |
|
|
|
|
|
Overall |
100 |
4,505 |
|
|
|
|
|
Credential status |
|
|
|
|
Has certificate, certification, or license1 |
24 |
1,092 |
|
|
|
|
Education attainment |
|
|
|
|
High school diploma or less |
66 |
2,995 |
|
Associates degree, some college, other less than BA |
6 |
277 |
|
|
|
|
Age |
|
|
|
|
18-30 |
23 |
1,023 |
|
31-45 |
29 |
1,328 |
|
46 and older |
48 |
2,155 |
|
|
|
|
By education attainment
|
|
|
|
Less than Bachelor’s degree |
|
|
|
|
Has certificate, certification, or license1 |
14 |
621 |
|
Hispanic |
11 |
481 |
|
Nonwhite, not Hispanic |
14 |
629 |
Associates degree, some college, other less than BA |
|
|
|
|
Has certificate, certification, or license1 |
2 |
105 |
|
Hispanic |
0.7 |
32 |
|
Nonwhite, not Hispanic |
1 |
58 |
|
|
|
|
1 The percentage of adults who reported their occupation has legal or professional requirements for continuing education or training.
NOTE: Expected numbers of completed interviews were calculated by applying the estimated percent of the adult population (accurate to the hundredths) to the total expected sample size of 4,505. The “less than Bachelor’s degree” subgroup contains all adults in the “Associates degree, some college, other less than BA” subgroup as well as those in the “high school diploma or less” subgroup.
SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Education for Work-Related Reasons Survey of the National Household Education Surveys Program (NHES), 2005.
There are no plans to release survey estimates from the ATES Pilot Test; the aim of the Pilot Test is to provide a large scale methodological evaluation of survey measures. However, to facilitate computation of preliminary estimates of numbers of C&C holders (overall and for key subgroups) so as to aid in the evaluation of the accuracy of the survey measures, weights will be computed to account for differential within-household selection probabilities. These weights will not account for biases that could be introduced due to differential nonresponse to the Screener or extended topical survey, or due to noncoverage of adults in households for which no telephone number was available.
The ATES Pilot Test will contain special methodology and design features to address household research concerns for the future use of the ATES items. These features are summarized below.
Proxy response. It is often difficult in household surveys to contact and interview a specific adult householder. For this reason, response from another adult in the household is sometimes used as a proxy for the specific sampled adult or for all adults in the household. There are concerns about the ability of proxy responders to report accurately about C&C. The ATES cognitive interview research has suggested that spouses and partners feel confident in reporting some basic information by proxy, such as whether or not their spouse or partner has a certification and the occupational field, but not detailed information such as credit hours or provider.
The Pilot Test will be used to empirically evaluate the feasibility of using a proxy responder to report on C&C. Because proxy response could potentially have a negative impact on data accuracy, we will keep the proxy interview separate from the main interview.
One approach to this would be to directly examine consistency between the self-report responses and the proxy responses for the same sampled adult. However, this approach requires greater resources in order to interview two adults in a household. Therefore, a less resource intensive alternative approach is proposed. We will conduct proxy interviews about adults and examine item-by-item differences between item response rates for self-reports versus for proxy reports. We will look at the percentage of “don’t know” responses. The plan is to identify proxy respondents using a two-pronged approach:
After the sampled adult has completed the interview about him/herself, ask him/her to complete an interview about a pre-indentified proxy subject (who is another adult household member).
Once the calling protocol has been fulfilled, if person contact is made on the final attempt but an interview cannot be completed with the sampled adult, ask a proxy respondent (another adult household member) to complete an interview about the sampled adult.
Both the sampled adult and the proxy subject will be identified during sample selection based on the adult household members enumerated in the Screener. Table 4 shows the numbers of completed proxy interviews required to support detection of differences in item response rates; various levels of item response rates and of differences in item response rates are presented in this table.
Table 4. Sample sizes (numbers of completed proxy interviews) required to support detection of various levels of differences in item response rates |
||||
Percentage point difference in item response rates (self-report vs. proxy report) |
Item response rate for self-reports |
|||
80 percent |
85 percent |
90 percent |
95 percent |
|
2 percent |
1,627 |
1,237 |
852 |
482 |
3 percent |
611 |
494 |
367 |
230 |
4 percent |
333 |
277 |
213 |
142 |
5 percent |
213 |
180 |
142 |
99 |
6 percent |
150 |
128 |
103 |
75 |
7 percent |
112 |
97 |
79 |
59 |
8 percent |
87 |
76 |
64 |
49 |
9 percent |
70 |
62 |
52 |
41 |
10 percent |
58 |
52 |
44 |
35 |
NOTE: Sample sizes presented here are based on the assumption that a one-tailed test is used; differences are expected to be detectable at a level α=0.05.
Seeded sample. An opportunity sample of approximately 1,250 respondents with known credentials will be included in the study in order to evaluate the rate of false negatives on reports of certificates, certifications, and licenses and to examine the measurement properties of items related to the classification of these credentials. We currently have data from two states (Maine and California), and are working with two national accrediting bodies (Institute for Credentialing Excellence and the American National Standards Institute) to develop a list sampling frame that includes name, address, and name/subject of credential. Data collection procedures for the seeded sample will be roughly comparable to those for the address-based sample (mail screener/telephone extended topical interview) with small modifications to personalize the data collection to the sampled adult. Because the sample drawn from these lists will have an unknown probability of selection relative to universe totals, we will not include the seeded sample in our calculations of representative estimates from the Pilot Test but will rather use this expanded sample to evaluate the discriminative properties of questionnaire items.
Promised Incentive. Households will be promised a $0, $10, or $20 incentive if the sampled person completes the extended topical survey. All adults assigned to the $10 or $20 group who complete the extended topical survey will be sent a check for the incentive amount. In order to provide information about the effect of the level of incentive and promise, the design will be tested experimentally. In 60 percent of the screener mailings (75% of the 80% of households assigned to the $10 or $20 group), the letter enclosed in the Screener mailings will notify the household that if an adult in the household is selected for the follow-up (extended topical) telephone survey, the adult will be offered a specified amount to complete the survey. Half of these respondents will be notified of a $10 incentive and half will be notified of a $20 incentive. Respondents will be reminded of the incentive at the start of the phone interview. In the remaining 25 percent of the households assigned to the $10 or $20 group, the respondent will be notified of the incentive amount only when they are contacted for the phone interview. Although promised incentives have been shown in random digit dial (RDD) surveys to be less effective than prepaid incentives, their relative effectiveness in two-phase surveys such as the ATES Pilot Test is unknown. Due to the fact that the household will have already received an incentive in the initial Screener mailing, and a relationship with the household will have already been established, it is possible that the relative effectiveness of the promised incentive to a prepaid incentive may be different in this context.
The ATES design incorporates a number of features to maximize response rates. This section discusses those features.
Total Design Method/Respondent Friendly Design. This approach combines the attributes of the least expensive and best methods available beginning with the least labor intensive mode to a mode requiring increasingly greater amounts of labor. While this places an emphasis on use of resources, these procedures create a respondent friendly approach that uses design attributes, a scheduled sequence of contacts, and survey mode to motivate and encourage survey participation. Surveys that take advantage of respondent friendly design have demonstrated increases in survey response (Dillman, Smyth, and Christian 2008; Dillman, Sinclair, and Clark, 1993).
Engaging Respondent Interest and Cooperation. The content of respondent letters and frequently asked questions (FAQs) will be focused on communicating the legitimacy and importance of the study. Interviewer training will focus on strategies for communicating the importance and legitimacy of the survey and gaining cooperation.
Nonresponse Followup. The data collection protocol includes several stages of nonresponse followup. In addition to the numbers of contacts, changes in follow-up method (mail, FedEx) are designed to capture the attention of potential respondents.
Flexibility in Scheduling Interviews. Whenever possible, telephone interviewers will attempt to complete extended topical interview at the time of the call. In situations where a respondent is unavailable, a call appointment will be entered into the CATI management system with notations on the best time to reach the respondent.
Incentives. Incentives will be used at both the screener and extended topical level. A prepaid incentive of $2 will be used at the screener, a $0 and a promised incentive of $10 or $20 will be used for the extended topical survey.
The ATES Pilot was preceded by 3 focus groups and 3 rounds of cognitive interviews. The 3 focus groups were conducted with individuals from the Washington, DC Metro area who self-identified as being certified, registered or licensed, or seeking this type of credential in the fields of information technology, healthcare, or business. The cognitive interviews were conducted by phone in three rounds of 20 people each. Participants were selected based on their self-reported credential status, education attainment, and age. Round 1 contained a quota of participants in the construction industry. Cognitive interview participants were located in areas around Washington, DC, Charlotte, North Carolina, and Minneapolis, Minnesota.
Key findings from the focus groups were:
Participants’ terminology and understanding of certifications was in line with the experts.
Obtaining a certification was seen as rigorous and required examination.
Certification and licensure are linked.
Certification is listed on a resume as “Professional Certifications.”
A certificate is often interpreted as a “certificate of completion” rather than completion of a program of study at an education institution.
Key findings from the cognitive interviews were:
Respondents generally report correctly about “professional certification” and “state or industry license.”
Respondents who have a certification will answer that they have a certificate if certificates are asked about first.
Respondents will report certificate of completion as an education certificate if no context or definition of a certificate is provided.
The shorter definition of certificate worked well for most people.
When certification is the process leading to licensure, the respondent uses the term “license.”
Some degree programs (i.e. nursing) lead to certification.
Some certificate programs are a set number of courses that also comprise part of a degree program which can cause confusion as to whether or not the training is “part of a degree program.”
Some education certificates lead to certifications.
Some industries like IT and Automotive require multiple certifications.
Individuals Responsible for Study Design and Performance
The people listed below participated in the study design and are responsible for the collection and analysis of the data.
Sharon Boivin, NCES 202/502-7627
Stacey Bielick, ESSI 202/403-6140
Kwang Kim, Westat 301/517-4078
Jill Montaquila, Westat 301/517-4046
Blumberg, S.J. and Luke, J.V. (2009). Wireless substitution: Early release of estimates from the National Health Interview Survey, January-June 2009. National Center for Health Statistics. December 2009. Available from: http://www.cdc.gov/nchs/nhis.htm (Accessed May 7, 2010).
Dillman, D.A., Sinclair, M.D., and Clark, J.R. (1993). Effects of questionnaire length, respondent-friendly design, and difficult questions on response rates for occupant-addressed Census mail surveys. Public Opinion Quarterly, 57, 289-304.
Dillman, D.A., Smyth, J.D., and Christian, L.M. (2008). Internet, mail, and mixed mode surveys: The Tailored Design Method. New York: Wiley.
Fahimi, M., Kulp, D., and Brick, J.M. (2009). A reassessment of list-assisted RDD methodology. Public Opinion Quarterly, 73(4): 751-60.
MacCallum, R.C., Widaman, K.F., Zhang, S., and Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1): 84-99.
U.S. Department of Labor. (2008). National Compensation Survey: Occupational Wages in the United States: July 2007. Washington, D.C.: Bureau of Labor Statistics. Available online at http://www.bls.gov/ncs/ocs/sp/nctb0298.pdf
1 While we are using the term “Screener” here (to adhere to conventional terminology), it is a bit of a misnomer in this context, in that it is actually an enumeration instrument and not a screening device.
2 USPS Priority Mail will be used for addresses that cannot be delivered via FedEx (e.g., P.O. Box addresses).
| File Type | application/msword |
| Author | WAITS_T |
| Last Modified By | #Administrator |
| File Modified | 2010-08-19 |
| File Created | 2010-08-19 |