Justification

Volume I:

Request for Clearance for Cognitive Interviews for New Items to Measure Professional Certifications and Educational Certificates among Adults in the United States

1850-0803 v.30

June 2, 2010

Justification

Recent OECD data indicate that the United States has fallen behind other nations in the educational attainment of the adult population, which has far-reaching consequences for our future economic prosperity and ability to compete globally. In his February 2009 State of the Nation address, President Obama stated an ambitious goal: by 2020 America will once again have the highest proportion of college graduates in the world. He then asked every American

“…to commit to at least one year or more of higher education or career training. This can be a community college or a four-year school, vocational training or an apprenticeship. But whatever the training may be, every American will need to get more than a high school diploma.”

The American Graduation Initiative was introduced by the President in July 2009 to directly support this goal by helping “an additional 5 million Americans earn degrees and certificates in the next decade.”

Attaining a postsecondary credential has become increasingly important for securing opportunities to get high-return jobs in the United States in the 21st century. However, NCES has traditionally only collected data on postsecondary certificates and degrees awarded through credit-bearing instruction in traditional institutions of higher education that participate in Title IV federal student aid programs. These comprise only a portion of subbaccalaureate education and training American adults seek and complete to learn the skills they need to find and keep good-paying jobs. In fact, a 2008 study using student unit record data from Florida¹ found that, in many cases, industry-recognized certifications have a greater economic value than associate degrees.

The importance of measuring educational attainment was underlined in a December 2000 report² from the Federal Interagency Committee on Measures of Educational Attainment:

Analyses of social and economic issues often use educational attainment as an explanatory variable. The importance of education in shaping life experiences and outcomes has been well documented in relation to health status, labor force experience, earnings, criminal activity, and participation in democratic processes as well as various income support programs. The importance accorded this measure is demonstrated by its inclusion in virtually all Federal social/demographic data collection efforts (including surveys, programmatic, and administrative data collections). Agencies that collect educational attainment data include the National Center for Education Statistics, the National Center for Health Statistics, the Bureau of the Census, the Bureau of Labor Statistics, the Division of Science Resources Studies of the National Science Foundation, the Department of Veterans Affairs, the Defense Manpower Data Center of the Department of Defense, and the Bureau of Justice Statistics.

Measuring progress toward the President’s goal and improving how we collect data on educational attainment in the federal statistical system require a coordinated effort to define and enumerate certificates and industry-recognized certifications.

This feasibility study tests a small set of items developed to enumerate the stock of certifications and certificates among the US adult population. These items would then become candidates for inclusion in a national household study of US adults.

The items proposed for testing were developed based on definitions and research questions recommended in a meeting held on November 3^rd and 4^th with over 50 experts from federal statistical and program agencies including Education, Labor, and Census; representatives from the Council of Economic Advisors; representatives from foundations; and representatives from nonprofit groups spanning workforce, education, industry, and standards-setting groups. Subsequent to the meeting, NCES conducted focus groups with certificate and certification holders and along with an inter-agency working group, developed items to address key properties of certificates and certifications. The items proposed for clearance result from these activities.

The scope of this effort includes certificates and certifications. We define “certifications” and “certificates” as follows:

Certificate: A credential awarded by a training program or educational institution based on completion of coursework. Knowledge-based certificates recognize a relatively narrow scope of specialized knowledge used in performing duties or tasks required by a certain profession or occupation and are issued after the individual passes an assessment instrument. Curriculum-based certificates are issued after an individual completes a course or series of courses and passes an assessment limited to the course content. A certificate is awarded once and carries no requirements for continuing education or repeated demonstration of knowledge. Certificates of attendance or participation are not in scope for this work.

Certification: A credential awarded by a certification body based on an individual demonstrating through a standardized examination process that they have acquired the designated validated knowledge, skills, and abilities. Certification is often voluntary but may be mandatory when tied to state licensure. The identified competencies must be derived from a formal process often called a job analysis. The examinations can be written, oral, or performance based, but must meet psychometric rigor to demonstrate the examinations are fair, valid, and reliable. Certification is a time-limited credential that is renewed through a re-certification process. A certification can be taken away from the individual for ethical violations or incompetence.

This request for clearance is for cognitive interviews to solicit clarification from respondents about their responses to specific questionnaire items designed to measure the prevalence of certificates and certifications or similar training among US adults. The main focus will be on certifications and certificates, however, we will also administer previously OMB-approved items on adult background characteristics and ask questions about a survey cover letter and short mail screener.

Design of Cognitive Interviews

The cognitive interviews will be designed as intensive, one-on-one interviews in which the adult respondent is asked to “think aloud” as he or she answers survey questions or to answer further questions and provide additional clarifications about their answers. Techniques include asking probing questions, as necessary, to clarify points that are not evident from the think-aloud comments and responding to scenarios. Probes that will be used include,

• probes to verify respondents’ interpretation of the question (e.g. asking for specific examples of activities in which the respondent reports participating),

• probes about the meaning of specific terms or phrases used in the questions, or

• probes for experiences or ideas that the respondent did not think were covered by the question but we would have considered relevant.

Content. The focus of the interviews will be on the items being developed by NCES.

Mode. Because we are testing only a few focused items, the interviews will be conducted via telephone. Appointments will be set in advance with the participants. We will mail participants the cover letter and short mail screener in advance.

Target population. Because we have only a few items of interest that address a very specific topic, we should be able to discern variations in the population with a fairly small number of well-targeted participants. We will interview 10 adults with a likely certificate or certification and 10 adults with less than a bachelor’s degree. Phone respondents will be from a location in the Midwest; a location in the South; and the DC metro area.

Estimated Response Burden

We expect the cognitive interviews to last up to one hour, depending on the number of credentials a person has. We estimate from the prior work on certifications that respondents with multiple credentials (common in IT and construction industries) and those with the ability to provide proxy responses will require the full hour while respondents with one credential and those not receiving the proxy questions will require less time. It is not possible to accurately determine these interview properties in advance of the full interview. We estimate, on average, the interviews will take about 40 minutes. Table 1 shows the main recruitment groups and their maximum burden hours.

Table 1. Maximum burden time for main recruitment groups

Respondents	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Recruitment Screener Interview	240	240	0.1	24
Mail screener	20	20	0.1	2
Adults with a credential	10	10	1	10
Adults with less than a bachelor’s degree	10	10	1	10
Total	240	260	1.2	46

Estimated Cost Burden

There is no direct cost to respondents.

Recruitment and payment to respondents

Participants completing the cognitive interview will receive $40. The recruitment and research firm, Shugoll Research, working in partnership with the American Institutes for Research, will recruit participants using their database of research volunteers in the Washington, DC metro area and the databases of research affiliates in the Midwest and South.

Assurance of confidentiality

Participation is voluntary and respondents will be read a confidentiality pledge before interviews are conducted. Shugoll Research and the American Institutes for Research are conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. This study is authorized by law under the Education Sciences Reform Act (Public Law 107-279). Your participation is voluntary. Your responses are protected from disclosure by federal statute (P.L. 107-279, Title 1, Part E, Sec. 183). All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose, unless otherwise compelled by law.

No personally identifiable information will be maintained after the cognitive laboratory analyses are completed.

Cost to Federal Government

We are requesting clearance for 20 interviews. The cost will be $20,120.40

Project Schedule

Activity	Timeline
Develop initial items and protocol	May 1, 2010 to June 2, 2010
Test items with cognitive interviews	June 10, 2010 (or upon clearance) to July 30, 2010

1 Jacobson, L. and Moker, C. “Pathways to Boosting the Earnings of Low-Income Students by Increasing Their Educational Attainment,” The Hudson Institute and CNA, November 2008.

2 “Federal Measures of Educational Attainment: Report and Recommendations,” prepared by The Federal Interagency Committee on Measures of Educational Attainment, December 2000.