Justification

Volume I:

Request for Clearance for Cognitive Interviews for New Items to Measure Industry-Recognized Certifications or Similar Training among Adults in the United States

1850-0803 v.21

February 9, 2010

Justification

Recent OECD data indicate that the United States has fallen behind other nations in the educational attainment of the adult population, which has far-reaching consequences for our future economic prosperity and ability to compete globally. In his February 2009 State of the Nation address, President Obama stated an ambitious goal: by 2020 America will once again have the highest proportion of college graduates in the world. He then asked every American

“…to commit to at least one year or more of higher education or career training. This can be a community college or a four-year school, vocational training or an apprenticeship. But whatever the training may be, every American will need to get more than a high school diploma.”

The American Graduation Initiative was introduced by the President in July 2009 to directly support this goal by helping “an additional 5 million Americans earn degrees and certificates in the next decade.”

Attaining a postsecondary credential has become increasingly important for securing opportunities to get high-return jobs in the United States in the 21st century. However, NCES has traditionally only collected data on postsecondary certificates and degrees awarded through credit-bearing instruction in traditional institutions of higher education that participate in Title IV federal student aid programs. These comprise only a portion of subbaccalaureate education and training American adults seek and complete to learn the skills they need to find and keep good-paying jobs. In fact, a 2008 study using student unit record data from Florida¹ found that, in many cases, industry-recognized certifications have a greater economic value than associate degrees.

The importance of measuring educational attainment was underlined in a December 2000 report² from the Federal Interagency Committee on Measures of Educational Attainment:

Analyses of social and economic issues often use educational attainment as an explanatory variable. The importance of education in shaping life experiences and outcomes has been well documented in relation to health status, labor force experience, earnings, criminal activity, and participation in democratic processes as well as various income support programs. The importance accorded this measure is demonstrated by its inclusion in virtually all Federal social/demographic data collection efforts (including surveys, programmatic, and administrative data collections). Agencies that collect educational attainment data include the National Center for Education Statistics, the National Center for Health Statistics, the Bureau of the Census, the Bureau of Labor Statistics, the Division of Science Resources Studies of the National Science Foundation, the Department of Veterans Affairs, the Defense Manpower Data Center of the Department of Defense, and the Bureau of Justice Statistics.

Measuring progress toward the President’s goal and improving how we collect data on educational attainment in the federal statistical system require a coordinated effort to define and enumerate certificates and industry-recognized certifications.

This feasibility study tests a small set of items developed to enumerate the stock of certifications among the US adult population. These items would then become candidates for inclusion in a national household study of US adults.

The items proposed for testing were developed based on definitions and research questions recommended in a meeting held on November 3^rd and 4^th with over 50 experts from federal statistical and program agencies including Education, Labor, and Census; representatives from the Council of Economic Advisors; representatives from foundations; and representatives from nonprofit groups spanning workforce, education, industry, and standards-setting groups. Subsequent to the meeting, NCES conducted focus groups with certification holders and along with an inter-agency working group, developed items to address key properties of certifications. The items proposed for clearance result from these activities.

The scope of this effort includes industry-recognized certifications and “other training” that may have the same properties as certifications. We define “certification” as follows:

Certification: A credential awarded by a certification body based on an individual demonstrating through a standardized examination process that they have acquired the designated validated knowledge, skills, and abilities. Certification is often voluntary but may be mandatory when tied to state licensure. The identified competencies must be derived from a formal process often called a job analysis. The examinations can be written, oral, or performance based, but must meet psychometric rigor to demonstrate the examinations are fair, valid, and reliable. Certification is a time-limited credential that is renewed through a re-certification process. A certification can be taken away from the individual for ethical violations or incompetence.

This request for clearance is for cognitive interviews to solicit clarification from respondents about their responses to specific questionnaire items designed to measure the prevalence of certifications or similar training among US adults.

Design of Cognitive Interviews

The cognitive interviews will be designed as intensive, one-on-one interviews in which the adult respondent is asked to “think aloud” as he or she answers survey questions or to answer further questions and provide additional clarifications about their answers. Techniques include asking probing questions, as necessary, to clarify points that are not evident from the think-aloud comments and responding to scenarios. Probes that will be used include,

• probes to verify respondents’ interpretation of the question (e.g. asking for specific examples of activities in which the respondent reports participating),

• probes about the meaning of specific terms or phrases used in the questions, or

• probes for experiences or ideas that the respondent did not think were covered by the question but we would have considered relevant.

Content. The focus of the interviews will be solely on the certification items being developed by NCES, plus an educational attainment item used on the American Community Survey (ACS) to provide additional context.

Mode. Because we are testing only a few focused items, the interviews will be conducted mostly via telephone with some interviews conducted in person at a research facility in Bethesda, Maryland. Appointments will be set in advance with the participants.

Target population. Because we have only a few items of interest that address a very specific topic, we should be able to discern variations in the population with a fairly small number of well-targeted participants. We will interview at least 15 adults ages 21-40 with a certification and at least 5 adults ages 21-40 with “other training” that may have the same key properties as certifications. Of these participants, 5 will have certifications related to construction; 5 phone respondents will be from a location in the Midwest; 5 phone respondents will be from a location in the South.

We are requesting clearance for up to 40 interviews in order to ensure that we are able to continue testing items if early interviews show the need for more information.

Estimated Response Burden

We expect the cognitive interviews to be approximately 60 minutes in length. Phone interviews may be somewhat shorter. Table 1 shows the main recruitment groups and their burden hours. The table includes the reserve respondent sample that we will use if needed and the screener used to determine subject’s eligibility to participate in the cognitive lab.

Table 1. Burden time for main recruitment groups

Respondents	Mode of Interview	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Adults with a certification	Phone	10	20	1	10
	In-Person	5	10	1	5
Adults with other training	Phone	5	10	1	5
	In-Person	0	0	1	0
Reserve respondents	Phone	10	20	1	10
	In-Person	10	20	1	10
Screener Recruitment Interview	Phone	480	480	0.1	48
Total	-	480	560	-	88

Estimated Cost Burden

There is no direct cost to respondents.

Recruitment and payment to respondents

Participants completing the cognitive interview will receive $40. The recruitment and research firm, Shugoll Research, working in partnership with the American Institutes for Research, will recruit participants using their database of research volunteers in the Washington, DC metro area and the databases of research affiliates in the Midwest and South.

Assurance of confidentiality

Participation is voluntary and respondents will be read a confidentiality pledge before interviews are conducted. Shugoll Research and the American Institutes for Research are conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. This study is authorized by law under the Education Sciences Reform Act (Public Law 107-279). Your participation is voluntary. Your responses are protected from disclosure by federal statute (P.L. 107-279, Title 1, Part E, Sec. 183). All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose, unless otherwise compelled by law.

No personally identifiable information will be maintained after the cognitive laboratory analyses are completed.

Cost to Federal Government

We are requesting clearance for 20 interviews plus a reserve of another 20. If we conducted all 40, the cost to the government will be $40,240.80. If we only conduct 20, the cost will be $20,120.40

Project Schedule

Activity	Timeline
Develop initial items and protocol	January 12, 2010 to February 5, 2010
Test items with cognitive interviews	February 12, 2010 to March 31, 2010

1 Jacobson, L. and Moker, C. “Pathways to Boosting the Earnings of Low-Income Students by Increasing Their Educational Attainment,” The Hudson Institute and CNA, November 2008.

2 “Federal Measures of Educational Attainment: Report and Recommendations,” prepared by The Federal Interagency Committee on Measures of Educational Attainment, December 2000.