Cognitive Interview Memo

Att_Certificates Cog Interviews Memo.doc

System Clearance for Cognitive, Pilot and Field Test Studies

Cognitive Interview Memo

OMB: 1850-0803

Document [doc]
Download: doc | pdf

Volume I:


Request for Clearance for Cognitive Interviews for New Items to Measure Educational Certificates or Similar Training among Adults in the United States


1850-0803 v.25
























March 24, 2010

Justification


Recent OECD data indicate that the United States has fallen behind other nations in the educational attainment of the adult population, which has far-reaching consequences for our future economic prosperity and ability to compete globally. In his February 2009 State of the Nation address, President Obama stated an ambitious goal: by 2020 America will once again have the highest proportion of college graduates in the world. He then asked every American


“…to commit to at least one year or more of higher education or career training. This can be a community college or a four-year school, vocational training or an apprenticeship. But whatever the training may be, every American will need to get more than a high school diploma.”


The American Graduation Initiative was introduced by the President in July 2009 to directly support this goal by helping “an additional 5 million Americans earn degrees and certificates in the next decade.”


Attaining a postsecondary credential has become increasingly important for securing opportunities to get high-return jobs in the United States in the 21st century. However, NCES has traditionally only collected data on postsecondary certificates and degrees awarded through credit-bearing instruction in traditional institutions of higher education that participate in Title IV federal student aid programs. These comprise only a portion of subbaccalaureate education and training American adults seek and complete to learn the skills they need to find and keep good-paying jobs. In fact, a 2008 study using student unit record data from Florida1 found that, in many cases, industry-recognized certifications have a greater economic value than associate degrees.


The importance of measuring educational attainment was underlined in a December 2000 report2 from the Federal Interagency Committee on Measures of Educational Attainment:


Analyses of social and economic issues often use educational attainment as an explanatory variable. The importance of education in shaping life experiences and outcomes has been well documented in relation to health status, labor force experience, earnings, criminal activity, and participation in democratic processes as well as various income support programs. The importance accorded this measure is demonstrated by its inclusion in virtually all Federal social/demographic data collection efforts (including surveys, programmatic, and administrative data collections). Agencies that collect educational attainment data include the National Center for Education Statistics, the National Center for Health Statistics, the Bureau of the Census, the Bureau of Labor Statistics, the Division of Science Resources Studies of the National Science Foundation, the Department of Veterans Affairs, the Defense Manpower Data Center of the Department of Defense, and the Bureau of Justice Statistics.


Measuring progress toward the President’s goal and improving how we collect data on educational attainment in the federal statistical system require a coordinated effort to define and enumerate certificates and industry-recognized certifications.


This feasibility study tests a small set of items developed to enumerate the stock of certifications and certificates among the US adult population. These items would then become candidates for inclusion in a national household study of US adults.


The items proposed for testing were developed based on definitions and research questions recommended in a meeting held on November 3rd and 4th with over 50 experts from federal statistical and program agencies including Education, Labor, and Census; representatives from the Council of Economic Advisors; representatives from foundations; and representatives from nonprofit groups spanning workforce, education, industry, and standards-setting groups. Subsequent to the meeting, NCES conducted focus groups with certificate and certification holders and along with an inter-agency working group, developed items to address key properties of certificates and certifications. The items proposed for clearance result from these activities.


The scope of this effort includes certificates, certifications, and “other training” that may have the same properties as certifications and certificates. We define “certifications” and “certificates” as follows:


Certificate: A credential awarded by a training program or educational institution based on completion of coursework. Knowledge-based certificates recognize a relatively narrow scope of specialized knowledge used in performing duties or tasks required by a certain profession or occupation and are issued after the individual passes an assessment instrument. Curriculum-based certificates are issued after an individual completes a course or series of courses and passes an assessment limited to the course content. A certificate is awarded once and carries no requirements for continuing education or repeated demonstration of knowledge. Certificates of attendance or participation are not in scope for this work.


Certification: A credential awarded by a certification body based on an individual demonstrating through a standardized examination process that they have acquired the designated validated knowledge, skills, and abilities. Certification is often voluntary but may be mandatory when tied to state licensure. The identified competencies must be derived from a formal process often called a job analysis. The examinations can be written, oral, or performance based, but must meet psychometric rigor to demonstrate the examinations are fair, valid, and reliable. Certification is a time-limited credential that is renewed through a re-certification process. A certification can be taken away from the individual for ethical violations or incompetence.


This request for clearance is for cognitive interviews to solicit clarification from respondents about their responses to specific questionnaire items designed to measure the prevalence of certificates and certifications or similar training among US adults. The main focus will be on respondents with certificates, however, we will also administer previously OMB-approved items on certifications if a respondent has a certification.


Design of Cognitive Interviews


The cognitive interviews will be designed as intensive, one-on-one interviews in which the adult respondent is asked to “think aloud” as he or she answers survey questions or to answer further questions and provide additional clarifications about their answers. Techniques include asking probing questions, as necessary, to clarify points that are not evident from the think-aloud comments and responding to scenarios. Probes that will be used include,

  • probes to verify respondents’ interpretation of the question (e.g. asking for specific examples of activities in which the respondent reports participating),

  • probes about the meaning of specific terms or phrases used in the questions, or

  • probes for experiences or ideas that the respondent did not think were covered by the question but we would have considered relevant.


Content. The focus of the interviews will be solely on the items being developed by NCES, plus an educational attainment item used on the American Community Survey (ACS) to provide additional context.


Mode. Because we are testing only a few focused items, the interviews will be conducted via telephone. Appointments will be set in advance with the participants.

Target population. Because we have only a few items of interest that address a very specific topic, we should be able to discern variations in the population with a fairly small number of well-targeted participants. We will interview at least 10 adults ages 21-40 with a certificate and at least 10 adults ages 21-40 from the general population. Phone respondents will be from a location in the Midwest; a location in the South; and the DC metro area.


We are requesting clearance for up to 40 interviews in order to ensure that we are able to continue testing items if early interviews show the need for more information.


Estimated Response Burden


We expect the cognitive interviews to last up to one hour, depending on the number of credentials a person has. We estimate from the prior work on certifications that respondents with multiple credentials (common in IT and construction industries) and those with the ability to provide proxy responses will require the full hour while respondents with one credential and those not receiving the proxy questions will require less time. It is not possible to accurately determine these interview properties in advance of the full interview. We estimate, on average, the interviews will take about 40 minutes. Table 1 shows the main recruitment groups and their maximum burden hours. The table includes the reserve respondent sample that we will use if needed and the screener used to determine subject’s eligibility to participate in the cognitive lab.








Table 1. Maximum burden time for main recruitment groups

Respondents

Number of Respondents

Number of Responses

Burden Hours per Respondent

Total Burden Hours

Adults with a certificate

10

10

1

10






General adults

10

10

1

10






Reserve respondents

20

20

1

20






Screener Recruitment Interview

480

480

0.1

48

Total

480

520

-

88



Estimated Cost Burden


There is no direct cost to respondents.


Recruitment and payment to respondents


Participants completing the cognitive interview will receive $40. The recruitment and research firm, Shugoll Research, working in partnership with the American Institutes for Research, will recruit participants using their database of research volunteers in the Washington, DC metro area and the databases of research affiliates in the Midwest and South.


Assurance of confidentiality


Participation is voluntary and respondents will be read a confidentiality pledge before interviews are conducted. Shugoll Research and the American Institutes for Research are conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. This study is authorized by law under the Education Sciences Reform Act (Public Law 107-279). Your participation is voluntary. Your responses are protected from disclosure by federal statute (P.L. 107-279, Title 1, Part E, Sec. 183). All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose, unless otherwise compelled by law.


No personally identifiable information will be maintained after the cognitive laboratory analyses are completed.




Cost to Federal Government


We are requesting clearance for 20 interviews plus a reserve of another 20. If we conducted all 40, the cost to the government will be $40,240.80. If we only conduct 20, the cost will be $20,120.40


Project Schedule



Activity

Timeline

Develop initial items and protocol

March 1, 2010 to March 24, 2010

Test items with cognitive interviews

April 6, 2010 (or upon clearance)

to May 30, 2010


1 Jacobson, L. and Moker, C. “Pathways to Boosting the Earnings of Low-Income Students by Increasing Their Educational Attainment,” The Hudson Institute and CNA, November 2008.

2 “Federal Measures of Educational Attainment: Report and Recommendations,” prepared by The Federal Interagency Committee on Measures of Educational Attainment, December 2000.

File Typeapplication/msword
File TitleVolume I:
AuthorNHES
Last Modified By#Administrator
File Modified2010-04-01
File Created2010-04-01

© 2024 OMB.report | Privacy Policy