Justification

Volume I:

Request for Clearance for Focus Groups and Cognitive Interviews About Adult Training and Education

1850-0803 v.69

August 1, 2012

Justification

The Departments of Education, Commerce, and Labor have been working jointly, via an Interagency Working Group (IWG), on the development of survey items to learn about adult training and education, and particularly, certificates and certifications, among the US adult population. The IWG’s work has recently been expanded to include the development of improved measures of adults’ participation in learning outside of postsecondary education programs. The proposed focus groups and cognitive interviews continue these lines of work. This effort builds on the IWG’s previous work on certificates and certifications, which were defined for that effort as follows:

Certificate: A credential awarded by a training program or educational institution based on completion of coursework. Knowledge-based certificates recognize a relatively narrow scope of specialized knowledge used in performing duties or tasks required by a certain profession or occupation and are issued after the individual passes an assessment instrument. Curriculum-based certificates are issued after an individual completes a course or series of courses and passes an assessment limited to the course content. A certificate is awarded once and carries no requirements for continuing education or repeated demonstration of knowledge. Certificates of attendance or participation are not in scope for this work.

Certification: A credential awarded by a certification body based on an individual demonstrating through a standardized examination process that they have acquired the designated validated knowledge, skills, and abilities. Certification is often voluntary but may be mandatory when tied to state licensure. The identified competencies must be derived from a formal process often called a job analysis. The examinations can be written, oral, or performance based, but must meet psychometric rigor to demonstrate the examinations are fair, valid, and reliable. Certification is a time-limited credential that is renewed through a re-certification process. A certification can be taken away from the individual for ethical violations or incompetence.

Although one focus of this research will be on certificates and certifications, we will also administer previously OMB-approved items and new experimental items on educational attainment and participation in learning in preparation for a pilot test of a new Department of Education survey of adults (for which a separate OMB submittal will be forthcoming later in 2012). This submittal requests approval for:

(1) focus groups on certificates. The focus groups will address problems in the measurement of educational certificates that were discovered in the IWG’s 2010 Adult Training and Education Study (ATES) pilot test. This pilot test resulted in high levels of variation in the educational certificate survey measures, which suggested potential problems with respondents’ understanding of the certificate construct. The focus groups will query individuals working in fields were certificates are a common job requirement, asking about their job qualifications in order to help us better understand how adults describe and use certificates for work.

(2) cognitive interviews of a draft questionnaire on adult learning. The cognitive interviews will test a self-administered, paper mail survey that expands the original ATES questionnaire, to include both more in-depth information on certificates and certifications, and a broader range of information on participation in learning. This questionnaire is being developed to test the feasibility of a mail survey for collecting information about adult training and education (the National Adult Training and Education Survey (NATES)). A mail-out paper survey is being used because the Department of Education’s National Household Education Survey has found telephone surveys infeasible as a method for interviewing households, particularly on the topic of adult education. We will also test a household screener which will be part of a small experiment and the initial survey invitation letter.

We are requesting clearance by August 10, 2012.

Research Design

We will conduct 3 focus groups during evening or lunch hours. A professional focus group moderator will conduct the groups.

The main focus group research questions are as follows:

1. What language do people use when they talk about their certificate? Do they “graduate” from a program? Do they receive a piece of paper, etc.?

2. How can we better ask about the relationship between certificates and other forms of education and training, including certifications, licenses, and degrees? Do certificates become “irrelevant” once people obtain these other qualifications?

3. How salient is their certificate to them in terms of their current or future employment? Where would they list it on their resume? If we needed to verify that they had a certificate, who would we speak to? Would their employer know?

4. How can we better ask about the level of effort required to obtain a certificate? In particular, was it earned in part-time or full-time program or through sequential versus non-sequential program attendance?

5. How can we better ask about certificate providers to distinguish between place/location of a program and its sponsor?

6. How can we better ask about the benefits/reasons for getting a certificate?

To adequately test the items, it is necessary to recruit participants who are known or likely certificate holders. To address this objective, we will target certificate holders in the following industries, as recommended by the Interagency Working Group (IWG): (1) Cosmetology; (2) Construction, Manufacturing, Mechanics, Transportation; and (3) Health Services, which represent the industries with the largest number of certificate holders. We will conduct one focus group for each industry grouping. We expect to conduct a total of 24 to 30 interviews in 3 groups of 8 to 10.

The cognitive interviews will be designed as intensive, one-on-one interviews in which the adult respondent is asked to “think aloud” as he or she answers survey questions and to provide additional clarification about their answers. Techniques include asking probing questions, as necessary, to clarify points that are not evident from the think-aloud comments and responding to scenarios. Probes that will be used include,

• probes to verify respondents’ interpretation of the question (e.g. asking for specific examples of activities in which the respondent reports participating),

• probes about the meaning of specific terms or phrases used in the questions, or

• probes for experiences or ideas that the respondent did not think were covered by the question but we would have considered relevant.

The focus groups and cognitive interviews will be conducted in-person. Participants will be recruited by contacting local businesses, college and university alumni associations, and through personal contacts.

Estimated Response Burden

We expect the focus groups to last 90 minutes. We expect the cognitive interviews to last 60 minutes.

Table 1. Estimated response burden

Respondents	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Recruitment Through Local Businesses and College/University Alumni Associations	20	20	0.05	1
Recruitment Screener for Individuals for Focus Groups and Cognitive Interviews	60	60	0.05	3
Focus Groups	30	30	1.5	45
Cognitive Interviews	15	15	1.0	15
Total	80	125	-	64

There is no direct cost to respondents.

Recruitment and Payment to Respondents

To thank the participants for their time, travel, and information, those completing the focus groups will receive $60 and those completing the cognitive interviews will receive $40.

Assurance of Confidentiality

Participation is voluntary and respondents will read a confidentiality pledge before interviews or groups are conducted. This statement is as follows: “The American Institutes for Research is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. This study is authorized by law under the Education Sciences Reform Act (ESRA, 20 U.S.C. §9543). Your participation is voluntary. Your responses are protected from disclosure by federal statute (20 U.S.C. §9573). All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose, unless otherwise compelled by law.”

No personally identifiable information will be maintained after the analyses are completed.

Cost to Federal Government

The anticipated cost for the target 45 interviews is $60,000.

Project Schedule

Activity	2012 Dates
OMB clearance	8/10
Questionnaire formatting	8/10 to 8/24
Recruitment	8/10 to 9/15
Conduct cognitive interviews	8/27 to 9/30
Conduct focus groups	9/1 to 9/30

6