Justification

Volume I

2017 National Household Education Surveys Program (NHES)

Adult Training and Education Survey (ATES)

Preparation for Certifications and Licenses

Cognitive Interviews and Focus Groups

OMB# 1850-0803 v.149

February 2016

National Center for Education Statistics (NCES)

Justification

The National Household Education Survey (NHES) is a data collection program of the National Center for Education Statistics (NCES) aimed at providing descriptive data on the educational activities of the U.S. population, focusing on topics that are appropriate for household surveys rather than institutional surveys. NHES topics have covered issues such as early childhood care and education, children’s readiness for school, parents’ perceptions of school safety and discipline, participation in before- and after-school activities, participation in adult and career education, parents’ involvement in education, school choice, homeschooling, and civic involvement. The NHES uses a two-stage design in which sampled households complete a screener questionnaire to enumerate household members and their key characteristics. Within-household sampling from the screener data determines which household member receives which topical survey. The NHES typically fields 2 to 3 topical surveys at a time, although the number has varied across its administrations. Surveys are administered in English and in Spanish.

A new topical survey, the Adult Training and Education Survey (ATES), has recently been developed to replace the original NHES survey on participation in adult and career education. The ATES focuses on the attainment of non-degree credentials. It identifies adults who have educational certificates, including the subject field of the certificate, its perceived labor market value, and its role in preparing for occupational credentialing; and counts adults who have an occupational certification or license, including the number of such credentials, type of work they are for, their perceived labor market value, and the role of education in preparing for these occupational credentials. The survey also includes a brief section on adult participation in work experience programs (such as apprenticeships). Out-of-high-school adults ages 16 to 65 are eligible to receive this topical survey.

The ATES topical survey has undergone extensive development work over the past five years. To develop the ATES instrument, NCES has: conducted focus groups in 2013 on characteristics of educational certificates and participation in work-related training; conducted cognitive interviews in 2013 and 2014 to test English versions of the ATES instrument; administered a response rate pilot study in early 2013; administered a feasibility study in 2014 to test processes for incorporating ATES into the NHES program; and in 2015 conducted a series of concept interviews and cognitive testing. These efforts led to the 2016 ATES instrument, currently being administered as part of the 2016 NHES.

Since the last cognitive testing of the ATES instrument, however, a number of new items have been added to the instrument that require additional testing. A few new follow-up items have been added to the certifications and licenses section to collect more detail on those credentials. Additionally, a new section has been added to find out who is in the pipeline for certification and licensure, and what activities the pipeline entails. Another new section asks about barriers to certification attainment (see survey items in Attachment 4). This request is to conduct cognitive interviews to test and revise the new ATES questions for inclusion in the ATES instrument in the 2017 NHES web pilot test, and to conduct focus groups with Spanish-speaking participants to understand any potential differences in terminology specific to Spanish speakers. In order to reduce respondents’ cognitive and time burdens, the cognitive test instrument includes only sections of the ATES questionnaire that contain items requiring testing.

The primary deliverable from this study will be the revised, final version of the ATES items for inclusion in the 2017 NHES web pilot. A report highlighting key findings will also be prepared.

Design

Cognitive Interviews

Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions, or to answer a series of questions about the items they just answered. Techniques include asking probing questions to clarify points that are not evident from the think-aloud comments, and responding to scenarios. In these interviews, probes will be used to:

• verify respondents’ interpretation of the question (e.g. asking for specific examples of activities in which the respondent reports participating);

• check respondents’ understanding of the meaning of specific terms or phrases used in the questions; and

• identify experiences or concepts that the respondent did not think were covered by the questions but that we consider relevant.

Interviews are expected to last about 1 hour and will be conducted by trained cognitive interviewers. This submission includes the protocol that will be used to conduct the interviews and the questionnaire to be tested (attachments 3 and 4). It is expected that the instrument and interview protocol will evolve during testing. The research will be iterative, in that question wording and format design may change during the testing period in response to problems identified during the interviews.

To adequately test the survey, it is necessary to distribute the cognitive interviews across respondents who represent the primary experiences of the target population while also obtaining sufficient numbers of respondents with similar characteristics to test each survey section. We propose to conduct 56 interviews in three rounds with adults ages 18 to 65 who are part of the work force (e.g., not retired) or who are students, with an emphasis on adults who work or study in fields that typically require a certification or license. The interviews will be distributed as follows:

• 36 adults who hold a current certification or license and

  • are not currently working on getting a new one (18) but received their certification or license in the past 5 years; or

  • are also currently working on getting a new one in addition to the certification or license they already have (18).

• 20 adults who are currently working on getting their first certification or license and

  • are currently students preparing for a certification or license by going to school after high school (10).

The interviews will be conducted in three rounds, with two rounds of revisions to the items after the first and second rounds, based on the results of interviews conducted up to that point. If at any point during a round an item is perceived as not working—as evidenced by at least 3 respondents making the same or similar error on that item—revisions may be made to that item on a case-by-case basis. Interviews will be audio-recorded. At their discretion, NCES staff may observe some of the interviews through a video-stream using GoTo Meeting video conferencing, hosted by the American Institutes for Research (AIR).

Focus Groups

The purpose of the Spanish-speaking focus groups is to learn more about the terminology Spanish-speaking credential holders use to talk about their credentials as well as how the concepts in the new items are understood. The new concepts of primary focus are:

• Different types and levels of related certifications or licenses (e.g. advanced, specialized),

• Preparation for certification or license and provider,

• Sources of financial support, and

• Barriers to certification or licensure for those who do not have these credentials.

Focus groups are expected to last about 1 hour and will be conducted by a trained focus group moderator. This submission includes the moderator’s guide that will be used to conduct the focus groups (attachment 3b).

We will hold 3 focus groups with a target of 8 participants in each for a total of 24 participants. Each focus group will have a different participant composition, as follows:

• Group 1: Spanish-speaking adults who hold a current certification or license,

• Group 2: Spanish-speaking adults who are currently working on getting their first certification or license, and

• Group 3: Spanish-speaking adults who hold a current certification or license and are currently working on getting an additional certification or license.

Consultations Outside the Agency

The ATES instrument was initially developed through the efforts of a government interagency working group – the Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA). GEMEnA has been involved with the ATES survey design since its inception and continues to provide expert advice on both policy and methodological issues. The new items tested here were developed in consultation with GEMEnA. For more information about GEMEnA’s developmental guidance on the ATES survey, please visit nces.ed.gov/surveys/gemena.

Recruiting and Paying Respondents

Participants will be recruited by AIR using multiple sources, including Craigslist ads, flyers on local community college and trade school campuses, and through personal and professional contacts. An example recruitment advertisement and e-mail are included in Attachment 1. People who have participated in cognitive studies or focus groups in the past 6 months and employees of the firms conducting the research will be excluded from participating. Interested individuals will be asked to complete an online or phone screening questionnaire (attachment 2). The 56 interviews will take place in the AIR offices in the DC-Metro area, and the 3 focus groups will take place in the AIR Austin, Texas office. To assure that participants agree to take part in the interviews and to thank them for their time and for completing the interview, each respondent will be offered $40.

Assurance of Confidentiality

Interviewers will read a confidentiality statement to respondents, and respondents will be given the opportunity to provide or decline consent before any interview is conducted (attachment 1). The consent form includes the following statement:

This cognitive interview has been approved by the Office of Management and Budget (OMB# 1850-0803). Your participation is voluntary. The information you give us will be combined with the responses of others in a summary report that does not identify you as an individual. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C. § 9573].

No personally identifiable information will be maintained after the interview analyses are completed. Data recordings will be stored on AIR’s secure data servers.

Estimate of Hour Burden

We expect the interviews to be approximately one hour in length. Screening potential participants will require 3 minutes per screening. We anticipate it will require 12 screening interviews per eligible participant (thus an estimated 960 screenings to yield 80 participants (56 cognitive interviews and 24 participants in focus groups). This will result in 48 hours of burden for the screener, and an estimated total 128 hours of respondent burden for the full study.

Table 1. Estimated response burden for ATES cognitive interviews and focus groups

Respondents	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Recruitment Screener	960	960	0.05	48
Cognitive Interviews	56	56	1.0	56
Focus Groups	24	24	1.0	24
Total	960	1,040	-	128

Estimate of Cost Burden

There is no direct cost to respondents.

Project Schedule

The project schedule calls for recruitment to begin as soon as OMB approval is received. Interviewing is expected to be completed within 2 months of OMB approval. After the interviews are completed, survey items will be finalized for the 2017 NHES web pilot.

Cost to the Federal Government

The cost to the federal government for this study is approximately $132,000.