Justification

Volume I

2021 Adult Training and Education Survey (ATES)

Cognitive Interviews – First Round

OMB# 1850-0803 v. 227

March 2018

National Center for Education Statistics (NCES)

Justification

The Adult Training and Education Survey (ATES) focuses on the attainment of non-degree credentials and was first administered in 2016 as part of the National Household Education Survey (NHES). It identifies adults who have an occupational certification or license, including the number of such credentials, type of work they are for, their perceived labor market value, and the role of education in preparing for these occupational credentials. The survey also includes a brief section on postsecondary certificates and a section on adult completion of work experience programs (such as apprenticeships). Out-of-high-school adults ages 16 to 65 are eligible to receive this survey.

The ATES instrument has undergone extensive development work over the past six years. To develop the instrument, NCES has: conducted focus groups in 2013 on characteristics of educational certificates and participation in work-related training; conducted cognitive interviews in 2013 and 2014 to test English versions of the ATES instrument; administered a response rate pilot study in early 2013; administered a feasibility study in 2014 to test processes for incorporating ATES into the NHES program; and conducted a series of concept interviews and cognitive testing in 2015 to develop the 2016 ATES instrument. After the first administration of the ATES instrument in 2016, a series of focus groups (in Spanish only), concept interviews, cognitive testing in both English and Spanish, and usability interviews were conducted to revise the instrument for a 2017 pilot test of an NHES web administration.

Since the 2017 web pilot (OMB# 1850-0803 v.163, 182, & 187) and the last cognitive testing of the ATES instrument (OMB# 1850-0803 v.195 & 205), a number of detailed analyses have been conducted on the 2016 ATES data, resulting in revisions that include reordering and rewording questions and adding new questions, all of which require additional testing. The certificate-related questions have been reordered to fall within questions about educational attainment; the certification and license questions have been reordered to follow and link to questions about employment; and the work experience program section questions were revised to better distinguish between apprenticeships and internships. New questions have been added to get more detailed information on current postsecondary education enrollments and to better link credential attainment to workers’ goals and outcomes. Additionally, ATES will adopt new education and employment questions derived from the National Survey of College Graduates (NSCG) and the American Community Survey (ACS). The intent of these changes is to improve the ability of analysts to use these data sources together to generate rich and robust analyses of work credentials in the United States, including credentials held by workers across all levels of the scientific workforce.

ATES will not be fielded as part of NHES:2019 and instead will undergo a pilot test in 2019 for a planned 2021 full administration. This request is to conduct cognitive interviews in English to test the new and revised questions for inclusion in the 2019 ATES pilot test. In order to reduce respondents’ cognitive and time burdens, the cognitive test instrument (Attachment 4) includes only sections of the ATES questionnaire that contain items requiring testing. Some background questions inquiring about education and employment have also been included to provide context for the (new and revised) certification, licensure, and work experience program items.

The primary deliverable from the requested here cognitive interviews will be revised questions for inclusion in the 2019 pilot test. A report highlighting key findings will also be prepared.

Design

Cognitive Interviews

Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions, or to answer a series of questions about the items they just answered. Techniques include asking probing questions to clarify points that are not evident from the think-aloud comments, and responding to scenarios. In these interviews, probes will be used to:

• verify respondents’ interpretation of the question (e.g. asking for specific examples of activities in which the respondent reports participating);

• check respondents’ understanding of the meaning of specific terms or phrases used in the questions; and

• identify experiences or concepts that the respondent did not think were covered by the questions but that we consider relevant.

Interviews are expected to last about 1 hour and will be conducted by trained cognitive interviewers. This submission includes the recruitment screener and interview protocol that will be used to conduct the interviews and the questionnaire to be tested (Attachments 3 and 4). It is expected that the instrument and interview protocol will evolve during testing. The research will be iterative, in that question wording and format design may change during the testing period in response to problems identified during the interviews.

To adequately test the survey, it is necessary to distribute the cognitive interviews across respondents who represent the primary experiences of the target population while also obtaining sufficient numbers of respondents with similar characteristics to test each survey section. We propose to conduct 60 interviews, all in English. These interviews will be held with employed adults ages 18 to 65.

The interviews will be distributed as follows:

• 8 interviews with adults who are currently enrolled in or taking courses from a college, university, technical or trade school, or other school after high school;

• 10 interviews with adults who have a subbaccalaureate credential (focusing on certificates);

• 22 interviews with adults who have a professional certification or license; and

• 20 interviews with adults who have participated in a work-experience program.

We will also screen for participants who work in a job that uses science, medicine, engineering, mathematics, or computer science.

The interviews will be conducted in three rounds, with revisions to the items being made between the rounds based on the results of interviews conducted up to that point. If at any point during a round an item is perceived as not working—as evidenced by at least 3 respondents making the same or similar error or comment on that item—revisions may be made to that item during the round on a case-by-case basis. Interviews will be audio-recorded. At their discretion, NCES and NSCG staff may observe some of the interviews through a video-stream using GoTo Meeting video conferencing, hosted by the American Institutes for Research (AIR).

Consultations Outside the Agency

The ATES instrument was initially developed through the efforts of a government interagency working group – the Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA). GEMEnA has been involved with the ATES survey design since its inception and continues to provide expert advice on both policy and methodological issues. For more information about GEMEnA’s developmental guidance on the ATES survey, please visit nces.ed.gov/surveys/gemena.

For the revisions being tested for the 2019 ATES pilot test, NCES has collaborated closely with NSCG experts at the National Science Foundation (NSF) through a series of planning and development meetings in early 2018. NCES will continue to work closely with NSCG staff throughout the 2019 ATES pilot.

Recruiting and Paying Respondents

Participants will be recruited by AIR using multiple sources, including Craigslist ads, flyers on local community college and trade school campuses, and through personal and professional contacts. An example recruitment advertisement and e-mail are included in Attachment 1. People who have participated in cognitive studies or focus groups in the past 6 months and employees of the firms conducting the research will be excluded from participating. Interested individuals will be asked to complete an online or phone screening questionnaire (Attachment 2). Most of the 60 interviews will take place in the AIR offices in the DC-Metro area, with a few interviews to occur in quiet, public places, such as a library or community center, in the Grand Rapids, Michigan and Jacksonville, Florida areas, or other areas near AIR in Austin, Texas; San Mateo, California; and Chicago, Illinois. To assure that participants agree to take part in the interviews and to thank them for their time and for completing the interview, each respondent will be offered $40.

Assurance of Confidentiality

Interviewers will read a confidentiality statement to respondents, and respondents will be given the opportunity to provide or decline consent before any interview is conducted (Attachment 1). The consent form includes the following statement:

This cognitive interview has been approved by the Office of Management and Budget (OMB# 1850-0803). Your participation is voluntary. The information you give us will be combined with the responses of others in a summary report that does not identify you as an individual. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

No personally identifiable information will be maintained after the interview analyses are completed. Data recordings will be stored on AIR’s secure data servers.

Estimate of Hour Burden

We expect the interviews to be approximately one hour in length. Screening potential participants will require 3 minutes per screening. We anticipate it will require 12 screening interviews per eligible participant (thus an estimated 720 screenings to yield 60 participants). This will result in an estimated total of 96 hours of respondent burden for this study.

Table 1. Estimated response burden for ATES cognitive interviews and focus groups

Respondents	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Recruitment Screener	720	720	0.05	36
Cognitive Interviews	60	60	1.0	60
Total	720	780	-	96

Estimate of Cost Burden

There is no direct cost to respondents.

Project Schedule

The project schedule calls for recruitment to begin in March 2018, as soon as OMB approval is received. Interviewing is expected to be completed within 5 months of OMB approval. After the interviews are completed, survey items will be finalized for the 2019 ATES pilot.

Cost to the Federal Government

The cost to the federal government for this study is approximately $88,375.