Justification

National Center for Education Statistics

Second Follow-up Field Test of the High School Longitudinal Study of 2009 (HSLS:09) – Recruitment and Screening for Cognitive Interviews

OMB# 1850-0803 v.114

Volume I

October 17, 2014

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803 v.114) that allows NCES to improve the methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, focus groups, and cognitive interviews. The request for approval described in this memorandum is to conduct recruitment for cognitive interviews in preparation for the Second Follow-Up Field Test of the High School Longitudinal Study of 2009 (HSLS:09). The data collection for this study is being carried out for NCES by RTI International – with Research Support Services (RSS) as a subcontractor – under contract to the U.S. Department of Education (Contract number ED-IES-14-R-0335).

The HSLS:09 Second Follow-Up will focus on respondents who have been out of high school for 2 and 3 years (including both high school graduates and students who dropped out). The survey will help to inform researchers, policymakers, and other stakeholders about issues of transition from high school to postsecondary education and the labor force, with a particular emphasis on sample members who are pursuing postsecondary education and careers in science, technology, engineering, and mathematics (STEM) fields. Cognitive interviews will help us to better understand respondent perspectives on a variety of issues and will be used to refine the survey questions to maximize the quality of data collected as part of the field test. This memo requests permission for Recruitment and Screening of potential cognitive testing participants. A subsequent memo will be submitted in late October 2014 to request approval of the cognitive interview protocol and the questions to be tested (the questions are being revised based on the feedback from Technical Review Panel (TRP) that met on October 15-16, 2014).

Design and Context

The current request is for approval to conduct recruiting, screening, and scheduling for a series of cognitive interviews planned to begin in November 2014. The cognitive research report based on the results will be available by March 2015. Recruitment and screening materials are provided in Attachments I, II, and III.

RSS (located in Evanston, Illinois) will recruit in the greater Chicago area for 30 cognitive interviews. The sample will include a mix of respondents from the following five categories – those who: are enrolled in a 4-year baccalaureate program; have completed or enrolled in a 2-year post high school program (associate’s degree); are employed; are not enrolled in any post high school program; and who left high school without a diploma. Respondents will also be selected to ensure a mix of racial and ethnic groups.

NCES is considering the inclusion of questionnaire items related to Lesbian, Gay, Bisexual, Transgender, or Questioning (LGBTQ) topics in the HSLS:09 Second Follow-Up, and RSS will attempt to recruit approximately ten participants who identify as LGBTQ. RSS has experience with recruiting sexual minorities, and their inclusion in the cognitive interviews will help to inform the potential use of these items in the field test and full-scale study. Table 1, below, provides a distribution of the desired minimum number of respondents by type, although a respondent may fit more than one category.

Table 1: Desired minimum number of respondents, by respondent type

Respondent type	Desired minimum number of respondents
Never enrolled in postsecondary education	5
Currently enrolled in postsecondary education Enrolled in a 2-year program Enrolled in a 4-year program	5 5
Working while enrolled	5
Working but not enrolled	5
Left high school without diploma or has GED	5
Identifies as LGBTQ	10

Trained RSS staff will conduct the cognitive interviews in rented conference rooms, with the goal of completing an average of five cases per day. The cognitive testing will involve intensive one-on-one interviews. Each interview will last a maximum of 60 minutes, and all participants will be offered an incentive of $40 for their participation. Audio recording of each interview will be available to NCES for review.

The organizing objective of the cognitive testing approach will be to identify the processes by which respondents answer draft survey questions and to pinpoint potential sources of question misinterpretation in their responses. For example, respondents will be asked to “think aloud” as they answer questions. Concurrent and retrospective protocols can provide a valuable source of evidence about the organization of information in memory, comprehension of the questions, strategies used in retrieving information, judgment processes that come into play, and other processes affecting the final answers to survey items. To elicit relevant responses, respondents may be asked to point out unfamiliar terms, to paraphrase the question or its accompanying instructions, to define a term, and to make judgments regarding the confidence they place in their answers. Typical probes – e.g., “How certain are you of your answer?” or “How easy or difficult was it to answer this question?” – seek to verify respondent interpretations, investigate the meaning of specific potentially ambiguous phrases, or elicit notions that the respondent thought were critically relevant to but absent from the question.

Immediately following the conclusion of each interview, methodologists will review the interview recordings and notes, and highlight potential themes that may have arisen. Following each interview, the digital audio recording will be archived for qualitative analysis. RSS will organize their observations and summarize the common themes, insights, and ideas emerging from each of the interviews into a report.

Attachment I provides additional detail about Recruitment Procedures. Attachment II presents the Screening Questions that will be used to determine Eligibility for cognitive lab participation. Attachment III presents the Sample Consent Forms.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and that their responses may be used only to help inform the survey design and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573). Participants will be assigned a unique student identifier (ID) that will be created solely for file management and used to keep all materials together. The participant ID will not be linked to the student’s name in any way. Participants will sign an informed consent form, which will be kept separate from the interview files and notes in a locked cabinet in a secure room for the duration of the study, and which will be destroyed after the final report is submitted.

Schedule for HSLS:09 Second Follow-Up Cognitive Testing Activities

RSS will begin recruiting for the cognitive testing upon receiving OMB clearance. The testing is scheduled to begin in November 2014. This timeline allows the project team to use the results from the qualitative research to improve the survey questions administered in the spring 2015 field test, and seek their clearance beforehand.

Estimate of Respondent Burden

To yield 30 completed interviews, we may need to interview 120 individuals to screen them for eligibility and to ensure that we are achieving the target goals in terms of respondent types. The screening process, on average, is estimated to take about 5 minutes per person.

Table 2: Estimate respondent burden

Activity	Number of respondents	Number of responses	Hours per respondent	Maximum total burden hours
Screening	120	120	0.083	10
Cognitive interview	30	30	1.0	30
Study Total*	120	150		40

* The total for this study is shown for information purposes only. Respondent burden for the cognitive interview portion of the study will be requested in late October 2014, in the subsequent request for approval of the cognitive interviews protocol and questions to be tested.

Estimate of Costs for Recruiting and Paying Respondents

In order to encourage participation by recent high school students in this study, and to thank them for their time and effort spent traveling to the interview facility and partaking in the interview, each respondent will be offered $40 for their participation.

Estimate of Cost Burden

Participants in the cognitive interviews must bear direct costs associated with travel by car or use of Chicago Transit Authority services to RSS’s rented conference rooms in Chicago.

Cost to Federal Government

The cost to federal governent for conducting these cognitive interviews will be $58,680, under the RSS subcontract to RTI, including recruitment, interviewing, analysis, report writing, and participant incentives.

2