1850-0882 NLTS 2012 Phase II_Supporting Statement PartB_revised 3-9-2016

Phase II of the National Longitudinal Transition Study 2012 (NLTS 2012)

Supporting Statement Part B

OMB #

Submitted by

National Center for Education Evaluation and Regional Assistance

U.S. Department of Education

December 2015

CONTENTS

INTRODUCTION 3

B. Collection Of Information Employing Statistical Methods 4

1. Respondent Universe And Sampling Methods 4

2. Statistical Methods For Sample Selection And Degree Of Accuracy Needed 4

3. Methods To Maximize Response Rates And Deal With Nonresponse 6

4. Tests Of Procedures And Methods To Be Undertaken 7

5. Individuals Consulted On Statistical Aspects Of The Design 9

TABLES

Table B.1. NLTS 2012 Phase II Sample by Disability Category 4

Table B.2. Current Consent Status for Phase II Transcript and SSA Records Collection 5

Table B.3: Increased consent experiment groups, treatments, and contact timing. 8

Table B.4: NLTS 2012 Phase I Technical Working Group 9

INTRODUCTION

The U.S. Department of Education (ED) is requesting Office of Management and Budget (OMB) approval for Phase II of the National Longitudinal Transition Study 2012 (NLTS 2012), which entails collection of high school and post-high school administrative data. These Phase II data will be linked to survey information from Phase I¹ and, together, constitute a key component of the Congressionally-mandated National Assessment of the Individuals with Disabilities Act (IDEA 2004).

NLTS 2012 is the third in a series of studies conducted by the U.S. Department of Education (ED), with the goal of describing the characteristics, secondary school experiences, transition, and outcomes of youth who receive special education services under IDEA. The current study is unique in that it includes representative samples not only of youth who had an individualized education plan (IEP) at the time of their participation in NLTS 2012, but also of youth who do not have an IEP but who have a condition that qualifies them for accommodation under Section 504 of the Vocational Rehabilitation Act of 1973, and of youth who have no identified disability.

The NLTS 2012 sample includes 21,959 students ranging in age from 13 to 21 in December 2011. The sample was selected to include sufficient students in each of the 12 federally defined disability categories, and adequate number of students without disabilities, including both students with a Section 504 plan and students with neither an IEP nor a Section 504 plan.

To meet the study’s objective, Phase II administrative data will be collected from the following sources: (1) school district records, including transcripts; (2) postsecondary enrollment information through the National Student Clearinghouse (NSC), (3) student financial aid data from ED’s Federal Student Aid Office (FSA), (4) employment, earnings, and benefits data from the Social Security Administration (SSA), either directly from SSA or from the Census Bureau through a joint agreement between the three agencies; and (5) information about vocational rehabilitative services and supports from the Rehabilitative Services Administration (RSA).

These data sources will yield information in three broad areas important to understanding outcomes for youth with disabilities: (1) high school course-taking and outcomes, (2) post-secondary education and training outcomes, and (3) employment and earnings outcomes after high school. This information will be used to address three research questions:

• To what extent do youth with disabilities who receive special education services under IDEA make progress through high school compared with other youth, including those identified for services under Section 504 of the Rehabilitation Act? For students with disabilities, has high school course taking and completion rates changed over the past few decades?

• Are youth with disabilities achieving the post-high school outcomes envisioned by IDEA, and how do their college, training, and employment rates compare with those of other youth?

• How do these high school and postsecondary experiences and outcomes vary by student characteristics, including their disability category, age, sex, race/ethnicity, English Learner status, income status, and type of high school attended (including regular public school, charter school, career/technical school, special education school, or other State or Federally-operated institution)?

Two important types of products will result from NLTS 2012 Phase II, once the data are assembled. There will be a series of descriptive reports examining the trajectories of students. In addition, there will be a restricted use file that meets federal privacy and confidentiality standards but that can be used by external researchers to answer questions or explore topics beyond those planned by NLTS 2012.

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

NLTS 2012 Phase II data collection will include different samples of students originally selected as part of Phase I of the study. The NLTS 2012 sampling plan was approved by OMB through previous Information Collection Requests (#1850-0882, 5/25/11; #1850-0882, 2/15/12; #1850-0882, NOC 7/31/12). Under the approved sampling plan, a total of 21,959 youth cases were released to achieve a nationally representative sample of students 13 to 21 years old (in December 2011) and enrolled in school district grades 7 through 12. Of the total released cases, 17,476 are youth with an IEP and 4,483 are youth without an IEP, including 1,168 youth with a Section 504 plan and 3,315 with neither an IEP nor a Section 504 plan (see table B.1).

Table B.1. NLTS 2012 Phase II Sample by Disability Category

	Total NLTS2012 Phase I sample
Sample member characteristics	Number	Percent
Total	21,959	100.0
Primary disability category
Students with IEPs	17,476	79.6
504 plan but no IEP	1,168	5.3
Neither 504 plan nor IEP	3,315	15.1

^a Includes youth who were reported as having an IEP but whose disability category was not indicated.

2. Statistical Methods for Sample Selection and Degree of Accuracy Needed

No further statistical methods will be used to determine the sample for Phase II of NLTS 2012; Phase II is a longitudinal follow-up of the sample selected for Phase I. The Phase I NLTS 2012 sampling design² placed a high priority on obtaining a sufficient number of respondents to achieve precise estimates for students with an IEP and for students in each of the federally defined disability categories.³ Other priorities were to obtain reasonably precise estimates for Section 504 students and for students with no IEP and no Section 504 plan.

a. Phase II Administrative Data Sample

Phase II will draw on the NLTS 2012 Phase I sample of 21,959 students who were selected to achieve a nationally representative sample of students 13 to 21 years old (in December 2011) and enrolled in school district grades 7 through 12. The sample sizes for transcript and student record collection, SSA record matching, and other administrative record matching will vary based on sample members’ consent status. We will attempt to increase the number of explicit consents through a new effort under Phase II (as described in Part A, section 2a and in Part B, below). This effort will also provide additional information on the last school(s) attended by sample members, which will aid in collection of transcripts.

Social Security Administration (SSA) data. Social Security Administration data (obtained either through SSA directly or by way of the Census Bureau) will be collected for the cases that provided explicit consent to collect those administrative data; this will include 3,499 cases who previously provided consent, plus additional cases who provide consent as part of Phase II increased consent procedures.

Transcripts and other administrative data. The transcripts, district records, and other educational records (FSA and NSC data) to be collected for NLTS 2012 Phase II are either internal to ED or are otherwise allowed under FERPA; therefore, explicit consent will not be required to collect these records, and they will be collected for all cases except those who explicitly refused consent. Currently there are only 2,312 such cases who have explicitly refused consent for transcript collection. Phase II will collect transcript, district, and other administrative data for the remaining 19,647 cases, plus any additional cases who provide consent during Phase II increased consent procedures. In round 1, school district records, including transcripts, will be collected from 430 school districts while other educational records will be collected directly from NSC and FSA. An optional second round collection to be conducted in 2018 would return to these districts, as well as additional schools attended by sample members (as identified from increased consent activities and district records). Table B2 below lists the Phase II sample by consent status; bold typeface indicates the sample members who will be included in the increased consent procedures.

Table B.2. Current Consent Status for Phase II Transcript and SSA Records Collection

				Consent status for SSA records
	Total sample			Explicit Consent			Explicit Refusal			No response (NR)
	Number	Percent of total		Number	Percent of total		Number	Percent of total		Number	Percent of total
Total sample	21,959	100.0		3,499	15.9		2,909	13.2		15,551	70.8
Transcript Explicit Refusal	2,312	10.5		237	1.1		1,555	7.1		520	2.4

b. Precision and Minimum Detectable Differences

An example of a key comparison of interest for the first follow-up will be the difference in the percentage of students who are enrolled in either secondary or postsecondary education between students with an IEP at baseline and students without an IEP at baseline. We anticipate that there will be approximately 8,910 youth completes at the first follow-up, of whom 80 percent will have had an IEP at baseline. This sample is expected to achieve a 95 percent confidence interval (around a point estimate of 0.80 for a proportion) of less than 0.015 for students with an IEP and 0.023 for students without an IEP, and a minimum detectible difference of 0.04 for a test of size alpha equals 5 percent and 80 percent power assuming a design effect from unequal weights and clustering of 2.5 for students with an IEP and of 1.5 for students without an IEP.

3. Methods to Maximize Response Rates and Deal with Nonresponse

District Participation

Phase II of NLTS 2012 will request school records and transcripts for all students sampled for the NLTS 2012 Phase I study except those who have explicitly refused consent. Since the success of this data collection is closely tied to active participation of school districts in which these students were enrolled, the consent and cooperation of school officials is essential. Procedures for working with these school districts will build upon the rapport developed during the NLTS 2012 Phase I. All school districts will have participated in the Phase I data collection and should recognize this study’s importance and the relationship of this request to their involvement with NLTS 2012. Approximately 90 percent of districts are expected to participate, and we anticipate collecting transcripts for over 90 percent of the student sample. These estimates are based on a transcript collection with a high school cohort for the High School Longitudinal Study of 2009, which had a school participation rate of over 96 percent, and collected transcripts for over 93 percent of the student sample. As much information as is available about prior contacts and data requests from the school districts will be acquired from IES project officers and NLTS 2012 Phase I contractors prior to communicating with the school districts.

At the start of data collection, each sampled district will receive an invitation packet. Follow-up calls will be made to ensure receipt of the request packet and to answer any questions about the study. The packet will contain clear, concise materials describing the purpose of the study, the type of data requested, and how to provide data. Districts will receive guidance on how to access, download, and upload files from a secure internet site. The site will include templates that districts can use to provide the needed information. File templates will contain the student’s name along with fields for each data item. Many districts and schools will be able to download these templates and create files from their current databases, which can then be uploaded to the secure internet site. Alternatively, districts can enter data directly into the template and upload their completed templates. A small number of districts are expected to have only hard copy records for requested data. In these cases, districts can either enter the data onto the template directly, scan and upload hardcopy documents, or send hardcopies via overnight delivery. District data collection materials are included in Attachment C.

Experienced staff will be assigned responsibility for a set of schools throughout the data collection process, which will help to maintain rapport with school staff and provide a reliable point of contact if districts have questions or need assistance. The staff will be thoroughly trained on the purposes and requirements of the study and on collecting school records.

Privacy and consent concerns may arise in the collection of school records data and staff assigned to a schools expressing concern will be familiar with the Family Educational Rights and Privacy Act of 1974 (FERPA), which permits schools to release student data to the U.S. Department of Education and its authorized agents without consent, and will be prepared to respond to concerns raised by high school staff. In compliance with FERPA, a notation will be made in the student record that all school record data and transcript data have been collected for use in NLTS 2012 Phase II.

To address concerns about nonresponse bias, analyses will be conducted to determine whether those who explicitly refused consent for other administrative data are different from the rest of the sample. In the case that nonresponse bias analysis results indicate potential bias, sample weights will be adjusted accordingly.

Consent from Students and Parents

Gaining sample member consent will be critical to the success of the SSA data collection. Prior to making contact, we will conduct batch database searches to ensure that we have the most up-to-date contact information for sample members and their parents. Mail and e-mail materials sent to sample members and parents will explain the purpose of the study, ask for their consent, and emphasize that, while voluntary, their participation is important to its success. Consent contacting materials are included in Attachment D.

For sample members or parents who do not respond to the mail or e-mail requests, we will follow-up with telephone calls to sample members or their parents. Outbound prompting calls will be managed through the Case Management System (CMS), which is equipped to automatically and efficiently schedule cases to be dialed, and maintain a complete record of locating information and contacting history for each case. For sample members or parents without current contact information, we will conduct intensive tracing. During interactive tracing, skilled tracers utilize all previously obtained contact information to make informed decisions about each case. These intensive interactive searches are completed using a special program that works with RTI’s CMS to provide organization and efficiency in the intensive tracing process. Sources that may be used, as appropriate, include credit database searches, such as Experian, various public websites, and other integrated database services.

4. Tests of Procedures and Methods to be Undertaken

District administrative records collection will draw heavily on extensively used procedures, including many from the HSLS:09 and ELS:2002. A formal pretest of the district record collection procedures is not planned; however, data collection will begin with a phased rollout of sampled districts. This method will allow us to assess how well the districts understand the data request, the challenges they face in compiling the requested data, and the efficiency of the various submission options. If problems are discovered, we will make adjustments to the secure upload site or instructions provided to institutions before beginning data collection on the full district sample.

NLTS 2012 Phase II will include an experiment to assess the effectiveness of an approach to increase the number of NLTS 2012 sample members for whom consent is granted for the collection of SSA data.  The approach is to send letters and emails to sample members’ households, asking that they provide consent for the collection of SSA data.  Methods for providing consent will include returning a self-addressed pre-paid mailer, indicating consent via a secure website, or calling a toll-free phone number.  An incentive to increase response will also be tested. 

The experiment will be performed with a random sample of 600 students. These students will be divided into three groups of 200 students each. All groups will be contacted via letter and email and asked to provide consent for collection of SSA data, and one group (treatment group 1) will be offered a $10 incentive with this initial contact. Approximately two weeks later, members of all groups who have not provided consent will be prompted to provide consent via a second letter and email. At this second contact, treatment group 2 will be offered the $10 incentive. Approximately two weeks later (approximately four weeks after the initial contact) all groups will be contacted again and prompted to provide consent. Upon the third contact, the treatment groups will be divided into halves of 100 students each, with one half receiving telephone prompting to request consent. Table B.3 provides an overview of the experiment structure, including timing and treatments.

Table B.3: Increased consent experiment groups, treatments, and contact timing.

Contact timing	Group 1 (n=200)		Group 2 (n=200)		Group (n=200)
Contact 1 Initial letter and email requesting consent	Includes offer of $10 incentive for response providing consent		No incentive offered		No incentive offered
Contact 2 Follow-up mailing and email Approximately two weeks after initial contact	Includes reminder of $10 incentive		Includes offer of $10 incentive		No incentive offered
Contact 3 Follow-up mailing and email Telephone prompting (for some cases) Approximately four weeks after initial contact	Includes reminder of $10 incentive		Includes reminder of $10 incentive		No incentive offered
	Group 1A Telephone prompting	Group 1B No telephone prompting	Group 2A Telephone prompting	Group 2B No telephone prompting	Group 3A Telephone prompting	Group 3B (Control) No telephone prompting

Any increases in the numbers of sample members with consent to collect SSA data will be assessed against the costs of requesting consent. Results will be tested for statistical significance of the difference between the treatment and control groups, comparing outcomes for those offered incentives as well as those receiving telephone prompting calls. Minimum detectable effect will also be calculated for comparisons between groups. If an assessment of any increases in consent, and comparison of the various treatments, indicates adequate value, an approach will be tailored for the remaining sample.

5. Individuals Consulted on Statistical Aspects of the Design

Phase II of NLTS 2012 will be carried out by RTI International (RTI), with subcontractors SRI International and Social Dynamics. ED will seek professional counsel on the statistical aspects of the design from key contractor study staff and the project’s Technical Working Group (TWG). The project’s TWG, to be identified after the study is underway, will include up to eight individuals with expertise in secondary and postsecondary education for youth with disabilities, knowledge of and experience using relevant administrative records, and expertise in research methodology and analytic strategies for descriptive studies.

During Phase 1 of NLTS2012, ED consulted with six researchers, one school district administrator, and one former school district administrator who made up the project’s Technical Working Group. The group met several times during the course of Phase 1. The members of the Phase I Technical Working Group are listed below in table B.4.

Table B.4: NLTS 2012 Phase I Technical Working Group

Brian Cobb, Ph.D. Interim Associate Director/Professor Emeritus College of Applied Human Sciences School of Education, Room 205 Colorado State University Fort Collins, CO 80523-1588 [email protected] 970-491-6835	Thomas Bailey, Ph.D. Professor of Economics and Education, and Director, Community College Research Center Teachers College, Columbia University 525 West 120th Street 400 Thorndike Hall, Box 174 New York, NY 10027 [email protected] 212-678-3091
Richard Luecking, Ed.D. President TransCen, Inc. 401 N. Washington St, Suite 450 Rockville, MD 20850 [email protected] 301-424-2002 x 230	Suzanne Lane, Ph.D. Professor, School of Education University of Pittsburgh 5916 Wesley W. Posvar Hall Pittsburgh, PA 15260 [email protected] 412-648-7095
Barbara M. Altman, Ph.D. Disability Statistics Consultant 14608 Melinda Lane Rockville, MD  20853 [email protected] 301-460-5963	Judy Elliott, Ph.D. EduLead, LLC 4925 Londonderry Drive Tampa, FL 33647 [email protected] 503-734-0306
Kalman Rupp, Ph.D. Senior Economist Division of Policy Evaluation Office of Research, Evaluation, and Statistics Social Security Administration [email protected] 202-358-6216	Markay Winston, Ph.D. Chief Officer of the Office of Special Education and Supports Chicago Public Schools 125 South Clark St, 8th fl Chicago, IL 60603 [email protected] 773-553-1804

1 Approval for NLTS 2012 Phase I data collection (#1850-0882) was obtained on January 31, 2012 and updated on July 31, 2012.

2 The NLTS 2012 sampling plan was approved by OMB through previous Information Collection Requests (#1850-0882, 5/25/11; #1850-0882, 2/15/12; #1850-0882, NOC 7/31/12).

3 Not included among the federally defined disability categories was the category “developmental delay”, which is used only for young children. Students served in the “developmental delay” category who continue to require special education services at ages 13 to 21 years would have been reclassified in one of the 12 disability categories included in the study.