Att_Supporting Statement Part B NLTS Baseline

Supporting Statement for Paperwork Reduction Act Submission National Longitudinal Transition Study 2012 Baseline Data Collection

Part B

October 11, 2011

Contract Number: ED-IES-10-C-0073 Mathematica Reference Number: 06876.381 Submitted to: Institute of Education Sciences U.S. Department of Education 555 New Jersey Ave., NW, Suite 502K Washington, DC 20208 Project Officer: Amanda DeGraff, Ph.D. Submitted by: Mathematica Policy Research P.O. Box 2393 Princeton, NJ 08543-2393 Telephone: (609) 799-3535 Facsimile: (609) 799-0005 Project Director: John Burghardt

Supporting Statement for Paperwork Reduction Act Submission National Longitudinal Transition Study (NLTS) 2012 Part B October 11, 2011

CONTENTS

INTRODUCTION 1

RESEARCH QUESTIONS 1

OVERVIEW 2

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS 4

1. Respondent Universe and Sampling Methods 4

2. Statistical Methods for Sample Selection and Degree of Accuracy Needed 5

3. Methods to Maximize Response Rates and Deal with Nonresponse 11

4. Tests of Procedures and Methods to Be Undertaken 14

5. Individuals Consulted on Statistical Aspects of the Design 15

APPENDICES

Appendix A	Individuals with Disabilities Education Improvement Act, Section 664(e)
Appendix B	Individuals with Disabilities Education Improvement Act, Section 664(a)
Appendix C	Federal Register Notice
Appendix D	Advance Mailing (Letter and Brochure) for Parent, Math/Language Arts Teacher, Principal, and School Program Respondent
Appendix E	Consent form from Districts requiring active consent of parents
Appendix F	Thank You Letters for Parent, Student, Math/Language Arts Teacher, Principal, and School Program Respondent
Appendix G	Documentation of Consent/Assent for Parent and Student
Appendix H	Mathematica Confidentiality Pledge
Appendix I	School Characteristics Survey Instrument (Principals)
Appendix J	Math or Language Arts Teacher Survey Instrument
Appendix K	School Program Survey Instrument
Appendix L	Baseline Parent Survey Instrument
Appendix L	Baseline Student Survey Instrument
Appendix M	Variables Measuring Student Characteristics, School Resources and Student Experiences, and Perceived Barriers

TABLES

B.1. NLTS 2010 Phase I Data Collection Plan 3

B.2. Population Sizes for Target Populations and Subpopulations Defined by Disability 4

B.3. Allocation of NLTS 2012 Sample to Small, Medium, and Large District Units 7

B.4. Sample Sizes, Precision, and Minimum Detectable Differences for Subpopulations Defined by Disability Category 10

B.5. Precision and Minimum Detectable Differences at Various Data Collection
Points 11

INTRODUCTION

The U.S. Department of Education (ED) is requesting Office of Management and Budget (OMB) approval for baseline data collection as part of the National Longitudinal Transition Study (NLTS) 2012, Phase I. NLTS 2012 is a longitudinal study focused on the educational experiences and transition from school of youth with disabilities between the ages of 13 and 21.

The main objectives of the study are to describe the background, secondary school, transition, postsecondary experiences, and outcomes of youth who currently have an individualized education plan (IEP) (and therefore receive special education services under the Individuals with Disabilities Education Act (IDEA)). The study will compare this group with three other groups: (1) youth who have no identified disability, (2) youth who do not have an IEP but who have a condition that qualifies them for accommodation under Section 504 of the Vocational Rehabilitation Act of 1973 and (3) similar cohorts of youth with an IEP who were studied in the past.

RESEARCH QUESTIONS

The study will obtain information on three broad areas important for understanding the experiences of transition-age youth: (1) the characteristics of youth and their families; (2) the experiences of youth in high school (including their academic program and the services they receive to support acquisition of academic proficiencies as well as transition); and (3) youth outcomes (high school completion status, access to postsecondary education and employment, persistence in postsecondary education and employment, independent living and integration into the community, and access to and use of services to support positive outcomes). NLTS 2012 will address the following research questions under the three broad objectives:

Describe Transition-Age Students with an IEP

1. What are the personal, family, and school characteristics of this group?

2. What are their courses of study, services and accommodations to support learning, and preparation for transition? What barriers and challenges do they encounter?

3. What are the key academic, social, and economic outcomes in school and after leaving school for youth with disabilities?

4. How do services, courses of study, barriers, and outcomes vary for subgroups defined by the nature of the youth’s disability, age, sex, race/ethnicity or characteristics of the student’s school or community?

5. How do academic, social, and economic outcomes for youth with disabilities vary by their course of study and receipt of services and accommodations, accounting for preexisting youth characteristics?

Compare Current Transition-Age Students with an IEP to Their Peers in Prior Cohorts

6. How does receipt of services and accommodations and youth outcomes of the current cohort of special education students differ from those of previous cohorts of special education students?

Compare Transition-Age Students with an IEP to Their Peers Who Do Not Have an IEP

7. What are the characteristics, school and transition experiences, and postsecondary outcomes of youth with plans that provide accommodations under Section 504 of the Rehabilitation Act of 1973?

8. How do characteristics, courses of study, receipt of services and accommodations, and key outcomes for transition-age youth with an IEP differ from students with Section 504 plans and from students with no Section 504 Plan and no IEP?

In NLTS 2012 Phase I, a sample of school districts and nationally representative sample of students will be selected and recruited for the study, and baseline data collection and first follow-up data collection will be completed. ED has not finalized the plan for Phase II. This Supporting Statement requests OMB clearance for securing consent and assent of students and their parents for participation in the study and conducting the baseline data collection. The next section provides an overview of Phase I of NLTS 2012.

OVERVIEW

The study will provide policymakers and educators with critical information that is not available from other sources. The study will provide up-to-date information on the barriers and challenges youth with disabilities encounter during and after high school; the services and support they receive to help them overcome these barriers from their families, community service providers, secondary and postsecondary schools, and employers; and the extent to which youth make a successful transition to postsecondary education, employment, and independent living. The study will examine these issues from multiple perspectives including those of school staff, parents, and the youth themselves. By comparing the experiences of a current cohort to those of previous cohorts, the study will be able to describe changes in the composition of students with disabilities over time as well as changes in their school experiences and outcomes.

A nationally representative sample of 15,000 students who are between the ages of 13 and 21 in December 2011 and enrolled in public school districts with grades 7–12 will be selected and recruited in two stages. The study team will first select and recruit a nationally representative sample of approximately 500 local education agencies (school districts, charter schools, and special schools). Using student lists provided by participating districts, the team will sample and recruit students. The student sample is designed to provide precision for describing all IEP students, and all students with no IEP, as well as for important subgroups including each of the 12 IDEA disability categories in which transition-age youth are served and students who have a Section 504 plan but do not have an IEP.

The first wave of data collection will begin in January 2012 and the second in January 2014, when sample members will be between 13 and 21 years old and 15 and 23 years old, respectively. Table B.1 summarizes the Phase I data collection design.

Table B.1. NLTS 2010 Phase I Data Collection Plan

Respondent	Mode	Timeline	Key Data
Parents	Telephone survey	Spring 2012	Characteristics of youth and educational expectations for child, involvement in transition planning
Students	Telephone survey (all) Academic assessment (16 and older)	Spring 2012	Experiences and perceptions of school career and educational expectations, engagement in school, community, self-determination and work Assess academic skills
Student’s School Principal	Web survey with telephone follow-up	Spring 2012	Policies, programs, staffing, and resources at student’s school
Student’s Math or Language Arts Teacher	Web survey with telephone follow-up	Spring 2012	Class and teacher characteristics, instructional practices, services supports and accommodations, student engagement
Student’s Special Education Teacher	Web survey with telephone follow-up (students with IEP)	Spring 2012	School program and supports for students with IEP
Student School Information	School records	Winter 2012	Student characteristics
Parent	Telephone survey	Spring 2014	Youth’s experiences in school and postsecondary
Student	Telephone survey (all) Academic assessment (16 and older)	Spring 2014 Spring 2014(if not done in Spring 2012)	Youth’s experiences in school and postsecondary Academic skills
Student’s Special Education Teacher	Telephone survey	Spring 2014	School program and supports for students with IEP
Student School Information	School records	Spring 2014	Attendance, transcripts

The study design and data collections for NLTS 2012 Phase I are similar to prior longitudinal studies of students with disabilities in order to address the third broad objective of comparing the characteristics, experiences, and outcomes of students with an IEP over time. However, ED seeks to improve on these prior studies in three important ways: (1) by using innovative methods of securing parental consent for youth participation to increase rates of sample participation, (2) by including students with no IEP (both students who have a condition that qualifies them for a Section 504 plan and students with no identified disability), and (3) by seeking more information on student barriers and activities that support transition.

As noted, this Supporting Statement requests OMB clearance to obtain parental consent and student assent to participate in the study, and to collect baseline data. OMB has approved a prior request to select and recruit districts and acquire lists of students for selecting the student sample (OMB 1850-0882). A future submission will request clearance for conducting first follow-up data collection.

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

This study is designed to collect information about youth between the ages 13 and 21 (as of December 2011) who are in schools serving grades 7 to 12 or who are in an ungraded school and in this age range. Within this target population there are three key groups of interest: (1) students identified as needing special education services—that is, those with IEPs; (2) students who have not been identified as needing special education services but who have a condition that qualifies them for accommodations under Section 504 of the Vocational Rehabilitation Act of 1973; and (3) students with no IEP and no Section 504 plan. Based on counts for the 2008–2009 school year, approximately 22,500,000 students are in the appropriate age and grade range for this study in the 50 states and the District of Columbia, and approximately 300,000 are in schools run by the Department of Defense and the Bureau of Indian Affairs or in the territories (Table B.2).¹ Of these, approximately 2,800,000 students have IEPs. Of the students who do not have IEPs, an estimated two percent (approximately 450,000 students) have Section 504 plans.²

Table B.2. Population Sizes for Target Populations and Subpopulations Defined by Disability

	Estimated Population Count
All students ages 13 to 21	22,500,000
All students without IEPs	19,720,000
Without Section 504 plans	19,270,000
With Section 504 plans	450,000
All students with IEPs	2,780,000
Specific learning disabilities	1,508,000
Other health impairments a	901,250
Speech or language impairments	113,200
Autism	95,000
Multiple disabilities	73,200
Hearing impairments	34,000
Orthopedic impairments	27,500
Traumatic brain injury	13,900
Visual impairments	12,200
Deaf-blindness	750

^aEstimates for this row also apply to the categories intellectual disability and emotional disturbance.

2. Statistical Methods for Sample Selection and Degree of Accuracy Needed

Two-stage sampling will be used to select approximately 15,000 youth ages 13 to 21 as of December 2011. Of these youth, approximately 12,000, or 80 percent, are expected to respond. The respondents will include approximately 9,600 students with IEPs and 2,400 students without IEPs. Of the 2,400 students without IEPs, approximately 600 will be students with Section 504 plans and 1,800 will be students with no IEP and no Section 504 plan.

The sampling design balances several objectives but places the highest priority on obtaining precise overall estimates for all students with IEPs and precise estimates for each of the federally defined disability categories. Other priorities are to obtain estimates for the Section 504 students and students with no IEP and no Section 504 plan.

The sampling design for this study was developed to support survey estimates with the precision needed for policy analysis for the 12 of 13 categories of students with disabilities specified in the Individuals with Disabilities Education Act in which transition-age students are served³. Among these disability categories, the prevalence of the disability varies substantially, with some disability categories being more prevalent (such as students with learning disabilities and students with intellectual disabilities) than others (such as students who are either deaf, blind, or both). For most of the students with the more prevalent disabilities, the school district is an efficient vehicle for identifying and selecting a sample of students, and we will use a two-stage sampling design for selecting these students. For students who are deaf and/or blind, a major portion of these students will be educated in state-sponsored schools for the deaf and/or blind. We will use these schools as a primary source of students in this disability category and supplement this sample with those selected through the district-based sample.

The primary sample will be selected in two stages. In the first stage, the study team formed primary sampling units of one or more districts (including charter schools) and randomly selected a sample of 600 district units using the Common Core of Data (CCD) from the National Center for Education Statistics, with a target of 300 district units participating in the study. We will initially recruit a randomly selected subsample of 300 district units, and the remaining district units will be a reserve sample. Additional district units will be randomly selected from this reserve sample in case a district unit (or part of the district unit) refuses to participate.

For the second stage, the study team will obtain lists of students with IEPs, students with a Section 504 plan, and students with neither an IEP nor a Section 504 plan from the approximately 500 participating districts (including charter schools). These lists will include all students whose education is provided by the sampled district, including students with an IEP whose district determines that the appropriate placement for the student is a private school. We will then allocate the sample among these strata of students in the disability categories, students with a Section 504 plan, and other students to select the student samples. Details of the proposed sample selection are described below.

a. District Sampling Frame

The sampling frame for the districts in the study comes from the CCD. Approximately 14,200 local education agencies (LEAs) nationwide serve students with IEPs in grades 7–12 or between the ages of 13 and 21. To achieve sufficient samples among the least prevalent disability categories, we estimate that the primary district-level sampling unit for the study will need to serve at least 375 students with IEPs. This number of students with an IEP for a primary sampling unit is necessary to ensure that the sample includes adequate numbers of students with low-incidence categories of disability to support descriptions of these key groups. This estimate includes an inflation adjustment to account for missing or suspect IEP counts in the CCD data file and loss of sample due to nonresponse.

Of the 14,200 LEAs in the nation, 5,140 districts serve fewer than 30 students with an IEP, and, in aggregate, serve less than 3 percent of all students with IEPs. Because at least 375 students with IEPs are needed in districts units, districts with fewer than 30 students with IEPs were excluded from the sampling frame for constructing the district units. The exclusion of these smaller districts will reduce the number of districts that will need to be recruited and better focus the resources for this study.

To support the data collection, the study team combined nearby districts into district units for sampling purposes. Specifically, districts with more than 30 and fewer than 375 students with IEPs were combined so that they contained at least 375 students with IEPs. Larger districts do not need to be combined but can serve as their own district unit.⁴ For the selection of students in district units consisting of multiple districts, the study team will obtain student lists from all component LEAs and sample from the combined student populations. Within district size strata, the sample of district units (and districts) was selected with probability proportional to a composite size measure that includes the total IEP and non-IEP population in the district unit. This measure increases the probability of selecting districts with more students with an IEP and can be used to provide nearly equal weighted samples of students within the federal disability categories in each district size stratum.

b. Stratification of the District Sample

The study team stratified the district units before sample selection. The primary explicit stratification of the district sample was by size of district. Approximately, 62 percent of students with IEPs attend school in districts with 375 or more students with IEPs (large districts); 16 percent attend districts with 200–375 students (medium districts), and 22 percent of students with IEPs attend districts with between 30 and 200 students (small districts). To keep the total number of districts to be recruited and the costs of data collection at reasonable levels, students attending the smallest districts were sampled at 50 percent of their proportion of the total population, and students in the large districts were sampled at about 118 percent of their proportion of the total population (see Table B.3). In addition, schools serving deaf and blind students will form a separate stratum. In Table B.3, we show the distribution of the students with IEPs across the three district size strata, an allocation of the sample across these strata using strict proportionality, the number of districts units to be selected in each stratum, and the estimated number of individual public school district and charter schools that will be recruited. By reducing the small district sample allocation from 2,117 to approximately 1,059 (a 50 percent reduction), we can compensate for this undersampling by increasing the sample allocation by only 18 percent for the larger districts. This will reduce the burden on the smaller districts by nearly 50 percent.

The team will use implicit stratification on geographic region and degree of urbanicity to ensure that the sample reflects the nationwide distribution of students along these dimensions. In implicit stratification, the sampling frame within a stratum is ordered by a factor such as region of the country, and by using a sequential selection procedure, the sample selected is approximately proportionally allocated across the regions of the country.

Table B.3. Allocation of NLTS 2012 Sample to Small, Medium, and Large District Units

	Proportional Sample Allocation		Revised Allocation
Stratum Definition (Projected Students With IEPs in District Unit)	Percentage of Students with IEPs	Number of Completed Interviews Among Students with IEPs	District Unitsa	Expected Number of Districts Recruited	Number of Completed Interviews Among Students with IEPs	Percentage of Sample	Sampling Rate	Students with IEPs/ District
Total	100%	9,600	300	521	9,600	100%
Large (375 or more)	61.9%	5,941	185	195	7,000	72.9%	1.18	38
Medium (200 to 374)	16.1%	1,542	48	125	1,541	16.1%	1.00	32
Small (30 to 200)	22.1%	2,117	67	201	1,059	11.0%	0.50	16

^aA district unit is a grouping of one or more neighboring individual public school districts or charter schools combined to ensure an adequate number of students with IEPs are enrolled to support the data collection and identification of sufficient number for each disability category.

The team will use implicit stratification on geographic region and degree of urbanicity to ensure that the sample reflects the nationwide distribution of students along these dimensions. In implicit stratification, the sampling frame within a stratum is ordered by a factor such as region of the country, and by using a sequential selection procedure, the sample selected is approximately proportionally allocated across the regions of the country.

c. Size Measure for District Selection

The study team will use a composite size measure to select the sample of districts and district units within a given stratum.⁵ The composite size measure will be based on the district-level counts of the number of students with IEPs, N(students with IEPs in district i), and the number of students without an IEP, N(students without IEPs in district i). The size measure is based on global sampling rates for students with IEPs, f(IEP), and those without an IEP, f(W/O IEP), using data available from the CCD. The size measure for the ith district will be of the form

Si = f(IEP) * N(students with IEPs in district i) + f(W/O IEP) * N(students without IEPs in district i)

As we expected, some districts (such as Los Angeles, Chicago, and parts of New York City) with large student populations were selected with certainty, and the study team used this size measure to identify these districts. The remaining districts within each stratum will be selected with probability proportional to the composite size measure and without replacement. This composite size measure can result in nearly self-weighting samples of students within the disability categories in each size stratum.

To enable the undersampling of students in districts with 30–200 students, the study team created “half-units” in the small district stratum that will include half of the target 375 students with an IEP. In this way, students in these districts will be selected to the sample at a rate that is 50 percent of their incidence in the population, and these district units will each contribute half the number of sample members that medium districts contribute.

d. Student Selection

Based on the federal reporting requirements and on the experience of NLTS2, we anticipate that all districts will maintain lists of students by federal disability category. Based on information from the ED Office of Civil Rights, we anticipate that most districts will also maintain a list of non-IEP students with Section 504 plans. Using these lists, the study team will assign each student age 13 to 21 to one of the strata (one of the IEP disability categories, the stratum of non-IEP students with Section 504 plans, or the stratum of non-IEP students without Section 504 plans). The study team will then draw a random sample from each stratum (controlling implicitly by grade level and school) at a rate designed to yield the target number of students in each stratum. The team will also select a reserve sample available for use to account for students who may be ineligible or choose not to respond.

We anticipate that some districts will not have lists of students by disability category or Section 504 status.⁶ In these districts, the study team will first select schools and then obtain the lists from the selected schools. The schools will be selected with probability proportional to size (such as the number of non-IEP students).

The study team expects to interview approximately 32 IEP students/parents and 8 non-IEP students/parents in each district or district unit. To obtain this many respondents from each district, the study team will select samples of approximately 40 students with an IEP and 10 students without an IEP, based on an anticipated response rate of 80 percent.

e. Precision and Minimum Detectable Differences

Table B.4 presents target sample sizes and estimates of precision for a set of disability category subgroups and the non-IEP sample (divided into Section 504 students and all other students). All of the sample sizes in this table represent the estimated number of youth (or parents) responding to the surveys. This sample allocation is designed to allow meaningful precision for survey estimates and minimum detectable differences of approximately 0.10 for proportions near 0.50 (for a two-sided test with alpha of 0.05 and 80 percent power) for most of the disability categories. The precision estimates are based on an allocation of 600 respondents with Section 504 plans. The table presents estimates of minimum detectable differences (MDDs) for comparisons between the subpopulations and two larger populations: all students with IEPs and all students without IEPs.

Phase II of the project is expected to continue following the NLTS 2012 sample after 2014. However, the schedule for data collection beyond 2014 has not been set. To provide insight on the precision of estimates at points beyond baseline of the study, the accompanying Table B.5 presents precision estimates for the second data collection point in spring 2014 and for a not yet planned, and therefore hypothetical, fifth data collection point (fourth follow-up) in spring 2020. We have selected 2020 for purposes of this discussion because it creates an eight-year follow-up period that will support comparisons between students in the NLTS 2012 and NLTS2 samples who were 13–16 at baseline and 21–24 at the time of this fourth follow-up survey.

The first set of columns of Table B.5 shows the half width of 95 percent confidence intervals at selected follow-up data collection points for selected subgroups of the full NLTS 2012 sample. The second set of columns shows MDDs for each data collection point. The first column in that second set shows the MDD between the row subgroup and all IEP students; the next column to the right shows the MDD between the row subgroup and all students with no IEP. Response rate assumptions are that 80 percent of the sample provides data at baseline, that data collection occurs every two years, and that sample available after each round is 94 percent of the sample available at the previous round.

The estimates in Table B.5 suggest the study will be able to estimate attributes of all students with IEPs and all students without IEPs at about +/− 2 percentage points at both the beginning and end of the eight-year period. Furthermore, the sample will be able to detect differences in the attributes of IEP and non-IEP students of about 9 percentage points at two years after baseline in spring 2014 and about 10 percentage points at eight years after baseline in spring 2020.

Table B.4. Sample Sizes, Precision, and Minimum Detectable Differences for Subpopulations Defined by Disability Category

		Half-Width of 95% Confidence Level at Selected Proportions			Minimum Detectable Differences (MDDs)
	Proposed/ Estimated Sample Size	.50	.10		With IEPs	Without IEPs
All Students without IEPs	2,400	0.022	0.013		0.079	--
Without Section 504 Plans	1,800	0.025	0.015		0.087	0.095
With Section 504 Plans	600	0.043	0.026		0.131	0.137
All Students with IEPs	9,600	0.018	0.010		--	0.079
Specific Learning Disabilities	1,600	0.026	0.016		0.090	0.098
Other Health Impairments	1,200	0.030	0.018		0.100	0.107
Intellectual Disabilities	1,200	0.030	0.018		0.100	0.107
Emotional Disturbance	1,200	0.030	0.018		0.100	0.107
Speech or Language Impairments	1,000	0.033	0.020		0.107	0.114
Autism	1,000	0.033	0.020		0.107	0.114
Multiple Disabilities	900	0.035	0.021		0.111	0.118
Hearing Impairments	520	0.046	0.027		0.140	0.145
Orthopedic Impairments	450	0.049	0..030		0.149	0.154
Traumatic Brain Injury	230	0.069	0.041		0.202	0.206
Visual Impairments	200	0.073	0.044		0.215	0.219
Deaf-Blindness	100	0.104	0.062		0.300	0.303

Note: MDDs apply to comparisons between the row subpopulation and either all students with IEPs (excluding those students in the specific row subpopulation) or all students without IEPs. The MDDs are computed for detecting a difference in a proportion near 0.50 for a test with alpha of 0.05 and 80 percent power.

Table B.5. Precision and Minimum Detectable Differences at Various Data Collection Points

Measure	Half-Width of 95 Percent Confidence Intervalsa				Minimum Detectable Difference Between Column and Row Groupsb
Age of Sample at Follow-Up	Age 13-21	Age 15-23	Age 21-29		Age 13-21	Age 13-21	Age 15-23	Age 15-23	Age 21-29	Age 21-29
	Base	FU 1c	FU 4d		Base	Base	FU 1	FU 1	FU 4d	FU 4e
Proportion of Sample with Data	r =.8	r= .752	r= .625		r =.8	r =.8	r= .752	r= .752	r= .625	r= .625
					vs.IEP	vs. No IEP	vs.IEP	vs. No IEP	vs.IEP	vs. No IEP
All Students Without IEPs	0.022	0.025	0.028		0.079		0.091		0.100
With Section 504 Plan	0.043	0.049	0.054		0.131	0.137	0.151	0.158	0.166	0.173
Without Section 504 Plan	0.025	0.029	0.032		0.087	0.095	0.100	0.109	0.110	0.120
All Students with IEPs	0.017	0.020	0.022			0.079		0.091		0.100
Other health impairments	0.030	0.035	0.038		0.100	0.107	0.115	0.123	0.126	0.135
Autism	0.033	0.038	0.042		0.107	0.113	0.123	0.131	0.135	0.144
Orthopedic impairments	0.049	0.057	0.062		0.149	0.154	0.171	0.177	0.188	0.194

^aShows confidence interval for attribute held by approximately half the population (p = .5).

^bShows minimum detectable difference for contrast between subgroup in row head and subgroup in column head for an attribute held by half the population, using a 95% confidence interval and 80 percent power. Thus, if the difference between groups for the population exceeds the value shown, the study will have an 80 percent chance of correctly rejecting the null hypothesis of no difference using 2-tailed test at the 95 percent confidence level.

^cFU 1 refers to first follow-up planned for spring 2004.

^dFU 4 refers to hypothetical, not yet planned fourth follow-up in spring 2020.

^eHypothetical fourth follow-up assumes a second follow-up (also not yet planned) would be conducted in spring 2016, a third in spring 2018, and a fourth in spring 2020, and that at each follow-up point data are available for 94 percent of the number of cases available at the previous round of interviewing. These assumptions are used solely to respond to the OMB reviewer’s question about sample precision at later stages of the study.

Table B.5 also shows the precision of estimates for the other subgroups including the specific federal disability categories and students with Section 504 plans. The precision of subgroup estimates is less than that for all IEP and non-IEP students. The precision for the rarest disability categories is the lowest.

3. Methods to Maximize Response Rates and Deal with Nonresponse

Achieving high response rates and retaining a youth sample over the life of a multiyear longitudinal study is challenging. For this study, the challenge of securing parental consent for students to participate and of completing data collection led us to set a response rate target of 80 percent of the sample for baseline data collection. Below we outline Mathematica’s experience in interviewing similar populations and describe why we believe the 80 percent targets are ambitious but realistic.

Mathematica has achieved response rates of 80 percent or higher on some surveys of disadvantaged youth in transition. Baseline and follow-up response rates were estimated based on those studies. For the Youth Transition Demonstration, the largest demonstration funded by the Social Security Administration to help young people with disabilities make successful transitions, Mathematica is collecting data from parents and youth at baseline, and 12 and 36 months after random assignment. For the baseline and 12-month surveys, Mathematica achieved response rates of approximately 87 percent. For the 36-month follow-up, the response rate was 82 percent. For the National Job Corps Study, sponsored by the U.S. Department of Labor, Mathematica followed more than 15,000 youth over four years. At baseline the response rate was 93 percent for the full research sample. After four years, Mathematica was able to locate and interview 78 percent of the sample.

Achieving high response rates to baseline and follow-up surveys will require a combination of techniques that Mathematica has refined over the past 40 years, including the following:

• Compelling advance materials, including brochures about the study, FAQs, and endorsements from leading organizations.

• Assurance to sample members the information they provide will be secure, treated confidentially, and used only for research purposes.

• Well-designed questionnaires, with cognitively tested and easy-to-answer questions.

• A toll-free help line for sample members to call with concerns or to schedule an appointment and well-trained interviewers able to address sample members’ concerns.

• Multiple attempts to reach respondents at various times of the day and week.

• Specialized refusal conversion efforts, as needed.

• Providing a monetary thank-you to show appreciation for participant’s time and effort.

• Expertise in conducting interviews with persons with disabilities—ensuring appropriate accommodations are in place to facilitate self-reporting whenever possible.

• Leading a team of highly trained professional interviewing staff to collect high-quality data, minimizing both unit and item nonresponse. Providing project-specific training to these staff to impart the importance of the research and enhance their familiarity with each instrument prior to administration with sample members.

We note that the surveys referred to above achieved approximately 80 percent response rates at the point of follow-up, whereas we are assuming a response rate of approximately 80 percent at baseline for NLTS 2012. These more conservative planning assumptions are appropriate for NLTS 2012 for two reasons. First, some districts may not be willing to provide contact information without prior consent from parents. This factor would reduce the percentage of the initial sample for whom we acquire consent and baseline data, relative to the situation where we are able to follow our basic plan of securing verbal recorded parental consent by telephone. The second factor is that the National Job Corps Study included an attempt to conduct in-person interviews with sample members who could be located but did not complete the interview by telephone. NLTS 2012 does not include in-person follow-up.

First follow-up data collection is estimated to be completed for approximately 75 percent of the original sample (and 94 percent of the sample for whom baseline data are obtained) at a point two years after the study baseline data collection. At first follow-up, attempts will be made to locate and interview all sample members, including both those who completed a baseline and those who did not complete a baseline. Based on similar studies conducted by Mathematica, including the two mentioned above, keeping sample attrition to approximately 6 percent during a two-year interval appears feasible. An important technique for retaining sample will be to collect a substantial amount of contact information from families at baseline. In addition to name, address, telephone numbers (landline and cellular), of sample members and their close friends or relatives who do not live with them, we will ask for email addresses to which we can send reminders. Additionally, we will ask permission to send text messages to cellular phones and to use social media networks to contact them with reminders about the follow-up survey. According to a 2009 survey from the Pew Research Center’s Internet & American Life Project, 73 percent of online American teens ages 12 to 17 used an online social network website, a statistic that has continued to climb from 55 percent in November 2006 and 65 percent in February 2008. Older online teens are more likely to report using online social networks than younger teens. By collecting all of these kinds of contact information, we believe Mathematica can achieve a response rate of approximately 75 percent in 2014.

Follow-up data collection will be conducted in spring 2014. Follow-up surveys will be administered to youth, their parents, and the special education staff most familiar with each student’s school program if he or she is still in school. The follow-up interval from baseline will be two years and will not vary by age or grade, except for the student direct assessment.

An interim contact is not planned because of the large amount of contact data collected at baseline. One-third of the cases are assumed to need to be located for the follow-up interviews. Text messaging and reminders sent through social media will begin about one month before interviewing. An advance letter will be sent to sample members not reachable through electronic media about one week before the interview. Students who reach the age of consent between the baseline and follow-up interviews will be asked to consent for themselves prior to answering follow-up questions.

At each data collection point, parent interviews are conducted prior to student interviews. If a youth no longer lives with his or her parent, we will ask the parent how we can reach the child. In most cases, parents will know where the child has relocated and will provide contact information. If a family has moved, various locating methods will be used. Searches using publicly available databases, contacts provided at baseline, and mail returned as undeliverable with forwarding addresses will be the starting point for these searches. Again, we expect that one-third of the sample will need some kind of locating prior to follow-up interviewing.

Given the response rates for similar types of studies, including NLTS2, we expect that 20 percent of the students selected for the sample will not respond at baseline. We will try to contact the entire selected sample for the follow-up interview regardless of whether the parent/child was reached for baseline interview. We expect that we will not be able to interview 25 percent at follow-up, but they will not necessarily include all of the 20 percent who did not respond to the baseline.

Our expectations about attrition are also consistent with a recent analysis of response rates conducted by the National Center for Education Statistics. The Educational Longitudinal Study (ELS), a longitudinal survey of 10th graders in 2002, found just over a 6 percent loss between baseline and the first follow up interviews. While the ELS experienced differential attrition for students with disabilities (about 10 percent) and those without disabilities (about 6 percent), we do not expect such a gap. Because the ELS was a study of primarily general education students, there was little extra or targeted effort devoted to retaining students with disabilities and their parents. In contrast, NLTS 2012 has the support of the special education community and its federal leaders. We continue to present at conferences and meetings about the study and are in the process of obtaining letters of support from stakeholder groups (for example, state special education coordinators) in addition to the Assistant Secretary of the Office of Special Education and Rehabilitative Services. We believe this extra backing will enable us to achieve higher rates of response for students with disabilities at the follow-up than was achieved by NLTS2. Our plan to rely heavily on administrative records and third-party data for key outcomes also mitigates the consequences of differential attrition in survey responses. However, if a differential were to persist across multiple interview waves, it would become a much larger problem and a serious threat to the study. To address this possibility, IES plans to monitor survey completion rates—overall and by IEP status—closely with Mathematica.

4. Tests of Procedures and Methods to Be Undertaken

The questionnaires for baseline and first follow-up data collection will draw heavily on extensively used items, including many from NLTS2 and HSLS:2009. Therefore, the pretests of these instruments are expected to focus on ensuring that the question flow works well and that the time required to complete the instrument is accurately estimated. Based on these considerations, each instrument has been pretested with a convenience sample of nine or fewer individuals during August 2011. The individuals included middle and high school principals, special education and mathematics and language arts teachers, and youth who spanned the age range of youth eligible for the study along with their parents. The pretest was conducted iteratively, in two stages, so obvious errors could be corrected before conducting more interviews. To avoid interviewer effects, four different interviewers conducted some of each parent and student questionnaire. The staff questionnaires were self-administered and administered by telephone. The main pretest finding was that the instruments were too long. Pretest results and discussions with IES, OSERS and an advocacy group representing deaf students were used to reduce the instruments to budgeted lengths and reasonable respondent burden. Main changes made to the pretest interviews (aside from deleting questions of lowest priority) include:

• Improving readability, flow, and minimize cognitive load of the barriers questions to future work, education, and independent living questions for parents and students.

• Deleting transitional activities from the middle School Characteristics questionnaire that were more suited for high school students.

• Eliminating long lists of yes/no items for the Student School Program Questionnaire that were causing fatigue for respondents.

5. Individuals Consulted on Statistical Aspects of the Design

The following people were consulted on the statistical aspects of the design:

John Burghardt, Ph.D. Project director and coprincipal investigator [email protected] 609-275-2395	Francis Potter, Ph.D. Task leader, sample selection [email protected] 609-936-2799
Joshua Haimson, Ph.D. Deputy project director and task leader, data analysis [email protected] 609-275-2208	David Johnson, Ph.D. Coprincipal investigator and task leader, analysis plan and reports [email protected] 612-624-1062

In addition, the following people will be responsible for the data collection and analysis:

John Burghardt, Ph.D. Project director and coprincipal investigator [email protected] 609-275-2395	David Johnson, Ph.D. Coprincipal investigator and task leader, analysis plan and reports [email protected] 612-624-1062
Anne B. Ciemnecki, M.A. Survey director [email protected] 609-275-2323	Joshua Haimson, Ph.D. Deputy project director and task leader, data analysis [email protected] 609-275-2208
Holly Matulewicz, M.A. Deputy survey director [email protected] 617-674-8362	Martha Thurlow Ph.D. Task leader, youth assessment analysis tasks [email protected] 612-624-4826

www.mathematica-mpr.com

Improving public well-being by conducting high-quality, objective research and surveys

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Washington, DC

Mathematica® is a registered trademark of Mathematica Policy Research

1 U.S. Department of Education, National Center for Education Statistics, Common Core of Data (CCD), “State Nonfiscal Survey of Public Elementary/Secondary Education,” 2008–2009, Version 1a. No information was available for the territories of American Samoa or Guam.

2 Available national data on the number of students with Section 504 plans available from the U.S. Department of Education Office of Civil Rights do not separately identify students by grade or age range that would support an estimate of students in this group who are between 13 and 21 years of age. The estimate of 2 percent is based on the findings of a survey reported in Rachel A. Holler and Perry A. Zirkel, “Section 504 and Public Schools: A National Survey Concerning ‘Section 504-Only’ Students,” National Association of Secondary School Principals Bulletin, volume 92, number 19, 2008. While this survey had a relatively low response rate, it is the only information we have identified. The authors report reported that 1.7 percent of middle school students and 1.6 percent of high school students had Section 504 plans only.

3 Not included is the category “developmental delay”, which is used only for young children. Students served in the “developmental delay” category who continue to require special education services at ages 13 to 21 years are classified will have been reclassified in one of the 12 disability categories included in the study.

4 In some metropolitan areas with large districts, charter schools, which were listed as separate LEAs on the CCD, were combined with the district to form the district unit. This will avoid the formation of district units of only charter schools that may be geographically dispersed.

5 Folsom, Ralph E., Francis J. Potter, and Steven R. Williams. “Notes on a Composite Size Measure for Self-Weighting Samples in Multiple Domains.” In Proceedings of the American Statistical Association, Section on Survey Research Methods. Alexandria, VA: American Statistical Association, 1987, pp. 792–796.

6 During the initial district recruiting phase, the study team will be able to determine more clearly the number of such districts.