Att_OMB-NEST-Recruiting-PartB 11-9-10

-

Supporting Statement for Paperwork Reduction Act Submission National Educational Transition Study (NEST)

Part B

November 9, 2010

Contract Number: ED-IES-10-C-0073 Mathematica Reference Number: 06876.381 Submitted to: Institute of Education Sciences U.S. Department of Education 555 New Jersey Ave., NW, Suite 502K Washington, DC 20208 Project Officer: Amanda DeGraff, Ph.D. Submitted by: Mathematica Policy Research P.O. Box 2393 Princeton, NJ 08543-2393 Telephone: (609) 799-3535 Facsimile: (609) 799-0005 Project Director: John Burghardt

Supporting Statement for Paperwork Reduction Act Submission National Educational Transition Study (NEST) Part B November 9, 2010

CONTENTS

PART B: SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION NATIONAL EDUCATIONAL TRANSITION STUDY(NEST) 1

B. Collection of Information Employing Statistical Methods 2

1. Respondent Universe and Sampling Methods 2

2. Statistical Methods for Sample Selection and Degree of Accuracy Needed 3

3. Methods to Maximize Response Rates and Deal with Nonresponse 7

4. Tests of Procedures and Methods to Be Undertaken 10

5. Individuals Consulted on Statistical Aspects of the Design 10

Appendix A: Individuals with Disabilities Education Improvement Act, Section 664(e)

Appendix B: Individuals with Disabilities Education Improvement Act, Section 664(e)

Appendix C: Recruiting Materials and Protocol

Appendix D: Mathematica Confidentiality Pledge

PART B:
SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION
NATIONAL EDUCATIONAL TRANSITION STUDY(NEST)

The U.S. Department of Education (ED) is requesting Office of Management and Budget (OMB) clearance for STOYD, a five-year longitudinal study focused on the educational and transitional experiences of youth with disabilities between the ages of 13 and 21. This study is being conducted by Mathematica Policy Research and its partners, the Institute on Community Integration, and Decision Information Resources (DIR), under contract with ED (contract number ED-IES-10-C-0073).

The main objectives of the study are to describe the background, secondary school, transition, and postsecondary experiences, and outcomes of youth who receive special education services and to gauge how the experiences of these youth differ from: (1) those who have no identified disability, (2) those who do not receive special education services but who have a condition that qualifies them for accommodation under Section 504 of the Vocational Rehabilitation Act of 1973 and (3) similar cohorts of youth receiving special education with disability studied in the past.

The study will provide policymakers and educators with critical information that is not available from other sources. The study will provide up to date information on the barriers and challenges youth with disabilities encounter during and after high school; the services and support they receive to help them overcome these barriers from their families, community service providers, secondary and postsecondary schools, and employers; and the extent to which youth make a successful transition to postsecondary education, employment, and independent living. The study will examine these issues from multiple perspectives including those of school staff, parents, and the youth themselves. By comparing the experiences of a current cohort to those of previous cohorts, the study will be able to describe changes in the composition of students with disabilities over time as well as changes in their school experiences and outcomes.

A national probability sample of 15,000 students will be selected and recruited in two stages. The study team will first select and recruit a nationally representative sample of approximately 300 school districts (from a pool of approximately 450 sampled districts); then the team will sample and recruit students from the 300 districts that have agreed to participate. The first wave of data collection will begin in January 2012 and the second in January 2014, when sample members will be between 13 and 21 and 15 and 23 years old, respectively.

The study data collection will draw on the following data sources each of which provide valuable information:

• Parent interviews will provide information on the characteristics of the family and youth (for example how their disabilities affect their ability to perform various tasks), parents educational expectations for their child, parent’s involvement in and perceptions of the transition planning process and supports their child receives in school.

• Youth interviews will provide information on their experiences and perceptions of school, their career and educational expectations, their engagement in school, and other key outcomes

• Principal surveys will cover school policies, programs, staffing levels, and other resources available in the school

• Teacher surveys completed both by the language arts teacher of each sample member and a special education teacher who is most familiar with each special education student’s school program will provide information on the students program of study, the classroom setting and the student’s classroom participation, the student’s participation in transition planning and activities designed to help them apply to postsecondary programs and jobs, and the services and other accommodations received by the student.

• Student school records will provide more detailed information on key outcomes (including attendance, courses taken, and test scores),

This OMB clearance request is the first of three for this study. The study schedule requires that district and school recruitment begin as early this spring as possible. This request seeks clearance for (1) the sample design and protocols for recruiting school districts into the study and (2) the process for securing from the selected school districts lists of students necessary to select the student sample and the contact information necessary to contact the parents of selected students and/or the students themselves in the case of students 18 or older for purposes of obtaining informed consent and conducting baseline data collection. The second clearance package, which ED plans to submit in spring 2011, will include all protocols and instruments for securing informed consent and conducting baseline data collection (which will begin in early 2012) and a more detailed analysis plan. The third clearance package, which is scheduled to be submitted in spring 2013, will include the follow-up data collection instruments (for surveys that will begin in early 2014). Initiating district recruitment in the spring of 2011 is critical to the goal of achieving a high rate of district participation and thereby minimizing the potential for nonresponse bias from district refusals. The study team plans to contact all districts before the summer recess and to continue the recruitment effort over the summer.

The study design and data collections described here are similar to prior longitudinal studies of students with disabilities conducted by ED but attempts to improve on them by 1) using innovative methods of securing parental consent for youth participation to improve participation, 2) limiting in school data collection and spreading the burden across schools in districts with multiple middle and high schools, 3) including students without IEPs (including both those with conditions that qualify them for a Section 504 plan as well as those with no disabilities), and 4) seeking more information on student barriers and activities that support transition.

B. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

This study is designed to collect information about youth between the ages 13 and 21 (as of December 2011) who are in schools serving grades 7 to 12 or who are in an ungraded school and in this age range. Within this target population there are three key groups of interest: (1) students identified as needing special education services—that is, those with IEPs; (2) students who have not been identified as needing special education services but who have a condition that qualifies them for accommodations under Section 504 of the Vocational Rehabilitation Act of 1973; and (3) students with no IEP and no section 504 plan. Based on counts for the 2008–2009 school year, approximately 22,500,000 students are in the appropriate age and grade range for this study in the 50 states and the District of Columbia, and approximately 300,000 are in schools run by the Department of Defense and the Bureau of Indian Affairs or in the territories (Table B.1).¹ Of these, approximately 2,800,000 students have IEPs. Of the students who do not have IEPs, an estimated two percent (approximately 450,000 students) have Section 504 plans.²

Table B.1. Population Sizes for Target Populations and Subpopulations Defined by Disability

	Estimated Population Count
All students ages 13 to 21	22,500,000
All students without IEPs	19,720,000
Without Section 504 plans	19,270,000
With Section 504 plans	450,000
All students with IEPs	2,780,000
Specific learning disabilities	1,508,000
Other health impairments a	901,250
Speech or language impairments	113,200
Autism	95,000
Multiple disabilities	73,200
Hearing impairments	34,000
Orthopedic impairments	27,500
Traumatic brain injury	13,900
Visual impairments	12,200
Deaf-blindness	750

^a Estimates for this row also apply to the categories mental retardation and emotional disturbance.

2. Statistical Methods for Sample Selection and Degree of Accuracy Needed

Two stage sampling will be used to select approximately 15,000 youth ages 13 to 21 as of December 2011. Of these youth, approximately 12,000 are expected to respond. The respondents will include approximately 9,600 students with IEPs and 2,400 students without IEPs. Of the 2,400 students without IEPs, approximately 400 will be students with Section 504 plans.

The sampling design balances several objectives but places the highest priority on obtaining precise overall estimates for all students with IEPs. Another priority is obtaining precise estimates for each of the federally defined disability categories. Other priorities are to obtain estimates for the Section 504 students and students with no IEP and no Section 504 plan.

The sample will be selected in two stages. In the first stage, the study team will randomly select 450 school districts using ED’s Common Core of Data (CCD) in the expectation that 300 will agree to participate in the study. (Additional districts will be randomly selected as a reserve just in case they are needed to recruit 300 participating districts). In the second stage, the study team will select samples of IEP students, students with a Section 504 plan, and students with neither an IEP nor a section 504 plan from the 300 participating districts, distributing the sample evenly across these districts to minimize the effects of clustering. Details of the proposed sample selection are described below.

a. District Sampling Frame

The sampling frame for the districts in the study comes from the CCD. About half of the nation’s 14,000 public school districts contain fewer than 100 students with an IEP across all ages. To support efficient data collection, the study team will combine nearby districts with fewer than 100 IEP students to form district units that contain at least 100 IEP students. For the selection of students, the study team will compile sample information from all component districts (typically two) and sample from the combined student populations. The sample of districts and district units will be drawn with probability proportional to a composite size measure that includes the IEP and non-IEP populations in the districts. This measure will increase the selection of districts with more students with an IEP and can provide nearly self-weighting national samples of students within disability categories. The final sample will include approximately 288 district units with perhaps 12 multi-district units; the study team projects that the sample will contain approximately 300 school districts.

b. Stratification of the District Sample

The study team will stratify the district units before sample selection. Stratification of districts serves two purposes: (1) ensuring that the representation of particular types of districts is not left to chance and (2) allowing the oversampling of rare district types that would occur too infrequently if all district types were sampled with the same probability. ED and the study team may consider as one stratification factor the poverty level of the district. This would ensure that districts that high poverty districts are included in the district sample and can be oversampled to enable the oversampling of the youths in high poverty districts. The team will also consider stratification by geographic region of the country, the degree of urbanicity, the extent of minority enrollment, and the number of IEP students.

The study team will use both explicit and implicit stratification. In explicit stratification, a portion of the sampling frame is grouped together (such as districts in major urban areas) to form a stratum and a specific sample size is allocated to the stratum. In implicit stratification, the sampling frame within a stratum is ordered by a factor such as district size (for example, four levels of district size), and by using a sequential selection procedure, the sample selected is approximately proportionally allocated across the four levels of district size.

c. Size Measure for District Selection

The study team will use a composite size measure to select the sample of districts and district units.³ The composite size measure will be based on the district level counts of the number of students with IEPs, N(students with IEPs in district i), and the number of students without an IEP, N(students without IEPs in district i). The size measure is based on global sampling rates for students with IEPs, f(IEP), and those without an IEP, f(W/O IEP), using data available from the CCD. The size measure for the ith district will be of the form

Si = f (IEP) * N(students with IEPs in district i) + f (W/O IEP) * N(students without IEPs in district i)

We expect that some districts (such as Los Angeles and Chicago) with large student populations will be selected with certainty, and the study team will use this size measure to identify these districts. The remaining districts will be selected with probability proportional to the composite size measure and without replacement. This composite size measure can result in nearly self-weighting samples of students within the disability categories.

To enable the oversampling of students in high poverty districts, the study team may assign higher global sampling rates for students with IEPs and for students without IEPs for high poverty districts than for other districts. These higher global sampling rates would allow the oversampling of students in high poverty districts, while maintaining nearly equal response burden across the selected districts. Because not all districts will be willing to participate, the study team will select a reserve sample of districts. As recruitment proceeds, the study team will keep track of how many sampled districts have been recruited. If additional districts are needed, either to complete the sample or to account for a district that withdrew from participation, additional districts will be randomly selected from the reserve list.

d. Student Selection

Based on the federal reporting requirements and on the experience of NLTS 2 we anticipate that all districts will maintain lists of students by federal disability category. Based on information from the ED Office of Civil Rights we anticipate that most districts will also maintain a list of non-IEP students with Section 504 plans. Using these lists, the study team will assign each student age 13 to 21 to one of the strata (one of the IEP disability categories, the stratum of non-IEP students with Section 504 plans, or the stratum of non-IEP students without Section 504 plans). The study team will then draw a random sample from each stratum (controlling implicitly by grade level and school) at a rate designed to yield the target number of students in each stratum. The team will also select a reserve sample available for use to account for students who may be ineligible or choose not to respond.

It is anticipated that a proportion of districts will neither maintain lists by disability category nor have lists of students with Section 504 plans.⁴ In these districts, the study team will first select schools and then obtain the lists from the selected schools. The schools will be selected with probability proportional to size (such as the number of non-IEP students).

The study team expects to interview approximately 32 IEP students/parents and 8 non-IEP students/parents in each of the districts or district units. To obtain this many respondents from each district, the study team will select samples of 40 IEP students and 10 non-IEP students, based on an anticipated response rate of 80 percent.

e. Precision and Minimum Detectable Differences

Table B.2 presents target sample sizes and estimates of precision for a set of disability category subgroups and the non-IEP sample (divided into Section 504 students and all other students). All of the sample sizes in this table represent the estimated number of youth (or parents) responding to the surveys. This sample allocation is designed to allow meaningful precision for survey estimates and minimum detectable differences of approximately 0.10 for proportions near 0.50 (for a two-sided test with alpha of 0.05 and 80 percent power) for most of the disability categories. The precision estimates are based on an allocation of 400 respondents with Section 504 plans. The table presents estimates of minimum detectable differences (MDDs) for comparisons between the subpopulations and two larger populations: all students with IEPs and all students without IEPs.

Three categories of disabilities (traumatic brain injury, visual impairments, and deaf-blind) are too rare to support reliable estimates individually without shifting too much sample from much larger categories. For some of the analysis these categories will be clustered with other disability groups to provide more reliable estimates. For example, the study team may group together the cluster of students with sensory impairments (namely, those with hearing impairments, visual impairments, and deaf-blindness).

Table B.2. Sample Sizes, Precision, and Minimum Detectable Differences for Subpopulations Defined by Disability Category

		Half-width of 95% Confidence Level at Selected Proportions			Minimum Detectable Differences (MDD)
	Proposed/ Estimated Sample Size	.50	.10		With IEPs	Without IEPs
All students without IEPs	2,400	0.021	0.013		0.076	--
Without Section 504 plans	2,000	0.023	0.014		0.080	--
With Section 504 plans	400	0.049	0.030		0.148	0.154
All students with IEPs	9,600	0.017	0.010		--	0.076
Specific learning disabilities	1,600	0.025	0.015		0.096	0.093
Other health impairments	1,200	0.029	0.017		0.101	0.101
Mental retardation	1,200	0.029	0.017		0.101	0.101
Emotional disturbance	1,200	0.029	0.017		0.101	0.101
Speech or language impairments	1,000	0.031	0.019		0.106	0.107
Autism	1,000	0.031	0.019		0.106	0.107
Multiple disabilities	900	0.033	0.020		0.110	0.111
Hearing impairments	600	0.040	0.024		0.126	0.129
Orthopedic impairments	450	0.046	0.028		0.142	0.145
Traumatic brain injury (TBI), visual impairments, and deaf-blindness	450	0.046	0.028		0.142	0.145
Disability Clusters
Sensory impaired a	818	0.035	0.021		0.114	0.116
Other health, orthopedic, and TBI b	2,100	0.023	0.014		0.096	0.088

Note: MDDs apply to comparisons between the row subpopulation and either all students with IEPs (excluding those students in the specific row subpopulation) or all students without IEPs. The MDDs are computed for detecting a difference in a proportion near 0.50 for a test with alpha of 0.05 and 80 percent power.

^aThe disability cluster of “sensory impaired” includes students in the categories of hearing impaired, visual impairments, and deaf-blindness. The sample count shown includes the expected proportion of students with visual impairments or deaf-blindness in the sample for the combined disability category of TBI, visual impairments, and deaf-blindness.

^bThe disability cluster of “other health, orthopedic, and TBI” includes the categories of other health impairments.

3. Methods to Maximize Response Rates and Deal with Nonresponse

Immediately after receiving OMB clearance, the study team will begin the process of recruiting district participants. Districts will receive a recruitment package that includes an introductory letter from ED and a study summary. The study team will begin calling district contacts within one week after they receive the recruitment package. The goals of the initial calls will be the following:

• Describe briefly the study and all participation requirements.

• Answer any immediate questions.

• Elicit the district’s agreement to participate.

• Arrange for the provision of student sampling variables for all students ages 13 to 21.

• Discuss the logistics of providing the following information for the 65 selected students: contact information for the student and parent, the student’s general and special education teacher, and the principal of the student’s school.

Subsequently, the study team will

• Follow up with appropriate personnel to discuss data needs and data collection activities.

• Secure participation agreement with a signed Memorandum of Understanding (MOU).

After that,

• The district provides de-identified sample frame data (via electronic upload).

• The study team will share ID numbers of 65 sampled students with the districts and arrange to obtain contact information for student, parent/guardian, teacher(s), and principals (providing a sample informed consent document if a district requires it).

Mathematica, our contractor, has developed effective methods to maximize response rates. This section focuses on the strategy for maximizing district participation. (Subsequent clearance requests will describe the approach for maximizing survey response rates of youth, parents, and school staff.) The approach for maximizing district participation includes enlisting the support and endorsement of key influential stakeholders; using experienced senior staff and a flexible approach to recruit districts; and identifying potential concerns and addressing them (including concerns about student privacy and confidentiality, the benefits and costs of participation, and staff burden). By securing OMB approval to initiate district recruitment this spring, the team will have sufficient time to take the following steps designed to achieve high rates of district participation:

• Mathematica and IES plan to work with OSERS and OSEP to engage stakeholder groups to ensure that the study represents their perspective and has their endorsement. The study team will send materials describing the study, make personal contacts with organization leaders to discuss the study, and request opportunities to make stakeholders aware of the study. Examples of groups whose support may be sought include Council for Exceptional Children; National Association of State Directors of Special Education; Council of Chief State School Officers; National Association of Secondary School Principals; Parents Alliance, Inc.; National Down Syndrome Society; and other condition-specific groups.

• We will follow a structured recruiting process that will allow flexibility in developing solutions with hesitant districts. Recruiting will be carried out by staff with the knowledge, experience, and authority to develop flexible solutions to meet districts’ participation requirements. The study team will develop written protocols for contacts and materials describing the study; delineating the roles of participating district and school staff, parents, and students; and answering frequently asked questions (FAQs). Study team members have contacts in nearly every state and in many large districts that will be sampled with certainty. If recruiters encounter resistance or difficulty getting access to decision makers, the team will use these contacts to identify staff in each district who have authority and who may facilitate district participation. The study team will execute an MOU and follow district research review requirements as necessary. As mentioned earlier, this information about contacts with district offices will be entered into an electronic tracking database. The study team will use the database to generate progress reports, detect and solve problems promptly, and ensure that no superfluous contacts with districts occur.

• Addressing Potential Obstacles to Participation. Achieving high participation rates hinges on having clear strategies to address familiar potential obstacles to district participation. Mathematica’s experience recruiting school districts for participation in various studies provides an understanding of the concerns often expressed by district staff.

District Concerns and Mathematica’s Approach to Addressing Them

Privacy of student body	For districts with privacy concerns, Mathematica will select the study sample from lists of eligible students without identifiers attached. The districts will provide only sample stratification variables with a unique identifier than can be linked to the student after sample selection. Thus, only students selected for the sample need to be identified. IES will issue a letter to districts reiterating that the study is covered by the Family Educational Rights and Privacy Act (FERPA).
Security of data	Mathematica will provide a secure (https) website for each school district to upload and view only its own data.
IRB approval and parent consent	A preliminary discussion with Mathematica’s IRB at Public/Private Ventures indicates that it will provide a waiver of written consent to be replaced by web-based consent or oral consent via telephone (digitally recorded). The study team will document these methods and mail a written copy of consent forms to those who consent. Mathematica will prepare separate IRB or research requests for districts when required.

Subsequent OMB packages will address the methods to encourage participation of survey respondents including school staff, youth, and parents. Proven methods, to be described in detail in subsequent submissions, include the following:

• Well-designed questionnaires, with cognitively tested and easy-to-answer questions

• Compelling advance materials

• Assurances to sample members that the information they provide will be secure, treated confidentially, and used only for research purposes

• A toll-free help line for sample members to call with concerns or to schedule an appointment and well-trained interviewers able to address sample members’ concerns

• Multiple modes of responding (web/telephone/fax for school staff)

• Reminder letters and emails, which will contain identification numbers and passwords so web respondents can “click through” to the web questionnaire

• Modest payments to compensate principals and teachers for their time

• Multiple attempts to reach respondents at various times of the day and week

• A small monetary thank you to show appreciation for parent’s and student’s time and effort

• Specialized refusal conversion and tracing as needed

• Focus on obtaining accurate contact information to facilitate locating for follow-up interviews

4. Tests of Procedures and Methods to Be Undertaken

Mathematica has worked with numerous school districts in 31 states and with charter schools in 15 states for different research projects in developing the methods to be used here. This experience has allowed us to hone our district recruiting procedures and methods. Individual instruments for school staff, parents, and teachers will be pretested before ED describes them in subsequent OMB submissions.

5. Individuals Consulted on Statistical Aspects of the Design

The following people were consulted on the statistical aspects of the design:

John Burghardt, Ph.D. Project director and co-principal investigator [email protected] 609-275-2395	Francis Potter, Ph.D. Task leader, sample selection [email protected] 609-936-2799
Joshua Haimson, Ph.D. Deputy project director and task leader, data analysis [email protected] 609-275-2208	David Johnson, Ph.D. Co-PI and Task leader, analysis plan and reports [email protected] 612-624-1062

In addition, the following people will be responsible for the data collection and analysis:

John Burghardt, Ph.D. Project director and co-principal investigator [email protected] 609-275-2395	Joshua Haimson, Ph.D. Deputy project director and task leader, data analysis [email protected] 609-275-2208
Anne B. Ciemnecki, MA Survey director [email protected] 609-275-2323	Eric Zeidman, Ed.M., Task leader, district recruitment [email protected] 609-936-2784
David Johnson, Ph.D. Co-PI and Task leader, analysis plan and reports [email protected] 612-624-1062	Martha Thurlow Ph.D. Task leader, youth assessment analysis tasks [email protected] 612-624-4826

www.mathematica-mpr.com

Improving public well-being by conducting high-quality, objective research and surveys

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Washington, DC

Mathematica® is a registered trademark of Mathematica Policy Research

1 U.S. Department of Education, National Center for Education Statistics, Common Core of Data (CCD), “State Nonfiscal Survey of Public Elementary/Secondary Education,” 2008–2009, Version 1a. No information was available for the territories of American Samoa or Guam.

2 Available national data on the number of students with section 504 plans available from the US Department of Education Office of Civil Rights do not separately identify students by grade or age range that would support an estimate of students in this group who are between 13 and 21 years of age. The estimate of 2 percent is based on the findings of a survey reported in Rachel A. Holler and Perry A. Zirkel, “Section 504 and Public Schools: A National Survey Concerning "Section 504-Only" Students”, National Association of Secondary School Principals Bulletin, Volume 92, number 19, 2008. While this survey had a relatively low response rate, it is the only information we have identified. The authors report that reported that 1.7 percent of middle school students and 1.6 percent of high school students had section 504 plans only.

3 Folsom, Ralph E., Francis J. Potter, and Steven R. Williams. “Notes on a Composite Size Measure for Self-weighting Samples in Multiple Domains.” In Proceedings of the American Statistical Association, Section on Survey Research Methods. Alexandria, VA: American Statistical Association, 1987, pp. 792–796.

4 During the initial district recruiting phase, the study team will be able to determine more clearly the number of such districts.