Att_4411 OMB_PartB_3_15_2011

School Improvement Status and Outcomes for Students with Disabilities Study

Part B (Description of Statistical Methods)

OMB CLEARANCE REQUEST

For Data Collection Instruments

December 17, 2010, updated March 15, 2011

Prepared for:

United States Department of Education

Contract No. ED-04-CO-0025/0013

Prepared by:

American Institutes for Research

Contents

Supporting Statement for Paperwork Reduction Act Submission 1

Description of Statistical Methods (Part B) 1

1. Respondent Universe and Sampling Methods 1

2. Procedures for Data Collection 4

3. Methods to Maximize Response Rate 8

4. Expert Review and Piloting Procedures 9

5. Individuals and Organizations Involved in the Study 9

Appendix A 1

Number of Accountable Schools with Achievement Data Between 2005–06 and 2007–08 1

Appendix B 1

Crosswalk of Survey Items With Constructs of Interest 1

Theory of Action 1

Mapping Current Survey Items to Constructs From the Theory of Action 2

List of Exhibits

Exhibit 9. Number of Accountable Schools Eligible for SWD Achievement Trend Analyses in Selected States in the EDFacts Sample, by Grade Level and State 3

Exhibit 10. Number of Schools Sampled for the School Survey in Selected States, by Accountability Status, Grade Level, and State 4

Exhibit 11. School Practices That Are Potentially Associated With SWD Outcomes 7

Exhibit 12. Organizations and Individuals Involved in the Study 9

Exhibit A–1. Number of Accountable Schools With Achievement Data Between 2005–06 and 2007–08, by Subject, Grade Level, and State 1

Supporting Statement for
Paperwork Reduction Act Submission

Description of Statistical Methods (Part B)

1. Respondent Universe and Sampling Methods

The overall sample for this study includes the following four subsamples, which will provide different types of data for addressing the research questions.

• School Accountability Sample (for Research Question 1): All public schools in the nation that reported to EDFacts school accountability and SWD performance data for
  2008–09 (67,596 public schools in 40 states for the interim study report) or 2009–10 (for the final study report)

• EDFacts Sample (for Research Question 3): A total of 2,553 accountable elementary schools from 11 selected states and 1,278 accountable middle schools from 8 selected states that will provide school-level EDFacts data for SWD achievement trend analyses.

• School Survey Sample (for Research Question 2): Up to 4,725 elementary schools and 1,913 middle schools from the EDFacts sample that will provide survey data on school practices with respect to SWDs.

• Longitudinal Student Sample (for Research Question 3): Students in elementary and middle schools in up to two selected states that will provide longitudinal student-level data.

Below we explain in detail how the four samples are constructed.

School Accountability Sample

The school accountability sample will provide data to address the first set of research questions about the inclusion of SWDs in the school accountability system and the AYP performance of schools accountable for the SWD subgroup. For the interim study report, the school accountability sample was created on the basis of the availability of the EDFacts data needed to address the first set of research questions. Specifically, to be included in the school accountability sample for the interim study report, a school must have reported to EDFacts the following types of data for the 2008–09 school year:

1. school accountability data, in particular, whether the school had a reportable number of SWDs and thus was held accountable for the performance of the SWD subgroup;

2. total number of SWDs tested in the school;¹ and

3. school AYP determination and reasons for missing AYP if applicable.

In total, 67,596 public schools in 40 states reported to EDFacts the data above for the
2008–09 school year, which constitute the school accountability sample for the interim study report. The actual number of schools and states included in a given analysis in the report, however, may differ depending on the nature of the specific analysis. The school accountability sample for the final study report will consist of all public schools in the nation that reported to EDFacts the relevant school accountability and SWD performance data for the 2009–10 school year.

EDFacts Sample

The EDFacts sample will provide school-level data for SWD achievement trend analyses in states selected based on the availability of necessary data. In constructing the EDFacts sample, we aimed to identify states with a relatively large number of schools accountable for the performance of the SWD subgroup so that it would be possible to compare groups of accountable schools that have different histories of school improvement status within individual states (e.g., ever‑identified schools vs. never-identified schools). At the same time, we aimed to include in the sample as many states as possible given data availability and budget constraints.

Specifically, based on the EDFacts data between 2005–06 and 2007–08 that we currently have, we first identified schools eligible for inclusion in the EDFacts sample as schools meeting the following criteria:

1. were accountable for SWD performance in 2005–06 and in at least one of the two subsequent years (2006–07 and 2007–08);

2. had the same school level designation (elementary or middle school) between
   2005–06 and 2007–08, AND

3. had three years of reading achievement data and/or three years of mathematics achievement data (2005–06 through 2007–08) for all three target grades (grades 3–5 for elementary schools and grades 6–8 for middle schools).

We then identified 11 states with at least 50 eligible elementary schools that have the achievement data needed for trend analyses in both reading and mathematics. Similarly, we identified 8 states with at least 50 eligible middle schools that have the achievement data needed for trend analyses in both reading and mathematics.² Exhibit 9 presents the number of accountable schools meeting the eligibility criteria for SWD achievement trends analyses in the states included in the EDFacts sample.

Exhibit 9. Number of Accountable Schools Eligible for SWD Achievement Trend Analyses in Selected States in the EDFacts Sample, by Grade Level and State

State	Elementary Schools (schools with grades 3–5)	Middle Schools (schools with grades 6–8)	Total
CA	231	246	477
FL	574	129	703
GA	318	245	563
IA	91	114	205
KS	71	–	71
MA	68	177	245
MD	773	185	958
MN	166	–	166
NC	125	52	177
ND	85	–	85
VA	51	–	51
WI	–	130	130
Total	2,553	1,278	3,831

Source: National AYP and Identification database (2005–06); 2005–06 to 2007–08 EDFacts data.

For the interim study report, we will analyze SWD achievement trends between 2005–06 and 2007–08 based on schools in the EDFacts sample. For SWD achievement analyses conducted for the final study report, a subset of the schools in the EDFacts sample—schools with the required achievement data for each year between 2005–06 and 2009–10—will be included.

School Survey Sample

Given the study’s focus on schools accountable for SWD performance, we will survey all accountable schools in our EDFacts sample described above. In addition, we will also survey nonaccountable schools (i.e., schools that were never accountable for the performance of the SWD subgroup between 2005–06 and 2007–08) in states with at least 30 such schools in our EDFacts sample. Given the budget constraints, the number of nonaccountable schools in the survey sample is capped at 350 schools in a given state. As shown in Exhibit 10, our survey sample includes 6,638 schools in total, of which 4,725 are elementary schools and 1,913 are middle schools.

Exhibit 10. Number of Schools Sampled for the School Survey in Selected States, by Accountability Status, Grade Level, and State

State	Elementary Schools (schools with grades 3–5)			Middle Schools (schools with grades 6–8)
	Accountable Schools	Non-Accountable Schools	Total	Accountable Schools	Non-Accountable Schools	Total
CA	231	350	581	246	350	596
FL	574	350	924	129	85	214
GA	318	350	668	245	62	307
IA	91	0	91	114	0	114
KS	71	0	71	–	–	–
MA	68	350	418	177	0	177
MD	773	0	773	185	0	185
MN	166	72	238	–	–	–
NC	125	350	475	52	48	100
ND	84	0	85	–	–	–
VA	52	350	401	–	–	–
WI	–	–	–	130	90	220
Total	2,553	2,172	4,725	1,278	635	1,913

Longitudinal Student Sample

To minimize data collection efforts, we plan to obtain, through data-sharing agreement, statewide longitudinally linked student-level data from up to two states that have already provided or will be providing such data to projects that are being conducted by AIR. All elementary school and middle school students in the target grades for this study (3–5 for elementary schools and 6–8 for middle schools) will be included in the longitudinal student sample. The extant data from the longitudinal student sample will inform questions about the achievement trends of SWDs (Research Question 3).

2. Procedures for Data Collection

The procedures for collecting both extant data and survey data for addressing the research questions of the study are discussed in detail below.

Collection of Extant Data

EDFacts Data

AIR currently has EDFacts data between 2005–06 and 2007–08 and incomplete EDFacts data for 2008–09. The study team will work with ED to request an extract of the complete EDFacts data for 2008–09 and 2009–10, when the data become available. The following data will be requested from EDFacts:

• School Title I status

• School status under the Individual with Disabilities Education Act (IDEA)

• Standard school identifiers, school operational status and type, grades offered, and contact information

• School improvement status under the Elementary and Secondary Education Act (ESEA) drawn from Consolidated State Performance Reports

• School performance on specific subgroup AYP targets

• The number of SWDs tested and the number or percentage of these students who scored proficient on state assessments

EDFacts relies on the states to submit the required data elements. In practice, many states have confronted challenges in submitting all their required data in a timely fashion. To maximize the number of states with usable data for this study, we propose to supplement, where possible, the EDFacts data with publicly available state AYP or achievement data from online sources.

State Longitudinal Student-Level Data

In addition to school-level data from EDFacts, we plan to collect longitudinally linked student‑level data on the following measures from up to two states:

• Performance (both scale scores and proficiency level) on state assessments in reading and mathematics and the type of assessment taken

• Grade retention/grade promotion/school dropout status

• Time spent in regular education classrooms

• Receipt of special education services (setting, type, and intensity)

• Student demographic characteristics (e.g., gender, race, free/reduced-price lunch status, special education status, and student disability category)

To minimize data collection efforts, we plan to obtain, with ED’s assistance, a data-sharing agreement with up to two of the states (e.g., Florida, North Carolina) that have provided or will be providing state longitudinal student-level data to projects that are being conducted by AIR (e.g., the ED-funded Evaluation of State and Local Implementation of Title III Standards, Assessments, and Accountability Systems). To minimize burden on the states and to consolidate our analytic work, we will make a single request to the selected states for longitudinal student‑level data files in summer 2011, and we will analyze these data in 2012 for the final study report. To maximize the knowledge gained from this study, we will make every effort to align the states included in the school survey effort with those providing longitudinal student-level data.

Collection of School Survey Data

Development of School Survey

The purpose of the school survey is to measure school practices—both regular education practices and special education practice—that may affect SWD outcomes. An initial draft of the survey was developed during the feasibility phase of this study. To identify and develop appropriate survey items, the feasibility team began by conducting a detailed review of existing school survey instruments that focus on programs and services for SWDs, asked experts in school improvement and special education to suggest constructs and items, and gathered specific suggestions and feedback from members of our Technical Working Group (TWG).

The survey has two versions—one for principals and one for special education designees. For each school in our survey sample, we will administer the principal survey to the principal and the special education designee survey to a staff member in the school whom the principal nominates as being knowledgeable about special education. The decision to have two survey respondents from each school arose out of a concern over whether the principal as a sole respondent could accurately present the implementation of school practices that focused specifically on SWDs. The principal survey mainly includes questions relevant to the education of all students in the school (including SWDs), and the special education designee survey focuses on questions relevant to all students and SWDs only in the school. The initial draft of the survey was piloted by Survey Research Management (SRM) in October 2008 in five elementary schools and four middle schools. In general, feedback from the pilot was positive, with minor suggestions for changing the content of the survey.

As described in the introduction to this document, the study team has embarked on a new direction for this study, based on the conclusions reached during the feasibility phase. As a result, the survey that was piloted in October 2008 has been revised to better align with the new study direction and the revised research questions. Before revising the survey, the study team grouped the school practices identified in the theory of action of the study (Exhibit 1) into two broad categories: (1) school practices focusing on SWDs and (2) school practices affecting all students (including SWDs). Each of the two broad categories has multiple subcategories (constructs) focusing on different aspects of school practices, which are detailed in Exhibit 11. In revising the survey, the study team carefully reviewed each survey item and made sure that each item was related to one or more of the constructs of school practice identified in the theory of action of the study.

Exhibit 11. School Practices That Are Potentially Associated With SWD Outcomes

School Practices Focusing on SWDs

Staff

• Quantity of staff serving SWDs

Professional Development

• Content and amount of professional development received by special education and regular education teachers

Access to the Regular Education Curriculum

• Educational placements for SWDs

• Instructional practices for SWDs (e.g., grouping strategies, behavior management, tutoring, test prep)

• Collaborative teaching and planning among special education and regular education teachers (e.g., co-teaching model, common planning)

• Development of standards-based Individualized Education Programs (IEPs)

Assessment of SWDs

• Number of SWDs taking regular assessments, regular assessments with accommodations, or alternate assessments

• Availability of test preparation

Identification of SWDs

• Composition of students who are identified as SWDs

School Practices Affecting All Students (Including SWDs)

Schoolwide Programs and Policies

Implementation of schoolwide initiatives including:

• Response-to-Intervention (RTI)/pre-referral

• Positive Behavior Interventions and Supports (PBIS)

• Use of data to inform instruction

• Changes in the amount of instructional time

• New curricular or instructional programs

• Ability grouping

• Technology

External Action

• Restructuring imposed from the outside

The revised survey was piloted by SRM in November 2009 in nine schools. The study team refined the survey on the basis of the pilot results and feedback from TWG members. The current draft surveys are presented in separate documents, and the item-construct mapping is presented in Appendix B.

Administration of School Survey

The spring 2011 survey planned for this study will be administed by NORC at the University of Chicago. In collaboration with NORC, the study team has discussed strategies for collecting school survey data that will maximize response rate and data quality at a reasonable cost. First, it will be essential to establish an understanding of the importance and value of the study with each respondent. We will do this by informing school district superintendents and/or state superintendents of the study and study goals and asking for a letter of support.

Second, we plan to offer the survey online (secure, ID- and password-protected) to principals and special education designees in all 6,638 schools in our survey sample and to follow up with those who do not respond to the online survey with printed surveys sent by mail. The printed surveys will be attractively designed and packaged, with a visually appealing project logo on both the package and the cover of the survey. The package will contain clear instructions for completing the survey, together with a postage-paid return envelope with clear directions for its return to NORC. A toll-free number will be included for those who have questions or who wish to complete the survey by phone.

As described above, each survey will contain two versions that need to be filled out by two respondents in each school—the principal and a special education designee. We will send the entire survey package to the principal, with a request that he or she designate a person in his or her school to fill out the special education designee version. The principal will serve as our main point of contact throughout the administration and completion of the survey.

We anticipate that many principals will not respond immediately and that one or more follow-up reminders will be required. To control costs, the initial reminders will be through e-mails and postcards (with the project-specific logo). Follow-up telephone contacts between NORC and the respondents will be managed with sensitivity to avoid making excessive demands on principals’ time. Often, making contact through another individual in the school, such as a secretary or an assistant, will facilitate working with the principal.

Finally, to encourage participation, we will offer each special education designee a $20 bookstore gift card after completing the survey. We will mail the gift cards together with thank‑you letters to all special education designees who complete the survey. The gift card is meant as a gesture of appreciation for their time and effort and a means to create a positive association with the survey and with the research project as a whole.

3. Methods to Maximize Response Rate

The anticipated response rate for the principal and special education designee surveys is approximately 80 percent. To ensure a high response rate and data quality at a reasonable cost, the study team will implement the following strategies for collecting data:

• Piloting and subsequently refining the survey instruments to ensure that they are streamlined, user-friendly, and easily understandable, all of which increase participants’ willingness to participate in the data collection activities and thus increase response rates.

• Establishing an understanding of the importance and value of the study with each respondent by informing school district superintendents and/or state superintendents of the study and study goals and asking for a letter of support.

• Offering the survey in web-based format and follow up with non-respondents with printed surveys that are attractively designed and packaged with clear instructions for completing the survey together with a postage-paid return envelope. A toll-free number will be included for those who have questions or who wish to complete the survey by phone.

• Following up regularly with non-respondents. To control costs, initial reminders will be through e-mails and post cards (with the project-specific logo). Follow-up telephone contacts between NORC and the respondents will be managed with sensitivity to avoid making excessive demands on principals’ time. Often, making contact through another individual in the school, such as a secretary or an assistant, will be more effective than attempting direct contact.

• Providing a small amount of compensation to key respondents in return for taking the time to complete the survey. Special education designees who complete the survey will be mailed a thank-you letter and a $20 bookstore gift card as a gesture of appreciation for their time and effort and a means to create a positive association with the survey and with the research project as a whole.

4. Expert Review and Piloting Procedures

Draft school surveys have been reviewed by both AIR’s internal experts and members of the study’s Technical Working Group. The surveys have also been tested through piloting procedures with fewer than 10 participants. These procedures have provided valuable data on the effectiveness of the instrument, the clarity and cognitive burden of the questions, and the ability of the survey to provide valid and reliable measures of key policy constructs.

5. Individuals and Organizations Involved in the Study

AIR is the prime contractor for the School Improvement Status and Outcomes for Students with Disabilities Study and, in collaboration with NORC at the University of Chicago, will carry out the study activities. Drs. Michael Garet and Louis Danielson will serve as co-principal investigators and Dr. Mengli Song as project director. School survey data collection will be supervised by Ms. Cynthia Simko (NORC), with content support by the deputy project director, Dr. Jenifer Harr-Robins. EDFacts data collection will be headed by Mr. Steven Hurlburt, with Dr. James Taylor serving as a senior advisor. The project officer and contracting officer’s representative at the U. S. Department of Education is Dr. Jonathan Jacobson of the Institute of Education Sciences, National Center for Education Evaluation. Contact information for the organizations and key staff involved in the study is presented in Exhibit 12.

Exhibit 12. Organizations and Individuals Involved in the Study

Responsibility	Contact Name	Telephone Number
American Institutes for Research (AIR)
Co-principal investigator	Dr. Michael Garet	202-403-5345
Co-principal investigator	Dr. Louis Danielson	202-403-5850
Project director	Dr. Mengli Song	202-403-5267
Deputy project director	Dr. Jenifer Harr-Robins	202-403-5360
EDFacts data task leader	Mr. Steven Hurlburt	202-403-6851
Senior policy advisor	Dr. James Taylor	202-403-5607
NORC at the University of Chicago
School survey task leader	Ms. Cynthia Simko	312-759-4066
U. S. Department of Education, Institute of Education Sciences (IES)
Project officer / contracting officer’s representative	Dr. Jonathan Jacobson	202-208-3876

Appendix A

Number of Accountable Schools with Achievement Data Between 2005–06 and 2007–08

Exhibit A–1. Number of Accountable Schools With Achievement Data Between 2005–06 and 2007–08, by Subject, Grade Level, and State

State	Elementary Schools (schools with grades 3–5)		Middle Schools (schools with grades 6–8)
	Mathematics	Reading	Mathematics	Reading
AR	19	35	25	31
CA	231	231	246	246
FL	574	574	128	129
GA	318	318	245	245
IA	91	91	114	114
KS	71	71	25	25
MA	68	68	177	177
MD	773	773	185	185
MN	166	166	14	14
MT	4	4	8	8
NC	125	123	52	51
ND	84	84	9	9
NH	1	0	23	17
OH	0	152	0	0
OK	19	0	69	0
RI	42	42	32	32
TN	51	0	98	0
VA	51	51	23	23
WI	25	25	130	130
Total	2,713	2,808	1,603	1,436

Source: National AYP and Identification database (2005–06); 2006–07 and 2007–08 EDFacts.

Note: “Accountable schools” are schools that were accountable for the performance of the SWD subgroup in 2005–06 and in at least one of the two subsequent years

Appendix B

Crosswalk of Survey Items With Constructs of Interest

The purpose of this appendix is to review the latest version of the school surveys against the theory of action guiding this study to identify key constructs of interest and potential survey items that are designed to measure those constructs.

Theory of Action

The box below contains the bulleted list of school practices that may be associated with school accountability for SWD subgroup performance and SWD outcomes.

School Practices Focusing on SWDs

Staff

• Quantity of staff serving SWDs

Professional Development

• Content and amount professional development received by special education and regular education teachers

Access to the Regular Education Curriculum

• Educational placements for SWDs

• Instructional practices for SWDs (e.g., grouping strategies, behavior management, tutoring, test prep)

• Collaborative teaching and planning among special education and regular education teachers (e.g., co-teaching model, common planning)

• Development of standards-based Individualized Education Programs (IEPs)

Assessment of SWDs

• Number of SWDs taking regular assessments, regular assessments with accommodations, or alternate assessments

• Availability of test preparation

Identification of SWDs

• Composition of students who are identified as SWDs

School Practices Affecting All Students (Including SWDs)

Schoolwide Programs and Policies

Implementation of schoolwide initiatives including:

• Response-to-Intervention (RTI)/pre-referral

• Positive Behavior Interventions and Supports (PBIS)

• Use of data to inform instruction

• Changes in the amount of instructional time

• New curricular or instructional programs

• Ability grouping

• Technology

External Action

• Restructuring imposed from the outside

Mapping Current Survey Items to Constructs From the Theory of Action

The table below shows which specific items from the principal survey and the special education designee survey are designed to measure each of the key constructs in the theory of action for this study.

Construct	Survey Items
Staff Quantity of staff serving SWDs	Principal Part B, Q5, h, j, n Part E, Q10, Q11
Professional Development Content and amount of professional development received by special education and regular education teachers	Principal Part B, Q5, l Part C, Q6, Q7 Sped designee Part C, Q12, Q13
Access to the Regular Education Curriculum Educational placements for SWDs	Sped designee Part A, Q6, Q9, p Part D, Q14
Access to the Regular Education Curriculum Instructional practices for SWDs (e.g., grouping strategies, behavior management, tutoring, test prep)	Sped designee Part B, Q7, a–h Part D, Q15
Access to the Regular Education Curriculum Collaborative teaching among special education and regular education teachers (e.g. co-teaching model)	Sped designee Part B, Q8, j, k Part D, Q14, c
Assessment of SWDs Number of SWDs taking regular assessments, regular assessments with accommodations, or alternate assessments	Sped designee Part E, Q16
Assessment of SWDs Availability of test preparation programs	Sped designee Part B, Q7, g
Assessment of SWDs Development of standards-based Individualized Education Programs (IEPs)	Sped designee Part E, Q17
Identification of SWDs Composition of students who are identified as SWDs	Sped designee Part A, Q4, Q6
School Programs and Policies Implementation of schoolwide initiatives including: Response-to-Intervention (RTI)/pre-referral	Sped designee Part B, Q7, a
School Programs and Policies Implementation of schoolwide initiatives including: Positive Behavior Interventions and Supports (PBIS)	Sped designee Part B, Q7, b
School Programs and Policies Implementation of schoolwide initiatives including: Use of data to inform instruction	Sped designee Part B, Q8, i

Construct	Survey Items
School Programs and Policies Implementation of schoolwide initiatives including: Changes in the amount of instructional time	Principal Part B, Q3, d, f, g Part D, Q8, Q9 Sped designee Part B, Q7, c, d, e Part D, Q15
School Programs and Policies Implementation of schoolwide initiatives including: New curricular or instructional programs	Principal Part B, Q3, e Sped designee Part B, Q9, l-o
School Programs and Policies Implementation of schoolwide initiatives including: Ability grouping	Sped designee Part B, Q7, f
School Programs and Policies Implementation of schoolwide initiatives including: Technology	Sped designee Part B, Q7, h
External action Restructuring imposed from the outside	Principal Part B, Q5, a–q

1 Ideally, we would want to know the total number of SWDs enrolled in each school for some of the analyses under Research Question 1. Such information, however, is not available in EDFacts. Therefore, for those analyses, we have to rely on the total number of SWDs tested in each school reported as part of SWD performance data in EDFacts.

2 The EDFacts data between 2005–06 and 2007–08 that we currently have are far from complete—across all three years, only 19 states had sufficient accountability and achievement data (see Appendix A), which has seriously limited the pool of states available for our EDFacts sample.