To select the sites, the contractor will use several existing databases. The contractor merged data from the NLSLSASD and the Common Core of Data (CCD), to obtain annual school-level student assessment scores, as well as school demographic information. These databases have also been linked to the Southwest Educational Development Laboratory (SEDL) CSR awards database to ensure the sample includes schools that received CSR awards. Once schools are selected, the contractor will verify their AYP (Adequate Yearly Progress) status with the National AYP and Identification database (NAYPI).
The overall site selection strategy will be as follows:
1. Identify CSR schools that had made gains in student achievement outcomes, identifying schools that had made rapid gains and those whose gains appeared more slowly over a number of years.
2. If an insufficient number of schools, across elementary and middle schools and with adequate representation of rural and urban schools, are identified among the pool of CSR schools, then NLSLSASD will be used to identify additional schools that show increases in student achievement outcomes.
Applying the strategy will involve the following procedures and criteria in selecting turnaround sites:
Within each school level, calculate the average gain in standardized achievement scores between each year from 2000 through 2005. The results of this analysis of gain score distributions will determine the cut-off points for what constitutes a significant achievement gain over time.
Identify an initial pool of schools that experienced large achievement gains that sustained for at least two years. Ideally, these schools will have significant gains in both reading and mathematics; however, the contractor will also consider schools that made significant progress in a single subject area. The initial pool will include a group that received CSR awards. For those schools in which the increases occurred in 2004, the contractor will draw on additional information to determine if the achievement gain sustained over two years (e.g. state and district data files).
From the initial pool, eliminate cases that experienced dramatic demographic shifts in terms of the number of free/reduced lunch program participants, as well as percentage of minority, ELL, and special education students.
From the pool, eliminate cases that experienced dramatic enrollment shifts.
Examine the remaining pool and determine the schools that are high poverty and were initially low performing. Depending on the distribution of schools across these categories, determine cut-off points for "initially low performing" and "high poverty."
Aim to select a large number of cases study sites that are high poverty and initially low performing.
Verify AYP status in 2004 and 2005 using the NAYPI database.
Once a pool of potential sites meeting all of the above criteria has been identified, the contractor will rank the schools in the pool according to the size of the gains in each subject area, the number of years it took to achieve the gains, the number of years of sustained progress, and the existence of a CSR award. The contractor will also consider school locale and other school demographics, such as high English Language Learner (ELL) populations, to ensure the final site selection includes cases representing a variety of school contexts.
The pilot study selection process indicated that the majority of turnaround schools were elementary schools. Consequently, the aim is to have representation across school levels, but the sample may consist largely of elementary schools.
The study of turnaround schools will benefit from the inclusion of comparison sites to understand the factors that contribute to successful school turnaround. When turnaround schools are contacted for study participation, the contractor will request names of additional schools engaged in similar activities as the turnaround sites. The contractor is using this snowball sampling technique, recommended by the TWG, to quickly identify comparison schools within the same state and district contexts of at least some turnaround schools. Then, achievement data for the list of nominated sites will be examined to select those schools that are engaged in similar activities but that have not experienced a turnaround in achievement. These comparison sites will serve as counterfactual evidence to help determine the particular factors that appear to be uniquely associated with successful school turnaround.
The targeted sample of schools selected for the field-based study consists of 20 turnaround schools and 10 schools that have not been successful based upon their student achievement levels. Before the site visit, the contractor will request documents related to school improvement from the principal. The contractor will conduct individual face-to-face interviews with the principal in the school during the turnaround period, the current principal (if different from the principal in the school during the turnaround period), assistant principal, one or two district officials responsible for the decisions and curriculum at the school, a school specialist or coach for mathematics and English/language arts (ELA), the math or ELA department chair (if applicable), and the guidance counselor.
The evaluation team will also conduct face-to-face focus group interviews of teachers, parents, community members, and the School Improvement Plan (SIP) or leadership team at each school site. Each focus group is expected to last one hour each. Teacher focus groups will include four sets of interviews: two focus groups with experienced teachers (at the school for five years or more) and two with less experienced teachers (at the school less than five years). Three to four teachers will participate in each focus group. For the other focus groups (e.g., parents, community members, leadership team), the contractor will ask for three to four participants with a longstanding relationship with the school (five or more years).
The contractor will visit the turnaround sites and comparison sites once in the fall and select the 10 turnaround sites to visit in the spring. Phone interviews will only be conducted if a key respondent is not available during the contractor’s visit.
Since this request is only for case study data collection and not survey data collection, the issue of response rates is moot. Additionally, in order to maximize the quality of data collected as part of the case studies, the contractor will visit 10 compelling schools a second time. These “compelling” schools will have a complex story that necessitates the follow-up visit to collect additional data.
No statistical tests will be conducted as part of the case studies proposed in this request. The contractor will rely on content analyses of the data collected from document reviews, interviews, and focus groups to describe the organizational structure, procedures, school culture, and the environment in which the school operates to generate individual case studies. The contractor will then examine the case studies to identify patterns across turnaround schools as well as unique experiences of turnaround schools. The analyses will be descriptive with no inference that the policies and practices described caused the gains. They will, however, provide qualitative data to generate hypotheses worthy of further investigation in other controlled studies.
B.5 Individuals Consulted on Statistical Aspects of Design
A TWG of methodology and content experts reviewed draft data collection instruments for clarity and precision, as well as to give feedback on the site selection criteria and process.
The approved analysis strategy includes two steps: the initial case report, which is a descriptive report of what is occurring at the site, and a cross-case analysis. The 10 site-specific reports will detail the individual school’s experience to illustrate the policy and processes within the complex school context. These field-based studies will integrate information from the documents reviewed, interviews, and focus groups held on site. As such, they can serve as stand-alone documents as well as the basis for the cross-site analysis.
To conduct the cross-case analysis, the evaluation team will use code common themes and patterns among the data from the 20 turnaround schools. In this exploratory study, the qualitative data analysis will be general—identifying major themes and patterns to inform hypothesis development of ideas presented in the Turnaround Evidence Review, Addendum, and the CSR program.
The inclusion of comparison schools in this study will result in an increased ability to explain the context of reform processes, implementation and outcomes. Collection of data in non-turnaround sites explicitly addresses patterns related to contextual conditions, as reflected in state and district policies and practices. Analysis of turnaround versus comparison cases will focus on finding differences between sites that may offer insight into the differences in student achievement trends.
| File Type | application/msword |
| File Title | PART B: COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS |
| Author | WestEd Staff |
| Last Modified By | DoED |
| File Modified | 2007-09-20 |
| File Created | 2007-09-20 |