LACIO supporting stmt Part A 091907 ver

LACIO supporting stmt Part A 091907 ver.doc

Longitudinal Analysis of Comprehensive School Reform Implementation and Outcomes (LACIO)(KI)

OMB: 1875-0222

Document [doc]
Download: doc | pdf

PART A. JUSTIFICATION

A1. Circumstances Making Collection of Information Necessary

Comprehensive School Reform under No Child Left Behind Act of 2001, P.L. 107‑110. This request is for OMB approval of revised data collection associated with the Longitudinal Assessment of the Comprehensive School Reform Program Implementation and Outcomes (LACIO). Sec. 1606 of the Elementary and Secondary Education Act (ESEA), as reauthorized by the No Child Left Behind Act (P.L. 107‑110) mandates activities to be conducted by K‑12 schools across the country under “comprehensive school reform,” and Sec. 1607 mandates the National Evaluation.1

Along with the new authorization, Congress appropriated $235 million for the Comprehensive School Reform program in fiscal year 2002.2 This level of funding supported reform activities at an estimated 2,000 schools. The vast majority of these schools were Title I schools “in need of substantially improving” their student achievement levels. 3

The federal funds were distributed on a formula basis to the states, which in turn made grants to districts to support the schools.

Each school received an average of over $70,000 per year for three years. The modest award amounts and limited award durations were intended to signal a catalytic role for the federal funds - helping a school to initiate or advance its reform efforts - rather than serving as a long‑term subsidy.

As of September 2005, the schools in the cohort studied through LACIO have completed the original funding cycle. Consequently, this request for the revision of OMB approval reflects a changed focus of the LACIO to address two concerns: (1) the extent to which the federal funds actually played the anticipated catalytic role and schools continue reform after CSR funding ended; and (2) the lessons from CSR that can be applied to school actions to successfully stimulate improved student achievement. This revision to the existing LACIO evaluation is for a new evaluation that includes a survey of a nationally representative sample of CSR and comparison schools approved by OMB (approval number 1875-0222). It draws on the findings from the study and poses additional questions that deepen knowledge of the role of comprehensive reform in changing outcomes in high-poverty, low-achieving schools.

The Longitudinal Analysis of the Comprehensive School Reform Program Implementation and Outcomes (LACIO) has involved three types of data collection and analysis:

  • Quantitative analyses of the relationship between scientifically based model adoption and school-level achievement using all 2002 CSR grantees;

  • Quantitative analysis of the longitudinal relationship between CSR awards and school-level achievement in the universe of 2002 Title I CSR grantees;

  • Quantitative and qualitative analyses of a survey of 500 CSR schools that received awards in 2002 and a matched set of 500 non-CSR schools, as well as case study data on 15 CSR and 15 matched non-CSR schools.


This extension will increase the number of sites for case studies and will place less emphasis on discrete programs and more on the role of CSR approaches in improving school performance. It will gather new data to determine what strategies and practices are implemented in improved schools. The contractor will collect the new data through document review and site visits.

The contractor will carry out the work described in this modification, culminating in a final report due October 27, 2008. The work to be completed is described below, along with a timeline for deliverables and other activities to support the data collection, analysis and reporting.

Evaluation Goals and Questions.

The No Child Left Behind legislation stipulates two broad goals for the LACIO:

1) To evaluate the implementation and results achieved by schools after three years of implementing comprehensive school reforms; and

2) To assess the effectiveness of comprehensive school reform in schools with diverse characteristics.

The original U.S. Department of Education (ED) solicitation for the National Evaluation articulated these two goals in terms of three specific evaluation questions:

1) To what extent have CSR schools made progress on state assessments in comparison to gains for schools in their state with similar characteristics?

2) How effective are various school reform activities, especially in diverse settings, and to what extent can school progress be linked to CSR reforms?

3) How have district policies and state policies affected CSR program implementation and comprehensive school reform?

In 2005, ED revised the LACIO to collect follow-up information on the extent to which reforms were sustained and to identify approaches for improving Title I schoolwide programs.

In 2006, ED determined that the LACIO could conduct additional analyses and on-site data collection about schools that changed from low-performing to meeting NCLB goals. Under the current contract modification, the contractor will examine the policies and processes implemented, actions taken, role of the 11 CSR components, and characteristics of lead actors in turnaround schools. The study will compare schools that are more or less successful in improving student achievement, focusing on the following new questions:

1) What is the role of the 11 CSR components in successful schools that turned around academically and narrowed achievement gaps between subgroups?

2) What combinations of strategies and promising practices did successful schools use?

3) Who or what was responsible for implementing these strategies?

4) How have successful schools overcome challenges in implementing turnaround strategies?

The contractor will gather data from a variety of sources to answer each question during site visits to successful turnaround schools and a small group of comparison schools.

Evaluation Design

The purpose of the new data collection is to identify promising practices in turnaround schools by documenting the role of CSR approaches and other strategies associated with improved student performance. Such schools demonstrate higher achievement levels and are making their Adequate Yearly Progress (AYP) targets.

The deliverables will include a series of ten site-specific reports that highlight and describe promising practices. In addition, the findings from these ten site-specific reports will be used to inform the final cross-site analysis and report. The final report will examine the extent to which the 11 CSR components and others identified in the Turnaround Evidence Review and the contractor’s supplement are apparent in successfully reforming schools.

The contractor’s supplement, also referred to as the Addendum to the Turnaround Evidence Review, focused on staff capacity, development, and leadership; external assistance; and, effective schools. Specifically, the literature referenced in the Addendum included the final report of the National Longitudinal Evaluation of Comprehensive School Reform (Aladjem et al., 2006), studies of the new American Schools (Berends et al., 2001 and 2002; Bodilly, 1996), as well as more general literature on comprehensive school reform and leadership (Desimone, 2000; Elmore, 2000; Hallinger, Bickman, & Davis, 1996).

Site visit protocols. The contractor developed a set of case study protocols to guide data collection on site. The instruments drew on existing research on CSR, LACIO findings and data collection experience, AIR’s data collection for the Study of State Implementation and Accountability and Teacher Quality under NCLB (SSI-NCLB), as well as the Turnaround Evidence Review.

The case study protocols will involve interviews and/or focus groups of district officials, principals, teachers, parents, community members, instruction/curriculum specialists, School Improvement Plan (SIP)/leadership team, department chairs, assistant principals, and guidance counselors.

A Technical Working Group (TWG) reviewed draft data collection instruments for clarity and precision, as well as to give feedback on the site selection criteria and process. The contractor pilot tested all data collection instruments. During these tests, which were administered to no more than nine respondents, the team assessed item comprehension, the effectiveness of the proposed strategies for gaining cooperation, and the length of time for respondents to answer questions in the instruments. Such information is critical for determining the burden associated with each instrument, which must be presented to all respondents before the administration of any federally sponsored research questionnaire to more than nine respondents.

The contractor delivered to ED an initial set of protocols in January 2006. Following ED review, the contractor pilot tested the revised protocols and submitted a report to ED detailing the results from the pilot test as well as suggested changes to the protocols in February 2007. Last, the contractor submitted a final set of protocols to ED in April 2007. Under Budget Services direction, the contractor revised the data collection protocols in May 2007.

Identify case study sites. The goal is to select schools that experienced a significant increase in student achievement and sustained it over time. The sites selected for the case studies of schools that show marked improvement in student achievement outcomes will be, to the extent feasible, sites that received CSR funds in any year of the program’s history. The contractor will select case study sites with preference given to schools that show rapid (within one to two years) or steady (within three or more years) gains that sustained for three to four years. The contractor will also consider schools with sustained academic growth for shorter periods of time (one to two years).

To select the sites, the contractor will use several existing databases. The contractor merged data from the National Longitudinal School-Level State Assessment Score Database (NLSLSASD) and the Common Core of Data (CCD), to obtain annual school-level student assessment scores from 2000 through 2005, as well as school demographic information. These databases have also been linked to the Southwest Educational Development Laboratory (SEDL) CSR awards database to ensure the sample includes schools that received CSR awards. Once schools are selected, the contractor will verify their AYP (Adequate Yearly Progress) status with the National AYP and Identification database (NAYPI).

More detail on the selection process is presented in section one of Part B: Respondent Universe and Sampling Methods.

Additional Details on Data Sources. The NAYPI includes 2004 and 2005 AYP data from all 50 states and the District of Columbia for 88,160 schools. Some data elements, such as the applicability of subgroups, were not available for all states. The database also does not include 2,529 schools for which states reported AYP as “not determined,” and about 4,000 schools that were not included in state-provided data files. The NAYPI is a source of AYP status rather than performance data.

The number of states included in the NLSLSASD varies across both school level and academic year (Exhibit 1). The largest number of states is represented in the 2001 through 2005 school years. The original suggestion was to examine gains from 2001 through 2005. We will include data from 2000 in the site selection. If 1999 data are included, the number of states represented is limited and requires designation of separate selection criteria. Therefore, the contractor will not include 1999 achievement data. The high school sample will be reduced to the 22 states for which there are consistent achievement data from 2000 to 2005.

Exhibit 1

Number of States with Valid Standardized Achievement Scores, by School Level and Academic Year


 

Elementary

 

Middle

 

High

Year

Math

Reading

 

Math

Reading

 

Math

Reading

1999

26

26

 

26

26

 

15

13

2000

38

39

 

39

39

 

22

21

2001

43

43

 

43

43

 

26

25

2002

45

45

 

45

45

 

26

26

2003

49

49

 

49

49

 

26

26

2004

48

48

 

48

48

 

28

29

2005

45

46

 

45

45

 

25

26

Disaggregated data are available in the NLSLSASD for the majority of states from 2003 forward. In states with well-established assessment systems, disaggregated data are also available for years prior to 2003, but the data are less consistent than data provided after 2003. To the extent feasible, we will use these data as selection criteria.

Conduct site visits. The contractor will conduct site visits at a sample of schools with evidence of turnaround. The case studies will determine the role of CSR approaches as described in the 11 components and other strategies and practices implemented in successfully turned-around schools. Evaluators will pay close attention to understanding what combinations of strategies and promising practices successful schools used to turn around academically and narrow achievement gaps between subgroups. In addition, the site visits also will examine how districts support turnaround efforts at each case study school.

The case study sample will include 20 successful turnaround schools, with as many as feasible (at least 10) being schools that received CSR awards, and 10 additional schools that have not been as successful. A two-person team of evaluators will visit each site for three to four days during the first visit. After the initial visit, the contractor and ED will identify 10 compelling sites to visit a second time. The contractor believes some of the turnaround sites’ success may be due to more complex phenomena than others. Therefore, it contends a follow-up visit will be necessary to collect all necessary data to describe the school’s story for a portion of the turnaround sites. The contractor will conduct a follow-up visit to 10 of these “compelling” sites. The second visit will involve only one staff member for a shorter time. The intent of this extended time is to provide in-depth descriptions of the turnaround phenomena.

The contractor developed a protocol for the site visits, which includes issues to be addressed during the visit and documents to collect. In addition, site visitors will meet prior to any visits to ensure there is a common understanding of the framework, questions, and issues to be addressed during the visits.

LACIO, and other national studies of CSR, have found limited relationships between CSR and improvement in student achievement. The studies found no clear differences between CSR and non-CSR schools in either gains or the extent of implementation of the 11 components. Consequently, these cases are an opportunity to generate hypotheses about the reform features—which may not be fully captured by the 11 CSR components--that are associated with success. The sample is purposive, focusing on schools that have successfully turned around so their students markedly increase achievement. The case studies will examine practices associated with increased student achievement. The framework for examining the features will include the 11 CSR components, augmented by research, particularly the Turnaround Evidence Review, which has noted additional features that may account for changes in schools that result in positive outcomes. Some of these are:

  • The extent to which the school has autonomy in allocating resources and selecting staff

  • The extent to which curriculum and instructional practices are aligned within the school and with district and state policies

  • The extent to which school and district leaders monitor practices within the school

  • The extent to which the community is engaged in school reform

  • The actions and capabilities of the school leader

To understand how districts support improvement efforts at each case study school, evaluators will arrange interviews with district staff who were involved in planning for turnaround or are currently involved with managing turnaround schools (including those responsible for managing contracts with EMOs [Educational Management Organizations] and/or charter schools). The contractor will also collect archival documents from the district and school relevant to the turnaround process, including but not limited to improvement plans, assessment descriptions, and staff resumes. Whenever possible, such documents will be secured prior to the site visit so that evaluators can begin the data analysis and focus the data collection while on-site.

The contractor will conduct the site visits after OMB approval, between September 25, 2007 and March 17, 2008. Findings from the case studies will be delivered to ED in a final report, which will identify promising practices in case study schools and include short vignettes describing each practice. Prior to the report, the contractor will provide ED a report outline for approval. The contractor will also submit 10 site-specific reports on the compelling turnaround experiences.

The contractor will submit to ED a draft outline for the 10 site-specific reports on February 1, 2008, the first drafts on April 1, 2008, and the final reports on August 15, 2008. For the cross-site report, the contractor will submit to ED a summary of findings and arrange to brief ED on key findings no later than May 15, 2008. The contractor will submit a draft report to the COR by June 23, 2008. After receiving comments by ED, the contractor will submit the first revised report by August 4, 2008, the second revised report by September 1, 2008, the third revisions to the report by September 29, 2008, and the final report by October 27, 2008.

A2. Purpose and Uses of the Data

ED and other interested parties will use the data from the LACIO to assess the sustainability of student achievement outcomes from the comprehensive school reform provisions as stated in Sec. 1606 of the ESEA. It will also provide information that ED can use to strengthen schoolwide Title I programs by pointing to important processes and lessons learned from efforts to turnaround failing schools. This information is intended to be useful to state and local school systems, including individual schools, in their efforts to achieve Adequate Yearly Progress (AYP) under the No Child Left Behind Act.

A3. Use of Technology to Reduce Burden

The evaluation team will use technology in a variety of ways, especially to reduce burden on schools. First, the contractor has obtained the student achievement data for the sample from ED databases, which contain outcome measures for schools in almost every state. Having access to these secondary data allow researchers to reserve data collection efforts at the school and district for only the most necessary data elements.

Communication between the evaluation team and selected school and district officials will occur through email, fax, and conference calls to take advantage of information technology and reduce burdens associated with paperwork. The communication will cover initial inquiries, the exchange of preliminary information, and scheduling and planning of site visits.

Throughout the evaluation, the contractor will provide a toll-free number and email addresses to respondents allowing them to contact the evaluation team with any questions or requests for assistance.

A4. Efforts to Identify Duplication

The LACIO design is built upon the survey and research questions posed by previous research efforts, including the National Longitudinal Survey of Schools (NLSS), the Field-Focused Study, the National Study of Title I Schools (NSTS), and the Longitudinal Evaluation of the Effectiveness of School Interventions (LEESI). However, the LACIO is unique in that it combines elements of each study into a comprehensive data collection effort that allows for comparisons of successful and unsuccessful efforts to improve low-performing schools. LACIO researchers will also describe how the implementation of such efforts is linked to education reform and student achievement.

A5. Methods to Minimize Burden on Small Entities

The LACIO will collect data from few small entities, as most of the data sources will be school organizations (district and local). The few small entities are likely to be associated with the external consultants and community members who are helping a school implement its reform. Only minimal information will be needed from these small entities; therefore, no significant impact on these data sources is expected.

A6. Consequences of Not Collecting the Data

The revision of LACIO will provide ED with a complete picture of the implementation and results achieved by low-performing schools after several years of implementing turnaround strategies and the extent to which both reforms and student achievement gains were achieved and sustained in failing schools. In addition, the evaluation will outline the effectiveness of school reform in schools with diverse characteristics. Such answers are necessary to understand how federal and state funds can better serve as a stimulus for school reform.

The data collection efforts focusing on turnaround schools will allow researchers and policy makers to better understand and comment upon what appears to turn around schools that were failing the students they serve. The LACIO's combination of student achievement data and intensive field-based study comprises a design that builds on the strength of the combined methods.

A7. Special Circumstances

This information collection fully complies with 5 CFR 1320.5(d)(2).

A8. Federal Register Comments and Persons Consulted Outside the Agency

The 60 day Federal Register notice was published in the Federal Register on May 7, 2007.

To date, no public comments have been received.

The evaluation team will seek the expertise of persons outside the agency through the creation of a Technical Working Group (TWG). The TWG will advise the evaluation team on issues of school reform from the perspective of various stakeholders as well as methodological issues in evaluating CSR. We expect that the TWG will meet several times during the course of the study, with such meetings being tied to important events or tasks within the evaluation. Each TWG member will receive an honorarium of $750 per day. The time commitment is relatively small, but the TWG will play an important role in providing insight and guidance to support the evaluation. The TWG members are listed in Exhibit 2.

Exhibit 2

Members of the LACIO Technical Working Group

Member

Affiliation

Areas of Expertise

Carolyn Temple Adger

Center for Applied Linguistics

English Language Learners

Geoffrey Borman

University of Wisconsin- Madison

Quantitative methods; comprehensive school reform

H.J. Green

Executive Director
Office of Secondary School Innovation
San Diego Unified School District

District policy; school improvement

Bryan Hassel

Center for Improvement and Innovation

Research on organizational turnaround

Elsie Leak

North Carolina Department of Public Instruction

State policy; school improvement

Valerie Lee

University of Michigan

Quantitative methods; school reform

Paul Ruiz

Education Trust

State and national policy; school improvement; student assessment

Jean Rutherford

National Center for Educational Accountability

Accountability, assessment

Sam Stringfield

University of Louisville

School improvement; comprehensive school reform

Malik Stewart

Delaware Department of Education

State policy; school improvement

Ken Wong

Brown University

Research on organizational turnaround

All TWG members offer project specific expertise and experience. The contractor and ED selected these TWG members for their breadth of expertise across multiple disciplines including: methodology, statistical analysis, education context and special issues, and knowledge of CSR. Several of the TWG members have expertise in CSR, including: Elsie Leak, Valerie Lee, Geoffrey Borman, and Sam Stringfield. Elsie Leak gives us a first-hand perspective of school reform at the state level as the Associate State Superintendent for Curriculum and School Reform Services in the North Carolina Department of Public Instruction. Valerie Lee has conducted much research on school reform, including a current study on high school curriculum reform in Chicago. Further, in 2007, she published a book with Douglas D. Ready entitled Schools Within Schools: Possibilities and Pitfalls of High School Reform. Geoffrey Borman is also an avid researcher on comprehensive school reform issues. Most significantly in 2003, he conducted a meta-analysis on the impact of comprehensive school reform of student achievement entitled Comprehensive School Reform and Achievement: A meta-analysis. Last, Sam Stringfield is an accomplished researcher in the field of comprehensive school reform. He is the Co-Director of the Program on Systemic Supports for School Improvement, Center for Research on the Education of Students Placed At Risk (CRESPAR) and a senior scientist at the Center for Social Organization of Schools at Johns Hopkins University. He authored Choosing Success, which provides guidance to schools considering a CSR program based on specific program objectives and research evidence.

A9. Payment or Gifts

The enormous pressures on school systems, in part due to increased assessment and accountability requirements, lead to their assigning a lower priority for participating in data collection efforts such as the LACIO. To indicate the importance of the work of the evaluation for informing federal, state, and district policies and practices on turnaround and reform, schools participating in the LACIO will receive a special monetary payment. Past research shows such payments are a cost‑effective way of increasing participation rates substantially (e.g., Dillman, 1991).

Each school will receive an honorarium of $200 to be used for purposes such as the purchase of books for the school library. This amount was approved by OMB in the existing data collection.

A10. Assurance of Confidentiality

The evaluators will carefully handle all data in a responsible manner so they are not seen by or released to anyone not working on the project, except as required by law. For the cross-site report, the evaluation team will ensure all data are reported in a summary fashion, so no specific individual or school may be identified. For the individual case-study reports, WestEd will not use school or individual respondent’s names. Finally, the evaluation team will maintain all data in secure and protected files that do not include personally identifying data.

The evaluators will not collect any information that would identify individual participants. Therefore, the evaluation team will not reference participants by name. The contractor will communicate an explicit statement regarding these processes to protect the data to any and all participants. Similarly, the student achievement data extracted from ED databases are aggregate school-level data and do not contain records of individual students.


A11. Justification of Sensitive Questions

The contractor will not ask questions that are of a sensitive nature.

A12. Estimates of Hour Burden

A revision is being made to the existing collection to add a new evaluation with 1,200 burden hours. The existing data collection (with 10,774 burden hours) has been completed—with the exception of 10 burden hours that will be completed this fall. (We anticipate that as of December 2007, an 83C will be able to be completed to delete these 10 hours.) However, the 1,220 hours that we are requesting for this evaluation being added—plus the 10 burden hours that remain to be completed with the collection—total the 1,230 hours being requested.

Exhibit 3

Hour Burden for Additional Data Collection

Task

Number of Respondents

Number of Responses

Hour Burden

Field-Based Study

1,140

1,200

1,220

Existing Study

10

10

10

TOTAL

1,150

1,210

1,230

Sampling and Gaining Cooperation

Thirty sites will be identified for the field-based component of the study utilizing the procedures previously outlined. The contractor will initiate cooperation through a telephone call explaining the study.

Field-Based Study

The targeted sample of schools selected for the field-based study consists of 20 turnaround schools and 10 schools that have not been successful based upon their student achievement levels. Before the site visit, the contractor will request documents related to school improvement from the principal (estimated .25 hours for this task). The contractor will conduct individual interviews with the principal in the school during the turnaround period, the current principal (if different from the principal in the school during the turnaround period), assistant principal, one or two district officials responsible for the decisions and curriculum at the school, a school specialist or coach for mathematics and English/language arts (ELA), the math or ELA department chair (if applicable), and the guidance counselor. Exhibit 5 presents the burden associated with this data collection.

The evaluation team will also conduct focus group interviews of teachers, parents, community members, and the School Improvement Plan (SIP) or leadership team at each school site. Each focus group is expected to last one hour each. Teacher focus groups will include four sets of interviews: two focus groups with experienced teachers (at the school for five years or more) and two with less experienced teachers (at the school less than five years). Three to four teachers will participate in each focus group. For the other focus groups (e.g., parents, community members, leadership team), the contractor will ask for three to four participants with a longstanding relationship with the school (five or more years).

All site visit interview protocols are presented in the Appendix with a crosswalk that demonstrates the link between protocol questions for each instrument and the 11 CSR components.

Exhibit 4

Estimated Burden for Participants in Field-Based Study for Additional Data Collection

Individual Interview Respondent

Number of Respondents

Number of times they Respond

Number of Responses

Time Estimate
(Hour)

Total Hours

Hourly Rate

Estimated Cost of Burden

Experienced Principal Interview

45

1

45

1.25

66.25

$36

$2,025

Current Principal/Assistant Interview1

35

1

35

1.25

43.75

$36

$1,575

10

2

20

1

20

$36

$720

Experienced Teacher Focus Group2

220

1

220

1

220

$30

$6,600

20

2

40

1

40

$30

$600

New Teacher Focus Group3

220

1

220

1

220

$30

$6,600

20

2

40

1

40

$30

$600

Curriculum/Instructional Specialist Interview Protocol

60

1

60

1

60

$30

$1800

English/Language Arts/Mathematics Department Chair Interview

30

1

30

1

30

$30

$900

Parent Focus Group

120

1

120

1

120

$-

$0

Community Member Focus Group

120

1

120

1

120

$-

$0

District Official Interview4

25

1

25

1

25

$52

$1,300

5

2

10

1

10

$52

$520

District Curriculum Specialist

25

1

25

1

25

$52

$1.300

5

2

10

1

10

$52

$520

School Improvement Plan (SIP)/Leadership Team Focus Group

120

1

120

1

120

$30

$3,600

Guidance Counselor Interview Protocol

30

1

30

1

30

$33

$990

Document Review Checklist5

30

1

30

1

30

$-

$0

Existing Study

10

1

10

1

10

$36

$360

TOTAL

1150

23

1210

-

1230

-

$31,210

1 10 of the 45 Current Principal/Assistant Principal Interviews will be conducted during the follow-up site visit.

2 20 of the 240 New Experienced Teacher Focus Group Interviews will be conducted during the follow-up site visit.

3 20 of the 240 New Teacher Focus Group Interviews will be conducted during the follow-up site visit.

4 10 of the 60 District Official Interviews will be conducted during the follow-up site visit.

5 Document Review Checklist is for site visitor use only. The burden estimate of 1 hour reflects the time a school staff member will spend collecting the documents.

A13. Estimate of Cost Burden to Respondents

There are no annual costs associated with recordkeeping or data reporting other than that reported in section A12. All other data will be collected directly by the contractor from existing ED data sources.

A14. Estimate of Annual Cost to the Federal Government

The total cost for this part of the evaluation is $911,627. Since a three-year clearance has been requested for purposes of ROCIS, we are showing an annual cost of $303,875.

A15. Program Changes or Adjustment

There is a program chane of –9,544 burden hours. All but 10 hours of the burden of the current data collection are complete. The revision of the evaluation adds 1,220 burden hours to the 10 remaining hours from the current collection. When this total of 1,230 burden hours is taken from the current OMB inventory of 10,744 burden hours, there is a reduction of –9,544 burden hours.

A16. Plans for Tabulation and Publication of Results

The study will produce 10 case-specific reports and one cross-case report describing promising practices and policies of turnaround. The timeline for the publication of these findings is outlined in Exhibit 6. The analyses for each data collection method are detailed below.


Exhibit 6

Schedule for Dissemination of Study Results

Activity

Due Date

10 Site-Specific Reports


Draft outline

February 1, 2008

Revised outline

March 3, 2008

Draft reports

April 1, 2008

First revision of reports

June 2, 2008

Second revision of reports

July 1, 2008

Final reports

August 15, 2008

Cross-site Report


Draft outline

April 15, 2008

Revised outline and summary of initial findings

May 15, 2008

Draft of report

June 23, 2008

First revised report

August 4, 2008

Second revised report

September 1, 2008

Third revised report

September 29, 2008

Final report

October 27, 2008

Student Achievement Data

In this case, student achievement data are assessed prior to the selection of the turnaround schools. Schools with the highest levels of performance will be selected for study and compared to non-improving schools to identify factors associated with improved student achievement and those that did little to facilitate turnaround efforts. School level performance data will be gathered from the most recent ED databases. These databases contain achievement data from nearly every school in the country.

Because states use different tests, comparisons of student achievement are problematic. The variability in norms from different test publishers makes the comparison of absolute performance across states difficult. Neither the content nor the criteria for determining proficiency are directly comparable from state to state. Also standards, assessments, and proficiency criteria often change, making scores within states difficult to compare over time. Therefore, to select the appropriate schools accounting for inconsistencies over time, the contractor will rely on standardized school-level achievement scores computed from the NLSLSASD database. The contractor will standardized average scale scores and percent proficient measures by calculating z-scores within each year. In those instances in which percentile ranks are the only available achievement measure, the percentile ranks will be converted into normal curve equivalents (NCEs) and subsequently standardized.

Field-Based Component Analysis

The planned field-based study analysis will include both within- and across-school issues. ED will prepare reports in a common format to ensure all relevant elements of the conceptual framework that focus on CSR approaches, as augmented by concepts in the Turnaround Evidence Review and supplement, are captured.

The 10 site-specific reports will detail the individual school’s experience to illustrate the policy and processes within the complex school context. These field-based studies will integrate information from the documents reviewed, interviews, and focus groups held on site. As such, they can serve as stand-alone documents as well as the basis for the cross-site analysis.


A17. Approval to Not Display OMB Expiration Date

The contractor is not requesting an exemption from displaying the expiration date.

A18. Explanation of Exceptions

This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.

References

Aladjem, D. K., LeFloch, K. C., Zhang, Y., Kurki, A., Boyle, A. et. al. (2006). Models Matter—The final report of the National Longitudinal Evaluation of Comprehensive School Reform. Washington, DC: American Institutes for Research.

Berends, M., Chun, J., Schuyler, G., Stockley, S., & Briggs, R. J. (2002). Challenges of conflicting school reforms: Effects of New American Schools in a high-poverty district. Santa Monica, CA: RAND.

Berends, M., Kirby, S. N., Naftel, S., & McKelvey, C. (2001). Implementation and performance in New American Schools: Three years into scale-up. Santa Monica, CA: RAND.

Bickman, L. (1987). The functions of program theory. In L. Bickman (Ed.), New directions for program evaluation, No. 33 (pp. 5-17). San Francisco: Jossey-Bass.

Bodilly, S. J. (1996). Lessons from New American Schools Development Corporation’s demonstration phase. Santa Monica, CA: RAND.

Center for Innovation and Improvement (CII). (2007). Turnaround evidence review. Report submitted to the U.S. Department of Education on February 28, 2007.

Desimone, L. (2000). Making comprehensive school reform work. (Urban Diversity Series No. 112). East Lansing, MI: Clearinghouse on Urban Education, Institute for Urban and Minority Education. (ERIC Document Reproduction Service No. 1637)

Dillman, D.A. (1991). The Design and administration of mail survey. Annual Review of Sociology, 17: 225-249.

Elmore, R.F. (2000). Building a new structure for school leadership. Washington, DC: The Albert Shanker Institute.

Hallinger, P, Bickman, L. & Davis, K. (1996). School Context, Principal Leadership, and Student Reading Achievement. The Elementary School Journal. 96(5), 527-549.



Appendix

Exhibit 7- Crosswalk between Protocol Questions and the 11 CSR Components

Site Visit Interview Protocols

Experienced Principal Interview

Current Principal/Assistant Principal Interview

Experienced Teacher Focus Group

New Teacher Focus Group

Curriculum/Instructional Specialist Interview Protocol

English/Language Arts/Mathematics Department Chair Interview

Parent Focus Group

Community Member Focus Group

District Official Interview

District Curriculum Specialist

School Improvement Plan (SIP)/Leadership Team Focus Group

Guidance Counselor Interview Protocol

Document Review Checklist

Exhibit 7

Crosswalk between Protocol Questions and 11 CSR Components

11 CSR Components

Experienced Principal Interview

Current Principal/Assistant Principal Interview

Experienced Teacher Focus Group

New Teacher Focus Group

Curriculum/ Instructional Specialist Interview

ELA/ Mathematics Department Chair Interview

Proven methods

2, 6, 14

2, 11

2

2, 3

2, 6, 14

2, 4, 9

Comprehensive design

2, 14

2, 11

2

2

2, 4, 14

2, 9

Professional development

2, 10

9, 10

8, 12

2, 7

9, 10

6, 7

Measurable goals

5, 11

5, 7

9

9

3, 5, 11

8

Support from staff

3, 8

4

7

3

3, 8

5

Support for staff

9, 10, 12

9, 10

8, 11, 13, 14

6, 10-13

9, 10, 12

6, 7

Parent and community involvement

13

2, 4, 11

16

15

3, 13


External assistance

9

9

8

6

8, 9

6

Evaluation

5, 11

4, 7

5

9

6, 11

8

Coordination of resources

7

2, 11

12


7


Scientifically based research

2, 14

2, 11

2

2, 3

2, 14

2, 4, 9



11 CSR Components

Parent Focus Group

Community Member Focus Group

District Official Interview

District Curriculum Specialist

SIP/Leadership Team Focus Group

Guidance Counselor Interview

Proven methods



2

1, 7

2

3

Comprehensive design



2

1, 7

2

3

Professional development



2, 6

1, 7, 8

6, 7

5

Measurable goals



10

9

5

9

Support from staff



3


3, 4

4

Support for staff



6

5, 10

7

5

Parent and community involvement

1, 4-7

1-3, 6-7

3



8

External assistance



2, 6

1, 5, 8

7

5

Evaluation



8, 10

9

5


Coordination of resources



7

5

6


Scientifically based research



2

1, 7

2

3



1 The new legislation continues comprehensive school reform initiatives started in FY1998 and originally established through appropriations, not authorizing legislation, the Appropriations Act for the U.S. Department of Education, P.L. 105‑78.

2 Total support for comprehensive school reform activities consisted of this appropriation plus another $75 million from an account under the Fund for the Improvement of Education (FIE). Thus, the total amount of funds available to support the activities was actually $310 million.

3 The earlier appropriations, (covering both comprehensive school reform and FIE allocations (see previous footnote) and number of schools supported are as follows: FY1998: $145 million (about 1,800 schools); FY1999: $145 million (to continue 1,800 schools along with about 441 new awards); FY2000: $220 million (to continue about 2,241 schools and also to fund about 568 new awards); FY2001: $260 million (to continue all the previous schools and fund about 139 new awards); and FY2002: $310 million (to continue all the previous schools and fund an estimated 2,000 new awards).

LACIO OMB Clearance Request DRAFT page 19

File Typeapplication/msword
File TitlePART A
AuthorFlaherty
Last Modified ByDoED
File Modified2007-09-20
File Created2007-09-20

© 2024 OMB.report | Privacy Policy