Memorandum

Date	March 7, 2008
To	Bridget Dooling, OMB Gil Garcia, IES Amy Feldman, IES
From	REL-SE SIOP Study Team
Subject	Response to OMB Questions on 200709-1850-002, SIOP

The following are responses to the questions raised by OMB on Supporting Statement A for the REL-SE SIOP Study. We address each question separately below, and have revised Supporting Statement A accordingly. Page numbers where responses have been inserted into Statements A and B are noted at the end of the response to each question.

1. Pages 6 and 7 present a number of scenarios describing possible exposure to SI by teachers in schools.

1. Can you clarify when and how you will make the decision that previous exposure to SI was sufficient to exclude the school from the random assignment?

We will exclude schools from participation if SIOP training has already been provided to teachers in the targeted grade levels (grades 4 and 5). We will make this decision at the time of verbal communication with the school as a part of determining whether the school is eligible. Schools are eligible for the study if they (a) have students in grade 4 and 5, (b) meet the percentage ELL criterion (5% or more), and (c) have not provided all or most of their teachers in grades 4 and 5 with SIOP training. Eligibility decisions will be made as study team members contact potentially eligible schools to provide information about the study and what study participation would entail.

[Inserted in Statement A, Overview, page 7.]

2. Is it possible that a teacher previously trained in SI will be trained again ("double dosing")?

It is possible that a teacher previously trained in SIOP will be trained again (though, as with the schools that have previously had SIOP training, these teachers are extremely unlikely to choose to participate), and it is also possible that a teacher previously trained in SIOP will be in a school assigned to the control group. It is not a problem that a small number of individual teachers have prior SIOP experience, either in the control group as part of the counterfactual or in the treatment group as the soon-to-be-double-trained, if one is trying to learn how much difference school-wide or grade-level-wide launching of SIOP will make in today’s Georgia environment. Having some SIOP exposure in these settings already is desirable, not problematic, when doing a study that will guide policy decisions about expanding or eschewing SIOP in just such environments.

[Inserted in Statement A, Overview, page 7]

3. Is your assumption that schools where funding was previously made available for SI training will opt out?

As indicated in our response to 1a, schools that made funding available to provide SIOP training to most or all of their teachers in grades 4 and 5 are not eligible for the study. Therefore, they will be excluded from the study and not given the opportunity to choose to participate or opt out.

[Inserted in Statement A, Overview, page 7.]

2. You note that SI has not previously been studied rigorously. What ELL interventions were evaluated by IES for OELA?

According to Cynthia Ryan, Director of Discretionary Grants at OELA (personal communication, March 2008), IES has not evaluated interventions for OELA. However, below we detail studies that have been conducted, including those funded by or conducted by IES, that might be of interest to OELA.

The What Works Clearinghouse (WWC) Topic Report on English Language Learners reviewed twelve interventions (WWC, 2007), none of which specifically looked at Sheltered Instruction. Many of the interventions themselves focus on elements included in SIOP, such as peer/group work (Instructional Conversations and Literature Logs, Peer Tutoring and Response Groups, and Peer-Assisted Learning Strategies) and vocabulary development (Vocabulary Improvement Program), but there has been no evaluation of a “sheltered instruction” intervention.

Of the articles comprising the What Works for English Language Learners (WWELL) bibliographic database (National Clearinghouse for English Language Acquisition and Language Instruction Educational Programs, 2008), the majority of studies were not related to sheltered instruction and ELLs. Of the six on approaches to instructing ELLs, the focus was on other approaches, such as bilingual education, English only, two-way immersion, English immersion, and transitional bilingual education, and not on any specifically operationalized model of sheltered instruction. The WWELL also included a Slavin and Cheung (2005) synthesis of 17 studies comparing bilingual and English only reading programs for ELLs that did not include SIOP specifically.

A search of the Research of the Effective Education of English Language Learners (REEELL) database (National Clearinghouse for English Language Acquisition and Language Instruction Educational Programs, 2008) yielded 35 studies funded by IES. Again, the majority had no substantive relation to SI and ELLs. Five pertained to reading development and ELLs, two to assessment accommodations, two to teacher quality, four to overall research in the field, one to adult ESOL, three on Success for All, and one on 21^st Century Community Learning Centers. Four were potentially relevant to the SIOP Study and required further examination.

The first two studies (Optimizing Education Outcomes for ELLs—David Francis (PI), and Project English Language Literacy Acquisition—Raphael Lara-Alecio (PI)) appear to be connected to Success through Academic Intervention in Language and Literacy (SAILL) (Francis, D., Carlson, C.D., Rivera, H., Short, D., Vaughn, S.R., and Linan-Thompson, S.), which we discuss in Section A.1 of Statement A, due to its inclusion of SIOP for part of the intervention. The third (Oracy/Literacy Development in Spanish-speaking Children—Vaughn, et al.) examines Spanish- and English-language reading interventions, while the final study (The Big-Picture—Kellie Rolstad) is a meta-analysis of bilingual and all-English programs. While some of the studies may contain elements of SIOP, only the SAILL study (described in Section A.1) examines sheltered instruction or SIOP specifically.

[Inserted in Statement A, item 1, pages 9-10.]

3. On page 9 you suggest that one of the ongoing SI studies will not meet the WWC criteria because it focuses on 7th grade students. Can you clarify what criteria for inclusion it will not meet?

The Impact of the SIOP Model on Middle School Science Language Learning study may indeed meet the WWC criteria of rigor. It would not have met the criterion for inclusion in the current WWC ELL Topic Report because it did not focus on the grades targeted in that report (elementary grades, K-6).

[Inserted in Statement A, item 1, page 9.]

4. On page 11-12, you describe your plan for classroom observation. You propose to use the SPC.

   1. To what extent are the teacher practices emphasized in the SPC those included in SI and SI training?

The SPC was developed to measure the five standards of effective pedagogy for diverse learners set out by Tharp and his colleagues (Tharp, Estrada, Dalton, and Yamauchi, 2000) within a socio-cultural constructivist theoretical framework. These are the same standards that formed the basis of the SIOP model. While the SIOP protocol delineates particular elements of the sheltered instruction model, the SPC focuses on the five underlying standards. Thus the SPC includes elements that we would expect to see in a highly implemented SIOP classroom as well as other aspects of instruction believed to be effective for diverse learners, including English Language Learners, that do not follow directly from SIOP. Appendix J shows a crosswalk of this overlap between the two instruments.

[Inserted in Statement A, item 2, page 13.]

2. While the SPC may be a good measure of general pedagogical skill, is it sufficiently sensitive to pick up changes in behavior as a result of the intervention?

A validation study of the SPC (Doherty, Hilberg, Epaloose, & Tharp, 2002) provided evidence of inter-rater reliability, concurrent validity, and criterion-related validity supporting the validity of interpretations of data gathered with the SPC. This study found that in a school serving predominantly poor Latino English Language Learners, higher SPC total scores were associated reliably with greater student achievement on end-of-year standardized tests (in this case, the SAT-9). We do not have strong evidence that the instrument will be sensitive to the kinds of changes that the professional development intervention might produce, but at this time, there is not an instrument with demonstrated sensitivity that is well matched to our research question.

[Inserted in Statement A, item 2, page 13.]

3. In addition, you plan to use the SI Observation Protocol to document fidelity of implementation. Why did you choose to use a checklist?

While the SIOP protocol appears to be a checklist in some respects, it is a 5-point rubric. It measures the 30 features of the eight components of instruction targeted by the SIOP professional development intervention. It is the only existing measure of fidelity of implementation and is the measure that is used in the field by the developers, trainers, peer coaches, and supervisors.

[Inserted in Statement A, item 2, page 14.]

4. Could you have used a protocol with established psychometric properties that could have been used in the statistical models?

We are intending to use the SIOP protocol for observation in treatment classrooms to document the extent of fidelity of implementation to the SIOP model as a descriptive analysis and in control classrooms to document any circumstantial contamination of similar practice elements, again, descriptively. We were not intending to use these measurements in statistical models of intervention impacts or student outcomes.

[Inserted in Statement A, item 2, page 14.]

Web-based sources referenced:

What Works Clearinghouse. (2007). Topic Report English Language Learners. Retrieved, March 6, 2008, from http://ies.ed.gov/ncee/WWC/reports/topic.aspx?tid=10.

What Works for English Language Learners. (2008). Retrieved, March 6, 2008, from http://www.ncela.gwu.edu/spotlight/4_whatworks.html.

National Clearinghouse for English Language Acquisition and Language Instruction Educational Programs. (2008). What Works For English Language Learners NCELA Bibliographic Database. Retrieved, March 6, 2008, from http://www.ncela.gwu.edu/rcd/search/.

National Clearinghouse for English Language Acquisition and Language Instruction Educational Programs. (2008). Research on the Effective Education of English Language Learners (REEELL) Database. Retrieved, March 6, 2008, from http://www.ncela.gwu.edu/databases/reeell/.

4

R

egional Educational Laboratory