12-3-07 memo

OMB Questions of November 21, 2007

OMB IC # 200709-1850-003, Early Literacy

1. Can you provide more detailed information about the intervention being studied (OWL), including information on its components/implementation and on how frequently it is currently being implemented and where? How is this intervention similar to or different from other early childhood interventions that IES or other entities have evaluated?

Opening the World of Learning (OWL) is a research-based preschool curriculum developed by Judith Schickedanz, Ph.D., and David Dickinson, Ed.D., in collaboration with Charlotte-Mecklenberg Schools. The curriculum package provides materials and guidance for implementing six thematically-organized units: Family, Friends, Wind and Water, The World of Color, Shadows and Reflections, and Things that Grow. The comprehensive curriculum is designed to support all aspects of development, including literacy, self-regulation and social development, through an activity-based daily schedule. The curriculum is comprised of the following seven components: (a) Morning Meeting, (b) Center Time, (c) Story Time, (d) Outdoor Play, (e) Songs, Word Play and Letters, (f) Small Group, and (g) Let’s Find Out/Talk About It. The curriculum focuses on the development of language and literacy skills by encouraging meaningful teacher-student verbal interactions related to daily lessons. Program elements are based on research that suggests that language development is correlated with the quantity and quality of conversations with adults, particularly daily interactions with teachers (Dickinson, 2001; McCartney, 1984). Phonological awareness and letter knowledge are key literacy skills that also are integrated with language development into the curriculum.

The authors of the curriculum have correlated the OWL Scope of Skills and Activities to the language and literacy Developmental Guidelines of the Work Sampling System Preschool-4. These guidelines are based on national, state, and local curriculum standards (see Meisels, 1997). The document is provided as an attachment.

Since 2003, OWL has been used in over 150 school districts in 39 states, with a heavier concentration in the south and central regions of the country. In 2005 and 2006 it was the research-based curriculum used in 5 Early Reading First grants. In 2007, OWL is the curriculum used in 8 of the 32 Early Reading First grants awarded across the nation.

This intervention complements the other early childhood interventions that IES has evaluated.

The Early Reading First evaluation conducted by NCEE used a quasi-experimental design with 3-5 year olds evaluating 2 curricula, High/Scope (Educating Young Children) and Creative Curriculum. The study found that the program had a positive impact on children's print and letter knowledge, but not on phonological awareness or oral language. The program had positive impacts on aspects of the classroom environment and teacher practices that are intended to support the development of language and literacy skills.

The Even Start Classroom Literacy Interventions and Outcomes (CLIO) Study currently being conducted by NCEE is assessing children's early literacy skills and parent's literacy and parenting behavior.

The OWL's predecessor, the Bright Beginnings Curriculum, was evaluated as part of NCER's Preschool Curriculum Evaluation Research (PCER) project. Individual researchers evaluated the curriculum at their own sites and then a cross-site analysis was conducted for NCER. The report for the cross-site PCER evaluation is in peer review. Bright Beginnings was an educational model as opposed to the earlier developmental models like Creative Curriculum mentioned above. This study examined children’s behavior in the classroom and their achievement test scores (on the Woodcock Johnson III and the PPVT).

This current study adds to the evidence base by focusing on the same or similar components of early reading and a preschool sample as the other IES studies, but is different with its more rigorous design than Early Reading First (an experimental randomized control trial design), its focus on teacher-child interactions in improving children’s literacy as opposed to CLIO, and its focus on the latest curriculum as opposed to PCER.

2. Education Innovations is identified in the consent form as the conductor of this study. The University of Memphis's Center for Research in Educational Policy is the entity that sought IRB approval. What are the roles of the various organizations in this study and how do they relate to the REL?

Education Innovations, LLC (EI) is the management entity for the study. EI subcontracted with The Center for Research in Educational Policy at The University of Memphis to provide staff observation and questionnaire tools as well as assessment expertise to conduct the study. Since those conducting the study are university staff, it appeared most appropriate to be in compliance with the University of Memphis’ IRB requirements. The University of Memphis IRB members also have extensive knowledge and understanding of the local school districts and their operation. Therefore, IRB approval was sought and received from The University of Memphis.

3. Please explain further what this means: "The schools agreed to random assignment on request of the district, independent of (but in consonance with) the needs of the randomized study" (Part B, page 3). How were schools approached, and how was it made clear that they were agreeing to something that was independent of but in consonance with something else?

It is agreed that the language in this statement is ambiguous. The intent was to indicate that the school administrative staffs were aware of the study and agreed to support it. Agreement to the randomization of the selection of intervention and control schools was based on administrators’ knowledge that the school district did not have the resources to provide the OWL program to all preschool sites simultaneously and that all sites were assured of receiving the program at a future time. District funds provide for implementation of the OWL curriculum in 25 new classrooms per year.

4. Page 7 of Part A says that the REL will also collect data for the follow-up assessment of children in cohort 1. Please clarify whether approval is also being sought for this activity, or whether that will be submitted as a separate ICR.

The initial design included follow-up of cohort 1 in kindergarten and first grade. The late start-up of the intervention (due to the significant influx of new pre-K classrooms and consequent late ordering of materials) led to a decision by the school district with guidance from the REL to revise the design so as not to proceed with follow-up for this cohort. The delay made full implementation of the curriculum unlikely. This circumstance made examination of a second cohort in the same schools preferable. Limited funding for the study was felt to be best applied to follow-up of a second cohort.

5. Relatedly, will the REL be taking over any functions that would-in the absence of the evaluation-be completed by the district? If so, why?

For cohort 2, the REL will be taking over the responsibility of the teacher and paraprofessional questionnaires. They would not normally be completed by the school district because they are not required by the Tennessee Department of Education, as are other assessments, such as classroom observations, parent survey, and the Peabody Picture Vocabulary Test. We need to collect the questionnaires because they provide important data for the study.

The REL is taking over the observation and assessment functions of the district by administering the ELLCO observations and the PPVT – III assessment. The district’s standard practices of using (a) the classroom teachers to test their own students on the PPVT and (b) literacy staff to conduct the observations would not meet the more rigorous IES requirements of scientifically-based research. The REL’s assumption of these functions avoids the risks of data collectors’ potential bias and lack of sufficient expertise needed for the RCT.

6. Is this a REL-sponsored study under contract with ED and if so why is only a small piece of it being submitted for OMB approval?  Did the management of the study change mid-way through?

The bulk of the study involves academic achievement testing and classroom observations, neither of which is considered burden by IES. Another component of the study is the administration of questionnaires to teachers and paraprofessionals. The OMB package was submitted for Year 2 because the responsibility of the administration of the questionnaires will be assumed by the REL. The Memphis City Schools initiated this study with technical assistance from the REL. The school district administered the questionnaires in accordance with the initial plan and had made no commitment to administer them for a second cohort. For the second cohort of the study, this component is being undertaken by the REL; hence, it is being submitted to OMB. EI and CREP will now assume a larger role in that we will carry out a second year of the study and collect questionnaire data from teachers and paraprofessionals. No change in management has occurred.

7. If the REL is only involved in a small part of the study, will it meet the strict standards required for inclusion in the WWC?  If not, why is the REL participating under its ED contract?

As described in responses to prior questions (particularly #’s 5 and 6), the involvement of the REL will be substantial in implementing cohort 2, encompassing all achievement testing, observations, surveys, and associated data analysis and reporting. The district has been substantially guided by the REL in Year 1, and we followed IES and WWC standards in every aspect of the study’s design, implementation, data collection, reliability checks, data analysis, and reporting. The study will clearly meet the standards of the WWC.

8. What confidentiality assurances, if any, did MSD offer to respondents when recruiting for the study? Was the REL involved in those decisions?  What role, if any, does the REL have in ensuring the confidentiality of information collected prior to the portion of the study for which is it seeking approval?

The Memphis City Schools had already provided its own clearly-stated assurances of confidentiality to parents when recruiting children for the study. With guidance from the REL, the district provided clearly-stated assurances of confidentiality to respondents when recruiting teachers and paraprofessionals for the study. These staff assurances were developed and reviewed jointly by the school district and the REL with the IES confidentiality and data protection rules applied, and they were provided to letters describing the study and requesting consent for participation. We have had an important role in ensuring the confidentiality of information collected prior to the portion of the study for which is the REL is now seeking approval. The REL sought and received written approval to conduct the study by the Memphis City Schools. Assurances of confidentiality were required for that approval. The REL similarly sought and received IRB approval, which required strict confidentiality assurances.

9. Is the second cohort strictly of students within the already identified schools and classrooms?  What is the purpose of a second cohort?  What is the size of the cohort?  Who will be obtaining consent for those students' participation?

The second cohort is strictly of students within the already identified schools. In Year 2, a second cohort was added to the study at the invitation of the school district. Typically, more than one year is required for teachers to fully implement a new curriculum. The REL considered advantageous the opportunity to evaluate a second cohort. The first cohort of students was comprised of 50 classrooms in 41 schools. From the original 41 schools, 3 schools closed or relocated their preschool classrooms at the close of the 2006-2007 school year, leaving 38 schools to participate in the study in the 2007-2008 school year. Additionally, the school district added 15 classrooms to their schools resulting in 12 more classrooms located at the 38 participating school sites. A collaborative decision between the district and the REL was reached to include these classrooms in the second cohort of the present study since the project was originally designed to include all preschool classrooms within each school and randomization at the school level. A total of 32 treatment and 27 control classrooms comprise cohort 2. The REL will assess approximately 1087 students. Permission forms were sent to parents by the school district, as with cohort 1. Parents returned forms if they chose to decline their children’s participation in the study.

10. Please provide the parental consent form used to gain children's participation in the study as well as all data collection instructions used in the first round of data collection.

Please see attached.

11. Has the REL or some other entity sought an extension to IRB approval, due to expire in January?

Application for an extension to the IRB approval was made to The University of Memphis on October 29, 2007. The approval was received on November 27, 2007 (see attached copy).

12. REL studies need to cite ESRA (not the Privacy Act) as their primary confidentiality statute and cite the statute somewhere on the collection instruments.

The following statement will be substituted for the current confidentiality statement on the questionnaires for teachers and paraprofessionals (see appendices).

Per the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, “Responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific district or individual. We will not provide information that identifies you or your district to anyone outside the study team, except as required by law. Any willful disclosure of such information for nonstatistical purposes, without the informed consent of the respondent, is a class E felony.”

13. The package seems to indicate that there will be at least 4 assessments of individual students. Are these assessments tests that the students will be taking anyway (e.g. as part of state requirements)? What is the burden entailed for children if they are assessments specific to this evaluation?

The four individual assessments are the Peabody Picture Vocabulary Test –Third Edition (PPVT - III), Phonological Awareness Literacy Screening for Preschoolers (PALS), Dynamic Indicators of Basic Early Literacy Skills (DIBELS) and the Woodcock-Johnson Tests of Achievement – Third Edition (WJ - III ACH). Only the PPVT- III is required by the state. The PALS and DIBELS are required by the school district. Only the WJ - III ACH is specific to this evaluation. Administration of the WJ - III ACH is important because no achievement tests are administered by the school district until Grade 3. The WJ - III ACH was selected for use by the district and the REL. The WJ-III ACH is individually administered to all participating students by trained EI site researchers in the fall and spring of the kindergarten year and the fall of the first grade year. The burden entailed for the WJ - III ACH is 30 minutes per child (four sub-tests). Attrition from fall to spring of each year is estimated to be 15 percent. The table shown below details the burden entailed for cohort 2.

Administration	Students	Administration Time	Burden
Fall 2008 Kindergarten	924	30 minutes (.5 hour)	27,720 minutes (460 hours)
Spring 2009 Kindergarten	785	30 minutes (.5 hour)	23,550 minutes (392.5 hours)
Fall 2009 First Grade	785	30 minutes (.5 hour)	23,550 minutes (392.5 hours)

6