School site visits to obtain a more detailed look at teacher and school leader practice will comprise our case study data collection efforts. For the case study sample, SRI will rely on a purposive approach to sampling. By focusing our fieldwork on schools where we know that many teachers are actively looking at student data, we greatly increase the likelihood of seeing effects of data use on practice, compared with a sample of schools drawn at random. During the spring of 2006, project staff engaged in an in-depth process of selecting appropriate schools and districts for site visits in the fall of the 2006-07 school year. School sites will be identified through three methods:
Polling of TWG members and other leaders in educational technology, researchers, vendors, and staff of professional associations such as the Consortium for School Networking (CoSN) and the State Education Technology Directors Association (SETDA), supplemented by a search of conventionally and Web-published literature on data-driven decision making, to identify schools and districts with reputations as leaders in this area.
Phone interviews with selected districts active in data-driven decision making during which districts will be asked to identify both their most exemplary school in terms of data use and a school that typifies data use at the school level within their district.
Self-nomination of schools at the 2006 National Educational Computing Conference (NECC) that brings together teachers, technology coordinators, administrators, policy makers and industry representatives to discuss education technology issues.
The list of nominated districts was then grouped by type of commercial data system used and systems developed locally. The plan is to select three locally developed systems and one district each using seven frequently used commercial systems (e.g., SchoolNet, eScholar). Publicly available data will also be reviewed to characterize nominated districts in terms of the demographics of their student bodies and in terms of their students’ achievement trends (i.e., we will not select any district that has declining achievement). Geographic diversity will also be sought across the 10 districts.
For each of the 10 districts selected for study, the district representative will be contacted by phone to gain recommendations for three site visit schools at either elementary or middle schools based on the following criteria:
One school that the district considers exemplary in its data use policies.
One school that has shown dramatic improvement in its use of data to improve instruction and student outcomes.
One school that is typical of the district with respect to use of data systems.
To the extent possible, each district will be asked to recommend schools that serve demographically similar student populations at the same grade levels. Priority will also be given to schools serving large numbers of low-income student and schools that have experienced improved student achievement.
Once the information concerning potential districts and schools falling into these categories has been collected, project staff will synthesize it and make phone contact with appropriate district and school staff as needed to confirm contact information and the extent to which these practices are being implemented. SRI then will provide the Department with site nominations, with a brief statement of the nomination source; what is known about the site, including demographic and academic achievement data; and the categorization of the site’s data-driven decision-making effort. SRI will provide the same kinds of information for a set of alternate sites that will serve as a back-up to the initial group in the event that some schools decline to participate. In nominating sites for case study, we will give priority to sites serving large numbers of low-income students and will seek geographic and demographic diversity across the 15 sites. We will also seek sites that have not experienced improved major declines in student achievement.
Teachers to be interviewed will be nested within the purposive sample of 15 case study schools. Even though schools will be selected because they use student data systems in instructional decision making, weWe expect considerable variability across teachers within a school with respect to how they are using data for instructional decision making. Research on schoolwide reform has shown that even when significant investments are made in reforms, variability within the school in implementation levels is high, and data-driven decision making is not likely to be an exception to this pattern. Therefore, we will request that the principal of each case study school nominate four to six active practitioners of data-driven decision making and four to six teachers who represent average use. In this way, we should be able to capture “best practices”the practices of the strongest data users within the school but still maintain a realistic perspective with regard to the pervasiveness of those practices. For middle schools, we will ask the principal to make nominations from the pool of English language arts/reading and mathematics teachers at the school.
As described in the first section of this document, the case studies are a component of an interrelated data collection plan that also includes a survey of a nationally representative sample of districts and review of secondary sources that address the same set of evaluation questions. SRI will conduct the data collection activities for which clearance is being sought according to the schedule shown in Exhibit 7. The timeline of our data collection activities is designed to inform subsequent collection activities. The case studies assist in the development of the district survey that will be submitted in October as a separate OMB package.
Exhibit
7
General Timeline of Data Collection Activities
|
Conduct Case Studies |
|
Fall 2006 |
|
|
Spring 2007 |
|
|
The first major data collection activity for this study will be the case studies in 30 schools located in 10 school districts. The case studies will provide both an opportunity to better understand the conditions and practices surrounding effective data-driven decision making and to identify key issues and generate hypotheses that can be tested more systematically through the district survey.
School site visits will be conducted in winter and spring of the 2006-07 school year. As noted above, in addition to informing the development of the district survey, the case studies will provide a much more in-depth understanding of how school staff use data systems and of the influence that their interpretations of data are having on instruction. For example, we will discuss with teachers the specific data they use from the data management systems available to them and the purposes for which they use data. We will also address how these data are interpreted and used to make student placement decisions and changes in classroom practice. Through the use of specific examples, we will gain a better understanding of the nature and quality of school-level data-driven decision making and the conditions and practices that facilitate or inhibit it. Topics to be explored in more depth with teachers and school leaders include those proposed in Exhibit 8.
Exhibit
8
Case Study Topics
Topic |
Subtopic |
Evaluation Question Addressed |
DDDM tools |
|
Q2: How prevalent are tools for generating and acting on data? |
Supports for DDDM |
|
Q3: How prevalent are state and district supports for school use of data systems to inform instruction? |
DDDM at the school level |
|
Q4: How are school staff using data systems? |
Using the data |
|
Q5: How does school staffs’ use of data systems influence instruction?
|
Focus groups and individual interviews with principals and teachers will support more in-depth investigation of questions about data-driven decision making. In addition, we will attempt to arrange individual teacher interviews in a location where the interviewee can show the system to the site visitor and demonstrate the types of data that he or she has examined in the past. Principals and teachers will be asked to describe their interpretation of the data they show the site visitor and to describe what decisions, if any, they would be likely to make on the basis of the data. They will be asked also to describe specific examples of how they have used data to make decisions that affected instruction. In addition, to provide comparable data across teachers, we plan to develop some mock-ups of hypothetical student data to present to teachers. Teachers will be asked to describe what the data tell them and how, if at all, they would change their teaching practices in light of such data. These mock-ups of hypothetical student data will be designed to resemble typical data representations in widely implemented systems and could be varied in the types of data displayed (e.g., one-time versus longitudinal data, state assessments versus school-administered tests). The data mock-ups will be designed in collaboration with TWG members who have developed similar materials.
Project-wide training for site visitors, including role play with the interview and focus group protocols, will be conducted in advance of the start of school site visits in fall 2006. Two site visitors will visit each school for two days. In most cases, one site visitor will conduct the focus group or interview while the other site visitor concentrates on recording the interview, with site visitors switching roles across respondents to avoid excessive fatigue. Focus groups and interviews are also audio taped as a back-up if additional information is required such as particular respondent quotes. For those schools that are implementing data-driven decision making as part of a district-initiated and supported effort, interviews with district staff will be scheduled as part of the site visit.
The use of secondary data sources will enhance our analysis and avoid duplication of efforts. We have currently identified two main sources of additional data on how states, districts, and schools are using data systems. Further collaboration with the U.S. Department of Education and the TWG may result in the identification of additional data sources available from other ongoing activities and studies on the topics of data management and DDDM.
The first source of data for secondary analysis is the NETTS study which is focusing on the implementation of the Enhancing Education Through Technology (EETT) program at the state and local levels. A teacher survey was completed through NETTS in January 2005 that gathered data from over 5,000 teachers in approximately 850 districts nationwide. Teachers were asked about their use of technology in the classroom, including the use of technology-supported databases. Questions about data systems addressed issues related to the accessibility of an electronic data management system with student-level data, the source of the system (state, district, school), the kinds of data and supports provided to teachers to access data from the system, and the frequency with which teachers use data to carry out specific educational activities, and the types of supports available to teachers to help them use student data. A copy of the relevant NETTS teacher survey items is included in Appendix B.
The second major source of data for secondary analysis is the National Center for Educational Accountability (NCEA) state survey, first administered in August 2005, which focused on data system issues related to longitudinal data analysis. The second administration of the NCEA state survey is expected to be completed by the end of September 2006 which updates data from the 2005 survey and adds some new items as well. The NCEA state survey will continue to be used as a secondary data resource for this study in the future. NCEA data provide key information on the data systems that states are building and maintaining as they gear up to meet NCLB requirements for longitudinal data systems (i.e., NCEA’s “ten essential elements”).1
Gaining the cooperation of district representatives, principals, and teachers is a formidable task in large-scale data collection efforts. Increasingly, schools are beset with requests for information, and many have become reluctant to participate. Our efforts will be guided by three key strategies to ensure adequate participation: (1) an introductory letter signed by the Department, (2) preparation of high-quality informational materials, and (3) personal contacts with nominated sites.
U.S. Department of Education Letter. A letter from the Department will be prepared that describes the purpose, objectives, and importance of the study (i.e., documenting the prevalence of data-driven decision making, identifying best implementation practices for effective data-driven decision makingaround the use of data in making instructional decisions, identifying challenges to implementation) as well as the steps that will be taken to minimize respondent burden.
High-Quality Informational Materials. Preparing relevant, easily accessible, and persuasive informational materials is critical to gaining cooperation. The primary component of the project’s informational materials will be a tri-fold brochure. This brochure will include the following information:
The study’s purpose
Information about the design of the sample and the schedule for data collection
The organizations involved in designing and conducting the study
A draft copy of the brochure is included in Appendix C. All informational materials will be submitted to ED for approval before they are mailed. Mailing of informational materials will begin in fall 2006, given OMB approval.
Contacting Districts and Schools. As noted earlier, initial contact with districts and schools will be made via phone to confirm contact information obtained during the site selection process and to verify the extent to which the data-driven decision making practices for which the school was nominated are being implemented. Once the final selection of case study schools is made by Department staff, SRI staff will follow up with contacts in the district and schools to determine their interest in participating as a case study site. SRI staff will describe the study’s purpose and why their district or school was selected to be a part of the case study sample. Any questions that respondents might have will be answered. If required, project staff will be prepared to submit proposals to district research committees. The initial phone call will be followed up with the Department letter described above, along with a copy of the study brochure. In the five districts where only school staff will be interviewed, we will send an informational letter to the Superintendent letting him or her know that one of the district’s schools will be asked to participate in the study. Draft copies of the notification letters are included in Appendix B.
We are selecting districts and schools because of their exemplary practices related to data-driven decision making, an approach that makes participation attractive to sites. In describing the data collection activities involved in the case study visits, every effort will also be made to minimize the burden on the districts and schools. Past experience has shown that working with district and school staff to schedule site visits during time periods that are convenient for them facilitates data collection activities. (Methods to maximize response rates for the district survey will be described in the subsequent OMB package.)
To improve the quality of data collection instruments and control the burden on respondents, all instruments will be pre-tested. Pre-testing of the site visit protocols will begin at the National Educational Computing Conference (NECC) in July 2006, where we will conduct focus groups of nine or fewer respondents to evaluate different sections of the protocols; therefore, prior approval from OMB will not be required. For example, one focus group will be asked about organizational supports for data-driven decision making while another group will be asked about district data usage. Focus group respondents will include district and school staff attending the conference who volunteer to participate in a focus group. These focus groups will provide an opportunity to assess how well the protocols elicit the desired information and any important omissions. The results of the pre-testing will be incorporated into revised instruments that will become part of the final OMB clearance package. If needed, the revised protocols will be piloted in a small set of local districts and schools with nine or fewer respondents prior to data collection.
The protocols will standardize data collection efforts across each site while being sufficiently open ended to allow the site visitor to customize them to the circumstances of individual sites. The site visit protocols can be found in Appendix D.
Dr. Barbara Means is the Project Director for the study. Her mailing address is SRI International, 333 Ravenswood Avenue, Menlo Park, CA 94025. Dr. Means can also be reached at 650-859-4004 or via e-mail at [email protected].
Christine Padilla is the Deputy Director for the study. Her mailing address is SRI International, 333 Ravenswood Avenue, Menlo Park, CA 94025. Ms. Padilla can also be reached at 650-859-3908 or via e-mail at [email protected].
1 Creating a longitudinal data system that will provide data to improve student achievement is one of the major goals of the Data Quality Campaign. The campaign is a national, collaborative effort to encourage and support state policymakers to improve the collection, availability and use of high-quality education data.
| File Type | application/msword |
| Author | Nicolette Mayes |
| Last Modified By | DoED |
| File Modified | 2006-10-18 |
| File Created | 2006-10-17 |