Vermont Classroom Observation Tool As Applies to Project Criss

APPENDIX B

VERMONT CLASSROOM OBSERVATION TOOL

AS APPLIED TO PROJECT CRISS

NWREL Experimental Study of Project CRISS

Appendix B

Vermont Classroom Observation Tool as Applied to Project CRISS

(VCOT-CRISS)

The Vermont Classroom Observation Tool (VCOT) is modeled after work done by the Science and Math Program Improvement (SAMPI), a research organization at Western Michigan University (1999), and Horizon, Inc (1997)., an NSF funded research group. The current tool was refined in collaboration with the Education Development Center and Vita-Learn. The observation tool can be used K-16 in any curricular area. The purpose of the VCOT is to provide data to educators to make program decisions and to monitor professional development needs. While it does generate information that might be useful for teachers in assessing the effectiveness of classroom practice, it is not meant as a teacher evaluation tool. The VCOT is part of a broader assessment process that includes pre and post observation interviews. The VCOT contains between 18-20 items around four primary criteria. Table B-1 shows the four criteria and how they relate to CRISS learning principles and teacher strategies.

In Classroom Observation Protocols: Potential Tools for Measuring the Impact of Technology in the Classroom, Dirr (2006) identifies seventeen observation protocols that have promise as reliable and valid classroom measures of classroom practice. The VCOT and its parent protocols have been listed as measures that:

• Have been carefully researched and developed using systematic methodology

• Can be applied in varied settings

• Provide normative data and evidence of reliability and validity

• Can be presented in enough detail and clarity to permit replication

The protocol has three major categories: implementation, content knowledge, and classroom culture. Each category has a set of indicators that are evaluated using a 5 point Likert-type scale. Observations are done in full classroom periods.

The VCOT is a constructivist based tool which makes it an appropriate choice for identifying changes in teacher practices that may occur as a result of the CRISS intervention. The CRISS principles and philosophy are grounded in the importance of developing metacognitive learners, teaching for in-depth understanding, and explanation and modeling with gradual release of responsibility to students. For this study, the VCOT developer/trainer modified the instrument using the key principles of CRISS. The indicators not only pertain to general learning best practices but emphasize the importance of integrating reading and writing best practices into content areas. These indicators were honed further after the initial VCOT five-day training (September 2006) provided by Dr. Nicole Saginor (formerly with Vermont Institutes and now a private consultant) to three senior CRISS study researchers, led by Dr. Maureen Carr of NWREL. We are calling this revised instrument the VCOT-CRISS.

Table B-1

VCOT Dimensions and Relation to Project CRISS

I. Planning/organization of the lesson

This section is concerned with the quality of the planning for and organization and structure of the lesson, NOT a written lesson plan.

CRISS [Content standards, purpose setting, assessments, preparing for understanding]

• Reading and content standards for unit/lesson identified

• Reading strategies ( and content material) for instruction identified and appropriate texts selected for study/practice

• Routines identified to facilitate discussion/guided practice/peer interaction/reflection/assessment.

• Classroom space and class schedule set to optimize physical qualities of the learning environment.

II. Implementation of the lesson

This section focuses on the effectiveness of instruction and learning that occur during the lesson. Observations of both teacher and student activity are noted.

CRISS [integrating strategies for understanding, checking for understanding, student engagement/involvement]

• Overarching purpose of the lesson is discussed with students

• Teachers and students engage in activities to activate useful prior knowledge to connect to the reading

• Teachers explain and model specific reading strategies to apply to current text

• Teachers engage students in guided practice of reading strategies

• Students work in pairs or small groups to discuss and practice reading strategies with a variety of texts

• Students complete product (e. g. word map, two column notes, summary, etc.) and discuss in small groups and in whole group

• Students discuss their understanding of the strategies in pairs and in teacher/student conferences

III. Content of the lesson

This section notes the accuracy, importance, level of abstraction, connections to other concepts of the content.

CRISS [active engagement, reorganizing information, discussion, metacognition, background knowledge]

• Teachers stress the importance of using reading strategies to optimize student understanding of content concepts and principles

• Teachers observe student application of reading strategies and provide corrective feedback

• Teachers use probing questions to assist students to determine the effectiveness of strategies for understanding the text

IV. Classroom culture

This section assesses the learning environment, the level of student engagement, the nature of the working relationships, and issues of equity.

CRISS [active engagement, discussion, pair/small group learning, setting learning goals, self regulation]

• Students participate in development of crucial questions for study

• Instructional conversations are an integral part of learning

• Students reflect on the effectiveness of their use of the reading strategies in facilitating comprehension of content text

The three CRISS study researchers who were trained and consulted with Dr. Nicole Saginor are currently conducting observation practice in three pilot schools. Later in the school year, inter-rater reliability will be established with these observers with the assistance of Dr. Saginor. Other observers will be trained in the VCOT-CRISS by Dr. Saginor (as required by the Vermont Institute) and reliability will be tested against the reliability set by the CRISS study researchers.

Part of the pilot test will be to collect data on whether the VCOT-CRISS is sensitive to the expected changes in teacher practice that should result from the intervention. In addition to collecting observational data from pilot schools in which teachers are just learning and trying out CRISS strategies, we are working with Carol Santa and Lynn Havens of Project CRISS to identify a small number of teachers and schools in the Northwest who are regarded as experienced, seasoned users of the CRISS philosophy and strategies. We will conduct pilot VCOT observations in these schools also, comparing beginning CRISS teachers to expert teachers in a small scale field trial to determine instrument sensitivity to Project CRISS. The final determination to use VCOT-CRISS will be made in spring 2007 based on its performance and practicality as a large-scale measure.

The VCOT has been used in some previous IES studies conducted by Mathematica, but with one problems. VCOT requires that Dr. Saginor train all observers, and Dr. Saginor was also responsible for doing reliability checks in the field for these studies. For our study, Dr. Saginor has agreed to come to NWREL in spring 2007 to train all of our new trainers, who will consist of three or four additional NWREL staff with teaching experience, for a total of six or seven trainers to cover 20 schools. Reliability checks will be conducted in the field by having an observer occasionally double-up with another observer to check reliability and provide quality control.

4