OMB Memo about survey revisions

Memorandum

TO: Sandra Garcia, REL COR

FROM: Trudy Cherasaro, REL Central

SUBJECT: OMB Control No: 1850-0903: A Study of Feedback in Teacher Evaluation Systems: Survey Revisions

DATE: 9/18/2014

Background

As approved in the OMB package 1850-0903, REL Central conducted a pilot study of the Examining Evaluator Feedback Survey in order to revise the survey for administration in the full study of feedback in teacher evaluation systems. REL Central completed the pilot study and has made minimal revisions to the survey. The survey revisions do not impact the time or cost burden that were originally proposed in the OMB package. The sections below provide a description of the survey revisions made at each step of the pilot study process. The original and revised surveys are provided as attachments.

Advisory Review

The survey was first reviewed by an advisory panel with 7 members including expert survey developers, state leaders and district leaders. Based on this review REL Central revised the directions, question stems, question wording, and/or response options on several questions for clarification and to increase ease in responding. REL Central also added three items to the original set of items. The changes did not impact the response burden.

Original Question #	Type of change	Description
2	Directions	Added the following note: If you have more than one evaluator please pick one and refer to that evaluator as you respond to the remaining questions.)
3	Directions/definition	Added walkthroughs as a type of evidence for which teachers may have received feedback.
4	Directions/definition	Added walkthroughs as a type of evidence for which teachers may have received feedback. Indicated that the written feedback could be in paper or electronically.
5-9	Created shorter question stem	Moved directions that asked teachers to “keep in mind feedback that was received throughout the current school year” to a page header. Pulled out repetitive question stem to allow for quicker responses (e.g. “My evaluator’s feedback …”).
5g	Question wording	Changed to “included recommendations” from “recommended next steps,” for a better fit with the stem and for clarity
6b	Question wording	Changed to include walkthroughs in addition to observations.
6c	Question wording	Changed “data” to “evidence”
7a-d	Question wording	Added “to effectively evaluate me” to the end of each item
7	Added question (7c in revised survey)	Added the item “knowledge of effective teaching.”
8b	Question wording	Revised question from “access to a coach/mentor” to “access to an instructional leader” with the following examples of an instructional leader (peer, coach/mentor, administrator).
8d	Question wording	Added example of “collaborative or individual planning.”
9d	Question wording	Revised question from “advice from a coach/mentor” to “advice from an instructional leader” with the following examples of an instructional leader (peer, coach/mentor, administrator).
9	Added question (9e in revised survey)	Added the item “I changed the way I plan instruction.”
10-13	Created shorter question stem	Pulled out repetitive question stems to allow for quicker responses (e.g. “My evaluator’s feedback …”) and rearranged questions to fit with question stems.
11b	Question wording (10i in revised survey)	Changed to include walkthroughs in addition to observations.
11c	Question wording (11a in revised survey)	Added “if they reviewed the same evidence” to the end of the item.
12a-d	Question wording	Added “to effectively evaluate me” to the end of each item.
12	Added question (12c in revised survey)	Added the item “knowledge of effective teaching.”
13b	Question wording	Revised question from “access to a coach/mentor” to “access to an instructional leader” with the following examples of an instructional leader (peer, coach/mentor, administrator).
14	Response options	Changed response options to: Not at all, a little and a lot.

Cognitive Interviews

The survey, revised based on the advisory review, was then completed by 9 teachers and follow up interviews were conducted with these teachers to ask about the clarity of the questions. Based on the feedback from these teachers, REL Central made minor adjustments to the language of 3 items. The table below shows these adjustments.

Original item	Revised item
In my opinion, my evaluator had sufficient knowledge of effective teaching to effectively evaluate me.*	In my opinion, my evaluator had sufficient knowledge of effective teaching practices to effectively evaluate me.
I had access to professional development that I needed in order to implement suggestions provided in my feedback.*	I had access to professional development (formal or informal) that I needed in order to implement suggestions provided in my feedback.
Because of the feedback I received from my evaluator, I sought professional development opportunities.	Because of the feedback I received from my evaluator, I sought professional development opportunities (formal or informal).

* These items were adjusted in both the questions with rating of agreement and the questions with rating of importance.

Pilot Administration

The revised survey was then completed by 196 teachers. Based on an analysis of reliability and validity using classical test theory, Rasch analysis, and confirmatory factor analysis REL Central identified two adjustments. First, the “somewhat agree” and “somewhat disagree” response options were collapsed into one response option “neither agree nor disagree.” Finally, we adjusted the wording of one item, presented below.

Original item	Revised item
My evaluator’s feedback was provided within an appropriate timeframe.	My evaluator’s feedback was provided in time for me to use it to inform my practice.

REL Central OMB 1850-0903: Survey Revisions 9/18/14 Page 2