Study of Emerging Teacher Evaluation Systems in the United States
Appendix II: Interview Protocols
School District Staff and Local Stakeholders
Interview Protocol
Introduction to the Interview
Introduce of the interviewer(s)
Explain purpose of the study and topics to be covered in the interview
Explain voluntary nature of participation, provisions for protecting respondent’s privacy, and/or explanation of opportunity to review text.
Interviewers will read the following statement to interview respondents at the beginning of each interview:
No individuals or schools will be identified by name in any reports or other communications about the study. Districts will be identified in profiles of individual teacher evaluation systems, although key study findings will be reported in the aggregate. Case study data will be maintained in secure files and will be accessible only to members of the study team. Information that identifies individuals will not be released except as may be required by law.
Advise respondent that the interview will last approximately 45 minutes.
Invite questions from respondents
Confirm respondent’s current roles and responsibilities within the agency/organization and in terms of teacher evaluation system development, implementation, and administration
Interview Questions
Note: Interviewers will ask all follow-up questions as specified in the protocol. In addition, interviewers will be directed to seek clarification of any response they judge to be vague or incomplete. The primary strategy for seeking clarification will be to ask respondents to provide specific examples to illustrate their responses. A second strategy will be to ask respondents to “say more” or to “expand on” or “explain the meaning of” a particular comment.
Planning the Teacher Evaluation System
Briefly describe the planning and implementation processes (e.g., meetings with advisory/planning group; outreach to teachers, teacher unions, school leadership; visits to schools; training for principals/teachers, etc.). What, in your opinion, were the key steps in system design and implementation?
When did your district begin developing the [name of teacher evaluation system]? To the extent you know, how long did the development process take? What steps were involved in that process, and were there some steps that were the most time consuming to complete?
[If applicable] Once the teacher evaluation system was developed, how many weeks, months, or years were required to fully implement the system? Was the system pilot tested? How long did the pilot testing take to complete?
What is the primary goal(s) of the new teacher evaluation system?
Improving instruction
Identifying effective teachers
Identifying ineffective teachers
Improving student learning
Other goal(s)
Follow-up questions for respondents who identify one or more goals: In your opinion, have these goals been communicated to all stakeholders (e.g., teachers, union representatives, district staff, and community members)? How have the goals been communicated to stakeholders? In your opinion, is there consensus about the goals?
What groups, (e.g., teachers, union leaders, building administrators, district staff/administrators, school board members, parents, business representatives) were involved in planning the new teacher evaluation system?
In your opinion, were there any groups or individuals who were not involved who should have been involved?
Follow-up question for respondents who say that key groups were not included in the planning process: In your opinion, what are the consequences (if any) of not including this group(s)/individual(s) in the planning process?
What difficult issues, if any, did the planning group confront?
Follow-up questions for respondents who say that there were difficult issues: Briefly, what were the difficult issues? How and to what extent were these issues resolved?
Who was involved in determining the relative weight assigned to various factors to be included in the overall rating of teacher effectiveness (e.g., changes in student learning, classroom instruction, professionalism, and community engagement)? What other issues did the planning group address in developing the teacher evaluation system? Follow-up questions for respondents who identify other issues: How were these issues resolved? What challenges did the group face in resolving them?
Components of the Teacher Evaluation System and Communication about These Components
Please describe how and to what extent the teacher evaluation model uses standards and/or a specific model of teaching practice (e.g., Charlotte Danielson’s framework, National Board for Professional Teaching Standards, Classroom Assessment Scoring System, state-developed standards/model) to assess teacher performance, including:
Whether and how an existing framework/standards has been tailored/modified to address district needs and priorities
Strategies for communicating about the framework/standards with teachers, school leaders, and other stakeholders
How was this framework/standards/domains communicated to stakeholders, especially teachers and principals?
In your opinion, do various stakeholders understand the framework and standards and agree that they are appropriate for teachers in this district?
Follow-up questions for respondents who say that not everyone is on board with the standards/framework: Which groups, if any, do not appear to understand the standards/frameworks? What concerns do various stakeholder groups have about the standards/framework?
How does the teacher evaluation system assess instructional practice, teacher planning and preparation, and/or other dimensions of professional practice?
What are the required and/or recommended strategies for collecting data on teacher performance in the areas included in the framework/standards (e.g., classroom observations, review of professional portfolios, peer review, student and/or parent surveys)?
What is the relative weight of measures of professional practice in the overall assessment of individual teachers?
How does the teacher evaluation system use gains in student achievement as a factor in rating teacher performance?
What is the relative weight of student achievement gains in the overall assessment of individual teachers?
Does the district administer additional standardized tests for the purpose of measuring teacher effects on student achievement for the teacher evaluation system, or is the district using existing tests (with no additional test administrations)? If additional tests are administered only for the purposes of the teacher evaluation, please explain which tests are administered, to whom and how frequently those tests are administered during the school year.
Do teachers, principals and others have questions and concerns about the way that gains in student achievement are used in the evaluation system?
Follow-up questions for respondents who say that there are concerns in this area: What are the questions and concerns about this component of the teacher evaluation system?
How, if at all, has the district tested the validity of the teacher evaluation system? That is, to what extent has the system’s capacity to validly measure teacher performance been tested?
Follow-up questions for respondents who indicate that the district has tested the validity of the teacher evaluation system: Has the teacher evaluation system been tested to determine whether teacher ratings are sensitive to external/contextual factors such as class size, curriculum, student demographics, etc.? Are teacher ratings relatively stable over time? Can the scores be manipulated? Who conducted the validity tests of using gains in student achievement and classroom observation data to rate teacher performance and what were the results? Were any adjustments made to the teacher evaluation system based on these results? To what extent are plans in place to regularly test the validity of the teacher evaluation system?
How, if at all, has the district assessed the validity and reliability of using classroom observations to rate teacher performance? That is, to what extent has the district determined whether classroom observation ratings are a valid predictor of value-added measures of student achievement? In addition, to what extent has the district tested the consistency of the ratings that principals and other observers (internal and/or external) assign teachers based on their observations of teachers’ classroom practices?
Follow-up questions for respondents who indicate that the district has assessed the validity and reliability of classroom observations to rate teacher performance: Who conducted the tests of reliability of classroom observation data and what were the results? Were any adjustments made to the system based on the results of the reliability tests? To what extent are plans in place to regularly test inter-rater reliability?
In your opinion, is the overall design of the teacher evaluation system appropriate for all types of teachers (e.g., regular education teachers, special education teachers, teachers of English language learners, teachers in core academic subjects, teachers in other subject areas, elementary school teachers, middle school teachers, high school teachers)?
Follow-up questions for respondents who indicate that the system may not be as appropriate for some groups of teachers as it is for other groups of teachers: For which groups of teachers is the system less appropriate? What are the specific reasons why it is less appropriate? To your knowledge, are there any efforts underway to address this issue(s)?
What options, if any, do teachers have to appeal ratings of their effectiveness?
Follow-up questions for respondents who indicate that teachers can appeal ratings of their effectiveness: What does the appeal process include and who serves as the final arbiter of the appropriateness of individual ratings?
Who is responsible for collecting information on a teacher’s classroom practice and other performance indicators?
How are these people prepared for their responsibilities?
Follow-up questions for respondents who are familiar with the preparation and/or who have participated in the process: In your opinion has the preparation been adequate? If not, what were the gaps in the preparation?
What mechanisms are in place to ensure that staff are carrying out their responsibilities successfully both in terms of quality and timeliness?
Questions for principals and other school-based staff responsible for collection data, providing feedback to teachers, and tasks associated with the teacher evaluation system:
In a normal week, approximately how much of your time is devoted to responsibilities associated with the teacher evaluation system and process?
What changes and accommodations, if any, have you made in other areas of your job to be able to devote time to responsibilities associated with the teacher evaluation system?
Administrative Costs of the Teacher Evaluation System
Does the district have any contracts with outside organizations for analysis of teacher evaluation data or for assistance with the administration of the evaluation system? If so, what are the purposes of these contracts and what are the annual costs of these contracts? If the contracts serve purposes beyond those of the teacher evaluation system, what percentage of the contracts’ total costs are estimated to be for teacher evaluation work?
What district-level staff are involved in the operation and administration of the teacher evaluation system? What are their responsibilities? Did the district create any new positions that are devoted largely or entirely for the new teacher evaluation system? If so, what are these positions? Were these positions added to the total number of positions in the district, or were existing positions converted into these positions? What responsibilities do people in these positions assume?
What type of staff are directly involved in a teacher’s annual evaluation in the current system and in the previous system? Please estimate the amount of time (in hours or minutes) that each of these staff spends on one typical teacher’s evaluation each year. To what extent do estimates vary for elementary versus secondary school teachers?a
Use of Teacher Evaluation System Results
How, if at all, does the district use (or expect to use) the results of the teacher evaluation system to inform decisions about:
Planning and providing teacher professional development
Retention or dismissal of individual teachers
Promotion of individual teachers to positions of increased authority and/or status
Teacher compensation
Deployment or re-deployment of highly effective teachers
Reporting to parents and other stakeholders on teacher quality in individual schools and the district as a whole
What challenges/problems, if any, has the district encountered, if any, in using the results for these decisions?
Follow-up questions for respondents who identify challenges in using the evaluation results: What has been done to overcome the challenges? How successful has the district been in these efforts?
Evaluation of the System and Possible Changes
What challenges/problems, if any, has the district encountered, if any, in implementing and operating the teacher evaluation system? What has been done to overcome the challenges? How successful has the district been in these efforts?
How have lessons from early implementation been translated into modifications to system design and operation?
What concrete evidence is available to suggest that the teacher evaluation system is achieving (or is likely to achieve) the goals you discussed earlier?
Based on your experience and information from principals and teachers, what, if anything, should the district have done differently regarding the design and implementation of its new teacher evaluation system?
What other changes in the district and/school organization and culture, if any, have you observed that you believe to be attributable to the teacher evaluation system?
Follow-up question: Has implementation of (name of teacher evaluation system) resulted in any unexpected changes or consequences in the district or individual schools?
Looking ahead the next school year, what changes or modifications to the system, if any, are likely?
Follow-up question for respondents who indicate that changes are likely: What assistance, if any, will be necessary to implement these changes?
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement for Paper Work Reduction Act Submissions |
| Author | JJB |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |