Interview Protocols

Study of Emerging Teacher Evaluation Systems in the United States

Appendix II: Interview Protocols

Protocol for Interviews with State Education Agency Staff and State-level Stakeholders

Background for Interview

Prior to each set of state-level interviews and site visit, site visitors will review all extant data for the site and determine which of the interview questions can be answered based on information that is in existing documents. In general, we anticipate that documents available prior to data collection will include, but not be limited to the following:

• Statutes/regulations governing the establishment and operation of teacher evaluation systems

• State requirements/guidance for local teacher evaluation systems, implementation plans, and plans for evaluations of new teacher evaluation systems

• State and local planning documents, including reports on planning activities, key recommendations for program design and operation, implementation timelines, membership on planning/steering/ and/or oversight committees, and proposals for funding of state and local evaluation system development and implementation

• Local program descriptions and manuals, including program overviews, and descriptions of key program components such as (a) program goals and objectives, (b) annual timelines for key activities and steps in the evaluation process, (c) descriptions of how data on student achievement contributes to teacher ratings, (d) strategies for using other indicators/data sources in assessing teacher performance, (e) program accommodations for evaluating special education teachers, teachers of English language learners, and new teachers, (f) teaching frameworks and standards, rubrics, and observation guides, and (g) training for principals and other staff on conducting and reporting the results of classroom observations, communicating evaluation results to teachers, and assisting teachers in individual professional development planning.

• Reports on program design and operations, reports on evaluation results and how they are used, and reports from external evaluations

• Technical reports and descriptions of student growth models used in rating teachers

• State and local plans and/or timelines for (a) full implementation of the Common Core State Standards and standards in other content areas and (b) full implementation of new state assessments aligned with the new standards

After reviewing these and other documents that may be available, site visitors will determine which of the questions in the protocols can be answered from the document review and which questions need to be included in the interviews.

Note that all site write-ups prepared for data analysis and to inform reporting will address all of questions included in the protocols, although it may not be necessary to ask all of the questions of all of the respondents.

Protocol for Interviews with State Education Agency Staff and State-level Stakeholders

Introduction to the Interview

• Introduce the interviewer(s)

• Explain the purpose of the study and topics to be covered in the interview

• Explain of voluntary nature of participation, provisions for protecting respondent’s privacy, and/or explanation of opportunity to review text.

Interviewers will read the following statement to interview respondents at the beginning of each interview:

No individuals or schools will be identified by name in any reports or other communications about the study. Districts will be identified in profiles of individual teacher evaluation systems, although key study findings will be reported in the aggregate. Case study data will be maintained in secure files and will be accessible only to members of the study team. Information that could be used to identify individuals will not be released except as may be required by law.

• Advise the respondent that the interview will last approximately 45 minutes.

• Invite questions from the respondent

• Confirm respondent’s current roles and responsibilities within the agency/organization and in terms of teacher evaluation system development, implementation, and administration

Interview Questions

Note: Interviewers will ask all follow-up questions as specified in the protocol. In addition, interviewers will be directed to seek clarification to any responses they judge to be vague or incomplete. The primary strategy for seeking clarification will be to ask respondents to provide specific examples to illustrate their responses. A second strategy will be to ask respondents to “say more” or “expand” on a particular comment.

State Framework for Teacher Evaluation

1. When did your state begin developing the [name of teacher evaluation system]? To the extent you know, how long did the development process take? What steps were involved in that process, and were there some steps that were the most time consuming to complete?

Follow-up questions for respondents who know about the development of the teacher evaluation system: Once the teacher evaluation system was developed, how many weeks, months, or years were required to fully implement the system? Was the system pilot tested? How long did the pilot testing take to complete?

2. Which of the following do you consider to be the primary goal(s) of (name of teacher evaluation system)?

• Improving instruction

• Identifying effective teachers

• Identifying ineffective teachers

• Improving student learning

• Other goal(s)

Follow-up questions for respondents who identify one or more goals: In your opinion, have these goals been communicated to all stakeholders (e.g., teachers, union representatives, district staff)? How have the goals been communicated to stakeholders? In your opinion, is there consensus about the goals?

3. What statutes and/or regulations, if any, are in place to guide and support the design of local teacher evaluation systems?

Follow-up questions for respondents in states in which there are statutes and/or regulations in this area: Based on your knowledge of the statutes and regulations, what components of teacher evaluation systems are required of all systems and where are there areas of flexibility? A number of factors or issues may contribute to decisions to pass legislation and/or issue regulations related to the establishment of teacher evaluation systems. In your opinion, what factors and/or issues led to the state statutes and/or regulations on teacher evaluation in this state?

4. What groups, (e.g., teachers, union leaders, building administrators, district staff/administrators, state education agency staff/leaders, parents, business representatives) were involved in developing the state model/framework/requirements for teacher evaluation systems and planning for state support for local programs?

5. In your opinion, were there any groups or individuals who were not involved who should have been involved?

Follow-up question for respondents who say that key groups were not included in the planning process: In your opinion, what are the consequences (if any) of not including this group(s)/individual(s) in the planning process?

6. Was the planning group involved in determining the relative weight assigned to various factors to be included in the overall rating of teacher effectiveness (e.g., changes in student learning, classroom instruction, professionalism, community engagement)?

Follow-up questions for respondents who say that the planning group did address the issue of weighting various factors related to teacher effectiveness: What specific issues did the group address? What were the main areas of agreement and disagreement?

7. What other issues did the group address in developing the state teacher evaluation model?

Follow-up questions for respondents who identify other issues: How were these issues resolved? What challenges did the group face in resolving them?

8. Please describe how and to what extent the state teacher evaluation model uses standards and/or a specific model of teaching practice (e.g., Charlotte Danielson’s framework, National Board for Professional Teaching Standards, Classroom Assessment Scoring System, state-developed standards/model) to assess teacher performance, including:

• Whether and how an existing framework/standards has been tailored/modified to address state needs and priorities

• Strategies for communicating about the framework/standards with teachers, school leaders, district staff, and other stakeholders

9. How does the teacher evaluation system assess instructional practice, teacher planning and preparation, and/or other dimensions of professional practice?

• What are the required and/or recommended strategies for collecting data on teacher performance in the areas included in the framework/standards (e.g., classroom observations, review of professional portfolios, peer review, student and/or parent surveys)?

• What is the relative weight of measures of professional practice in the overall assessment of individual teachers?

10. How does the teacher evaluation system use gains in student achievement as a factor in rating teacher performance?

• What is the relative weight of student achievement gains in the overall assessment of individual teachers?

11. How, if at all, has the state tested the validity of the teacher evaluation system? That is, to what extent has the system’s capacity to validly measure teacher performance been tested?

12. In your opinion, is the overall design of the teacher evaluation system appropriate for all types of teachers (e.g., regular education teachers, special education teachers, teachers of English language learners, teachers in core academic subjects, teachers in other subject areas, elementary school teachers, middle school teachers, high school teachers)?

Follow-up questions for respondents who indicate that the system may not be as appropriate for some groups of teachers as it is for other groups of teachers: For which groups of teachers is the system less appropriate? What are the specific reasons why it is less appropriate? To your knowledge, are there any efforts underway to address this issue(s)?

State Support to Districts

13. A state education agency can provide various kinds of training and technical assistance to support local implementation of new teacher evaluation systems (e.g., training in the application and use of student growth models teacher evaluation, training on the development and use of observation protocols and other data collection tools, technical assistance in planning and conducting training for principals and other district staff in conducting observations, providing feedback, and assisting in individual professional development planning). What kinds of training and technical assistance does this agency provide or make available to support implementation of local teacher evaluation systems?

14. Does the state education agency require school districts to submit plans for new teacher evaluation systems?

Follow-up questions for respondents who indicate that the state education agency requires district plans: What are district plans expected to include (e.g., an implementation timeline, strategies for communicating about key components of the new teacher evaluation system, plans for developing guides, rubrics, and other artifacts)? How does the state review and provide feedback on the plans?

15. Does the state education agency require school districts to evaluate their new teacher evaluation systems?

Follow-up questions for respondents who indicate that the state education agency requires districts to evaluate their new teacher evaluation systems: What specific questions and/or issues are local evaluations required to address?

16. What specific plans does the state education agency have for using the results of local evaluations?

Challenges, Lessons, and the Future

17. Based on your experience and information available from districts, what are the key lessons learned from local implementation of the state model/requirements for teacher evaluation systems?

18. How have lessons from early implementation been translated into modifications to system design and operation?

19. Again, based on your experience and information from districts, what, if anything, should the state education agency have done differently to support local implementation and operation of new teacher evaluation systems?

20. Looking ahead, do you anticipate changes in the state model/requirements/ for local teacher evaluation systems?

Follow-up questions for respondents who say that they anticipate changes in the state model/requirements for local teacher evaluation systems: What specific changes do you anticipate? When will the changes be made?

Follow-up questions for respondents who say that they anticipate changes and who work in states that have adopted the Common Core State Standards(CCSS) and/or that are members of one of the national assessment consortia: How, if at all, will local implementation of the CCSS and/or new state assessments aligned with the standards affect and/or alter the design and operation of local teacher evaluation systems?

8