Alternative student outcomes for growth measures case studies

OMB Supporting Statement Part B: Alternative Student Growth Measures for Teacher Evaluation: Case Studies of Early Adopters

Appendix E: Principal Interview Protocol

Instruction to interviewer: when the respondent is unable to respond completely to a question, please always follow up by asking for the names and contact information of other staff in the district who can provide more information.

Introduction and Consent for Audio Recording

The purpose of this section is to introduce the respondent to the study and request consent for audio recording.

1. Thank you for taking the time for the interview today. My name is [name] and I work for Mathematica Policy Research, which is conducting this study for the Mid-Atlantic Regional Educational Laboratory (REL). The Mid-Atlantic REL’s mission is to “build research alliances with education practitioners and policymakers to help states and districts use data to answer important issues of policy and practice.” The results of this study will inform publically available reports aimed at states and districts considerting implementing measure of teacher performance that include student growth. As I mentioned [in my email/when we spoke on the phone], we are conducting case studies of districts implementing measures of teacher performance that include student growth, but that measure growth on outcomes other than state assessments. These alternative outcome measures might include nationally normed assessments, end-of-course curriculum-based assessments, or “student learning objectives” (SLOs) developed by individual teachers for their own students. Student growth might be measured through a simple subtraction of end-of-year achievement from beginning-of-year achievement, through a student growth percentile method, through a value-added model, or implicitly in SLOs that are set based on an understanding of baseline achievement levels in a class at the beginning of the school year. We plan to examine what alternative outcome measures are used, how the growth measures are implemented, challenges and obstacles in implementation, how the measures are being used, and the effectiveness of these measures in differentiating teacher performance. We expect our findings to benefit states and districts considering the use of such alternative growth measures. The purpose of this interview is to gather information on the implementation of [insert specific growth measures used in the district] in the district from your perspective as a district administrator. We will be interviewing teachers, principals, and union representatives in the district to gather information on their perspectives as well. We will not identify you by name in the study reports.We also will not identify the districts included in the study in subsequent reports.

Before beginning, I would like to get your permission to record the interview. This will ensure that we accurately capture information about district policies and will prevent me from having to ask questions and take notes at the same time. The audio recordings from all individuals interviewed within your district will be compiled into a single district summary that is for internal analysis purposes only and will not be published or shared with anyone outside the study team. You will not be identified in the district summary. This summary will then be aggregated with other district summaries and analyzed for trends and variations across multiple districts. Do I have your permission to record the interview?

Type of Student Outcome and Growth Measure Used

1. Are teachers in your school measured based on [identify the alternate outcome growth measure(s) discussed in district interview]?

2. (If the district uses SLOs) As the principal, what role do you play in the development of SLOs? Is there a district-wide template used by teachers? What is the process for approving SLOs submitted by teachers? Are teachers required or permitted to submit more than one SLO for a class? Does this vary by grade and subject area? From your perspective, are SLOs of comparable difficulty across all teachers in your school? Do SLOs in your school involve classwide targets or separate targets for each student? How much time do you devote to the development of SLOs?

3. Can you describe the process for rolling out the alternative measure(s) in the district? What role did you play in the decision-making process? Who else was involved in the decision-making process? Did you receive any training or professional development related to the implementation of the measure?

Implementation of Data Collection and Analysis

1. I would like to learn more about the administration of these alternative student assessments in the district. (For each assessment) How is the assessment administered (e.g., timing, frequency, who administers)? What is the process for collecting assessment results?

2. Are you aware of any quality control measures in place in the district for the collection and analysis of the teacher performance data collected through the alternate measure? (If district uses SLOs) For example, what documentation of student performance do your teachers need to submit for your SLOs, and are teachers subject to any audits, such as auditing SLOs to ensure rigor? (If district uses a VAM/growth model) With respect to quality control for VAM/growth model: for example, do teachers administer or score the student assessments or is administration and/or scoring conducted by independent proctors or an external firm?

3. What kinds of obstacles, if any, have you encountered in preparing for and administering the alternate student assessments or SLOs? How many extra hours, if any, have you had to devote to these tasks, per teacher? Does the amount of time devoted to these tasks vary by teacher?

4. Have there been any changes to the district’s approach to using alternative student growth measures? Have these changes resulted in notable improvements?

Differentiation produced by the alternative growth or VA measure

1. What proportion of teachers in your school is in each performance category according to the alternate growth measure? (If respondent is unsure across all categories) Approximately what proportion of teachers is in the top and bottom categories?

2. Have you found the performances of individual teachers in your school to be consistent across years and measures? Does the measure accurately reflect teachers’ contributions to student learning? Is the measure capturing the appropriate student learning standards?

Use of Alternative Growth or VA Measure

1. For what purposes is your school using each alternative student outcome measure used in assessing teacher performance? Is it used for evaluating teachers? In what grades and subject areas? Do teachers receive targeted professional development based on their performance? Does the measure affect teacher assignments? Are there any other ways in which you are using these measures?

2. Is the performance of teachers within your school used for evaluating your performance as a principal? Have you taken any school-wide or teacher-level actions as a principal based on this performance information?

3. Do teachers receive any feedback regarding their performance on each measure? What type of feedback? When and how is this feedback provided? Who provides this feedback to teachers?

4. Does a teacher’s performance rating on the alternative measure affect whether he/she is awarded tenure?

Benefits and Drawbacks

1. Could you summarize some of the main benefits, from your perspective, of implementing this alternative VAM or growth measure of teacher performance in your school? For example, have you found that receiving this kind of information about teacher performance in your school has had any benefits for your effectiveness as a principal?

2. What do you perceive as the primary drawbacks or costs (e.g., time) of implementing the measure? Do you think the measure allows for a sufficient level of comparability across teachers on performance? If SLOs are used, do you think all teachers are setting sufficiently high standards for student growth?

3. How do you perceive the district’s communication of the use of the alternative measure in evaluations to teachers and principals?

District Context

1. Have there been any changes in the district administration or policies that may have affected the implementation of these alternative measures of teacher performance (for example, change in superintendent or implementation of a performance-based pay system)?

Is there anything else that you think might be important to know about the development and implementation of this alternative growth or value-added measure in the district? Are any other key stakeholders in the district with whom you think it would be important for us to speak?

OMB Package: Alternative Student Outcomes for Growth Measures Case Studies 1