Part_B_of_Supporting_Statement_TES_OMB_package.3-22-12

Supporting Statement for Paperwork Reduction Act Submissions for the Study of Emerging Teacher Evaluation Systems in the United States

B. Collection of Information Employing Statistical Methods

B.1. Respondent Universe and Sample Size

Through the use of internet research, recommendations of the TWG, and suggestions from Department staff who are familiar with state and local efforts to design and implement new teacher evaluation systems, the research team will compile a list of candidate sites to be considered for the final sample. Next, we will contact the person who directs local teacher evaluations for an informal discussion of the characteristics of the systems and to gauge the extent to which the system reflects the following characteristics specified by PPSS:

• The use of measures of student achievement and measures of teacher effects based on value-added measures or other calculations of gains in the achievement of students

• The capacity to make distinctions among teachers at different levels of performance

• A formative component that provides timely feedback to teachers to help them improve

• Alignment with other parts of the district human capital system

• Requirement for annual evaluations of all teachers

• Data on individual student growth and student growth aggregated to the class and school levels available to teachers and principals

• Use of evaluation data to inform decisions in areas such as professional development, tenure, promotion, and compensation

We will select nine sites for the study, including four that are in the early phase of system implementation and five that are fully operational, based on how well the district’s system meets the selection criteria. In addition, diversity of geography, size, poverty level, demographics, urbanicity, and state policy climate will be taken into consideration when selecting sites.

Once PPSS has approved the list of sites to be considered for the study and OMB has approved the case study data collection, the research team will send each superintendent a letter soliciting district participation in the study. These letters will explain the study and expectations for district participation. (Appendix I includes a draft of the recruitment letter.)

B.2. Procedures for Collection of Information

Two members of the study team will visit each of the nine sites selected for the study sample. Researchers will interview up to 753 individuals in various roles in the fully operational sites and up to 24 individuals in sites that are in the implementation phase, as shown in Exhibit 3.

Exhibit 3: Expected Number of Respondents

in Each Participating Site, by Role

Role	Number of Respondents in Fully Operational Sites	Number of Respondents in Sites in Early Implementation Phase
District leaders and staff (e.g., superintendent, director of research, director of human resources, director of teacher evaluation system, budget director, and principals)	220 individuals	20 individuals
Teachers who participated in the design process
Local stakeholders (e.g., union leaders, community representatives,)
Teacher focus groups (made up of 5-7 teachers, with each group made up of teachers in the same role)	7 groups (up to 49 individuals)	None
State education agency staff and state-level stakeholders (e.g., union leaders, members of state boards of education, legislators, representatives of professional organizations) (interviewed by phone)	4 individuals	4 individuals

The specific number and role of respondents will vary across districts, depending on how the teacher evaluation systems were planned and operate at the time of the site visits. In all cases, district leaders and staff who have specific responsibilities for administering the system and/or system operations, teachers involved in system planning and design, local-level stakeholders, state education agency staff, and state-level stakeholders will be included as respondents.

In sites that are fully operational, we will conduct seven focus groups, with each group including five to seven teachers. Following PPSS’s specifications for this part of the data collection activities, each focus group will include teachers of the same role, with one focus group for each of the following categories:

• Special education teachers who may not have standardized student achievement tests as part of their evaluation measures

• Teachers who teach English language learners in language instructional education programs

• Teachers in grades 4 and 5

• Middle school teachers who teach in at least one tested subject and grade

• Middle school teachers who teach non-tested core academic subjects and grades

• High school teachers who teach in the tested grades and subjects

• High school teachers who teach in non-tested academic subjects and grades

Each individual interview will last 45-60 minutes and each focus group will last 75 minutes. (The three protocols that will guide the interviews are included in Appendix II.) The research team will audio-record and take notes during each interview to ensure an accurate record.

Upon receipt of final approval for the case study sites and after the sites have agreed to participate, the study team will work closely with the site liaisons, who will be assigned by superintendents as each district’s primary contact for the study, to identify appropriate respondents. The first step will be to ensure that the site liaisons understand the purpose of the study and the research questions that we will address. Next, we will explain the data collection needs and request the liaisons to identify appropriate respondents and to begin scheduling the interviews.

For the teacher focus groups, we will request that the liaisons include teachers from at least two schools (and more, if possible) in each focus group and that all teachers have participated in at least one complete cycle of the evaluation process.¹ We will also ask that teachers who were involved in planning the evaluation system and who might be selected as a respondent for a district staff interview not be included among those identified for participation in a focus group.² In addition, we will ask the liaisons to include teachers with different kinds of assignments in each of the focus groups. For example, an ideal special education teacher focus group could include teachers who work with students with serious cognitive impairments, teachers who work with students with physical handicaps, and teachers who work with regular education teachers in inclusion classrooms. Similarly, the ideal focus group of high school teachers who teach in tested subjects and grades would include a mix of teachers from core academic subjects as well as teachers who are members of subject area teams and those who teach in completely self-contained classrooms. Although it may not be true in every site, the general pattern of state testing suggests that these teachers would teach 10th-graders in all or some of their classes. Although it is not possible to provide specific formulae for the composition of the focus groups, we will work closely with the site liaisons to ensure that each focus group includes teachers with a range of experience and perspectives. We will also work with the liaisons to identify appropriate external stakeholders to be interviewed. Depending on local circumstances, we will either ask the liaisons to schedule the interviews with the external stakeholders or arrange to contact them directly to schedule the interviews.

We will almost certainly contact union representatives directly, although the liaisons will assist in scheduling the interviews with other external stakeholders. At the state level, we will contact the deputy chief state school officer to identify the member of the agency staff who is responsible for the state’s efforts to guide and support local evaluation systems. We will work with this individual to identify other state-level respondents. Once these individuals have been identified, we will contact them directly to schedule and conduct the telephone interviews. Here, again we will contact state union leaders directly to identify appropriate respondents and to schedule the telephone interviews. Wherever possible, the state-level interviews will take place prior to the district-based data collection although it is possible some follow-up interviews with state-level respondents will be necessary after the district-level data collection.

1. Statistical Methodology

This study involves collection of qualitative data. A discussion of statistical methodology is not applicable to this study.

2. Analysis Methods

All documents and interview data will be subjected to qualitative analysis procedures that rely on text recognition software and appropriate coding processes to identify cross-cutting themes and key details of (a) system planning and design, (b) state policy context and state education agency support for local system design and operation, (c) structural characteristics and operations, (d) planned and actual uses of system outputs, and (e) implementation challenges and solutions.

3. Degree of Accuracy Needed

The research team will do everything possible to maximize the accuracy of the data collected for each of the case studies. First, as described below, we have pilot-tested the interview protocols and revised them to increase the efficiency of the interview process and to ensure that we collect all of the data necessary for the case studies. Second, prior to data collection, we will provide a thorough orientation to the members of the site visit teams to familiarize them with the overall study design and their responsibilities for data collection and case write-ups. This orientation will include special attention to the interview protocols, and responsibilities and strategies for tailoring and conducting individual interviews and focus group interviews. Third, all interviews will be recorded for later transcription and data analysis, and interviewers will also take detailed notes during each interview for use in checking the accuracy of the transcriptions.

4. Use of Periodic Data Collection

Data collection will occur only one time.

B.3. Methods for Maximizing Response Rate and Dealing with Nonresponse

With the assistance of the liaison in each district, interviews with each respondent or group of respondents will be scheduled in advance. In cases where selected respondents are unable to schedule a meeting during the site visit or become unavailable on short notice, we will conduct interviews by phone at a later date. Because the research team will work closely with the site liaisons to select respondents based on their role and because we will have a considerable degree of flexibility in scheduling the time and location of the interviews, we anticipate a high response rate is anticipated. In addition, the recruitment letter to each superintendent will request his/her support in recruiting and ensuring respondent participation in the study, and once that superintendent has advised the study team that the district will participate, we will advise prospective interview respondents of this support.

B.4. Test of Procedures and Methods

In addition to consulting with the TWG, we received feedback from Department staff on drafts of the three interview protocols. (The versions of the protocols for which we seek approval are included in Appendix II.) Feedback from the Department included suggestions that we:

• Add follow-up prompts for interviewers to use in asking respondents for more detailed information about the teacher evaluation systems and their experience and perspectives in either carrying out responsibilities associated with system design and implementation or in going through the evaluation process itself.

• Include additional details in the introduction to the interviews to indicate how long the interviews will last and provisions for protecting respondents’ privacy.

After making the necessary changes, we pilot-tested the protocols with a total of nine respondents, representing various roles at the state and district levels. Specifically, we interviewed

• Five teachers from a large urban district with a fully-operational teacher evaluation system

• Two principals from two different school districts. One district is a large urban district with a fully operational teacher evaluation system and the other is a district that is in the first year of implementing a new teacher evaluation system. Both systems feature the design characteristics envisioned by the Department

• Two district leaders responsible for leading the planning and implementation of new teacher evaluation systems. One person is from a large district and the other is from a small district that enrolls students from a medium-sized city and a surrounding rural area. This person also serves as a member of a statewide planning group that is advising the state education agency on the design of key features of local teacher evaluation systems.

• A member of a state board of regents who also serves as the vice-president of the state teachers association. Her state is implementing a statewide teacher evaluation system and in her union role she leads a six-district initiative fosters district-union partnerships to implement new teacher evaluation systems.

All interviews were conducted by telephone in November 2011. Consistent with overall plans for the case studies, interviewers conducted preliminary reviews of state and local documents to prepare for the interviews.

In general, the respondents did not have difficulty responding to questions about the particular elements of the respective teacher evaluation systems with which they were familiar. However, as we expected, not all respondents were familiar with all of the components of the systems and therefore could not answer some of the questions. For example, several respondents were unable to answer questions about the value-added measures used to rate teacher effectiveness.

Despite the fact that respondents did not have trouble answering the interview questions, several of their responses have led to the following modifications to the protocols.

• Each protocol now begins with the following question:

Which of the following do you consider to be the primary goal(s) of (name of teacher evaluation system)?

• Improving instruction

• Identifying effective teachers

• Identifying ineffective teachers

• Improving student learning

• Other goals

Rationale: Although documents describing teacher evaluation systems typically begin by describing system goals or purposes, asking this question of all respondents will help understand the extent to which there is consensus about system goals as well as areas of disagreement or uncertainty.

• Both the district staff and teacher focus group protocols now include the following question about system design and operation:

What options, if any, do teachers have to appeal ratings of their effectiveness?

Follow-up questions for respondents who indicate that teachers can appeal ratings of their effectiveness: What does the appeal process include and who serves as the final arbiter of the appropriateness of individual ratings?

Rationale: This modification was initially suggested by a principal, who noted that the decision to include an appeal process was an important step in the planning process and one which contributed to teacher buy-in to the new system. We note that this question also complements a question in the teacher focus group protocol about fairness of the new teacher evaluation systems as well as one in the district protocol that asks respondents about the appropriateness of the system for all teachers.

• The teacher focus group protocol now includes the following question about teacher perceptions about the impact of participating in the new teacher evaluation system.

How, if at all, has participating (name of teacher evaluation system) influenced your work as a teacher?

Rationale: The pilot-test version of this protocol asked teachers about the impact of participation on instruction. Several teachers interpreted this quite broadly and offered a range of examples of how participating in the system influenced them. The new question, which replaces the original question, is intended to elicit the broader range of responses that teachers may offer.

• The district staff and stakeholder protocol now includes a follow-up to Question 21 which asks about changes in the district that are attributable to the new teacher evaluation system:

Has implementation of (name of teacher evaluation system) resulted in any unexpected changes or consequences in the district or individual schools?

Rationale: One of the pilot test respondents offered an anecdote about how the new teacher evaluation system that is being implemented has led the district to revise some its school improvement planning requirements associated with setting school goals and priorities to accommodate the use of a schoolwide value-added measure. According to this respondent, the district had not anticipated the need to make this change but had concluded that it was important to both support the new evaluation process and to increase teacher buy-in.

B.5. Consultations on Statistical Aspects of the Design

Members of the study team who will be responsible for data collection and analysis are listed in Exhibit 4; although there will be no statistical analyses, the staff will conduct qualitative analyses.

Exhibit 4: Staff Contact Information

Name	Organization	Title	Telephone
Bruce Haslam	Policy Studies Associates	Managing Director	202-939-5333
Leslie Anderson	Policy Studies Associates	Managing Director	202-939-5327
Andrea Palmiter	Policy Studies Associates	Research Analyst	202-939-5332
Alisha Butler	Policy Studies Associates	Research Analyst	202-939-5318
Mariann Lemke	American Institutes for Research	Principal Research Analyst	202-403-5000
Meredith Ludwig	American Institutes for Research	Principal Research Analyst	202-403-5000
Peter Youngs	Michigan State University	Associate Professor	517-775-6791

References

Kane, T. (2009, October 30). Identifying effective teaching. Presentation made by Thomas Kane on behalf of the Bill and Melinda Gates Foundation at the Center for American Progress.

Mathers, C., Oliva, M., & Laine, S. (2008). Improving instruction through effective teacher evaluation: Options for states and districts. Washington, DC: National Comprehensive Center for Teacher Quality

National Council on Teacher Quality. (2011). 2010 state teacher policy yearbook: National summary. Washington DC: National Council on Teacher Quality.

U.S. Department of Education, Office of Planning, Evaluation and Policy Development. (2010). ESEA Blueprint for Reform, Washington, DC: author.

Weisberg, D., Sexton, S., Mulhern, J., & Keeling, D. (2009). The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness. New York: The New Teacher Project. Retrieved February 23, 2010, from http://widgeteffect.org/downloads/TheWidgetEffect_execsummary.pdf.

1 Typically, this will include two or more classroom observations and feedback from the observers, receipt of a value-added score, if such scores are included in the evaluation system, and receipt of a final performance rating. In some systems, the cycle may also include an initial goal setting activity and interim or benchmark progress reviews.

2 We will use the district staff protocol for interviews of these teachers. We will not ask them about their experiences in the evaluation process.