TSL OMB Part B

^{A nchor}

Design of an Impact Evaluation to Inform the Teacher and School Leader Incentive Program

Part B: Collection of Information Employing Statistical Methods

August 26, 2019

Submitted to:

U.S. Department of Education

Institute of Education Sciences

550 12th Street, S.W.

Washington, DC 20202

Project Officer: Elizabeth Warner
Contract Number: 91990018C004

Submitted by:

Mathematica Policy Research

P.O. Box 2393

Princeton, NJ 08543-2393

Telephone: (609) 799-3535

Facsimile: (609) 799-0005

Project Director: Phil Gleason
Reference Number: 50714

CONTENTS

PART B. SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION 1

Collection of information employing statistical methods 2

B1. Respondent universe and sampling methods 2

B2. Procedures for the collection of information 2

1. Statistical methods for sample selection 2

2. Data collection 3

3. Estimation procedures 5

4. Degree of accuracy needed 8

5. Unusual problems requiring specialized sampling procedures 9

6. Use of periodic (less frequent than annual) data collection cycles to reduce burden 9

B3. Methods to maximize response rates and deal with nonresponse 9

B4. Tests of procedures or methods to be undertaken 12

B5. Individuals consulted on statistical aspects of the design and on collecting and analyzing data 13

REFERENCES 14

TABLES

B.1 Minimum detectable effect sizes with 100 schools 9

B.2 Individuals consulted on statistical design 13

B.3 Individuals responsible for data collection and analysis 13

PART B. SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION

This package requests clearance for the initial data collection activities to support an evaluation of the Teacher and School Leader Incentive Program (TSL). The Institute of Education Sciences (IES), U.S. Department of Education (ED) has contracted with Mathematica and its partners Public Impact, Applied Engineering Management Corporation, Decision Information Resources, and Dr. Jason Margolis to conduct this evaluation.

To address a Congressional mandate to conduct an independent evaluation of the TSL program, this study will include two evaluation components.

• The first evaluation component will describe TSL grantees’ programs.¹ Data collection for this first component includes district interviews for all FY17 TSL grantees to obtain information on TSL grantees’ programs and experiences.

• The second evaluation component includes: (i) a random assignment study in non-TSL schools that will rigorously analyze the implementation and impacts of a teacher leader role, a common strategy among TSL grantees;² and (ii) data collection from TSL grantee schools, focusing on their implementation of the teacher leader role in order to connect the impact findings within the non-TSL schools to the TSL implementation experience. Data collection for the second evaluation component includes: teacher leader applicant background forms, teacher leader activity logs, school information questionnaires, and principal and school-day teacher surveys.

This package provides a detailed discussion of both evaluation components. However, the package only requests clearance for the district interview data collection to be conducted in the first evaluation component along with the initial data collection activities to be conducted in the second evaluation component—the teacher leader applicant background form and the school information questionnaire. A separate package will be submitted at a later date for all other instruments and administrative data collection activities.

Collection of information employing statistical methods

B1. Respondent universe and sampling methods

The study will examine two groups of districts and schools: (1) the universe of 2017 TSL grantee districts along with a random sample of schools within the TSL districts funding teacher leaders, and (2) a purposive sample of 100 schools across 10 school districts for the random assignment evaluation. The TSL grantee sample will include all 14 TSL grantees and the 25 districts covered by a 2017 TSL grant. For the 100 schools participating in the random assignment evaluation, we will match schools within each district into pairs prior to random assignment. The schools will be matched based on the similarity of their grade span, grades and subjects that they propose for teacher leader roles, and student characteristics. In spring 2020, we will randomly assign one school in each pair to the treatment group and one to the control group. The treatment group receives the teacher leader intervention, including funding and training for two teacher leaders for two years (2020-2021 and 2021-2022 school years). The control group will proceed with business-as-usual.

B2. Procedures for the collection of information

1. Statistical methods for sample selection

Below we explain the samples for the 2017 TSL grantees and for the random assignment evaluation.

a. 2017 TSL grantees

TSL grantee districts, and principals and teachers from schools covered by TSL grants, will be asked to complete surveys. Below we explain the samples for those surveys.

Selection of school districts for district grantee surveys for first evaluation component. We will not be sampling 2017 TSL school districts. We will ask all 25 districts covered by a 2017 TSL grant to participate in the district interviews.

Selection of schools for second evaluation component. We will randomly select 100 schools from the 23 TSL districts that received and are using TSL funds to support a teacher leader role. Each principal of these 100 schools will be asked to complete a principal survey.

Selection of teachers for second evaluation component. We will randomly select 2 eligible teachers from each of the 100 TSL-supported schools that were randomly selected to identify the principal sample. Eligible teachers will be those in the same grades as those implementing teacher leader roles for the random assignment evaluation (for example, 4th and 5th grades) and who teach math or ELA.

b. Component 2: a random assignment evaluation

The random assignment evaluation will include a purposive sample of approximately 10 districts that together include about 100 schools that are eligible for and willing to participate in the study. Below we explain in more detail how we will select districts, schools, teachers and students for the study.

Selection of school districts. We will recruit districts and schools that meet the study eligibility criteria. Eligibility requirements include: schools that are high-need, the district uses a teacher evaluation system based in part on student learning, and the school does not have teacher leader or teacher support systems in place that is similar to the study’s teacher leader model, which relies on frequent and systematic observation and coaching to improve instructional practices. We will recruit, on average, 10 schools from each district. Based on the Common Core of Data and National Council on Teacher Quality (2018), 358 districts in 29 states meet these criteria. This will result in a purposive sample of districts that are willing to participate and schools from within those districts that are willing and eligible to participate. Although we will not be able to generalize to all schools, we will obtain valid estimates of the impact of the intervention for a policy-relevant sample of schools that meet our eligibility requirements and are willing to participate.

Selection of schools. Within the participating districts we will invite eligible schools to participate in the study. We will include 100 schools for the 2020–2021 and 2021-2022 school years. Schools eligible and agreeing to participate in the evaluation will be randomly assigned to the treatment or control group as described in section B1 above.

Selection of teachers. Within the treatment schools, we will include all teachers on the two teacher leaders’ teams. This includes the teacher leaders and the 2 to 5 other teachers in their same grade or grade/subject. Within the control schools, we will include all teachers in the same grades and subjects as the treatment schools. Teachers will participate in the study for the 2020–2021 and 2021-2022 school years.

Selection of students. To estimate impacts on student outcomes after one year of the intervention, we will define the student analysis sample as the students who are on study teachers’ rosters at the beginning of the first intervention school year (fall 2020). To estimate the second year impacts, we will define the student analysis sample as the students enrolled in study schools in fall 2020 who are projected to be in the grades covered by the study teams in fall 2021, along with the new students assigned to study teachers in fall 2021. The study team will have access to administrative data on student characteristics and test scores through a Memorandum of Understanding (MOU) established with each participating district.

2. Data collection

This study includes multiple data collection efforts summarized here. At this time, we are only requesting clearance for the district interview protocol, teacher leader application form, and the school information questionnaire. The grantee interview has been completed and all other data collection activities will be part of a separate OMB package and are described here only for context in understanding the full study.

a. Data collection under current clearance request

District interviews. To describe the program features, perceptions, and challenges of TSL grantees, we will conduct one-hour interviews in spring 2020 and spring 2021 with all 25 districts covered by a 2017 TSL grant (Appendix A). We will collect standardized information related to the components of each grantee district’s human capital management system and the strategies the district is using to improve educator effectiveness. The interview in 2021 will feature many of the same topics as in 2020, but will focus on experiences since the first year interview.

Teacher leader application form. To examine the extent to which teachers selected for the teacher leader roles were the applicants with the strongest teaching and coaching skills, principals in treatment schools will complete a teacher leader application form (Appendix B) for each teacher leader applicant. This hardcopy form will be collected in spring and summer 2020 (and in spring and summer 2021 if any teacher leaders are replaced after the first study year) and will take 30 minutes to complete across all candidates in each school. The form will collect candidates’ background characteristics, principals’ ratings of candidates’ written applications, in-person responses to interview questions, and demonstration of coaching and feedback skills and candidates’ scores on their district’s evaluation system.

School information questionnaire. Prior to random assignment in spring 2020, all study school principals will be asked to complete a thirty minute, hardcopy school information questionnaire (Appendix C). The questionnaire will be used to collect details on high-priority teachers for coaching, potential teacher leaders, potential grade (or grade-subject) for teacher leader teams, schools’ staffing structure (departmentalized or self-contained), the number of teachers in each grade-subject, and the number of expected vacancies. These data will be used to help create matched pairs for random assignment and to assess whether teacher leader roles improve the effectiveness of high-priority teachers, such as novice and low-performing teachers.

b. Data collection under future clearance request

Teacher leader activity logs. To monitor and describe teacher leader activities, teacher leaders will complete a weekly web-based checklist during the 2020-2021 and 2021-2022 school years to document the types of leadership activities conducted and the teachers they supported. For each 30 to 60 minute block of time per week that teacher leaders were released from their classroom, teacher leaders will indicate what they did, whom they supported (specific teachers, the full team, a subset of the team), and the focus of the activity. It is expected that teachers will spend 10 minutes per week completing the log, for a total of 5.3 hours each school year.

Principal surveys. We will administer a 30-minute web-based survey in spring 2021 to all principals of treatment and control schools and 100 randomly selected schools that receive TSL funds to support teacher leader roles. We will also survey all principals of treatment and control schools in spring 2022. The principal survey will collect information related to the types and frequency of coaching, mentoring, common lesson planning, and professional development in the principal’s school, strategies for recruiting and retaining teachers, activities of teacher leaders, and principal satisfaction.

Teacher surveys. To learn about teachers’ perspectives and practices, we will administer a 30-minute web-based survey in spring 2021 to teachers in treatment and control schools and 100 randomly selected schools that receive TSL funds to support teacher leader roles. We will administer a second round of the teacher survey in spring 2022 to teachers in the treatment and control schools. The teacher survey will ask about demographic and background characteristics, the types and frequency of support, coaching, mentoring, and professional development teachers receive or provide.

District administrative records on teachers. In fall 2022, we will collect district administrative data on teachers in treatment and control schools. We will collect teachers’ district evaluation scores from the year before the intervention (the 2019–2020 school year) to assess whether principals chose teacher leaders with strong evidence of effectiveness, examine whether teacher leader roles enabled schools to attract and retain more effective teachers, and to examine whether impacts are greater for teacher teams led by teacher leaders with stronger prior effectiveness as a teacher. We will collect district data on teachers’ assignments to examine the impact of the retention of effective teachers. To describe the study sample, we will also collect district data on teachers’ demographic characteristics.

District administrative records on students. To estimate the impact of teacher leaders on student achievement, we will collect administrative data on students from each district participating in the random assignment evaluation. These data include state test score data in reading and math from the end of each implementation year (2020-2021 and 2021-2022), as well as student test scores from the baseline year (2019–2020). In order to assess the impact of teacher leaders on the achievement of specific teachers’ students, we will obtain student achievement data linked to their teachers. We will also collect information on students’ demographic and socioeconomic characteristics (for example, gender, age, and free and reduced-price lunch eligibility) to describe the students in the study and compare the characteristics of students in treatment and control schools.

3. Estimation procedures

This study will conduct analyses for an implementation brief for the first evaluation component and a final report for the second evaluation component. This submission requests clearance for collecting data that will be used for the implementation brief. We include a description of the analyses for the final report (based on data collection instruments for which we will request clearance in a later submission), only for context in understanding the full study.

Analyses for implementation brief. The implementation brief will describe TSL grantee districts and their programs. A key goal of this part of the study is to enhance our understanding of TSL-supported programs and their experiences. We will describe several aspects of grantee districts and their programs. For example, we will describe the location and size of grantee districts, the types of activities supported by TSL grants, and districts’ challenges in implementing and sustaining TSL activities.

Analyses for final report. The final report includes four broad sets of analyses: (1) impact analyses, estimating the effects of the study interventions on student and educator outcomes; (2) subgroup analyses, estimating the effects of the interventions on various subgroups of interest; (3) nonexperimental analyses, including correlational analyses to examine how features of the teacher leader roles may be related to student achievement, and a cost-effectiveness analysis; (4) implementation analyses of the teacher leader program in evaluation sites.

Impact analyses. We will use regression models to estimate the impact of teacher leader roles on principal, teacher, and student outcomes after each year of implementation. Key outcomes include, principals’ strategies for recruiting teachers, the retention and recruitment of effective teachers, principals’ and teachers’ satisfaction, and students’ math and ELA achievement.

To estimate impacts on student, teacher, and principal outcomes, we will use the following regression model:

(1) ,

where y_isb is the outcome for individual i (either student, teacher, or principal) in school s and random assignment block b; T_sb indicates whether the school was assigned to the treatment group to receive funding for up to two teacher leaders; X_isb is a set of student-level covariates (included in the student analyses only); Z_b is a set of indicators for the study’s random assignment blocks (matched pairs of schools); ε_isb is an individual-level error term; and γ and δ are vectors of coefficients. The coefficient β represents the average impact of the teacher leader roles. We will estimate robust standard errors that account for the clustering of outcomes within schools in the teacher and student analyses.

For the student-level regressions, the covariates will include baseline math and reading achievement, student demographic characteristics (such as gender, race, ethnicity, free or reduced-price lunch eligibility, special education status, and Limited English proficiency). We do not plan to include covariates in the principal- or teacher-level regressions because these analyses will include smaller numbers of individuals, and we expect that once we include the block fixed effects, these covariates will explain little of the variation in outcomes.

In our main analysis, we will give equal weight to each school (regardless of the number of students or teachers in the analysis sample in each school). Estimates that give equal weight to each school will reflect the impact of teacher leader roles on the average school in the sample. However, as a sensitivity test, we will also estimate the impacts of the teacher leader role for the average teacher or student by giving equal weight to each student or teacher in the analysis.

We will use the standard (frequentist) statistical approach to estimate the impact of teacher leader roles (β) along with the standard error, statistical significance, and p-value of this estimate. However, to avoid common misinterpretation of p-values (Wasserstein and Lazar 2016; Greenland et al. 2016), we will complement these estimates with a Bayesian measure of the probability that the intervention had a positive effect.

Subgroup analyses. To help districts and schools identify which teachers and students might benefit the most from teacher leader roles, we will estimate impacts for subgroups defined by teacher and student characteristics, including:

1. Teachers identified by their principals (prior to random assignment) to have the greatest potential to benefit from the coaching

2. Novice teachers (less than 3 years of teaching experience) and those with low baseline evaluation scores (as measured by their district evaluation system)

3. Teacher leaders (those chosen to be a teacher leader in a treatment school and those identified prior to random assignment by principals of control schools as likely to be chosen to be a teacher leader)

4. Students with high-needs (based on their eligibility for free or reduced-price lunch) or low baseline achievement (below median achievement in the district’s study schools at baseline)

Non-experimental analyses. We will conduct non-experimental analyses to better understand the effectiveness of teacher leader roles and their potential application for policy and practice.

1. Correlational analyses. To help districts and schools decide how to adopt or refine their teacher leader roles, we will conduct several correlational analyses to explore how various characteristics of the teacher leader roles might be related to the impacts of the teacher leader on student achievement. In particular, we will examine whether estimated impacts tend to be larger in matched pairs of schools in which the treatment school teacher leader has particular characteristics (such as having better baseline evaluation scores) or engages in particular activities (such as doing more modeling of teaching practices).

2. Assess cost-effectiveness of teacher leader roles. If we find a positive impact on student achievement, districts choosing how to spend limited resources will benefit from knowing whether this strategy is cost-effective relative to other policies. The cost of implementing teacher leader roles includes (1) teacher leader stipends, (2) compensation for teachers who cover teacher leader’s release time, (3) support to help principals select teacher leaders, and (4) initial and follow-up training for teacher leaders and their principals. We will calculate a cost-effectiveness ratio—total per-student cost of this strategy divided by its total impact—and compare it to the ratios for other policies that have been rigorously evaluated and that districts could implement instead of teacher leader roles, such as pay-for-performance or incentives to work in low-performing schools. Thus, these findings will yield lessons about the most promising components to emphasize in human capital management systems.

Implementation analyses for the second evaluation component. Two important goals of the implementation analyses are to highlight overarching lessons for the TSL program and to provide context for the impact results. We will conduct several implementation analyses. First, we will compare TSL grantee and evaluation districts, including district, principal and teacher characteristics and aspects of their teacher leader roles. This comparison will help policymakers understand the extent to which findings based on the schools in the random assignment evaluation may be relevant for TSL grantee districts. Second, we will describe implementation of teacher leader roles in the treatment schools, such as teacher leader qualifications and characteristics, the composition of teacher teams, and how teacher leaders spend their time. A detailed understanding of the teacher leader roles implemented will provide important information for districts considering implementing similar teacher leader roles, support replication of these roles in other districts, and provide context for impact findings. Third, we will describe and compare the amount and types of supports teachers in treatment and control school receive, such as teachers’ exposure to coaching and mentoring. A clear description of what the control group schools are doing compared to the treatment schools can help identify possible reasons for the presence or absence of impacts.

4. Degree of accuracy needed

In the analysis for the implementation report, we will conduct interviews with all 2017 TSL grantee districts. Thus, the resulting descriptive statistics based on data from these interviews will cover the relevant universe and not be subject to sampling error.

In the analysis contributing to the impact report (and based on data collection instruments for which we will request clearance in a later submission), the target sample size of 100 schools will enable the study to detect impacts on student test scores as small as 0.11 standard deviations in the full sample—students of teacher teams in treatment schools and those from the identical grades and subjects in the paired control schools (Table B.1). This effect is smaller than the average effect of coaching (0.18) documented in a meta-analysis by Kraft et al. (2018), so the evaluation will be able to detect effects equal to or even a bit smaller than the typical coaching intervention. Among students of high-priority teachers—about half of the sample in each school—the minimum detectable effect will still be 0.11. For teacher satisfaction and retention, the minimum detectable impacts of 11 and 10 percentage points are similar to the average effects of other teacher-focused interventions (Chiang et al. 2017 and Clotfelter et al. 2008).

Table B.1 displays the minimum detectable effect sizes for the full sample of teachers as well as a 50 percent subsample. The full sample will include 100 schools, 50 in the treatment group and 50 in the control group. The study design will maximize power to detect impacts by matching schools with similar characteristics into blocks within each district. The characteristics used to match schools will include grade span, grades and subjects that they propose for teacher leader roles, and average school baseline performance. We will also use students’ prior test scores as covariates in the impact analysis to increase statistical power.

The calculations in Table B.1 assume the following: (1) 80 percent power and 5 percent significance level for a two-tailed test; (2) an average of 8 teachers per school and 22 students per teacher; (3) a response rate of 95 percent for district records and 85 percent for the teacher survey; (4) 87 percent of teachers receive the top two evaluation scores; and (5) a sample of 100 schools, with schools split evenly between the treatment and control groups. For student achievement, the calculations assume that 16 percent of the variation in outcomes occurs across schools, and 80 percent of the cross-school variation and 40 percent of the within-school variation can be explained by covariates (Deke et al. 2010). For teacher satisfaction and retention outcomes, assumptions on school-level clustering of outcomes and explanatory power of covariates come from Wellington et al. (2016). For teacher retention, assumptions on the percentage of teachers that receive the top two evaluation scores come from Garet et al. (2017).

Table B.1. Minimum detectable effect sizes with 100 schools

Outcome	Data source	Units	Minimum detectable effect
			Full sample	50 percent subsample of schools	50 percent subsample of teachers in all schools
Students’ reading and math test scores in spring 2021	District records	Standard deviations	0.11	0.15	0.11
Percentage of teachers who are satisfied with their jobs in spring 2021	Teacher survey	Percentage points	11	16	14
Retention of effective teachers (based on baseline evaluation scores) from spring 2020 to fall 2021	District records	Percentage points	10	15	14

5. Unusual problems requiring specialized sampling procedures

We do not anticipate any unusual problems that require specialized sampling procedures.

6. Use of periodic (less frequent than annual) data collection cycles to reduce burden

In order to limit respondent burden as much as possible, we have carefully considered what the minimum amount of data is needed to answer the research questions and how to structure the data collection. For example, we will request administrative data no more than once a year, and whenever possible, we will request multiple years of data within a single request to reduce the number of separate requests.

B3. Methods to maximize response rates and deal with nonresponse

The study will employ multiple strategies to maximize response rates while minimizing burden on respondents. The following only refers to methods to deal with nonresponse related to the instruments requesting clearance at this point. Strategies include establishing positive relationships with respondents and school and district staff; sending letters to respondents to alert them to upcoming requests to complete the surveys. To reassure respondents on the confidentiality of the data they provide, we will include a statement on confidentiality and data collection requirements (Education Sciences Reform Act of 2002, Title I, Part E, Section 183) in all letters and data collection instruments. Finally, we will include a reminder for TSL grantee districts that their participation in the district interview for this ED-sponsored study is expected as a condition of their grant. For other respondents on other data collection instruments, we will include a statement indicating that participation is voluntary, yet we will also emphasize the importance of each response for the study findings.

District interviews. Flexibility in scheduling and conducting district interviews will help us obtain high response rates. Members of the study team will identify the contact at each district best suited to respond to the interview protocol. Initial emails will be sent to each contact to identify a time that would be most convenient for their schedule. Reminder emails will be sent to non-responders, highlighting the importance of their participation and our flexibility to meet their scheduling needs. The use of a semi-structured interview protocol will allow flexibility in focusing on key concepts and themes, while staying within the confines of the one-hour interview. We anticipate that these qualities, in addition to ED support, will facilitate a response rate of 100 percent.

Teacher leader application form. Teacher leader application forms will be distributed to principals in all treatment schools as an attachment sent via email. Principals will complete the form for each teacher leader applicant and return to the study team through an upload to the study’s secure file transfer site (due to teacher PII collected on the form). The study team will coordinate with principals throughout the fielding period to confirm that all forms are being returned in a timely manner via periodic email and phone check-ins. Utilizing the communication channels and relationships necessary for implementation of the study intervention, as well as the development of a succinct form, we anticipate a response rate of 100 percent. The study team will identify nonresponse and reporting errors by checking for complete and reasonable answers as soon as forms are received and will follow-up with respondents as needed for correction.

School information questionnaire. School information questionnaires will be distributed to principals in all study schools prior to random assignment, as an attachment sent via email. Principals will complete the form and return to the study team through an upload to the study’s secure file transfer site (due to teacher PII collected on the form). The study team will coordinate with principals throughout the fielding period to confirm that questionnaires are being returned in a timely manner via periodic email and phone check-ins. Utilizing the communication channels and relationships necessary for implementation of the study intervention, as well as the development of a succinct form, we anticipate a response rate of 100 percent.

To ensure high quality data, we will pilot the instrument during the recruitment of the earliest study sites. A traditional pretest with similar sites would not accurately identify issues or further revisions needed for the forms, as the data collected will be very specific to study participation. We do not anticipate needing to make adjustments to the form that would impact burden, but may need to make some revisions to enhance the usefulness of the data for the study. We expect that the instrument will take 30 minutes to complete. The study team will identify nonresponse and reporting errors by checking for complete and reasonable answers as soon as forms are received and will follow-up with respondents as needed for correction.

B4. Tests of procedures or methods to be undertaken

We did not pretest the school information questionnaire, but will pilot the instrument during the recruitment of the earliest study sites. A traditional pretest with similar sites would not accurately identify issues or further revisions needed for the forms, as the data collected will be very specific to study participation. We do not anticipate needing to make adjustments to the form that would impact burden, but may need to make some revisions to enhance the usefulness of the data for the study.

B5. Individuals consulted on statistical aspects of the design and on collecting and analyzing data

The following individuals were consulted on the statistical aspects of the study:

Table B.2. Individuals consulted on statistical design

Name	Title	Telephone Number
Hanley Chiang	Senior Researcher, Mathematica	617-674-8374
Phil Gleason	Associate Director, Human Services Research and Senior Fellow, Mathematica	202-264-3443
Kristin Hallgren	Senior Researcher, Mathematica	609- 275-2397
Jason Margolis	Professor of Education, Duquesne University	412-396-6106
Alison Wellington	Senior Researcher, Mathematica	202-484-4696

The following individuals will be responsible for data collection and analysis:

Table B.3. Individuals responsible for data collection and analysis

Name	Title	Telephone Number
Julie Bruch	Senior Researcher, Mathematica	617-301-8964
Megan Davis-Christianson	Lead Program Analyst, Mathematica	609-275-2361
Phil Gleason	Associate Director, Human Services Division and Senior Fellow, Mathematica	202-264-3443
Kristin Hallgren	Senior Researcher, Mathematica	609- 275-2397
Alicia Harrington	Survey Researcher, Mathematica	609-750-3193
Mariesa Herrmann	Senior Researcher, Mathematica	609-716-4544
Jason Margolis	Professor of Education, Duquesne University	412-396-6106
Alison Wellington	Senior Researcher, Mathematica	202-484-4696
Eric Zeidman	Associate Director, Human Services Division and Senior Survey Researcher, Mathematica	609-936-2784

REFERENCES

Chiang, Hanley, Cecilia Speroni, Mariesa Herrmann, Kristin Hallgren, Paul Burkander, and Alison Wellington. “Evaluation of the Teacher Incentive Fund: Final Report on Implementation and Impacts of Pay-for-Performance Across Four Years.” NCEE 2017-4004. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, 2017.

Clotfelter, Charles T., Elizabeth Glennie, Helen F. Ladd, and Jacob L. Vigdor. “Would Higher Salaries Keep Teachers in High-Poverty Schools? Evidence from a Policy Intervention in North Carolina.” Journal of Public Economics, vol. 92, no. 5-6, 2008, pp. 1352-1370.

Deke, John, Lisa Dragoset, and Ravaris Moore. “Precision Gains from Publically Available School Proficiency Measures Compared to Study-Collected Test Scores in Education Cluster-Randomized Trials.” NCEE 2010-4003. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, 2010.

Garet, Michael S., Andrew J. Wayne, Seth Brown, Jordan Rickles, Mengli Song, and David Manzeske. “The Impact of Providing Performance Feedback to Teachers and Principals.” NCEE 2018-4001. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, 2017.

Kraft, Matthew A., David Blazar, and Dylan Hogan. “The Effect of Teacher Coaching on Instruction and Achievement: A Meta-Analysis of the Causal Evidence.” Review of Educational Research, vol. 88, no. 4, 2018, pp. 547–588.

Wellington, Alison, Hanley Chiang, Kristin Hallgren, Cecilia Speroni, Mariesa Herrmann, and Paul Burkander. “Evaluation of the Teacher Incentive Fund: Implementation and Impacts of Pay-for-Performance After Three Years.” NCEE 2016-4004. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, August 2016.

1 Although a prior study (Chiang et al. 2017) provided information on the 2010 Teacher Incentive Fund (TIF) grantees’ programs- a precursor program to TSL, the program has evolved since then. This component of the study will update our knowledge of the current grantees’ activities.

2 The rationale for including non-TSL schools is two-fold. First, since most TSL grantees are currently funding teacher leader positions and have already started implementing their TSL programs (including teacher leader roles), rigorously assessing the impact of the strategy in the context of the grants is not feasible. In recent years, the teacher leader role also has grown as a strategy funded by Title II, Part A in ESSA. Thus, learning about the effectiveness of the teacher leader role strategy in a more general context is useful for TSL and other policy.