MTSS-B OMB Part B

Impact Evaluation of Training in Multi-tiered Systems of Support for Behavior (MTSS-B)

OMB Clearance Request:

Data Collection

Supporting Statements B

April 2015

Prepared for:

Institute of Education Sciences

United States Department of Education

Contract No. ED-IES-14-C-003

Prepared By:

MDRC

16 East 34^th Street, 19^th Floor

New York, NY 10016

Fred Doolittle, Project Director

[email protected]

(212) 340-8638

TABLE OF CONTENTS

INTRODUCTION 3

THEORY OF ACTION AND RESEARCH QUESTIONS 4

CHARACTERISTICS OF THE TREATMENT TO BE TESTED 5

STUDY DESCRIPTION 9

SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT

SUBMISSION 17

B. DESCRIPTION OF STATISTICAL METHODS 17

1. Respondent Universe and Sampling Methods 17

2. Procedures for Data Collection 20

3. Procedures to Maximize Response Rates 27

4. Testing of Data Collection Procedures 28

5. Individuals Consulted on Statistical and Methodological Aspects

of Data Collection 29

REFERENCES 30

APPENDICES

1. Site Visit Interview Protocols: Administrators

2. Site Visit Interview Protocols: Behavior Team Leader

3. Site Visit Interview Protocol: Students

4. Site Visit Interview Protocol: Staff

5. Phone Interview: MTSS-B Coach

6. Phone Interview: Administrator and Behavior Team Leader

7. Staff and Teacher Survey

8. Student Survey

9. Parent Informed Consent Form

10. Teaching Ratings of Student Behavior

11. District Records Data Collection Request Letter

12. Excerpt from IDEA, P.L. 108-446

INTRODUCTION

Creating and maintaining safe and orderly school and classroom environments to support student learning is a top federal priority, as indicated by the U.S. Department of Education and the U.S. Department of Justice jointly issued guidance on school discipline. The implementation of Multi-Tiered Systems of Support for Behavior (MTSS-B) is seen as a promising approach to improve school and classroom climate, student behavior, and academic achievement, and potentially reduce the inappropriate identification of students for special education. MTSS-B is a multi-tiered, systematic framework for teaching and reinforcing positive behavior for all students that includes universal, school wide supports (often called Tier I), targeted interventions to address the problems of students not responding to the universal components (often called Tier II), and intensive interventions (Tier III) for students not responding to Tier II services.

The widespread use of MTSS-B nationally contributes to the relevance of this evaluation. The U.S. Department of Education’s (ED’s) Office of Special Education Programs has supported technical assistance around MTSS-B since 1998. Recent data from the Office of Special Education Programs TA Center on Positive Behavior Interventions and Support indicate that as of 2014, MTSS-B is reportedly implemented in over 21,000 schools across the nation.¹

As the support for and implementation of MTSS-B have increased, researchers have been developing an evidence base for it and other related programs. Numerous descriptive and correlational studies have associated aspects of MTSS-B implementation, or MTSS-B implementation overall, with student outcomes. More recently, researchers have begun to conduct causal studies of MTSS-B, although some of these initial studies have been on a relatively small scale (or conducted within a single state) or have been somewhat limited in duration and outcomes.² Much of this research has focused on individual programs and not examined the impact of a large-scale implementation of both universal, school-wide programs and targeted programs within an MTSS-B framework.

Therefore, this evaluation focuses on a comprehensive MTSS-B program that includes core MTSS-B components: school-wide and classroom-level strategies (universal supports, or Tier I) and individual strategies (targeted intervention or Tier II), with appropriate infrastructure (staffing sand data system). This evaluation will not support intensive intervention for students not responding to Tier II interventions (i.e., there will be no training in Tier III interventions). The goal of the evaluation is to implement an MTSS-B program with fidelity and determine whether universal supports (Tier I) and a targeted intervention (Tier II) with appropriate infrastructure are effective when implemented in a large number of districts and elementary schools. IES has contracted with MDRC, AIR, Harvard University, and Decision Information Resources (DIR) to conduct the Impact Evaluation of MTSS-B.

Using a school-level random assignment design, this evaluation will examine the impact of training elementary school staff in the implementation of an MTSS-B framework including staff and data infrastructure with universal, school-wide components, as well as classroom-based activities (Tier I) in the first year of implementation, as compared to outcomes for schools undertaking typical behavior support practices. These schools are hereafter called business-as-usual or “BAU schools.” In addition, the study will also examine the impact of this Tier I training plus additional training in a targeted intervention (Tier II) in a second year of implementation. Schools that receive MTSS-B training and support are hereafter referred to as “Program schools.” The study will also investigate the differences in behavioral support services between program and BAU schools. Finally, the evaluation team will assess whether the MTSS-B training and support activities are implemented as intended, as well as the fidelity of schools’ implementation of the intended MTSS-B Tier I and Tier II activities.

THEORY OF ACTION & RESEARCH QUESTIONS

The implementation of MTSS-B is ultimately intended to improve student outcomes. The conduct of school-wide and classroom strategies (Tier I) and individual or small group strategies (Tier II), supported by appropriate infrastructure, is designed to improve school staff behavior practices as well as school and classroom climate. The improved practices and climate are, in turn, hypothesized to benefit student behavior and academic outcomes. The individual and small group strategies may affect the students who need additional support (Tier II) as well as all students if these strategies lessen the disruption of students needing additional support. Finally, improved student behavior could lead to better student academic outcomes (see Exhibit A-1).

Exhibit A-1: Conceptual Model of MTSS-B and Intended Outcomes

The evaluation is anchored on this conceptual model and will focus on the following research questions:

• What MTSS-B training and support activities were conducted? What MTSS-B activities occurred in the schools receiving MTSS-B training? How do these MTSS-B activities differ from those in schools that do not receive the training?

• What is the impact on school staff practices, school climate and student outcomes of providing training in the MTSS-B framework plus universal (Tier I) positive behavior supports and a targeted (Tier II) intervention?

• What are the impacts for relevant subgroups (e.g., at-risk students)?

CHARACTERISTICS OF THE TREATMENT TO BE TESTED

• In the spring of 2014, MDRC and partner organizations issued a Request for Proposals to providers of training in MTSS-B. In November 2014, we announced the winner of this competition, The Center for Social Behavior Support (CSBS), which is a collaboration between The Illinois-Midwest PBIS Network at the School Association for Special Education in DuPage, Illinois (SASED) and the PBIS Regional Training and Technical Assistance Center at Sheppard Pratt, in Maryland. The training and support activities proposed by CSBS are described below. Following that summary, the memo describes CSBS’s plan for how the MTSS-B Tier I and Tier II program will be implemented in schools.

CSBS Training and Support Activities

• CSBS will mostly follow a “train the trainer” model with some direct support to all school staff and on-site support to the school staff and Coaches throughout implementation. Below we describe the key phases of their work with schools.

Readiness Period - Training and Support by CSBS to Develop Infrastructure: The first step is the development of school-level infrastructures that will support implementation of Tier I and Tier II program components, which include a staffing structure and data system. Tier I infrastructure will be put in place in the late spring of 2015 for all Program schools. Infrastructure for Tier II will be developed in the spring of 2016.

Staffing structure: CSBS trainers will work with each school to identify a representative School Leadership Team (SLT)³ and each district to identify MTSS-B Coaches. This will be accomplished during a two-day site visit to each district as well as webinars, email exchanges and phone calls. In addition to helping schools and districts identify the SLT and MTSS-B Coaches, CSBS will explain the MTSS-B program to the district coordinator, administrator, SLT and MTSS-B Coaches, as well as the roles and functions of each individual. During the spring of 2016, CSBS will work with schools to identify a Targeted Team (TT) to support Tier II implementation. Often, the TT is a subset of the SLT.

Behavioral monitoring data system: During the spring of 2015, CSBS trainers will support the SLT in putting in place the data infrastructure that relies on a web-based behavior monitoring data system called the School Wide Information System (SWIS). Minimally three SLT members and MTSS-B Coaches will be trained to use the SWIS data system to drive decision-making about issues such as which teachers may need additional supports with classroom management, and which locations in the school at what time may need more school staff to monitor student behavior. Appropriate staff will be identified and trained to enter data and generate reports. The data infrastructure will be augmented in the spring 2016 to include the data system for Check In Check Out (CICO), the primary Tier II intervention.

• Initial and Booster Summer Trainings: During the summer of 2015, CSBS will host training within each district for the SLT, MTSS-B Coaches and administrators in all Program schools. The goal of the initial CSBS Tier I training during the summer of 2015 is to introduce the key school-level implementers (MTSS-B Coaches, SLT and administrators) to the core features of MTSS-B Tier I,⁴ as well as the tools to track implementation fidelity and engage in data-based problem solving. During the summer training each school team drafts a school-level implementation plan, and each school team should leave the summer training prepared to further develop and implement the school-level implementation plan. During the summer of 2016, the key Tier I implementers engage in a booster training in which they review core components, reflect on the previous year and develop a year 2 school-level implementation plan. During this booster training, new school staff are also introduced to the core components of MTSS-B Tier I. Initial summer training for Tier II also takes place in the summer of 2016 and includes training for the MTSS-B Coaches, Targeted Team and administrator in the core features of Check In Check Out⁵ and how to use CICO-SWIS data to monitor student progress. The goal of this initial training is for schools to develop and implement their CICO implementation plan.

Ongoing Training and Support: During the 2015-2016 and 2016-2017 school years, CSBS will provide ongoing support to the MTSS-B Coaches, SLT and administrators and some direct trainings to all school staff as schools implement Tier I activities. CSBS trainers will visit all Program schools four times over the course of the school year. During these visits, they will work with the SLT, administrator and MTSS-B Coaches to introduce Tier I core components to staff and to train staff in their version of the Good Behavior Game, referred to as the Positive Behavior Game (PBG), which helps teachers to teach and acknowledge classroom rules aligned with the school-wide expectations. Additionally, they will gather and use data regarding schools’ implementation fidelity to help the SLT to create and modify schools’ implementation plans.⁶ During site visits, monthly webinars and email/phone exchanges, CSBS will support Coaches and the SLT to train teachers in the eight CSBS classroom components.

CSBS’s Eight Classroom Components

1. Arrange orderly physical environment

2. Define, teach, and acknowledge rules aligned with school-wide expectations by modeling desired behaviors and using a gaming strategy;

3. Explain and teach routines;

4. Provide specific and contingent praise for appropriate behavior;

5. Use class-wide group contingencies;

6. Provide error correction through prompt, re-teach, and provision of choices;

7. Employ active supervision – move, scan, interact;

8. Provide multiple opportunities to respond.

CSBS’s provision of ongoing support for the Tier II intervention during the 2016-2017 school year will follow the same model as their provision of ongoing support for Tier I. However, CSBS will conduct an extra site visit during year 2 of the study so they have sufficient time to provide ongoing support for the implementation of Tier I and Tier II.

MTSS-B Program Implementation

The key components of the Tier I and Tier II MTSS-B program evaluated in this study are described in the paragraphs below.

Tier I Practices: The school-level implementation of Tier I practices will occur in all Program schools during both the 2015-2016 and 2016-2017 school years. These practices include school-wide activities as well as specific classroom components that are overseen by the SLT and MTSS-B Coaches.

Leadership provided by SLT and MTSS-B Coaches: The SLT and MTSS-B Coaches are the lead implementers of the program and are responsible for introducing the program model and implementation plan to other staff members with support from CSBS trainers. They meet regularly to monitor implementation of the school action plan and, based on data collected on student behavior and implementation fidelity, provide support to other school staff to improve implementation.

Ongoing support for teachers by the SLT and MTSS-B Coaches: The SLT and MTSS-B Coaches provide ongoing training and coaching to teachers. These supports include collection of data on teacher performance, as well as the provision of targeted feedback to teachers and opportunities to practice the Positive Behavior Game.

School-wide implementation of program: A key component of the Tier I practices of MTSS-B is that the program is implemented school-wide and not just by particular administrators, teachers, or staff members. All school staff will be responsible for implementing the school’s 3-5 behavioral expectations, responding appropriately to positive and problem student behavior, and using student behavior data to develop solutions to specific problems.

Classroom components: Within the classroom, teachers establish and maintain a classroom management system, which includes eight specific components (see text box on page 9).

Tier II Practices: Tier II will begin in all Program schools during the 2016-2017 school year. During this period, the SLT and MTSS-B Coaches will review behavioral monitoring data to identify students who are not responding to Tier I supports. They will also solicit referrals from school staff and families. These students will receive the Check In Check Out (CICO) intervention, which is the Tier II intervention in this study.

CICO is a small group intervention that reinforces and extends the universal expectations by systematically providing a higher frequency of scheduled prompts, pre-correction, and acknowledgement to participating students. The program consists of students daily “checking in” with an adult at the start of school day to retrieve a goal sheet and receive encouragement and help in setting specific behavioral goals (including skills to practice) for the day. Throughout the day, teachers (or other school staff) provide feedback to the student and document student behavior on the goal sheet. Students “check out” at the end of the day with an adult, who summarizes the feedback and recognizes accomplishments. The Targeted Team (TT), usually a subset of the SLT, monitors students’ progress on CICO and establishes systems of communication with families.

STUDY DESCRIPTION

Study Design

School level random assignment is appropriate for this study because by definition, MTSS-B is an intervention delivered at the school level. The analysis of MTSS-B impacts will rely on a random assignment design in which schools within each participating district (approximately 9) are randomly assigned to the following two groups:

1. Program Schools: Training, infrastructure and support to implement Tier I components in year 1 (SY2015-2016) and Tier I and Tier II components in year 2 (SY2016-2017).

2. Business As Usual (BAU) Schools: Continue with any existing practices related to student behavior, and will not receive any additional training and support in MTSS-B from the evaluation.

First year impacts of Tier I will be calculated by comparing first year outcomes in all 58 Program schools implementing Tier I to first year outcomes in the 31 BAU schools. The second year impacts of Tier I plus II will be calculated by comparing second year outcomes in the Program schools to second year outcomes in the BAU schools. A baseline teacher assessment of student behavior (in fall 2015) allows for the identification of a high risk subgroup and the estimation of the net impact for high risk students of Tier I (year 1) and Tier I plus II services (year 2) compared to BAU schools.

The study team has recruited 89 elementary schools across approximately nine districts that are willing and able to comply with the requirements of MTSS-B training providers and the research and data collection procedures involved in the evaluation. Decisions regarding how many schools and districts to recruit were based upon desired minimum detectable effect sizes (MDES) for each outcome. We worked with IES to set the target minimum detectable effect size (MDES) based on reasonable and policy relevant expected impacts on various types of outcomes for this kind of intervention. We then calculated the required sample size for the target MDES, using parameter assumptions based on a review of existing literature.⁷

The impact study will examine several outcomes of MTSS-B – teacher practice, school climate, student behavior and academic achievement – following the theory of action presented above. The study will also consider the difference in behavior support practices in Program and BAU schools (service contrast). Finally, the evaluation will describe CSBS’s training and support activities as well as the implementation of MTSS-B in Program schools.

Data Collection Strategy & Analytic Approach

The data collection activities planned for this study are essential for the measurement of the key outcomes of interest—teacher practice, school climate, student behavior and academic achievement. Each data source contributes in a unique way to the measurement of these outcomes. For example, the student survey is the only measure of student’s perception of classroom climate and the staff survey is the only measure of staff’s perception of school climate.

Exhibit A-2 provides an overview of data collection activities. Data sources that present a burden to students and/or staff are indicated with an asterisk. A longer description of and justification for the data sources that present a burden is provided in the paragraphs below the table.

Exhibit A-2. Overview of MTSS-B Data Sources
				Timeline
	# of Respondents per School	# of Schools	Fall 2015		Spring 2016	Summer 2016	Fall 2016	Spring 2017
Site visits [SET/ISSET]*
School tour to observe school rules	0	89	X					X
Document review	0	89	X					X
Administrator interview 45 minutes per wave	1	89	X					X
Team leader interview 30 minutes per wave	1	89	X					X
Student interview (stratified random sample) 2 minutes per wave	15	89	X					X
Staff interview (stratified random sample) 5 minutes per wave	15	89	X					X
Structured interviews with key implementers (phone interviews)*
MTSS-B Coach phone interview 45-60 minutes per wave	.20 ˅	58			X			X
Program school principal phone interview 45 minutes per wave	1	58			X			X
Team leader phone interview 45 minutes per wave	1	58			X			X
Review of web-based program forms (Coach logs, meeting minute forms and TIPS school forms)	0	58	X		X	X	X	X
Review of sample of CSBS records (CSBS trainer logs, attendance records, webinars, fidelity data)	0	58	X		X	X	X	X
Review of MTSS-B behavior monitoring and fidelity data (SWIS data, SWIS-CICO data & PBIS Apps data)	0	58	X		X	X	X	X
School staff survey (all school staff)* 30 minutes for non-teaching staff 35 minutes for teachers	60	89			X			X
Student survey (all students in grades 4-5)* 20 minutes to administer survey [include time for set-up and reading survey aloud]	200	89						X
Classroom observations with the ASSIST and possibly CLASS [augmented in Program schools to included assessment of implementation fidelity]	0	89						X
Teacher ratings of student behavior (all teachers in grades 1-5; all students in class at baseline and sample of students in spring 2016 and spring 2017)* 5 minutes per student per wave	25	89	X		X			X
District records data collection (9 districts)	0	NA	X		X			X
Note: Activities that present a time burden to students and staff are indicated with an asterisk(*). Fielding of the staff survey in spring 2016 and collection of district records in Spring 2016 is not definite. ˅ The MTSS-B Coach role is designated as 0.2 FTE for each Program school. On average, however, districts will need 2 different MTSS-B coaches, though one of them may be only part-time. This increases the number of respondents for data collection purposes, but does not change the actual staffing.

Description of Data Sources that Present a Burden to Staff and Students

Site Visits: We will conduct site visits in Program and BAU schools twice over the study period (fall 2015 and spring 2017). The site visit protocols involve document review, observation and interview. The times required for the interviews conducted as part of the site visit are indicated in Exhibits A-2 and A-3. The data collected from these visits will serve the following purposes: (1) Assess the difference in behavior support practices in Program and BAU schools (service contrast) and (2) Assess implementation fidelity to the core feature of the MTSS-B Tier I and Tier II programs in Program schools. To develop the protocols for these site visits, the study team is using the interview protocols as part of the School-wide Evaluation Tool (SET), which has been found to be a valid and reliable measure of MTSS-B Tier I implementation fidelity.⁸ It has also been used in prior randomized control trials of MTSS-B as a measure of service contrast between Program and BAU schools and found to accurately discriminate between schools implementing the core features of MTSS-B and those not.⁹ The team is combining the SET protocol with the I-SSET, which assesses the fidelity of Tier II and Tier III implementation.¹⁰ These two protocols have been combined in prior studies to provide a complete picture of the three tiers of MTSS-B implementation.¹¹ The interview protocols for the site visits can be found in Appendices A-D.

Structured Phone Interviews with Key Implementers: We will conduct phone interviews with key program implementers (MTSS-B Coach, Administrator and Behavior Team Leader) in each of the 58 Program schools twice during the study period (spring 2016 and spring 2017) to assess these individuals’ perceptions of their schools’ implementation fidelity and the quality/utility of CSBS training and support. Understanding staff’s perceptions of the quality and utility of CSBS training and support in the Program schools is critical to fully understanding implementation fidelity and to identifying challenges associated with the implementation of MTSS-B in a large number of elementary schools and districts. The interview protocols for the structured interviews with key program implementers can be found in Appendices E and F.

Staff Survey: School staff (teachers, administrators and other staff) in all study schools will be asked to participate in a web-based survey about school climate, staff practices related to behavior, and training for behavior support practices and programs in the spring of 2016 and spring of 2017. The surveys also include a number of questions related to individual background characteristics and experiences. The survey for non-teaching staff will take approximately 30 minutes to complete and the survey for teaching staff will take approximately 35 minutes to complete.

Improved school climate is theorized to be the proximal outcome of the intervention and is therefore critical to measure to assess staff’s perceptions of school climate. We have adapted two well-validated measures: Organizational Health Inventory (OHI) and the Malasch Burnout Inventory. ¹² Prior studies of MTSS-B have found impacts on the OHI.¹³

To assess staff practices related to behavior, we have adapted and abbreviated the Effective Behavior Support Survey (EBS).¹⁴ These items assess the extent to which school staff report implementing core features of MTSS-B. We have also adapted questions from the multi-site evaluation from IES’s Social and Character Development study to identify differences between BAU and Program schools in the extent to which staff report that they have received training for and/or are implementing school-wide and classroom-based programs similar to the MTSS-B approach being implemented in Program schools.¹⁵ We have also designed questions to assess staff’s reports of exposure to training and coaching related to behavior support practices. Staff’s answers to these questions will inform our understanding of service contrast; fully understanding service contrast is critical to the interpretation of the impact estimates.

The survey fielded in Program schools includes a limited number of items designed to assess staff’s perception of implementation fidelity and their perception of the quality and utility of the training and support they have received. Some of these questions have been adapted from prior studies of MTSS-B.¹⁶ The staff survey can be found in Appendix G.

Student Survey (Grades 4-5): Students in grades 4 and 5 in Program and BAU schools will be asked to complete a survey in the spring of 2017. The survey will take approximately 20 minutes for all fielding. This survey will provide critical information about students’ perceptions of their own behavior and classroom climate, which are both predicted by the theory of action to be positively impacted by MTSS-B. Most of the scales in the survey were borrowed from the student survey developed by Catherine Bradshaw and colleagues for the Maryland Safe and Supportive School Initiative. The high school version of this survey has been validated in prior studies.¹⁷We have also included a limited number of questions to assess students’ awareness of core features of MTSS-B implementation, such as the establishment of school rules. These questions will be asked in the survey administered in BAU and Program schools so they can inform the assessment of implementation and service contract. These questions were adapted from the student survey used for IES’s Violence Prevention study¹⁸. The student survey can be found in Appendix H.

Teacher Ratings of Student Behavior: Teachers of students in grades 1-5 in Program and BAU schools will be asked to rate their students’ behavior in fall of 2015.¹⁹ In the spring of 2016 and the spring of 2017, teachers will be asked to rate the behavior of a high-risk sub group (identified from the fall 2015 fielding) and a random sample of other students. Teachers will only be asked to rate the behavior of students in their classes with consent. The rating form is expected to take approximately five minutes per student to complete. If a teacher has twenty consenting students in her class, she may need to spend up to 100 minutes completing this rating form in Fall of 2015. In the Spring of 2016 and 2017, we are only asking teachers to rate a sample of their students and so the teachers will only need to spend up to 40 minutes on ratings each Spring.

The teacher ratings form will be used to assess the impact of the program on student behavior. The study team has adapted the Teacher Observation of Student Adaptation-Checklist (TOCA-C) for this instrument. The TOCA-C has been shown to be a valid and reliable measure of students’ behavior (concentration problems, disruptive behavior and prosocial behavior).²⁰ It has been used in prior evaluations of MTSS-B where it has been shown to be sensitive to the intervention.²¹ The instrument measures key domains of student behavior closely tied to the theory of change of MTSS-B —prosocial behavior, concentration problems, and disruptive behavior. The study team is augmenting this measure with scales to assess additional policy-relevant concepts that are also closely tied to the MTSS-B theory of action-- bullying, students’ emotional regulation, and internalizing behaviors, as well as receipt of behavioral support services and referrals related to behavior. These additional items have been used in Catherine Bradshaw and colleagues’ ongoing evaluations of the MTSS-B interventions and shown in unpublished technical reports to have adequate reliability.²² The teacher ratings instrument is the most reliable and sensitive way to collect useful student behavior information. The teacher ratings form can be found in Appendix J.

District Records Data Collection: The study team will request extant data from school district records regarding student background characteristics, teacher background characteristics and experiences, student behavior incidences, student academic achievement (grades 3-5) and special education. Data will be collected at the end of each school year: baseline (SY 2014-2015), year 1 (SY 2015-2016) and year 2 (SY 2016-2017). Requests will be made of district research offices. Appendix K provides a sample of the letter that will be sent to districts requesting these data.

SUPPORTING STATEMENT FOR PAPERWORK

REDUCTION ACT SUBMISSION

1. Respondent Universe and Sampling Methods

The sample for this school-level random assignment study includes 89 schools across approximately nine school districts. To arrive at this sample from the universe of schools districts and elementary schools within districts, study team and the Department developed a set of district and school eligibility guidelines, which are summarized in Exhibit B-4.

Exhibit B-4: Screening guidelines used for recruitment
District Screening Guidelines:	A district must have 10 public schools serving students in Kindergarten-Grade 5 or Grade 6 that meet the following criteria: School Size: Have at least 50 students per grade, on average. Family Income: 25% of students qualify for Free and Reduced Priced Lunch or are Title 1 Eligible [according to 2011-2012 CCD or district data]. Current Behavior Support Strategies: Are not currently implementing MTSS-B Tier I or Tier II programs. Interest: The district must be interested in implementing the MTSS-B program in nominated schools. The district must fully understand the random assignment process and the burden of data collection.
School Screening Guidelines:	Within an eligible district, a school will be considered for nomination if they meet the following criteria: School Size: Do not have a single grade serving fewer than 40 students [according to 2011-2012 CCD or district data]. Across all nominated schools, there will be an average of at least 60 students per grade. Family Income: Across all nominated schools within a district, the percentage of FRL students must be greater than 25% with preference given to schools that have at least 15% of students qualify for Free and Reduced Priced Lunch [according to 2011-2012 CCD or district data]. Current Behavior Support Practices: Are not currently implementing MTSS-B Tier I and Tier II programs. Interest: The schools must be interested in implementing the MTSS-B program if they are selected as a program school and be willing to continue with business-as-usual if they are selected as a BAU school. Schools must fully understand the random assignment process and the burden of data collection.

Site Recruitment Process: To recruit districts and schools and to determine eligibility, the study team first identified 480 districts across the country that met the first two of the final screening criteria (see Exhibit B-4). The evaluation team then contacted potential sites and provided them with information on the project and the research design. The districts discussed their current behavioral systems in place, as well as the district's ability and willingness to implement research procedures related to random assignment and data collection. Selection of districts and schools took into account the advantage for the policy relevance of results of having sites that include a variety of regions, urban/suburban/rural settings, and race/ethnicity of the student body. The research team worked closely with the project officer in the Department of Education to make final choices. Nine school districts and 89 schools have been accepted into the study and signed a memorandum of understanding. These nine districts are located in six states and have between 7 and 16 participating schools.

Sampling Methods

In the paragraphs below we describe the sample and anticipated response rate for all data sources that present a burden to respondents.

Site visits: Site visits will be conducted in fall 2015 and spring 2017 in the 89 study schools. There are 89 principals and 89 Behavior Team Leaders in the sample for the site visit interviews. The site visit protocol includes 1-2 minute interviews with 2-3 randomly selected students per grade in each school (up to 15 students total per school) who answer two questions to assess whether they know the schools rules and the behavior reward system. The 15 students in each school will be randomly selected (within grade) for brief interviews as site visit assessors walk through non-classroom settings of the school (e.g. hallway, playground) at a time that the principal says will be least disruptive to instruction. In addition, a stratified random sample of 15 staff in each school will be selected for a brief interview as site visit assessors walk through non-classroom settings of the school when staff are not engaged in instruction (e.g. hallway, playground, staff break room) at a time that the principal says will be least disruptive to instruction. In identification of staff for the interviews, assessors will be instructed to try to attain representation from each grade level, special subject areas and non-teaching staff. At least three of the staff interviewed should be members of the Behavior Team. For all site visit interviews, we expect a 90% response rate.

Structured Interviews with Key Implementers: We will conduct phone interviews with key program implementers (MTSS-B Coach, administrator and Behavior Team Leader) in each of the 58 program schools twice during the study period (spring 2016 and spring 2017). The 58 Behavior Team Leaders and 58 principals in Program schools are included in the interview sample. Based on the number of Program schools, we estimate that there will be 15 MTSS-B Coaches in this study (some working part-time). All MTSS-B Coaches will be sampled for the phone interviews. For all interviews we expect a 90% response rate.

Staff Survey: School staff (teachers, administrators and other staff) in all study schools will be asked to participate in a web-based survey in the spring of 2016 and 2017. All school staff will be included in the survey sample. Based on numbers of teaching and non-teaching staff reported by recruited districts, we assume a sample of 60 staff per school. We expect 90% of the staff sample to complete the survey.

Student Survey (Grades 4-5): Students in grades 4 and 5 in Program and BAU schools will be asked to complete a survey in the spring of 2017. Based on the CCD for the recruited schools, we estimate that there are, on average, 200 students in grades 4 and 5 in each of the study schools. We assume a 90% response rate for the survey.

Teacher Ratings of Student Behavior: Teachers of students in grades 1-5 in Program and BAU schools will be asked to rate their students’ behavior in fall 2015. Based on the CCD for the recruited schools, we estimate that on average there are 25 classroom teachers of grades 1-5 in the 89 sampled schools. Based on CCD, we estimate that, on average, there are 100 students per grade in each of the study schools. We estimate that 90% of a teacher’s students will provide consent to be rated and that, on average, 90% of teachers will rate all of their consenting students. In total, we expect to have 81 students per grade in each of the schools rated in fall 2015. In total, we expect 23 teachers per school to rate all of their consenting students.

In the spring of 2016, we will draw two samples of the students based on spring 2016 rosters of consenting grades 1-5 students. The first sample will consist only of students who were identified as being at highest risk of future behavior problems based on teachers’ ratings in fall 2015 and are still attending a study school. We expect this sample not to exceed 20 students per grade per school, on average. In addition, we will draw a random sample of up to 30% of all consenting students from the spring 2016 rosters (grades 1-5), which will include consenting students who were not rated in fall 2015. We expect that this sample will consist of no more than 25 students per grade per school, on average. We assume that 90% of the 25 classroom teachers will rate all of their sampled students. In total we expect 23 teachers per school to rate all of their consenting students.

In spring 2017, we will draw two samples of consenting grade 1-5 students from the spring 2017 roster. The first sample will be the high-risk sub-group identified in fall 2015, excluding students who have moved onto 6^th grade. We assume that we will lose 20% of the high-risk sub-group between year 1 and year 2 due to students moving to a school outside the study sample.²³ We expect this high-risk sample of students not to exceed 16 students per grade per school, on average. In addition we will draw a random sample of up to 30% of all consenting students from the spring 2016 rosters (grade 1-5). We expect that this sample will consist of no more than 25 students per grade per school, on average. We assume that 90% of teachers will rate all of their sampled students. In total we expect 23 teachers per school to rate all of their consenting students.

District Records Data Collection: The study team will request extant data from school district records regarding student background characteristics, teacher background characteristics and experiences, student behavior incidences, student academic achievement (grades 3-5) and special education. Data will be collected at the end of the baseline school year (SY 2014-2015), the end of year 1 (SY 2015-2016) and the end of year 2 (SY 2016-2017). Requests will be made to the nine district research offices in the study. We anticipate 100% response rate for these requests.

2. Procedures for Data Collection

MDRC will work with partner organization Decision Information Resources (DIR) to conduct the data collection activities. The paragraphs below describe the data collection strategy for the data sources that impose a burden on students or school staff: site visits to Program and BAU schools; staff surveys; student surveys; and teacher ratings of student behavior.

The following sections present our approach for each data collection effort, including information on sample and timing, instruments, and training. More information about data sources can be found in the Description of Data Sources beginning on page 13.

Site Visits: The site visits include document review, observations, and interviews with Behavior Team Leaders, and administrators. The data collected from site visits serves two purposes: (1) to assess the difference in behavior support practices between Program and BAU schools; and (2) to assess implementation fidelity of the core features of the MTSS-B Tier I and Tier II programs in Program schools. As described in the introduction to Supporting Statement A, the site visit protocol draws from the School-wide Evaluation Tool (SET) and the Individual Student Systems Evaluation Tool (ISSET), which assess MTSS-B implementation fidelity and service contrast.²⁴ These protocols have been augmented with structured interview protocols in Program schools to assess the quality and utility of CSBS training and support.

Site visits to all schools will occur twice over the course of the study, during the fall of the first year (2015-2016), and during the spring of the second year (2016-2017). Site visits involve a 45-minute interview with an administrator and a 30-minute interview with the Behavior Team Leader. Additionally, there will be interviews with a stratified random sample (by staff type) of 15 staff members (5 minutes each) and a stratified random sample (by grade) of 15 students (2 minutes each). Interviews will be conducted by DIR staff with training by Catherine Bradshaw’s team at the University of Virginia and Johns Hopkins University²⁵.

The data collection team understands the challenges involved with conducting site visits at schools during the school day. Therefore, our approach will be flexible to allow for a range of options that will accommodate known or unknown challenges. The team will work closely with school liaisons to arrange access to each school, identify the availability of space, determine best times for the individual interviews, and identify any “restricted” times or days (for example, field trips, school closings, student testing, and so forth).

Site visitors will be required to have experience conducting site visits in school settings and have at least a Bachelor’s degree. Preference will be given to individuals with a background in education, child development, or similar educational background.

All site visitors will attend a 3-4 day training in which they will go over details about the study and site visit procedures. A portion of the training will include visits to non-study schools to assess the site visitors' reliability with the protocols

The training agenda will include:

• An overview of the overall study

• General information about conducting site visits

• Proper administration of the measurement tool

• Quality assurance

• Scoring each tool

• Practice sessions immediately following the training

• Administrative procedures, responsibilities, and compensation

• Review of the certification process and criteria to meet established standards

Attendees will also review the instrument, professional conduct, policies and guidelines for reporting child maltreatment, and administrative policies and guidelines. A manual including information from the training will be provided to attendees. Site visitors who meet the certification requirements will be selected to conduct the site visits.

The specific site visit activities will be carried out as follows:

• The team will work with the school liaison to schedule the visit, set up the required interviews, and notify them of the required documents/procedures that will be reviewed. The agreed upon schedule will be provided to the school electronically and a reminder will be sent one week and one day prior to the scheduled visit.

• On the day of the site visit, the site visitor will arrive 30-60 minutes prior to the scheduled start time, and upon arriving, they will check-in with the school liaison and review the schedule for the day and ensure that all necessary documents are available for review. In addition, the site visitor will confirm that the planned interviews will be carried out with the specified people.

• After the schedule is set, the site visitor will begin carrying out the various required steps including selection of a random sample of students and staff for brief interviews (students interview is 1-2 minutes per student, and the staff interview is 3-5 minutes long). The procedures of this random sampling will be worked out with the district and the school liaison ahead of time so that every effort is made to not disrupt instruction.

• Upon completion of all pieces, the site visitor will alert the school liaison that the visit is finished, or if there were missing elements, that will be reviewed at that time.

• Site visitors will be required to conduct the scoring of both protocols on the same day the visit was conducted.

Scored protocols will be sent back to DIR and MDRC, reviewed for quality control, and then entered into a database.

Staff Survey: The staff survey is designed to collect data on the school climate, staff practices related to behavior, and training for behavior support practices.

This survey will be provided to teachers and non-teaching staff at all study schools in a web-based format in spring 2016 and spring 2017. DIR is managing the fielding of this survey. The survey will be administered to all teachers in grades 1-5, as well as to all non-teaching staff and administrators (n=60/per school). School staff will be given the option to complete all surveys by using the web version or paper version, or a combination, which will maximize the attainment of the desired 90 percent (or higher) response rate. School staff emails will be obtained through the district and school data collection liaison. DIR will format the questionnaire for self-administration via the web or hardcopy. Prior to completing the survey, teachers and other school staff will be asked to provide consent, and they may also withdraw their consent at any time.

Training for the staff survey administration will be nominal. From experience, most school staff will opt to complete the report using the web-based version. For those opting to use hardcopies, DIR staff will disseminate hardcopies of the instrument to school staff by mailing them to school liaisons. School staff will be asked to put all completed hardcopies in a DIR-provided envelope, seal the envelope, and return them to the school liaison. They will be shipped to the Houston office and logged as completed. Field personnel will be trained in the areas of confidentiality, data security (including collecting, securing, and transporting data), and tracking or logging receipt of complete and incomplete instruments. Staff member names will not be included on the completed paper-based surveys sent through the mail.

DIR will send hardcopy questionnaires (through school liaisons) to staff that request to complete the questionnaires using the paper version hardcopies. At the start of week 4, hardcopy questionnaires will be sent to school liaisons who will deliver them to the non-responders who requested to complete using the web version. A reminder postcard will be designed (to be delivered in person or via regular mail) and sent to all non-responders, and additional email reminders will be sent to teachers with email addresses during the last two weeks of the fielding period. DIR’s previous experience with attempting to reach teachers and school staff by telephone has not been very successful. Typically, messages are left without any certainty of their delivery. Thus, DIR will ask the school liaisons to hand deliver reminder postcards and give the teachers and school staff a verbal reminder at specific times during the fielding period. Each of these steps will help to ensure that the desired response rate is reached by the end of the fielding period.

In summary, DIR will complete the following steps in fielding the staff survey:

• Two weeks prior to launching the web survey, DIR will send staff a personalized letter (via email and regular mail) with a description of the importance of the questionnaire, stressing the confidentiality of their responses, explaining how the results will be used, describing the honoraria, and indicating when they will receive their individualized link to complete the web version. DIR will also thank them in advance for their participation.

• On the day of the survey launch, DIR will send staff an email announcing the launch of the survey. This email will contain a unique identification number and a link to the survey. For staff without an email address, DIR will send the same information via regular mail.

• At the 3-day, 10-day, and 14-day mark, DIR will send reminder emails and/or postcards to thank staff for their participation or encourage them to respond if they have not already done so. At the 14-day mark, the option to complete a hardcopy survey will be included in the reminder. Upon request, hardcopy surveys will be sent directly to the teacher via FedEx overnight delivery, along with a pre-paid return envelope.

DIR will ask school liaisons to send a reminder to staff to express district/school support for the study and encourage participation. DIR will provide the specific text for them to send, in an effort to minimize the burden on the liaisons.

Student Survey (Grades 4 and 5): This survey will provide critical information about students’ perceptions of their own behavior and classroom climate, which are both predicted by the theory of action to be positively impacted by MTSS-B.

DIR will field the survey to all students in grades 4 and 5 in 89 participating elementary schools across nine districts during spring 2017.The survey will take approximately 20 minutes for all fielding (the actual survey will only take approximately 17 minutes to complete). Data collection in each school will occur over a 2-3 day period. The study team will collect active parental consent for this activity. See question A-9 and B-3 for more detail on our proposed strategy for maximizing response rates on the consent form.

DIR will recruit and hire a team of 8–10 proctors per district. Each team will be led by an assessor with extensive classroom experience. Candidates will be required to have experience working with children between the ages of 8-11 and have some college education. Preference will be given to individuals with a child development or similar educational background.

All proctors will attend a training in which they will go over details about the study and data collection procedures. The training agenda will include:

• An overview of the evaluation and student survey

• General assessment techniques for working with children ages 8-11

• Proper administration of the measurement tool

• Quality assurance

• Live practice sessions immediately following the training

• Administrative procedures, responsibilities, and compensation

• Review of the certification process and criteria to meet established standard

Attendees will also review the instrument, proctors’ responsibilities, professional conduct, policies and guidelines for reporting child maltreatment, and administrative policies and guidelines. A manual including information from the training will be provided to all proctors.

The following approaches will be used to collect the student survey data:

• DIR will work closely with school liaisons to arrange access to each school, identify the availability of space, determine best times for assessments, and identify any “restricted” times or days (for example, field trips, school closings, and so forth).

• DIR staff will call approximately 3 business days before the scheduled visit to confirm the date, the school contact (if different from the liaison), will verify the location for the administration, and gather any additional necessary information.

• DIR will assign 2 proctors to all fourth and fifth grade classrooms. For each day of data collection, DIR staff will arrive 1 hour before the actual administration is set to begin. One proctor will distribute the materials and verify all materials match the classroom roster and are distributed accurately. The second proctor will confirm that all students have their assigned materials, distribute pencils, walk the classroom during the survey administration, and collect all materials at the end of the survey administration assigned.

• Survey instructions and all questions will be read aloud to students.

• Completed surveys will be logged and returned to the DIR offices for scoring.

Teacher Ratings of Student Behavior: The teacher ratings will be used to assess the impact of the program on student behavior and the students’ referral to and receipt of targeted supports for behavior and special education services.

Teachers of students in grades 1-5 in all study schools (n=89) will be asked to rate all of their students’ behavior in fall 2015. In spring 2016 and spring 2017, teachers will be asked to rate the behavior of a random sample of students (average of 25 students sampled per grade), drawn from the spring 2016 roster of consenting students. Additionally, teachers will be asked to rate the behavior of a high risk subgroup identified through the fall 2015 teacher ratings (average of 20 students per grades in each school will be identified for the high-risk subgroup). DIR field staff will contact all teachers of students in grades 1-5 in 89 participating schools across nine districts. The rating form is expected to take approximately five minutes per student to complete and each teacher will rate all of their students. The study team will collect active parental consent for this activity. See questions A-9 and B-3 for more detail on our proposed strategy for maximizing response rates on the consent form.

The rating checklist will be administered electronically with DIR using paper-based surveys to follow-up with non-responders. Fielding procedures for this instrument are consistent with the web-based staff survey described above. In summary, the teacher rating checklist data collection activities will be carried out as follows:

• Two weeks prior to launching the web link, DIR will send teachers a personalized letter (via email and regular mail) with a description of the importance of the rating checklists, stressing the confidentiality of their responses, explaining how the results will be used, and indicating when they will receive their individualized link to complete the web version. DIR will also thank them in advance for their participation.

• On the day of the web launch, DIR will send teachers an email announcing the launch of the web version. This email will contain a unique identification number and a link to complete the ratings checklist. For staff without an email address, DIR will send the same information via regular mail.

• At the 3-day, 10-day, and 14-day mark, DIR will send reminder emails and/or postcards to thank teachers for their participation or encourage them to respond if they have not already done so. At the 14-day mark, the option to complete a hard-copy version of the ratings checklists will be included in the reminder. Upon request, hardcopy versions will be sent directly to the teacher via FedEx overnight delivery, along with a pre-paid return envelope.

• DIR will ask school liaisons to send a reminder to teachers to express district/school support for the study and encourage participation. DIR will provide the specific text for them to send, in an effort to minimize the burden on the liaisons.

3. Procedures to Maximize Response Rates

The goal will be to achieve a 90 percent response rate or higher in each site for each data collection effort. Procedures for obtaining the maximum degree of cooperation include:

• Sending respondents cover materials that address the importance and value of the study.

• Collecting all parental consent forms for student data collection activities that require parental consent at the start of the school year.

• Working closely with school principals and/or liaisons to develop a schedule that will facilitate administration to all students.

• Training proctors to maintain direct contact with respondents and provide assurances to reluctant responders.

• Providing proctors packets of student surveys with blank labels in the event that new students are added to the class.

• Allowing school staff and teachers multiple formats (web and paper-pencil) for completing the instruments.

• Offering appropriate payments to school staff and teachers.

Maximizing the response rate for the parental consent will require a multilingual and multistep approach. All materials will be prepared in both English and Spanish and distributed in packets by classroom to teachers with a request to distribute the forms to all children in their classrooms. Forms will be printed on 2-ply carbonless paper so that parents could keep a signed copy. The consent packet will include a written explanation of the study and a request that the parent or guardian allow their child’s participation in the study by signing the forms. As an incentive, teachers or schools data collection liaisons (whoever is responsible for distributing the consent forms) will be offered a $25 gift card if they achieve a 90 percent response rate or better return of parental consent forms.

4. Testing of Data Collection Procedures

Most of the questions proposed for this survey are either identical to questions used in prior MDRC or AIR evaluations,²⁶ or are similar to questions used in previous national surveys or major evaluations.²⁷ Consequently, many of the items have been thoroughly tested on larger samples.

The MTSS-B surveys and interview protocols have already undergone a number of revisions, following critiques by internal staff, by project consultants, and by staff at IES. MDRC will also work closely with DIR and AIR senior staff to conduct formal pre-tests of these surveys and interview protocols, using fewer than nine appropriate respondents.

5. Individuals Consulted on Statistical and Methodological Aspects of Data Collection

This project is being conducted under contract to the Department of Education by MDRC, AIR, Harvard University, and DIR. The data collection strategy and instruments were developed by Fred Doolittle and Barbara Condliffe of MDRC; Anja Kurki, Louis Danielson, Muna Shami and Gail Chan of AIR; Stephanie Jones of Harvard Graduate School of Education; and Catherine Bradshaw of the University of Virginia. Pei Zhu of MDRC is leading the statistical design of the study and the estimation of the impacts of the MTSS-B training and support.

The organizations responsible for data collection activities are as follows:

Organization	Primary Contact	Phone Number
MDRC	Fred Doolittle	212-340-8638
American Institutes for Research (AIR)	Anja Kurki	202-403-6022
Decisions Information Resources (DIR)	Russell Jackson	713-650-1425

Individuals outside these organizations who were consulted on the design of data collection strategy are as follows:

Name	Organization	Phone number
David Cordray	Vanderbilt University, Department of Psychology and Human Development	615-343-2699
Brian Flay	Oregon State University, School of Social and Behavioral Health Sciences	541-737-3837
Mary Louise Hemmeter	Vanderbilt University, Department of Special Education	615-322-5648
Sara E. Rimm-Kaufman, Ph.D.	University of Virginia, Curry School of Education and Center for the Advanced Study of Teaching and Learning	434-982-2863
Lori Newcomer, Ph.D.	University of Missouri, Missouri Prevention Center / Department of Educational School and Counseling Psychology	573-882-7731   314-605‐0235
Catherine Bradshaw, Ph.D.	University of Virginia Johns Hopkins University	434-924-8121
Stephanie Jones, Ph.D.	Harvard University	617 496-2223

The contracting offer for this study is Dr. Lauren Angelo of the National Center for Education Evaluation (IES). Her phone number is: 202-219-2180.

REFERENCES USED IN PART B

Anderson, C. M., Teri Lewis-Palmer, Anne W. Todd, Robert H. Horner., George Sugai, N.K. Samson. 2008. "Individual student systems evaluation tool, version 2.6." Educational and Community Supports, University of Oregon.

Bradshaw, Catherine P., Celine E. Domitrovich, Jeanne Poduska, Wendy Reinke, and Elise T. Pas. (2009). Measure of Coach and Teacher Alliance – Teacher Report. Unpublished Measure. Johns Hopkins University. Baltimore, MD.

Bradshaw, Catherine P., Tracey E. Waasdorp, Katrina J. Debnam, and Sarah Lindstrom Johnson. (2014). “Measuring school climate: A focus on safety, engagement, and the environment.” Journal of School Health, 84, 593-604. DOI: 10.1111/josh.12186

Bradshaw, Catherine P., Katrina J. Debnam, and Philip J. Leaf. (2009). Teacher Observation of Classroom Adaptation-Expanded Checklist (TOCA-EC). Unpublished Measure. Johns Hopkins University. Baltimore, MD.

Bradshaw, Catherine P., Christine W. Koth, Katherine B. Bevans, Nicholas Ialongo, and Philip J. Leaf. 2008a. “The Impact of School-Wide Positive Behavioral Interventions and Supports (PBIS) on the Organizational Health of Elementary Schools.” School Psychology Quarterly, 23, 4: 463-473.

Bradshaw, Catherine P., Mary M. Mitchell, and Philip J. Leaf. 2010. “Examining the effects of School-Wide Positive Behavioral Interventions and Supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools.” Journal of Positive Behavior Interventions, 12, 133-148.

Bradshaw, Catherine P., Wendy M. Reinke, Louis D. Brown, Katherine B. Bevans, and Philip J. Leaf. 2008b. “Implementation of school-wide Positive Behavioral Interventions and Supports (PBIS) in elementary schools: Observations from a randomized trial.” Education and Treatment of Children, 31, 1–26.

Bradshaw, Catherine P., Tracy E. Waasdorp, and Philip J. Leaf. 2012. "Effects of School-Wide Positive Behavioral Interventions and Supports on Child Behavior Outcomes." PEDIATRICS 130, 5: e1136-e1145.

Bradshaw, Catherine P., Tracey E. Waasdorp, Katrina J. Debnam, and Sarah Lindstrom Johnson. (2014). “Measuring school climate: A focus on safety, engagement, and the environment.” Journal of School Health, 84, 593-604. DOI: 10.1111/josh.12186.

Debnam, Katrina J., Pas, Elise T., and Bradshaw, Catherine P. (2012). Secondary and tertiary support systems in schools implementing school-wide positive behavioral interventions and supports: A preliminary descriptive analysis. Journal of Positive Behavior Interventions, 14(3), 142–152.

Horner, Robert H. 2014. “The Role of District Leadership Teams in PBIS Implementation”. Presentation. Available at http://www.pbis.org/Common/Cms/files/pbisresources/1-Implementing%20PBIS.pptx.

Horner, R. H., Todd, A. W., Lewis-Palmer, T., Irvin, L. K., Sugai, G., & Boland, J. B. 2004. The school-wide evaluation tool (SET): A research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Interventions, 6, 3–12.

Horner, Robert H., George Sugai, Keith Smolkowski, Lucille Eber, Jean Nakasato, Anne W. Todd, and Jody Esperanza. 2009. "A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools." Journal of Positive Behavior Interventions 11, 3: 133-144.

Hoy Wayne K. and Clemens John Tarter. 1997. The Road to Open and Healthy Schools: A Handbook for Change, Elementary Edition. Thousand Oaks, CA: Corwin Press.

Hurrell, Joseph J. and Margaret A. McLaney. (1988). “Exposure to job stress: A new psychometric instrument.” Scandinavian Journal of Work Environment and Health, 14, 27-28.

Johnson,Sarah Elise T. Pas, and Catherine P. Bradshaw. (2015). “Identifying Factors Relating to the Coach-Teacher Alliance as Rated By Teachers and Coaches.” Manuscript submitted for publication.

Jones, Stephanie M., Joshua L.Brown and J. Lawrence Aber. 2011a. The longitudinal impact of a universal school-based social-emotional and literacy intervention: An experiment in translational developmental research. Child Development 82,2: 533-554.

Jones, Stephanie M., Joshua L. Brown, and J. Lawrence Aber. 2011b “Two-Year Impacts of a Universal School-Based Social-Emotional and Literacy Intervention: An Experiment in Translational Developmental Research”. Psychology Faculty Publications. Paper 131. (http://fordham, bepress.com/psych_facultypubs/131).

Jones, Stephanie M., Joshua L. Brown, Wendy LG Hoglund, and J. Lawrence Aber. 2010. "A school-randomized clinical trial of an integrated social–emotional learning and literacy intervention: Impacts after 1 school year." Journal of Consulting and Clinical Psychology 78, 6: 829.

Koth, Christine W., Catherine P. Bradshaw, and Philip J. Leaf. 2009. "Teacher Observation of Classroom Adaptation—Checklist: Development and factor structure." Measurement and Evaluation in Counseling and Development 42, 1: 15-30.

Maslach, Christina, and Susan E. Jackson. 1981. "The measurement of experienced burnout." Journal of Organizational Behavior 2, 2: 99-113.

OSEP Center on Positive Behavioral Interventions and Supports. 2007. “Is School-Wide Positive Behavior Support An Evidence-Based Practice? A Research Summary.” Website: www.pbismaryland.org.

Silvia, Suyapa, Jonathan Blitstein, Jason Williams, Chris Ringwalt, Linda Dusenbury, and William Hansen. "Impacts of a Violence Prevention Program for Middle Schools: Findings after 3 Years of Implementation.” NCEE 2011-4017." National Center for Education Evaluation and Regional Assistance (2011).

Sugai, George, Todd, Anne W., and Horner, Robert H. (2000). Effective Behavior Support (EBS) Survey: Assessing and planning behavior supports in schools. Eugene, OR: University of Oregon.

1 Horner (2014).

2 Bradshaw, Mitchell, and Leaf (2010); Horner et al. (2009); OSEP Center on Positive Behavior Interventions and Supports (2007).

3 Note that the School Leadership Team is a term used by CSBS but will more often be referred to as a “behavior team” in business as usual schools. We refer to the SLT when describing the CSBS program but the “behavior team” when describing the data collection strategy (e.g. interviews with behavior team leader).

4 Core components of MTSS-B Tier I include teaching and reinforcing specific school-wide behavioral expectations, a classroom management system to support behavioral expectations, and the use of fidelity data to monitor and improve implementation.

5 Core components of MTSS-B Tier II (CICO) include use of data to identify and progress monitor students needing additional supports, additional instruction and time for students to develop behavioral skills, additional structures with increased opportunity for feedback from staff, and sharing information about student behavior between school and families.

6 CSBS will use fidelity measures to monitor schools’ progress, identify areas in need of improvement, and make adjustments in training when necessary. CSBS plans to use the Effective Behavior Support (EBS) survey and the Tiered Fidelity Inventory (TFI) implementation tool for this purpose. CSBS will use the Behavior Education Program Fidelity of Implementation (BEP-FIM) to assess fidelity of implementation for the Tier II intervention.

7 For academic achievement, the MDES for the year 2 estimates are 0.189 for reading and 0.195 for math. The MDES for behavior ratings in year two is 0.089 for the high-risk subgroup and 0.069 for the random sample of all students. The MDES for teacher practice outcomes range from 0.118 to 0.201 depending on the parameter value assumptions and number of classrooms observed. The MDES for school climate measures from the teacher survey in year 2 is between 0.186 and 0.410 depending on the parameter value assumptions. The MDES for the school climate measures from the student survey (grades 4&5) is between 0.149 and 0.285 depending on the parameter value assumptions.

8 Horner et al. (2004).

9 Bradshaw, Waasdorp, and Leaf (2012).

10 Anderson et al. (2008).

11 Debnam, Pas and Bradshaw (2012).

12 Hoy and Tartar (1997); Maslach and Jackson (1981). We have also drawn items from the NIOSH’ Job Stress Questionnaire to complement our measure of burnout (Hurrell & McLaney, 1988).

13 Bradshaw et al., 2008.

14 Sugai, Todd, and Horner (2000).

15 Social and Character Development Consortium, 2010.

16 Debnam, Pas &Bradshaw, 2012; Bradshaw et al., 2009; Johnson, Pas & Bradshaw, 2015

17 Bradshaw et al., 2014.

18 Silvia et al., 2011

19 Fielding this survey at baseline will allow the study team to identify a high-risk sub-group at baseline. Identification of this subgroup is necessary for answering one of the study’s research questions regarding the effects of MTSS-B on at-risk students.

20 Koth, Bradshaw and Leaf (2009).

21 Bradshaw, Waasdorp, and Leaf (2012).

22 Bradshaw, Debnam & Leaf, 2009.

23 Assumptions about student mobility are based on recent student mobility data from ongoing MDRC studies.

24 Anderson et al., 2008; Horner et al., 2004; Debman, Pas and Bradshaw, 2012.

25 Bradshaw and her colleagues have used the site visit protocols being used in this study in multiple other studies of MTSS-B; for example, see Debman, Pas, and Bradshaw (2012).

26 Some examples of such evaluations include MDRC's current Evaluation of Response to Intervention for Elementary School Reading (report due in 2015; http://ies.ed.gov/ncee/projects/evaluation/disabilities_rti.asp) and AIR's Good Behavior Game Professional Development Study (http://www.air.org/project/good-behavior-game-professional-development-study).

27 For example,. Anderson et al. (2008); Bradshaw et al. (2008a); Bradshaw et al. (2008b); Bradshaw, Mitchell, and Leaf (2010); Bradshaw, Waasdorp, and Leaf (2012); Jones, Brown, Hoglund, and Aber (2010); Jones, Brown, and Aber (2011a); Bradshaw et al., 2014; Jones, Brown, and Aber (2011b); Koth, Bradshaw, and Leaf (2009 Silvia et al., 2010).

11