An Impact Evaluation of Training in Multi-Tiered Systems of mSupport for Behavior (MTSS-B)

Appendix E: Phone Interview Protocol for MTSS-B Coach Interview

Protocol A: MTSS-B Coach(es) in Each District (Spring 2016 and Spring 2017)

Date:

Interviewer_ID: District_ID: Coach_ID:

Introduction [same introduction repeated in Spring of 2016 and Spring of 2017]

Good morning/good afternoon. My name is [name] and I work with the Multi-Tiered Systems of Support for Behavior (MTSS-B) study team.

We are conducting interviews with MTSS-B coaches to hear how implementation is going in each of your schools and learn your thoughts about the training and support they have received to help school staff to implement the MTSS-B program. Your participation in the interview is completely voluntary. We hope you will participate because your perspective is essential to understanding how to improve behavior support practices in schools across the country. We anticipate that the interview will take about 45-60 minutes.

Before we begin, I would like to ask you to review this consent form and let me know if you agree to be interviewed and I can audio-record the interview. We record the interviews to ensure that we do not miss any relevant information. All the information we collect is confidential, and you, your school, or anyone working at or attending your school will not be identified by name in any of the reports that will be published. You can ask me to stop recording at any time. (Review consent form.)

Consent Form: Emailed to coach and returned electronically

AGREEMENT TO COMPLETE INTERVIEW FOR THE EVALUATION OF

MULTI-TIERED SYSTEMS OF SUPPORT FOR BEHAVIOR (MTSS-B)

What is the purpose of the study?

This study is evaluating Multi-Tiered Systems of Support for Behavior (MTSS-B), an approach to teaching and reinforcing appropriate behavior for all students and providing additional supports for students with greater needs. Some schools in the study have been randomly selected to receive training and support in MTSS-B and other schools in the study were randomly selected to continue with their existing behavior support practices. Data is being collected to understand how this program works across schools and districts; it will not be used to evaluate individual schools, teachers, or staff members.

The U.S. Department of Education has picked MDRC, a research group, to carry out the study. MDRC is working with partner organizations to collect data for this research study—American Institutes of Research (AIR), Decisions Information Resources (DIR) and Harvard Graduate School of Education. These four organizations make up the MTSS-B study team.

What am I being asked to do?

You are invited to complete an interview about behavior support practices. It will take up to 60 minutes.

What are the benefits from taking part in this interview?

We hope that you will feel satisfied knowing that you are helping to improve teaching and learning in your school and throughout the country.

What are the possible risks of this interview?

There are no anticipated risks of participating in this interview. Your responses and personal information will be stored securely and will not be released to any person outside of the research team. Information collected will not be used to evaluate you or other individuals in your school.

Who will know about my participation in this interview?

Information collected for this study comes under the confidentiality and data protection requirements of the Institute of Education Sciences. All information from this study will be kept confidential as required by the Education Sciences Reform Act of 2002 (Title I, Part E, Section 183). Responses to this data collection will be used only for statistical purposes. Personally identifiable information about individual respondents will not be reported. We will not provide information that identifies you, your school, or your district to anyone outside the study team, except as required by law.

Is my participation in this interview voluntary?

Your participation in the interview is completely voluntary and your decision will not affect your current or future status in your school. We hope you will participate because your views and experiences are very important to us. If you begin the interview and change your mind, you can withdraw at any time.

If I have additional questions, how can I get them answered?

You can contact Fred Doolittle of MDRC at (212) 340-8638 or [email protected].

************************************************************************

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number.  The valid OMB control number for this information collection is #1850-NEW.  The time to complete this interview is estimated to be 45-60 minutes, including time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. The obligation to respond to this collection is voluntary.  If you have comments or concerns regarding the status of your individual submission of this form, application or survey, please write directly to Lauren Angelo, U.S. Department of Education, Institute of Education Sciences, 555 new Jersey Avenue, NW, Suite 502h, Washington, D.C. 20208 or email [email protected]

VOLUNTARY CONSENT

I have read this form, and all of my current questions have been answered. I understand that I am encouraged to ask questions about any aspect of this research during the course of the study, and that such future questions will be answered by a qualified individual. Any questions I have about either the study or my rights as a research participant will be answered by the Project Director at MDRC (Fred Doolittle) – (212) 340-8638 or [email protected].

By signing this form, I agree to participate in this interview. A copy of this consent form will be given to me. I know that participation in these data collection activities is voluntary and that all information about me will be kept private to the extent allowed by law. I know that I can refuse to provide any information or stop participating in data collection activities at any time.

___________________________________ ______________________

Name of Participant (please print) District

___________________________________ _____________________

Participant’s Signature Date

Do you have any questions before we begin?

1. Classroom components—general (TFI 1.8)

For each of the schools in which you coach, please estimate how many teachers are partially or fully implementing the CSBS classroom components. (none, less than half, more than half, all)

How useful were the CSBS trainings and supports in preparing you to train teachers in the CSBS classroom components (e.g., 8 classroom components or monthly classroom practices)? (not at all useful, somewhat useful, useful, very useful)

Have you encountered any challenges in training teachers in the classroom components? (yes, no) If yes, please describe the challenges you encountered in training teachers in the classroom components. (e.g., no time for professional learning communities to meet; low teacher buy-in; low principal support; I could not attend the monthly webinars [probe for which schools faced which challenges])

How helpful has the CSBS trainer been in addressing these challenges? (not at all helpful, somewhat helpful, very helpful) What, if anything, have they done to help? (e.g., additional training, individualized coaching, sharing of materials)

What additional supports could the CSBS trainer have provided to help you support the implementation of the classroom components? (e.g., additional time, additional content, additional opportunities to practice)

2. Positive Behavior Game

For each of the schools in which you coach, please estimate how many teachers are partially or fully implementing the Positive Behavior Game. (none, less than half, more than half, all)

How useful were the CSBS trainings and supports in preparing you to support teachers in the use of the Positive Behavior Game? (not at all useful, somewhat useful, useful, very useful)

Have you encountered any challenges in training/supporting teachers in their use of the Positive Behavior Game? (yes, no) If yes, please describe any challenges you encountered in training/supporting teachers in the positive behavior game. (e.g., low attendance at initial training; low teacher buy-in; low principal support [probe for which schools faced which challenges])

How helpful has the CSBS trainer been in addressing these challenges? (not at all helpful, somewhat helpful, very helpful) What, if anything, have they done to help? (e.g., additional training, individualized coaching, sharing of materials)

What additional supports could the CSBS trainer have provided to help you support the implementation of the Positive Behavior Game? (e.g., additional time, additional content, additional opportunities to practice)

3. Identifying teachers for targeted coaching supports

How useful were the CSBS trainings and supports in preparing you to identify teachers who need additional support? (not at all useful, somewhat useful, useful, very useful)

Have you encountered any challenges in identifying teachers who need additional support? (yes, no) If yes, please describe the challenges you encountered in identifying teachers for additional support. (e.g., teachers don’t self-nominate; team and admin not collecting data; no time in meetings to identify teachers for additional support [probe for which schools faced which challenges])

How helpful has the CSBS trainer been in addressing these challenges? (not at all helpful, somewhat helpful, very helpful) What, if anything, have they done to help? (e.g., additional training, individualized coaching, sharing of materials)

What additional supports could the CSBS trainer have provided to help you identify teachers for additional support? (e.g., additional time, additional content, additional opportunities to practice)

4. Coaching teachers

How useful were the CSBS trainings and supports in preparing you to use the 6 step Classroom CheckUp process with teachers who need additional support? (not at all useful, somewhat useful, useful, very useful)

Have you encountered any challenges in using the Classroom CheckUp? (yes, no) If yes, please describe any challenges you encountered in using the classroom check-up. (e.g., teachers resistant to observation; no time to complete process [probe for the schools in which they faced a challenge])

How helpful has the CSBS trainer been in addressing these challenges? (not at all helpful, somewhat helpful, very helpful) What, if anything, have they done to help? (e.g., additional training, individualized coaching, sharing of materials)

What additional supports could the CSBS trainer have provided to help you use the 6 step Classroom CheckUp process with teachers who need additional support? (e.g., additional time, additional content, additional opportunities to practice)

5. Teams

For each of the schools in which you coach, please describe the frequency with which the school is fully using the CSBS TIPS process for School Leadership Team meetings. (never, sometimes, all the time)

How useful were the CSBS trainings and supports in preparing you to support school teams in their use of the TIPS process? (not at all useful, somewhat useful, useful, very useful)

Have you encountered any challenges in supporting team to use the TIPS process? (yes, no) If yes, please describe the challenges you encountered in supporting schools use of the TIPS process. (e.g., my schedule does not allow me to attend meetings; teams resist protocol; trouble with online team meeting minutes form; low principal-support [probe for which schools faced which challenges])

How helpful has the CSBS trainer been in addressing these challenges? (not at all helpful, somewhat helpful, very helpful) What, if anything, have they done to help? (e.g., additional training, individualized coaching, sharing of materials)

What additional supports could the CSBS trainer have provided to help you support the teams in their use of the TIPS process? (e.g., additional time, additional content, additional opportunities to practice)

6. Classroom walkthroughs

How useful were the CSBS trainings and supports in preparing you to conduct classroom walkthroughs?

Have you encountered any challenges in completing classroom walkthroughs? (yes, no) If yes, please describe challenges you have face in completing classroom walkthroughs. (e.g., don’t understand protocol; no time; teacher resistance to observations [probe for the schools in which you faced these challenges])

How helpful has the CSBS trainer been in addressing these challenges? (not at all helpful, somewhat helpful, very helpful) What, if anything, have they done to help? (e.g., additional training, individualized coaching, sharing of materials)

What additional supports could the CSBS trainer have provided to help you conduct walkthroughs? (e.g., additional time, additional content, additional opportunities to practice)

7. CSBS trainings

Overall, what components of CSBS training and support have been most helpful in fulfilling your role as an MTSS-B Coach? (e.g., specific training sessions [team training, SY site visits, webinars]; specific modules; opportunities to practice; relationship with trainer; opportunities to connect with others in the district)

How could CSBS training and support be improved to better support your role? (e.g., materials more tailored to my schools; more opportunities for practice; more access to trainers; more engaging training sessions)

8. PBIS Principal Support

For each of the schools in which you coach, please describe the level of commitment (low commitment, some commitment, full commitment) to MTSS-B implementation that the principal has in regards to:

• Allocating time and resources to support MTSS-B implementation

• Developing and implementing MTSS-B activities and programs

• Participating in the School Leadership Team

• Promoting MTSS-B among school staff

Please describe any challenges the school principal has faced in fulfilling his or her role in the implementation of MTSS-B. (e.g., too many competing initiatives; poor relationship with staff; new principal; no time to attend training; low levels of resources)

9. Tier II (spring 2017 only)

For each of the schools in which you coach, please estimate how many teachers are partially or fully implementing the Check In Check Out Tier II intervention (providing feedback to their CICO student multiple times per day)? (e.g., none, less than half, more than half, all)

How useful were the CSBS trainings and supports in preparing you to support teachers in their use of CICO? (not at all useful, somewhat useful, useful, very useful)

Have you encountered any challenges in supporting teachers in their use of CICO? (yes, no)

If yes, please describe the challenges you have faced in supporting teachers in their use of CICO.

(e.g,. students not identified; teachers don’t provide feedback; I have no time to monitor Tier II [probe for which schools faced which challenges])

How helpful has the CSBS trainer been in addressing these challenges? (not at all helpful, somewhat helpful, very helpful) What, if anything, have they done to help? (e.g., additional training, individualized coaching, sharing of materials)

What additional supports could the CSBS trainer have provided to help you support teachers in their implementation of CICO? (e.g., additional time, additional content, additional opportunities to practice)

10. Tier II Team (spring 2017 only)

For each of the schools in which you coach, please describe the frequency with which the school is fully using the CSBS TIPS process for Targeted Team meetings. (never, sometimes, all the time)

How useful were the CSBS trainings and supports in preparing you to support school teams in their use of the TIPS process? (not at all useful, somewhat useful, useful, very useful)

Have you encountered any challenges in supporting schools’ use of the TIPS process during Targeted Team meetings? (yes, no) If yes, please describe the challenges you have faced in supporting schools’ use of the TIPS process during Targeted Team meetings. (e.g., my schedule does not allow me to attend meetings; teams resist protocol; trouble with online team meeting minutes form; low principal-support [probe for which schools faced which challenges])

How helpful has the CSBS trainer been in addressing these challenges? (not at all helpful, somewhat helpful, very helpful) What, if anything, have they done to help? (e.g., additional training, individualized coaching, sharing of materials)

What additional supports could the CSBS trainer have provided to help you support the Targeted Teams in their use of the TIPS process? (e.g., additional time, additional content, additional opportunities to practice)