Impact Evaluation of Training in Multi-Tiered Systems of Support for Reading in Early Elementary School

MTSS-R
Team Leader Interview

Treatment School Opening Script and Consent

Interviewer name: _____________________	District ID: _____________________
Date of interview: _____________________	School ID: _____________________
	Staff ID: _____________________

Thank you for taking the time to speak with us today. My name is [name], and I’m joined by my colleague [name]. We are conducting interviews with each MTSS-R team leader in a school implementing [ECRI/CORE Partners] in order to learn how MTSS-R implementation is going and to get your perceptions on the training and support you and your school received from the [PROVIDER] and your assigned MTSS-R coach. We hope you will participate because your perspective is essential to understanding how to support MTSS-R implementation in schools across the country.

Both [name] and I work as part of the evaluation team—that is, we do not work for [PROVIDER], and we will not share any information you give us with [PROVIDER], the MTSS-R coach, or district leadership. We are interested in knowing about your experiences with MTSS-R trainings and ongoing technical assistance and implementation of MTSS-R in Grades 1 and 2 to inform the field. Therefore, please feel free to be candid in your response.

We anticipate that the interview will take approximately 40 minutes, but we scheduled an hour to make sure we will have enough time.

Before we ask you to participate, we want you to know that participation in this interview is voluntary. All your responses will be kept private and will only be used for this study and related research. All data that we collected will be reported in aggregate form only and will not identify individuals, schools, or districts. Information collected for this study falls under the confidentiality and data protection requirements of the Institute of Education Sciences. All information from this study will be kept confidential as required by the Education Sciences Reform Act of 2002 (Title I, Part E, Section 183). Responses to this interview will be used only for statistical purposes. Personally identifiable information about individual respondents will not be reported. The study team will not provide information that identifies you, your school, or your district to anyone outside the study team, except as required by law.

Do you agree to participate in this interview?

 Interviewee agrees to participate in the interview

 Interviewee does not agree to participate in the interview.

If interviewee agrees to participate, say: I’d like to record the rest of this call, as a backup for my notes. You can ask me to stop recording at any time. Do I have your permission to record the interview?

 Yes  No

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid Office of Management and Budget (OMB) control number. The valid OMB control number for this information collection is XXXX-XXXX. The time to complete this interview is estimated to average 40 minutes, including time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. The obligation to respond to this collection is voluntary. If you have comments or concerns regarding the status of your individual submission of this form, application, or survey, please write directly to Lauren Angelo, U.S. Department of Education, Institute of Education Sciences, 550 12th Street, SW, Room 4104, Washington, DC 20004 or email [email protected].

Note: All interviews will involve one interviewer and one note taker. The note taker will take notes within the word document, and notes will be transferred to database for analysis. The interviews will be conducted through a video/phone interview. The interviewer will read the item options out for the interviewee.

1. Professional Development: Provider Training and Ongoing Support

First, we’d like to ask how helpful the trainings and supports have been for implementing MTSS-R in Grades 1 and 2, by having you rank each training activity on a scale of 0 to 4, with 0 being ‘I don’t know/don’t recall’, 1 being not at all helpful and 4 being very helpful. Also, please describe why you chose that rating.

1. Please rate and then describe how helpful you found the following components of training by [PROVIDER]:

1. How helpful were the meetings held in late spring or early summer of 2021 by the [PROVIDER] in ensuring that the school has adequate infrastructure for MTSS-R implementation (e.g., the MTSS-R team and an adequate screening and progress monitoring system in place).

I don’t know/ don’t recall	Not at all helpful			Very helpful
0	1	2	3	4

Explanation:

2. How helpful were the summer or early fall of 2021/2022/2023 teacher trainings by [PROVIDER] to improve Tier I instruction (e.g., explicit instruction and data-based differentiated instruction)

I don’t know/ don’t recall	Not at all helpful			Very helpful
0	1	2	3	4

Explanation:

3. How helpful were the summer or early fall of 2021/2022/2023 interventionist/school staff member training by [PROVIDER] to support delivery of Tier II intervention as intended.

I don’t know/ don’t recall	Not at all helpful			Very helpful
0	1	2	3	4

Explanation:

4. How helpful were the summer or early fall of 2021/2022/2023 MTSS-R team training by [PROVIDER] to lead and monitor MTSS-R implementation.

I don’t know/ don’t recall	Not at all helpful			Very helpful
0	1	2	3	4

Explanation:

5. How helpful were the site visits this past year by [PROVIDER] to help MTSS-R teams, teachers, and interventionist to problem solve challenges and foster MTSS-R model implementation as intended.

I don’t know/ don’t recall	Not at all helpful			Very helpful
0	1	2	3	4

Explanation:

6. How helpful were the monthly support events (e.g., webinars, or consultation sessions) by [PROVIDER] to support MTSS-R implementation.

I don’t know/ don’t recall	Not at all helpful			Very helpful
0	1	2	3	4

Explanation:

7. How helpful were the other videoconferences, phone calls, and e-mails you participated in with [PROVIDER] to address specific MTSS-R implementation issues/challenges.

Not applicable	Not at all helpful			Very helpful
0	1	2	3	4

Explanation:

2. Do you think that any of the following [PROVIDER] activities would have helped your school to more effectively address MTSS-R implementation challenges?

• Trainings that were more customized to match the needs of my school

• Practice and planning time during and after the training events

• More engaging, active learning tasks during the trainings

• The [PROVIDER] visiting my school more often

• The [PROVIDER] focusing more on working with teachers

• The [PROVIDER] focusing more on working with school staff delivering Tier II intervention

• The [PROVIDER] focusing more on working with the MTSS-R team

• The [PROVIDER] focusing more on how to support students at risk for reading difficulty (e.g., students receiving Tier II intervention)

• The [PROVIDER] focusing more on how to support students with a learning disability in reading

• The [PROVIDER] focusing more on how to support English learners

• More focus on another topic (specify): _______________________________

• Other (specify) ____________________________________

2. Tier I Instruction

We would now like to ask how well you think classroom teachers are implementing strategies that they were trained to use by [PROVIDER] to implement Tier I classroom reading instruction.

3. Thinking across all the 1^st and 2^nd grade teachers who teach reading, we’d like to know how many are implementing explicit instruction strategies and data-based differentiated instruction strategies at an adequate level or higher. Please select one of the following six responses: all, most, about half, less than half, few, or none. Let’s start with ‘explicit instruction strategies’ and then move to ‘differentiated instruction strategies’.

1. How many 1^st or 2^nd grade teachers are implementing explicit instruction strategies at an adequate level or higher? By explicit instruction strategies, we mean modeling, providing feedback to students, using precise language, having students practice, and providing opportunities for students to respond.

 All	 Most	 About half
 Less than half	 Few	 None

2. How many 1st or 2nd grade teachers are implementing data-based differentiated instruction

strategies at an adequate level or higher? By data-based differentiated instruction strategies we mean using data on a regular basis to identify students who need to work on particular aspects of reading or who excel with reading, and grouping students based on data and tailoring instruction to each group.

 All	 Most	 About half
 Less than half	 Few	 None

3. How do you, or how does the MTSS-R team, monitor teachers’ explicit instruction and data-based differentiation?

Describe:

4. How many 1st or 2nd grade teachers are implementing the core reading program as intended, with a high level of fidelity?

 All	 Most	 About half
 Less than half	 Few	 None

5. How many 1st or 2nd grade teachers are implementing the [PROVIDER] approach to Tier instruction as intended, with a high level of fidelity? [For ECRI this means using ECRI lessons plans to replace parts of the core reading program lessons; for CORE Partners this means enhancing core reading program delivery by using strategies and materials from the CORE Teacher Resource Book].

 All	 Most	 About half
 Less than half	 Few	 None

6. Have teachers in your school expressed any challenges, or have you witnessed any challenges, regarding implementation of explicit instruction, data-based differentiated instruction, or the core reading program?

• Yes

• No  Skip to next section (Tier II Supplemental Reading Intervention)

1. Please describe the greatest challenges teachers expressed or that you’ve witnessed (i.e., up to 3 major challenges. (For probes and instant coding, see challenge categories below).

Challenge #	Challenge Description (list below)
1
2
3

Challenge categories (to be assigned by interviewer):

• Teachers lacked sufficient reading content foundational reading content knowledge to implement explicit phonemic awareness, phonics or fluency instruction

• Teachers lacked sufficient reading content knowledge to implement explicit vocabulary or reading comprehension instruction

• Teachers lacked understanding of how to apply differentiated instruction strategies to their lessons

• Teachers lacked understand how to implement [PROVIDER’s] instructional strategies

• Teachers lacked understanding of how to use data to group students for differentiated/small group instruction

• Teachers struggled to deliver differentiated instruction due to large differences in students reading skills (e.g., some students were more than one grade level below or above)

• Teachers struggled to deliver differentiated instruction due to classroom management problems

• Teachers lacked access to the right data to inform their instruction

• Teachers lacked understanding of the core reading program components and expected scope and sequence

• Teachers lacked access to needed core reading program materials

• The structure of the reading block did not allow for sufficient time to implement the [PROVIDER’s] model as intended

• Teachers were resistant to [the PROVIDER’s] model

• Teachers expressed that their students’ needs were too great and they could not meet their needs by following the [PROVIDER’s] model

• Teachers struggled to implement these strategies with SWDs

• Teachers struggled to implement these strategies with ELs

• Other, specify: _________________________________

2. You mentioned that [fill in the blank] was a challenge. I will read a list of activities that the [PROVIDER] or the MTSS-R coach may have engaged in to address the challenge. After I have read aloud each activity, could you let me know whether the [PROVIDER], the MTSS-R coach or both conducted the activity to address the challenge, [fill in the blank]?

	Activities	Addressed by either [PROVIDER] or the MTSS-R Coach
1	Delivered additional training to teachers or the MTSS-R team about [CHALLENGE]	 Yes  No
2	Delivered additional individualized supports to teachers about [CHALLENGE]	 Yes  No
3	Modelled delivery of [PROVIDER’s] Tier I instructional strategies	 Yes  No
4	Provided more materials (e.g., example videos about blending, templates for data-based decision making) related to [CHALLENGE]	 Yes  No
5	Worked with the school leadership and teachers to ensure that the [PROVIDER] model for fit the school’s need	 Yes  No
6	Worked with the school and district leadership to change instructional structures and schedules to ensure that [PROVIDER] model could be implemented	 Yes  No
7	Describe other activities that the provider or MTSS-R coach conducted to address the challenge. Write down:	 Yes  No

3. Were the activities mostly conducted by the [PROVIDER], the MTSS-R coach or were most of the activities conducted by both the [PROVIDER] and the MTSS-R coach?

• The activities were mostly conducted by the [PROVIDER]

• The activities were mostly conducted by the MTSS-R coach

• Most of the activities were conducted both by the [PROVIDER] and the MTSS-R coach

[If the interviewee listed more than one major challenge, questions 4b and 4c will be repeated for each listed major challenge (up to three challenges)]

4. How helpful was the [PROVIDER] or the MTSS-R coach in addressing the [up to three] major challenges you mentioned, [fill in the blank], using the scale of very helpful, somewhat helpful, or not at all helpful?

	Level of helpfulness
	Very helpful	Somewhat helpful	Not at all helpful
Challenge #1: fill in challenge, activities #1-7 selected above			
Challenge #2: fill in challenge, activities #1-7 selected above			
Challenge #3: fill in challenge, activities #1-7 selected above			

5. Other than activities from [PROVIDER] or the MTSS-R coach that we have discussed, what did your school do to overcome the challenges related to implementing explicit instruction, data-based differentiated instruction or the core reading program in Grade 1 or 2 classrooms?

• Addressed the challenges in grade-level or other MTSS-R team meetings

• Addressed the challenges with district personnel/leadership

• District/School provided other trainings on these topics to teachers

• Other, specify: __________________________

6. Would you say that your school was successful, somewhat successful, or not at all successful in addressing these challenges?

 Successful  Somewhat successful  Not at all successful

3. Tier II Supplemental Reading Intervention

We would now like to ask how well you think the staff who provide supplemental reading intervention (or Tier II intervention) are implementing the strategies they were trained in by [PROVIDER].

7. How many of the staff that provide supplemental reading intervention to 1^st or 2^nd grade students are delivering intervention at an adequate level or higher? Again, please respond using one of the following six options: all, most, about half, less than half, few, or none

 All	 Most	 About half
 Less than half	 Few	 None

1. How do you, or how does the MTSS-R team, monitor interventionists’ performance?

Describe:

8. Have school staff who provide supplemental reading intervention expressed any challenges, or have you witnessed any challenges, in implementing the Tier II intervention model that they were trained in by [PROVIDER]?

• Yes

• No  Skip to next section (MTSS-R Infrastructure: Team Meetings)

1. Please describe the biggest challenges teachers expressed or that you’ve witnessed (i.e., up to 3 major challenges) related to delivering supplemental Tier II reading intervention. (For probes and instant coding, see challenge categories below).

Challenge #	Challenge Description
1
2
3

Challenge categories (to be assigned by interviewer):

• Adjusting instructional schedules to deliver Tier II intervention was difficult

• Aligning content of Tier II intervention with the core reading program scope and sequence was difficult

• Not enough staff to deliver Tier II intervention

• Staff delivering Tier II intervention did not have access to needed materials

• School staff delivering Tier II intervention were resistant to provider’s Tier II intervention model/program

• Staff delivering Tier II were overwhelmed with the large number of students to serve

• Staff struggled implementing Tier II intervention with ELs

• Staff struggled implementing Tier II intervention with SWDs

• Other, specify: _________________________________

2. You mentioned that [fill in the blank] was a challenge. I will read a list of activities that the [PROVIDER] or the MTSS-R coach may have engaged in to address the challenge. After I have read aloud each activity, could you let me know whether the [PROVIDER], the MTSS-R coach or both conducted the activity to address the challenge, [fill in the blank], you mentioned above.

	Activities	Addressed by either [PROVIDER] or the MTSS-R Coach
1	Delivered additional training to school staff delivering Tier II intervention about [CHALLENGE]	 Yes  No
2	Delivered additional individualized supports to school staff delivering Tier II intervention about [CHALLENGE]	 Yes  No
3	Modelled delivery of [PROVIDER’s] Tier II intervention	 Yes  No
4	Provided more materials (e.g., example videos of implementation)	 Yes  No
5	Worked with the school leadership and staff to ensure that the [PROVIDER] Tier II intervention fit the school’s need	 Yes  No
6	Worked with the school and district leadership to change instructional structures, schedules and staffing to ensure that Tier II intervention could be implemented as intended	 Yes  No
7	Describe other activities that the provider or MTSS-R coach conducted to address the challenge. Write down:	 Yes  No

3. Were the activities mostly conducted by the [PROVIDER], the MTSS-R coach or were most of the activities conducted by both the [PROVIDER] and the MTSS-R coach?

• The activities were mostly conducted by the [PROVIDER]

• The activities were mostly conducted by the MTSS-R coach

• Most of the activities were conducted both by the [PROVIDER] and the MTSS-R coach

[If the interviewee listed more than one major challenge, questions 6b and 6c will be repeated for each listed major challenge (up to three challenges)]

4. How helpful was the [PROVIDER] or the MTSS-R coach in addressing the [up to three] major challenges you mentioned, [fill in the blank], using the scale of very helpful, somewhat helpful, or not at all helpful?

   	Level of helpfulness
   	Very helpful	Somewhat helpful	Not at all helpful
   Challenge #1: fill in the challenge, activities #1-7 selected above			
   Challenge #2: fill in the challenge, activities #1-7 selected above			
   Challenge #3: fill in the challenge, activities #1-7 selected above			

5. Other than activities from [PROVIDER] or the MTSS-R coach that we have discussed, what did your school do to overcome the challenges related to implementing Tier II intervention?

• Addressed staffing issues with district personnel

• Changed instructional schedules

• Provided additional trainings or support to teachers or interventionists delivering Tier II intervention

• Other, specify: __________________________

6. Would you say that your school was successful, somewhat successful, or not at all successful in addressing these challenges?

 Successful  Somewhat successful  Not at all successful

4. MTSS-R Infrastructure: Team Meetings

9. Would you say that your school fully implemented the MTSS-R team meeting process recommended by [PROVIDER], mostly implemented this process, partially implemented this process, or did not implement this process at all?

 Fully implemented	 Partially implemented
 Mostly implemented	 Not at all implemented

10. Has your team encountered any challenges fulfilling the key responsibilities of the MTSS-R team or following the meeting procedures (e.g., creating agendas, documenting meetings, informing other school staff about decisions) that they were trained in by [PROVIDER]?

• Yes

• No  Skip to next section (Screening)

1. Please describe the biggest challenges MTSS-R team members expressed or that you’ve witnessed (i.e., up to 3 major challenges) related to fulfilling the responsibilities of the MTSS-R team or following the meeting procedures (e.g., creating agendas, documenting meetings, informing other school staff about decisions) that they were trained in by [PROVIDER]. (For probes and instant coding, see challenge categories below).

Challenge #	Challenge Description
1
2
3

Challenge categories (to be assigned by interviewer):

• Team struggled fulfilling team positions

• There was not enough time to hold meetings during school days

• The team members wanted to meet less often

• The team meeting processes as laid out by [PROVIDER] did not align with the typical meeting procedures staff have been used to using in the past

• Team struggled with making instructional or tier movement decisions based on data

• Team struggled with collecting, interpreting and/or using MTSS-R fidelity data to guide implementation

• Other, specify: _________________________________

2. You mentioned that [fill in the blank] was a challenge. I will read a list of activities that the [PROVIDER] or the MTSS-R coach may have engaged in to address the challenge. After I have read aloud each activity, could you let me know whether the [PROVIDER], the MTSS-R coach or both conducted the activity to address the challenge, [fill in the blank], you mentioned above.

	Activities	Addressed by either [PROVIDER] or the MTSS-R Coach
1	Delivered additional training or support about [CHALLENGE]	 Yes  No
2	Participated in /helped the MTSS-R team to run meetings	 Yes  No
3	Assisted with analyzing data (screening, fidelity data)	 Yes  No
4	Provided more materials (e.g., example videos, agenda templates)	 Yes  No
5	Worked with the school leadership and teachers to ensure that the [PROVIDER] model fit the school’s need	 Yes  No
6	Worked with the school and district leadership to change instructional structures and schedules to ensure that [PROVIDER] MTSS-R team can meet during school hours	 Yes  No
7	Describe other activities that the provider or MTSS-R coach conducted to address the challenge. Write down:	 Yes  No

3. Were the activities mostly conducted by the [PROVIDER], the MTSS-R coach or were most of the activities conducted by both the [PROVIDER] and the MTSS-R coach?

• The activities were mostly conducted by the [PROVIDER]

• The activities were mostly conducted by the MTSS-R coach

• Most of the activities were conducted both by the [PROVIDER] and the MTSS-R coach

[If the interviewee listed more than one major challenge, questions 8b and 8c will be repeated for each listed major challenge (up to three challenges)]

4. How helpful was the [PROVIDER] or the MTSS-R coach in addressing the [up to three] major challenges you mentioned, [fill in the blank], using the scale of very helpful, somewhat helpful, or not at all helpful?

   	Level of helpfulness
   	Very helpful	Somewhat helpful	Not at all helpful
   Challenge #1: fill in the challenge, activities #1-7 selected above			
   Challenge #2: fill in the challenge, activities #1-7 selected above			
   Challenge #3: fill in the challenge, activities #1-7 selected above			

5. Other than the activities from [PROVIDER] or the MTSS-R coach that we have discussed, what did your school do to overcome the challenges related to implementing the MTSS-R team meeting process?

• Found a way to hold meetings during school hours

• Shortened or reduced frequency of meetings

• Change of MTSS-R team members/roles

• Adapted the meeting process to better suit the school’s needs

• Participated in additional trainings organized by the school/district about MTSS-R team roles and meetings

• Other, specify: __________________________

6. Would you say that your school was successful, somewhat successful, or not at all successful in addressing these challenges?

 Successful  Somewhat successful  Not at all successful

5. Screening

We’d now like to turn to your school’s use of screening data to identify students for Tier II intervention.

11. Would you say that your school fully implemented the screening process for identifying students for Tier II recommended by [PROVIDER], mostly implemented this process, partially implemented this process, or did not implement this process at all? For the purpose of this interview full implementation means that “screening is administered at multiple times a year, staff enter the screening data in a timely manner, the team reviews the screening data, the team uses a common cutoff point for identification of students for Tier II intervention, and the team documents all exceptions, alternative-decision rules, and additional data sources used in decision making.”

 Fully implemented	 Partially implemented
 Mostly implemented	 Not at all implemented

12. Has your school encountered any challenges in using the screening process?

• Yes

• No  Skip to next section (Progress Monitoring)

1. Please describe the biggest challenges MTSS-R team members, teachers and staff members in have expressed or that you’ve witnessed (i.e., up to 3 major challenges) related to screening. (For probes and instant coding, see challenge categories below).

Challenge #	Challenge Description
1
2
3

Challenge categories (to be assigned by interviewer):

• Not enough trained staff to conduct screenings efficiently

• The screening was delayed

• There was a long lag between screening students and when the data were available to analyze

• The decision-making rules for identifying students for Tier II supplemental reading intervention were not clear

• The decision-making rules for identifying students for Tier II supplemental reading intervention identified more students than could be served given staff capacity and schedule

• Staff wanted to use a different screening tool

• The screening process did not work for EL students

• Challenge specific to SWDs

• Other, specify: _________________________________

2. You mentioned that [fill in the blank] was a challenge. I will read a list of activities that the [PROVIDER] or the MTSS-R coach may have engaged in to address the challenge. After I have read aloud each activity, could you let me know whether the [PROVIDER], the MTSS-R coach or both conducted the activity to address the challenge, [fill in the blank], you mentioned above.

	Activities	Addressed by either [PROVIDER] or the MTSS-R Coach
1	Delivered additional training about [CHALLENGE]	 Yes  No
2	Helped to collect the screening data	 Yes  No
3	Helped to identify students eligible for Tier II intervention	 Yes  No
4	Helped to place students in the correct starting point for Tier II program	 Yes  No
5	Provided more materials (e.g., example videos)	 Yes  No
6	Worked with the school leadership and teachers to ensure that the [PROVIDER] model for screening fit the school’s need	 Yes  No
7	Describe other activities that the provider or MTSS-R coach conducted to address the challenge. Write down:	 Yes  No

3. Were the activities mostly conducted by the [PROVIDER], the MTSS-R coach or were most of the activities conducted by both the [PROVIDER] and the MTSS-R coach?

• The activities were mostly conducted by the [PROVIDER]

• The activities were mostly conducted by the MTSS-R coach

• Most of the activities were conducted both by the [PROVIDER] and the MTSS-R coach

[If the interviewee listed more than one major challenge, questions 10b and 10c will be repeated for each listed major challenge (up to three challenges)]

4. How helpful was the [PROVIDER] or the MTSS-R coach in addressing the [up to three] major challenges you mentioned, [fill in the blank], using the scale of very helpful, somewhat helpful, or not at all helpful?

	Level of helpfulness
	Very helpful	Somewhat helpful	Not at all helpful
Challenge #1: fill in the challenge, activities #1-7 selected above			
Challenge #2: fill in the challenge, activities #1-7 selected above			
Challenge #3: fill in the challenge, activities #1-7 selected above			

5. Other than activities from [PROVIDER] or the MTSS-R coach that we have discussed, what did your school do to overcome the challenges related to the screening process?

• Received more training or support from the district to conduct screening, identifying students for Tier II intervention

• Worked with the district to make the data collections and data availability more timely

• Added quality control step(s) to the data collection process

• Modified data-based decision-making rules to be more relevant for my school’s context

• Other, specify: __________________________

6. Would you say that your school was successful, somewhat successful, or not at all successful in addressing these challenges?

 Successful  Somewhat successful  Not at all successful

6. Progress Monitoring

We’d now like to turn to your school’s use of progress monitoring data to identify students to exit the supplemental Tier II reading intervention or to be referred to other or more intensive Tier III intervention.

13. How frequently does your school typically administer the progress monitoring assessment to each student who receives Tier II intervention?

• A few times per year

• About once a month

• About twice a month

• Nearly every week

• At least once a week

14. How often does the MTSS-R team review or analyze the progress monitoring data?

• A few times per year

• About once a month

• About twice a month

• Nearly every week

• At least once a week

15. Have staff in your school experienced any challenges in using progress-monitoring data to inform instructional decision making?

• Yes

• No  Skip to next section (MTSS-R Infrastructure: Fidelity Data)

1. Please describe the biggest challenges MTSS-R team members, teachers and other school staff members in have expressed or that you’ve witnessed (i.e., up to 3 major challenges) related to progress monitoring. (For probes and instant coding, see challenge categories below).

Challenge #	Challenge Description
1
2
3

Challenge categories (to be assigned by interviewer):

• Not enough trained staff to conduct progress monitoring as intended

• The data were not collected with intended frequency

• The data were not always recorded by teachers/interventionists

• The data was not available in a timely manner for school staff to use it to make decisions about discontinuing Tier II intervention

• The decision-making rules for data-based decision making were not clear

• Challenge specific to ELs

• Challenge specific to SWDs

• Other, specify: _________________________________

2. You mentioned that [fill in the blank] was a challenge. I will read a list of activities that the [PROVIDER] or the MTSS-R coach may have engaged in to address the challenge. After I have read aloud each activity, could you let me know whether the [PROVIDER], the MTSS-R coach or both conducted the activity to address the challenge, [fill in the blank], you mentioned above.

	Activities	Addressed by either [PROVIDER] or the MTSS-R Coach
1	Delivered additional training about [CHALLENGE]	 Yes  No
2	Helped to collect progress monitoring data	 Yes  No
3	Helped to analyze the progress monitoring data	 Yes  No
4	Helped to create or clarify decision making rules to move student back to Tier I instruction or to more intensified Tier III intervention	 Yes  No
5	Provided more materials (e.g., example videos)	 Yes  No
6	Worked with the school leadership and teachers to ensure that the [PROVIDER] model for progress monitoring fit the school’s need	 Yes  No
7	Describe other activities that the provider or MTSS-R coach conducted to address the challenge. Write down:	 Yes  No

3. Were the activities mostly conducted by the [PROVIDER], the MTSS-R coach or were most of the activities conducted by both the [PROVIDER] and the MTSS-R coach?

• The activities were mostly conducted by the [PROVIDER]

• The activities were mostly conducted by the MTSS-R coach

• Most of the activities were conducted both by the [PROVIDER] and the MTSS-R coach

[If the interviewee listed more than one major challenge, questions 13b and 13c will be repeated for each listed major challenge (up to three challenges)]

4. How helpful was the [PROVIDER] or the MTSS-R coach in addressing the [up to three] major challenges you mentioned, [fill in the blank], using the scale of very helpful, somewhat helpful, or not at all helpful?

	Level of helpfulness
	Very helpful	Somewhat helpful	Not at all helpful
Challenge #1: fill in the challenge, activities #1-7 selected above			
Challenge #2: fill in the challenge, activities #1-7 selected above			
Challenge #3: fill in the challenge, activities #1-7 selected above			

5. Other than activities from [PROVIDER] or the MTSS-R coach that we have discussed, what did your school do to overcome the challenges related to using progress-monitoring data to inform instructional decision making?

• Received more training or support from the district to conduct progress monitoring

• Discussed the decision-making rules with district personnel to adjust them to better fit school needs

• Revamped the schedule so that staff had adequate time to collect and analyze data

• Provided additional training and support to interventionists and teachers on how to use data to inform instruction and student movement between tiers

• Provide more materials

• Other, specify: __________________________

6. Would you say that your school was successful in addressing these challenges, somewhat successful, or not at all successful in addressing these challenges?

 Successful  Somewhat successful  Not at all successful

7. MTSS-R Infrastructure: Fidelity Data

16. How often did you and the MTSS-R team members review fidelity data that [PROVIDER] had you collect to monitor MTSS-R?

• A few times this year

• About once a month

• About twice a month

• Nearly every week

17. How did you use the fidelity data that [PROVIDER] had you collect to monitor MTSS-R?

• To understand how well the different components of MTSS-R were being implemented

• To guide changes made to MTSS-R implementation

• To plan additional trainings and support with the [PROVIDER]

• To plan additional trainings and support with the MTSS-R coach

• Other, specify: __________________________

18. Could you please describe, briefly, how your team assesses overall implementation fidelity?

Description:

19. Were any of the following among the three biggest challenges that MTSS-R team encountered any challenges in collecting or analyzing fidelity data?

• Yes

• No  End interview.

1. Please describe the biggest challenges MTSS-R team members have expressed or that you’ve witnessed (i.e., up to 3 major challenges) related to fidelity of MTSS-R. (For probes and instant coding, see challenge categories below).

Challenge #	Challenge Description
1
2
3

Challenge categories (to be assigned by interviewer):

• There was not enough time to collect fidelity data

• There was not enough time to analyze fidelity data

• School staff found the [PROVIDER] fidelity protocols confusing

• School staff didn’t receive enough training on how to analyze or interpret the data

• School staff did not know how to adjust MTSS-R implementation based on the fidelity data

• Teachers found it difficult to change their core reading program delivery to attain higher level of fidelity

• Staff delivering Tier II intervention found it difficult to change their program delivery to attain higher level of fidelity

• Other, specify: _________________________________

2. You mentioned that [fill in the blank] was a challenge. I will read a list of activities that the [PROVIDER] or the MTSS-R coach may have engaged in to address the challenge. After I have read aloud each activity, could you let me know whether the [PROVIDER], the MTSS-R coach or both conducted the activity to address the challenge, [fill in the blank], you mentioned above.

	Activities	Addressed by either [PROVIDER] or the MTSS-R Coach
1	Delivered additional training to MTSS-R team about [CHALLENGE]	 Yes  No
2	Provided more materials (e.g., example videos)	 Yes  No
3	Collected the fidelity data for the MTSS-R team	 Yes  No
4	Analyzed the fidelity data for the MTSS-R team	 Yes  No
5	Worked with the school leadership and teachers to ensure that the [PROVIDER] model for fidelity data collection and use fit the school’s need	 Yes  No
6	Describe other activities that the provider or MTSS-R coach conducted to address the challenge. Write down:	 Yes  No

3. Were the activities mostly conducted by the [PROVIDER], the MTSS-R coach or were most of the activities conducted by both the [PROVIDER] and the MTSS-R coach?

• The activities were mostly conducted by the [PROVIDER]

• The activities were mostly conducted by the MTSS-R coach

• Most of the activities were conducted both by the [PROVIDER] and the MTSS-R coach

[If the interviewee listed more than one major challenge, questions 17b, and 17c will be repeated for each listed major challenge (up to three challenges)]

4. How helpful was the [PROVIDER] or the MTSS-R coach in addressing the [up to three] major challenges you mentioned, [fill in the blank], using the scale of very helpful, somewhat helpful, or not at all helpful?

	Level of helpfulness
	Very helpful	Somewhat helpful	Not at all helpful
Challenge #1: fill in the challenge, activities #1-6 selected above			
Challenge #2: fill in the challenge, activities #1-6 selected above			
Challenge #3: fill in the challenge, activities #1-6 selected above			

5. Other than activities from [PROVIDER] or the MTSS-R coach that we have discussed, what did your school do to overcome the challenges related to collecting and using fidelity data?

• The MTSS-R team adjusted the process of collecting fidelity data

• The MTSS-R team collected data less frequently

• The MTSS-R team decided to use another protocol to measure fidelity

• The MTSS-R team requested more support from the district or other schools on how to analyze or interpret fidelity data

• The MTSS-R team requested more support from the district or other schools on how to change implementation of MTSS-R based on data

• Other, specify: __________________________

6. Would you say that your school was successful, somewhat successful, or not at all successful in addressing these challenges?

 Successful  Somewhat successful  Not at all successful

Ending

These are all the questions we have. Do you have any additional feedback to the study team about the MTSS-R model or the training and supports that you were provided as a part of the study? Thank you so much for your time.

35