School leader Interview Questions

Appendix G: PRISMS Toolkit Evaluation School Leader Interview Questions

Thank you so much for taking the time to talk with me today about your experience with the toolkit. The research team will protect the confidentiality of all information collected for the study and will use it for research purposes only. Only the evaluation team members with training in how to deal with sensitive and confidential data will be allowed access. None of your responses will be individually attributed to you or your school or district and will be used for statistical purposes only. You may opt out from responding to a question or the entire measure at any time with no consequences.

Virtual meeting/Conference recording notice 

The American Institutes for Research^® (AIR^®) allows for the recording of audio, visuals, participants, and other information sent, verbalized, or utilized during business-related meetings. By joining a meeting, you automatically consent to such recordings. Any participant who prefers to participate via audio only should disable their video camera so that only their audio will be captured. Video and/or audio recordings of any AIR session shall not be transmitted to an external third party without the permission of AIR. 

First, I’d like to get to know a little bit about your background. 

1. Please tell me your name and a brief description of your role at the school. 

   1. How long have you been in this role at your school?

   2. What was your job before moving into your current role?

The PRISMS Toolkit was designed to support implementation of the recommendations from the What Works Clearinghouse (WWC) practice guide, Providing Reading Interventions for Students in Grades 4–9. Prior to this interview, we shared a one-page overview of the toolkit components with you, so you can reference it during this interview. [Note: We will provide the toolkit resource roundup infographic. This infographic will be part of component 4 of the toolkit.]

2. How, if at all, did you engage with the PRISMS Toolkit? [If needed, use probes listed in 2a and 2b.]

   1. Which tools did you use personally?

   2. Did you support the use of the toolkit by teachers, coaches, or other staff? If so, how (for example, scheduling time and space for use of online modules or professional learning community [PLC] meetings, watching online modules, attending PLC meetings, delegating use of the school leader walk-through tool or checklist for school leaders to a coach)?

Now, we are going to discuss each component of the toolkit so that I can better understand your perceptions of its utility and quality as well as any barriers you had in using it.

[If the respondent has indicated familiarity with the teacher’s fidelity of implementation tool in the answer to question 2, proceed with this section. Otherwise, skip and go on to the next section.] Let’s start by discussing component 1, the teacher’s fidelity of implementation tool.

3. How, if at all, did you engage with the teachers’ fidelity of implementation tool?

   1. How, if at all, did you use the resulting information (for example, provided space or time for teachers to use it, viewed resulting information, used resulting information to assess current practices in the school or district and how they align with WWC recommendations, used resulting information to plan for professional development or instructional coaching)?

   2. Were there aspects that worked well about the fidelity of implementation tool? If so, what were they?

   3. Were there aspects of the fidelity of implementation tool that were challenging? If so, what were they?

4. Overall, how useful was the teacher’s fidelity of implementation tool in advancing teachers’ understanding of the instructional recommendations from the WWC practice guide?

   1. Did teachers discuss their impressions of and/or takeaways from use of the fidelity of implementation tool with you? If so, what did they say? 

5. Is there anything you would recommend changing to support use of this tool based on your use of the tool or impressions you heard from your teachers or coaches? [If needed, use probes listed in 5a and 5b.]

   1. Was the length of time needed to use the implementation fidelity tool appropriate?

   2. Was language used in the implementation fidelity tool accessible and easy to understand?

[If the respondent has indicated familiarity with the online professional learning modules in the answer to question 2, proceed with this section. Otherwise, skip and go on to the next section.] Now, let’s talk about component 2, the online professional learning modules.

6. How, if at all, did you engage with the online professional learning modules (for example, completed them, participated in discussions with my staff about them, set up a time and place for staff to complete them)?

   1. What worked well about the online modules (for example, duration, mode of presentation, visual scaffolds, modeling of instructional routines, accessibility)?

   2. What was challenging?

7. Overall, how useful were the professional development modules and the resources, activities, and instructional strategies included within these modules in supporting teachers to implement the instructional recommendations from the WWC practice guide?

   1. Did teachers discuss their impressions of and/or takeaways from the professional learning modules with you? If so, what did they say? 

8. Is there anything you would recommend changing to support use of these online modules based on your use of the modules or impressions you heard from your teachers or coaches? [If needed, use probes listed in 8a and 8b.]

   1. Was the length of time needed to take the online modules appropriate?

   2. Was the language used in the online modules accessible and easy to understand?

[If the respondent has indicated familiarity with the PLC tools in the answer to question 2, proceed with this section. Otherwise, skip and go on to the next section.] Now, let’s talk about component 3, the professional learning community (PLC).

9. How, if at all, did you engage with the PLC (for example, set up a time and place for staff to complete them, attended at least one, attended all, read meeting minutes)?

   1. What worked well about the PLCs? (If needed, use prompts listed in 9ai–aix.)

      1. Consistent scheduling of PLCs?

      2. Systems and structures to guide PLCs (for example, use of an agenda, PLC roles, colleagues’ following through on next steps, strong facilitation)?

      3. Duration of PLCs?

      4. Mode of presentation?

      5. Facilitators modeling instructional routines?

      6. Teacher collaboration?

      7. Teachers planning to implement instructional routines in their classrooms?

      8. Improved intervention?

      9. Improved use of student data?

   2. What was challenging? [If needed, use probes listed in 9bi–bxi]

      1. Time constraints?

      2. Schedule challenges?

      3. Role of team members who participated in PLCs?

      4. Communication issues?

      5. Lack of training?

      6. Lack of buy-in from your colleagues, leaders, and/or other staff in your building?

      7. Too many students in need of intervention?

      8. Toolkit did not adequately address the needs of certain student groups (for example, multilingual learners, students with significant disabilities)?

      9. Information conflicted with previous understandings of or beliefs about literacy intervention?

      10. Lack of alignment between WWC recommendations and the school’s curriculum?

      11. Lack of systems and structures to guide PLCs?

10. Overall, how useful were the PLCs and the resources, activities, and instructional strategies included within these PLCs in supporting teachers to implement the instructional recommendations from the WWC practice guide?

1. Did teachers discuss their impressions of and/or takeaways from the PLCs with you? If so, what did they say? 

11. Is there anything you would recommend changing to support the PLCs based on your participation or impressions you heard from your PLC facilitator, teachers, or coaches? [If needed, use probes listed in 11a–11c.]

    1. Was the length of time needed to plan and use the PLC facilitation guide appropriate?

    2. Was the language used in the PLC facilitation guide accessible and easy to understand?

    3. Did you have sufficient staff to support implementation of the PLCs? Please elaborate.

[If the respondent has indicated familiarity with the checklist for school leaders, school leader walk-through tool, toolkit resource roundup, and manual in the answer to question 2, proceed with this section. Otherwise, skip and go on to the next section.] Now, let’s talk about component 4, the checklist for school leaders, school leader walk-through tool, toolkit resource roundup, and manual.

12. How, if at all, did you engage with these tools (for example, used the checklist, used the school leader walk-through tool, read the toolkit resource roundup infographic, read the manual, supported coaches or teachers in using the tools)?

    1. What worked well about these tools (for example, how long they took, mode of presentation, accessibility)?

    2. What was challenging?

    3. About how much time did you need to support implementation of the toolkit?

       1. How much planning time before the school year began?

       2. How many hours per week during implementation?

       3. Time spent on additional structures to implement toolkit recommendations (for example, training teachers to administer and analyze literacy diagnostic assessments)?

13. Overall, how useful were these tools from component 4 in supporting teachers to implement the instructional recommendations from the WWC practice guide?

1. Did coaches discuss their impressions of and/or takeaways from these tools with you? If so, what did they say? 

14. Is there anything you would recommend changing to support use of these tools from component 4 based on your use or impressions you heard from teachers or coaches? [If needed, use probes listed in 14a–14c.]

    1. Was the length of time needed to plan and use the tools appropriate?

       1. If not, what length of time would you recommend?

    2. Was the language used in the tools accessible and easy to understand?

       1. If not, what recommendations do you have to improve the language used?

    3. Did you have sufficient staff to implement the school walk-throughs and checklist for school leaders?

       1. If not, what additional staff or support would you need?

Next, I would like to discuss your impressions of the overall toolkit as you and your teachers, coaches, and PLC facilitators worked to implement the toolkit in your school.

15. How, if at all, did the implementation of the toolkit inform teachers’ instruction this year?
    Across a typical month of classroom observations, how frequently did you observe teachers acting upon the toolkit’s instructional recommendations in their classrooms?

1. Can you provide an example of what you observed?

16. What support was available to you and your teachers as you implemented the toolkit? [If needed, use probes listed in 16a–16g.]

1. School leadership?

2. District-level support?

3. Teacher collaboration?

4. Trusting relationships?

5. Student data?

6. Systems and structures to guide PLCs (for example, consistent, dedicated time to meet; use of an agenda; PLC roles; colleagues’ following through on next steps; strong facilitation)?

7. Alignment with other professional learning priorities in the school or district?

17. To what extent did your staff face any of the following barriers to implementing the toolkit? [If needed, use probes listed in 17a–17h.]

1. Time constraints?

2. Schedule challenges?

3. Role of team members who participated in implementation?

4. Communication issues?

5. Lack of training?

6. Staff buy-in? 

7. Lack of systems and structures to guide PLCs?

8. Misalignment with other professional learning priorities in the school or district?

18. [Skip if respondent does not mention challenges.] How, if at all, were you able to address or overcome these challenges?

We are approaching the end of our time together today. To wrap up: 

19. Is there anything else you think we should know about the toolkit that we have not covered? 

Thank you for your time today.

Regional Educational Laboratory Southwest Toolkit Evaluation 6