Justification

Regional Educational Laboratory, Midwest

Usability and Feasibility Study for the Data-Informed Leadership for Equity (DILE) Approach

OMB# 1850-0952 v.6

Volume I

Supporting Statement

Submitted by:

National Center for Education Evaluation (NCEE)

Institute of Education Sciences (IES)

U.S. Department of Education

Washington, DC

June 2024

Attachments

Attachment I. Culturally responsive practices training interview protocol

Attachment II. School leadership team training, coaching, and ABM interview protocol

Attachment III. Staff survey

Table of Contents

1) Submittal-Related Information 3

2) Background 3

3) Design and Context 4

4) Recruitment and Data Collection 5

5) Estimated Respondent Burden 5

6) Estimate of Costs for Recruiting and Paying Respondents 6

7) Cost to federal government 6

8) Assurance of Confidentiality 6

9) Justification for Sensitive Questions 6

10) Project Schedule 6

1. Submittal-Related Information

Please note that this Generic Information Collection Request (GEN IC) was previously approved under 1850-0952.

The following material is being submitted under the National Center for Education Evaluation (NCEE) generic clearance agreement (OMB# 1850-0952), which provides NCEE the capability to collect preliminary or exploratory information to aid in study design by: (1) fielding brief, quick turnaround surveys, extracting test case administrative data, administering interviews, or conducting “mini-experiments” in advance of a study for the purpose of determining feasibility, a random assignment or comparison group strategy, or a data collection approach most suitable for a potential or planned evaluation; and (2) developing, testing, and improving its survey and assessment instruments, methodologies, and study dissemination strategies.

This request is to conduct interviews with teachers, other school staff, and school leadership team members participating in the Data-Informed Leadership for Equity (DILE) approach field test. These interviews will be used to understand participants’ perceptions of the usability, feasibility, and accessibility of the DILE approach in order to inform refinements to the DILE components to be examined in a future optimization and efficacy study. Interviews will begin in February 2023.

In addition to interviews, REL Midwest is requesting to administer a staff survey to all staff in schools participating in the DILE approach and schools in the same districts not participating in the DILE approach (comparison schools). REL Midwest will administer the staff survey twice, once in September 2024 prior to the student-focused culturally responsive practice training (pre-measure) and once in May 2025 at the end of the school year (post-measure). The staff survey includes six Likert scales: use of data, control over student outcomes, educating all students, collaboration with and view of colleagues, view of the school’s supports provided to teachers and staff, and professional learning about equity. These data will be useful to measure the short-term outcomes of DILE. The staff survey will look at the effects of DILE on staff outcomes in DILE schools compared with non-DILE schools.

2. Background

School leaders’ use of a variety of data elements, including information about students’ educational opportunities and experiences, can facilitate progress toward equity-related goals by helping leaders identify disparities that they can work to address. Furthermore, having systems that facilitate efficient collection and analysis of these data elements can support educators’ decisionmaking about student, classroom, and school needs. Regional Educational Laboratory (REL) Midwest will partner with two to three districts in the Midwest region to develop, field test, and refine a school-level approach that aims to support educators with processes, knowledge, and skills to use real-time local data on racial and other group disparities in sense of belonging and engagement (suspensions and attendance) and to make decisions about strategies to reduce those disparities. This multicomponent approach, which is referred to as Data-Informed Leadership for Equity (DILE), will incorporate professional learning for school leaders, classroom teachers, and other school staff in culturally responsive practices as well as training and tools specifically designed to support the use of data to act upon student disparities in sense of belonging, suspensions, and attendance. The DILE approach uses data to understand where inequities in student outcomes exist, use culturally responsive practices to increase student sense of belonging, and explore and intervene on the system-level factors that might be contributing to differences in experiences. Specifically, the DILE approach has three components: 1) training for staff and school leaders in student-focused culturally responsive practices and the concept of sense of belonging; 2) action-based monitoring (ABM) that consists of adaptive data analysis and reporting algorithms to facilitate monitoring of students’ self-reported sense of belonging, suspensions, and attendance, along with associated written protocols to guide staff in identifying and responding to issues flagged in the reports; 3) training plus coaching for school leadership team to support implementation of student-focused culturally responsive practices and respond to ABM alerts.

REL Midwest is proposing to conduct a field test study of the DILE approach to provide the DILE development team with critical, timely information about participants’ experiences with the DILE components (usability), the contextual factors that support or hinder implementation (feasibility), and the perceived value of the DILE approach (acceptability). This information will guide changes to the approach with an aim to increase usability, feasibility, and acceptability before moving into an efficacy study in the future. The proposed study allows for testing and refining the DILE approach across multiple cohorts. The cohort 1 study provided some promising evidence that the components of the DILE approach that were implemented and studied have potential to benefit educators and students. Unfortunately, despite implementation team efforts to exercise flexibility and minimize burden, ICCSD requested that REL Midwest pause DILE implementation activities with cohort 2 schools. As a result, REL Midwest did not implement DILE in cohort 2 schools. The proposed study will add a third cohort to test the usability, feasibility, and acceptability of the full DILE approach over an entire school year. REL Midwest will collect data from cohort 3 during the 2024/25 school year and use the cohort 3 findings to inform adaptations to DILE and to test for preliminary effectiveness of DILE.

This request is to conduct interviews with teachers, other school staff, and DILE leadership team members participating in DILE to understand what makes (or does not make) DILE a usable, feasible, and acceptable approach that can improve sense of belonging and engagement among students. Through the interviews, teachers, other school staff, and leadership team members will reflect on the extent to which the DILE components and content are easy to understand and implement; will offer suggestions on how the DILE development team can improve the quality and accessibility of DILE content and structure; will highlight contextual factors that support or hinder successful implementation of DILE; will reflect on how the dosage, duration, and density of DILE supported or hindered their participation; and will reflect on how worthwhile and helpful DILE is for support educators with processes, knowledge, and skills to use real-time local data on racial and other group disparities in sense of belonging and engagement (suspensions and attendance) and to make decisions about strategies to reduce those disparities. The culturally responsive practices training interview protocol and the school leadership team training, coaching, and ABM interview protocol are presented in Attachments I and II, respectively. The interview collection during the field test will inform the design of the DILE components.

In addition, this request is to administer staff surveys to staff in DILE and non-DILE schools in the same district to examine exploratory impacts of DILE on teachers outcomes. The staff survey includes six Likert scales: use of data, control over student outcomes, educating all students, collaboration with and view of colleagues, view of the school’s supports provided to teachers and staff, and professional learning about equity. These data will be useful to measure the short-term outcomes of DILE. The staff survey will look at the effects of DILE on staff outcomes in DILE schools compared with non-DILE schools. Thes staff survey scales are presented in Attachment III.

Staff survey. The research team will administer a staff survey to all school staff in all middle schools—DILE schools and non-DILE schools in the participating districts. The staff survey will be administered twice in cohort 3, once in September 2024 prior to the student focused practices training (pre-measure) and once in May 2025 at the end of the school year (post-measure).

Teacher and other staff interviews. At one time point per cohort, the REL Midwest research team will conduct interviews with 9 teachers and other staff members in one school in cohort 1 and up to 4 teachers and other staff members in each of 4 schools in cohort 2, for a total of 16 interviews in cohort 2. In cohort 3, the REL Midwest research team will conduct interviews with 4-6 randomly selected staff members in each of four schools for a total of 16 to 24 interviews. These teacher/other staff interviews will occur once toward the end of the spring semester in each cohort.

School leadership team interviews. REL Midwest will conduct two interviews during the spring semester (2023) with all 6 members of the school leadership team in cohort 1, for a total of 12 interviews in cohort 1. REL Midwest will conduct one interview during the fall semester (2023) and two interviews during the spring semester (2024) with two members of each school’s leadership team in cohort 2, for a total of 36 interviews in cohort 2. REL Midwest will conduct one interview during the fall semester (2024) and one interview during the spring semester (2025) with 4 members of each school’s leadership team in cohort 3, for a total of 32 interviews in cohort 3.

Interviews will be conducted via videoconference and will take no more than 45 minutes to complete.

3. Design and Context

The survey and interviews for which this clearance is requested will be used to understand teacher, other staff, and DILE leadership team perceptions of DILE’s usability, feasibility, and accessibility as well as preliminary impact of DILE on staff outcomes. REL Midwest will use the findings generated from these interviews to inform and develop hypotheses about optimal delivery of DILE for different contexts. Hypotheses will be related to combinations of modalities, dosages, and other DILE components that will be efficacious under different school and district conditions (e.g., efficacious under resource constraints).

Specifically, the interviews will focus on:

• Determining the extent to which the DILE content covered in the trainings and coaching sessions are easy to understand, easy to implement, and facilitates student-focused instruction and use of data,

• Collecting suggestions for improving the quality of DILE content, the accessibility of DILE content, and the modes of delivery for DILE content (i.e., training in student-focused practices, coaching for school leadership team, ABM),

• Identifying district and school contextual factors that support or hinder successful implementation of DILE,

• Determining how the dosage and duration of the trainings and coaching affect participants’ ability to participate and engage fully in DILE, and

• Determining the extent to which teachers, other staff, and DILE leadership team members perceive DILE as worthwhile and helpful for improving instructional practice, use of data, and student outcomes.

The interviews will be a semi-structured discussion with teachers, staff, and leadership team members from one participating school in cohort 1 and four participating schools in cohort 2, and four participating schools in cohort 3. The interviews are intended to maximize the quality of data collected by asking probing questions to better understand what makes (or could make) DILE a usable, feasible, and accessible approach to improve data use and student outcomes.

The interview protocols will have sections covering the following topics:

1. The perceived purpose of DILE and general impressions of DILE.

2. Experiences participating in each DILE component.

3. Experiences implementing the strategies learned in trainings in their classroom or school.

4. Challenges and successes in implementing DILE in their classroom or school.

5. Community, district, and school contextual factors that are necessary to support successful implementation of DILE.

The staff survey will include the following scales:

1. The Use of Data scale is a seven-item scale that focuses on use of data on sense of belonging and engagement (Bos et al., 2022; alpha = 0.71).

2. The Control Over Student Outcomes scale is a seven- item scale about staff’s capacity to influence their students’ students’ sense of belonging (Brady et al., in preparation; alpha = 0.74),

3. The Educating All Students scale is a nine-item scale developed by Panorama about staff’s perceptions of their readiness to address issues of diversity,

4. The Collaboration With and View of Colleagues scale includes 8 items about the extent to which teachers work together and trust each other (Bos et al., 2022; alpha = 0.84),

5. The View of the School’s Supports Provided to Teachers and Staff scale includes 4 items about the extent to which the school provides students and staff resources and support services (Bos et al., 2022; alpha = 0.82).

6. professional learning about equity

The staff survey will look at the effects of DILE on staff outcomes in DILE schools compared with non-DILE schools.

4. Recruitment and Data Collection

REL Midwest will be responsible for selecting DILE participants to interview, obtaining their agreement to be interviewed, conducting the interviews, compiling interview responses, and summarizing findings. Interviewers will ask questions in an open-ended manner using the protocols in Attachments I and II while allowing for new questions or ideas to be brought up during the interview as a result of the interviewee’s responses. Two members of the research team will review the interview transcripts individually and code them separately. Each coder will draw on deductive approaches to help organize data into pre-specified groups or themes (e.g., ease of use, ease of implementation, and cultural responsiveness of DILE) and inductive approaches to make meaning from the data and allow for the possibility that groups or themes other than those identified in advance might emerge. To ensure intercoder reliability, a 20 percent sample of interview transcripts will be double coded. If 95 percent agreement on key groups or themes is not met, the coders will reconcile their codes in a meeting with each other and adjudicate to a single response.

REL Midwest will obtain the list of DILE leadership team members from the DILE development team. The Research team will interview all leadership team members in cohort 1 and will randomly select four leadership team members from each school in cohorts 2 and 3. REL Midwest will obtain from the DILE development team attendance records for the teacher and staff DILE student focused practices trainings. The research team will randomly select 9 teachers and other staff who participated in the training in cohort 1, 4 teachers in each of the 4 schools in cohort 2, and 4-5 teachers in each of the 4 schools in cohort 3 to participate in interviews.

The interviews will be conducted via video conference and will be audio and/or video recorded so that researchers can reference recordings to supplement written notes taken during the interview.

For the staff survey, REL Midwest will work with administrators at each school to obtain staff rosters with email addresses. REL Midwest will invite all school staff in the district schools to participate in the survey via email and will request consent at the start of the survey. REL Midwest will send out up to four reminder emails to increase response rates as needed.

To examine the effects of DILE on teacher outcomes from the staff survey, the research team will examine the extent to which respondents of the staff survey in DILE schools and non-DILE schools are equivalent on the pre-test of the outcome and demographic characteristics (e.g., years of teaching experience, highest degree earned, gender, race/ethnicity). If there is a standardized mean difference (which is a measure of the extent to which differences exist between the two groups) that is greater than 0.05 on any of the observed variables, the team will account for baseline differences by including inverse probability weights (IPTWs) calculated using propensity scores with the WeightIt Package in R (Greifer, 2021). IPTWs reweight the sample so that the distribution of baseline measures (that is, demographic characteristics and outcomes measures at baseline) are more similar across the treatment and comparison groups. Assuming the measured background characteristics accurately capture the important preexisting differences between treatment and comparison staff, IPTWs allow for valid estimates about what DILE school staff would have experienced if they had been in a non-DILE school.

The estimated propensity scores will be used to calculate IPTWs for comparison staff, such that a comparison staff weight equals the staff’s predicted odds of treatment assignment:

equals 1 for staff attending a DILE school and 0 for staff attending a comparison school. With this weight, the comparison group will be weighted to represent the treatment group to facilitate estimation of the average treatment effect on the treated. The IPTW has a value of 1 for all staff attending a DILE school.

The research team will carry out treatment effect estimation using weighted ordinary least squares to examine differences between DILE staff and non-DILE staff on self-efficacy:

Y_i is the outcome for staff i, T_ij is an indicator for being in the treatment (versus business-as-usual) group, is the precision-weighted estimate of the impact of being in a DILE school instead of the comparison group, X_ij is the same vector of teacher characteristics used in the propensity score model, λ_c are district fixed effects, and is an idiosyncratic error. The model includes the same set of covariates used to create the weights to control for any remaining differences between the treatment and comparison groups and to increase precision. Standard errors are clustered at the school level, and the research team also will check the sensitivity of these results to using bootstrapped standard errors given the limited number of schools in the study. The team will consider coefficients of p < .05 as substantively different and also will examine the strength of the effect using Hedges’ g.

This design allows for a rigorous and cost-efficient evaluation of teacher outcomes across multiple schools served by DILE, in which random assignment of schools is not practical. Empirical within-study comparisons have demonstrated that studies using propensity score methods can approach the results of randomized experiments (Pohl et al., 2009; Shadish et al., 2008). This analysis will serve as an exploratory analysis during the pilot phase of DILE to provide evidence of whether the early versions of DILE improve staff self-efficacy and collaboration. Results will be interpreted with caution given the early stages of implementation, and the discussion of results will take into account any implementation challenges identified as part of this study.

5. Estimated Respondent Burden

Table 1 details the respondent burden for the interviews in cohorts 1 and 2. The estimates assume that it will take 15 minutes to schedule interviews and that interviews will take 45 minutes. Each teacher or other staff will be interviewed once. Each DILE leadership team member will be interviewed twice in cohort 1 and three times in cohort 2. On average across the two cohorts and across respondent type, we will have 11 respondents and 18 burden hours.

Table 2 details the respondent burden for the survey and interviews in cohort 3. The interview estimates assume that interviews will take 45 minutes. Each DILE leadership team member will be interviewed twice. Each staff will be interviewed once. The survey estimates assume that it will take 15 minutes to complete the survey. A total of 312 staff across 8 schools (4 DILE schools, 4 comparison schools) will complete the survey twice, once in fall and once in spring. In total we will have 344 respondents and 204 burden hours.

Please note that this Generic Information Collection Request (GEN IC) was previously approved under 1850-0952. This request adds 167 annual burden hours.

Table 1. Estimated Participant Burden SY 23

Activity	Number of Respondents	Number of Responses	Minutes per respondent	Total burden hours
Cohort 1/SY 23 teacher and other staff interviews	9	9	45	9
Cohort 1/SY 23 DILE school leadership team interviews	6	12	45	12
Cohort 2/SY 24 teacher and other staff interviews	16	16	45	16
Cohort 2/SY 24 DILE school leadership team interviews	12	36	45	36
Study Total	43	73	45	73
Cohort 1 and 2 Average Annual Burden	22	37	45	37

Note: Numbers have been rounded to whole numbers.

Table 1. Estimated Participant Burden SY 25 (Cohort 3)

Activity	Number of Respondents	Number of Responses	Minutes per respondent	Total burden hours
SY 25 DILE school leadership team interviews	16	32	90	32
SY 25 teacher and other staff interviews	16	16	45	16
SY25 Staff survey	312	624	30	156
Cohort 3 annual burden	344	672	60-105	204

Note: Numbers have been rounded to whole numbers.

6. Estimate of Costs for Recruiting and Paying Respondents

REL Midwest will provide principals who are on the DILE leadership team with $50 gift cards for participating in each interview. REL Midwest will provide teachers and other staff $30 gift cards for participating in each interview.

REL Midwest will provide school staff who complete the survey with $50 gift cards for their participation.

7. Cost to federal government

The total cost to federal government for conducting the interviews in cohorts 1 and 2 will be $71,072. This cost includes scheduling interviews, conducting interviews, and analysis and reporting.

The total cost to the federal government for conducting the interviews and administering the surveys in cohort 3 will be $287,014. This cost includes scheduling interviews, conducting interviews, programming the survey, sending communications about the survey, analysis and reporting.

8. Assurance of Confidentiality

Information collected for this study comes under the confidentiality and data protection requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Respondents will be informed that their participation is voluntary and that all information they provide will not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). At the start of the interview, this information will be conveyed verbally to respondents by the interviewer.

The REL Midwest research team will follow confidentiality and data protection protocols in place at the American Institutes for Research (AIR). They will submit and maintain a data governance plan. All data will be housed on a secure drive, and access to this drive will be restricted to staff assigned to the project. Any data obtained for this study will be used only for descriptive analyses.

9. Justification for Sensitive Questions

The interview protocols do not include questions that might be considered sensitive.

10. Project Schedule

Scheduling for interviews will begin in January 2023 and data collection activities will continue through May 2025.