Justification Memo

Regional Educational Laboratory, Midwest

Usability and Feasibility Study for the Making Equitable Schools Audit (MESA) Approach

OMB# 1850-0952 v.7

Volume I

Supporting Statement

Submitted by:

National Center for Education Evaluation (NCEE)

Institute of Education Sciences (IES)

U.S. Department of Education

Washington, DC

August 2024

Attachments

Attachment I. MESA approach interview 1: Pre-equity audit leadership team interview protocol

Attachment 2. MESA approach interview 2: Equity audit interview protocol

Attachment 3. MESA approach interview 3: Post-equity audit leadership team interview protocol

Attachment 4. MESA approach interview 4: Action and Monitoring school leader interview protocol

Table of Contents

1) Submittal-Related Information 3

2) Background 3

3) Design and Context 4

4) Recruitment and Data Collection 4

5) Estimated Respondent Burden 5

6) Estimate of Costs for Recruiting and Paying Respondents 5

7) Cost to federal government 5

8) Assurance of Confidentiality 5

9) Justification for Sensitive Questions 6

10) Project Schedule 6

1. Submittal-Related Information

The following material is being submitted under the National Center for Education Evaluation (NCEE) generic clearance agreement (OMB# 1850-0952), which provides NCEE the capability to collect preliminary or exploratory information to aid in study design by: (1) fielding brief, quick turnaround surveys, extracting test case administrative data, administering interviews, or conducting “mini-experiments” in advance of a study for the purpose of determining feasibility, a random assignment or comparison group strategy, or a data collection approach most suitable for a potential or planned evaluation; and (2) developing, testing, and improving its survey and assessment instruments, methodologies, and study dissemination strategies.

This request is to conduct interviews with teachers, other school staff, and school leadership team members participating in the Making Equitable Schools Audit (MESA) approach field test. These interviews will be used to understand participants’ perceptions of the usability, feasibility, and accessibility of the MESA approach in order to inform refinements to the MESA components to be examined in a future optimization and efficacy study. Interviews began in March 2023.

The Generic information collection (IC) request for the REL Midwest Usability and Feasibility Study for the Making Equitable Schools Audit (MESA) Approach was originally approved on January30, 2023. In accordance with the Paperwork Reduction Act of 1995 (44 U.S.C. chapter 3501 et seq.), ED is submitting a simple change to the existing information collection for the following changes outlined in this memo. The request increases the number of staff interviews, resulting in 16 additional burden hours.

1. Increase number of staff interviews

The study team has proposed to conduct interviews with additional school leadership team members and equity audit participants participating in MESA. The original request was for one interview with 9 teachers and other staff members and two interviews with 6 members of the school leadership team in one school in cohort 1, for a total of 21 interviews in cohort 2. In addition, we requested one interview with up to 4 teachers and other staff members and three interviews with two leadership team members in each of 4 schools in cohort 2, for a total of 52 interviews in cohort 2.

The MESA approach was adapted following the second cohort and the revised MESA approach is being piloted with a third cohort of schools. The study team is proposing to conduct an additional set of interviews in cohort 3 to develop evidence of the usability, feasibility, and acceptability of the updated MESA approach. Gathering this information is a critical step in the development process to prepare for a future efficacy study. The study team will conduct interviews with the principal and one member of the leadership team per school prior to the equity audit for a total of 6 interviews. The study team will conduct up to 8 interviews per school after the equity audit sessions for a total of 24 interviews. Additionally, after the leadership coaching sessions have concluded, the study team will conduct interviews with up to 5 leadership coaching participants per school for a total of 15 interviews. Finally, the team will conduct three interviews with principals after the action and monitoring to learn about what plans schools put in place. The additional cohort 3 interviews will allow the study team to learn about usability, acceptability, and feasibility of MESA over an entire school year and in a larger sample of schools.

In total, the study team estimates 48 respondents and 36 burden hours in cohort 3. This represents a request for a change in annual burden hours from 20.38 across cohorts 1 and 2 to 36 in cohort 3 under OMB Control Number 1850-0952 v.7.

2. Background

Students learn best when they feel physically and emotionally safe, connected to school, academically challenged, and respected by adults (Chang et al., 2019). However, many students, especially students of color, report that they do not feel safe and supported at school. Demographic group differences in student perceptions of safety and support in school can be attributed to a range of factors, but one of the most significant factors relates to student experiences with school discipline (Lacoe, 2015). In Dayton Public Schools, school staff assigned 85 percent of out-of-school suspensions to Black students, a significant overrepresentation given that Black students make up only 65 percent of the student body. Previous research shows that educators use subjective criteria, such as disobedient or disruptive behavior and harassment or intimidation, in two-thirds of these suspensions of Black students (Sweigart, 2018). School staff often define policies using a cultural bias that negatively impacts students of color (Staats, 2014), demonstrating the subjectivity in school discipline. In addition, school staff are more likely to interpret and formally enforce discipline policies with students of color, even when they are exhibiting the same behaviors as White peers (Welsh & Little, 2018).

To address the demographic disparity in student perceptions of safety and support at school, the Regional Educational Laboratory (REL Midwest) will work with its partners in Akron Public Schools (APS) and the Ridgewood Community High school District to improve equity in school discipline practices by targeting the perceptions held by school staff and the ways they institutionalize these perceptions with school discipline policies. REL Midwest and its partners will design and develop the Making Equitable Schools Audit (MESA). The Making Equitable School Audit (MESA) approach strives to ensure that all members of a school community—including students, teachers, and staff—experience a safe and supportive school environment. MESA is a yearlong facilitated and collaborative approach designed to help schools use existing data and leverage communitywide relationships to move toward more equitable and belonging-filled experiences for all in a school community.

The MESA approach is designed to do the following:

• Use existing school data to identify equity gaps in discipline, student belonging, and high-quality learning opportunities.

• Bring together leadership and diverse representatives from across a school community to look at data, identify common themes, and prioritize findings for action.

• Engage in an intentional planning process to boost student sense of belonging, reduce the use of exclusionary discipline, and improve the overall school environment.

The MESA intervention occurs in three phases:

1. Onboarding for the school principal to ensure smooth operations and full buy-in to the project and pre-audit leadership coaching, where a diverse school staff leadership team participates in preparation and knowledge-building sessions. Pre-audit sessions introduce central concepts relating to systems change, equitable and culturally responsive educational practices, and the use of school data to identify equity gaps.

2. Equity audit sessions where representatives from the leadership team, family members, and students review school data, identify equity gaps, and identify a set of central root cause themes to inform school-level action planning for change.

3. Post-audit leadership coaching that reconvenes the school staff leadership team, this time to develop an implementation action plan. As part of the planning process, Regional Educational Laboratory (REL) Midwest supports the school team to identify research-based strategies that address the priority root cause topics identified during the equity audit. In addition, REL Midwest facilitates two action and monitoring “hand-off” sessions with the school administrators. In these sessions, REL Midwest staff work with the school leaders to determine specific monitoring metrics and a timeline to help ensure that the school can effectively implement its action plan and measure impacts.

REL Midwest is proposing to conduct a field test study of the MESA approach to provide the MESA development team with critical, timely information about participants’ experiences with the MESA components (usability), the contextual factors that support or hinder implementation (feasibility), and the perceived value of the MESA approach (acceptability). This information will guide changes to the approach with an aim to increase usability, feasibility, and acceptability before moving into an optimization and efficacy study in the future. The proposed study allows for testing the MESA approach across two cohorts: cohort 1 (February through August 2023) and cohort 2 (August 2023 through June 2024). Cohort 2 participated in full implementation of the MESA approach over a full school year. The sample differs across the two cohorts. Cohort 1 included school leaders, classroom teachers, and other staff in one APS high school. Cohort 2 included school leaders, classroom teachers, and other staff in the two other APS high schools. REL Midwest collected data from cohort 1 during the spring semester of the 2022/23 school year and use the cohort 1 findings to inform any adaptations to MESA before implementation and data collection with cohort 2. REL Midwest collected data from cohort 2 during the 2023/24 school year and use the cohort 2 findings to inform adaptations to MESA.

Cohorts 1 and 2 were conducted during the 2022/23 school year and the 20223/24 school year respectively. Based on the results from Cohort 1 and 2, we are seeing promising evidence that this approach has potential to influence educator and student outcomes, and we have learned strategies for how to refine the approach. Based on the data collected, we think we need to continue to pilot test refinements to the approach before moving to a larger efficacy study. It is important to examine the degree to which participants in the refined MESA approach find the approach useful, feasible, and acceptable. We will add a third cohort to test the usability, feasibility, and acceptability of the full MESA approach over an entire school year. REL Midwest will collect data from cohort 3 during the 2024/25 school year and use the cohort 3 findings to inform any additional adaptations to MESA.

This request is to conduct interviews with teachers, other school staff, and MESA leadership team members participating in MESA to understand what makes (or does not make) MESA a usable, feasible, and acceptable approach that can improve sense of belonging and engagement among students. Through these interviews, teachers, other school staff, and leadership team members will reflect on the extent to which the MESA components and content are easy to understand and implement; will offer suggestions on how the MESA development team can improve the quality and accessibility of MESA content and structure; will highlight contextual factors that support or hinder successful implementation of MESA; will reflect on how the dosage, duration, and density of MESA supported or hindered their participation; and will reflect on how worthwhile and helpful MESA is for supporting educators with processes, knowledge, and skills to use real-time local data to inform decisions about school policies and practices related to student discipline and student perceptions of belonging. The MESA Usability, Feasibility, and Acceptablitiy (UFA) study teacher and school leader interview protocol is presented in Attachment I. The interview collection during the field test will inform the design of the MESA components.

The research team will conduct up to four rounds of semistructured, 30-minute interviews for each cohort to gather information about the usability, feasibility, and acceptability of MESA (research questions 1–9).

Pre-equity audit leadership coaching interviews: The research team will conduct interviews with the principal and one member of the leadership coaching team prior to the equity audit per school for a total of 6 interviews. These interviews will occur during the fall semester.

Equity audit interviews: The research team will conduct interviews with up to eight participants per school after the equity audit sessions for a total of 24 interviews. These interviews will occur during the winter and early spring semester.

Post-leadership coaching interviews: The research team will conduct interviews with up to five leadership coaching members per school for a total of 15 interviews. These interviews will occur during the Spring semester.

Action and monitoring coaching interviews: The research team will conduct interviews with one principal per school for a total of 3 interviews. These interviews will occur 6 months after the conclusion of action and monitoring planning.

Interviews will be conducted via videoconference and will take no more than 30 minutes to complete.

3. Design and Context

The interviews for which this clearance is requested will be used to understand teacher, other staff, and MESA leadership team perceptions of MESA’s usability, feasibility, and accessibility. REL Midwest will use the findings generated from these interviews to inform and develop hypotheses about optimal delivery of MESA for different contexts. Hypotheses will be related to combinations of modalities, dosages, and other MESA components that will be efficacious under different school and district conditions (e.g., efficacious under resource constraints).

Specifically, the interviews focus on:

• Determining the extent to which the MESA content covered in the audit and coaching sessions are easy to understand, easy to implement, and facilitates culturally responsive instruction and use of data,

• Collecting suggestions for improving the quality of MESA content, the accessibility of MESA content, and the modes of delivery for MESA content (i.e. coaching for school leadership team, equity audit),

• Identifying district and school contextual factors that support or hinder successful implementation of MESA,

• Determining how the dosage and duration of the trainings and coaching affect participants’ ability to participate and engage fully in MESA, and

• Determining the extent to which teachers, other staff, and MESA leadership team members perceive MESA as worthwhile and helpful for improving instructional practice, use of data, and student outcomes.

The interviews will be a semi-structured discussion with teachers, staff, and leadership team members from one participating school in cohort 1 and two participating schools in cohort 2. The interviews are intended to maximize the quality of data collected by asking probing questions to better understand what makes (or could make) MESA a usable, feasible, and accessible approach to improve data use and student outcomes.

The interview protocol will have sections covering the following topics:

1. The perceived purpose of MESA and general impressions of MESA.

2. Experiences participating in each MESA component.

3. Experiences implementing the strategies learned in trainings in their classroom or school.

4. Challenges and successes in implementing MESA in their school.

5. Community, district, and school contextual factors that are necessary to support successful implementation of MESA.

4. Recruitment and Data Collection

REL Midwest will be responsible for selecting MESA participants to interview, obtaining their agreement to be interviewed, conducting the interviews, compiling interview responses, and summarizing findings. Interviewers will ask questions in an open-ended manner using the protocols in Attachment I while allowing for new questions or ideas to be brought up during the interview as a result of the interviewee’s responses. Two members of the research team will review the interview transcripts individually and code them separately. Each coder will draw on deductive approaches to help organize data into pre-specified groups or themes (e.g., ease of use, ease of implementation, and cultural responsiveness of MESA) and inductive approaches to make meaning from the data and allow for the possibility that groups or themes other than those identified in advance might emerge. To ensure intercoder reliability, a 20 percent sample of interview transcripts will be double coded. If 95 percent agreement on key groups or themes is not met, the coders will reconcile their codes in a meeting with each other and adjudicate to a single response.

REL Midwest will obtain the list of MESA leadership team members from the MESA development team. REL Midwest will obtain from the MESA development team attendance records for the MESA equity audit sessions. Using administrative teacher data provided by the district, the research team will purposively sample participating teachers and school leaders on the basis of years of teaching experience and classroom demographics (e.g., percentage of students eligible for the national school lunch program and English learner students), if applicable.

The interviews will be conducted via video conference and will be audio and/or video recorded so that researchers can reference recordings to supplement written notes taken during the interview.

5. Estimated Respondent Burden

Table 1 details the respondent burden for the interviews. The estimates assume that it will take 15 minutes to schedule interviews and that interviews will take 30 minutes. Each teacher or other staff participating in the equity audit sessions will be interviewed once.

Table 1. Estimated Participant Burden (cohorts 1 and 2)

Activity	Number of Respondents	Number of Responses	Minutes per respondent	Total burden hours
Cohort 1 teacher and other staff interviews (equity audit)	8	8	45	6
Cohort 2 teacher and other staff interviews (equity audit)	16	16	45	16
Cohort 1 MESA school leadership team interviews (post-audit coaching)	5	5	45	3.75
Cohort 2 MESA school leadership team interviews (post-audit coaching)	10	20	45	15
Study Total	39	49	180	40.75
Cohort 1 and 2 Average Annual Burden	19.5	24.5	90	20.38

Note: Numbers have been rounded to whole numbers.

The study team estimates 48 respondents and 36 burden hours in cohort 3 (see Table 2). This represents a request for a change in annual burden hours from 20.38 across cohorts 1 and 2 to 36 in cohort 3 under OMB Control Number 1850-0952 v.7. ED is only requesting 16 responses and 16 burden hours for the additional interviews be added to the existing/approved responses and hours for this generic IC to avoid double counting.

Table 2. Estimated Participant Burden (cohort 3)

Activity	Number of Respondents	Number of Responses	Minutes per respondent	Total burden hours
Cohort 3 Pre-equity audit leadership coaching interviews	6	6	45	4.5
Cohort 3 Equity audit interviews	24	24	45	18
Cohort 3 post-leadership coaching interviews	15	15	45	11.25
Cohort 3 MESA principal interviews (action and monitoring)	3	3	45	2.25
Study Total	48	48	180	36

Note: Numbers have been rounded to whole numbers

6. Estimate of Costs for Recruiting and Paying Respondents

REL Midwest will provide interview participants with $50 gift cards for participating in each interview.

7. Cost to federal government

The total cost to federal government for conducting the interviews will be $118,094. This cost includes scheduling interviews, conducting interviews, and analysis and reporting.

8. Assurance of Confidentiality

Information collected for this study comes under the confidentiality and data protection requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Respondents will be informed that their participation is voluntary and that all information they provide will not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). At the start of the interview, this information will be conveyed verbally to respondents by the interviewer.

The REL Midwest research team will follow confidentiality and data protection protocols in place at the American Institutes for Research (AIR). They will submit and maintain a data governance plan. All data will be housed on a secure drive, and access to this drive will be restricted to staff assigned to the project. Any data obtained for this study will be used only for descriptive analyses.

9. Justification for Sensitive Questions

The interview protocols do not include questions that might be considered sensitive.

10. Project Schedule

Scheduling for interviews began in October 2023, as soon as OMB approval is granted, and continue through June 2025.