Justification Memo

Regional Educational Laboratory, Midwest

Usability and Feasibility Study for the Strategies to Improve Reading (STIR) Approach

OMB# 1850-0952 v.1

Volume I

Supporting Statement

Submitted by:

National Center for Education Evaluation (NCEE)

Institute of Education Sciences (IES)

U.S. Department of Education

Washington, DC

August 2024

Attachments

Attachment I. Teacher interview protocol

Attachment II. School leader interview protocol

Attachment III. Instructional coach interview protocol

Table of Contents

1) Submittal-Related Information 3

2) Background 3

3) Design and Context 4

4) Recruitment and Data Collection 4

5) Estimated Respondent Burden 5

6) Estimate of Costs for Recruiting and Paying Respondents 6

7) Cost to federal government 6

8) Assurance of Confidentiality 6

9) Justification for Sensitive Questions 6

10) Project Schedule 6

1. Submittal-Related Information

The following material is being submitted under the National Center for Education Evaluation (NCEE) generic clearance agreement (OMB# 1850-0952), which provides NCEE the capability to collect preliminary or exploratory information to aid in study design by: (1) fielding brief, quick turnaround surveys, extracting test case administrative data, administering interviews, or conducting “mini-experiments” in advance of a study for the purpose of determining feasibility, a random assignment or comparison group strategy, or a data collection approach most suitable for a potential or planned evaluation; and (2) developing, testing, and improving its survey and assessment instruments, methodologies, and study dissemination strategies.

This request is to conduct interviews with teachers, school leaders, and district leaders participating in the Strategies to Improve Reading (STIR) professional learning opportunities field test. These interviews will be used to understand participants’ perceptions of the usability, feasibility, and accessibility of the STIR professional learning opportunities in order to inform refinements to the STIR components to be examined in a future optimization and efficacy study. Interviews began in February 2023.

The Generic information collection (IC) request for the REL Midwest Usability and Feasibility Study for the Strategies to Improve Reading (STIR) Approach was originally approved on December 20, 2022. In accordance with the Paperwork Reduction Act of 1995 (44 U.S.C. chapter 3501 et seq.), ED is submitting a simple change request to the existing information collection for the following changes outlined in this memo. The Department has assessed the upcoming changes and believe based on previously executed change requests to the information collection that these changes fall under that guise. The request increases the number of staff interviews and adds an instructional coach interview protocol, resulting in 15 additional burden hours.

 

1. Increase number of staff interviews 

The study team has proposed to conduct interviews with teachers, principals, and instructional coaches participating in STIR. The original request was for two interviews with 10 teachers, 4 school leaders, and 2 district members for two cohorts, for a total of 64 interviews (32 interviews for cohorts 1 and 2). The STIR approach was adapted following the second cohort and the revised STIR approach is being piloted with a third cohort of schools. The study team is proposing to conduct an additional set of interviews in cohort 3 to develop evidence of the usability, feasibility, and acceptability of the updated STIR approach. Gathering this information is a critical step in the development process to prepare for a future efficacy study. The study team will conduct two interviews with 6 teachers, 2 principals, and 2 instructional coaches, for a total of 20 interviews. All interviews will occur twice during the school year with each participant, once during the fall semester (2024) and once during the spring semester (2025). The additional cohort 3 interviews will allow the study team to learn about the usability, acceptability, and feasibility of STIR over the entire school year following changes to the STIR approach from cohort 2 to cohort 3.

2. Add an instructional coach interview protocol

 

The study team is requesting to administer a new instructional coach interview protocol for cohort 3. In cohort 3, the STIR development team adjusted its training approach to focus more on training district instructional coaches to lead the STIR training. To better understand this change, the study team developed a separate protocol based on the existing school leader interview protocol to better understand instructional coaches experiences separate from principals experiences. The study team estimates that 2 instructional coaches will participate in cohort 3.

2. Background

Recent National Assessment of Educational Progress results demonstrate a need to improve students’ reading outcomes in Michigan. To address low reading proficiency and inequities in reading outcomes, the Michigan Department of Education has prioritized improving literacy instruction, with a particular focus on phonics and phonological awareness in early literacy. The Strategies to Improve Reading (STIR) partnership aligns with the Michigan Department of Educations’ strategic priorities by developing and offering a series of professional learning opportunities developed by Regional Education Laboratory (REL) Midwest to address low reading proficiency and inequality in reading outcomes. Through STIR, educators and school leaders will receive training and coaching on four literacy-specific content areas: (a) features of effective, evidence-based instructional practices for teaching phonics and phonological awareness, including using instructional routines that are explicit, data-driven, systematic, and culturally responsive; (b) decision-driven literacy data collection, analysis and action to inform development of literacy instruction and interventions; (c) a diagnostic and implementation monitoring tool that school leaders can use to provide teachers with formative feedback on their use of effective instructional practices in phonics and phonological awareness, and (d) effective practices for school leaders to implement at the school level around literacy data use.

REL Midwest will support the development and field testing of the STIR professional learning opportunities by conducting a usability/feasibility study to understand participants’ perceptions of different components of STIR (usability), the contextual factors that support or hinder implementation of STIR (feasibility), and perceptions of the value of STIR (acceptability). The usability/feasibility study will include three cohorts of educators (kindergarten, grade 1, and grade 2 teachers), school leaders (school administrators and instructional coaches) and district leaders from elementary schools in three Michigan school districts. Each cohort will participate in a Summer Institute, monthly professional learning communities, and quarterly site visits. REL Midwest collected data from the first cohort during the 2022/23 school year and use the cohort 1 findings to inform any adaptations to STIR before implementation and data collection with the second cohort. REL Midwest collected data from the second cohort during the 2023/24 school year and use the cohort 2 findings to inform adaptations to STIR before implementation and data collection with the third cohort. REL Midwest will collect data from the third cohort during the 2024/25 school year and use findings to inform adaptions to STIR.

This request is to conduct interviews with educators, school leaders, and district leaders participating in STIR to understand what makes (or does not make) STIR a usable, feasible, and acceptable professional learning opportunity that can improve literacy instruction, literacy practice, and generate equitable literacy outcomes in kindergarten, grade 1, and grade 2. Through these interviews, educators, school leaders, and district leaders will reflect on the extent to which STIR content is easy to understand and implement;, will offer suggestions on how the STIR development team can improve the quality and accessibility of STIR content and structure; will highlight contextual factors that support or hinder successful implementation of STIR; will reflect on how the dosage, duration, and density of STIR supported or hindered their participation; and will reflect on how worthwhile and helpful STIR is for developing instructional practice in their classroom, school, or district. The educator, school leader, and district leader interview protocols are presented in Attachments I, II, and III, respectively. The interview collection during the field test will inform the design of the STIR components and how it will be studied during a future optimization and efficacy study.

At two time points per cohort, the REL Midwest research team will conduct interviews with up to four kindergarten teachers, four grade 1 teachers, two grade 2 teachers, four school leaders, and two district leaders, across three Michigan school districts. To obtain rich interview data, the research team will conduct two interviewers with each of these 16 practitioners in each cohort, for a total of 32 interviews per cohort. The interviews for the first cohort were conducted in February 2022 and May 2022. The interviews for the second cohort were conducted in February 2023 and May 2023. The interviews for the third cohort will be conducted in November 2024 and March 2025. Interviews will be conducted via videoconference and will take no more than 30 minutes to complete.

3. Design and Context

The interviews for which this clearance is requested will be used to understand educator, school leader, and district leader perceptions of STIR’s usability, feasibility, and accessibility. REL Midwest will use the findings generated from these interviews to inform and develop hypotheses about optimal delivery of STIR for different contexts. Hypotheses will be related to combinations of modalities, dosages, and other STIR components that will be efficacious under different school and district conditions (e.g., efficacious under resource constraints).

Specifically, the interviews will focus on:

• Determining the extent to which the STIR content covered in the three professional learning opportunities is easy to understand, easy to implement, and facilitates culturally responsive instruction,

• Collecting suggestions for improving the quality of STIR content, the accessibility of STIR content, and the modes of delivery for STIR content (i.e., Summer Institute, professional learning communities, and site visits),

• Identifying district and school contextual factors that support or hinder successful implementation of STIR,

• Determining how the dosage, duration, and density of the professional learning opportunities affect participants’ ability to participate and engage fully in STIR, and

• Determining the extent to which educators, school leaders, and district leaders perceive STIR as worthwhile and helpful for improving instructional practice and student outcomes.

The interviews will be a semi-structured discussion with educators and school leaders from four participating schools and district leaders from both school districts. REL Midwest will interview educators and school leaders from schools that are struggling with implementation of STIR and schools that are successfully implementing STIR. The interviews are intended to maximize the quality of data collected by asking probing questions to better understand what makes (or could make) STIR a usable, feasible, and accessible professional learning opportunity to improve early literacy instruction and student outcomes.

The interview protocols for each group of participants (i.e., educators, school leaders, district leaders) will have sections covering the following topics:

1. The perceived purpose of STIR and general impressions of STIR.

2. Experiences participating in each STIR professional learning opportunity.

3. Experiences implementing the strategies learned in STIR in their classroom or school.

4. Challenges and successes in implementing STIR in their classroom, school, or district.

5. Community, district, and school contextual factors that are necessary to support successful implementation of STIR.

4. Recruitment and Data Collection

REL Midwest will not recruit STIR participants but will be responsible for helping to select STIR participants to interview, obtaining their agreement to be interviewed, conducting the interviews, compiling interview responses, and summarizing findings. Interviewers will ask questions in an open-ended manner using the protocols in Attachments I, II, and III while allowing for new questions or ideas to be brought up during the interview as a result of the interviewee’s responses. Two members of the research team will review the interview transcripts individually and code them separately. Each coder will draw on deductive approaches to help organize data into pre-specified groups or themes (e.g., ease of use, ease of implementation, and cultural responsiveness of STIR) and inductive approaches to make meaning from the data and allow for the possibility that groups or themes other than those identified in advance might emerge. To ensure intercoder reliability, a 20 percent sample of interview transcripts will be double coded. If 95 percent agreement on key groups or themes is not met, the coders will reconcile their codes in a meeting with each other and adjudicate to a single response.

REL Midwest will use existing preliminary data from a Stakeholder Feedback Survey to identify schools where school leaders and teachers have struggled with STIR implementation and schools where school leaders and teachers have been successful with STIR implementation. In cohorts 1 and 2, the research team will review this information with district partners from two districts and work together to identify one school in each participating district that has struggled with STIR implementation and one school in each district that has been successful with STIR implementation. The research team will then conduct interviews with 2 district leaders, 4 school leaders and 10 teachers across these four schools. In addition, REL Midwest will work with district partners to develop a selection strategy for district leaders. In cohort 3, the research team will identify one school in each participating district to participate in an interview. The research team will then conduct interviews with 1 instructional coach, 1 school leader, and 6 teachers across these two schools.

The interviews will be conducted via video conference and will be audio and/or video recorded so that researchers can reference recordings to supplement written notes taken during the interview.

5. Estimated Respondent Burden

Tables 1a and 1b details the annual respondent burden for the interviews, which corresponds to the respondent burden for a single cohort. Cohorts 1 and 2 are shown separately from cohort 3. The estimates assume that it will take 15 minutes to schedule interviews, that interviews will take 45 minutes, and that each respondent will be interviewed twice. As mentioned earlier, this is a change to an approved Gen IC; therefore, to avoid overcounting, we are requesting the difference in hours with this change which results in 15 hours. Responses will be recorded in the ROCIS system as 1 since the number of responses have decreased from the previous Gen IC.

Table 1a. Estimated Annual Participant Burden (cohorts 1 and 2)

Activity	Number of Respondents	Number of Responses	Minutes per respondent	Total burden hours
Teacher interviews	10	20	45	15
School leader interviews	4	8	45	6
District leader interviews	2	4	45	3
Study Total	16	32	1,440	24

Note: Numbers have been rounded to whole numbers.

Table 1b. Estimated Annual Participant Burden (cohort 3)

Activity	Number of Respondents	Number of Responses	Minutes per respondent	Total burden hours
Instructional coach interviews	2	4	45	3
Teacher interviews	6	12	45	9
School leader interviews	2	4	45	3
Study Total	10	20	900	15

Note: Numbers have been rounded to whole numbers.

Table 2 details the respondent burden for the interviews across both cohorts of the study. As in Table 1, the estimates assume that it will take 15 minutes to schedule interviews, that interviews will take 45 minutes, and that each respondent will be interviewed twice.

In total, the study team estimates 42 respondents, 84 responses, and 63 burden hours across the three cohorts. This represents a request for an increase from 48 burden hours to 63 burden hours (although a reduction in the annual burden hours from 24 burden hours for Cohorts 1 and 2 to 15 burden hours for Cohort 3 under OMB Control Number 1850-0952 v.1. There is a net increase of 15 burden hours.

Table 2. Estimated Total Participant Burden, Across Three Cohorts

Activity	Number of Respondents	Number of Responses	Minutes per respondent	Total burden hours
Teacher interviews	26	52	45	39
School leader interviews	10	20	45	15
District leader interviews	4	8	45	6
Instructional coach interviews	2	4	45	3
Study Total	42	84	3,780	63

6. Estimate of Costs for Recruiting and Paying Respondents

Respondents will receive a $50 gift card for participation in the interviews.

7. Cost to federal government

The total cost to federal government for conducting the interviews will be $69,450. This cost includes scheduling interviews, conducting interviews, and analysis and reporting.

8. Assurance of Confidentiality

Information collected for this study comes under the confidentiality and data protection requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Respondents will be informed that their participation is voluntary and that all information they provide will not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). At the start of the interview, this information will be conveyed verbally to respondents by the interviewer.

The REL Midwest research team will follow confidentiality and data protection protocols in place at the American Institutes for Research (AIR). They will submit and maintain a data governance plan. All data will be housed on a secure drive, and access to this drive will be restricted to staff assigned to the project. Any data obtained for this study will be used only for descriptive analyses.

9. Justification for Sensitive Questions

The interview protocols do not include questions that might be considered sensitive.

10. Project Schedule

Scheduling for interviews began in January 2023 and will continue through June 2025.