Change Memo

U.S. Department of Education

Agency Information Collection: Efficacy and Implementation Evaluation of the Secondary Writing Toolkit

AGENCY: Department of Education (ED), Institute of Education Sciences (IES).

OMB Control Number: 1850-0985

­­

SUMMARY:

In accordance with the Paperwork Reduction Act of 1995 (44 U.S.C. chapter 3501 et seq.), ED is submitting a simple change request to the existing and approved information collection as outlined in this memo.

To date, we have been unable to recruit schools for the approved randomized controlled trial (RCT) study of a middle school writing toolkit to support evidence-based writing instructional practices (see Appendix A for more detail on recruitment efforts). At this point in the school year, we no longer have enough time to complete the intervention before students take their state standardized assessment (our student outcome measure) in spring 2025. We therefore no longer have a student outcome measure aligned to our program logic model and only implementation of the toolkit, not impact, can be measured before the end of our Regional Educational Laboratory (REL) Pacific contract cycle. We note that, within the time allowed by our project contract, this study must be conducted within the 2024–25 school year.

Our study recruitment team indicates that the difficulty in recruiting was largely due to:

1. Lack of interest due to the RCT design.

2. Small number of eligible and interested schools in Hawai’i, our original study location.

3. Perception that the incentives we offered were too small, both relative to the lift asked of schools and teachers, and relative to other studies in which schools participated.

We have proposed four changes to our study to adjust to these circumstances that have been reviewed and approved by the REL COR and Branch Chief:

1. Change the study to an implementation study, while continuing to use the existing, OMB-approved research instruments. We will only collect these data from implementing schools. There will be no comparison group from which data will be collected. These changes will be made as a result:

   1. Eliminating the student outcome analysis. The proposed student outcome was to be students’ writing sub-score on state ELA assessments (typically administered in March or April of each year). Given that no schools have consented to participate, implementation would not begin prior to January, meaning teachers would complete the 4-5 month curriculum of the toolkit implementation in late April, after the administration of the state ELA assessments. We therefore propose eliminating the collection and analysis of student outcome data, as we will not be able to capture the full impact of the intervention (if such an effect exists). The elimination of the student data will also reduce requests to school and district offices, as they will not need to fulfill a data request from the study team.

   2. Eliminate baseline teacher instructional log data collection. The new recruitment timeline and 4-5 months required to implement the toolkit does not allow time for baseline teacher instructional log data collection before the end of school calendars in June. Further, we do not feel that pre-test data will be informative in this particular study, as ELA teachers typically teach argument writing (the focus of the toolkit) in spring. There is no baseline data to collect since any data collected will be capturing teacher practice when teaching but not necessarily teaching the topic of the instrument.

   3. Restricting data collection to implementing schools. We will collect data on the implementation and uptake of the toolkit from educators at implementing schools. This will allow us to recruit schools into a known intervention condition, which responds to schools’ hesitation to participate in an RCT study.^[1]

2. Expand our recruitment into California, to allow for a larger number of eligible schools.

   1. We originally planned for this study to take place in Hawai’i as the only state in the REL Pacific jurisdiction which could support the sample size required and conducted extensive recruitment efforts within the state that were not successful (see Appendix A). There were several schools who were interested in the intervention but expressed substantial concern with regard to both the RCT design and the paucity of incentives for the school and teachers. We therefore propose expanding our recruitment into California as well, as (1) the state has over 7,000 middle schools, (2) California also uses the Smarter Balanced ELA assessments, which will ensure that the toolkit, as developed for Hawai’i’s schools, will align to the local context and (3) the intervention’s developers are located in California, providing them with better contextual knowledge and relationships to facilitate recruitment and implementation.

3. Increase financial incentives for both schools and research participants to better align to (1) the study context, (2) the extent to which the intervention would be seen as an adequate incentive for implementing schools, (3) the study timeline, and (4) higher incentives for focus groups to cover the costs of attendance.

   1. In designing the incentives, we followed guidelines from the U.S. Department of Education’s National Center for Education Evaluation and Regional Assistance (NCEE, May 2022). These guidelines indicate that in a comparison group study, while comparison schools may receive incentives to participate in the study, as a default, the intervention schools’ only incentive to participate should be the intervention itself, not additional money. The guidelines indicate that any proposal for additional incentives should be discussed with the project COR.

   2. In recruitment conversations, we consistently heard that the incentive payments were too low for both schools and educators, and were given examples from other similar studies recently completed by these schools where they and their educators received higher incentives. We therefore propose new incentives, which better consider the current:

      1. Context: We are recruiting in Hawai‘i and California, two of the areas with the highest cost of living in the country. However, our originally proposed incentives used an hourly rate lower than a teacher’s average wage in California. Further, our proposed incentives were set using guidance from over two years ago; given substantial inflation in the past few years, an adjustment for inflation is warranted for all award amounts.

      2. Extent to which the intervention would be seen as an adequate incentive: This intervention provides lighter-touch supports than are typically seen in many rigorous studies. Specifically, the intervention itself is a toolkit designed to support teachers in self-study to adopt evidence-based practices in writing. The innovation behind testing this approach is to better understand what teachers can learn from well-designed instructional materials and minimal external support. As a point of contrast, the studies from which the evidence-based practices were derived—using similar RCT designs—typically included regular, ongoing coaching directly to the teachers (see, e.g., Gallagher, Arshan, & Woodworth, 2017, in which teachers were provided with 40 hours of professional learning each year over two years). That is: the toolkit is, by design, (1) free to schools, (2) affordable to implement, and (3) dependent on teacher self- and collaborative study. The receipt of a high-touch intervention that would typically be unaffordable to the school (e.g., intensive, on-going coaching) would likely be an adequate incentive for most schools. Based on our conversations with schools, we understand that school leaders do not perceive the intervention being tested in this study to meet this bar; financial incentives for implementing schools are therefore warranted.

      3. Study timeline: As we are recruiting in the middle of the school year, the study timeline is substantially compressed: we are asking principals to make a mid-year change in their plans for their ELA teachers’ professional learning on short notice. Further, we cannot be assured that all recruited schools will have a full four months in which to implement the intervention, which may require principals to cover teachers’ time with substitutes to ensure teachers have adequate time to complete the intervention work. We therefore believe that providing some additional resources to schools to ease concerns about unanticipated needs (e.g., covering teacher time with substitutes) is warranted.

      4. Focus group incentives to cover potential costs of attendance. Abt commonly uses higher incentive awards for focus groups, as they require participants physical attendance at the most mutually agreeable time. Some participants may find the time inconvenient (if, for example, they need to cover the cost of childcare or transportation costs). Higher incentives for focus groups therefore ensure higher quality data that is more representative of the population being studied.

In sum, our overall changes to incentive payments are to:

• Use Bureau of Labor Statistics (BLS) averages for teacher and administrator salaries in California as the basis for an hourly rate.

• As applicable, adjust incentive amounts for inflation using the BLS Consumer Price Index or adjust non-wage amounts using the Commerce's Bureau of Economic Analysis Regional Price Parity estimates.

• Provide financial incentives to implementing schools adequate to support implementation.

• Follow our internal IRB guidance on incentives for participation in focus groups.

• Round values for each task to round numbers, which are more easily communicated and understood by research participants.

We estimate an average hourly wage of $50 for teachers and $75 for administrators. Additionally, we calculate school incentives as the replacement cost of three teachers’ and one peer facilitator’s time for each potential hour spent implementing the intervention; we use three teachers as we anticipate the average school we recruit will have 2–4 teachers. We provide the approved and proposed amounts in Table A. See Appendix B for more detail on how the proposed amounts were calculated.

Table 1: Approved and Proposed Incentives

Respondent Type	Activity	Frequency	Time per Respondent	Approved Total Incentive/ Respondent	Proposed Total Incentive/ Respondent
Individual Incentives
Teacher	Baseline Instructional log	5 times in fall	75 minutes (15 mins/time)	$50	N/A
Teacher	Implementation instructional log	5 times in spring	75 minutes (15 mins/time)	$50	$100
Teacher (excludes peer facilitators)	Online tracker of intervention activities	8 times during implementation	80 minutes (10 mins/time)	$50	$100
Peer facilitator	Online tracker of intervention activities	8 times during implementation	200 minutes (25 mins/time)	$100	$300
Teacher	Focus group	once in spring	60 minutes	$30	$150
Peer facilitator	Focus group	once in spring	60 minutes	$30	$150
Administrator	Survey	once in spring	10 minutes	$50	$75
School/LEA Incentives
Treatment schools	Participation in evaluation	NA	NA	$0	$5,500
Control schools	Participation in evaluation	NA	NA	$2,500	N/A

4. Update the response choices on a survey questionnaire

   1. The study team has proposed to make edits to a survey instrument for teachers and peer leaders participating in the Efficacy and Implementation Evaluation of the Secondary Writing Toolkit. These surveys, the “professional learning trackers,” are designed to record the content and participation in each of 8 professional learning sessions. The study has been approved for 8 responses per person: one following each of 8 professional learning sessions. The instrument is customized to respondent and session, to ensure we only ask information relevant to the professional learning experience being asked about. The instrument will be administered to 86 teachers and 18 peer facilitators. We estimated .17 hours per response for teachers and .20 hours per response for peer facilitators, for a total burden of 120 hours, and a total response cost of $3,584.

   2. These professional learning trackers, as approved, have approximately 5–9 questions in each administration (depending on respondent and the specifics of the professional learning session the instrument follows). Most questions have a yes/no or 5-scale Likert response. A small number have drop down lists of responses from which the respondent chooses an answer. These drop-down lists did not explicitly list the response options. Instead, they indicated that the respondent would choose from drop-down list of Module-specific options. This description was provided in recognition that the modules were still undergoing peer-review and any list of options would need to be updated once they were finalized.

      1. The study team is proposing following the response options for Question 2e, which will follow four PLC sessions:

         1. Classroom Video Lesson

         2. Retrofitting an Existing Lesson or Extending Your Practice

         3. Create Your Own Lesson, including Going Deeper

         4. Create Your Own Lesson, not including Going Deeper

In total, the study team estimates 393 respondents and 668 burden hours in the study. This does not represent a request for a change in annual burden hours under OMB Control Number 1850-0985.

Appendix A: Recruitment Efforts and Challenges

We designed this study to take place in Hawaiʻi, as it is the only entity in the REL Pacific region large enough to provide enough schools for implementation and evaluation of the toolkit. This left us with a limited number of schools available to recruit: Only 110 Hawaiʻi Department of Education (HIDOE) schools met study criteria, meaning that we needed to recruit 36 percent of eligible schools to meet our recruitment goal. The sample was further limited due to the wildfires in Maui. We removed these schools from eligibility, both because we felt the ask was too much for the schools and because they would not have represented ideal implementation conditions, as required for an initial efficacy study.

The initial recruitment plan employed an efficient and resource-conscious strategy. This included a ranking of schools by number of English language arts (ELA) teachers and island. Because approximately 70 percent of the eligible schools are located on the island of Oahu (Less than 20 percent of eligible schools are located on the island of Hawai‘i, and less than 10 percent combined are located on the state’s other islands), it was anticipated that more schools would be recruited from Oahu. Schools with two or more teachers of grades 6–8 were targeted. The original recruitment timeline called for initial recruitment efforts to start in September 2023, with initial contact with the 15 complex area superintendents of the 110 schools via email and a follow-up phone call shortly thereafter (within two days of the email). The initial email would provide a brief introduction to the study that emphasized the purpose and relevance of the toolkit for Hawai‘i educators and their students and the alignment of the toolkit with the state standards. A timeline for the study, study expectations and incentives for participation, a link to a website with additional information, a Frequently Asked Questions sheet (as an attachment), and assurances of support and research approval from HIDOE were included in the initial invitations. The follow-up phone calls were intended to focus on answering questions about the study, gaining approval to reach out to school leaders, confirming contact names and information for each targeted school, and ascertaining how best to reach out to school-level leaders to provide information about the study opportunity.

The above sequence of recruitment steps and efforts was slightly delayed due to Office of Management and Budget (OMB) authorization, but the REL director sent initial invitations to all 15 Hawaiʻi complex area superintendents (spanning all HIDOE school districts) and the executive director for the State Charter School Commission in December 2023.

In early January 2024, recruiters followed up with phone calls to all complex area superintendents and the charter schools’ executive director. REL Pacific sent a second email invitation and conducted follow up phone calls in February 2024. As an additional option, invitations for in-person meetings were extended to all complex areas.

Collectively, these efforts yielded responses from three superintendents who gave authorization for the REL Pacific team to proceed by extending invitations to school principals. Invitations were sent to a total of 36 school principals (36 eligible schools out of 39 possible schools), while phone call efforts to reach non-responsive complex area superintendents continued into March 2024, with an in-person meeting request from Director Christina Tydeman. None of these invitations were answered.

REL Pacific also requested an in-person meeting with HIDOE’s assistant superintendent from the Office of Curriculum and Instructional Support to explore other recruitment strategies from within HIDOE. At the same time, REL Pacific sent follow-up emails and made phone calls to the principals in the three complex areas for which REL Pacific had authorization.

On March 25, 2024, REL Pacific recruitment leads met with the team from the Office of Curriculum and Instructional Support to discuss the toolkit study and explore ways to share with ELA teachers and directors. This resulted in announcements in the weekly HIDOE superintendent and principal newsletter.

In April 2024, efforts to recruit school principals continued where authorized. Other efforts to gain access from additional superintendents to begin outreach in other eligible public schools continued as well.

Expanded recruitment efforts to private schools in Hawaiʻi began in May 2024. Also, expanded recruitment was considered for other possible Pacific jurisdictions. REL Pacific began follow-up phone calls to private schools in mid-May, but none of these efforts yielded interested schools.

In June 2024, REL Pacific secured subcontracting support for recruitment and expanded recruitment to Guam. REL Pacific sent invitation emails and toolkit information to Guam’s superintendent and two deputy superintendents. REL Pacific also leveraged internal networks to gain access to three additional complex areas, and invitations were sent to nine additional school principals. At the same time, REL Pacific continued to explore other possible recruitment angles in Hawaiʻi with the deputy superintendent and the assistant superintendent overseeing HIDOE’s academic services. Following this meeting, and at the request of the deputy superintendent, the REL Pacific team presented toolkit recruitment materials to teachers and principals at one of their HIDOE leadership meetings.

Given the concerns about recruiting in the Pacific, REL Pacific’s recruitment efforts pivoted to include California public middle schools in August 2024, and REL Pacific sent email invitations and conducted follow-up phone calls in September 2024. Given the relatively late start of this recruitment, we have targeted small districts, which has allowed us to focus our efforts on schools where we might be able to move quickly to sign memoranda of understanding (MOUs) and schedule professional learning due to lighter administrative burden.

We have found recruitment difficult and, to date, have not been able to recruit any schools willing to sign up for an RCT.

Appendix B: Calculation of incentives

Here we provide more detail about how we calculated our current incentives.

Hourly Rates

The hourly rates for teachers and educators form the basis of all incentives other than focus group participation (which is based off guidance from Abt’s IRB). NCEE guidance for project incentives provides a suggested rate of $30/hour for teacher and $47/hour for principals. They also recommend consulting the Bureau of Labor Statistics (BLS) Occupational Outlook handbook to confirm current salaries when determining incentive amounts. We consult both the BLS Occupational Outlook for educator wages in California and, given the high rates of inflation over the past few years, use the BLS Consumer Price Index (CPI) to adjust for inflation. Below we describe our process for deriving our hourly rates of $50/hour for teachers and $75/hour for administrators.

Hourly Rates for Teachers

According to the BLS, the average Middle School (non-career and technical education, non-special education) teacher in California made $95,860 in May 2023 (April 2024). We divided this by 2080 hours in a work year, following guidance from U.S. Department of Education’s National Center for Education Evaluation and Regional Assistance (NCEE, May 2022), providing an average hourly wage of $46.09. Using the BLS CPI, we then adjusted this number for inflation between May 2023 and September 2024 (the most recent month for which data are available), resulting in an hourly wage of $47.78 (October 2024). We then rounded this wage to $50 for ease of communication with research participants.

Hourly Rates for Administrators

According to the BLS, the average education administrator (Kindergarten through Secondary) in California made $141,370 in May 2023 (April 2024). We divided this by 2080 hours in a work year, following guidance from U.S. Department of Education’s National Center for Education Evaluation and Regional Assistance (NCEE, May 2022), providing an average hourly wage of $69.97. Using the BLS CPI, we then adjusted this number for inflation between May 2023 and September 2024 (the most recent month for which data are available), resulting in an hourly wage of $72.54 (October 2024). We then rounded this wage to $75 for ease of communication with research participants.

Calculation of Incentives for Research Activities

We describe below how we calculated the incentives for each individual research activity.

• Instructional log. The instructional log is completed by teachers during their daily instruction. We therefore used the teacher hourly rate as a basis for computing the amount. The log is administered 5 times over the course of one week and we estimated teachers may spend 15 minutes for each administration, for a total of 75 minutes, which we rounded to 2 hours. This brings us to $100, which we plan to prorate according to the number of administrations to which teachers respond ($20 per administration).

• Teacher professional learning tracker. The teacher professional learning tracker is completed by teachers who are not serving as peer facilitators. We therefore used the teacher hourly rate as a basis for computing the amount. The professional learning tracker is administered 8 times—once after each professional learning community session—and we estimated teachers may spend 10 minutes for each administration. This brings us to a total of 80 minutes, which we rounded to 2 hours. This brings us to $100, which we plan to prorate according to the number of administrations to which teachers respond ($12.50 per administration).

• Peer facilitator professional learning tracker. The peer facilitator professional learning tracker is completed by educators serving as peer facilitators. Because the role will be held by both administrators and teachers, we used the administrator rate for this instrument, even if the peer facilitator in that school is employed by the schools as teachers. The professional learning tracker is administered 8 times—once after each professional learning community session—and we estimated educators may spend 25 minutes for each administration. This brings us to a total of 200 minutes, which we rounded to 3 hours. This brings us to $300, which we plan to prorate according to the number of administrations to which teachers respond ($37.50 per administration).

• Administrator survey. We used the administrator rate for the administrator survey, as the most typical respondent will be the school principal. We estimated 20 minutes for administrators to fill out this survey. However, NCEE guidelines for incentives recommend that, “The minimum incentive, no matter how much time the research activity requires, is one full hour of wages.” We therefore propose a $75 incentive.

• Focus group participation. As noted in the main text, Abt typically provides higher incentive payments for focus groups to cover the cost of physical attendance at a time convenient to most—but potentially not all—participants. For example, a teacher or district coach may work part time and need to cover childcare and come into the office to meet during a lunch period or at the end of the day. If we do not cover their total cost of attendance, we risk systematically missing participants who represent unusual cases, thereby skewing our data. Given concerns about cost of living in California and recent inflation, we err on the generous side with this incentive. Abt typically pays $1–$2/minute for participation in a focus group. Our focus group is estimated to take 60 minutes. Using the $2/minute rate puts us at an incentive of $120, which we rounded to $150 for ease of communication and assurance that it will cover the cost of attendance in areas with high costs of living.

Calculation of School Incentives

We have noted that (1) schools may be hesitant to cede control of their schedule in an RCT design and (2) this intervention represents a novel approach to implementation. The size of our redesigned school incentive was determined in response to these two concerns. Specifically, we calculated the school incentive as the labor cost of 3 teachers’ time for the full 20 hours of training and preparation expected of them during the implementation of the toolkit. We used 3 teachers because we anticipate most schools to have 2–4 teachers. We also provided labor costs for 1 peer facilitator, who have an additional 8 hours of time in meeting with the program developers, bringing them to 28 hours. For the peer facilitator, we use an administrative rate, assuming that district or school ELA coaches may fill this role. This brings us to $5,100, which we rounded to $5,500 for ease of communication with administrators.

Providing the schools with this level of incentive has several benefits:

• It reduces the uncertainty inherent in testing a novel approach to teacher training. Specifically, it gives school leaders the resources to either (1) hire substitutes if they find that teachers and peer leaders need additional time during the workday or (2) compensate teachers for additional training time outside of school hours, if necessary.

• Together with the lighter-touch supports provided in the intervention being tested, it replaces the incentive typically provided by the kind of high-touch, resource-intensive programs typically considered to be an adequate incentive for participation in a research evaluation. For example, the typical cost of instructional coaching for a single teacher is between $3,260–$5,220 in the 2009–2010 school year. Adjusted for inflation from September 2009 to September 2024 and cost of living in California, this range currently would be $5,354.31–$8,573.48.
  
  

References

Bureau of Economic Analysis (2023, December). Regional price parities by state and metro area. https://www.bea.gov/data/prices-inflation/regional-price-parities-state-and-metro-area

Bureau of Labor Statistics (2024, April). Employment of middle school teachers, except special and career/technical education by state, May 2023. https://www.bls.gov/oes/current/oes252022.htm#st

Bureau of Labor Statistics (2024, April). Employment of education administrators, kindergarten through secondary by state, May 2023. https://www.bls.gov/oes/2023/may/oes119032.htm#st

Bureau of Labor Statistics (2024, October). CPI inflation calculator (administrator wages). https://data.bls.gov/cgi-bin/cpicalc.pl?cost1=69.97&year1=202305&year2=202409

Bureau of Labor Statistics (2024, October). CPI inflation calculator (school incentives). https://data.bls.gov/cgi-bin/cpicalc.pl?cost1=2500&year1=202205&year2=202409

Bureau of Labor Statistics (2024, October). CPI inflation calculator (teacher wages). https://data.bls.gov/cgi-bin/cpicalc.pl?cost1=46.09&year1=202305&year2=202409

Gallagher, H. A., Arshan, N., & Woodworth, K. (2017). Impact of the National Writing Project’s College-Ready Writers Program in high-need rural districts. Journal of Research on Educational Effectiveness, 10(3), 570–595. https://doi.org/10.1080/19345747.2017.1300361

Knight, D. S. (2012). Assessing the cost of instructional coaching. Journal of Education Finance, 38(1), 52–80. http://www.jstor.org/stable/23259121

National Center for Education Evaluation and Regional Assistance (2022, May). Guidelines for incentives for REL research studies: May 2022.

^[1] We note that the teacher outcomes were not designed to meet the U.S. Department of Education’s What Works Clearinghouse standards.