Part A: Supporting Statement for Paperwork Reduction Act Submission
REL Pacific Efficacy and Implementation Evaluation of the Secondary Writing Toolkit
March 2023
Prepared for:
Elizabeth Nolan
U.S. Department of Education
550 12th Street, SW
Washington, DC 20024
Submitted by:
REL Pacific
745 Fort Street, Suite 1915
Honolulu, HI 96813
Table of Contents
Part A: Supporting Statement for Paperwork Reduction Act Submission i
A.1 Circumstances Making the Collection of Information Necessary 1
A.2 Purposes and Use of the Information Collection 2
A.3 Use of Information Technology and Burden Reduction 5
A.3.2 Teacher, Peer Facilitator, and School Administrator Surveys 5
A.3.3 Teacher and Peer Facilitator Focus Groups 5
A.4 Efforts to Identify Duplication 5
A.5 Efforts to Minimize Burden in Small Businesses 6
A.6 Consequences of Not Collecting the Information 6
A.7 Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6 6
A.8 Consultation Outside the Agency 6
A.9 Payments or Gifts to Respondents 7
A.10 Assurance of Confidentiality 7
A.11 Questions of a Sensitive Nature 9
A.12 Estimate of Response Burden 9
A.14 Estimates of Costs to the Federal Government 12
A.16 Plans for Analysis, Publication, and Schedule 12
A.16.2 Publication and Schedule 13
A.17 Approval to Not Display Expiration Date 14
The U.S. Department of Education’s (ED) Institute of Education Sciences (IES) requests clearance for data collection activities to support an evaluation of efficacy and implementation of the Secondary Writing Toolkit that is being developed by the Regional Educational Laboratory Pacific (REL PA). Specifically, this request covers (1) recruiting schools and teachers; (2) requesting administrative data from the Hawai‘i Department of Education (HIDOE); (3) collecting survey data in the form of an instructional log and a professional learning tracker from teachers and peer facilitators (who are also teachers within schools); (4) collecting survey data from administrators; and (5) conducting focus groups with teachers and peer facilitators.
Literacy, including writing, is closely tied to student success throughout K–12 education, which impacts high school graduation (National Institute for Literacy, 2008; NCES, 2020) and, ultimately, income beyond graduation (U.S. Bureau of Labor Statistics, 2019). Despite the importance of writing to life and learning, teachers report that the training they receive on teaching writing, both prior to entering the field and while teaching, is minimal or insufficient (Graham, 2019). To address this problem, the REL PA toolkit development team is developing a Secondary Writing Toolkit to support teachers in implementing evidence-based instructional strategies to improve writing among students in grades 6–8. The instructional strategies and resources in the Toolkit that will be selected and developed are based on the three recommendations in the Teaching Secondary Students to Write Effectively What Works Clearinghouse (WWC) Practice Guide: 1) explicitly teach appropriate writing strategies using a Model-Practice-Reflect instructional cycle; 2) integrate reading and writing to emphasize key writing features; and 3) use assessments of student writing to inform instruction and feedback. The toolkit is being developed in collaboration with district and school partners in Hawai‘i.
The Toolkit is designed to support interactions among teachers collaborating in Professional Learning Communities (PLCs) facilitated by one of the teachers in the school (peer facilitator). The Toolkit provides facilitators all of the materials they need to lead a PLC and includes all of the professional development content, in the form of four professional learning modules and other resources, to help teachers learn new instructional skills in the PLCs. It also includes instructional resources to support teachers’ use of evidence-based strategies in their classrooms. The specific components of the Toolkit include: 1) an initial diagnostic; 2) a Facilitator’s Guide and four professional learning modules; 3) ongoing monitoring tools; and 4) steps for institutionalizing supports (an Administrator Sustainability Guide and a chapter on sustainability in the Facilitator Guide). All four components of the Toolkit address the three Practice Guide recommendations: 1) explicitly teaching writing strategies; 2) integrating reading and writing; and 3) assessing writing to inform instruction and feedback. As required by ED, the Toolkit will be “manualized” such that it can be readily replicated elsewhere with fidelity if shown to be efficacious.
The REL PA toolkit study team is requesting clearance to conduct an independent evaluation that will assess the efficacy of the Toolkit. The evaluation will also assess how teachers and facilitators implement the toolkit to provide context for the efficacy findings and guidance to improve the toolkit and its future use. The evaluation will take place in 40 schools in Hawai‘i and focus on all students in grades 6–8.
The current authorization for the REL Program is under the Education Sciences Reform Act (ESRA) of 2002, Part D, Section 174, (20 U.S.C. 9564), administered by the Institute of Education Sciences’ (IES) National Center for Education Evaluation and Regional Assistance (NCEE). This authorization established a network of large-scale labs that focuses on three main activities in partnership with key education stakeholder groups: (1) conduct applied research, (2) provide technical assistance, and (3) disseminate scientifically valid research that improves learner outcomes from early childhood to adulthood. The information collected through this study is focused on the required activity of conducting applied research, which should be change-oriented, supporting consequential local or regional decisions about policies, programs, and practices designed to improve learner outcomes.
As part of the REL solicitation request (Solicitation #91990020R0032), IES required each applicant to develop at least one research-based toolkit to support educators’ use of evidence-based practices, and to conduct an independent efficacy and implementation evaluation of the toolkit.
Per the solicitation:
“IES is invested in developing practitioner-friendly toolkits to help educators use evidence-based practices in classrooms – from preschool through postsecondary settings. Some of the best evidence available is consolidated in the WWC Practice Guides, in which researchers and practitioners review the evidence from the most rigorous studies available, develop recommendations for practice, and create action steps for how to use the recommended practices. To help get this evidence into the hands of stakeholders, RELs shall partner with educators and postsecondary instructors (if relevant) to develop one toolkit based on an assigned WWC Practice Guide, which shall include all materials necessary for effective implementation.”
The solicitation also states that RELs must evaluate the efficacy and implementation of the professional development resources in the finished toolkit. According to the solicitation, “(t)he evaluation shall examine changes in teacher practice and may also include measures of teacher knowledge and/or teacher self-efficacy.”
In response to the REL solicitation request, this data collection measures the efficacy and implementation of the Secondary Writing Toolkit that is currently being developed by REL PA. The Toolkit is focused on providing grade 6–8 English Language Arts (ELA) teachers with professional development and resources for improving writing instruction. REL PA has contracted with WestEd to develop the Toolkit and Abt Associates (Abt) to conduct the independent evaluation of the Toolkit. The toolkit evaluation will produce a report for district and school leaders in search of strategies to improve writing among secondary students.
Results from the efficacy evaluation will help states, districts, and schools implement evidence-based practices, as emphasized by the Every Student Succeeds Act (ESSA), in two ways. First, the evaluation will provide evidence of the efficacy of the current toolkit on whether it improves teacher practices and student writing achievement. Second, regardless of the results of the efficacy evaluation, the collected information and associated analyses will provide insights into how the toolkit can be improved and adapted and serve as a resource to help states, districts and schools implement ESSA.
The purpose of this study is to conduct an evaluation of the efficacy and implementation of the Secondary Writing Toolkit developed by REL PA. IES has contracted with McREL International (prime contractor), which has contracted with Abt (the “study team”) to conduct an external evaluation of the Toolkit.
There are two components to the evaluation: an efficacy evaluation and an implementation evaluation. The specific research questions for each of these evaluations are presented in Exhibit A1 below. The efficacy evaluation will use the most rigorous design—a randomized controlled trial (RCT)—with schools as clusters. The study team will recruit and randomly assign 40 schools that employ at least two grade 6–8 ELA teachers to the treatment condition (Toolkit) or business as usual (comparison). Schools assigned to the treatment will receive training and support from REL PA for teachers, peer facilitators, and administrators to use the instructional strategies and resources in the Toolkit. Comparison schools will continue with any professional development offered by their schools and with their writing instruction as usual. The implementation evaluation will assess fidelity of implementation of the Toolkit and provide descriptive and contextual information that will be used to improve the Toolkit for future implementations.
Exhibit A1: Research Questions
Efficacy Evaluation Research Questions E1. What is the impact of having access to the Toolkit components and professional learning activities on grade 6–8 students’ writing ability? E2. What is the impact of having access to the Toolkit components and professional learning activities on grade 6-8 English Language Arts teachers’ use of effective strategies for teaching writing identified in the Practice Guide?
Implementation Evaluation Research Questions I1. Implementation Description
I2. Implementation Fidelity: To what extent were the Toolkit professional learning activities implemented as intended by peer facilitators, teachers, and administrators? I3. Treatment-Comparison Contrast
|
To recruit schools for the study, we will document research approval from HIDOE’s Office of Curriculum and Instructional Design, in addition to support from the HIDOE State Superintendent in a statement provided by HIDOE, which will be included in the recruitment materials. Following HIDOE’s research policy (and after OMB clearance is received for this study), the study team will make initial contact with the 15 superintendents of the 110 schools via email, with a phone call follow-up shortly thereafter (within two days of the email) to apprise them of the opportunity to participate in the study, gauge interest, and obtain buy-in and approval to reach out to school administrators. With complex area superintendent support, we will target all eligible schools with two or more teachers of grades 6–8 on any island. The study team will leverage connections of on-island REL Pacific staff and study liaisons to reach out via email and phone calls to school administrators of the 110 eligible schools.
Schools will need to meet two criteria to be included in the sample: 1) school needs to agree to participate; and 2) at least 80 percent of the ELA teachers in grades 6–8 will need to agree to participate. The study team, in close coordination with the study liaisons, will work with school administrators to obtain consent from eligible teachers and to execute a Memorandum of Understanding between REL Pacific and each school to clarify with schools what participation entails and provide an outline of activities and timing. After the school administrator expresses interest and agrees to participate, the study team will ask school administrators about the best approach to continuing the conversation with eligible teachers (as a condition of participation, school administrators will have already apprised teachers of the opportunity) and will be flexible in accommodating each school’s suggestions on how to present the study to teachers, which we believe may include REL Pacific staff emailing teachers directly, attending a school staff meeting (virtually or in person), and/or hosting an informational webinar.
The study will result in a final report on the efficacy and implementation of the Toolkit. The study team will report on the impact of the Toolkit on grade 6–8 students’ writing ability, measured as students’ writing scores on the statewide standardized Smarter Balanced Assessment (SBA), and teachers’ use of effective strategies for teaching writing collected from teachers’ instructional logs. In addition, the study team will describe the experiences of staff involved in implementing the Toolkit, the fidelity of implementation, and how experiences differed between staff in treatment and comparison schools.
Exhibit A2 lists the type of administrative and primary data to be collected and describes the purpose of each. The planned data collection and final report are designed to fulfill the requirements of the current REL solicitation that is authorized under the Education Sciences Reform Act (ESRA) of 2002, Part D, Section 174, (20 U.S.C. 9564), administered by the Institute of Education Sciences’ (IES) National Center for Education Evaluation and Regional Assistance (NCEE), and provide actionable information for program improvement.
Exhibit A2. Data Sources
Data Source |
Mode |
Timing of data collection |
Use(s) in Study |
Administrative data |
|||
School-level SBA writing scores |
Data extraction by Hawai‘i Department of Education (HIDOE) |
Spring 2024, Spring 2025 |
E1. Use as key student outcome for the efficacy evaluation analyses |
Public school teacher (such as demographic data; professional and educational data) and school data (such as school-level student demographic characteristics; student enrollment by grade, student retention and attendance rates) |
Data extraction by Hawai‘i Department of Education (HIDOE) |
2023 to 2025 |
E1. Use as covariates in efficacy evaluation analyses |
Primary data |
|||
Teacher instructional logs (peer facilitators, teachers)
|
Online instructional log completed daily for one week over two periods
|
Fall 2024, Spring 2025 |
E2. Use as key teacher outcomes for the efficacy evaluation analyses I3. Use to assess treatment–comparison contrast in writing instruction |
Professional Learning Tracker (peer facilitators, teachers)
|
Online survey
|
Fall 2024 to Spring 2025 |
I1. Understand implementation of the Toolkit (for example, professional learning and instructional resources used, activities were participated in, and helpfulness of these components) I2. Assess fidelity of implementation of the Toolkit |
Focus Groups (peer facilitators, teachers)
|
Virtual focus groups
|
Spring 2025 |
I1. Understand experiences of participating in PLCs and implementing the Toolkit; Identify challenges in implementing the Toolkit, strategies for addressing challenges, and recommendations for improving Toolkit implementation and use |
Administrator survey
|
Online survey
|
Spring 2025 |
I1. Understand implementation of the Toolkit I2. Assess the extent of Toolkit implementation participation and fidelity of implementation I3. Use to assess treatment–comparison contrasts in professional learning supports |
Training attendance records (WestEd) |
Electronic records collected by Toolkit developer |
Fall 2024 to Spring 2025 |
I2. Assess fidelity of implementation of the Toolkit |
The data collection plan is designed to obtain information in an efficient way that minimizes respondent burden. Where feasible, the study team will collect all possible data from administrative sources, rather than through primary data collection. To minimize burden on study participants, the study team will use strategies that have proven successful in past studies that the team has conducted with similar populations of school administrators, students, and parents (such as the Study of Enhanced College Advising in Upward Bound, the Study of Student Messaging in GEAR UP, the Impact Evaluation of Academic Language Interventions, Evaluation of the DC Opportunity Scholarship Program). General strategies to minimize burden using technology are described below for each type of data collection.
To minimize burden on organizations providing administrative data to the study team, administrative data will be collected in electronic form using a secure file-sharing site with FedRAMP security authorization, and web-based interface with drag-and-drop capability and user-friendly administrative controls to efficiently manage internal and external user permissions and levels of access. Organizations will have the flexibility to submit electronic data in a wide range of file formats (Excel, csv, SAS, etc.).
The study team will ensure that surveys are as low burden and secure as possible for respondents. Burden is reduced for all respondents by requesting only the minimum information through well-designed and concise surveys and by allowing respondents to complete tasks at their convenience, while meeting the study objectives. For both the efficacy and implementation evaluations, all instruments will be administered online, which will reduce response time and limit burden. The teacher instructional log and professional learning tracker (PLT) will be administered using Abt’s online survey software (Survey Designer), which allows specific burden-reducing features to be incorporated, including automated skip patterns and validation checks. The study team will use accessible themes in Survey Designer, which include the right amount of color contrast and brightness to ensure surveys are accessible to most people. The study team will also utilize the automated checks available in Survey Designer to verify that all questions are compliant as well as manually run the survey through a screen reader to check for any non-compliant parts.
The Survey Design platform is hosted on Abt Associates’ Data Collection and Analytic Computing Environment (DC-ACE) environment built within Amazon Web Services. DC-ACE is built specifically for collecting, storing, and analyzing sensitive information. DC-ACE complies with HIPAA, FERPA, and FISMA Moderate standards. The surveys themselves are protected via encryption (at rest and in transit) and any web page (including web-based surveys) on DC-ACE are built to and scanned against the OWASP Top 10 standard for web application security.
Log and PLT items will use close-ended response categories, which will improve data quality and lower respondent burden. In addition, to reduce burden by teachers completing logs, we will randomly select a single course section per teacher to serve as the reference for the log entries in the fall and spring. Finally, the study team will provide respondents with a toll-free line they can call with any questions on the instruments, and the study team will answer both substantive and logistical questions to increase ease of completion. Liaisons to each school will also be available to answer respondents’ questions (see below).
To minimize the time and cost burden of travel, all focus groups will be conducted virtually using the WebEx platform. In addition, to ensure that focus group facilitators capture responses accurately and can conduct efficient content analysis of the interviews, the study team will request permission from respondents to digitally record the focus groups and will use the WebEx transcripts, along with notes taken during the focus groups, to create a final set of notes for analysis. The WebEx platform utilizes several features which will protect the privacy of our respondents and the security of the data collected during interviews. First, direct meeting links and passwords will be utilized to ensure only invited meeting participants are able to join the data collection sessions. WebEx offers robust end-to-end encryption “zero-trust” security capabilities based on Automated Certificate Management Environment (ACME) protocols through end-to-end key-based encryption, which protects access to the meeting and the content of video, audio, and text messages transmitted during the meeting.
Whenever possible, the study team will use administrative data to gather the information needed to address the study’s research questions (see Exhibit A1 for the research questions and Exhibit A2 for the types and sources of administrative data to address each question). For example, rather than assess individual students on writing, we are using statewide administrative data on students’ SBA writing scores that are collected annually by HIDOE. However, the information to be collected in the instructional logs, PLTs, focus groups, and administrator surveys is not available from other sources.
No information for this study will be collected from small businesses. The primary small entities for this study are (a) participating schools, and (b) HIDOE. The data collection procedures have been designed to minimize burden on these entities. First, the study team will hire temporary data collection staff in Hawai‘i, such as retired educators, who will serve as study liaisons to each school during the 2024/25 school year. The liaisons will support a wide variety of tasks, such as assisting with collecting teacher consent forms during recruitment, alerting the study team about missing instructional logs, following up with peer facilitators and teachers about completing the PLT, and scheduling focus groups. The liaisons will also be available to answer any questions about data collection procedures and study instruments, which will facilitate the completion of all data collection activities. The study team has used this approach in the past, and it has resulted in efficient and secure data collection. Second, the study team is only requesting school-by-grade level averages and standard deviations from administrative data, rather than individual student data, which substantially decreases burden for HIDOE. In addition, Abt will limit requests for administrative data from HIDOE to the minimum set of variables and records needed to conduct the study. All other entities are individual school personnel, such as administrators, teachers, and peer facilitators (who are also teachers within schools).
The Education Sciences Reform Act of 2002 states that the central mission and primary function of the regional educational laboratories are to support applied research and provide technical assistance to state and local education agencies within their region (ESRA, Part D, section 174[f]). If the proposed data were not collected, REL Pacific would not be fulfilling its central mission to serve the states in the region and provide support for evidence-based research. The systematic collection and analysis of the data described above are required to accomplish the goals of the research project approved by IES. Participation in all data collection activities is voluntary. Information for site recruitment will be collected using the process described in response to question A2. This is a one-time study (that is, not recurring), and therefore periodicity is not addressed.
There are no special circumstances concerning the collection of information in this study. Data collected will be conducted in a manner consistent with the guidelines in 5 CFR 1320.5.
A 60-day Federal Register Notice was published on 4/25/2023 with no comments (88 FR 24979). A 30-day notice was published on 07/31/2023 with no comments (88 FR 49456).
In addition, throughout the course of this study, we will draw on the experience and expertise of Dr. Michael Hebert—Associate Professor and Writing Project Director in the School of Education at the University of California, Irvine, and the subject matter expert for the evaluation—and REL peer reviewers.
To maximize the success of our data collection effort, we propose to provide incentives to offset teachers’, peer facilitators’, and school administrators’ time and effort with completing the data collection activities. Incentives are also proposed because high response rates are needed to make the study findings reliable, and prior evidence suggests the importance of providing an incentive. Monetary incentives reduce non-response bias and improve survey representativeness (Goritz, 2006; Groves et al., 2006; James, 1997; Singer & Kulka, 2002; Singer & Ye, 2013). The proposed compensation amounts are consistent with current guidance from IES and link dollar amounts to the extent of burden.1 However, there is one exception: the teacher log compensation amount exceeds the amount calculated using the 2021 approximate hourly wage rates for teachers identified in the IES Guidelines for Incentives for REL Research Studies (May 2022) by $10, which is proposed because it is particularly burdensome to complete a log every day, and less than $10/day would not likely be a sufficient incentive for this daily effort. This amount was approved by IES by email on 6/1/2022. See Exhibit A3 for a breakdown of participant compensation by data collection activity.
Exhibit A3. Data Source and Compensation
Data Source |
Compensation |
Administrative data from HIDOE |
$0 |
Teacher instructional logs (peer facilitators, teachers) |
$10/daily log (maximum of $100 if all logs are completed) |
Professional learning tracker (peer facilitators, teachers) |
$50 |
Focus groups (peer facilitators, teachers) |
$30 |
Administrator survey |
$15 |
Training attendance records (WestEd) |
NA |
Incentives will be distributed electronically (a link to a gift card) after respondents complete the data collection instruments. Schools randomly assigned to the comparison group will receive $2,500 to be used on activities unrelated to the intervention; this amount has been approved by IES to incentivize participation and create balance across the treatment and comparison groups. This incentive amount is necessary to encourage participation in the study and has been approved by IES.
Abt will conduct all data collection activities for this study in accordance with relevant regulations and requirements, which are:
The Privacy Act of 1974, P.L. 93-579 (5 U.S.C. 552a).
The Family Educational Rights and Privacy Act (FERPA) (20 U.S.C. 1232g; 34 CFR Part 99).
The Protection of Pupil Rights Amendment (PPRA) (20 U.S.C. 1232h; 34 CFR Part 98).
The Education Sciences Institute Reform Act of 2002, Title I, Part E, Section 183.
The study team and the U.S. Department of Education (ED) will protect the confidentiality of all data collected for the study and will use it for research purposes only. All data will be kept in secured locations. Paper files will be stored in a locked file cabinet and all digital files will be password protected so that only project researchers can access them. As soon as all data are received, the study team will replace personally identifiable information (PII) with study-specific identifiers, retaining this PII separately from responses to data collections. PII will be destroyed as soon as it is no longer required. All members of the study team having access to the data will be trained and certified on confidentiality and data security procedures. Reports will present data only in aggregate form, such that reports do not identify individuals and schools. No individual student data will be collected for this evaluation, only school-by-grade writing scores.
The following is an example of the type of statement that will be included in the teacher instructional log, the Professional Learning Tracker (PLT), and the administrator survey since those data will be released in a restricted use dataset by ED:
“Abt is committed to keeping your personal information private. There is a small risk of loss of confidentiality. Abt has many procedures in place to reduce this risk. Paper files will be stored in a locked file cabinet and all digital files will be password protected so that only project researchers can access them. After Abt collects all data, all personally identifiable information (PII) will be replaced with study-created identifiers and all PII will be destroyed. Abt will never include your name or your individual information in any report. Responses to all data collections will be used for statistical purposes only and responses will not be associated with a specific school or individual. Abt will report findings only at the district level and will name HIDOE as the district. Abt will not use your information for any other purpose than this study.
Abt will create a dataset for the U.S. Department of Education (ED) that will include de-identified data from this survey. These data could be distributed to another investigator for future research studies without additional informed consent from you. Abt will destroy all data stored at Abt in 2031, five years after the end of the study, as required by ED.”
The following safeguards are routinely employed by Abt to ensure confidentiality, and they will be consistently applied to this study:
All employees sign a confidentiality pledge that emphasizes the importance of confidentiality and describes employees’ obligations to maintain it.
Personally identifiable information (PII) is maintained on separate forms and files, which are linked only by sample identification numbers. All documents with PII will be destroyed after the study team replaces PII with study-created identifiers.
Access to hard copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are collected in secure, company-provided shredding bins. The study team contracts services from a data records and management company that empties the contents of the shredding bins into large totes, which are then transported to the company’s facility, where the contents are securely shredded.
All digital files will be password protected so that only project researchers can access them.
To ensure the security of administrative data requested from HIDOE, the study team uses a FIPS 140-2-compliant data sharing tool (MOVEit) that encrypts data in transit and at rest. This data sharing tool allows the study team to manage user permissions for uploading, viewing, editing, or downloading data and restrict unauthorized access to data.
The study will take several steps to safeguard respondent information:
All contractor staff will comply with the security investigation requirements governed by their risk/sensitivity level, as detailed in the ED Contractor Vetting Security Requirements (11-1-2019).
All contractor staff will receive instruction in the privacy requirements of the study.
Access to any data with identifying information will be limited to contractor staff directly working on the study. Access to electronic data will require individual usernames and passwords.
Names and other identifying information for survey respondents will be replaced with numerical identifiers after the data are collected and prior to analysis. A key linking the names to the identifiers will be kept in a separate location, with access for Abt staff on a need-only basis.
Any quotations from responses used in public reporting will be edited to ensure that the identity of the respondent cannot be ascertained, with the caveat that responses from program operator staff will be attributable to the program operator (but not an individual staff member).
The study team will house survey data on the Confirmit platform, which is Federal Information Security Management Act Moderate compliant. For analyses, the study team will store de-identified data within Abt’s Analytic Computing Environment (ACE), which complies with HIPAA, FERPA, and FISMA Moderate standards implemented in FedRAMP-certified Amazon Web Services (AWS) environments. Additional security features include encrypted storage, intrusion detection, and audit log aggregation. ACE is monitored seven days a week, 24 hours a day using advanced monitoring and alerting tools. Expert security and IT staff continuously review audit logs, conduct regular scans, and apply patches. Redundant backups are stored both on AWS and at a remote data center.
Abt Associates Inc. (IORG#0000913) holds a current Federal-Wide Assurance (FWA) for the Protection of Human Subjects from the U.S. Department of Health and Human Services’ Office for Human Research Protections (FWA#00000664; expires 1/12/2026) and maintains its own internal Institutional Review Board (IRB) [IRB#00001281]. The Abt IRB has determined that this study is eligible for IRB exemption.
The study does not include questions of a sensitive nature. Teachers completing the instructional log will be asked about their writing instruction practices, such as the length of time spent on and types of writing instruction and writing activities, use of assessments of student writing, and use of evidence-based writing practices. Teachers and peer facilitators completing the professional learning tracker will be asked about experiences with the toolkit and PLC sessions, and implementation fidelity of the toolkit. Teachers and peer facilitators participating in focus groups will be asked about their experiences and challenges with toolkit implementation, along with recommendations for future toolkit implementation. Administrators completing the administrator survey will be asked about their implementation of and participation with the Toolkit, usefulness of trainings and resources, background characteristics, and writing professional development provided to teachers.
The study team will use teacher background characteristics such as gender, race/ethnicity, years of teaching experience, and educational credentials from teacher administrative data that is collected by the state in two ways. First, the study team will aggregate these data across study participants to describe the sample of teachers and write about the extent to which the sample is similar to all teachers in Hawaii in terms of demographic characteristics. Second, the study team will construct school-level aggregates of these background characteristics and will use them as covariates in the impact estimation models.
The data collection plan has been designed to maximize efficiency, accuracy, and convenience for respondents and to minimize their burden. Over the two-year recruitment and data collection period, the total respondent burden associated with this study is 831 hours, and the total cost of burden is $27,721. The recruitment burden is 163 hours, and the cost of recruitment burden is $7,298. The data collection burden is 668 hours, and the cost of data collection burden is $20,423. The annualized respondent burden is 416 hours, and the cost is $13,861. Exhibit A4 presents the total and annualized estimated time and cost burden for the study. The following assumptions informed these burden estimates: Our sample will include 40 schools in Hawai‘i selected as a convenience sample from the pool of 110 schools with at least two teachers serving at least one target grade (6–8). The schools will be randomly assigned to the treatment group (20 schools) or comparison group (20 schools).
Teachers and Peer Facilitators
Using HIDOE data, the arithmetic average of the number of grade 6–8 ELA teachers is seven per school. In order for schools to be eligible for the study, the study team requires at least 80 percent of teachers in schools to agree to participate in the study.
Given these parameters and to be conservative, the estimated number of teachers participating per school is five, which yields an estimate of 200 teachers in the study sample (= 5 teachers X 40 schools), with 100 teachers in the treatment group and 100 teachers in the comparison group. Of the 100 teachers in the treatment group, 20 are expected to be peer facilitators (one teacher per treatment school). The number of respondents reported in Exhibit A4 account for response rates, which the study team estimates as 85 percent for teachers and 90 percent for peer facilitators.
Administrators: The estimated number of administrators in the study sample is 40, where one administrator in each of the 40 schools will complete the administrator survey. The study team expects a response rate of 85 percent for administrators.
Hourly wage estimates for the HIDOE administrator fulfilling the data request are based on a median annual salary of $96,710 per year (Hourly wage: $47) in 2021 from the 2021 Bureau of Labor Statistics Occupational Outlook Handbook for Database Administrators. Hourly wage estimates for teachers ($30), peer facilitators ($30), and administrators ($47) are based on guidance from IES.
Exhibit A4. Estimate of Respondent Time and Cost Burden
Activity/Output |
Number of Respondents |
Responses per Respondent |
Number of Responses |
Hours per Response |
Total Burden Hours |
Total Cost of Respondent Burden a |
|||
Recruitment Burden |
|||||||||
Superintendent Email |
15 |
1 |
15 |
0.05 |
1 |
$35 |
|||
Superintendent Email (follow-up) |
15 |
1 |
15 |
0.05 |
1 |
$35 |
|||
Superintendent Phone Call |
15 |
1 |
15 |
0.16 |
2 |
$113 |
|||
Complex Area Superintendent Total |
45 |
|
4 |
$183 |
|||||
School Administrator Email |
110 |
1 |
110 |
0.05 |
6 |
$259 |
|||
School Administrator Email (follow-up) |
110 |
1 |
110 |
0.05 |
6 |
$259 |
|||
School Administrator Phone Call |
110 |
1 |
110 |
0.16 |
18 |
$827 |
|||
School Administrator Information Session |
110 |
1 |
110 |
1 |
110 |
$5,170 |
|||
Administrator Total |
440 |
|
139 |
$6,514 |
|||||
Teacher Email |
200 |
1 |
200 |
0.05 |
10 |
$300 |
|||
Teacher Email (follow-up) |
200 |
1 |
200 |
0.05 |
10 |
$300 |
|||
Teacher Total |
400 |
|
20 |
$600 |
|||||
Total Recruitment Burden |
885 |
|
163 |
$7,298 |
|||||
Data Collection Burden |
|||||||||
HIDOE Administrative Data |
1 |
1 |
1 |
16 |
16 |
752 |
|||
HIDOE Total |
1 |
|
16 |
$752 |
|||||
Administrator Survey |
34 f |
1 |
34 |
1 |
6 |
$266 |
|||
Cognitive Testing of Administrator Survey |
2 |
1 |
2 |
1 |
2 |
$71 |
|||
Administrator Total |
36 |
|
7 |
$337 |
|||||
Teacher Log |
170 b |
10 |
1,700 |
0.25 |
425 |
$12,750 |
|||
PLT for Teachers |
68 c |
8 |
544 |
0.17 |
91 |
$2,720 |
|||
Teacher Focus Group |
68 c |
1 |
68 |
1 |
68 |
$2,040 |
|||
Cognitive Testing of Teacher Log |
6 |
1 |
6 |
1 |
6 |
$180 |
|||
Cognitive Testing of PLT for Teachers |
2 |
1 |
2 |
1 |
2 |
$60 |
|||
Cognitive Testing of Teacher Focus Group |
2 |
1 |
2 |
1 |
2 |
$60 |
|||
Teacher Total |
2,322 |
|
594 |
$17,810 |
|||||
PLT for Peer Facilitators |
18 d |
8 |
144 |
0.20 |
29 |
$864 |
|||
Peer Facilitator Focus Group |
18 d |
1 |
18 |
1 |
18 |
$540 |
|||
Cognitive Testing of PLT for Peer Facilitators |
2 |
1 |
2 |
1 |
2 |
$60 |
|||
Cognitive Testing of Peer Facilitator Focus Group |
2 |
1 |
2 |
1 |
2 |
$60 |
|||
|
Peer Facilitator Total e |
166 |
|
51 |
$1,524 |
||||
|
Total Data Collection Burden |
2,525 |
|
668 |
$20,423 |
||||
|
TOTAL BURDEN OVER TWO YEARS |
3,410 |
|
831 |
$27,721 |
||||
|
ANNUALIZED BURDEN |
1,705 |
|
416 |
$13,861 |
||||
a Hourly wage estimates for the HIDOE administrator fulfilling the data request are based on a median annual salary of $96,710 per year (Hourly wage: $47) in 2021 from the 2021 Bureau of Labor Statistics Occupational Outlook Handbook for Database Administrators. Hourly wage estimates for teachers ($30), peer facilitators ($30), and administrators ($47) are based on the IES Guidelines for Incentives for REL Research Studies (May 2022). |
|||||||||
b Includes 170 teachers, the estimated number of teachers in the study sample (n = 200) multiplied by the 85 percent response rate. |
|||||||||
c Includes 68 teachers, the estimated number of teachers in the treatment group (excluding peer facilitators; n = 80) multiplied by the 85 percent response rate. |
|||||||||
d Includes 18 peer facilitators, the estimated number of peer facilitators in the study sample (n = 20) multiplied by the 90 percent response rate. |
|||||||||
e Peer facilitators also complete the teacher portal section of the PLT, which is included in their burden estimate. |
|||||||||
f Includes 34 administrators, the estimated number of administrators in the study sample (n = 40) multiplied by the 85 percent response rate. |
|||||||||
There are no annualized capital/startup or ongoing operation and maintenance costs involved in the collection of the proposed data.
The total cost to the federal government of the evaluation over all five years is $2,058,103. This cost includes staff time for REL Pacific to recruit schools and teachers and for the independent evaluator to collect, clean, analyze, and report on primary and extant data. It also includes costs for REL Pacific staff and the independent evaluator related to study preparation and submission of the study information to IES (from proposed research design through reporting of results). The average annual cost to the federal government is $411,621.
This is a request for a new collection of information.
The study team will use regression models, generalized mixed models, descriptive analyses, and applied thematic analyses to address the study’s research questions:
Efficacy Evaluation
Regression analyses will be used to estimate the impact of access to the Toolkit components and professional learning activities on school-level averages of students’ writing scores. Analyses will control for school-level averages of students’ writing scores from the prior year, in addition to school and teacher characteristics data.
Generalized mixed models will be used to estimate the impact of access to the Toolkit components and professional learning activities on teachers’ practices, which are based on totals/count variables and yes/no responses. Totals/count variables will be analyzed using a Poisson distribution with a log-link function, or dichotomous indicators would be analyzed using a binomial distribution with a logistic function. Analyses will control for school and teacher characteristics data.
Implementation Evaluation
Descriptive analyses will be used to summarize all administrative and survey data. For measures using continuous scales, the study team will calculate means and standard deviations to describe central tendency and variation. For categorical scales, the study team will use frequency distributions and percentages.
Applied thematic analyses will be used to analyze the qualitative data from the teacher and peer facilitator groups. This approach is structured by the research questions, while allowing new themes to emerge from the data. These analyses support the identification of common themes related to the experiences of teachers and peer facilitators in participating in and implementing the Toolkit.
Regression via generalized mixed models will also be used to examine differences in teacher receipt of professional development and practices by treatment condition.
Per reporting requirements in the REL solicitation request (Solicitation #91990020R0032), this evaluation will result in one 15-page report that summarizes the results from the efficacy and implementation evaluation conducted during the 2024 to 2025 school year, to be released in May 2027.
A restricted use data file (RUF) will be made available with the de-identified data that the study team collects for this study. The production and distribution of the RUF is to ensure other researchers can replicate the findings presented in the report or answer additional research questions.
All restricted use files are required to be reviewed by IES’ Disclosure Review Board (DRB). The DRB is comprised of members from each National Center for Education Statistics Division, representatives from the Statistical Standards Program, and a member from each of the IES Centers. The DRB will review disclosure risk analyses conducted by the REL contractor to ensure that data released do not disclose the identity of any individual respondent. The DRB approves the procedures used to remove direct identifiers from restricted use data files.
The timeline for the activities in the evaluation and implementation study, including recruitment, data collection, analyses, and reporting, is in Exhibit A5.
Exhibit A5. Project Timeline
Timeline for Efficacy and Implementation Evaluation of Secondary Writing Toolkit |
||||||||||||
Activity/Milestone |
2022 |
|||||||||||
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 |
|
Proposal revisions |
|
|
|
|
|
|
|
|
|
|
|
|
Instrument development and revisions |
|
|
|
|
|
|
|
|
|
|
|
|
McREL & SME review |
|
|
|
|
|
|
|
|
|
|
|
|
Proposal submission to IES |
|
|
|
|
|
|
|
|
|
|
|
|
IES proposal review (6 months) |
|
|
|
|
|
|
|
|
|
|
|
|
Abt/REL PA start talking to HIDOE about data agreement |
|
|
|
|
|
|
|
|
|
|
|
|
Activity/Milestone |
2023 |
|||||||||||
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 |
|
IES proposal review (continued) |
|
|
|
|
|
|
|
|
|
|
|
|
IRB submission & approval |
|
|
|
|
|
|
|
|
|
|
|
|
Start drafting OMB package (after one round of RPR) |
|
|
|
|
|
|
|
|
|
|
|
|
OMB package to IES and revisions |
|
|
|
|
|
|
|
|
|
|
|
|
OMB approval (6 months) |
|
|
|
|
|
|
|
|
|
|
|
|
Pilot study for teacher log |
|
|
|
|
|
|
|
|
|
|
|
|
Cognitive interviews & revisions |
|
|
|
|
|
|
|
|
|
|
|
|
HIDOE Research Application and Data Sharing Agreement |
|
|
|
|
|
|
|
|
|
|
|
|
Obtain letter of support for study from HIDOE State Superintendent |
|
|
|
|
|
|
|
|
|
|
|
|
Study registration at Registry of Efficacy and Effectiveness Studies (REES) after IES proposal approval |
|
|
|
|
|
|
|
|
|
|
|
|
Complex Area recruitment (schools in Hawai‘i are organized into Complex Areas) |
|
|
|
|
|
|
|
|
|
|
|
|
School recruitment, including MOUs and teacher consent |
|
|
|
|
|
|
|
|
|
|
|
|
Activity/Milestone |
2024 |
|||||||||||
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 |
|
School recruitment, including MOUs and teacher consent |
|
|
|
|
|
|
|
|
|
|
|
|
Online programming of teacher log and PLT |
|
|
|
|
|
|
|
|
|
|
|
|
Random assignment of schools |
|
|
|
|
|
|
|
|
|
|
|
|
Request and obtain grades 6–8 school-level SBA Writing and demographic data for study schools from HIDOE |
|
|
|
|
|
|
|
|
|
|
|
|
Request teacher class schedules from participating schools |
|
|
|
|
|
|
|
|
|
|
|
|
Random selection of teacher class sections for logs |
|
|
|
|
|
|
|
|
|
|
|
|
Treatment and comparison teachers complete log (daily for one week in September prior to beginning of Toolkit implementation) |
|
|
|
|
|
|
|
|
|
|
|
|
Peer facilitator training |
|
|
|
|
|
|
|
|
|
|
|
|
Monthly coaching for peer facilitators |
|
|
|
|
|
|
|
|
|
|
|
|
PLCs meet (October 2024–February 2025) |
|
|
|
|
|
|
|
|
|
|
|
|
Treatment teachers and peer facilitators complete professional learning tracker (October 2024–February 2025) |
|
|
|
|
|
|
|
|
|
|
|
|
Cleaning and analysis (ongoing as data are collected) |
|
|
|
|
|
|
|
|
|
|
|
|
Activity/Milestone |
2025 |
|||||||||||
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 |
|
Monthly coaching for peer facilitators (continued) |
|
|
|
|
|
|
|
|
|
|
|
|
PLCs meet (continued) |
|
|
|
|
|
|
|
|
|
|
|
|
Treatment teachers and peer facilitators complete professional learning tracker (continued) |
|
|
|
|
|
|
|
|
|
|
|
|
Treatment and comparison teachers complete logs (daily for one week in February) |
|
|
|
|
|
|
|
|
|
|
|
|
Focus groups with teachers and peer facilitators, survey with administrators |
|
|
|
|
|
|
|
|
|
|
|
|
Cleaning and analysis (ongoing as data are collected) |
|
|
|
|
|
|
|
|
|
|
|
|
Request and obtain grades 6–8 school-level SBA Writing data for study schools from HIDOE |
|
|
|
|
|
|
|
|
|
|
|
|
Report writing |
|
|
|
|
|
|
|
|
|
|
|
|
Activity/Milestone |
2026 |
|||||||||||
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 |
|
Report writing |
|
|
|
|
|
|
|
|
|
|
|
|
Draft report submission to IES |
|
|
|
|
|
|
|
|
|
|
|
|
IES Phase I Review (est. 10 months) |
|
|
|
|
|
|
|
|
|
|
|
|
Activity/Milestone |
2027 |
|||||||||||
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 |
|
IES Phase I Review (continued) |
|
|
|
|
|
|
|
|
|
|
|
|
IES Phase II Review (est. 3 months) |
|
|
|
|
|
|
|
|
|
|
|
|
Report released |
|
|
|
|
|
|
|
|
|
|
|
|
No exemption is requested. The data collection instruments will display the expiration date.
The submission describing data collection requires no exemptions to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).
Goritz, A. (2006). Incentives in web studies: Methodological issues and a review. International Journal of Internet Science, 1(1), 58–70.
Graham, S. (2019). Changing how writing is taught. Review of Research in Education, 43(1), 227–303. doi.org/10.3102/0091732X18821125
Graham, S., Bruch, J., Fitzgerald, J., Friedrich, L., Furgeson, J., Greene, K., Kim, J., Lyskawa, J., Olson, C. B., & Smither Wulsin, C. (2016). Teaching secondary students to write effectively (NCEE 2017-4002). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.
Groves, R. M., Couper, M. P., Presser, S., Singer, E., Tourangeau, R., Acosta, G., & Nelson, L. (2006). Experiments in producing nonresponse bias. Public Opinion Quarterly, 70(5), 720–736.
James, T. (1997). Results of Wave 1 Incentive Experiment in the 1996 Survey of Income and Program Participation. Proceedings of the Survey Research Section, American Statistical Association, 834–883.
National Center for Education Statistics (NCES). (2020, September 21). National Assessment of Educational Progress (NAEP): Writing. nces.ed.gov/nationsreportcard/writing/
National Institute for Literacy. (2008). Developing early literacy: Report of the National Early Literacy Panel. National Center for Family Literacy. lincs.ed.gov/publications/pdf/NELPReport09.pdf.
Singer, E., & Kulka, R. A. (2002). Paying Respondents for Survey Participation. In Studies of welfare populations: Data collection and research issues. Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs, edited by M. Ver Ploeg, R. A. Moffitt, & C. F. Citro. Committee on National Statistics, Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press, pp. 105–128.
Singer, E. & Ye, C. (2013). The use and effectives of incentives in surveys. Annals of the American Academy of Political and Social Science, 645(1), 112–141.
U.S. Bureau of Labor Statistics. (2019). Median weekly earnings $606 for high school dropouts, $1,559 for advanced degree holders. U.S. Department of Labor. www.bls.gov/opub/ted/2019/median-weekly-earnings-606-for-high-schooldropouts-1559-for advanced-degree-holders.htm
U.S. Bureau of Labor Statistics. (2021). Occupational Employment and Wages. Retrieved from www.bls.gov/oes/current/oes119151.htm
1 IES consulted the Bureau of Labor Statistics (BLS) Occupational Outlook Handbook to identify the most current information (currently from 2021) about educator wages to calculate reasonable incentive amounts. Across classroom educator (teacher) categories, the 2021 approximate annual wage is $61,500. Across principals, the approximate annual wage for 2021 is $98,420. By dividing the annual wages by 2080 hours, IES arrived a teacher hourly rate of $30/hour and a principal hourly rate of $47/hour.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Allan Porowski |
| File Modified | 0000-00-00 |
| File Created | 2024-11-27 |