ED Response to OMB Comments

MEMORANDUM

TO: Sandra Garcia DATE: April 18, 2014

FROM: Russell Gersten, Rebecca Newman-Gonchar, Joe Dimino, Madhavi Jayanthi

SUBJECT: Response to public comments from National School Boards Association on

ED Notice of Information Collection; Docket ID: ED-2013-ICCD-0157

Title of Collection: The Impact of Professional Development in Fractions for Fourth Grade

OMB Control Number: 1850-NEW

This memo summarizes ED’s response to the comments from the National School Boards Association (NSBA) regarding the OMB package for The Impact of Professional Development in Fractions for Fourth Grade study, received on April 7, 2014.

In this memo, we have included the comments from the NSBA’s letter and responded in blue font to each comment. For each comment that required changes to the OMB package, we have provided page numbers from the revised OMB package where the change was made. As you can see, most of these changes are clarifications of material in the earlier OMB package.

NSBA Comment 1:

NSBA supports providing opportunities for teachers to receive PD to become better educators for their students. However, NSBA is concerned that this Notice goes much farther than merely requesting permission to collect data. To obtain the data sought, ED will need fourth grade teachers to participate in a PD program that would be squeezed into eight sessions during the already-short first semester of the coming 2014-2015 school year (SY). More specifically, ED states that, at the front end, it will seek to have approximately 252 teachers from certain Georgia and South Carolina elementary schools go through this PD training consisting of 8 three-hour sessions, plus additional time for homework, all during just the fall semester, and have those teachers fill out nine monthly PD surveys and take a pre- and post-test on fractions. Then, at the back end, administer a fractions assessment to the students of those teachers during the spring semester of the 2014-2015 SY to determine if there is any improvement in their use of fractions.

REL-SE Response 1:

The text has been changed to reflect the proposed length of the professional development (PD) program, rather than the semester during which the PD will take place. Wherever the text referred to the PD sessions taking place during the Fall semester (e.g., page D-3 of Appendix D), the text now reads that the PD sessions will take place periodically during a 6-month period of the school year (beginning typically in September 2014 and ending typically in February 2015). The study team will work with district and school personnel in order to coordinate the specific dates of the sessions to avoid conflicts with other commitments. The timeframe during which the PD sessions will take place may be extended at the request of a school or district.

The timelines in Part A, sections A14 and A15 were corrected to reflect the possibility of the sessions taking place during the 6-month period of the school year.

In addition to attending PD sessions during this timeframe, all teachers (both experimental and control group) will be asked to complete a 12-minute survey every month for 9 months of the school year, a <12-minute demographic survey, and <60-minute fractions measure at the beginning of the year, and a <60-minute fractions measure at the end of the PD program. In total the measures will take teachers no more than 4 hours over the course of the school year. (Note that the calculations of estimated burden include an extra hour for each fractions measure to allow teachers to travel to the PD site to complete the measure, to account for the possibility that the measure is not administered at their school site.)

Monetary Concerns

NSBA Comment 2:

When will the 8 three-hour sessions of PD be conducted during the Fall 2014 semester?

After School Hours? If the PD is to take place after school hours, it is unclear from the Notice who is going to pay for the teachers’ time. Technically, teachers are “off contract” once the school day ends, and are not required to engage in any duties without being paid overtime above and beyond their contracted salaries. As part of its “randomized control trial study,” is ED going to compensate these teachers for their 24 hours of PD class time plus the time they spend on the “additional homework lessons”?

Right now, school districts are wrapping up their work on the budget process for the 2014-2015 SY. It is unlikely that such additional compensation has been included in the budget drafts for the yet-to-named affected school districts in Georgia and South Carolina. And given the tight time constraints teachers already face in balancing their work and home lives during the school year, it is unlikely that the desired number of teachers would voluntarily participate in the PD program requiring that much of a time commitment without compensation.

During School Hours? If the teachers would be expected to attend the PD during the school day, it is unclear from the Notice itself who is going to pay for the costs of the substitute teachers needed to cover the classrooms of those 252 teachers. Supplemental documents state that the ED trial study is not going to pay for the time of substitute teachers. This is a big expense that will have a direct financial impact on school districts, though ED states in its materials that it will not. Such teacher absences will result in considerable expense for the affected school districts since 24 clock hours of PD x 252 teachers = 6,048 hours of substitute teacher coverage that will be required to permit the teachers’ attendance. Typically, substitute teachers are not paid by the hour, but by the half- or full-day of coverage. Again, this is a cost that the affected school districts probably have not factored into their 2014-2015 SY budgets already in the approval process now.

REL-SE Response 2:

The text has been changed to clarify that

1. The PD program includes

   1. 24 hours of PD (eight 3-hour units) that will be presented during five PD sessions over a 6-month period of the school year. Unit pairs 1 and 2, 4 and 5, and 7 and 8 will be presented in three 6-hour PD sessions, and the two more difficult units (3 and 6) will be presented individually.

   2. Approximately 8 total hours of assignments that teachers will need to complete before attending the five PD sessions, such as reading PD materials. When sessions take place during the work day, teachers will have time to complete these assignments during the work day since sessions will last only 3 or 6 hours of the full work day.

2. The five PD sessions will be presented over a combination of Release Days and Saturdays. The specific schedule and structure of the sessions will be worked out in consultation with the DMI PD facilitators and local school and district personnel. The two parties will decide what will work best for the facilitators and the teachers, given the school and district calendars. It may be that they decide to present the five PD sessions over five Release Days or over five Saturdays or some combination of the two. (Note the estimated burden in Part A, Section A12 of the revised OMB package is now based on the maximum time teachers may be expected to participate in PD and complete assignments outside of their normal work day (i.e., five Saturday PD sessions), reflecting a jump in annualized burden from 595 hours to 2112 hours.) Holding PD sessions after school does not appear to be advisable given the nature of the PD. None of the research alliance members indicated after school sessions as a possible preference. All have indicated districts will most likely want a mix of Release Days and some Saturday sessions.

3. If any of the five PD sessions take place during the work day and teachers are released from their teaching responsibilities, we will share the cost of the substitutes with the school or district. In most districts and schools, teachers are required to attend in-services during the work day and their substitutes are paid for by the school or district. If any of the five PD sessions are scheduled to replace typical district- or school-mandated PD, districts or schools will likely pay for the substitute. For additional PD days, if schools or districts cannot afford the cost of the substitutes, we will offer to share that cost. (Note that the cost of paying substitutes in those cases where districts or schools refuse to pay for them has been included in the cost of the study and is not presented as an additional cost or incentive for this study.)

4. Teachers will be paid their typical hourly rate (varies by state and district and often by seniority) for any time they spend attending PD sessions outside of their work day (i.e., on Saturdays). If teachers attend sessions on Saturdays, they will receive their hourly rate for the time they spend attending the session, completing the preparation assignment, and traveling to the PD site. (Note that the cost of paying teachers for the time they attend PD sessions or complete assignments has been included in the cost of the study and is not presented as an additional cost or incentive for this study.)

Changes were made on pages A-4 and A-5 of Appendix A, D-3 of Appendix D, E-2 and E-3 of Appendix E, L-10 and L-11 of Appendix L, and 18, 19 and 20 of Part A. A complete explanation of the burden the DMI PD program may place on teachers can now be found in Appendix J.

NSBA Comment 3:

Where is the PD to take place?

If the PD is to take place at a central location, since the 84 schools will be spread out across two states, will ED be reimbursing the teachers for their mileage, food, and housing costs (if any) that they incur to travel to the training site? Or is the PD to take place at each teacher’s school? The Notice does not provide the logistical details of how this entire project is to be completed.

REL-SE Response 3:

As stated on page D-3 of Appendix D and E-3 of Appendix E, the PD will take place at a convenient location within each district. Teachers will not be reimbursed for their mileage, food, or housing costs as they will not be traveling far from their school or district. Thus, there will be multiple implementations of PD across the states (and not in one central location). As noted above, if teachers attend sessions on Saturdays, they will receive their hourly rate for the time they spend attending the session, completing the preparation assignment, and traveling to the PD site.

Non-Monetary Concerns

NSBA Comment 4:

Interruption of Student Learning

Typically, elementary school teachers do not teach just one subject during the day. Each teacher teaches most, if not all, areas of the district’s fourth grade curriculum during the course of the day/week. Thus, if the PD takes place during the school day necessitating the teacher’s absence, those teachers’ students will be taught by a substitute teacher for three clock hours a week on eight separate occasions over the course of the short 18-week Fall 2014 semester. Also, it is highly probable that it will be a different substitute teacher for each period of absence. Though substitute teachers are a valued part of the education system, not all districts require substitute teachers to have a teaching certificate or even any post-secondary education. Some only require a high school diploma. Thus, some of the instruction being provided in solid three-hour blocks to those affected fourth graders during the regular teacher’s absences may be by a person who is not licensed and/or has no instruction or experience in how to teach students in any subject area whatsoever.

Since the PD is to take place during just the Fall 2014 semester, the students of the 252 identified teachers will be receiving 24 clock hours of instruction from less than fully qualified persons. Assuming an average class size of 20 students, a little over 5,000 students’ educational growth would be negatively impacted during the fall semester of their fourth grade year, and not just in math instruction, but in instruction in a multitude of content areas depending on the curricula of the affected school districts.

REL-SE Response 4:

As stated above, the five PD sessions will be presented over a combination of Release Days and Saturdays. The specific schedule and structure of the sessions will be worked out in consultation with the DMI PD facilitators and local school and district personnel. The two parties will have to decide what will work best for the facilitators and the teachers, given the school and district calendars. It may be that they decide to present the five PD sessions over five Release Days or over five Saturdays or some combination of the two. (Note we estimated burden based on the maximum time teachers may be expected to participate in PD and complete assignments outside of their normal work day (i.e., five Saturday PD sessions).)

The professional development we are offering may prepare teachers to more effectively teach fractions to their students. Not only will teachers benefit from 24 hours of high quality professional development aimed at improving their understanding of fractions, but their students will hopefully gain a deeper understanding of the mathematics that underlie fractions from the higher quality fractions instruction they receive as a result of the professional development their teachers attend. So while students may be taught by substitutes a maximum of 5 days spread out over 6 months of the school year, they will be gaining more effective instruction and a deeper understanding of fractions in the long run.

NSBA Comment 4:

Validity of Data Collected from Assessment of Students

From the Notice, it appears that the students of the teachers going through the PD will be tested during the Spring 2015 semester to ascertain any student achievement in working with fractions. However, it is unknown to what data the assessment results will be compared to determine whether there has been student achievement. It would be unreasonable to compare the fourth graders’ outcomes in Spring 2015 to testing results of fourth grade classes from previous years, because they are different students. Also, given that the teachers would have just received the PD on fractions in the previous semester, it is questionable how much improvement the students of those teachers would demonstrate in working with fractions at the end of the very next semester.

Working with fractions is a skill that is expanded upon over several years as students progress through a school district’s mathematics curriculum. It is unclear what one assessment at the end of the fourth grade year will show to justify the disruption in the educational growth of those students in the other areas of the curriculum.

REL-SE Response 4:

As in medical research, participants in this study (i.e., schools) will be randomly assigned to the group that receives the intervention or the group that does not. That means, half of the interested schools will receive the DMI mathematics professional development and half will receive the typical professional development offered by their district (i.e., they will be in the control group). To determine the impact of the professional development on student learning, 4^th grade students in all the participating schools (both treatment and control) will be asked to take a fractions posttest several months after the professional development has ended (typically 4-6 weeks before the end of the school year). After statistically controlling for entry-level mathematics ability (i.e., students’ prior spring 3rd grade math achievement scores on the state standardized assessment) using covariance methods, 4^th grade students’ scores on the fractions posttest from the experimental group (i.e., whose teachers received the DMI professional development during the 2014-2015 school year) will be compared to 4^th grade students’ scores on the fractions posttest from the control group (i.e., whose teachers received typical district professional development during the 2014-2015 school year). This design allows us to conduct a scientifically rigorous study of the impact of this type of professional development on student knowledge of fractions and to ensure that the results take into account students’ mathematics proficiency prior to the beginning of the study.

The rigorous literature review¹ conducted in preparation for this study provides evidence that studies of this nature can detect impacts of professional development on student achievement within a school year. This literature review is available at

http://ies.ed.gov/ncee/edlabs/regions/southeast/pdf/REL_2014010.pdf.

1 Gersten, R., Taylor, M. J., Keys, T. D., Rolfhus, E., & Newman-Gonchar, R. (2014). Summary of research on the effectiveness of math professional development approaches. (REL 2014–010). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southeast. Retrieved from http://ies.ed.gov/ncee/edlabs.