4.11 OMB B

SUPPORTING STATEMENT B:

REQUEST FOR CLEARANCE OF INFORMATION COLLECTION FORMS FOR

“Understanding the Impact of Providing Information to Parents about the Role of Algebra II: An Opportunistic Study”

January 2014

Submitted to: Submitted by:

U.S. Department of Education SEDL

Institute of Education Sciences 4700 Mueller Blvd.
555 New Jersey Ave. NW, Rm. 308 Austin, TX 78723

Washington, DC 20208 Phone: (800) 476-6861

4700 Mueller Blvd. Austin, TX 78723

800-476-6861

www.relsouthwest.org

This publication was prepared for the Institute of Education Sciences (IES) under contract ED-IES-12-C-00012 by Regional Educational Laboratory Southwest, administered by SEDL. The content of the publication does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government. The publication is in the public domain. Authorization to reproduce in whole or in part for educational purposes is granted.

Content

s

1. Respondent Universe and Sampling Methods 6

2. Description of Procedures for the Collection of Information 8

Data Sources 8

3. Description of Procedures for Maximizing Response Rates 9

4. Description of Tests, Procedures, and Methods 10

5. Individuals Consulted on Statistical Aspects of the Design and Key Staff 11

Attachment B-1. School Agreement to Participate Form 13

The U.S. Department of Education (ED) requests clearance for data collection under the Office of Management and Budget (OMB) clearance agreement (OMB number [IES to complete]) for activities related to the Regional Educational Laboratory (REL) Program. ED, in consultation with SEDL, intends to conduct a timely, policy relevant opportunistic experiment to determine if directly providing parents/guardians, prior to students’ selection of their courses, with information on the importance of completing algebra II for college access and success has an impact on the percentage of students who enroll in and complete algebra II by the end of their junior year.

In 2013, the governor of Texas signed House Bill (HB) 5—the Foundation High School Program—into law. HB changed the high school graduation requirements and reduced the number of required state secondary school exams for public high school students in Texas. Prior to enactment of HB 5, high school students were required take four courses each in English, mathematics (including algebra II), science, and social studies—earning the credits they needed to be admitted to most state universities and colleges. Now, only one of the three new graduation plans¹, created for the Foundation High School Program requires completion of algebra II in order to graduate. Despite the change in high school graduation requirements, the public universities in Texas will still require students to have completed algebra I, geometry, and algebra II in order to gain admission

Thus, a relevant question is whether providing parents/guardians with information about the importance of completing algebra II will help prevent students who aspire to postsecondary degrees from opting to skip algebra II in high school. This study builds on research being conducted in a companion study that investigates the trends in the percentage of students in a school who complete algebra II by the end of their junior year for the graduating classes of 2009-10 through 2016-17.

The results of this study will provide Texas Education Agency (TEA) and the Texas Higher Education Coordinating Board (THECB) with information about the role that information dissemination can play in influencing students’ course selection and completion. Information from this study will provide TEA with valuable information that will inform their attempts to stem the number of students who wish to go on to postsecondary education, but may not be aware of the importance completing algebra II has toward this goal. The intervention is relatively inexpensive, so if it is successful, it would demonstrate the benefit of low-cost interventions for influencing student course selection. TEA will be able to use this information to answer questions about changes in course taking patterns and course failure rates that may arise from parents, education practitioners and administrators, policymakers, and researchers. It will also allow them to respond to inquiries regarding any variation in how districts responded to the HB 5 graduation requirements.

Additionally, if the results of the study show that schools in which students’ parents were provided with information describing changes to the Texas high school graduation requirements, with a specific emphasis on changes to the mathematics requirements and how these changes may impact college access, have higher algebra II completion rates, TEA could recommend that districts conduct an information campaign similar to this one. The Texas Association for School Administrators has already expressed an interest in providing the materials to its members—all school principals in Texas. Similarly, if this study shows positive results, TEA could produce materials for districts to disseminate to parents when statewide policy changes occur. This would allow for a clear and consistent message that is currently lacking, as schools and districts must create their own materials based on their own interpretations of the policy.

Finally, results of our study will provide evidence to other states about the impact of providing parents/guardians with information about course completion. Due to the size and demographic makeup, many states look to Texas for guidance when making decisions about implementing statewide policies and programs. The findings of this study should be of particular interest to those states.

OMB approval is being requested to recruit public high schools in Texas to participate in this study, as well as to collect extant student data records from TEA.

This study will address the following research questions:

Confirmatory Research Questions

1. Does providing parents/guardians of students entering grade 9 after the enactment of House Bill 5 with information describing changes to the Texas high school graduation requirements, with a specific emphasis on changes to the mathematics requirements and how these changes may impact college access, have an impact on:

   1. The percentage of students in a school who enroll in algebra II by the end of their junior year?

   2. The percentage of students in a school who complete algebra II by the end of their junior year?

Exploratory Research Questions

2. Are there differential impacts of providing information describing changes to the Texas high school graduation requirements, with a specific emphasis on changes to the mathematics requirements and how these changes may impact college access, on students’ algebra II enrollment for:

   1. Schools with a high percentage of minority students?

   2. Schools with a high percentage of low-income students?

   3. Schools with a high percentage of low-achieving students?

3. Are there differential impacts of providing information describing changes to the Texas high school graduation requirements, with a specific emphasis on changes to the mathematics requirements and how these changes may impact college access, on students’ algebra II completion for:

   1. Schools with a high percentage of minority students?

   2. Schools with a high percentage of low-income students?

   3. Schools with a high percentage of low-achieving students?
      
      
      

4. To what extent are schools implementing the intervention with fidelity?

Data collection for this project consists of recruitment activities (e.g., telephone calls, email messages) and collection of extant data files from TEA. Specifically, in this OMB clearance package, ED is requesting clearance for the following data collection approach:

• Recruitment activities, including telephone calls and email messages, targeted at public school districts and public high schools in Texas

• Verification of dissemination of study materials to parents

• Extant data collection consisting of student records data to be obtained from the Texas Education Agency

ED believes that the data collections for which clearance is being requested represent the bare minimum necessary to determine if providing parents/guardians, prior to students’ selection of their courses, with information describing changes to the Texas high school graduation requirements, with a specific emphasis on changes to the mathematics requirements and how these changes may impact college access, has an impact on the percentage of students who enroll in and complete algebra II by the end of their junior year.

1. Respondent Universe and Sampling Methods

Respondent Universe

In this study, ED’s contractor, REL Southwest will investigate differences in algebra II enrollment and completion by the end of the junior year between students in study schools who receive information describing changes to the Texas high school graduation requirements, with a specific emphasis on changes to the mathematics requirements and how these changes may impact college access, (treatment group) and students in study schools who receive information describing changes to the Texas graduation requirements and providing parents with links to additional resources describing these changes included on state and regional websites (control group). To do so, ED’s contractor will follow the grade 9 students who begin high school in study schools during the 2014–15 school year—the first cohort of students required to graduate under the state’s new diploma plans—and assess differences in algebra II enrollment and completion by the end of junior year between students in treatment schools and students in control schools. All grade 9 students, including special education and English learners who enroll in either algebra I or geometry will be included in the study. All public high schools in the state of Texas that are not automatically placing their incoming grade 9 students into the Distinguished graduation plan² will be recruited to participate in the study.

Power

For this study, ED’s contractor wishes to be able to detect at least a 5 percentage point difference in algebra II completion between our treatment and control groups—it is not believed that a difference smaller than this would be meaningful. We used Optimal Design version 3.01 (Spybrook, Bloom, Congdon, Hill, Liu, Martinez & Raudenbush, 2013) to determine the number of schools needed for our study. Within the software package, ED’s contractor estimated a two-level cluster RCT design with binary outcomes³, with the following assumptions:

Probability of success in treatment group: 0.70

Probability of success in control group: 0.65

Number of students in each school: 170 (median number of students in grade 9 in Texas public high schools)

Plausible value range: 0.60—0.90

Power: 0.80

Thus, for this study, ED’s contractor determined that it will need to include a total of 144 schools in the RCT, with 50 percent of the schools allocated to the treatment and control groups, to be able to detect a 5 percentage point difference between treatment and control schools⁴.

Random Assignment

Once schools have been consented into the study, ED’s contractor will use cluster analysis to assign schools to strata for randomization. Cluster analysis allows for creation clusters of schools that are similar across multiple variables—rather than just one or two—which should increase precision. Conducting random assignment within strata reduces the likelihood of obtaining treatment and control groups that differ significantly on covariates of interest. For this study, we will use k-means clustering, which creates k strata and then assigns each unit to one stratum, so that a measure of similarity is maximized. Since the covariates are both categorical and continuous, how close the units are to one another will be determined using the general similarity measure (distance) outlined in Glower (1971). To determine the optimal number of strata, ED’s contractor will generate and evaluate results for up to 20 strata and then plot the between-cluster variability in an elbow plot, which is similar to a scree plot in principal components analysis. ED’s contractor will use this plot to determine where the rate of change in the proportion of variance accounted for by each additional strata slows. There are 6 school-level covariates that are likely to predict whether or not students will complete algebra II. These covariates, shown in Table 1, will be used in the cluster analysis to define strata.

Table 1. Covariates used for cluster analysis

Percent Black, White, Hispanic, Other	School size	Percent special education
Average incoming test scores in math and reading (that is, grade 8)	School region (such as suburban, rural, urban)	Percent English language learners

After the schools have been placed into strata, one-half of the high schools selected from each strata will be randomly assigned to serve in the treatment group, while the other half of the schools within each strata will serve as the control group for the impact study.

2. Description of Procedures for the Collection of Information

ED’s contractor will recruit schools, manage data collection and ensure quality and timeliness. ED’s contractor proposes to collect secondary data only. This study will utilize extant district, school and student-level data collected and archived by TEA. OMB clearance is being sought for extant data collection, as well as study recruitment activities.

Data Sources

To answer research questions 1 through 3, ED’s contractor will utilize longitudinal student- and school-level data sets available from TEA. Specifically, the study will use data from TEA’s Public Education Information Management System (PEIMS), statewide assessment files, and Texas Academic Performance Report (TAPR) files. PEIMS contains student-level data on student enrollment and demographic characteristics, special program participation, and course completion data. TAPR contains organizational data for schools and districts. To answer research question 4, ED’s contractor will use the envelopes mailed to the contractor by participating high schools. REL Southwest will also utilize data from a survey administered to all districts in Texas to describe the how study districts are disseminating information about the new graduation requirements to parents and students. The survey achieved a response rate of over 80 percent.

3. Description of Procedures for Maximizing Response Rates

ED is committed to obtaining complete data for this evaluation. This impact study relies on administrative data. ED’s contractor anticipates a 95-percent response rate from TEA on student variables in the administrative data. That is, ED’s contractor assumes 5 percent or less of the data will be missing on key variables. ED’s contractor has a strong working relationship with representatives from TEA. All administrative data files will be reviewed for consistency and completeness. The response rate from schools regarding the returned envelopes will be, by definition, 100 percent.

Based on its prior experience with recruiting schools to participate in RCTs, ED’s contractor expects to have minimal difficulty recruiting high schools to participate in the study. ED’s contractor will utilize a two-part process to make schools aware of the project, invite them to participate, and answer questions that schools might have about the study.

Recruitment will be conducted via email and telephone. Around the third week of October, ED’s contractor will send an email message to all high school principals in the state introducing the study and inviting them to participate. The emails will outline the goals of the study, the content of the informational materials, and the school’s role in information dissemination if randomized into the treatment condition. Principals will be made aware that this study is a randomized controlled trial (RCT) and that only half of participating schools will receive the treatment informational materials initially, while the control schools will receive information describing changes to the Texas graduation requirements and providing parents with links to additional resources describing these changes included on state and regional websites; however, they will also be informed that their schools will receive the materials, if desired, after the conclusion of the study—the 2017-18 school year. Interested principals will be directed to contact REL Southwest for additional information. A follow-up email will be sent to all non-responding principals one week after the initial email, with a second follow-up e-mail sent to all non-responding principals one week after the first follow-up. One week after distribution of the second follow-up email, if enough schools have not agree to participate in the study, REL Southwest will begin conducting telephone recruiting calls. REL Southwest anticipates that it will be able to recruit about 80 high schools via email. REL Southwest also anticipates needing to contact approximately 150 high school principals via telephone in order to recruit an additional 64 high schools. To recruit additional high schools, they will begin by contacting other high schools within the same districts as those that have already agreed to participate. If this does not result in a sufficient number of high schools, random numbers will be assigned to each of the high schools in Texas that have not responded to the recruitment emails, and the schools will be sorted in chronological order by these random numbers. Starting with the smallest number, high schools will be contacted via telephone and asked to participate in the study.

During the recruiting process, REL Southwest researchers will ensure that principals understand the nature of the study and the responsibilities of participating schools. Once principals have agreed to participate in the study, they will be asked to sign a memorandum of understanding/consent form (Attachment B-1) indicating that their school agrees to participate in the study.

While ED’s contractor does not intend to provide principals or schools with incentives to participate in the study, the cost of materials and material dissemination will be reimbursed. As part of the recruiting process principals will be informed that they will be provided with all of the materials necessary to distribute the informational brochures. In January, principals will be asked to provide ED’s contractor with the number of current students who entered grade 9 during the 2014–15 school year. Principals will be provided with enough study materials for that number of students, plus a few extra. That is, ED’s contractor will send each participating school a box of postage-paid envelopes containing the informational brochures, and we will ask principals to have their administrative staff create mailing labels, affix the labels to the envelopes, and have the envelopes picked up during the regular mail service. Schools will not need to supply any resources other than staff time.

Along with these materials, schools will be provided with one postage-paid envelope addressed to ED’s contractor. The envelope will contain the name of the school from which it was sent, and schools will be asked to mail this envelope at the same time as the informational materials. ED’s contractor will use these envelopes to determine which high schools have mailed the informational materials and when they were sent. If schools have not mailed the informational materials within one week of the mailing deadline, ED’s contractor will conduct reminder telephone calls to high school principals. Schools will be reimbursed up to $150.

4. Description of Tests, Procedures, and Methods

Data analyses for this study will consist of a confirmatory analysis examining the impact of providing parents/guardians with information describing changes to the Texas high school graduation requirements, with a specific emphasis on changes to the mathematics requirements and how these changes may impact college access, on algebra II course enrollment and completion by the end of junior year and a set of exploratory analyses investigating differences in the effect of providing this information on students’ mathematics course completion for various subgroups.

To answer our confirmatory research question, we will utilize multilevel models to compare algebra II enrollment and completion rates by the end of junior year for students in schools that received information describing changes to the Texas high school graduation requirements, with a specific emphasis on changes to the mathematics requirements and how these changes may impact college access, and students in schools that received information describing changes to the Texas graduation requirements and providing parents with links to additional resources describing these changes included on state and regional websites. To answer our exploratory research questions, we will include interaction terms into the model used to analyze our confirmatory research question in order to look for differential impacts for schools with high minority populations and high low-income populations. We will also assess whether there are differential impacts in schools with a higher percentage of low-achieving students.

Although the specific email messages and telephone scripts that will be used in this study have not been pilot tested, they are modeled email messages and telephone scripts that ED’s contractor has used for recruitment in other similar randomized controlled trials. Only minor changes have been made to reflect the nuances of this study. The time burden estimate used here is based on experiences from these similar studies.

The treatment group informational materials to be used in the study were pilot tested with parents of students who were enrolled in grade 9 during the 2013-14 school year—these parents and students cannot be included in this study. Several different versions of the brochure were pilot tested with nine parents. These parents were asked to review the brochure. Then, they were asked to identify which brochure was the easiest to read, easiest to understand, most eye catching, and most informative. We used feedback from these parents to create the final version of the brochure.

5. Individuals Consulted on Statistical Aspects of the Design and Key Staff

The following individuals were consulted on the statistical, data collection, and analytic aspects of this study through REL Southwest’s Technical Working Group (TWG):

Dan Goldhaber, Ph.D.

Director, CALDER (National Center for Analysis of Longitudinal Data in Education Research)

Vice President, American Institutes for Research (AIR)

Director, Center for Education Data & Research (CEDR), University of Washington Bothell

Co-Editor, Education Finance and Policy

3876 Bridge Way N,Suite 201

Seattle, WA 98103

Ph: 206-547-1562

Fax: 206-547-1641

E-mail: [email protected]

Geoffrey Borman, Ph.D.

Professor of Education, University of Wisconsin—Madison

Deputy Director of the University of Wisconsin's Predoctoral Interdisciplinary Research Training Program

Senior Researcher, Consortium for Policy Research in Education.

348 Education Building
1000 Bascom Mall
Madison, WI 53706-1326

Ph: 608-263-3688

Fax: 608-265-3135

E-mail: [email protected]

Johannes M. (Hans) Bos, Ph.D.

Vice President and Program Director, International Development, Evaluation, and Research (IDER) Program

American Institutes for Research

2800 Campus Drive, Suite 200
San Mateo, CA 94403

Ph: 650-843-8100

Fax: 650-843-8200

E-mail: [email protected]

W. Steven Barnett, Ph.D.

Board of Governors Professor and Director of the National Institute for Early Education Research

Rutgers University

73 Easton Avenue
New Brunswick, NJ 08901

Ph: 848-932-4350 x23132

Fax: 732-932-4360

E-mail: [email protected]

Attachment B-1. School Agreement to Participate Form

See attachment

1 The STEM endorsement included in the Foundation plus Endorsement Plan also requires completion of algebra II.

2 The Distinguished graduation plan requires students to complete algebra II in order to earn a Distinguished Level of Achievement.

3 The Optimal Design software does not allow for inclusion of estimates of the amount of variation taken into account by either level-1 or level-2 variables. As such, there is no ICC estimate, and the power for this study is likely an underestimate.

4 A difference of less than 5 percentage points between treatment and control groups is not meaningful.

3