Part A
Supporting Justification Request for
OMB Clearance of Information Collection Forms for:
The Effects of a Hybrid Secondary School Course in Algebra I
On Teaching Practices, Classroom Quality and Adolescent Learning
January 11, 2008
Submitted to: Submitted by:
U.S. Department of Education REL Appalachia / CNAC
Institute of Education Sciences 4825 Mark Center Drive
555 New Jersey Ave., NW, Rm. 506 Alexandria, VA 22311
Washington, DC 20208 703-824-2828
202-219-1597
Project Officer: Project Director:
Sandra Garcia Linda Cavalluzzo
CONTENTS
Justification of Data Collection Items 27
60-day and 30-day Federal Register Notices 30
The Effects of a Hybrid Secondary School Course in Algebra I
on Teaching Practices, Classroom Quality and Adolescent Learning
This submission is a request for approval of a data collection plan for an evaluation of an Algebra I intervention the applicant proposes to implement in 60 high needs high schools across the Commonwealth of Kentucky. The project is sponsored by the Institute of Education Sciences within the US Department of Education and will be conducted by the Regional Educational Laboratory Appalachia (REL-A) administered by The CNA Corporation (CNAC).
This intervention involves implementation of a hybrid instructional approach in all high school Algebra I classrooms in intervention schools. The hybrid approach combines online instruction with face-to-face classroom instruction for students. Intervention teachers will be supported by extensive and ongoing professional development focused on implementing effective hybrid classes and on research-based instructional practices for Algebra I.
Algebra I has emerged in recent years as a critical gatekeeper course, necessary to prepare students for the rigorous mathematics curriculum required for high school graduation and successful post-secondary experiences. Therefore, providing Algebra I teachers with the very best resources and professional development to ensure effective instruction has become a priority in Kentucky and across the nation. This research study is designed to evaluate, through a randomized control trial, experimental design, an approach that combines online and technology enhanced instruction with face-to-face classroom instruction to address this need. This hybrid or “blended” approach has shown promising results in Kentucky and in research elsewhere.
CNAC, as the lead organization for the research study, has comprised a team supported by researchers at Education Innovations (EI) and the University of Virginia (UVA). In addition, the Collaborative for Teaching and Learning (CTL) and Kentucky Virtual High School (KVHS) will implement the intervention and support recruitment efforts. Teachers receiving the intervention will apply the hybrid approach using the online course material in Algebra I selected by the Kentucky Department of Education (KDE) and KVHS for their online course in this subject. This courseware is an off-the-shelf product created by university faculty affiliated with the University of California College Prep Online, and the Center for Digital Innovation, UCLA and was chosen for its quality and content and because it is customizable, allowing educators to modify the content as needed. This last feature has particular appeal for going to scale if the intervention is shown to be effective, as the content can be tailored to meet the unique needs of different school systems. The courseware has gone through an external quality control protocol by the National Repository of Online Courses and has been reviewed by curriculum specialists at the KDE for quality and alignment with national and state standards for Algebra I instruction. The results on improved instructional practices, classroom quality, and student learning will be compared to those in control sites in which Algebra I instruction will continue as it has with normal classroom instruction.
Teachers will receive professional development through face-to-face training led by a master teacher in mathematics and distance training sessions supported by an online curriculum. Spotlight On Algebra I, developed by the Southern Region Education Board with funding from the AT&T Foundation, is the online courseware that provides the framework for professional development. Training and support for teachers will begin in the summer and continue throughout the intervention school year. Use of “Spotlight” is expected to improve teacher skills and instructional methods in Algebra I, and is expected to work synergistically with the hybrid curriculum for the following reasons: (1) the professional development experience provides models of instructional methods for teachers using online content; (2) it focuses on Algebra I, allowing teachers to focus on and improve their subject-specific instructional methods; and (3) it provides a vehicle (framework) for regular sustained activities and discussions for participating teachers (the community of learners). Further, coupling professional development in research-based instructional practices with hybrid instruction is expected to change the classroom environment, helping teachers break old habits of instruction and facilitating adoption of improved, research-based practices. Finally, the hybrid model provides tools to adopt more effective instructional approaches, including flexible instruction and formative assessments to help teachers meet the needs of different learners.
Research Hypotheses
This study poses five hypotheses focusing on the impact of the Algebra I intervention on teaching practices, classroom quality, and student learning.
The Hybrid Algebra I approach will:
Hypothesis 1: Increase the use of research-based best practices for Algebra I as documented in the Algebra I instructional standards.1
Hypothesis 2: Improve classroom quality as seen in increased levels of student interest and engagement and academically focused class time.
Hypothesis 3: Increase student achievement in learning Algebra I content and skills.
Additional hypotheses:
4: Achievement gains associated with the online curriculum may vary among students with different characteristics (gender, race/ethnicity, income as measured by free or reduced-price lunch status, LEP, LD/SPED, age).
5: The beneficial effects of the intervention on student outcomes will extend to the post-intervention year, as measured in terms of improved performance on the 10th grade PLAN assessment, mathematics course-taking, mathematics course grades, and improved high school continuation rates.
Methodology
To address the hypotheses in this evaluation, the research team will: (1) identify the universe of schools in Kentucky that meet the criteria for inclusion in the study; (2) recruit eligible schools; (3) randomly assign schools to treatment and control conditions; (4) collect administrative and survey data, conduct classroom observations, and administer a post-treatment assessment of Algebra I knowledge and skills; and (5) analyze the data and report the findings from our analyses. A summary of key activities appears in Table 1.
Identifying eligible schools. The study focuses on the impacts of hybrid instruction in high needs schools in which Algebra I is taught in the 9th grade. For the purposes of this study, we define “high needs” in terms poor performance in mathematics. In particular, schools that are considered to be poor performers in mathematics are identified as those in which fewer than 65 percent or students are proficient in mathematics, as indicated by either the nationally norm-referenced CTBS/5 exam in mathematics for 9th graders, or by the criterion-referenced Kentucky exam (the KCCT) for 8th or 11th graders that share the school with the 9th graders. These criteria result in a list of 210 eligible schools that will be contacted to determine interest in participating in the study.
Recruiting eligible schools. Based on our power analysis (described below), we established a target sample size of 60 schools.
The full initial sample (that is, prior to attrition or teachers’ non-response) of 60 participating schools will include approximately 120 Algebra I teachers and about 13,500 students who are enrolled in 9th grade Algebra I classes. We will ask district superintendents and building administrators to sign letters of commitment agreeing to support the intervention and data collection activities. Copies of these documents can be found in Exhibit A.
Randomly assigning schools. The study plan proposes to randomly assign schools, rather than teachers, to treatment or control conditions for practical reasons, as well as for reasons related to scientific rigor. In particular, we are able to minimize the potential for spillover effects between treatment and control teachers if teachers in the two conditions are physically separated. Further, we are better able to protect the sample from bias that could be introduced after randomization, if students within schools do not have an option to move between treatment and control conditions. From a practical standpoint, we also found that among six school principals we contacted, all preferred to have teachers within their school in the same treatment condition.
Collecting data. From school or district administrative records, we will collect information on student enrollments in Algebra I, demographic attributes, and mathematics test scores in the year prior to the intervention as well as student outcomes, including mathematics assessments and school dropout information in the year following the intervention (10th grade for most students). A brief one-time survey will be given to teachers in treatment and control groups in the spring of the implementation year. Finally, researchers will conduct classroom observations during one-day visits to each school during the spring of the intervention year.
Table 1. Schedule of Activities
Activity
|
Schedule |
Create District and School Pool for Site Selection Recruitment |
Spring - Fall 2007 |
District and School Recruitment (pending OMB approval) |
Winter-Spring 2008 |
District IRB
|
Spring - Fall 2008 |
District and School MOUs
|
Spring 2008 |
Random Assignment
|
Spring 2008 |
|
|
Start Intervention
|
Spring 2008 (Teachers) |
Start Collection of administrative data
|
Fall 2008 |
Collect Classroom observations, Teacher Survey and Algebra I Post-test |
Spring 2009 |
Collect administrative data on longer-term outcomes |
Winter 2010 |
Final Report of Findings |
Fall 2010 |
Analyzing data and reporting results. The evaluation will compare outcomes in the treatment and control groups to estimate the impact of the intervention on indicators of instructional practices, classroom quality and students’ academic outcomes. Following careful review of the analysis and findings by internal reviewers for REL-A and external evaluators for IES, the research team will make necessary revisions for re-review by the external evaluators. The approved study will be posted on the IES website for wide distribution, and disseminated to target audiences within Kentucky, the Appalachian region, and the nation in fall 2010.
The Regional Educational Laboratories are authorized under the Education Sciences Reform Act of 2002 (Pub. L. 107-279) Part D., Section 174 (20 U.S.C. 9564), and are administered by the Institute of Education Sciences’ National Center for Education Evaluation and Regional Assistance. A copy of this legislation is found at the following website: http://www.ed.gov/policy/rschstat/leg/PL107-279.pdf. The Regional Educational Laboratory Appalachia (REL-A) is one of ten regional labs funded by the U.S. Department of Education, (Number: ED-06-CO-0021). The REL is required to carry out randomized control trials or rigorous studies that examine the effects of proposed policies, programs, or practices on academic achievement that are related to the high-priority needs of the region. High quality curriculum in a core subject such as math, professional development, and education technology approaches are listed among the examples given for topical areas of research. Further, REL-A is mandated to follow IES standards for scientifically based research, as described in the IES authorizing legislation. In summary, we will abide by the standards outlined to: (i) apply rigorous, systematic, and objective methodology to obtain reliable and valid knowledge relevant to education activities and programs and (ii) present findings and make claims that are appropriate to, and supported by, the methods that have been employed. As is appropriate to our research being conducted, the term includes: (i) employing systematic, empirical methods that draw on observation or experiment; (ii) involving data analyses that are adequate to support the general findings; (iii) relying on measurements or observational methods that provide reliable data; (iv) making claims of causal relationships only in random assignment experiments or other designs (to the extent such designs substantially eliminate plausible competing explanations for the obtained results); (v) ensuring that studies and methods are presented in sufficient detail and clarity to allow for replication or, at a minimum, to offer the opportunity to build systematically on the findings of the research; (vi) obtaining acceptance by a peer-reviewed journal or approval by a panel of independent experts through a comparably rigorous, objective, and scientific review; and (vii) using research designs and methods appropriate to the research question posed.
Regional Need
The hybrid approach we propose to evaluate offers significant promise to Kentucky, and more generally to the Appalachian region, and to school systems nationally. In particular, the 2005 Regional Advisory Committee for Appalachia listed the improvement of teacher quality and the identification of evidence-based curricula/programs as two of the top five areas of need in the region. (Appalachia Regional Advisory Committee (RAC) Report to the U.S. Dept. of Education, 2005, The CNA Corporation) While teaching quality was one of the seven identified topical concerns of every region, (Synthesis RAC Report to the U.S. Dept. of Education, 2005, The CNA Corporation), the challenge is particularly daunting in rural areas, like those in Kentucky and elsewhere in the Appalachian region. In particular, survey data reveal that rural areas have difficulty attracting teachers from outside the local area because those areas are often economically depressed, and lacking the cultural and social activities to which teaching candidates from more urbanized areas are accustomed (McClure, Redfield, & Hammer, 2003). Thus, low performing rural schools are left with a limited supply of teachers, many of whom received much of their own education in the same low performing schools. The RAC report for Appalachia also indicated that a focus on mathematics instruction and the use of technology were important areas of need in the region.
Discussions between REL Appalachia researchers and education leaders at the Kentucky Department of Education corroborated these needs, emphasizing the role that widespread use of hybrid instruction could play in resolving them.
Conceptual Basis for the Intervention
The hybrid approach immediately brings to Kentucky students a curriculum that has been reviewed and selected by Kentucky Department of Education curriculum specialists and is fully aligned with state and national standards for Algebra I. In addition, the subject matter is presented in a variety of ways, helping to deepen understanding of Algebraic rules and meet the needs of students who collectively have multiple styles of learning. Endowed with a deeper understanding of Algebra, students should be more able to retain what they have come to understand (rather than memorized) for longer periods and be more able to construct accurate solutions to algebraic problems than students without this depth of knowledge. In addition to the immediate benefit to students offered by the online curriculum, teachers who use the curriculum in their classrooms are exposed to a range of ways to present mathematics to adolescent students. Moreover, the sustained professional development dimension of the intervention helps ensure that teachers take away improved pedagogical skills and an increased conceptual understanding of Algebra I content. Further, by changing the context for instruction from traditional direct instruction to a hybrid approach focusing more on student-centered activities, the hybrid classroom is expected to facilitate adoption of research-based instructional practices.
As stated in the study overview, the professional development (PD) program is thorough and is expected to work well with the hybrid curriculum. During the summer, intervention teachers will attend a one-day orientation where they will be introduced to the online software for Algebra I instruction, Spotlight on Algebra for PD, and Blackboard, an electronic platform for management of online courses. A two-day face-to-face training will follow the orientation and will lead teachers through four Spotlight sessions, as well as Horizon Wimba, a communication system used by KDE for online conferencing. Over the next five weeks, teachers will conduct activities and meet online for discussion guided by a master teacher. During the intervention school year, weekly distance sessions will be held to discuss classroom practices. These are formal professional development sessions. They are designed to deepen understanding and encourage application of pedagogical practices that were introduced during the summer sessions. Each session has a topic area or focus and may require a short reading or introduce an instructional tool for classroom use. The building of collaborative professional relationships among participating teachers is also encouraged through these sessions. Teachers discuss challenges and share solutions to problems they face, or things they learned as they use the hybrid approach in their classrooms. Teachers will be asked to attend these sessions at least once per month throughout the year. At the end of the school year, formal PD will end. This PD program is consistent with a substantial empirical literature that characterizes key structural features of effective professional development programs. This research indicates that effective professional development programs are school-based and job-embedded, continuous and ongoing, content focused, organized around groups of teachers, designed around active learning, and coherent, in the sense that the programs align with key aspects of the educational system, such as school and district goals and student content and performance standards. (Joyce, Bruce & Showers, Beverly, 1988; Joyce, Bruce & Calhoun, Emily, 1996; Fullen, Michael, 2001; Loucks-Horsley, Susan, Hewson, Peter, Love, Nancy, & Stiles, Katherine, 1998; Elmore, Richard & Burney, Deanna, 1999; Supovitz, Jonathan & Turner, Herbert, 2000; Supovitz, Jonathan, Mayer, Daniel, & Kahle, Jane, 2000, Garet, Porter, Desimone, Birman and Yoon, 2001; Desimone, Porter, Garet, Yoon, & Birman, 2002.)
Role for Rigorous Research
While the potential for hybrid instruction appears to be substantial, growth in use of the approach is outstripping the development of rigorous research that can quantify its effectiveness, and identify contexts for its successful use. For example, a recently released survey of online and blended learning found that one-third of reporting school districts had at least some students taking blended/hybrid courses in school year 2005-2006; two thirds of districts expected enrollments to grow in these types of courses (Picciano and Seaman, 2007, p 8). And, while there is evidence of efficacy of the hybrid approach, to date, there has been no rigorous study of the causal effect of the hybrid instructional model, with extensive professional development, as will be provided in this intervention. A recently released study using a RCT approach to evaluating fifteen different educational software products for reading and mathematics targeted to different grade levels found no statistically significant results for the set of products overall. However, the products were diverse in their subject range and implementation practices, potentially masking individual positive findings. In addition, product providers offered implementation supports, but not the extensive subject-specific professional development that characterizes the current intervention. (Dynarski et al., 2007)
This application is for a new collection of information to support the proposed REL-A Hybrid Algebra I study. Findings from the evaluation will inform IES, regarding the impact of the Hybrid Algebra I intervention on instructional practices, classroom quality, and Algebra I learning among adolescents in predominantly rural schools across the state of Kentucky. In addition, the study will add to a sparse literature, rigorously derived evidence of the impact of a relatively new and rapidly growing instructional approach. This information is expected to be of substantial value to the Kentucky Department of Education (KDE), which is considering expansion of hybrid instructional methods in its public schools, as well as other states, districts and schools across the nation. After appropriate review, results will be distributed to state and local education agencies and schools, and more broadly through posting on the IES website for access by the general public.
The study will involve four primary types of data collection: (1) collection of administrative data; (2) direct classroom observations; (3) brief one-time surveys of teachers; and (4) assessment of student Algebra I knowledge and skills. Because the research team will conduct classroom observations, they do not present a data collection burden to study participants. Similarly, the post-test of student knowledge does not create burden requiring OMB review. To help reviewers understand the research design and full scope of the analysis, we explain their role in the discussion below. Also attached in separate files are the treatment and control teacher questionnaires. Appendix A shows how specific items in the teacher questionnaires support our research design.
Data collection details are presented below in association with each research hypothesis.
Hypothesis 1: The Hybrid Algebra I approach will increase the use of research-based best practices for Algebra I as documented in the Algebra I instructional standards.2
As seen in Table 2, Hypothesis 1 will be addressed through analysis of data from four sources. Education Innovations will employ and train external researchers to conduct direct classroom observations, and administer teacher surveys.
Table 2. Hypothesis 1: How, by Whom, and for What Purpose Data are Collected
Data Source and Purpose |
How and By Whom Data are Collected |
Classroom observation tools
Questionnaire
|
Observations of full (approximately 1 hour) Algebra I classes will be conducted in up to 5 classrooms during one 1-day visit to each Treatment and Control school. The 1) SOM and 2) AQA will be used by researchers to record information during the classroom observations. Education Innovations will conduct classroom observations under direction of Dr. Deborah Lowther.
3) Teacher surveys will be collected from teachers by external researchers on the day of their school visit
|
Hypothesis 2: Improve classroom quality as seen in increased levels of student interest and engagement and academically focused class time.
Hypothesis 2 will be addressed through analysis of data from two sources: The SOM classroom observation instrument and the teacher surveys. Please see Table 3.
Classroom observations will be scheduled so as not to conflict with student test days. Exhibit C shows a sample scheduling form that will be filled out by teachers and used by observers for their classroom visits.
Table 3. Hypothesis 2: How, by Whom, and for What Purpose Data are Collected
Data Source and Purpose |
How and By whom Data are Collected |
|
Same as Table 2. |
Hypothesis 3: Increase student achievement in learning Algebra I content and skills.
As seen in Table 4, an Algebra I Post-test will be administered by researchers to assess knowledge in May of the intervention school year. To minimize the need for additional testing, test scores from an assessment of mathematics knowledge (the KCCT) given to all 8th graders in Kentucky, will be gathered by researchers from district or school administrative records and used as a covariate in the statistical analyses to control for prior performance in mathematics. Enrollment records for Algebra I courses will be used to control for the amount of student exposure to the intervention in the analysis.
Table 4. Hypothesis 3: How, by Whom, and for What Purpose Data are Collected
Data Source and Purpose |
How and By whom Data are Collected |
1) 8th grade KCCT Math Scores
2) Algebra I Post-test scores
3) Student enrollment data |
1) Administrative records of student 8th grade KCCT math scores will be gathered by researchers from districts and used as a covariate, to improve the accuracy of the estimated impact of the intervention, in a statistical analysis of program impact on student knowledge of Algebra I. 2) Trained external proctors will conduct onsite administration of the Algebra I Post-test. 3) Researchers will collect student enrollment data from schools at the end of each marking period during the intervention year. These data will be used to control for the amount of student exposure to the intervention in the analysis. |
Additional hypotheses:
Hypothesis 4: Achievement gains associated with the online curriculum may vary among students with different characteristics (gender, race/ethnicity, income as measured by free or reduced-price lunch status, LEP, LD/SPED, age).
This hypothesis will be addressed with a comprehensive 2-level HLM analysis using student test scores from the 8th grade KCCT math test, the Algebra I assessment, and the listed learner characteristics (Table 5).
Table 5. Hypothesis 4: How, by Whom, and for What Purpose Data are Collected
Data Source and Purpose |
How and By whom Data are Collected |
1) 8th grade KCCT Math Scores
2) Algebra I Post-test scores
3) Student enrollment data
4) Administrative records of student-level variables for gender, race/ethnicity, and income as measured by free or reduced-price lunch status, LEP, LD/SPED, and age.
|
1) Same as Table 4 above
2) Same as Table 4 above
3) Same as Table 4 above
4)) Same as Table 4 above
|
Hypothesis 5: The beneficial effects of the intervention on student outcomes will extend to the post-intervention year, as measured in terms of improved performance on the 10th grade PLAN assessment, mathematics course-taking, mathematics course grades, and improved high school continuation rates in the post-intervention school year.
Data from the 10th grade PLAN exam, mathematics course grades, and information regarding high school continuation rates and enrollment in math courses will be used in a HLM analyses to quantify sustained treatment effects of the Hybrid Algebra I approach (Table 6).
Table 6. Hypothesis 5: How, by Whom, and for What Purpose Data are Collected
Administrative data stored in electronic databases will be used wherever possible to reduce the burden to respondents. In particular, benchmark (8th grade) assessments, 10th grade follow-up indicators, student enrollment data, and student demographic data will be collected by researchers from KDE, districts, or schools as appropriate, to build the required data set for the statistical analyses of the effects of the intervention on student outcomes.
In order to maximize the return rate to questionnaires, teacher surveys will be administered as paper-based documents and collected during researcher visits to the schools. Specifically, teacher surveys will be distributed by mail to each treatment and control teacher prior to the onsite visit at his/her school. The observers will collect the completed surveys from each teacher as his/her class is observed.
The proposed study uses data that are available in administrative databases, as well as unique data that are not available from previous studies, from the Kentucky Department of Education (KDE), or from the participating school districts. The unique data are required for execution of the RCT study. These data include information to be collected by researchers during direct classroom observations of Treatment and Control classes, as well as Treatment and Control teacher about the instructional practices and classroom quality in the Algebra I classrooms. The research team examined KDE documents on the state’s assessment program and discussed with KDE assessment experts, what assessments would be available in order to avoid imposing additional assessments on students in the sample. We learned that Kentucky has suspended use of a 9th grade norm-referenced assessment of mathematics skills and knowledge that had been in place for several years. In addition, the implementation of a new statewide end-of-grade exam in mathematics for 9th graders has been postponed. As a result, it is necessary for us to administer an Algebra I assessment (post-test) to participating students. Our plan is to use the Educational Testing Service assessment for Algebra I.
The primary entities for this study are schools, their associated school districts, and all Algebra I teachers and students at the participating schools. All data collection procedures have been designed and scheduled to minimize burden on the schools, districts, teachers, and students. Specifically, all surveys are based on valid and reliable instruments and have been field tested with 9 teachers for item clarity and completion time. The field test results revealed that the control teacher survey took an average of 8 minutes to complete, and the treatment teacher survey required approximately 9 minutes. In addition, teachers complete only one survey. Classroom observations involve only one day at each school and will be scheduled to avoid conflict with testing schedules. Trained proctors will administer the Algebra I post-test, which releases teachers from this burden. Requests for school district data will clearly delineate the data that are being requested, and will provide a contact person from the research team who can answer questions, will be respectful of each district’s unique context, and will reinforce the confidentiality measures that will be enforced when handling their data. The electronic data will be accepted in a layout that is easiest for each district to provide.
This is a one-time data collection. But consequences of not conducting the collection are substantial. Large numbers of high school students are now taking one or more courses online. Such courses support a variety of needs, including credit recovery and advancement, and bringing elective and core courses to schools that would otherwise not be able to offer them. In addition, use of hybrid courses---courses that use online curriculum to support learning in traditional classrooms guided by a qualified classroom teacher---is expanding rapidly. This latter use is seen as a way to individualize instruction to meet the unique learning needs of different students and to enhance the pedagogical skills and practices of traditional teachers, and therefore, student learning.
In Kentucky, KVHS furnished hybrid courses in a total of 26 classrooms, reaching a combined total of over 500 students in SYs 2005-2006 and 2006-2007. Without the ability to conduct a randomized study, information would not be available to assess the effectiveness of this approach, and whether its effectiveness justifies expansion of the program. Findings from the study will inform decision makers regarding this growing classroom practice.
There are no special circumstances that require the collection to be conducted in a manner inconsistent with OMB guidelines.
Federal Register Announcement. A 60-day notice to solicit public comments will be published in the Federal Register by ED to allow public comment. A 30-day notice will follow. Drafts of these announcements are placed in Appendix B. We will have addressed the comments received during the 60-day announcement period and will insert any further documentation needed at that time.
Consultants Outside the Agency. Extensive consultation has been undertaken prior to formulating the proposed collection. Discussions have been held over a period of one year with administrators at KDE, including curriculum specialists, assessment specialists, KVHS online teachers, administrators and professional development providers, and Algebra I teachers from Kentucky secondary schools. In particular, the following people engaged in discussions and provided feedback and advice regarding this study:
Ann Bartosh, Math Consultant, Division of Curriculum Development, Kentucky Department of Education (KDE)
William Bush, Professor and Director, Center for Research in Mathematics and Science Teacher Development, University of Louisville
Ghenna Chernovski, KVHS teacher, former classroom teacher
Terri DeYong, Senior Consultant, KVHS, KDE
Linda France, Deputy Commissioner, KDE
Sarah L. Friedman, REL-A Director
Bob Hackworth, Senior Consultant, Kentucky Virtual High School (KVHS), KDE
Michael Hansen, Field Scientist, (REL Appalachia), Bureau of Learning & Results Services, KDE;
Jack Hunter, Spotlight on Algebra I Instructor
Donna Lynch, KVHS teacher, online course developer and classroom teacher
Ellen Mandinach, REL-A, Senior Researcher, The CNA Corporation
Linda Pittenger, Director, KVHS, KDE
Collectively, these individuals are intimately familiar with various aspects of the proposed study. For example, the KDE representatives provided information regarding data availability in KDE administrative records as well as the assessments that can be used to measure student learning. KVHS representatives reviewed the implementation plan and provided information regarding the intervention materials. In addition, the teacher survey instrument was pre-tested in Kentucky by nine teachers to ensure its clarity, ease of response and to measure the level of effort required to complete them.
To provide expert advice on the study, a Technical Working Group (TWG) of researchers with knowledge and experience in designing, conducting, and reporting results of Randomized Control Trials was formed. The TWG conducted detailed reviews of the study plan and supporting documents and provided feedback and recommendations in face-to-face meetings as well as provided support through email and phone conferences. The TWG members and affiliations are listed below:
Johannes M. Bos, President and CEO, Berkeley Policy Associates
Laura M. Desimone, Professor of Public Policy and Education, Vanderbilt University
Barbara Goodson, Senior Researcher, Abt Associates, NY, NY
Rebecca A. Maynard, University Trustee Chair, Professor of Education and Social Policy, University of Pennsylvania
Samuel
C. Stringfield,
Nystrand Center of
Excellence
in Education, University of Louisville
In addition, the original
study plan, and revisions to it, received careful review by an
external
evaluation team funded by IES and led by Michael Puma,
Chesapeake Research Associates, LLC.
Participating schools assigned to the treatment group will receive the Spotlight on Algebra I intervention, including the training of all participating teachers and follow-up support throughout the year. Treatment teachers will not receive payment for data collection activities. Control teachers will be offered dinner and reimbursement for local travel for participating in an evening orientation meeting where they will be given a study overview and description of their responsibilities as participants in the study. Alternative sessions will be offered for teachers unable to attend the initial session.
Assurance of confidentiality is provided in writing in all letters, brochures and other study documents. Exhibits A and B display recruiting materials that will be distributed to potential participants and information sheets that will be distributed at orientation sessions for participating teachers respectively.
The following ESRA pledge language is included on all information collections (Please see Exhibits D and E.)
Per the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, "Responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific district or individual. We will not provide information that identifies you or your district to anyone outside the study team, except as required by law.”
The following PRA statement is also included on all information collections (Exhibits D and E).
Paperwork Burden Statement: According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is xxx-xxx. The time required to complete this information collection is estimated to average 12 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, D.C. 20202-4700. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: Sandra Garcia, The Institute for Education Sciences), U.S.
The CNA Corporation, Education Innovations and all REL Appalachia partners follow the confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). The CNA Corporation and REL Appalachia will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released. Information from participating institutions and respondents will be presented at aggregate levels in reports. Information on respondents will be linked to their institution but not to any individually identifiable information. All institution-level, identifiable information will be kept in secured locations and identifiers will be destroyed as soon as they are no longer required. The CNA Corporation and REL Appalachia obtain signed NCEE Affidavits of Nondisclosure from all employees, subcontractors, and consultants that may have access to this data and submits them to our NCEE COR.
All members of the study team having access to study data have been certified by Western Institutional Review Board® (WIRB®) as having received training in the importance of confidentiality and data security. The following confidentiality language appears on all letters, brochures, and other study materials:
Responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific district, school or individual. We will not provide information that identifies you or your district to anyone outside the study team, except as required by law.
Both CNAC and EI execute and maintain a rigorous policy on Human Subjects Research, which will be the standard for this study. These policies result from CNAC and EI insistence upon full compliance with governing statues and regulations and from their commitment to safeguard the rights and welfare of human participants in all research with which they are associated. Both CNAC and EI have a designated Human Subjects Officer (HSO) to review research involving human subjects and determine whether that research needs to be assessed by a formal Institutional Review Board (IRB) before proceeding. CNAC and EI follow the instructions of the Institutional Review Board with regard to any additional disclosures or issues specifying the type of consent required from the research participant. Specifics regarding confidentiality of data for CNAC and EI are provided below.
The CNA Corporation: Confidentiality of Data
CNAC’s policy ensures strict confidentiality for data access and management. Our handling procedure for all projects that involve individually identifiable data is as follows:
For projects that require no exchange of data with external personnel, all sensitive data will be stored and utilized on servers that are segregated from the corporate Windows 2000 domain and all other domains on the corporate network.
Projects that require external exchange of sensitive data will use RADCON01 or other secure means of data exchange for this purpose. This server exists in a separate Windows 2000 domain (CNACCON) that was established to give non- CNAC employees the ability to exchange data securely with CNAC employees.
Access to servers that have sensitive data is granted through access to the domain by a separate user account (separate from the corporate network account) on an as-needed basis. Access may be further restricted to a particular server if necessary.
Servers that contain sensitive data must use warning banners to post security reminders/warnings.
A separate user account (from overall corporate network account) and password are required to access all servers that contain sensitive data.
Access control lists (ACLs) are used to restrict access to data. User access is restricted to the minimum necessary to perform the job. User rights are granted on an as-needed, need-to-know basis.
Project personnel can move only sanitized data, which have been stripped of personal identifiers, to user folders.
Any data that comes in on tape, floppy, CD Rom, or zip disk will be returned to the client as soon as possible. Only necessary copies of these data will be made, and these sensitive data will not be copied from the server to any removable media.
No sensitive data will be removed from the CNAC premises.
Access to data from remote sites from an authorized computer is permitted as long as the data is not copied to any media.
Typically, all data will be backed up using secure procedures. Individual projects can request that their data not be backed-up if it is deemed too sensitive in nature.
Any concerns regarding human subject research will be reported to the HSO, and he/she will determine the appropriate corrective actions. All complaints will be taken seriously and fully evaluated.
Education Innovations: Confidentiality of Data
EI has strict confidentiality procedures for research data received in two formats: physical and electronic. Once collected research data is handled and stored according to the most appropriate of the two methods below:
Physical Data – Physical data such as questionnaires, observation materials, documents collected as part of a document review, interview tapes from focus groups and interviews or any other data collected for the purposes of completing the research in question is treated carefully and confidentially at all times. Upon receipt of the physical data, it is logged in and immediately stored in a file folder associated with the project in question located in a private office that is locked each evening and is only accessible to project staff during the day. Upon completion of the project all related files are sealed and moved to an archive office located on site and then to a secure offsite archive a year later where it is kept until the data is a total of seven years old before then being destroyed. The data is always in possession of organizational staff and is never at any time shared with anyone, including the project sponsor, in its original form; it’s presented as a summary where all names and other identifying information has been removed.
Electronic Data – Electronic data exists as a result of collecting data directly from research subjects through online questionnaires or student achievement databases, or it exists as a summary of physical data collected during the course of research. In either case it is treated carefully and confidentially at all times. Upon receipt of the data it is immediately stored on a central file server accessible only to project staff and located behind a secure firewall that has all unessential ports blocked. Accessing the data requires domain name authentication of both the user and the computer and can only occur from within the offices located behind the firewall except for the rare occasion that project staff may be given access through the use of a secure Virtual Private Network connection also requiring user authentication. Computer use and access to the data is limited to regular office hours and all computers are located in offices that are secured each evening.
No private or sensitive information such as sexual behavior and attitudes or religious beliefs is requested in this collection.
The estimate of hour burden for data collection is derived from the estimate of time needed for review of informational materials about the study by members of the eligible universe and for treatment and control teachers to: participate in orientation meetings; assist research staff in scheduling classroom observations; and complete surveys. As seen in Table 7, the frequency of response for teacher participants will be one time during the spring of the intervention school year. The teacher survey takes approximately ten minutes to complete. As mentioned in the answer to question 5, the time estimates for the survey is based on field trials of the instrument conducted with nine Algebra I teachers from Kentucky schools.
In addition to the time it takes to complete the questionnaires, both treatment and control teachers will attend orientation meetings where their roles and responsibilities under the study will be reviewed. Forty minutes will be set-aside for treatment teachers at their first face-to-face training session for this orientation. During the session, they will (1) hear a presentation reviewing the study goals, research design, and their responsibilities as participants; (2) have an opportunity to ask questions; and (3) read and review an information sheet restating this information. Teachers in the control group will be invited to attend an evening orientation session where they will go through the same agenda. Meeting places for the orientation sessions for control teachers are planned for multiple locations and expected to be no more than 50 miles from the home of any participating teacher, or 25 miles from teachers' homes on average (40 minutes travel time). For control teachers, the round-trip travel time plus orientation creates a burden of 120 minutes per teacher.
In addition to time for orientation, we include burden for (1) all teachers, principals and superintendents in the eligible universe for the review of materials (brochure and commitment statement) related to the study. The estimated time for this activity is 15 minutes per person; (2) For the 100 teacher respondents that remain in the study throughout the intervention year, there will be a burden estimated at 10 minutes per teacher for scheduling of researcher visits for classroom observations and student testing.
Construction of the estimated burden to respondents shown in table 7 is based on the following assumptions regarding sample sizes: (1) 210 principals for the schools eligible for participation and their 166 district Superintendents will receive the study awareness materials and commitment statements, (see Exhibit A for a copy of these forms); (2) at two teachers per school, principals will forward the materials, brochure and teacher commitment statement, to their 420 teachers; (3) 60 participating schools with two teachers per school on average, randomly and evenly assigned to treatment (T) and control (C) groups will complete the orientation (Please see Exhibit B for a copy of the information sheet that will be distributed at the orientation meeting.); (4) an estimated response rate of 83 percent for teachers, yielding a sample of 100 teachers (50 treatment, 50 control) who will complete the teacher questionnaire and the scheduling form for classroom observations.
Table 7. Respondent Burden Estimates (numbers account for respondent attrition)
Activity |
Estimated Total By Group |
Average Completion Time in Hours (Minutes) |
Total Annual Burden in Hours (Minutes) |
Estimated Annual Costs* |
|
Group T=Treatment C=Control |
Est. # |
(Est. # x Avg. Hrs) 3 |
Annual hrs x:
$30.88/hr for teachers; $47.25/hr for principals; $70.00/hr for Supers |
||
Study Awareness Materials and Commitment Statements for eligible universe = 1 |
Superintendent |
166 |
.25 (15) |
13.83 (830) |
$968.33 |
Principal |
210 |
.25 (15) |
17.5 (1,050) |
$826.88 |
|
Teachers |
420 |
.25 (15) |
35 (2,100) |
$1,080.80 |
|
Subtotal |
|
796 |
|
66.33 hours |
$2,876.01 |
Subgroup Activities |
|
|
|
|
|
Teacher Orientation Meetings = 1 |
T |
60 |
.667 (40) |
13.34 (800) |
$411.94 |
C |
60 |
2.0 (120) |
40 (2,400) |
$1,235.20 |
|
Scheduling Form (Classroom Observations and Algebra I Post Test) = 1 |
All Teacher Respondents |
100 |
.167 (10) |
5.57 (333.33) |
$171.90 |
Algebra I Teacher Questionnaire (Hybrid/Treatment and Control)= 1 |
T |
50 |
.167 (10) |
2.78 (167) |
$ 85.85 |
C |
50 |
.167 (10) |
2.78 (167) |
$85.85 |
|
Subtotal 320 |
64.47 hours |
$1,990.74 |
|||
TOTAL 796 respondents 372 annual responses* *372= (796+320)/3 |
130.8 hours |
$4,866.75 |
|||
*2007 Statistical Abstract: The National Data Book, Table 240: Public Elementary and Secondary Schools-Number and Average Salary of Classroom Teachers, 1990 to 2004, and by State, 2004 and Table 241: Average salaries and wages paid in public school systems: 1985 – 2005. http://www.census.gov/compendia/statab/education/. Data for this analysis came from the following website. http://data.bls.gov/cgi-bin/surveymost
No additional costs are imposed on respondents or record-keepers. Data collectors are employed to obtain data from classroom observations.
Table 8 summarizes by year the total cost of the evaluation. Costs are disaggregated by evaluation costs, which include refinement of the evaluation plan, recruiting, data collection and analysis, reporting, and cost of implementing the intervention. The total average annual costs for the first four years of the study are estimated at $702,694 (That is, $2,810,775/4). Average annual costs associated with the first four years of the evaluation (that is, excluding intervention costs) is $601,495.
Table 8. Hybrid Study Budget Summary
|
Study Years |
||||
|
Year 1 |
Year 2 |
Year 3 |
Year 4 |
Total |
Evaluation costs |
$266,291 |
$883,629 |
$965,080 |
$290,981 |
$2,405,980 |
Intervention costs |
$57,808 |
$346,987 |
$0 |
$0 |
$404,795 |
Total |
$324,099 |
$1,230,616 |
$965,080 |
$290,981 |
$2,810,775 |
Not applicable since this is a new collection.
The data collected for the Hybrid study will be analyzed and tabulated to address each of the research hypotheses. Details of the data analysis to address each hypothesis are presented below and the schedule of activities can be found in Table 1 located in the Study Overview. Example tabulation tables are presented in Appendix C. The results will be disseminated in strict accord with the policies and permissions of the Regional Educational Laboratory Appalachia and the Institute of Education Sciences. A proposed publication schedule is presented subsequent to the analyses descriptions.
Hypothesis 1: The Hybrid Algebra I Approach/District Algebra I curriculum will increase the use of research-based best practices for Algebra I as documented in the Algebra I instructional standards.
Observation data (instruments not included in this package, because they create no burden) will be analyzed using independent t-tests in which between-group (hybrid vs. control) differences will be investigated for each of the strategies assessed. The teacher survey data will also be analyzed by means of independent t-tests. To protect against the increased likelihood of Type I error, the Bonferroni-Hochberg step-down procedure to adjust statistical significance will be used. Effect sizes will also be computed using Cohen’s d. Example tabulation tables are in Appendix C.
Hypothesis 2: Improve classroom quality as seen in increased levels of student interest and engagement and academically focused class time.
Hypothesis 2 will be addressed with data from two sources: the classroom observations and teacher surveys. The data analyses will be the same as those reported in relation to Hypothesis 1; however, the data to be analyzed will be limited to items directly related to Hypothesis 2. Data selected from the hybrid teacher survey (Exhibit D) and control teacher survey (Exhibit E) will include Item 5 - Use of the Hybrid Algebra I approach increases student interest and engagement and Item 11 – Use of the Hybrid Algebra I approach increases academically focused class time. (Please see example tabulation Table C.3 in Appendix C.)
Hypothesis 3: Increase student achievement in learning Algebra I content and skills.
As described in item Table 4, the KCCT exam in mathematics for 8th grade students will be used as a covariate in an analysis examining the impact of the intervention on student learning. A 2-level HLM will be used as a baseline model to assess the overall effect of the treatment while controlling for the school average performance on the pretest (school-level covariate) and length of exposure to treatment. Example tabulation tables for the student achievement results are presented in Appendix C.
Additional Hypotheses:
Hypothesis 4: Achievement gains associated with the online curriculum may vary among students with different characteristics (gender, race/ethnicity, income as measured by free or reduced-price lunch status, LEP, LD/SPED, age).
The Algebra I achievement analysis involves examining the results by the following subgroups: gender, race/ethnicity, eligibility for free or reduced-price lunch, Limited English proficiency, and Learning Disabled/Special Education services. This hypothesis will be addressed with a comprehensive 2-level Hierarchical Linear Modeling (HLM) using student 8th grade test scores from the KCCT mathematics assessment and the listed learner characteristics as covariates at the school and student level respectively.
Hypothesis 5: The beneficial effects of the intervention on student outcomes will extend to the post-intervention year, as measured in terms of improved performance on the 10th grade PLAN assessment, mathematics course-taking, mathematics course grades, and improved high school continuation rates.
Follow-up data from the PLAN, given to all students in Kentucky in the fall of 10th grade, will be used in a comprehensive 2-level HLM to identify any sustained treatment effects on student learning of the Hybrid Algebra I approach. Other indicators of longer-term effects that will be examined in the subsequent school year, mathematics course grades in the first and second marking periods, and continued high school enrollment.
Publication of Results
Throughout the study, there is strong monitoring and frequent reporting of the study’s progress. Included in this process are the submissions of monthly progress reports as well as updated annual plans in December of Years 1 2, and 3 that will document any changes in the research. At the end of the study, a final report of the findings will undergo an extensive peer-review process before dissemination occurs. Initially, REL-A researchers will review the report internally. Once suggested revisions are completed, the report will be sent to IES for external review of the study analyses and findings. The research team will make necessary revisions based on the IES review. Once the study is approved by IES, it will be posted on the IES website for wide distribution, and disseminated to target audiences within Kentucky, the Appalachian region, and the nation in winter 2010.
Not applicable
There are no certification exceptions identified with this information collection.
Desimone, L., Porter, A., Garet, M., Yoon, K., and Birman, B. (2002) Effects of Professional Development on Teacher's Instruction: Results from a Three-Year Longitudinal Study, Educational Evaluation and Policy Analysis, Vol 24, No 2, pp. 81-112.
Dynarksi, Mark, Roberto Godini, Sheila Heaviside, Timothy Novak, Nancy Carey, Larissa Campuzano, Barbara means, Robert Murphy, William Penuel, Hal Javitz, Deborah Emery, and Willow Sussex. Effectiveness of Reading and Mathematics Software Products: Findings From the First Student Cohort, Washington, D.C.: U.S. Department of Education, Institute of Education Sciences, 2007.
Elmore, R. & Burney, D. (1999). Investing in Teacher Learning: Staff Development and Instructional Improvement. In L. Darling-Hammond & G. Sykes (Eds.), Teaching as the Learning Profession: Handbook of Policy and Practice (San Francisco, CA: Jossey-Bass.
Fullen, M. (2001). The New Meaning of Educational Change. New York: Teachers College Press.
Garet, M., Porter, A., Desimone, L., Birman, B., and Yoon, K. (2001), What Makes Professional Development Effective? Results from a National Sample of Teachers, American Education Research Journal, Vol. 38 No. 4, pp. 915-945.
Joyce, B. & Calhoun, E. (1996). Learning Experiences in School Renewal: An Exploration of Five Successful Programs. Eugene, OR 97403-5207: Editor, ERIC Clearinghouse on Educational Management, 5207 University of Oregon.
Joyce, B. & Showers, B. (1988). Student Achievement Through Staff Development. White Plains, NY: Longman Press.
Loucks-Horsley, S., Hewson, P., Love, N., & Stiles, K. (1998). Designing Professional Development for Teachers of Science and Mathematics. Thousand Oaks, CA: Corwin Press.
McClure, C. T., Redfield, D., & Hammer, P. C. (December, 2003). Recruiting and Retaining High-Quality Teachers in Rural Areas, AEL Policy Brief. http://www.edvantia.org/publications/index1.cfm?§ion=publications&area=publications&id=482
Picciano, A., and Seaman, J. (2007) K-12 Online Learning, A Survey of U.S. School District Administrators, Sloan Consortium.
Supovitz, Jonathan, Mayer, Daniel, and Kahle, Jane (2000). Promoting Inquiry-Based Instructional Practice: The Longitudinal Impact of Professional Development in the Context of Systemic Reform. Educational Policy, 14, 331-356
Supovitz, Jonathan and Turner, Herbert (2000). The effects of professional development on science teaching practices and classroom culture. Journal of Research in Science Teaching, 37, 963-980.
Virtual High School, School Year Report Card (2004-2005 and 2005-2006). www.ode.state.oh.us/reportcard/ [report card not cited in part A, but plenty of references to the virtual high school]
Justification of Data Collection Items
Justification of Data Collection Items
In the table that follows, columns 1 and 2 show specific item numbers from the teacher questionnaires, copies of which are contained in Exhibits D and E for treatment and control teachers, respectively. The last two columns describe the information in those items and how they pertain to the analyses of instructional practices and classroom quality (hypotheses 1 and 2).
Hybrid Algebra I
Justification of Data Collection Items
Item Numbers by Instrument |
|
|||
Treatment Teacher |
Control Teacher |
Data Collection Items |
Algebra I Instructional and Technology Standards |
|
|
|
Teacher activities |
|
|
16 |
16 |
Ask "Why…. And What if…” questions |
Use questioning as a means of determining student understanding and/or to elicit thinking/understanding |
|
17 |
17 |
Use number lines, graphs, or diagrams to explain Algebra |
Connect numerical (tables and charts), algebraic (equations and formulae), and graphical strategies to solve and represent functions |
|
18 |
18 |
Use Computers to explain Algebra |
Use virtual tools, such as manipulatives, models [number lines, graphs (interactive), charts, tables] as a way to assist students with conceptualizing, visualizing, and understanding specific content |
|
Use other software that enhances, expands and connects student understanding and learning of Algebra |
||||
|
|
Student Activities |
|
|
19 |
19 |
Working in groups |
Engaging in Meaningful Conversation: strategies that promote discussion of important mathematics and encourage reflection on learning (questioning, cooperative groups, think-pair-share…) |
|
20 |
20 |
Write to explain algebra (descriptions, poetry, songs reflections) Use Exit Slips |
Writing to Learn: strategies that help students use writing to learning new mathematical content |
|
Writing to Demonstrate Learning: strategies that assist students show what they have learned through writing and to reflect on their learning |
||||
21 |
21 |
Talk to explain algebra |
Engaging in Meaningful Conversation: strategies that promote discussion of important mathematics and encourage reflection on learning (questioning, cooperative groups, think-pair-share…) |
|
Vocabulary Development: strategies that promote understanding and effective application of important mathematical vocabulary |
||||
22 |
22 |
Using things like algebra tiles or blocks |
Use manipulatives (Algebra Tiles, Geoboards, Miras, colored chips, Base Ten Tiles….) and/or models (pictures….) as a means to conceptualize the underpinnings of content |
|
|
|
Use discovery/inquiry as students construct meaning and look for multiple approaches and solutions to problems |
||
23 |
23 |
Using activities such as “guess and check”, estimating, or drawing |
Use multiple strategies to solve problems:
|
|
24 |
24 |
Using graphing calculators |
Use to assist with problem solving and reasoning and visualizing the connections to the tabular/numerical, graphic, and algebraic solutions to equations |
|
25 |
25 |
Using computers to learn algebra |
Use virtual tools, such as manipulatives, models [number lines, graphs (interactive), charts, tables] as a way to assist students with conceptualizing, visualizing, and understanding specific content |
|
26-30 |
26-29 |
Other professional development activities |
Questions about other PD to identify other factors that could impact student outcomes in participating schools. |
|
*Online Algebra I: Quality Assessment, Teacher Questionnaire, Control Teacher Questionnaire
60-day and 30-day Federal Register Notices
For Proposed Information Collection Requests
(DRAFTS TO BE EDITED UPON SUBMISSION)
60-Day Register Notice -- Draft
DEPARTMENT OF EDUCATION
Notice of Proposed Information
Collection Requests
AGENCY: Department of Education.
SUMMARY: The Leader, Information
Management Case Services Team,
Regulatory Information Management
Services, Office of the Chief Information
Officer, invites comments on the
proposed information collection
requests as required by the Paperwork
Reduction Act of 1995.
DATES: Interested persons are invited to
submit comments on or before DATE
SUPPLEMENTARY INFORMATION: Section
3506 of the Paperwork Reduction Act of
1995 (44 U.S.C. Chapter 35) requires that the
Office of Management and Budget (OMB) provide interested Federal agencies and the public an early
opportunity to comment on information
collection requests. OMB may amend or
waive the requirement for public
consultation to the extent that public
participation in the approval process
would defeat the purpose of the
information collection, violate State or
Federal law, or substantially interfere
with any agency’s ability to perform its
statutory obligations. The Leader,
Information Management Case Services
Team, Regulatory Information
Management Services, Office of the
Chief Information Officer, publishes that
notice containing proposed information
collection requests prior to submission
of these requests to OMB. Each
proposed information collection,
grouped by office, contains the
following: (1) Type of review requested,
e.g. new, revision, extension, existing or
reinstatement; (2) Title; (3) Summary of
the collection; (4) Description of the
need for, and proposed use of, the
information; (5) Respondents and
frequency of collection; and (6)
Reporting and/or Recordkeeping
burden. OMB invites public comment.
The Department of Education is
especially interested in public comment
addressing the following issues: (1) Is
this collection necessary to the proper
functions of the Department; (2) will
this information be processed and used
in a timely manner; (3) is the estimate
of burden accurate; (4) how might the
Department enhance the quality, utility,
and clarity of the information to be
collected; and (5) how might the
Department minimize the burden of this
collection on the respondents, including
through the use of information technology.
Dated: XX/XX/XXXX
Angela C. Arrington,
Leader, Information Management Case
Services Team, Regulatory Information
Management Services, Office of the Chief
Information Officer.
Institute of Education Sciences
Type of Review: New.
Title: The Effects of a Hybrid Secondary School Course in Algebra I on Teaching Practices, Classroom Quality and Adolescent Learning.
Frequency: One-time collection
Affected Public: Kentucky public schools; Algebra I teachers; math students.
Reporting and Recordkeeping Hour
Burden:
Responses: 10,900.
Burden Minutes: 98,581.
Abstract: Providing Algebra I teachers with the very best resources and professional development to ensure effective instruction has become a priority in Kentucky and across the nation. This research study is designed to test, through a rigorous experimental design, an approach that combines online and technology enhanced instruction with face-to-face classroom instruction to address this need. Participating schools will be randomly assigned to either an intervention group or a control group in and participating teachers will assume the intervention or control status assigned to their school. This submission includes recruitment of districts, schools and students. Requests for copies of the proposed information collection request may be
accessed from http://edicsweb.ed.gov,
by selecting the ‘‘Browse Pending
Collections’’ link and by clicking on
link number XXXX. When you access the information collection, click on ‘‘Download Attachments’’ to view. Written requests for information should be addressed to U.S. Department of Education, 400 Maryland Avenue, SW., Potomac Center, 9th Floor, Washington, DC 20202–4700. Requests may also be
electronically mailed to the Internet address OCIO[email protected] or faxed to 202–245–6621. Please specify the complete title of the information
collection when making your request.
Comments regarding burden and/or
the collection activity requirements
should be directed to Kathy Axt at her
e-mail address [email protected].
Individuals who use a telecommunications
device for the deaf (TDD) may call the Federal Information Relay Service (FIRS) at 1–800–877–
8339.
[FR Doc.# Filed DATE; TIME]
BILLING CODE
30-Day Register Notice -- Draft
DEPARTMENT OF EDUCATION
Submission for OMB Review;
Comment Request
AGENCY: Department of Education.
SUMMARY: The IC Clearance Official,
Regulatory Information Management
Services, Office of Management invites
comments on the submission for OMB
review as required by the Paperwork
Reduction Act of 1995.
DATES: Interested persons are invited to
submit comments on or before MONTH DAY, YEAR.
ADDRESSES: Written comments should
be addressed to the Office of
Information and Regulatory Affairs,
Attention: Rachel Potter, Desk Officer,
Department of Education, Office of
Management and Budget, 725 17th
Street, NW., Room 10222, New
Executive Office Building, Washington,
DC 20503 or faxed to (202) 395–6974.
SUPPLEMENTARY INFORMATION: Section
3506 of the Paperwork Reduction Act of
1995 (44 U.S.C. Chapter 35) requires
that the Office of Management and
Budget (OMB) provide interested
Federal agencies and the public an early
opportunity to comment on information
collection requests. OMB may amend or
waive the requirement for public
consultation to the extent that public
participation in the approval process
would defeat the purpose of the
information collection, violate State or
Federal law, or substantially interfere
with any agency’s ability to perform its
statutory obligations. The IC Clearance
Official, Regulatory Information
Management Services, Office of
Management, publishes that notice
containing proposed information
collection requests prior to submission
of these requests to OMB. Each
proposed information collection,
grouped by office, contains the
following: (1) Type of review requested,
e.g. new, revision, extension, existing or
reinstatement; (2) Title; (3) Summary of
the collection; (4) Description of the
need for, and proposed use of, the
information; (5) Respondents and
frequency of collection; and (6)
Reporting and/or Recordkeeping
burden. OMB invites public comment.
Dated: MONTH DAY, YEAR.
Angela C. Arrington,
IC Clearance Official, Regulatory Information
Management Services, Office of Management.
Institute of Education Sciences
Type of Review: Revision.
Title: The Effects of a Hybrid Secondary
School Course in Algebra I on Teaching Practices,
Classroom Quality and Adolescent Learning.
Frequency: One-time collection.
Affected Public: Kentucky Public schools; Algebra I teachers and students
Reporting and Recordkeeping Hour
Burden:
Responses: 10.900.
Burden Minutes: 98,581.
Abstract: Providing Algebra I teachers with the very best resources and professional development to ensure effective instruction has become a priority in Kentucky and across the nation. This research study is designed to test, through a rigorous experimental design, an approach that combines online and technology enhanced instruction with face-to-face classroom instruction to address this need. Participating schools will be randomly assigned to either an intervention group or a control group in and participating teachers will assume the intervention or control status assigned to their school. This submission includes recruitment of districts, schools and students. (The identification and
recruitment phase was cleared in a
previous OMB submission.)
Requests for copies of the information
collection submission for OMB review
may be accessed from http://
edicsweb.ed.gov, by selecting the
‘‘Browse Pending Collections’’ link and
by clicking on link number XXXX. When
you access the information collection,
click on ‘‘Download Attachments’’ to
view. Written requests for information
should be addressed to U.S. Department
of Education, 400 Maryland Avenue,
SW., Potomac Center, 9th Floor,
Washington, DC 20202–4700. Requests
may also be electronically mailed to
[email protected] or faxed to 202–
245–6623. Please specify the complete
title of the information collection when
making your request.
Comments regarding burden and/or
the collection activity requirements
should be electronically mailed to
[email protected]. Individuals who
use a telecommunications device for the
deaf (TDD) may call the Federal
Information Relay Service (FIRS) at 1–
800–877–8339.
[FR Doc. xx–XXXX Filed DATE; TIME]
BILLING CODE xxxxxxx
(Provided in Response to Question 16)
Table C.1 Example Tabulation of Results From Teacher Surveys
Hybrid n = XX
Control n= XX
Items |
Group |
Strongly Disagree & Disagree |
Neutral |
Strongly Agree & Agree |
Mean |
Standard Deviation |
|
Hybrid Control |
|
|
|
|
|
|
Hybrid Control |
|
|
|
|
|
|
Hybrid Control |
|
|
|
|
|
|
Hybrid Control |
|
|
|
|
|
|
Hybrid Control |
|
|
|
|
|
|
Hybrid Control |
|
|
|
|
|
|
Hybrid Control |
|
|
|
|
|
|
Hybrid Control |
|
|
|
|
|
|
Hybrid Control |
|
|
|
|
|
|
Hybrid Control |
|
|
|
|
|
|
Hybrid Control |
|
|
|
|
|
|
Hybrid Control |
|
|
|
|
|
|
Hybrid Control |
|
|
|
|
|
|
Hybrid Control |
|
|
|
|
|
|
Hybrid Control |
|
|
|
|
|
*Hybrid Teacher Questionnaire; **Control Teacher Questionnaire
Table C.1 (Continued)
While implementing the Hybrid Algebra I Approach/District Algebra I curriculum this past year:
how often did YOU do the following during DIRECT Instruction: |
Group |
Never to Rarely |
Occasionally |
Frequently to Extensively |
Mean |
Standard Deviation |
16. Ask "Why…” and “What if…" questions |
Hybrid Control |
|
|
|
|
|
17. Use number lines, graphs, or diagrams to explain Algebra |
Hybrid Control |
|
|
|
|
|
18. Use a computer to explain Algebra |
Hybrid Control |
|
|
|
|
|
how often did YOUR STUDENTS… |
Hybrid Control |
|
|
|
|
|
19. Work in groups |
Hybrid Control |
|
|
|
|
|
20. Write to explain algebra (e.g., descriptions, poetry, songs, reflections) |
Hybrid Control |
|
|
|
|
|
21. Talk to explain algebra |
Hybrid Control |
|
|
|
|
|
22. Use things like algebra tiles or blocks |
Hybrid Control |
|
|
|
|
|
23. Use activities such as “guess and check”, estimating, or drawing |
Hybrid Control |
|
|
|
|
|
24. Use graphing calculators |
Hybrid Control |
|
|
|
|
|
25. Use computers to learn Algebra |
Hybrid Control |
|
|
|
|
|
26. Use “Exit Slips” |
Hybrid Control |
|
|
|
|
|
Math-Related Professional Development (PD) Not counting Hybrid and Spotlight training, indicate the number of math-related PD activities you completed during the past 12 months as… |
|
None |
1 to 2 |
3 or more |
Mean |
Standard Deviation |
26. Workshops |
Hybrid Control |
|
|
|
|
|
27. Extended (non-graduate school) seminars or coursework |
Hybrid Control |
|
|
|
|
|
28. Graduate coursework |
Hybrid Control |
|
|
|
|
|
[Hybrid Teachers Only] |
|
Not at all |
Some |
A lot |
Mean |
Standard Deviation |
29. To what degree did the (non-Hybrid/Spotlight) math-related PD change the way you teach Algebra I |
Hybrid Control |
|
|
|
|
|
30. To what degree did the Hybrid/Spotlight PD change the way you teach Algebra I |
Hybrid Control |
|
|
|
|
|
Table C.2 Example Tabulation of Teacher Survey Results: Significant Differences Between Hybrid and Control
|
Hybrid (n = XX) |
Control (n = XX) |
|
|
|
||||
Teacher Survey Items |
M |
SD |
M |
SD |
t (XX) |
p* |
ES |
||
|
|
|
|
|
|
|
|
||
|
|
|
|
|
|
|
|
||
|
|
|
|
|
|
|
|
||
|
|
|
|
|
|
|
|
||
|
|
|
|
|
|
|
|
||
|
|
|
|
|
|
|
|
||
Scale will be presented for items with significant differences
*Bonferroni - Hochberg adjustment was used.
Table C.3 Example Tabulation of Results Teacher Survey for Item 5 and 11
Hybrid n = XX
Control n= XX
Items |
Strongly Disagree & Disagree |
Neutral |
Strongly Agree & Agree |
Mean |
Standard Deviation |
5. Use of the Hybrid Algebra I Approach/my District Algebra I curriculum increases student interest and engagement. |
|
|
|
|
|
11. Use of the Hybrid Algebra I Approach/my District Algebra I curriculum increases the amount of academically focused class time. |
|
|
|
|
|
*Hybrid Teacher Questionnaire; **Control Teacher Questionnaire
Table C.4 Example Tabulation of Teacher Survey Results for Item 5 and 11: Significant Differences Between Hybrid and Control
|
Hybrid (n = XX) |
Control (n = XX) |
|
|
|
||||
Survey Items |
M |
SD |
M |
SD |
t (XX) |
p* |
ES |
||
|
|
|
|
|
|
|
|
||
|
|
|
|
|
|
|
|
||
|
|
|
|
|
|
|
|
||
Scale will be presented for items with significant differences
*Bonferroni - Hochberg adjustment was used.
Table C.5 Example Tabulation of Student Achievement Results
Table C.5 Student Achievement Results on Algebra Post-Test |
|||
Parameter |
Estimate |
SE |
Effect Size |
Intercept |
|
|
|
|
|
|
|
School-Level Effects |
|
|
|
School Pretest |
|
|
|
Treatment |
|
|
|
|
|
|
|
Variance Components |
|
|
|
within school |
|
|
|
residual |
|
|
|
* p<.05 |
|
|
|
Effect size is given by Cohen’s d |
|||
Table C.6 Example Tabulation of Student Achievement Results by Subgroup
Table C.6 Student Achievement Results on Algebra Post-test by Subgroup |
|||
Parameter |
Estimate |
SE |
Effect Size |
Intercept |
|
|
|
|
|
|
|
School-Level Effects |
|
|
|
School Pre-test |
|
|
|
Treatment |
|
|
|
|
|
|
|
Individual-Level Effects |
|
|
|
Student pre-test |
|
|
|
Free/Reduced Lunch Status |
|
|
|
Underserved Minority |
|
|
|
|
|
|
|
Interaction Effects |
|
|
|
School Pretest*Student Pretest |
|
|
|
School Pretest*Student Free/Reduced Lunch |
|
|
|
School Pretest*Underserved Minority |
|
|
|
Treatment*Student Pretest |
|
|
|
Treatment*Student Free/Reduced Lunch |
|
|
|
Treatment*Underserved Minority |
|
|
|
|
|
|
|
Variance Components |
|
|
|
within school |
|
|
|
residual |
|
|
|
* p<.05 |
|
|
|
Effect size is given by Cohen's d |
|
|
|
1 Based on National Council of Teachers of Mathematics, North American Council for Online Learning (NACOL), ISTE NETS-T and NETS-S, and KY Algebra I standards.
2 Based on National Council of Teachers of Mathematics, North American Council for Online Learning (NACOL), ISTE NETS-T and NETS-S, and KY Algebra I standards.
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT |
| Author | Richard Roberts |
| Last Modified By | Sheila.Carey |
| File Modified | 2008-01-29 |
| File Created | 2008-01-29 |