ED Response to OMB Comments

ED/OESE Response to OMB comments on Promoting Student Success in Algebra 1 Project Data Collection

1) How will the ten sites be selected? Why are they limited only to HSGI grantees?  Are a sufficient number of HSGI grantees implementing the five topic areas effectively enough to be highlighted as models?

 

Response: This project is being funded by the HSGI and many HSGI grantees have implemented approaches to support students struggling in Algebra I.  A preliminary review of documents suggests that the AIR project team should be able to identify some, but not all sites from the HSGI grantee pool.   We do anticipate that to identify a diverse set of exemplar sites, we will need to expand the set of candidate sites beyond the HSGI grantees.  Because some HSGI grantees were awarded to state education agencies (for example, the Colorado Department of Education) this widens the pool of candidate sites.

 

As described in the OMB clearance document, the project team will seek a pool of sites that reflect variation in terms of enrollment, urbanicity, geographic region, and student demographics.   All final decision regarding which sites to be profiled will be made by ED. 

 

To supplement the information on the sampling procedures described in the OMB clearance submission, the project team proposes the following:

 

Sampling within the HSGI grantees:

• As a first step, the project team will contact the math coordinator for all HSGI districts (or the districts in which HSGI schools are nested) to inform them of the project and inquire about activities that would align with the purposes of the project.

        The project director or his designee with appropriate math expertise will conduct informal phone conversations with the math coordinator for each site to determine the level of implementation, perceptions of impact, and alignment with research-supported approaches.

        Each candidate site must also send documentation to provide supplementary evidence of the strategy or approach, including additional descriptive information of the approach, documentation of implementation and evidence suggesting that the approach has contributed to improved outcomes of interest.

        Based on the interview and documentation, dimensions of each strategy or approach will be rated on a 3 point-scale as minimally (1), moderately (2), or extensively (3) meeting specific criteria for each dimension.

        If an HSGI grantee receives an average rating of 2 points or higher across the dimensions identified by the project team, the grantee will be nominated for inclusion in the sample. 

Sampling outside of HSGI grantees:

After identifying all candidate sites among the HSGI grantees, the project team will identify non-HSGI grantees, drawing on AIR’s extensive contacts among mathematics experts, state and district mathematics coordinators, comprehensive centers, regional labs, and other organizations.  In addition, the AIR team will consult with sites that participated in the project’s TWG meeting in 2013 and with members of the National Council of Supervisors of Mathematics (NCSM), the largest national organization of state and district math coordinators and directors.  After identifying a set of districts or schools with potentially interesting, relevant and promising approaches, the team will follow the same procedures as above, namely:

        Conducting phone conversations with math representatives from the candidate districts or schools and requesting documentation

        Rating the quality of the strategy or approach based on the phone conversation and documentation.

        Identifying sites that meet minimum standards for inclusion in the study

2) Did the study consider selecting sites that are carrying out more than one of the topics effectively?  The rationale being that these topics are inter-related and a model site would have all (or most) of these topics integrated seamlessly.    

 

Response:  This is an excellent point and one that we are currently working to address in FY 2014. Unfortunately, funding was limited in FY 2012 and the HSGI program office first wanted to examine the in-depth implementation of specific practices prior to looking at the implementation of multiple practices in one setting.  A review of one practice allows for the development of technical assistance tools that provide rich descriptions of the implementation process and that can help program practitioners and developers  support change at HSGI sites or at other sites with similar needs.  HSGI staff is working with the AIR project team to expand the scope of the current project to include an additional component that would examine sites with multiple, integrated topics and would enable us to address this request directly.  While the outcome will also be the development of technical assistance tools, the audience for this tool is system administrators.

 

For the scope of work described in the OMB clearance submission, ten sites would be profiled.  We do not intend to obscure the connections among practices.  For example, a site that is implementing a promising instructional approach may support this approach through professional development, but the primary emphasis of the profile would be the implementation of the instructional practices. The professional development would be discussed as part of the profile, but as supplemental information. Since professional development is likely to be a cross-cutting topic area, we expect that one or more of the profiles will include supplemental information about professional development. As with all topical areas, the profiles that focus primarily on professional development will be selected because they illustrate successful implementation of one or more of the key findings from the research.

 

3) One critical area of policy interest is identifying concrete ways to help with the transition to college-and career-ready standards.  Are your profiles of practice directly addressing this area?  We suggest adding questions on this topic in Exhibit 1 and integrating this into the technical assistance products.  For example, is it possible to ask whether any of the practices being studied, where relevant, have helped with transitioning to teaching new college- and career-ready standards?

 

Response: We agree that consideration of college- and career-ready standards (most often the Common Core standards) is an important topic and one that we intend to address in data collection activities, the practice profiles, and associated tools. However, the primary focus of this project is on helping struggling students succeed in Algebra I, a specific course identified by HSGI grantees as a needed area of technical assistance. Algebra I is considered a “gateway” to college- and career-ready high school courses and the math content and instructional emphases of the Common Core State Standards for Mathematics (CCSSM) also reflect the college- and career-ready standards, so this project will be directly relevant to districts and schools who are making the transition to college- and career-ready standards.   

 

In terms of mathematics content, each of the current topics of the profiles of practice, which reflect the study team’s recent review of the relevant research, is directly related to helping students transition to college- and career-ready standards.   For example, the content of the CCSSM represent the college- and career-ready math content standards in the 44 states that have adopted the CCSSM.  The CCSSM reflects both procedural fluency and conceptual understanding in mathematics, which dovetails with this project’s focus on instructional practices that emphasize procedural fluency and conceptual understanding.  Thus, as respondents provide information about the transition to new instructional practices with this dual emphasis, this will inform our understanding of concrete ways to facilitate the transition to college- and career-ready standards.  Similarly, the Common Core standards meet the definition of a vertically-aligned curriculum, so our interview and focus group questions on the curricular alignment topic will yield data that could inform Common Core implementation.

 

4) Could the study team look at sites that have made the transition to college-and career-ready standards?

 

Response: As indicated in our response to Question 3, 44 states have adopted the CCSSM, so we expect that the majority of our sites will be in these states and will therefore be making the transition to college- and career-ready standards.  Yet there is likely to be variation across sites within and across these states in the degree to which they have fully transitioned to these standards. During site selection, the phone conversations and documentation review process will include specific prompts to determine whether sites have college- and career-ready initiatives that extend beyond the adoption of the CCSSM, and this information will be considered as part of the selection criteria.  However, as mentioned previously, the primary focus of these profiles is to provide information and tools to practitioners who are focused on helping struggling students succeed in Algebra I. 

 

5) How did the study team determine what to cover in the Profiles of Practice for each topical area?  For example, why is the study only focusing on double-dose algebra programs for the Supplementary Learning Opportunities Profile of Practice?

 

Response: The topics and practices covered for each profile of practice were informed by what was learned from the reviews of existing research on each topic (soon to be published in the resulting research briefs).  These reviews highlight promising practices and strategies with the strongest evidence to support success in Algebra I.  Sites selected for all profiles of practice will be required to demonstrate high-quality implementation aligned with research-supported approaches as detailed above.

The Supplementary Learning Opportunities Profile of Practice is a special case. The scope of programs broadly classified as supplementary learning opportunities is too large to showcase in a profile of practice.  Because the profiles of practice are designed to provide an in-depth picture of a specific practice or strategy, HSGI staff and the project team have narrowed the focus of this profile of practice to examine double-dose algebra as a prominent strategy to support struggling students in Algebra I.  Districts and schools, including HSGI grantees, are increasingly developing and implementing double-dose programs to support struggling students and there is a rigorous base of evidence to support its effectiveness. 

 

6) In conducting the literature review on the five topical areas, did OESE consult with IES?  Were the What Works Clearinghouse standards used to systematically determine which studies to include in the literature review?  If not, what was the protocol the study team followed to determine which studies to include?

 

Response: All of the literature reviews were informed by relevant IES practice guides and included a wide range of articles involving descriptive, theoretical, and explanatory research that spanned different methodological approaches (e.g., high-quality experiments, quasi-experimental studies, descriptive studies, case studies), sources (e.g., educational journals, research organizations, national content-specific organizations), and disciplines. In addition to conducting a rigorous search of existing literature, we contacted experts in the field who are conducting research on these educational programs to identify research findings not yet published and included them in this review. We used a four-part, hierarchical selection process as the basis for including the studies summarized in this brief: subject (algebra vs. mathematics vs. other subjects), grade level (Grades 6–9 vs. Grades 1–5), year of publication (since 2005 vs. before 2005), and level of evidence (strong vs. moderate vs. low, based on standards informed by the What Works Clearinghouse (WWC; see http://ies.ed.gov/ncc/wwc/). We prioritized studies that focused on algebra or mathematics in Grades 6–9, that were published since 2005, and that had strong or moderate evidence per WWC guidelines.

 

Subject: OMB comments on Promoting Student Success in Algebra 1 Project

 

 

1)      How will the ten sites be selected?  Why are they limited only to HSGI grantees?  Are a sufficient number of HSGI grantees implementing the five topic areas effectively enough to be highlighted as models?

2)      Did the study consider selecting sites that are carrying out more than one of the topics effectively?  The rationale being that these topics are inter-related and a model site would have all (or most) of these topics integrated seamlessly.    

3)      One critical area of policy interest is identifying concrete ways to help with the transition to college-and career-ready standards.  Are your profiles of practice directly addressing this area?  We suggest adding questions on this topic in Exhibit 1 and integrating this into the technical assistance products.  For example, is it possible to ask whether any of the practices being studied, where relevant, have helped with transitioning to teaching new college- and career-ready standards?

4)      Could the study team look at sites that have made the transition to college-and career-ready standards?

5)      How did the study team determine what to cover in the Profiles of Practice for each topical area?  For example, why is the study only focusing on double-dose algebra programs for the Supplementary Learning Opportunities Profile of Practice?

6)      In conducting the literature review on the five topical areas, did OESE consult with IES?  Were the What Works Clearinghouse standards used to systematically determine which studies to include in the literature review?  If not, what was the protocol the study team followed to determine which studies to include?