Memo

U NITED STATES DEPARTMENT OF EDUCATION

OFFICE OF PLANNING, EVALUATION AND POLICY DEVELOPMENT

TO: Rachel Potter

Education Desk Officer

Office of Information and Regulatory Affairs

Office of Management and Budget

FROM: Michael Fong

Project Officer

Policy and Program Studies Service

Office of Planning, Evaluation and Policy Development

U.S. Department of Education

SUBJECT: Change to the OMB-Approved Study of the Implementation of Research-Based Programs and Practices to Prevent Youth Substance Abuse and School Crime (Safe and Drug-Free Schools and Communities Program)

REFERENCE #: OMB No. 1875-0216

Requesting Approval for Change to List of Research-Based Prevention Programs

The U.S. Department of Education (ED) is respectfully requesting approval of a minor change to the study’s list of research-based prevention programs. We are proposing to add up to three additional prevention programs to the list (currently there are 19) for a total of 22 programs. Since the evidence reviews were conducted in 2004 and early 2005, new sources of information have shed light on up to three additional prevention programs that are relevant to the study and have potentially positive effects.

Documents Included in Transmittal

• Completed OMB Form 83-C

• Attachment A: List of 19 research-based programs identified by the study

• Attachment B: OMB’s memo on reconciling the HAY list with the Westat study list

Background

The process for reviewing evidence of program effectiveness involved the use of rigorous research standards in order to apply uniform criteria systematically to the literature. These evidence reviews form the basis of how we measure the prevalence of research-based programs in schools, one of the major purposes and tasks of the study. The study team identified 19 prevention programs that have strong research evidence of positive behavioral outcomes (see Attachment A).

Recent evidence released by the What Works Clearinghouse (WWC) from September 2006 to the present and the Helping America’s Youth initiative (HAY) from late 2005 to the present have led us to consider updating the study’s list of research-based programs based on their work. The proposed approach for updating the study’s list:

1. Identify relevant school-based prevention programs from the WWC and HAY lists, particularly those that are itemized in the study’s survey.

2. Cross-check each program’s citations list as annotated in the WWC intervention reports and HAY website to see if each article or report was reviewed by the study team. If yes, then document the results and explain how the literature was coded. If no, then go to #3.

3. Apply the same review criteria that the study team used for the main evidence review process (resulting in the 19 programs) to any new literature identified by the WWC and HAY. For example, the literature needs to focus on behavioral outcomes as opposed to attitudinal outcomes or outcomes that are out-of-scope like academic achievement. Another example is that articles or reports must be based on more than one independent data set. Under this criterion, one must count multiple studies using the same data set (sometimes known as a re-analysis) as only one independent study.¹

4. Revise list of research-based programs and calculate the prevalence rate based on this list.

Because the WWC and HAY examined different program types and adopted different research standards and review processes, the literature identified by these projects varies widely. We propose the above approach in order to maintain consistent standards with the study’s main evidence reviews but still allowing for flexibility for other programs with compelling evidence. We view the WWC and HAY lists as a good opportunity to update the current list.

The WWC and HAY plan to continue rolling out information as evidence is reviewed and summaries and/or ratings completed. The deadline for submitting PART and GPRA information is fast approaching, so we propose that all programs identified by the WWC and HAY must have been released by the end of October 2006.

WWC Character Education Programs

As of October 2006, the WWC reviewed evidence of 13 “Character Education” programs with varied results. While many of the character education programs reviewed by the WWC focus on character traits such as honesty, kindness, or ethics, five programs have direct bearing on preventing drug/alcohol/violence behaviors and show “positive effects” or “potentially positive effects.” These five programs include (1) Lion’s Quest Skills for Adolescence, (2) Positive Action, (3) Too Good for Drugs and Violence, (4) Too Good for Drugs, and (5) Too Good for Violence.

Positive Action (#2) is one of the 19 programs already identified by the study team as being research-based, so no additional review of this program is necessary. The other four programs have been or are currently going through the updated review process identified on page 2. Too Good for Drugs (#4) meets all of the study’s screens, so we propose adding it to the list. This increases the study’s list from 19 programs to 20.

One other program, Lion’s Quest Skills for Adolescence (#1), has potential to pass these screens. The review has not yet been completed, so Lion’s Quest Skills for Adolescence is considered a possible 21^st program.

The WWC intervention reports on character education programs can be downloaded from the following website:

http://www.whatworks.ed.gov/Topic.asp?tid=12&ReturnPage=default.asp

HAY Programs

OMB carefully compared our study’s criteria against the HAY criteria to identify differences in the evidence review standards and in how programs were ultimately coded/rated. In a memo titled “Reconciling the HAY list and the Westat study list” sent July 10, 2006, OMB provides a breakdown of relevant HAY programs by the stage in which the program was eliminated during the Westat evidence review process. The memo also outlines possible action steps should ED decide to revise the list in consideration of the HAY process (see Attachment B).

All programs identified in the OMB memo have been previously screened by Westat producing results that are documented in the study’s database of evidence reviews. Westat applied the same set of screening criteria to each of these programs. Many of the HAY list of citations for the programs were in fact reviewed by the study’s review team. HAY did uncover some citations that were not reviewed by the study team. The same is true that our study team found additional citations that the HAY did not list.

A review of the HAY citations show that almost every article or report was reviewed but did not meet the study’s screens for the following more common reasons: (1) articles or reports are based on student attitudes, knowledge, or other outcomes rather than student behavior; (2) articles or studies were based on only one independent data set; (3) articles or studies were unpublished or not peer reviewed; and (4) articles and studies were published between 2005 to the present or “still in press.”

The only program from the reconciliation memo that should be added is Skills, Opportunities, and Recognition (SOAR). The reason the program was excluded was due to a lack of implementation materials that is germane to the measurement of program fidelity (Phase 2 of the study). Prior to this determination, the study team reviewed the evidence and found that the program did in fact meet all of the screens. This program would be appropriate to include and increases the list to 22 programs.

Summary

This memo requests a small change to the list of research-based programs, increasing the number from 19 to possibly 22 programs (21 if Lion’s Quest Skills for Adolescence does not meet the screens). If you have any questions, please contact Michael Fong at 202-401-7462; email [email protected].

Thank you!

Attachments

1 Please note that in our discussions with Allison Cole in June and July 2006 about the HAY programs, we incorrectly noted that the study’s evidence reviews looked at articles and reports published in 1996 or later. The study actually reviewed articles and reports published in 1983 or later, a roughly 20-year period consistent with the WWC.

4