Reconciling the HAY list and the Westat Study List
The goal of this process is to determine and reconcile the sources of difference in the two lists.
Steps Westat has already completed:
Westat determined which programs included in Level I and II of the HAY list are within the scope of the Safe and Drug Free Schools (SDFS) program. These programs were all (a) focused on youth substance abuse or school crime and (b) applicable to school-age youth.
Result: 58 HAY level I and level II programs were identified as within the scope of the SDFS program.
2. Westat identified 4 programs that were on their study list that were not included in the HAY list.
On Westat Study List, NOT on HAY list |
Anger Coping Program |
Brainpower |
Child Development Project |
Earlscourt Social Skills Group Program |
Action Step: These programs could be considered for inclusion in the HAY list. These programs should be retained in Westat’s data analysis.
Westat identified 3 programs that were on their study list that were included in Level III of the HAY list.
Programs on Level III of the HAY list which were also on Westat’s Study List |
Adolescent Alcohol Prevention Trial All Stars |
Know Your Body |
Second Step |
Action Step: These programs might be considered for inclusion in Level I or Level II of the HAY list because the research evidence supporting this program was determined by Westat to be strong. These programs should be retained in Westat’s data analysis.
Next Steps for Reconciling the HAY list and the Westat Study list:
Westat’s work has identified 65 programs which were included in either the HAY list or Westat’s study list, or both, as within the scope of the SDFS program. 19 of these programs already appear on the Westat study list and will be included in Westat’s data analysis. 46 programs remain for further analysis.
11 of these programs were not included in list of programs reviewed by Westat for their study and were not included in the Program Survey Questionnaire (Westat reviewed 317 programs and found 89 of these programs to be with the scope of the SDFS program). These programs first appeared in the 12/05 version of the Mihalic Metrix used by Westat to construct the list of relevant programs for the study. Had these programs been included on the list at an earlier date, they would have been eligible for the study.
Programs that were not included in the Westat review |
Academic Tutoring and Social Skills Training |
American Indian Life Skills Development |
Caring School Community Program |
Families in Action |
HeadOn |
Not-on-Tobacco (NOT) |
Project EX |
SMART Team (Students Managing Anger and Resolution Together) |
Spit Tobacco Intervention for High School Athletes |
Success in Stages |
Too Good for Violence |
Action Step: These 11 programs could be reviewed to determine the strength of the research evidence of their effectiveness. If research evidence is sufficient, these 11 programs could be included in the data analysis where they were included in write-in responses.
2 of the remaining 35 programs were not included in list of programs reviewed by Westat for their study but were included in the Program Survey Questionnaire.
Programs not reviewed but included in survey |
Preventive Treatment Program |
Peacemakers Program (Grades 4-8) |
Action Step: These 2 programs could be reviewed to determine the strength of the research evidence of their effectiveness. If research evidence is sufficient, these 2 programs could be included in the data analysis.
1 of the remaining 33 programs is included in the HAY level II list but was eliminated from Westat’s list of research based programs because they did not have implementation materials available for the fidelity study.
Programs without implementation materials |
Skills, Opportunities, and Recognition (SOAR) |
Action Step: This program could be reviewed to determine the strength of the research evidence of its effectiveness. If research evidence is sufficient, this program could be included in the data analysis.
5 of the remaining 32 programs are included in the HAY level II list but were eliminated from Westat’s list of research based programs because they did not have 2 or more empirical studies which evaluated the program.
Programs with only 1 evaluation |
Al's Pals: Kids Making Healthy Choices |
Bicultural Competence Skills Approach |
Keep A Clear Mind (KACM) |
Protecting You/Protecting Me |
School Violence Prevention Demonstration Program |
The review conducted to construct the HAY list found more than 1 study for 4 of these 5 programs.
Action Step: The additional studies used in the construction of the HAY list could be reviewed to determine the strength of the research evidence of the effectiveness of these 4 programs (see Appendix 1). If research evidence is sufficient, these 4 programs could be included in the data analysis.
5 of the remaining 27 programs are included in HAY’s Level I or Level II but were eliminated by Westat for not meeting stringent methodological standards.
Programs that did not meet Westat’s stringent methodological standard |
Bullying Prevention Program (BPP) |
Guiding Good Choices (Preparing/ Drug Free Years) |
I Can Problem Solve (Interpersonal Cognitive Problem Solving) |
Strengthening Families Program |
Strengthening Families Program Parents and Youth 10-14 |
Action Step: These 5 programs could be included in the data analysis.
The 22 remaining programs are included in the HAY list but were eliminated by Westat for not having research support that met at least minimally acceptable methodological standards. These standards included
Whether each study reported quantitative results on behavioral outcomes for the program of interest;
Whether or not any statistically significant results favored the program on any of these outcomes;
Whether an “acceptable” research design was used: randomized control trial, pretest/posttest with comparison group, regression discontinuity, or interrupted time series; and
Whether at least two independent samples were studied.
A number of these programs have been identified as research based, exemplary, or promising by studies conducted previously by Education, by other Federal agencies, or by RAND. The programs where this is the case are also identified below.
Programs that did not meet Westat’s minimal methodological standards |
Level I on HAY list |
1. Athletes Training and Learning to Avoid Steroids* n |
2. Coping Power n |
3. FAST Track n |
4. Good Behavior Game/Baltimore Mastery Learning |
5. Keepin It Real |
6. Linking the Interests of Families and Teachers (LIFT) |
7. PeaceBuilders |
8. Project Northland*** |
9. Safe Dates |
10. Towards No Tobacco Use (TNT)*** |
|
Level II on HAY list |
11. CASASTART* |
12. Chicago Child-Parent Center and Expansion Program |
13. Children of Divorce Intervention Program |
14. Consistency Management and Cooperative Discipline |
15. First Step to Success |
16. Growing Healthy** |
17. Lions-Quest Skills for Adolescence** n |
18. Resolving Conflicts Creatively |
19. Schools and Families Education Children (SAFE Children) |
20. Sembrando Salud |
21. SMART Leaders |
22. Social Competence Promotion Program for Young Adolescents (SCPP-YA) |
*Identified by Education as an exemplary program. |
**Identified by Education as a promising program. |
***Identified by RAND as a model program and by Education as an exemplary program. |
n Identified by NIDA as a evidence based program. |
Action Step: Determine whether the research evidence for these programs should be reviewed again.
Notably, the criteria used by Westat and the criteria used in the construction of the Hay list are similar. One source of difference that could be considered by a second review is that, according the HAY criteria, any studies published after 1950 were included in the construction of the HAY list; Westat used only studies published after 1995.
| File Type | application/msword |
| File Title | Reconciling the HAY list and the Westat Study List |
| Author | cole_a |
| Last Modified By | Michael.Fong |
| File Modified | 2006-11-09 |
| File Created | 2006-11-09 |