Att_Supporting Statement for 2nd OMB_Section A (01-26-09)_Final

Request for Clearance
“Study of the Implementation of the Safe and
Drug-Free Schools and Communities Act
Program State Grants”

Submitted to:
The Office of Management and Budget
Human Resource Programs
Education Branch
Washington, DC 20503

Submitted by:
U.S. Department of Education
Office of Planning, Evaluation, and Policy Development
Policy and Program Studies Service
Washington, DC 20202
Contract No. ED-04-CO-0059

Table of Contents
Section

Page

A
A.1

JUSTIFICATION ............................................................................................

1

Need for Collecting This Information..............................................................

1

A.1.1
A.1.2
A.2
A.3
A.4
A.5
A.6
A.7
A.8

Overview of Proposed Study ..............................................
Instrumentation for Examining SDFSCA Programs ...........

3
11

Purposes and Uses of the Data.........................................................................
Improved Information Technology..................................................................
Efforts to Identify Duplication.........................................................................
Methods Used to Minimize Burden on Small Entities ....................................
Consequences of Less Frequent Collection .....................................................
Special Circumstances for Data Collection .....................................................
Compliance with 5 CFR 1320.8 and Consultations.........................................

13
13
14
14
14
15
15

A.8.1
A.8.2
A.9
A.10
A.11
A.12
A.13
A.14
A.15
A.16
A.17
A.18
B
B.1

Compliance with 5 CFR 1320.8..........................................
Consultations.......................................................................

15
15

Payment of Stipends to Participants ................................................................
Assurance of Confidentiality ...........................................................................
Justification for Questions of a Sensitive Nature.............................................
Estimates of Burden Hours ..............................................................................
Estimates of Annual Cost Burden to Respondents ..........................................
Estimates of Annualized Cost to the Federal Government ..............................
Changes in Burden...........................................................................................
Publication Plans and Time Schedule..............................................................
A.16.1 Time Schedule.....................................................................
A.16.2 Data Analysis Plans.............................................................
Approval Not to Display Expiration Data for OMB Approval........................
Adherence to the Guidelines in 5 CFR 1320.9 ................................................

17
17
18
20
20
20
21
21
21
21
25
25

DESCRIPTION OF STATISTICAL METHODOLOGY ...............................

26

Respondent Universe .......................................................................................

26

B.1.1
B.1.2
B.2

Schools and Districts...........................................................
Research-Based Prevention Programs ................................

26
27

Stratification and Sample Selection .................................................................

29

B.2.1
B.2.2
B.2.3
B.2.4

Selection of District and School Samples ...........................
Selection of Research-Based Prevention Programs ............
Expected Levels of Precision ..............................................
Estimation Procedures.........................................................

ii

29
31
33
34

Table of Contents
Section

Page
B.3

Methods to Maximize Response Rates and Deal with Nonresponse...............
B.3.1
B.3.2

35

Survey Response .................................................................
Item Response and Data Quality.........................................

36
36

B.4

Tests of Procedures and Methods ....................................................................

37

B.5

Statistical Consultation and Implementation of the Study...............................

37

List of Appendices ........................................................................................................

38

List of Tables
Table
1

Crosswalk of study questions, study components, and domains of
variables ...........................................................................................................

8

Data collection activities and schedule for the Study of Implementation of
Safe and Drug-Free Schools and Communities Act Program State Grants .....

12

3

Estimates of annual response burden ...............................................................

20

4

Anticipated project schedule............................................................................

22

5

Distribution of schools in 2005-06 CCD file by instructional level,
metropolitan status, and percent minority enrollment .....................................

28

Expected samples for the Prevalence Study by level, metropolitan status,
and percent minority enrollment......................................................................

31

Expected sample sizes based on an initial sample of 6,000 schools and
prevalence rates reported in the Study of the Implementation of ResearchBased Programs to Prevent Youth Substance Abuse and School Crime .........

33

95 percent confidence interval half-widths for an estimate of a 50-percent
characteristic ....................................................................................................

34

2

6
7

8

iii

List of Exhibits
Exhibit
1

Westat confidentiality pledge ..........................................................................

19

List of Figures
Figure
1

Conceptual framework for understanding adoption and implementation of
prevention programs in schools .......................................................................

iv

6

REQUEST FOR CLEARANCE
STUDY OF THE IMPLEMENTATION OF THE SAFE AND DRUG-FREE SCHOOLS
AND COMMUNITIES ACT PROGRAM STATE GRANTS
SECTION A. JUSTIFICATION

A.1.

Need for Collecting This Information
The purpose of the Study of the Implementation of the Safe and Drug-Free Schools and

Communities Act Program State Grants is to gauge progress in increasing the prevalence and quality of
research-based prevention programs in public elementary and secondary schools nationally and those
funded by the Safe and Drug-Free Schools and Communities Act (SDFSCA) Program. As such, this study
will collect critical information about the SDFSCA Program, as authorized by Title IV, Part A, Subpart 1
of the Elementary and Secondary Education Act of 1965, as amended (ESEA). Specifically, the
information will allow the Department of Education (ED) to assess the overall quality of activities that are
being implemented by grantees and to provide followup data for performance measures to meet
Government Performance and Results Act (GPRA) and the Program Assessment Rating Tool (PART)
review requirements for the Program. A previous study sponsored by ED provided the initial or baseline
data for these measures, which are: (a) the percentage of drug and violence prevention programs/practices
supported with SDFSCA state grants funding that are research-based, and (b) the percentage of SDFSCAfunded research-based drug and violence prevention curriculum programs that are implemented with
fidelity; the proposed study will provide follow-up data for the measures. We are requesting that OMB
approve a set of revised survey instruments that were previously approved for obtaining the required
baseline data.
The U.S. Department of Education is authorized by Section 4121 of the ESEA to evaluate
activities funded through the SDFSCA Program. In addition, the ESEA requires that ED report on several
aspects of illegal drug use and violence in elementary and secondary schools, including their incidence
and prevalence. This study will examine one type of activity that receives SDFSCA-funding, prevention
programs, by measuring the prevalence and fidelity of research-based programs. 1 The study will not
examine other types of activities that receive SDFSCA funding, which include but are not limited to: (a)
community-wide planning and organizing activities to reduce violence and illegal drug use, (b) acquiring
1

Defining quality in terms of whether activities are research-based is consistent with the No Child Left Behind Act requirement that SDFSCAfunded activities adhere to the “Principles of Effectiveness,” one of which is that activities should be research-based. For more information,
see:, ESEA, Section 4115.

1

and installing metal detectors or other related equipment and technologies, and (c) reporting criminal
activities committed on school property.
The Study of the Implementation of the Safe and Drug-Free Schools and Communities Act
Program State Grants is part of a comprehensive set of studies sponsored by ED that will help to meet it
goals for the SDFSCA Program. In particular, this study will build on the foundation established by
previous studies, including the Study of the Implementation of Research-Based Programs to Prevent
Youth Substance Abuse and School Crime (i.e., precursor study that provided baseline data on the
GPRA/PART measures for the SDFSCA Program), National Study of School Violence and Prevention
(SSVP) and School-Based Drug Prevention Programs: A Longitudinal Study in Selected School Districts,
that together focused on examining the types and funding sources of programs being implemented and the
extent to which sound planning and implementation processes are used for the programs. The goal of the
current study is to provide the first followup data on the extent to which SDFSCA-funded prevention
efforts in schools are based on sound, research-based principles.
As a part of the national effort to provide programming in the schools and, through other
social service agencies, to prevent youth alcohol, tobacco, and other drugs (ATOD) use and school crime
in this country, the SDFSCA Program provides funding to states to support prevention programs. Each
state has both a State Education Agency (SEA) Program and a Governors’ Program. Under the current
legislation, the SEAs allocate funds to school districts by formula (based on Title I grants and student
enrollment); the Governors’ Programs award grants to community agencies and public and private
nonprofit entities.

The districts and other grantees support prevention activities at the school and

community levels. Providing nearly $300 million in state grants and reaching schools in practically all
school districts, this program is the largest and broadest school-based youth ATOD use and school crime
prevention program nationally. The current study will examine only the SEA Program.
A clarification on the use within this study of various selected terms is useful here. The term
SDFSCA Program refers to the formula grant program administered by the U.S. Department of Education
and implemented in school districts and communities through SEAs. A prevention program is an
intervention or set of interventions put in place with the intention of reducing problem behavior in a
population of youth or to establish and maintain a safe and orderly learning environment. A school-based
prevention program is one that, regardless of funding source, is primarily delivered in a school building
(even if outside of school hours) or is implemented by school staff or under school or school system
auspices, and includes any of the levels from kindergarten through high school.

2

A.1.1.

Overview of Proposed Study
In this section, we introduce the current study, which follows closely the methods used by

the previous study that provided baseline data for the required GPRA/PART measures (Study of the
Implementation of Research-Based Programs to Prevent Youth Substance Abuse and School Crime). The
introduction includes a description of the study questions, the conceptual framework of the study, and
study design.

Study Questions
In addition to using information from the proposed study to meet legislative requirements,
ED is interested in obtaining information that will help it to improve administration of this program. To
meet these objectives, the study will seek to answer the following three research questions.

1.

Which youth ATOD use and/or school crime prevention programs and practices are
research-based and have been shown to be efficacious or effective?

1a.

Which youth ATOD use and/or school crime prevention programs have been
identified previously as research-based, and does research conducted from 2004
to 2008 continue to support these programs and/or identify additional
programs?

1b.

What evidence does literature from 1983 to 2008 offer as to the efficacy and
effectiveness of these programs and practices?

The first research question focuses on the extent to which currently available evidence
qualifies programs and practices as research-based, and on the magnitude of effects found for such
programs and practices. Research subquestion 1a is driven by the need to identify research-based
programs that will be the subject of activities designed to answer questions 2 and 3. Subquestion 1b is
intended to provide information on the magnitude of effect of the research-based programs and of general
practices that cut across specific programs and contribute to the achievement of desired outcomes.

2.

What proportion of youth ATOD use and/or school crime prevention programs,
nationally and among those receiving funding from the SDFSCA Program, are
research-based?

2a.

What proportion of schools is implementing research-based youth ATOD use
and/or school crime prevention programs?

2b.

What proportion of SDFSCA-funded programs is research-based?

3

2c. What factors are associated with the adoption of research-based youth ATOD use
and/or school crime prevention programs?
This research question tackles the important issue of adoption by schools of research-based
prevention programs. Subquestion 2a focuses on programs regardless of their funding sources, while
subquestion 2b narrows the scope to include only those receiving funding from the SDFSCA Program. As
mentioned, ED has particular interest in the use of SDFSCA Program funding for research-based
prevention programs and other authorized activities (e.g., community-wide planning and organizing
activities to reduce violence and illegal drug use), because the ESEA requires that those funds be used to
support only research-based activities. Subquestion 2c focuses on the factors that are associated with
adoption by schools of research-based programs. The answers may point to possible administrative
improvements (e.g., related to the availability of information from ED on specific research-based
programs) that can boost adoption.

3.

To what extent, nationally and among those receiving funding from the SDFSCA
Program, are research-based youth ATOD use and/or school crime prevention
programs implemented with fidelity?

3a.

What proportion of research-based youth ATOD use and/or school crime
prevention programs is being implemented with fidelity?

3b.

What proportion of research-based SDFSCA-funded programs is being
implemented with fidelity?

3c.

What factors are associated with implementing with fidelity research-based
youth ATOD use and/or school crime prevention programs?

The third question recognizes that achieving desired outcomes requires both offering
research-based programs and implementing them well. Subquestion 3a pertains to all research-based
programs, while subquestion 3b targets those research-based programs receiving SDFSCA Program
funds. Subquestion 3c focuses on the factors that are associated with the quality of implementation of
programs adopted by schools. The answers may point to possible administrative improvements (e.g.,
related to quality and amount of training on implementation) that can enhance the quality of
implementation.

Conceptual Framework
We plan to use a conceptual framework to help guide the study. This conceptual framework
identifies the main topics of interest: adoption and implementation of research-based programs in schools

4

and the factors associated with them (see Figure 1). The study will measure and analyze variables from
each of the following components.
•

Prevention programs (Area 1)—This is the universe of interventions put in place in
schools with the intention of preventing or reducing youth ATOD use and school
crime. As indicated by the box enclosing Area 1, prevention programs are among a
diverse range of activities that can be used to achieve those objectives.

•

Research-based prevention programs (Area 2)—A subset of all prevention
programs in schools, these programs are supported by rigorous research that indicates
they consistently prevent or reduce youth ATOD use or school crime. They vary on
quality of implementation.

•

Prevention programs implemented with fidelity (Area 3)—A subset of all
prevention programs in schools, these programs are implemented in ways that meet
the standards established by their developers. They include research-based programs
and other programs. Of all the prevention programs operating in schools, researchbased programs implemented with fidelity are the most likely to successfully prevent
or reduce youth ATOD use and school crime.

•

Prevention programs funded by the SDFSCA Program (Area 4)—A subset of all
prevention programs in schools, these programs receive funding from the SDFSCA
Program. They include research-based programs and other programs; and they vary on
quality of implementation. As mentioned in Section A.1, the Principles of
Effectiveness require that all SDFSCA-funded programs and other activities be
research-based.

•

Factors associated with adoption of research-based prevention programs
(Area 5)—These factors can potentially affect whether schools adopt research-based
prevention programs. They include relatively direct factors, such as the availability of
funding and aspects of the program selection process that operates at the district level.
They also include relatively indirect factors, such as whether district staff receive
technical assistance from SEAs on program selection and federal policy supporting
the use of research-based programs.

•

Factors associated with implementation with fidelity of research-based
prevention programs (Area 6)—These factors can potentially affect the quality of
implementation of the research-based prevention programs operating in schools. They
include proximal factors, such as the use of implementation materials by program
providers, and the quality and amount of training on the programs. They also include
distal factors, such as the extent to which district staff monitor and evaluate programs,
and districts and SEAs are accountable for the performance of their prevention
programs.

5

Figure 1. Preliminary conceptual framework for understanding adoption and implementation of prevention programs in schools
All prevention
activities

6

5. Factors
associated with
adoption of
research-based
prevention
programs

1. Prevention programs

3. Prevention
programs
implemented
with fidelity

2. Researchbased
prevention
programs

4. Prevention
programs
funded by
SDFSCA
Program

7. Contextual factors

6. Factors
associated with
implementation
with fidelity of
research-based
programs

•

Contextual factors (Area 7)—These factors represent the environment in which all
of the other components in the framework operate. They include school, district, and
community characteristics, such as: school instructional level and student
characteristics, district enrollment, and urbanicity. Such factors remind us that the
prevention programs in schools operate within dynamic organizations and settings that
can indirectly facilitate or impede them.

Study Design
The design of the study has three main components—Identification Study, Prevalence Study,
and the Fidelity Study. For the Prevalence Study and Fidelity Study components, the study will collect
information about the SDFSCA SEA Program. Each of the components parts is described in this section
and Table 1.
The Identification Study will provide answers for the first research question concerning the
programs and practices that are deemed effective. The previous study (Study of the Implementation of
Research-Based Programs to Prevent Youth Substance Abuse and School Crime) indicated that
identifying research-based practices and specific named programs required separate approaches. To
identify practices, we will update the review of meta-analyses (conducted for the previous study) that
provided quantitative results, across many studies, on the effectiveness of practices and general program
types. To identify specific programs, we will update the review (conducted for the previous study) of over
1,000 individual studies on programs that were judged to be effective by external sources.
To identify practices that consistently yield sizable positive effects on behavior problems,
we will update the review of meta-analyses on the prevention of youth ATOD use and school crime
(including violence). The previous review covered meta-analyses published from 1983 to 2004 on efforts
to prevent or reduce youth ATOD use and school crime; we will extend it to include meta-analyses
published through 2008. Based on those meta-analyses, we will update the database of the reported effect
sizes. 2 This database consists of over 200 effect sizes, organized by type of outcome and by different
practices and general program types. We will review the findings for practices and general program types
to flag those that had consistently high effect sizes across diverse types of outcomes.
Developing a list of programs for the study will entail compiling and screening existing lists
of research-based prevention programs, reviewing literature on the programs that pass the screens, and
making judgments on whether the programs achieve high levels of effectiveness. We discuss these tasks
in the subsections that follow.

2

Effect sizes are standardized measures of treatment effectiveness, typically in relation to a comparison group.

7

Table 1. Crosswalk of study questions, study components, and domains of variables
Study question

Study component

Domain of variables*

1.

Which youth ATOD use and/or
school crime prevention programs
and practices are research-based and
have been shown to be efficacious or
effective?

Identification Study
ƒ Systematic review of existing
meta-analyses on effective
prevention practices
ƒ Systematic review of existing
literature on research-based
prevention programs
ƒ Meta-analysis on research-based
prevention programs indicated by
the systematic review of programs

Research-based prevention programs
(Area 2)

2.

What proportion of youth ATOD use
and/or school crime prevention
programs, nationally and among
those receiving funding from the
SDFSCA Program, are researchbased?

Prevalence Study
ƒ Prevalence Survey
ƒ District Survey

Intersection of:
ƒ Prevention programs (Area 1)
ƒ Research-based prevention
programs (Area 2)

3.

To what extent, nationally and among
those receiving funding from the
SDFSCA Program, are researchbased youth ATOD use and/or school
crime prevention programs
implemented with fidelity?

Intersection of:
ƒ Prevention programs (Area 1)
ƒ Research-based prevention
programs (Area 2)
ƒ Prevention programs funded by
SDFSCA Program (Area 4)

Fidelity Study
ƒ Provider Survey
ƒ Interviews with program
developers
ƒ Results from Prevalence Study

Factors associated with adoption of
research-based programs (Area 5)
Intersection of:
ƒ Prevention programs (Area 1)
ƒ Research-based prevention
programs (Area 2)
ƒ Prevention programs
implemented with fidelity (Area
3)
Intersection of:
ƒ Prevention programs (Area 1)
ƒ Research-based prevention
programs (Area 2)
ƒ Prevention programs
implemented with fidelity (Area
3)
ƒ Prevention programs funded by
SDFSCA Program (Area 4)
Factors associated with
implementation with fidelity of
research-based programs (Area 6)

* The domains of variables are based on the conceptual framework (see Figure 1).

8

•

Compiling and screening lists. This step will focus on identifying programs that have
been added to lists since the previous study. While we had planned to include only those
lists that met explicit criteria on stringent inclusion criteria, those criteria are often
vague. We also became concerned that even lists that used less stringent criteria could
include both highly effective and less effective programs. Hence, to be comprehensive,
we will begin with a master list developed by Mihalic in 2007. 3 This master list
aggregates 12 existing lists of programs intended to prevent problem behavior. Even
though the 12 individual lists frequently overlap on programs, the master list contains
over 300 programs, some of which are not focused on school-based prevention. To
reduce the list to the potentially most relevant programs, two research staff will
independently screen each of the programs added to the master list on the following
criteria: (a) school based, (b) focused on prevention of ATOD use or school
crime/delinquency, and (c) applicable to school-age youth.

•

Reviewing literature and making judgments on programs. We plan to gather and
review research literature on the newly added programs that survive the screening
process; we also will gather and review any literature on the programs considered in the
review for the previous study. First, we will conduct extensive automated searches for
the published literature. Second, we will further screen the programs on whether they
had studies on them that met at least minimally acceptable methodological standards and
on whether mechanisms are in place to support widespread dissemination of the
programs. Third, two mid- to senior-level staff with strong methodology skills will
independently review each research document on the surviving programs. The
reviewers will use a standardized coding form to capture information from studies on
several methodological criteria (e.g., used experimental design or strong quasiexperimental design). Fourth, for the programs with two independent studies that meet
these criteria, we will conduct an in-depth assessment of the level of effectiveness
indicated by the studies. We will assess whether the preponderance of available
evidence supports the effectiveness of each program, and recommend that the programs
with this level of support be included on the list of research-based programs for the
study.

•

Determining the final list. Once the research-based programs have been identified, a
panel--of survey design experts and experts in the fields of youth ATOD use and school
crime prevention programs--will convene to review the selected programs. The panel
will be asked to examine the appropriateness of the programs selected for inclusion and
to suggest other programs that may have been missed through this process. Those
programs remaining on the list will form the basis of analysis for the Prevalence Study
and the Fidelity Study.

The Prevalence Study will consist of two separate data collections—a national survey of
schools and a national survey of districts. Results of the survey of schools will identify those schools that
have research-based programs. They are the schools that will participate in the Fidelity Study.
The school and district portions of the Study of the Implementation of the Safe and DrugFree Schools and Communities Act Program State Grants will be based on a national probability sample
3

See: http://www.colorado.edu/cspv/blueprints/matrixfiles/matrix.pdf.

9

of schools and conducted in several steps. Once the Identification Study identifies the research-based
programs, the next steps will be to survey schools with the “Prevalence Survey”—on whether researchbased programs are operating in the schools--and survey districts in which the responding schools are
located with the “District Survey”--on funding and district-wide prevention efforts.
Conducting the Prevalence Survey.

The school prevention program coordinator will

receive a questionnaire that will ask for information about youth ATOD use and school crime prevention
programs that operate in his/her school. Specifically, the respondent will be asked to identify all youth
ATOD use and school crime prevention programs from a list, to indicate whether the program is funded
by SDFSCA, and to identify the persons most knowledgeable about each program. The questionnaire
will be web based, with many skip patterns that can be easily navigated by the school prevention program
coordinator. For those coordinators without access to the web, a paper-and-pencil version of the
instrument will be provided. The results of this study will provide answers for the second research
question concerning the proportions of programs and schools with research-based programs, including
those programs receiving support from the SDFSCA Program.
Conducting the District Survey. Districts with schools that respond to the Prevalence
Survey Westat will be included in the District Survey. A district prevention coordinator in those districts
will be surveyed on whether each of the programs reported by schools in the Prevalence Survey is funded
by SDFSCA. The District Survey, which will be web-based, also will obtain information about districtlevel prevention policies and programming that may have an impact on a school’s prevention activities.
The Fidelity Study will consist of two separate data collections—a survey of schools and
interviews with developers of research-based programs. The survey of schools—“Provider Survey”--will
assess the degree to which the schools using research-based curriculum programs (as identified by the
Identification Study) implement them with fidelity to the research on which they are based. 4 The
interviews with the developers of those research-based curriculum programs will provide information on
the standards for implementing their programs.
Conducting the Provider Survey. Those schools with research-based curriculum programs
will be asked to complete a web-based program provider questionnaire. We will ask the program’s
representative to respond to a questionnaire that asks about a variety of program dimensions: program
goals/objectives, planning and training, and aspects of implementation of the program (e.g., content,
methods of delivery, extent of use, and degree of student exposure).
4

The previous study indicated that the majority of prevention programs implemented were curriculum-based

10

Interviewing Program Developers.

The developers of the research-based programs

identified in the Identification Study will be asked to provide information over the telephone on how their
programs should be implemented to achieve their intended objectives. This information will permit a
comparison of how a specific program is implemented in a school—based on the Provider Survey—
against standards for implementation—based on the interviews with program developers. If program
developers are unable or unwilling to provide information on their programs, the study will extract it from
program implementation materials.

A.1.2.

Instrumentation for Examining SDFSCA Programs
To provide an accurate and comprehensive picture of the implementation of SDFSCA

programs nationwide, the study will use tested data collection methodologies. The respondents and
instruments for each major study component are summarized in the following sections. Copies of the
draft instruments are included as appendices to this clearance request. After OMB approval is received,
data will be collected for the Prevalence Study in spring/summer 2009 and for the Fidelity Study in fall
2009. Table 2 provides information about the data collection activities to be conducted using these
instruments.

Instruments for the Prevalence Study
For the Prevalence Study, the respondents for the study are the school staff in the schools
who are most knowledgeable about prevention efforts in their schools and district prevention program
coordinators for the responding schools. The instruments for this study component are as follows. (See
Appendices A-1 and A-2 for copies of the instruments.)
•

Prevalence Questionnaire (one per school), and

•

District Questionnaire (one per district).

11

Table 2.

Data collection activities and schedule for Study of Implementation of Safe and DrugFree Schools and Communities Act Program State Grants
Number of
Data
respondents/
collection
responses
period
Prevalence Study
6,000 school
Spring
prevention
2009
program
coordinators
(i.e., school staff
in each sampled
schools who is
most
knowledgeable
about prevention
efforts in school)

Format and
length of
instruments

Topics covered

Web-based
survey with
telephone
followup; 30
minutes

Used to obtain information
on programs operated
during the 2008-09 school
year, contact information
for program coordinators

Appendix A2: District
Questionnaire

3,800 district
prevention
program
coordinators

Web-based
survey with
telephone
followup; 30
minutes

District characteristics;
planning, implementation,
and evaluation of district
prevention program;
SDFSCA funding

Appendix A3: Provider
Questionnaire

2,000 school
prevention
providers

Web-based
survey with
telephone
followup; 30
minutes

Quality of implementation
of research-based
prevention programs
(content, methods,
technical quality, extent of
use, degree of student
exposure); factors
associated with quality of
implementation of
research-based programs
(e.g., provider
characteristics)

Appendix A4: Program
Developer Protocol

Approximately
25 program
developers

Telephone
interviews;
45 minutes

Quality of implementation
of research-based programs
required (content, methods,
technical quality, extent of
use, degree of student
exposure)

Instrument

Appendix A1: Prevalence
Questionnaire

Springsummer
2009

Fidelity Study
Fall 2009

Fall 2009

12

Instruments for the Fidelity Study
The Fidelity Study will only be conducted in those schools that are found to have researchbased programs based on the results of the Prevalence Study. The respondents for the study are school
staff in the schools who are most knowledgeable about specific curriculum prevention programs in the
school. In addition, developers of the research-based prevention programs will be interviewed. The
instruments for this study component are as follows. (See Appendices A-3 and A-4 for copies of the
instruments.)
•

Provider Questionnaire (one per school), and

•

Program Developer Protocol (one per program).

All instruments to be used for this study are shown in Table 2, with the maximum number of
respondents, the data collection period, and the topics covered by that instrument.

A.2.

Purposes and Uses of the Data
The purpose of the study is to provide information on the extent to which the SDFSCA

Program is funding research-based programs for youth ATOD use and school crime prevention programs.
After determining the proportion of schools implementing research-based programs, this study will
examine the fidelity of the implementation of programming supported by SDFSCA funds.

A.3.

Improved Information Technology
We expect that most potential respondents have access to computers and are comfortable

participating in surveys via the Internet. For that reason and given the large numbers of respondents
expected, we anticipate using web-survey technology for each of the three survey instruments: Prevalence
Questionnaire, District Questionnaire, and Provider Questionnaire.

(Potential respondents who lack

access to or prefer not to respond via computers may complete and return paper versions of the
instruments.) However, the small number of program developer respondents expected does not warrant
using web-survey technology to gather information from them; hence, interviewers will use a paper
version of the Program Developer Protocol.

13

A.4.

Efforts to Identify Duplication
This study is part of ED’s comprehensive plan for evaluating the effectiveness of the

SDFSCA program.

The study has been designed specifically to address the gaps in the available

information regarding the quality of school-level programming supported by SDFSCA funds at the
district and school levels. The previous study on the prevalence and quality of implementation of
research-based programs (Study of the Implementation of Research-Based Programs to Prevent Youth
Substance Abuse and School Crime) provided baseline data on these issues; the proposed study will
provide follow-up data on the same issues. A review of literature indicates that no other research or
evaluation efforts have been undertaken.

A.5.

Methods Used to Minimize Burden on Small Entities
Burden is minimized for all survey respondents in districts and schools by keeping the

questionnaires relatively short (approximately 30 minutes completion time) and straightforward,
restricting questions to generally available information, and limiting the number of open-ended responses.
At each school, attempts will be made to avoid asking any one person will to complete more than two
questionnaires, for a total burden of approximately 60 minutes. Alternate contacts will be used as much
as possible for questionnaires on additional programs at the school. Additionally, no principal will be
asked to complete any questionnaire unless he or she has specifically asked to be the person to provide
information about a particular program and is not willing to designate anyone else.
Burden is minimized for the program developers by keeping the protocol relatively short
(approximately 45 minutes completion time) and straightforward. For example, the interviewer will
permit the respondent to direct the flow of the open-ended questions; this approach will make the task
simpler for the respondents and reduce the time required to complete the interview.

A.6.

Consequences of Less Frequent Collection
This study is authorized by Title IV of the ESEA.. Under Section 4121, the Act permits a

national evaluation of drug- and violence-prevention programming and provides funds for that purpose.
If this study is not conducted, ED will be severely hampered in its efforts to build upon previously
collected data in order to address important remaining programmatic issues (i.e., the quality of youth
ATOD use and school crime prevention programs in schools). Without the addition of evaluative data to
14

the previously collected descriptive data, ED will have difficulty fulfilling its mandate to support relevant
and effective prevention activities; it also will be unable to provide follow-up data for the SDFSCA
Program GPRA/PART measures.

A.7.

Special Circumstances for Data Collection
Data collection will be conducted in a manner consistent with all guidelines described in

5 CFR 1320.

A.8.

Compliance with 5 CFR 1320.8 and Consultations

A.8.1.

Compliance with 5 CFR 1320.8
The 60-day Federal Register notice for the study was published on November 26, 2008

(Volume 73, Number 229, page 72035). ED received no comments during the public comment period.

A.8.2.

Consultations
Consultation on the design of the study and on sampling issues has been provided on this

study. The two groups of consultations are identified as follows.

15

Consultation on Sampling
Consultation on sampling issues has been gathered from the following statisticians:
•

Adam Chu
Westat
1650 Research Boulevard, RE 442
Rockville, MD 20850
(301) 251–4326

•

Ralph DiGaetano
Westat
1650 Research Boulevard, RE 490
Rockville, MD 20850
(301) 294–2062

•

Pam Broene
Westat
1650 Research Boulevard, RE 484
Rockville, MD 20850
(301) 294–3817

Consultation on Study and Questionnaire Design
The Technical Work Group (TWG), which met on November 18, 2008, provided
consultation on the study and questionnaire design. The names, affiliations, contact information, and areas
of expertise of the TWG members are as follows.
•

Pierre Foy
Lynch School of Education
Boston College
Chestnut Hill, MA 02467
(617) 552-1600
Area of expertise: Statistical methods

•

Roger Tourangeau
Joint Program in Survey Methodology
University of Maryland
1218 LeFrak Hall
College Park, MD 20742
(301) 314-7911
Area of expertise: Survey methods

16

A.9.

•

Chris Ringwalt
Pacific Institute for Research and Evaluation
1516 E. Franklin Street, Suite 200
Chapel Hill, NC 27514-2812
(919) 265-2613
Area of expertise: Prevention research

•

Michael Furlong
The Gevirtz School
University of California, Santa Barbara
Santa Barbara, CA 93106-9490
(805) 893-3338
Area of expertise: Prevention research

•

David Wilson
Administration of Justice George Mason University
10900 University Boulevard, MS-4F4
Manassas, VA 20110-2203
(703) 993-4701
Area of expertise: Systematic review/meta-analysis

•

Harris Cooper
Duke University
249 Psych, Box 90086
Durham, NC 277
(919) 660-3167
Area of expertise: Systematic review/meta-analysis

Payment of Stipends to Participants
No stipends are offered for school staff for this study.

A.10.

Assurance of Confidentiality
Data results from this study will be presented in aggregate statistical form only. School

identifiers are collected, but no data will be released with individual identifiers attached, nor will names
of individuals or schools be used in any reports. A privacy statement is included in all cover letters
accompanying the questionnaires and on the instruction page in each questionnaire. This statement reads:
Responses to this data collection will be used only for statistical purposes.
Reports prepared for this study will summarize findings across the sample and
will not associate responses with a specific district, school, or individual.

17

Information that identifies the district, school, or respondent will not be
provided to anyone, except as required by law.
For the Prevalence Questionnaire, we will add the following caveat to the end of this
statement:
One exception is that school districts will be provided a list of prevention
programs identified by their schools as operating during the 2008-09 school
year.
We will request that respondents provide their names and contact information as this
information is crucial to the success of the data retrieval process (to collect missing information and
clarify responses) conducted after receipt of questionnaires from respondents. All personnel working on
the study will be required to sign the contractor’s confidentiality pledge (see Exhibit 1).
The contractor’s Institutional Review Board (IRB) reviewed this study. This IRB found that
the study was exempt from the scope of the human subjects regulations under 45 CFR 46.101 (b)(5) for
public service programs, as the study focuses on the agency and not the characteristics or opinions of the
study subjects.

A.11.

Justification for Questions of a Sensitive Nature
Questions included in the instruments for this study are not considered sensitive.

18

Exhibit 1.

Westat confidentiality pledge
WESTAT, INC.
EMPLOYEE OR CONTRACTOR’S ASSURANCE OF CONFIDENTIALITY
OF SURVEY DATA

Statement of Policy
Westat is firmly committed to the principle that the confidentiality of individual data obtained through Westat surveys must be protected.
This principle holds whether or not any specific guarantee of confidentiality was given at time of interview (or self-response), or whether or not
there are specific contractual obligations to the client. When guarantees have been given or contractual obligations regarding confidentiality have
been entered into, they may impose additional requirements which are to be adhered to strictly.
Procedures for Maintaining Confidentiality
1. All Westat employees and field workers shall sign this assurance of confidentiality. This assurance may be superseded by another
assurance for a particular project.
2. Field workers shall keep completely confidential the names of respondents, all information or opinions collected in the course of
interviews, and any information about respondents learned incidentally during field work. Field workers shall exercise reasonable caution to
prevent access by others to survey data in their possession.
3. Unless specifically instructed otherwise for a particular project, any employee or field worker, upon encountering a respondent or
information pertaining to a respondent that she/he knows personally, shall immediately terminate the activity and contact her/his supervisor for
instructions.
4. Survey data containing personal identifiers in Westat offices shall be kept in a locked container or a located room when not being
used each working day in routine survey activities. Reasonable caution shall be exercised in limiting access to survey data to only those persons
who are working on the specific project and who have been instructed in the applicable confidentiality requirements for the project.
5. Ordinarily, serial numbers shall be assigned to respondents prior to creating a machine-processible record and identifiers such as
name, address, and Social Security number shall not, ordinarily, be a part of the machine record. When identifiers are part of the machine data
record, Westat’s Manager of Data Processing shall be responsible for determining adequate confidentiality measures in consultation with the
project director. When a separate file is set up containing identifiers or linkage information which could be used to identify data records, this
separate file shall be kept locked up when not actually being used each day in routine survey activities.
6. When records with identifiers are to be transmitted to another party, such as for keypunching or key taping, the other party shall be
informed of these procedures and shall sign an Assurance of Confidentiality form.
7. Each project director shall be responsible for ensuring that all personnel and contractors involved in handling survey data on a project
are instructed in these procedures, have signed this pledge and comply with these procedures throughout the period of survey performance.
When there are specific contractual obligations to the client regarding confidentiality, the project director shall develop additional procedures to
comply with these obligations and shall instruct field staff, clerical staff, consultants, and any other persons who work on the project in these
additional procedures. At the end of the period of survey performance, the project director shall arrange for proper storage or disposition of
survey data, including any particular contractual requirements for storage or disposition. When required to turn over survey data to our clients,
we must provide proper safeguards to ensure confidentiality up to the time of delivery.
8. Project directors shall ensure that survey practices adhere to the provisions of the U.S. Privacy Act of 1974 with regard to surveys of
individuals for the Federal Government. Project directors must ensure that procedures are established in each survey to inform each respondent
of the authority for the survey, the purpose and use of the survey, the voluntary nature of the survey (where applicable) and the effects on the
respondents, if any, of not responding.
PLEDGE
I hereby certify that I have carefully read and will cooperate fully with the above procedures. I will keep completely confidential all
information arising from surveys concerning individual respondents to which I gain access. I will not discuss, disclose, disseminate, or provide
access to survey data and identifiers except as authorized by Westat. In addition, I will comply with any additional procedures established by
Westat for a particular contract. I will devote my best efforts to ensure that there is compliance with the required procedures by personnel whom
I supervise. I understand that violation of this pledge is sufficient grounds for disciplinary action, including dismissal. I also understand that
violation of the privacy rights of individuals through such unauthorized discussion, disclosure, dissemination, or access may make me subject to
criminal or civil penalties. I give my personal pledge that I shall abide by this assurance of confidentiality.
Signature
Date

19

A.12.

Estimates of Burden Hours
Table 3 presents Westat’s estimates of the annual response burden for the Study of

Implementation of the Safe and Drug-Free Schools and Communities Act Program State Grants.

A.13.

Estimates of Annual Cost Burden to Respondents
This study will not require respondents to record and maintain additional data outside their

current requirements. Hence, no additional hour burden on the part of school personnel will be required.
No equipment costs will be incurred by participating schools as a result of this study.
Table 3.

Estimates of annual response burden

Instrument

Number of
responses

Prevalence Questionnaire
District Questionnaire

6,000
3,800

Provider Questionnaire
Program Developer Protocol
Total (for all Prevalence and
Fidelity Study questionnaires)

2,000
25

A.14.

11,825

Length of
instrument
Frequency
Prevalence Study
30 min.
1
30 min.
1
Fidelity Study
30 min.
1
45 min.
1
N/A

N/A

Total
burden

Hourly
rate

Total
cost

3,000 hrs.
1,900 hrs.

$75
$75

$225,000
$142,500

1,000 hrs.
19 hrs.

$30
$75

$30,000
$1,425

5,919 hrs.

N/A

$398,925

Estimates of Annualized Cost to the Federal Government
The full study is estimated to cost the federal government $2.4 million for contractual

services, over 2 years. Hence, the annualized cost related to data collection, data analysis, and reporting is
$1.2 million.

These costs include personnel, benefits, overhead, supplies, and indirect costs.

The

estimate is based on expected time to produce, train on, and administer the instruments, and to analyze
and report the data. These estimates are grounded in Westat’s previous experience managing data
collection efforts of this type.

20

A.15.

Changes in Burden
This request is a reinstatement with change of a previously approved collection for which

approval has expired (OMB #1875-0216). Three of the four instruments are shorter versions of those
previously approved. The fourth instrument (Program Developer Protocol) is new; it adds very little to
the overall burden, because it will be used with only approximately 25 respondents. The change in burden
is summarized in Table 3.

A.16.

Publication Plans and Time Schedule
In this section, we summarize the time schedule for data collection, and our analysis plans

for the study.

A.16.1.

Time Schedule
Data collection for the Prevalence Study—which includes the Prevalence Survey

and

District Survey—is scheduled for spring 2009, after OMB approval is received (see Table 4). Sample
selection and recruitment for these surveys will occur during winter 2009. Results from the Prevalence
Study will be used to select schools and programs for the Fidelity Study.

A.16.2.

Data Analysis Plans
For the analysis of survey data, four types of analytic techniques will be employed: (a) data

reduction techniques (e.g., composite/scale score building); (b) psychometric analysis (e.g., reliability and
validity analysis); (c) descriptive statistics (e.g., measures of central tendency, dispersion, maximum and
minimum values, and frequencies); and (d) inferential statistics (e.g., cross-tabulations, t-tests, and
regression analyses). In this section, the techniques and relevant examples are presented for the study
questions.
Data Reduction Techniques. The first step in conducting the data analysis for the current
study will be to review the data for variables that need to be combined to simplify the analysis and
enhance the psychometric properties of the measures. Data reduction will be based on the techniques
used in earlier studies, in which the variable of the same underlying construct are then combined into
21

smaller components (i.e., scales or composite measures). These components will link directly to the
domains in our conceptual framework (see Figure 1).
Table 4.

Anticipated project schedule
Task

Begin systematic reviews

2008
Identification Study
October

2009

Initial findings on systematic reviews

July

Begin meta-analysis

March

Initial findings on meta-analysis

July

Final report on systematic reviews, metaanalysis*

November

2010

Prevalence Study
Notify districts and schools

January, February

Prepare forms for data collection

February, March

Begin data collection

March

End data collection

September

Initial findings

October

Final study report*

July

Deliver data files

August
Fidelity Study

Begin data collection

September

End data collection

February

Initial findings

February

Final study report*

July

Deliver data files

August

*Two final reports will be issued, one on the systematic reviews and meta-analysis and one on the Prevalence Study and Fidelity Study.

22

The composite measures will be made up of several different questions from a specific
questionnaire and will usually possess higher reliability and validity than single questionnaire items.
Combining related variables can greatly facilitate data analysis and the interpretation of the results. As an
example, several separate survey questions to measure quality of training are combined into one
composite measure, the Quality of Training scale. Each respondent will then have a Quality of Training
score.
In some cases, not all of the items measuring the same construct (e.g., quality of training)
will share an adequate amount of variance. Items that do not share variance with the other items
measuring quality of training will not be used in the composite score because they either contribute
insignificant amounts of information and/or have low construct validity.
Psychometric Analysis. Psychometric analysis will be performed next to determine the
reliability and validity of the measures. Internal consistency measures such as Cronbach’s alpha and/or
split-half correlations will be examined to determine the reliability of the measures.
Valid composite measures are those that measure what they are supposed to measure.
Convergent and discriminant validity will be emphasized for this study. Convergent validity involves
examining the relationship between measures that should be related to one another, while discriminant
validity involves examining the relationship between two measures that should not be related to one
another. To conduct this type of analysis, we will construct and examine correlation matrices for the
relevant measures.
The final report for the national surveys will include a table showing the composite measures
and the questionnaire items that make up each measure, and the reliability and validity of the items.
Single item indicators will also be shown in this list of measures. Composite measures found to have
poor psychometric (reliability and validity) properties will not be used in the descriptive or inferential
data analysis. This step will require reevaluation of the variables included in the domains identified in the
conceptual framework. Some variables may have to be reconstructed or eliminated.
Descriptive Statistics. Descriptive statistics of composite and/or scale scores, which will be
employed to describe characteristics of the prevention activities, schools, and the districts, are useful for
answering questions regarding ranges of values, averages, counts, and percentages. These statistics
include frequencies and measures of central tendency (such as the mean or median) for individual
questionnaire items and composite scores. The descriptive statistics will also be broken down by school
level, urbanicity, and SDFSCA funds; district/school size; and other variables important to the study.
23

Other statistics that will be produced for both individual items (when appropriate) and composite
measures include the standard error and minimum and maximum values.
An example of the descriptive statistics that we will use pertains to measuring the fidelity of
implementation of research-based programs. For each specific research-based program (e.g., Life Skills
Training), we plan to compare the developer’s specifications for implementing the program against the
survey findings on that program. For example, we might report the developer’s requirement for number
of sessions implemented and the mean number of sessions actually implemented based on the Fidelity
Study. This analysis would indicate the extent to which the programs were implemented as intended by
the developers who created them.
Inferential Statistics. Statistics calculated to determine whether a statistical relationship
has occurred by chance are called inferential statistics. While some of the study questions are easily
answered using descriptive statistics, others can only be answered using inferential statistics. These types
of statistics vary in complexity, with cross-tabulations representing the simplest type and multiple
regressions representing the more complex. The type of inferential statistic used will depend on the study
question and the type of data.
An example of the inferential statistics that we will use continues the example on measuring
the fidelity of implementation of research-based programs. We could go one step further to examine the
extent to which the mean number of sessions implemented differs by district or school characteristics. In
this case, we could compare the means between two groups (e.g., middle and high schools) using t
statistics or compare the means among several groups (e.g., urban, suburban, and rural schools) using F
statistics. We also are likely to develop multivariate models that consider the relationship between a
measure of fidelity of implementation and several independent variables, including school characteristics
and the “predictors of program quality” that we highlight in our conceptual framework. By examining
inferential statistics for the individual independent variables, we will be able to gauge their relative
strength of association with the measure of fidelity.
When possible, a straightforward approach such as cross-tabulations will be used to address
study questions rather than a more complicated technique, since results from the former are often easier to
understand and communicate. Using data developed through these different types of analysis, we will
prepare draft, interim, and final reports.

24

A.17.

Approval Not to Display Expiration Date for OMB Approval
This section is not applicable to the current study. The expiration date for OMB will be

displayed on all survey instruments.

A.18.

Adherence to the Guidelines in 5 CFR 1320.9
No exceptions to the certification statement identified in Item 19, “Certification for

Paperwork Reduction Act Submissions,” of OMB Form 83-1 are requested.

25