Att_Supporting Statement for 2nd OMB_Section B (01-26-09)_Final

REQUEST FOR CLEARANCE
STUDY OF THE IMPLEMENTATION OF THE SAFE AND DRUG-FREE SCHOOLS
AND COMMUNITIES ACT PROGRAM STATE GRANTS
SECTION B. DESCRIPTION OF STATISTICAL METHODOLOGY

B.1

Respondent Universe
The Study of the Implementation of the Safe and Drug-Free Schools and Communities Act

Program State Grants has two separate populations of inference corresponding to the Prevalence Study
and the Fidelity Study. For the Prevalence Study, the target population will include what are essentially
all “regular” public elementary and secondary schools in the United States with the exception of a few
types of schools. The target population for the Fidelity Study will be the subset of public elementary and
secondary schools that have implemented research-based prevention programs (i.e., programs intended to
prevent youth ATOD use and school crime) during the 2008-09 school year. A school will be asked to
participate in the Fidelity Study if it both participates in the Prevalence Study and offers one of the
research-based prevention programs identified during the Identification Study component of the study.

B.1.1

Schools and Districts
The respondent universe for the Prevalence Study will consist of public schools that provide

instruction in any of grades 1 through 12 and are located in the 50 states and the District of Columbia.
The 2006–07 NCES Common Core of Data (CCD) Public Elementary and Secondary School Universe
file will be used to construct the school sampling frame. Although the 2006-07 CCD file has not yet been
released by NCES, it is expected to be available in early 2009. However, rather than starting directly from
the CCD files, the 2006-07 National Assessment of Educational Progress (NAEP) national sample frame
(which will be derived from the 2006-07 CCD file) will be used if it is available. The advantage of using
the NAEP frame is that it will have undergone many edits to eliminate closed and other types of
“ineligible” schools (e.g., vocational schools with no enrollment, and ungraded, special education,
hospital, and prison schools). In addition to the types of schools already eliminated from the NAEP
sampling frame, other types of schools that are ineligible for the Prevalence Study will be eliminated as
part of the establishment of the sampling frame. These include state-run schools, federal Department of
Defense and Bureau of Indian Affairs schools, schools with a grade no higher than kindergarten, and
schools outside the 50 states and the District of Columbia. Certain types of ineligible schools cannot be
26

identified in advance of sampling and must be eliminated later when schools and school districts are
contacted about participation in the study. For example, only those sampled vocational-technical schools
whose students attend only the vocational-technical school will be considered eligible for the study. As
indicated in Table 5, an estimated 86,000 schools are expected to be included in the final sampling frame.
The corresponding respondent universe for the District Survey (which is part of the
Prevalence Study) will include those public school districts with at least one school meeting the eligibility
criteria described above. The district survey will collect information about district-level policies and
programs that will be used to confirm whether the research-based programs reported by their schools have
been implemented and whether the reported research-based programs have received SDFSCA program
funding. Although data from the District Survey will be used to characterize the research-based programs
reported by schools in the Prevalence Survey, the study has no plans to develop separate district-level
estimates from the survey.

B.1.2

Research-Based Prevention Programs
The respondent universe for the Fidelity Study will consist of those public elementary and

secondary schools that have implemented one or more eligible research-based prevention programs
during the 2008-09 school year. The list of eligible research-based programs will be developed in the
Identification Study. Compilation of this list will start with the over 300 potentially eligible prevention
programs assembled for the previous Study of the Implementation of Research-Based Programs to
Prevent Youth Substance Abuse and School Crime. 1 Any additional SDFSCA-relevant programs
identified by examining external sources will be added to this list. Consistent with the approach used in
the previous study, each potentially eligible program will be screened to determine whether it is (a)
entirely school-based or has separable school-based components, (b) focused on prevention of youth
ATOD use or school crime, and (c) applicable to school-age youth. Those programs meeting these criteria
will be eligible for the Provider Survey (which is part of the Fidelity Study).

1

Crosse, S., Williams, B., Hagen, C., Harmon, M., Ristow, L., DiGaetano, R., Broene, P., Alexander, D., Tseng, M., and Fong, M. (2008, under
review). Study of the Implementation of Research-based Programs to Prevent Youth Substance Abuse and School Crime: Final report.
Rockville, MD: Westat.

27

Table 5.

Distribution of schools in 2005-06 CCD file by instructional level, metropolitan status,
and percent minority enrollment
Instructional
level*
Elementary

Middle

High

Metropolitan
status

Percent minority
enrollment†

City

0 to 10%
11 to 60%
Over 60%

1,311
5,562
7,617

Suburban/Town

0 to 10%
11 to 60%
Over 60%

8,528
9,712
4,677

Rural

0 to 10%
11 to 60%
Over 60%

10,032
4,007
1,017

City

0 to 10%
11 to 60%
Over 60%

324
1,509
1,941

Suburban/Town

0 to 10%
11 to 60%
Over 60%

2,828
3,030
1,181

Rural

0 to 10%
11 to 60%
Over 60%

2,771
1,316
335

City

0 to 10%
11 to 60%
Over 60%

355
1,548
1,862

Suburban/Town

0 to 10%
11 to 60%
Over 60%

3,031
2,720
866

Rural

0 to 10%
11 to 60%
Over 60%

5,577
2,000
447

Total
*

Number of
schools in 200506 CCD**

86,104

Elementary: schools with a low grade of 1-3 and a high grade of 8 or less, or with a low grade of 4-6 and a high
grade of 6 or less. Middle: schools with a low grade of 1-3 and a high grade of 9 or 10, or with a low grade of 47 and a high grade of 10 or less, or with a low grade of 8-9 and a high grade of 9 or less. High: schools with a
low grade of 1-7 and a high grade of 11-12, or with a low grade of 8-12 and a high grade of 10 or higher.

†

Percent of students in the school who are black or Hispanic.

** Counts from the 2005-06 CCD file are given for illustration. The actual sampling frame will be constructed
from the 2006-07 CCD file (or NAEP sampling frame if it is available).

28

B.2

Stratification and Sample Selection
Stratification has two main (and sometimes conflicting) purposes. The first is to improve

overall sampling precision. The second is to help ensure that certain key subsets of the population are
adequately represented in the sample for subgroup analyses. Hence, we propose to design the sample to
permit analysis of subgroups of schools defined by instructional level, metropolitan status, and percent
minority enrollment. Stratification by metropolitan status and percent minority is important because,
based on earlier studies, issues related to ATOD use and school crime are expected to be correlated with
these variables. 2 Stratification by instructional level is important because research-based prevention
programs are known to differ appreciably by instructional level. Hence, a total of 27 school strata will be
formed by a cross-classification of three categorical variables with three values each: instructional level
(elementary, middle, and high), metropolitan status (central city, other urban including suburban, rural),
and percent minority (defined as the percentage of students who are black or Hispanic: 0-10% minority,
11 to 60% minority, over 60% minority). Within a given stratum, schools will be sampled at rates
designed to achieve specified levels of precision for the major analytic domains of interest. In the
following sections, we summarize the sampling methodology for the study.

B.2.1

Selection of School and District Samples
Although data will be collected from staff who coordinate youth ATOD use and school

crime prevention activities at the district level, the primary focus of the study is on the implementation of
research-based youth ATOD use and school crime prevention programs in schools. Hence, the sample
design will be geared toward producing precise school-level estimates. Specifically, we will select a
stratified sample of approximately 6,000 public schools that avoids undue clustering by district and that
achieves minimum precision levels for selected domains of interest. The resulting stratified school sample
will then be used to identify the associated sample of districts. We estimate that the proposed stratified
sample of 6,000 schools will be associated with approximately 3,800 unique districts.
The target school sample size of 6,000 will be allocated across the 27 categories (i.e., strata)
indicated in Table 6 with the goal of achieving two objectives: to maximize the precision of estimates,
and to keep the precision roughly constant across the marginal levels of the three school stratification
2

For examples of support, see: http://www.cdc.gov/healthyyouth/yrbs/pdf/yrbs07_us_disparity_race.pdf;
http://www.oas.samhsa.gov/2k4/ruralYouthAlc/ruralYouthAlc.pdf;
http://nces.ed.gov/programs/crimeindicators/crimeindicators2007/ind_02.asp;
http://nces.ed.gov/programs/crimeindicators/crimeindicators2007/figures/fig_02_2.asp; and
http://nces.ed.gov/programs/crimeindicators/crimeindicators2007/ind_18.asp.

29

variables (i.e., level, metropolitan status, and minority status). The sampling rates within the 27 strata will
be set so as to obtain an overall sample size of approximately 6,000 schools. Note that within the 27
primary strata indicated in Table 6, further stratification (either implicitly through sorting or explicitly
through formation of detailed substrata) may be employed. For example, sorting the schools by percent of
students eligible for free/reduced-price lunch prior to sampling will induce an implicit stratification that
will help ensure that all income levels are appropriately represented in the sample. Similarly,
substratification by enrollment size will ensure that schools of all sizes are represented in the sample.
However, in the case of enrollment size, formation of explicit size classes for sampling
purposes rather than implicit stratification might be preferable because it would permit the use of
differential sampling rates by size class. In particular, this would allow the larger schools in a stratum to
be oversampled if desired. Where differential sampling rates are used, the weights of the responding
schools will be adjusted accordingly to reflect the overall probabilities of selection (see Section B.2.4).
While the use of differential sampling rates will increase the variation in weights and design effects for
broad subgroups of schools, it is necessary to ensure that the resulting sample sizes for key subgroups are
adequate to meet the overall analytic objectives of the study.
Assuming an 85 percent response rate and a 95 percent eligibility rate, an initial sample of
6,000 schools will yield approximately 4,800 eligible responding schools for the Prevalence Study. Table
6 summarizes the sample sizes to be expected under the proposed design.

30

Table 6.

Expected samples for the Prevalence Study by level, metropolitan status, and percent
minority enrollment

Instructional
level
Elementary

Middle

High

Metropolitan
status

Percent minority
enrollment

City

0 to 10%
11 to 60%
Over 60%

81
51
47

65
41
38

Suburban/Town

0 to 10%
11 to 60%
Over 60%

310
254
199

250
205
161

Rural

0 to 10%
11 to 60%
Over 60%

404
403
493

326
325
398

City

0 to 10%
11 to 60%
Over 60%

322
195
168

260
157
136

Suburban/Town

0 to 10%
11 to 60%
Over 60%

315
251
188

254
203
152

Rural

0 to 10%
11 to 60%
Over 60%

167
189
200

135
153
162

City

0 to 10%
11 to 60%
Over 60%

619
226
204

500
182
165

Suburban/Town

0 to 10%
11 to 60%
Over 60%

283
108
88

229
87
71

Rural

0 to 10%
11 to 60%
Over 60%

113
57
65

91
46
52

6,000

4,845

Total

B.2.2

Number to be
sampled

Expected
number of
eligible
respondents

Selection of Research-Based Prevention Programs
As noted in the previous section, the focus of the Prevalence Study will be to identify

schools with eligible research-based prevention programs. In the Study of the Implementation of
Research-Based Programs to Prevent Youth Substance Abuse and School Crime, 42 percent of
elementary schools, 46 percent of middle schools, and 33 percent of high schools indicated that they used

31

research-based programs in the 2004-05 school year. However, among the roughly 3,000 programs that
were subsampled for the follow-up study of program characteristics, about 44 percent did not meet the
eligibility criteria for the study. This left about 900-1,000 eligible research-based programs. Moreover,
because of the subsampling of programs in the previous study, the researchers were not always able to
characterize responding schools as having one or more eligible programs. For example, if a school had
eight programs and four were subsampled for the follow-up study and all four turned out to be ineligible,
one still could not definitively conclude that the school did not have an eligible research program. To
avoid the potential for understating the prevalence of schools with eligible programs, all programs
identified by the responding schools (rather than a subsample) will be included in the proposed Fidelity
Study.
The prevalence estimates from the Study of the Implementation of Research-Based
Programs to Prevent Youth Substance Abuse and School Crime suggest that an initial sample of 6,000
schools will yield approximately 2,000 schools reporting research-based programs (assuming a schoollevel response rate of 85 percent and school eligibility rate of 95 percent). In Table 7, we present
estimates of the expected numbers of schools by selected subgroups assuming a total initial sample size of
6,000 schools and sampling rates similar to those used in the previous study. This table is intended to
illustrate the rough orders of magnitude of the sample sizes to be expected under the proposed design. The
actual sample sizes will depend on the final design to be specified for the study and may differ from those
shown in the table.

32

Table 7.

Expected sample sizes based on an initial sample of 6,000 schools and prevalence rates
reported in the Study of the Implementation of Research-Based Programs to Prevent
Youth Substance Abuse and School Crime*

Subgroup of sample

Number of schools to
be sampled

Expected number of
responding schools

Expected number of
schools reporting
research-based programs

Total sample

6,000

4,845

2,000

Instructional level
Elementary
Middle
High

2,242
1,995
1,763

1,810
1,611
1,424

764
762
473

Minority status
0 to 10%
11 to 60%
Over 60%

2,614
1,734
1,652

2,111
1,400
1,334

816
613
571

Metropolitan status
Central city
Suburban
Rural

1,913
1,996
2,091

1,545
1,612
1,688

670
650
680

Size
< 300
300-999
1000+

1,426
3,723
851

1,151
3,006
687

434
1,286
280

Free Lunch
< 25%
26-55%
55%+

1,853
2,034
2,113

1,496
1,642
1,707

529
696
775

* Crosse, S., Williams, B., Hagen, C., Harmon, M., Ristow, L., DiGaetano, R., Broene, P., Alexander, D., Tseng, M., and Fong, M. (2008, under
review). Study of the Implementation of Research-based Programs to Prevent Youth Substance Abuse and School Crime: Final report.
Rockville, MD: Westat.

B.2.3

Expected Levels of Precision
In Table 8, we present the 95 percent confidence interval half-widths for a prevalence

estimate of 50 percent that would be expected using the proposed stratification and sample allocation
given in Table 6. These results provide a rough indication of the sampling precision to be expected from
the proposed design. We based the design effects used to calculate the effective sample size on actual
design effects obtained from the previous study for the estimated proportion of schools with a researchbased program. The design effect, which is defined to be the ratio of the variance of an estimate based on
a disproportionate stratified sample to the variance of an estimate based on a self-weighting sample of the
33

same size, primarily reflects the variation in weights resulting from the disproportionate allocation of
schools to strata under the proposed design. To a lesser extent, it also reflects the impact of the differential
nonresponse weighting adjustments described in Section B.2.4.
Table 8.

95 percent confidence interval half-widths for an estimate of a 50-percent characteristic
Expected number of
responding, eligible
schools (n)

Effective
school sample
size (n/deff)*

95% CI
half-width
for P=50%

Instructional level
Elementary
Middle
High

1,810
1,611
1,424

1,509
1,342
1,186

2.59%
2.69%
2.88%

% Minority
0-10%
11-60%
60%+

2,111
1,400
1,334

1,759
1,167
1,112

2.38%
2.93%
2.98%

Metro status
Central City
Suburban
Rural

1,545
1,612
1,688

1,287
1,343
1,407

2.79%
2.69%
2.69%

Size
<300
300-999
1000+

1,151
3,006
687

959
2,505
573

3.27%
2.02%
4.13%

% Eligible
free/reducedprice lunch
< 25%
26-55%
55%+

1,496
1,642
1,707

1,247
1,368
1,423

2.79%
2.69%
2.69%

Total

4,845

3,461

1.73%

*Average design effect (deff) for category = 1.2; deff for total=1.4.

B.2.4

Estimation Procedures
For estimation purposes, sampling weights reflecting the overall probabilities of selection

will be attached to each data record. These weights will include upward adjustments for nonresponse at
both the school and program levels. To compensate for school nonresponse, weight adjustment factors
will be computed within subgroups or “cells” defined by the 27 school strata and other school-level
variables, such as percent of students eligible for free/reduced price lunch and enrollment size. The

34

adjustment factor to be applied to the school base weight will be computed as the ratio of the weighted
count of schools in the sample to the corresponding weighted count of the responding schools. The
adjustment will have the effect of distributing the weight of the nonresponding schools in the cell to the
responding schools, hence bringing the total weight of the responding schools to the level of the original
sample. Adjustments for program nonresponse will be handled in a similar manner. Since all eligible
programs identified in the Prevalence Study will be included in the study, the “base” weight for a program
is simply the corresponding nonresponse-adjusted school weight. In this case, the adjustment cells to be
used to compensate for program response will be defined by relevant program-level variables, such as
program focus (e.g., ATOD use, school crime, or both), in addition to the school-level variables
mentioned earlier.
To properly reflect the complex features of the sample design, standard errors of the surveybased estimates will be calculated using jackknife replication. Under the jackknife replication approach,
50-100 subsamples or “replicates” will be formed in a way that preserves the basic features of the full
sample design. A set of estimation weights (referred to as “replicate weights”) will then be generated for
each jackknife replicate. Using the full sample weights and the replicate weights, estimates of any survey
statistic can be calculated for the full sample and each of the jackknife replicates. The mean square error
of the replicate estimates then provides a measure of the variance (standard error) of the survey statistic.

B.3.

Methods to Maximize Response Rates and Deal with Nonresponse
In this section, we discuss specific methods that will be used to maximize response rates and

the procedures that will be used to deal with nonresponse for each of the major components of the study.
Key strategies for maximizing response include: (a) comprehensive recruitment of schools with
notification of SEAs and districts to mitigate against potential nonresponse at the school level; (b) use of
well-tested procedures and experienced staff for completing applications to conduct research in “special
clearance” districts; (c) use of a web-based questionnaires whenever feasible, which provides
convenience and is likely to reduce the time spent answering survey questions; and (d) extensive
telephone follow-up of survey and item nonresponse, including the use of experienced telephone center
staff who will help convert initial and/or repeated refusals.

Westat has substantial experience in

administering national education surveys. We expect that the response rate for this study will be 85
percent based on previous experiences with national surveys of school prevention programs.

35

B.3.1.

Survey Response
Recruitment efforts will begin with an introductory letter to Chief State School Officers,

State Prevention Coordinators, and District Superintendents encouraging district and school participation
from a senior ED official. Negotiations with schools and special clearance districts (i.e., districts with a
formal application and review process for requests to participate in studies) about participating in the
survey will stress the legitimacy of the overall study and emphasize the importance of their participation
in this particular study component. Westat will contact individual districts in the study sample to inform
them about the surveys. Next, Westat will obtain approval from each sampled school’s principal, and
work with the principal or the principal’s designee in the school to help identify appropriate school-level
respondents. Contact and descriptive information for these potential respondents will be entered into the
study’s database for use in producing survey materials such as letters and labels, as well as for tracking
the progress of the survey.
Highly trained telephone interviewers will follow-up with the respondents who do not
submit a completed survey within a three-week timeframe. These staff have been very successful in
negotiating participation for many education and prevention-related studies such as the National Center
for Education Statistics’ (NCES) Fast Response Survey System, the NCES School Survey on Crime and
Safety, the National Study on School Violence and Prevention, and the Study of the Implementation of
Research-Based Programs to Prevent Youth Substance Abuse and School Crime. Each staff member will
be assigned to monitor a particular set of schools and will retain those contacts throughout the data
collection period. This continuity of support has proven very successful in gaining and maintaining
rapport with busy school and program administrators who do not have the time to re-explain their
problems or questions every time they call.

B.3.2.

Item Response and Data Quality
To ensure data quality, manual editing will be performed directly on the survey response

forms. For data provided on web-based questionnaires, edits will be performed in real time by special
computer software that is programmed with built-in data checks.

For data provided on paper

questionnaires, manual edits are designed to check each document for completeness, inter-item
consistency, extraneous remarks, and proper adherence to any skip instructions; range checks (checks on
responses beyond the anticipate range of response) will also be performed at this time. Whenever
possible, sources outside the survey will be used to aid in checking data for accuracy and consistency.

36

Although these procedures are designed to maximize item response rates, the analysis will
need to confront the issue of missing data.

Experience with similar surveys indicates that some

respondents will omit responses to some specific items (e.g., those viewed as reflecting negatively on a
program), although they may have provided most of the data requested. By employing good survey data
collection practices, including use of respondent contact information to conduct follow-up, the amount of
missing data on any single variable will be minimized–-still the most desirable solution. Where missing
data still cannot be obtained, for analyses involving just one or two variables, the problem will be handled
by omitting the cases with missing data, or, in the case of categorical response variables, by using an
explicit “missing” or “unknown” category.

B.4

Tests of Procedures and Methods
In December 2008 and January 2009, Westat conducted pretests on the instruments for the

study’s surveys. The pretest participants included personnel in six schools along with the corresponding
district officials for those schools. The sample of participating schools, which was selected from the
CCD, represented a range of instructional levels (elementary, middle, and high schools) and district sizes
(large student enrollment and small to medium student enrollment). As a result of this process, all of the
pretested instruments were revised as were some of the planned survey procedures.

B.5.

Statistical Consultation and Implementation of the Study
The following statisticians were consulted on the statistical aspects of the design and

analysis of the study:
„

Adam Chu, Westat;

„

Ralph DiGaetano, Westat; and

„

Pam Broene, Westat.

The study is being conducted by Westat, Battelle Memorial Institute, and ISA Group, under
contract with ED (Contract Number ED-04-CO-0059).

37

LIST OF APPENDICES

A.

Proposed Instruments
Prevalence Questionnaire
District Questionnaire
Provider Questionnaire
Program Developer Protocol

B.

Proposed Notification Letters and Related Materials
Letter to Chief State School Officer
Letter to State Coordinator of Safe and Drug-Free and Communities Act Program
Letter to District Superintendent
Letter to District Coordinator
Web Survey Information Sheet
Letter to Principal
Letter to Program Provider

38