Feasibility Study

attachment_7__final_version__9.22.2008.pdf

Drug Free Communities Support Program National Evaluation

Feasibility Study

OMB: 3201-0012

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 3201-0012 can be found here:

Document [pdf]

Download: pdf | pdf

Attachment 7
Feasibility Study

DRAFT

REPORT ON THE FEASIBILITY OF ACCESSING
SCHOOL-BASED SUBSTANCE ABUSE DATA FOR THE
DRUG FREE COMMUNITIES SUPPORT PROGRAM
NATIONAL EVALUATION
January 19, 2007

By:

BATTELLE

In Collaboration With

The Association for the Study and
Development of Community
444 N. Frederick Avenue
Suite 315
Gaithersburg, MD 20877

Ballelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

Table of Contents
1. Background and Introduction ......................................................................................... 3
2. Goals of Feasibility Assessment ..................................................................................... 4
3. Data Collection Method .................................................................................................. 4
4. Findings ........................................................................................................................... 5
4.1 Available Questionnaires .......................................................................................... 5
State-Specific Questionnaires ..................................................................................... 5
National Youth Risk Behavior Survey (YRBS) ......................................................... 5
Communities That Care (CTC) Youth Survey ........................................................... 5
PRIDE Risk and Protective Factors Survey ................................................................ 5
Youth Tobacco Survey ............................................................................................... 6
Center for Substance Abuse Prevention's (CSAP) Substance Abuse Risk and
Protective Factor Survey ............................................................................................. 6
Bach-Harrison Youth Survey ...................................................................................... 6
Frequency Of Questionnaire Use By States ................................................................ 7
4.2 Data Utility ................................................................................................................ 8
Adequate Coverage of Core Measures ....................................................................... 8
Data Collection by Grade ............................................................................................ 9
Frequency of Data Collection ................................................................................... 10
Unit of Analysis ........................................................................................................ 10

Feasibility Study for the DrugFree Communities Support Program National Evaluation
January 19, 2007

Baltelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

4.3 Availability And Quality Of Data ........................................................................... 11
Feasibility of Using Sub-State Level Data ................................................................ 11
State Level Data ........................................................................................................ 13
5. Feasibility Study Results: Issues and Challenges ......................................................... 15
6. Recommendations ......................................................................................................... 16
References ......................................................................................................................... 18
Appendix A Comprehensive List of Questionnaires Administered by State ................... 19
Appendix B Frequency of Survey Administration by State ............................................. 22
Appendix C Data Availability, Data Format, and Smallest Unit of Available Data by
State ................................................................................................................................... 24

Feasibility Study for the DrugFree Communities Support Program National Evaluation
Januaiy 19, 2007

Ballelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

1. Background and Introduction
The Drug Free Communities (DFC) Support Program is a community-based substance
abuse prevention effort funded by The Executive Office of the President, Office of
National Drug Control Policy (ONDCP) and managed by the Substance Abuse and
Mental Health Services Administration (SAMHSA). The DEC National Evaluation is
conducted by the Battelle Institute in collaboration with the Association for the Study and
Development of Community (ASDC). The goal of the National Evaluation is to assess
the effectiveness of substance abuse prevention coalitions ("DFC coalitions"), and as
such, school-based data provide the best and most reliable source of information. The
National Evaluation intends in this study to determine substance abuse trends among
youth in grades 6 to 12 at the local, state, and national levels by combining data collected
by the DFC coalitions with substance abuse data collected from the 50 states and the
District of Columbia. The DFC National Evaluation is specifically interested in
questionnaires that capture the four core measures for youth substance abuse. These
measures are: 1) past 30-day use, 2) average age of onset, 3) perception of risk, and 4)
perception of parental disapproval, for alcohol, tobacco, and marijuana.
The DFC National Evaluation relies on data provided by DFC Support Program grantees
to address the core measures. Grantees are required by ONDCP to collect and report data
on the Government Performance and Results Act (GPRA) core measures for youth
substance abuse every two years, minimally, and for at least two middle or high school
grades. Preliminary analysis of the grantees' data indicates that a significant number of
grantees are unable to provide complete or reliable community- or school-level data that
meet the GPRA requirements. Concerns about the quality, consistency, and reliability of
the data provided by the grantees have been raised by the Battelle Institute, External
Review Group for the National Evaluation, and the Office of Management and Budget.
To address these concerns, the Battelle Institute conducted a feasibility assessment as part
of its scope of work to determine if consistent and reliable community- and school-level
data on the four core measures could be collected from more reliable sources at the state
and national levels.
Between May and July 2006 ASDC, on behalf of the Battelle Institute, conducted
telephone interviews with state data managers and/or administrators of school-based
surveys that target students in 6th through 12th grades from 50 states and the District of
Columbia. These contacts were identified by a representative of each state's principal
substance abuse prevention agency. This report describes the data collection process,
findings, challenges, and recommendations for accessing the required data directly from
states and other sources in order to improve the quality of data available to the DEC
National Evaluation.

Feasibility Study for the Drug-Free Communities Support Program National Evaluation
January 19, 2007

Ballelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

2. Goals of Feasibility Assessment
The goals of the feasibility assessment are to:
•

•

Determine if all 50 states and the District of Columbia collect complete and reliable
community- and school-level data on youth substance abuse that capture the four core
measures for alcohol, tobacco and marijuana; and
Determine if the data can be made available to the DFC National Evaluation.

The Battelle Institute identified key individuals responsible for administering surveys
about substance abuse to 6th through l2 grade students in all 50 states and the District of
Columbia. Team members interviewed these key individuals by telephone, using a
protocol based on the following six questions:
1. What agencies and individuals are responsible for collection of data on substance use
among youth in your state?
2. What instruments do states use to collect this information?
If the instrument properly captures the required core measures, then:
3. How consistent is data collection across state(s) for this instrument (i.e., what
geographic areas are covered or not covered)?
4. What is the smallest geographic level of the data that can be reliably provided to the
DFC National Evaluation (e.g., school, school district, county)?
5. How frequently is data collected from the same geographic area?
6. What next steps, if any, can be used to acquire data for use in the DFC National
Evaluation?

3. Data Collection Method
The Battelle Institute interviewed 102 survey administrators and/or managers from 50
states and the District of Columbia. Original contacts were identified by a representative
(usually the director) of the state's principal substance abuse prevention agency. In 15
cases, the contact identified by the state representative was not the appropriate one. There
were 51 additional contacts whose input was needed to get the full picture of the surveys
administered in each state. In some instances the original interviewee was responsible for
administration and logistics, but a methodologist within the agency was responsible for
data analysis and data management. There were 35 cases where the data analysis and
management was handled by an independent consultant (e.g., University of Washington,
PRIDE Institute, Bach-Harrison) or a company designated by the funding agency (e.g.,
Westat manages data for states and local areas affiliated with the National Youth Risk
Behavior Survey).
States can have more than one questionnaire and therefore, may have more than one manager or analyst.
Feasibility Study for the Drug-Free Communities Support Program National Evaluation
January 19, 2007

Baflelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

4. Findings
4.1 Available Questionnaires
A variety of questionnaires about substance abuse among youth are used in each state.
Some are standardized, validated instruments that have been used to capture substance
abuse information for several years; others have been tailored to the standards that a state
or locale has to meet. Below is a brief description of questionnaires most commonly cited
by state data administrators/managers:

State-Specific Questionnaires
Most states (n=29) develop their own youth substance abuse questionnaires to address
specific community needs or access information specific to their communities. These
questionnaires are usually adapted from existing instruments, such as the Youth Risk
Behavior Survey (YRBS) questionnaire or the Communities That Care (CTC) Youth
Survey questionnaire. In adapting an existing questionnaire, a state might replace general
questions with state- or community-specific questions and/or remove a set of questions
that community stakeholders might consider too sensitive (e.g., sexual behavior
questions).

National Youth Risk Behavior Survey
The Centers for Disease Control and Prevention (CDC) is responsible for the
administration and analysis of the National Youth Risk Behavior Survey (YRBS) data in
49 states and the District of Columbia, and 23 local areas (cities, counties, and
municipalities) (National Center for Chronic Disease Prevention and Health Promotion,
2006). The YRBS monitors priority health risk behaviors that contribute to the leading
causes of death, disability, and social problems among youth and adults in the United
States. Conducted every two years during the spring semester, the survey provides data
that is representative of students in grades 9 through 12 in public and private schools
throughout the United States (National Center for Disease Prevention and Health
Promotion, 2005). See Appendix A for the complete list of states and cities participating
in the National YRBS.

Communities That Care (CTC) Youth Survey
The CTC Youth Survey consists of 350 self-report items which are hypothesized to
measure 21 risk factors and 11 protective factors as well as current levels of substance
abuse, violence, and delinquency (Arthur, Hawkins, Pollard, Catalano, & Baglioni,
2002). The CTC Survey is designed for students in grades 6 to 12 and helps communities
identify the risk and protective factors that pose the most significant challenges and
opportunities for the community.

PRIDE Risk and Protective Factors Survey
PRIDE Risk and Protective Factors Survey questionnaire, a shortened version of the CTC
Youth Survey instrument was created in 2005 by the International Survey Associates.
Feasibility Study for the Drug-Free Communities Support Program National Evaluation
January 19, 2007

Ballelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

The questionnaire was created in response to local schools' and coalitions' request for a
shorter version of the CTC Youth Survey instrument. The PRIDE Risk and Protective
Factors Survey questionnaire contains the risk and protective factor items that show the
strongest correlation to drug use. The survey is administered in several states and
communities, including some with DFC coalitions.

Youth Tobacco Survey
The Youth Tobacco Survey (YTS) is a surveillance tool to measure change in prevalence
of current cigarette smoking and other key indicators of youth tobacco use. The YTS
questionnaire covers cigarette smoking and other tobacco use among young people;
knowledge and attitudes of young people towards cigarette smoking; role of the media
and advertising on young people's use of cigarettes; access to cigarettes; tobacco-related
school curricula; environmental tobacco smoke; and cessation of cigarette smoking.
Several states administer a state-specific YTS to middle and high school youth, using
either standard YTS questions or questions tailored to local communities. The
inconsistent design and use of the YTS across states made it difficult to determine the
extent to which the survey addresses the four core GPRA measures. Further, the Battelle
Institute was not able to access copies of the YTS for most states.

Center for Substance Abuse Prevention's (CSAP) Substance Abuse Risk
and Protective Factor Survey
The Substance Abuse Risk and Protective Factor Survey (SARPF) is a broad-based
questionnaire used to assess student attitudes, perceptions, and behaviors regarding the
use of alcohol, tobacco and other drugs (ATOD). SARPF asks basically the same
questions as the CTC Youth Survey. It is a self-contained, classroom-administered
questionnaire designed to: 1) measure levels of ATOD use among middle and high
school students; and 2) measure risk and protective factors in four social domains
(community, school, family, and individual/peer).

Bach-Harrison Youth Survey
The Bach-Harrison Youth Survey, also known as the Prevention Needs Assessment
(PNA) Risk and Protective Factor Youth Survey, measures the need for prevention
services among youth in the areas of substance abuse, delinquency, antisocial behavior,
and violence (Bach-Harrison, 2006). The questions on the survey questionnaire ask youth
about the factors that place them at risk for substance use and other problem behaviors
along with the factors that offer them protection from problem behaviors. The survey
questionnaire also inquires about the use of alcohol, tobacco, and other drugs and
participation in various antisocial behaviors.

Feasibility Study for the Drug-Free Communities Support Program National Evaluation
January 19, 2007

Ballelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

Frequency of Questionnaire Use By States
Of the questionnaires administered by the states and the District of Columbia:
• 49 states and the District of Columbia administer the National YRBS,
• 25 states administer YTS,
• 17 states administer the Prevention Needs Assessment,
• 10 states administer the PRIDE Risk and Protective Factors Survey, and
• 8 states administer CTC,
• 7 states administer Smart Track,
• 2 states administer SARPF, and
• 29 states administer state specific questionnaires (e.g., Illinois Youth Survey,
Maryland Adolescent Survey).
Table 1 shows the total number and percentage of the most commonly reported
questionnaires across the 50 states and the District of Columbia. This study identified a
total of 41 school-based substance abuse questionnaires being administered in 50 states
and the District of Columbia. Twenty-two states have sub-areas (e.g., cities or counties)
that participate in the National YRBS, and are not included in this study's analysis. This
is also true for the District of Columbia. Appendix A lists the questionnaires administered
in each state.

Table 1: Questionnaires Administered Across States and the District of Columbia
o

-->-

0
(.

Ce
-

•

DC.')

)
E
o

:
o

c 1o

C.)
I)

CeQ

-H

Total States2

Percentage by
State3

98%

49%

33%

20%

16%

14%

57%

_________

_______

_________

_______

__________

______

2 Total states for all charts includes all 50 states and the District of Columbia.
Percentage by state for all charts was calculated using a denominator of 51 which includes the 50 states
and the District of Columbia.
Feasibility Study for the Drug-Free Communities Support Program I'ktional Evaluation
January 19, 2007

Baltelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

4.2 Data Utility
The quality of the school-based substance abuse data is as important as its availability to
the DFC National Evaluation. In order to determine the utility or usefulness of the data
from each state the following issues were investigated:
•

•
•
•

Adequate coverage of core measures (past 30-day use, average age of onset,
perception of risk, and perception of parental disapproval for alcohol, tobacco,
and marijuana);
Frequency of data collection;
Data collection by grade; and
Unit of analysis.

Adequate Coverage of Core Measures
DFC Support Program grantees are required by ONDCP to collect information on the
four core GPRA measures, as described previously. This study set out to determine the
extent to which these core measures are adequately addressed in questionnaires
administered in the 50 states and the District of Columbia, and the geographic level for
which the data are available (e.g., school, school district, county, municipality).
Questionnaires were reviewed to see if they included questions related to the four core
measures for the three substances of interest, and whether these questions were asked in a
manner compatible with the GPRA requirements and the National Evaluation's
standards.
Forty-three states administered questionnaires that capture "past 30 day use" for alcohol,
while 44 capture this core measure for tobacco and 43 for marijuana. Forty-one states
capture the "average age of onset" for alcohol and marijuana, while 43 capture this core
measure for tobacco. For "perception of risk," 40 states have questionnaires that capture
this measure for alcohol, tobacco, and for marijuana. Finally, 39, 40, and 38 states have
questionnaires that capture "perception of parental disapproval" for alcohol, tobacco, and
marijuana respectively. States that use the CTC Youth Survey, PRIDE Risk and
Protective Factors Survey, or the SARPF Survey capture all four core measures for
alcohol, tobacco and marijuana. The National YRBS covers only "average age of onset"
and "past 30-day use" for the three substances. Table 2 specifies the extent to which the
above questionnaires cover the core measures.
The YTS is included in Table 2 even though its questions vary significantly across states
in relation to the core measures covered, including states that go beyond tobacco related
questions to include questions about alcohol and marijuana, The YTS in the table
represents the usual YTS tobacco related questions. State specific surveys are included in
Table 2 for the same reason.

Feasibility Study for the Drug-Free Communities Support Program National Evaluation
Januaiy 19, 2007

Ballelle
CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

Table 2. Core Measures by Questionnaire a

___________

Past 30-Day Use
Alco
Tob
Mar

•

•
•

•
•
•

______

•
•

•

Prevention
_______

_______

SARPF
SmartTrack

YRBS

YTS

State Surveys
Other

o
o

___

o
o

Parental_Disapproval
Alco Tob
Mar

•

Needs

_______

Perception_of Risk
Alco Tob
Mar

•

CTC

Assessment
PRIDE

Age of Onset
Alco Tob
Mar

o
o

•
•
•

___

•
•
•

•
•

___ ___ ___

___

o
o

___

o
o

a Open circles indicate variance among questionnaires capturing core measures. Closed circles indicate data
collection is standardized by questionnaire.

Data Collection by Grade
States administer a variety of school-based substance abuse questionnaires targeting
youth in grades 6 through 12. Some questionnaires are administered across all grades,
while others assess alternating middle and high school grades (i.e., grades 6, 8, 10, and
12) or specific grades in middle and high school (e.g., grades 8 and 11 or only grades 9
through 12). High school grades are more frequently surveyed than middle school grades.
Table 3 shows the total and percentage of states that collect data from students in grades
six to 12.
Table 3: Number and Percentage of States that Collect Data from Students in
Grades Six to 12
(n=41)

_____

Total
States

Percentage
of States

Grades
I 9

_____ _____

______ ______ _______ ______ ______ ______

88% 61%

95%

68% 90% 68% 93%

Feasibility Study for the Drug-Free Communities Support Program National Evaluation
Januaiy 19, 2007

Ballelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

Frequency of Data Collection
The DFC National Evaluation is interested in the frequency of data collection in order to
track substance abuse trends among youth. The DFC National Evaluation requires data
reporting at least every two years. All states and the District of Columbia administer
questionnaires at least every two years or more frequently, depending on the agency
administering the questionnaire, state mandates, and community or school district
requests.
In several states, surveys are intentionally administered every other year such that all
relevant data are captured consistently. For example, in Maryland, the National YRBS is
administered in 2001, 2003, 2005, and so on, while the Maryland Adolescent Survey is
surveyed in 2000, 2002, 2004, 2006, and so on. In some states, one questionnaire could
be administered annually, while another one biennially (e.g., in Arkansas, the YTS is
administered each year and the National YRBS every other year). Appendix B lists the
frequency of survey administration by state.

Unit of Analysis
DFC coalitions are intended to impact local target areas (e.g., school, school district), and
as such, the DFC National Evaluation requires data that can be analyzed at the same
levels targeted by DFC coalitions. This study identified the smallest geographical unit of
analysis available from each state. Some states have data available at more than one
geographic level (e.g., school and county levels), others have data only at one level.
Table 4 indicates the smallest unit of data available by state.
Thirty-eight states and the District of Columbia collect data at the school level, while
three states collect data at the county level and no states collect data at the school district
or municipality levels (e.g. city, town). Additionally, one state has data available in the
"other" category (e.g., individual classroom). Appendix C lists state specific data
availability by unit.

Table 4: Sub-state Level Data Collection (n43)

County

Municipality

Othef

___________

School
District

____________

______________

_____________

School
______________

Total States

Other could be health district, region, program level, or individual classroom level.

Feasibility Study for the Drug-Free Communities Support Program National Evaluation
January 19, 2007

Balleile

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

4.3 Availability And Quality Of Data
In 35 states and the District of Columbia, data from questionnaires administered
statewide can potentially be made available from state or local data managers and
analyzed for the DFC National Evaluation. In most cases the data are owned by a state
agency and can be accessed through formal or informal agreements. In two of three cases,
the data are owned by the participating communities (e.g., school district). The Battelle
Institute, on behalf of ONDCP would need to request the data directly from the school
district; release of the data could be subject to policies in place or at the discretion of the
heads of the school districts. In cases where data cannot be released, the Battelle Institute,
on behalf of ONDCP might negotiate with data managers to run specific analyses.
Twenty-nine states and the District of Columbia can provide the DFC National
Evaluation with raw data and two states can provide data in aggregate form. Table 5
shows the total and percentage of states for which data are available in raw or aggregate
form.
Table 5: Data Availability and Raw/Aggregate Status
(n51)
Data Available
__________

Total States
Percentage of
States

Data Form

Yes

Unknown

Raw

71%

29%

59%

Aggregate

The majority of the school-based substance abuse data available from states are stored
and analyzed in a data management package that is compatible with DFC National
Evaluation management packages (e.g., SPSS, SAS, Excel).

Feasibility of Using Sub-State Level Data
Sub-state level (e.g., school, school district, county, and municipality) data are most
useful to the DFC National Evaluation, as discussed previously. With these data the
National Evaluation can compare trends in substance abuse outcomes between states with
and without DFC coalitions, using the four core measures. Forty-four states collect this
infonnation statewide at the sub-state level, and can make the information available (40
for the entire state and four for parts of its state).
States that do not collect information statewide do so for various reasons. For example, a
school district might decide not to participate in a particular survey because of concerns
that the results would portray the community in a negative light. Funding issues can also
limit the administration of the survey across the state (i.e., a state might receive a small
grant to administer the survey only in one or two areas). Some communities decline to

Feasibility Study for the Drug-Free Communities Support Program National Evaluation
January 19, 2007

Balleile

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

administer a certain survey to protect students from additional surveys and overburdened
teachers and school staff from additional responsibilities.
Table 6a and 6b present the key findings of this study. In order for state collected data to
be adequate for use in the DFC National Evaluation it must be compatible with the core
measures used in the DFC National Evaluation, be collected every two years of more
frequently, be administer to youth in grades 6-12, be available for use by others, and be
able to be analyzed at sub-state areas contiguous to DFC coalition target areas.
Representatives from 38 states have reported that their states could meet these criteria for
all areas throughout their state. The District of Columbia has data on two of the four core
measures available city-wide through the National YRBS. As noted previously, not all
states have all sub-state areas available for analysis by the DFC National Evaluation.
Four states have been reported to have local community data available for some parts of
their state. The analysis of the adequacy of data from these states is presented in Table
6b.
Tables 6a and 6b demonstrate that state collected data can not be used instead of grantee
provided data in an evaluation that would include DFC coalitions from all 50 states and
the District of Columbia. Although 38 states reported coverage of all four core measures,
this data would not be able to capture the efforts of all DFC grantees which operate
across the nation. Without getting an official approval at the state level there is no
guarantee that state representatives can deliver the information reported as available in
the format reported.
Table 6a: States with Adequate Core Measures (Total state with sub-areas)
n=44
Perception of
Age of onset
Perception of nsk
Past 30 day use
parental_disapproval
T
M
A
T
M
A
T
M
A
M
A
T
States with total
coverage of sub44
43
41
43
41
40
40
40
39
40
38
43
_____

_____ _____

_____ _____ _____

______

_____

______

_____

36
25
38
27
36
27
35

36
25
39
28
37
28
37

36
25
38
27
36
27
35

34
25
37
25
35
27
37

35
25
39
27
37
28
38

_____

state units

Data collected
every two years or
less
Grades:

6
7
8
9
10
11
12

_____

34
25
37
25
35
26
37

33
23
36
24
34
25
34

______

26
_____

32
22
35
23
33
24
33

______

_____

35
24
37
26
35
26
36

31
21
34
22
32
23
32

This table illustrates states administering statewide questionnaires.

Feasibility Study for the Drug-Free Communities Support Program National Evaluation
Januaiy 19, 2007

Baflelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

Table 6b: States with Adequate Core Measures (Part of state with sub-areas)
n=13
Age of onset

Past 30 day use
_____ ______ _____

Perception of nsk

_____ _____ ______

_____ ______ _____

Perception of
parental_disapproval

_____

______

_____

______

_____

______

_____

______

_______

_____

6
7
8
9
10
11

7
6
8
6
8
6

2
1
3
1
3
1

2
1
3
1
3
1
3

7
6
8
6
8
6
8

______

States with partial
coverage of substate units
Data collected
every two years or
less

Grades

_____

2
1
3
1
3
1
3

+ This table illustrates states administering questionnaires to partial regions of the state.

State Level Data
Although availability of sub-state level data is most relevant to the National Evaluation,
the National YRBS, which is available at the state level is unique in the context of the
other surveys. One advantage of the YRBS data is that the Battelle Institute, on behalf of
ONDCP, can access it from the CDC without going to states individually. Because 49
states and the District of Columbia, and 23 locales send their data to the CDC for
management and analysis, it is more efficient to access this dataset.
Details about the National YRBS include:
•

The National YRBS collects information on two of the four core measures-past 30
day use and average age of onset for alcohol, tobacco, and marijuana.

The National YRBS does not include measures of perception of risk or perception of
parental disapproval for the three substances. Some participating states have added
these measures to their instruments in order to capture all the core measures for all
three substances.
The dataset provided by the CDC includes information about responses to all risk
behavior questions, some of which could be used to compare substance abuse and
other behaviors by race/ethnicity and grade.

Feasibility Study for the Drug-Free Communities Support Program National Evaluation
January 19, 2007

Ballelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

•

The National YRBS has been administered every other year since 1991; however, the
frequency and quality of data varies from state to state where some states (e.g.,
Alabama, Utah) have administered the YRBS consistently every other year since
1991, while others sporadically (e.g., 1991 and 2005 in Maryland and 2003 and 2005
in Oklahoma).
The National YRBS data set includes weighted and unweighted data. For the
National YRBS, weighted results means that the questionnaire got an overall
response rate of at least 60%. These results are representative of all students surveyed
in grades 9 to 12. Unweighted data represent only the students who completed the
questionnaire and are not representative of the population of interest. CDC does not
include unweighted data in its analyses.

•

Data from 1991 until 2005 can be made available to the DFC National Evaluation.
Data collected before 2005 are only available if the state has given permission. The
CDC requires states participating in the National YRBS to administer the
questionnaire to Grades 9 to 12 statewide. However, a few participating states
administer the YRBS to grades 6 to 8 as well.
The National YRBS is collected at the school level. However, when the states send
the school-level data to the CDC, all school identifiers are removed. Data are only
available at the state level. The DFC National Evaluation is interested in data in units
smaller than the state level (i.e., school, school district, county and municipality) in
order to combine with or compare it to DFC coalition data. Because all identifiers are
removed, the DFC National Evaluation would not be able to do within state
comparisons.
The DFC National Evaluation could use the state-level data to make comparisons
between states or conduct analyses by region (e.g. northeast, southwest). Also, the
National Evaluation might consider city/county comparisons using data from the 23
YRBS participants in local communities.

•

The CDC will share the YRBS code book and the full data set by state (for states that
have given signed permission) with the DFC National Evaluation, making it
relatively easy to access and use the data.

•

YRBS is also available online where the Battelle Institute can access state-level data
by race, gender and grade.

Feasibility Study for the Drug-Free Communities Support Program National Evaluation
January 19, 2007

Ballelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

5. Feasibility Study Results: Issues and
Challenges
Several important issues and challenges came about as a result of this feasibility study:

Adequate data are not available from all states and therefore not all DFC coalitions
could be included in the same investigation using state collected data. The primary
design of the DFC National Evaluation and GPRA reporting require data on all DFC
program grantees and their communities. Therefore, under current conditions, the
use of state collected data or sources other than grantees is not feasible at this time.
•

Data from the National YRBS is only available at the state level. The National YRBS
is collected at the school level, but school identifiers are removed when states send
the data to the CDC. The data can be used to compare states or region; otherwise, it
has limited use to the DFC National Evaluation (i.e. as a comparison for core
measure trends. The National Evaluation could also consider city/county
comparisons using the data from the 23 YRBS participants in local communities for
secondary analysis purposes. However, the YRBS only adequately includes two of
the four core measures.

•

Some states administer different questionnaires to different populations in their
states, some with overlapping participants. The use of multiple questions would be
another data coordination and cleaning task if state collected data was to be used.
Frequency of data collection across and within states is so varied that only
comparisons of trends would be possible. Dates of data collection and the intervals
between them are inconsistent (e.g., two or three years apart). Having diverse data
collection times limits the opportunity to look at absolute changes (i.e., a net change
between two times) in local, state and youth substance abuse. The National
Evaluation would have to compare trends instead, requiring the collection of
multiple data points. Comparing trends would allow the evaluation to indicate how
many DFC coalitions report change, the average change they report, and difference
in trends between DFC communities and national, state, or local communities.
However, the evaluation would not be able to quantify the actual net change that
DFC coalitions have on their communities from 2004 to 2009. Grantee provided data
faces the same challenge.

Feasibility Study for the Drug-Free Communities Support Program National Evaluation
January 1 2007

Ballelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

6. Recommendations
1. The DFC National Evaluation could consider accessing substance abuse
outcomes from states based on the findings of the feasibility study for verifying
accuracy of grantee-provided data, filling in where grantees are missing data,
and conducting secondary studies of DFC communities. Verifying accuracy
would help determine the quality of data provided by checking the submissions of
some grantees with the data collected from states. In addition, state data may be used
to fill in values for grantees that do not provide data. Secondary data could be
collected on a sub-group of grantees and possibly comparison communities to
investigate differences between DFC and non-DFC communities. This data would be
secondary because the current goals and plan for the DFC Program Evaluation call
for reporting on all grantees, which is not possible with state reported data. Even
though type of access, core measure coverage, frequency, and diversity of
questionnaires vary by state, there are potential uses of these data for DFC National
Evaluation purposes. These ideas could be discussed with the DFC National
Evaluation External Review Group. However, currently there are no evaluation funds
allocated to obtain additional state data.
2. CDC can be encouraged to update the national YRBS instrument so that it
captures all four core measures. Currently the National YRBS asks average age of
onset and past 30-day use for alcohol, tobacco and marijuana for students in grades 9
through 12. It does not ask the perception of risk or the perception of parental
disapproval questions.
3. Investigate the feasibility of using other surveys to access data from the states for
which sub-state area data is not available, such as the PRIDE Risk and
Protective Factors Survey and the American Drug and Alcohol Survey. Recent
versions of the questionnaires used in these school-based surveys meet all the criteria
for use in the DFC National Evaluation. The American Drug and Alcohol Survey
(ADAS) developed and managed by the Rocky Mountain Behavioral Science
Institute, Inc., is an example of a questionnaire administered in several communities
around the country. ADAS has surveyed more than 1.5 million students in 47 states
since 1987 and it recently was updated to capture the four core measures for alcohol,
tobacco, and marijuana. The PRIDE Institute survey regularly collects data from
8,000 school districts. The PRIDE Institute has expressed interest in working closely
with ONDCP and Battelle Institute in providing data for the DFC National
Evaluation.
4. ONDCP should consider how to develop a national substance abuse surveillance
and data system to address the fragmentation, inconsistencies, and barriers
accessing quality data by local communities. Substance abuse data lags behind
other public health and law enforcement data (e.g., Unified Crime Reporting) in
regards to standards for measurement of vital statistics such as the core measures. The
Feasibility Study for the Drug-Free Communities Support Program National Evaluation
January 19, 2007

Ballelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION
lack of availability of these data for sub-state areas is another major problem. While
44 states have data available to be analyzed at these smaller units, the investigators in
this feasibility study found that these data are not easily available to communitybased groups, such as DFC coalitions. This observation is confirmed through the
technical assistance requests of DFC grantees to the Battelle Institute team. If
community-based prevention efforts are central to the national drug control strategy,
the elimination of this barrier is essential.

Feasibility Study for the Drug-Free Communities Support Program National Evaluation
January 19, 2007

Ballelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

References
Arthur, M. W., Hawkins, J. D., Pollard, J. A., Catalano, R. F., & Baglioni, A. J. (2002).
Measuring risk and protective factors for substance use, delinquency, and other
adolescent problem behaviors: The Communities That Care Youth Survey.
Evaluation Review, 26(6), 575-601.
Bach Harrison, L.L.C.(2006). Prevention Needs Assessment Youth Survey.
Accessed January 19, 2007 http://www.bachharrison. comlBhResources/PnaSurvey. aspx.
National Center for Disease Prevention and Health Promotion (2005). Youth Risk
Behavior Surveillance System Survey Participation Map, 2005. Retrieved July 21,
2006 from http://www.cdc.gov/HealthyYouth/YRBS/map.htm
National Center for Chronic Disease Prevention and Health Promotion (2006). Youth Risk
Behavior Surveillance System-United States 2005. Retrieved August 7, 2006 from
http ://www.cdc.gov/mmwr/PDF/SS/SS 5505 .pdf

Feasibility Study for the Drug-Free Communities Support Program National Evaluation
January 19, 2007

Baftelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

Appendix A
Comprehensive List of Questionnaires Administered by State
YRBS4

State

YTS5

Alabama

- -

Alaska

- -

Arizona
_______________

•

- -

_________

________

PNA6

PRIDE7

- -

•

SARPF9

- -

________

_________

- -

California
•

________________ _________

- -

________

_______

________

- -

________

__________

- -

Colorado
•

________________

__________ ________

- -

________ _________

•

Connecticut
•

________________ _________

- -

________ ________ ________

________

- -

_________

- -

_________

- -

__________

•
_________

Delaware
•

________________ _________

•

District of
Columbia

- -

•

_______________

Georgia

________

•

_______________

________

- -

________ _______

- -

________

_________

- -

Arizona Youth
Survey
- -

• California Healthy
Kids Survey
• California Student
Survey
• Healthy Kids
Colorado Survey
(Middle & High
School)
• Monitoring the
Future
• School Health
Survey
• Delaware Secondary
School Student
Assessment and
Survey

Florida

Hawaii

- -

________ ________ ________

- -

Other

•
- -

________ ________

Arkansas

CTC8

- -

________ _______

•

- -

_________

- -

________ _______
-

- -

_________

- -

_________

- -

_________

- -

• Florida Youth
Substance Abuse
Survey
• Department of
Education instrument
• Tobacco Health
Survey
- -

Youth Risk Behavior Survey
Youth Tobacco Survey
6 Prevention Needs Assessment
Pride Institute Risk and Protective Factors Survey
8 Communities That Care
CSAP Substance Abuse Risk and Protective Factor Student Survey
Feasibility Study for the Drug-Free Communities Support Program National Evaluation
January 19, 2007

Ballelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION
YRBS4

State

YTS5

Idaho

PNA
- -

PRIDE7

CTC8

- -

SARPF9
- - -

Illinois

•

- -

Indiana

Iowa

________ ________ _________

_________

________

_______

- -

Kansas
Kentucky

_______________

_________

Louisiana

•
________ ________ ________
- -

________

_________

- -

_________
- -

Maine
•

_______________

Maryland

_________

•

_______________

_________

- -

________ ________ ________

- -

________

_________

________

_________

- -

•
________ ________ ________

Massachusetts
- -

- -

Michigan

Minnesota

- -

5
- -

- -

- - -

Mississippi

- -

Missouri

•
•

Montana

- -

•

- -

•

- -

Nebraska
_______________

•

_________

________

_______

- -

Nevada
New Hampshire
New Jersey
New Mexico

•
•

- -

•
- -

_______________ _________

Carolina

North Dakota
Oklahoma
Oregon

•

5
5
5

•

- -

________

_________

- -

- - -

New York

Ohio

• Substance Abuse
Survey
. Illinois Youth Survey
• Afternoons R.O.C.K.
in Indiana
. Alcohol Tobacco &
Other Drugs
• Iowa Youth Survey
- -

• ADAS

•

North

Other

- -

- - -

•

________ ________ _________

- _________

• Kentucky Incentives
for Prevention
Survey
- -

• Maine Youth Drug
and Alcohol
Survey/YTS
• Maryland Adolescent
Survey
• Massachusetts Youth
Health Survey
- -

• Minnesota Student
Survey
- -

• Missouri Student
Survey
- -

• Nebraska Risk &
Protective Factor
Survey
- -

- -

• Middle & High
School Survey
• Youth Risk &
Resiliency Survey
• OASAS School
Survey

_______

- -

5
5
5
- -

- -

•
- -

- - -

Feasibility Study for the Drug-Free communities Support Program National Evaluation
January 19, 2007

- -

• Oregon Healthy Teen
Survey
20

Ballelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

State
______________

Pennsylvania
________________

YRBS4

YTS5

PNA6

PRIDE7

CTC8

________

_______

•

__________ ________

Smart
Track

SARPF9

_______

_________

• Pennsylvania Youth
Survey
• Survey Assessment
for Learning &
Teaching
• South Carolina
Youth_Survey

- -

________

__________

- -

________

Rhode Island
•

_________________

South Carolina
South Dakota
Tennessee

__________

•
•
•

________

- -

•

•
•

Texas
•

Utah

•

Vermont

Virginia

•

.
_______

Wisconsin

5
5

Wyoming
Total States &
DC
Percentage by
State '°

- -

________ ________ _________

- -

•

- -

• Texas School Survey
of Substance Abuse

- -

__________

Washington
West Virginia

__________

_______

Other

_________ ___________________

- -

. Youth Health Survey
• Virginia Community
Youth_Survey
• Healthy Youth
Survey

_________

- -

_______

______

7
_______

2
_______

29
________________

98%

49%

33%

20%

16%

14%

57%

___________

_________

__________

____________

_________________________

The percentage was calculated using a denominator of 51 which includes the 50 states and the District of
Columbia.
Feasibility Study for the Drug-Free Communities Support Program National Evaluation
Januaiy 19, 2007

Baflelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

Appendix B
Frequency of Survey Administration by State
State

Year
Began

______

Frequency (n=41)

______

Once a
Year

Every 2
Years

Every 3 to 5
Years

Another
Interval

2002

- -

Arizona

2002

- -

Arkansas

2002

- -

California

1985

- -

Colorado

1993

- -

5
S

- -

___________

- ________

- _________

1989

Alabama
Alaska

Connecticut
Delaware
District of
Columbia

1995

1991
_________

- -

- _____________ ________
- -

________ ________

- ___________

- -

- ________

Florida

1998

Georgia
Hawaii

1991

1984

- -

Idaho

1996

- -

Illinois

1990

- -

Indiana

1991

- -

Iowa

1975

- -

Kansas

- -

S
S

- -

1995

- -

Kentucky

- -

Louisiana

1998

- -

Maine

1995

- -

Maryland

1992

- -

Massachusetts

2004

- -

Michigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada

S
S
S

- -

1989

- -

2000

- -

2000

- -

1998

- -

2003
- -

- -

S
S
S

- -

Feasibility Study for the Drug-Free Communities Support Program National Evaluation
.Januay 19, 2007

Ballelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

State

Year
Began

______

Frequency (n=41)

______

Once a
Year

Every 2
Years

Every 3 to 5
Years

Another
Interval

________

____________

________

- -

New
Hampshire

__________

New Jersey

1995

New Mexico

1999

- -

NewYork

1990

- -

North
Carolina

_________

_____________

_________

- -

Ohio

1993

- -

Oklahoma

2004

- -

Oregon

1994

- -

Pennsylvania

1989

- -

Rhode Island

2001

- -

North Dakota

____________ _________

- -

South
Carolina

_____________ ________

___________

________

_________

South Dakota

- -

Tennessee

- -

Texas

1988

Utah

1995

- -

Vermont

1993

- -

Virginia

2000

- -

Washington

1988

- -

West Virginia

2003

- -

Wisconsin
Wyoming
Total States
Percentage of
States

2001
N/A

- -

N/A

27%

71%

____________

_________

__________

______________

_________

Feasibility Study for the Drug-Free Communities Support Program National Evaluation
January 19, 2007

Ballelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

Appendix C
Data Availability, Data Format, and Smallest
Unit of Available Data by State

State

Data Available
(n=51)

Data Format
(n- )

Smallest Unit of
Available Data by State
43

_______

_____________
Alabama
Alaska

Unknown

- -

Colorado

- -

Georgia
Hawaii

- -

Kansas

- -

____________

________

- -

5
5

- -

Louisiana

- -

______

- -

__________

- -

Minnesota

- -

Indiana

Michigan

- -

1assachusetts

- -

Illinois

Maryland

- -

School

- -

- ______

Florida

Maine

- -

California

Kentucky

- -

Iowa

- -

Arkansas

Idaho

- -

District of
Columbia

Other

Aggregate

Arizona

Connecticut
Delaware

County

Raw

Yes

- -

Mississippi

- -

Missouri

- -

Montana

5
5

Feasibility Study for the Drug-Free Communities Support Program National Evaluation
January 19, 2007

Ballelle

CENTERS FOR PUBLIC HEALTH RESEARCH & EVALUATION

State

Data Available
(n=51)

Data Format
(n=51)

Smallest Unit of
Available Data by State
(43)

_______

_____________
Nebraska
Nevada
New
Hampshire
NewJersey

Yes

Unknown

Raw

- -

______

__________

Aggregate

School

- -

_______

County

Other

- -

______

____________

________

________
- -

- -

NewMexico

- -

NewYork

- -

North Carolina

- -

North Dakota

- -

Ohio

- -

Oklahoma

- -

Oregon

- -

Pennsylvania

- -

Rhode Island

- -

South Carolina

- -

South Dakota

- -

Tennessee

- -

Texas

- -

Utah

- -

Vermont
Virginia

Washington

- -

West Virginia

- -

Wisconsin

- -

Wyoming
Total States
Percentage of
States

36
71%
_______

- -

29%

59%

- -

____________ ________ ______________

39
91%

_________

__________

Feasibility Study for the Drug-Free Communities Support Program National Evaluation
January 19, 2007

File Type	application/pdf
File Modified	0000-00-00
File Created	2008-08-26