Download:
pdf |
pdf
Attachment 2
Evaluation Design Document
UPDATED EVALUATION DESIGN DOCUMENT
DRUG‐FREE COMMUNITIES SUPPORT PROGRAM
NATIONAL EVALUATION
Submitted to:
Executive Office of the President
Office of Administration
Office of National Drug Control Policy
Submitted by:
Battelle
505 King Avenue
Columbus, OH 43201
Originally Issued: June, 2005
Updated: September, 2008
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
TABLE OF CONTENTS
EXECUTIVE SUMMARY....................................................................................................................................................i
1.0
INTRODUCTION AND OBJECTIVES ...................................................................................................................1
2.0
EVALUATION FRAMEWORK, TYPOLOGY, AND DESIGN OVERVIEW.................................................................3
2.1
2.2
EVALUATION TYPOLOGY ...................................................................................................................4
2.1.1
Defining Coalition Stages of Development ........................................................................5
2.1.2
Operalization of Typology...................................................................................................6
OVERVIEW OF EVALUATION DESIGN ..............................................................................................12
2.2.1
3.0
4.0
Limitations of the Study ...................................................................................................13
DATA COLLECTION AND MANAGEMENT.......................................................................................................13
3.1
DATA SOURCES ...............................................................................................................................13
3.2
DATA COLLECTION APPROACH .......................................................................................................15
3.3
DATA ASSESSMENT AND MANAGEMENT .......................................................................................16
3.3.1
Data Quality Assessment ..................................................................................................16
3.3.2
Data Management ............................................................................................................17
STATISTICAL ANALYSIS USING SELF‐REPORTED DATA...................................................................................17
4.1
EXPLORATORY ANALYSIS.................................................................................................................18
4.2
ASSESSMENT OF STAGE‐OF‐DEVELOPMENT...................................................................................19
4.2.1
4.3
Creating and Refining a Classification Algorithm..............................................................22
MODELING SUBSTANCE ABUSE OUTCOMES...................................................................................23
4.3.1
Identifying Factors Associated with Substance Abuse Outcomes among DFC
Coalitions ..........................................................................................................................23
4.3.2
Comparison of DFC Communities to Non‐DFC Communities ...........................................24
4.4
ASSESSMENT OF ADDITIONAL EVALUATION QUESTIONS AND HYPOTHESES.................................30
4.5
ANALYSES TO SUPPORT GPRA REPORTING.....................................................................................32
5.0
STATISTICAL ANALYSIS OF EXTERNAL DATA..................................................................................................35
5.1
5.2
MODELING SUBSTANCE ABUSE OUTCOME DATA COLLECTED THROUGH EXTERNAL SOURCES ....35
5.1.1
Feasibility Assessment ......................................................................................................36
5.1.2
Statistical Methods ...........................................................................................................37
EVALUATING DISTAL OUTCOMES ...................................................................................................39
6.0
SUMMARY
..........................................................................................................................................40
7.0
REFERENCES
..........................................................................................................................................42
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
EXECUTIVE SUMMARY
The Office of National Drug Control Policy (ONDCP) funds the Drug‐Free Communities Support
Program (DFC) to build community capacity to prevent substance abuse among our nation’s
youth. The DFC program has two primary goals: (1) to reduce substance abuse among youth by
addressing local risk and protective factors to minimize the likelihood of subsequent substance
abuse in the community, and (2) to support community anti‐drug coalitions by establishing,
strengthening, and fostering collaboration among public and private nonprofit agencies, as well
as federal, state, local, and tribal governments to prevent and reduce substance abuse.
Currently, 769 community anti‐drug coalitions are receiving DFC grants.
ONDCP commissioned a national evaluation of the DFC program with the overall goal of
assessing the program’s implementation and effectiveness. Three primary objectives of the
evaluation are to (1) assess whether the DFC program has made an impact on reducing the
substance abuse outcomes at the community, state, and national level; (2) determine if there
are specific factors that can be identified that are related to increases in substance abuse
prevention; and (3) assess whether the DFC program has increased the capacity and
effectiveness of substance abuse coalitions. Within these broad objectives, there are a number
of specific questions and hypotheses that will be addressed by the evaluation.
An evaluation framework that is based on a review of scientific literature for conducting
evaluations of substance abuse coalitions will
be used to guide the evaluation. This
framework is based on a maturation of
development stage typology that
hypothesizes a causal chain between DFC
coalitions’ functions and activities and
immediate, intermediate, substance
use/abuse, and long‐term outcomes as the
coalition “matures” and develops its capacity
(see Figure E1). Each component of the
Figure E1. Evaluation Framework
framework represents a measurable
hypothesis for the national evaluation to assess.
Using the evaluation framework as a foundation, the evaluation will focus on assessing both a
direct and indirect link of the success of the DFC
Program in assisting coalitions in impacting
substance abuse in their communities (see Figure
E2). First, to estimate the effectiveness of the DFC
program, we will examine whether the activities,
initiatives, strategies, etc. of DFC coalitions have a
direct impact on substance abuse outcome
measures of interest, such as the proportion of
youth who report using tobacco in the last 30
Figure E2. Overview of Evaluation Design
i
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
days. However, the greatest warrant for estimating the efficacy of the program will come from
an outcomes analysis focused on those mature and sustaining coalitions that might be expected
to have the greatest impact on youth substance use. Second, the indirect impact of the DFC
program on enhancing grantee coalitions’ ability to influence change in the community will be
assessed by evaluating the degree to which coalitions participating in the DFC Program mature
into advanced coalitions (i.e., build capacity). If coalitions that participate in the DFC program
can be found to advance into mature coalitions, and if the link between mature coalitions and
substance abuse outcomes can be established, then it would be logical and scientifically
appropriate to conclude that the DFC program is effective in reducing substance abuse
outcomes, directly for mature coalitions and indirectly for beginning coalitions, by fostering an
environment where less mature coalitions can become mature coalitions.
The data for this evaluation will be drawn from semi‐annual progress reports submitted by
coalitions. This information will be collected through
the use of a web‐based system referred to as the
Coalition On‐line Management and Evaluation Tool
(COMET). As illustrated in Figure E3, the COMET will
capture information from coalitions on aspects
related to structure and characteristics, internal
capacity, intended functions, immediate and
intermediate outcomes, and substance abuse
outcomes in the coalition’s targeted community.
Semi‐annual progress reports will contain
information that may change on a semi‐annual basis
Figure E3. Sources of Data for
(such as coalition membership). An annual progress
the Evaluation
report will provide information on items that are
expected to change less frequently, such as intermediate outcomes. Finally, a separate module
of the COMET, the Coalition Classification Tool (CCT), will capture information that can be used
to classify each coalition into their respective stage‐of‐development. Information from external
sources will also be collected and utilized as part of the evaluation, particularly to obtain
explanatory variables such as funding levels or comparative outcome measures, such as
statewide substance abuse rates.
Although the information will be collected via the COMET in a standardized format, all of the
information collected by the COMET represents self‐reported information by coalitions.
Therefore, as with any self‐reported data, the quality of the data needs to be assessed prior to
being used in the evaluation. One key concern is the consistency and reliability of the self‐
reported substance abuse outcome information. Many coalitions use different survey
instruments, methods, and sample sizes for collecting and reporting substance abuse outcome
information for their targeted communities. To be useful for the evaluation, inconsistencies in
the various survey instruments need to be identified and resolved or these cases must be
excluded. Coalitions that have employed unreliable or suspect methods when providing data
will also be identified and excluded from the analysis if necessary. Finally, the variability and
reliability associated with each outcome will be assessed through examining the sample sizes
ii
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
that serve as a basis for each reported percentage. Outcomes that are based on too few
samples to be statistically reliable also may be excluded from the analysis.
Within the evaluation framework, six different analyses will be conducted as part of the core
evaluation effort: 1. exploratory/preliminary analyses, 2. assessing stage‐of‐development,
3. modeling substance abuse outcomes, 4. assessing additional evaluation questions and
hypotheses, 5. analyses to support GPRA reporting, and 6. analysis of external data. Together,
the analyses conducted for these six key areas will address the three primary objectives of the
evaluation and their corresponding evaluation questions and hypotheses.
1. Exploratory/Preliminary Analyses. One implicit objective of this evaluation is to provide
information to ONDCP and others regarding the status of the DFC program and the
characteristics of the participating coalitions. This information will represent a useful
context and background for the remaining analyses. It will also provide an initial overview
of how the coalitions’ characteristics (e.g., size, structure, degree of formalization, etc.)
change over time. This information is important for refining the overall evaluation
framework and stage‐of‐development typology.
2. Assessing Stage‐of‐Development. There are a number of steps that will be performed to
assess whether DFC coalitions are maturing and to identify the characteristics associated
with maturation. First, using information collected primarily through the CCT, coalitions
will be classified into one of five specific stage‐of‐development groups; either 1) Pre‐
coalitions, 2) Establishing coalitions, 3) Functioning coalitions, 4) Maturing coalitions, or
5) Sustaining coalitions. This classification will be implemented using a theory‐driven scale
based on the hypothesized typology. Next, the quality of the classification will be assessed
and the hypothesized typology refined. Significant predictors of a coalition’s stage‐of‐
development such as a coalition structure, capacity, and characteristics will be identified
using statistical models. Finally, the progress of DFC coalitions in moving from the initial
stages of development to more advanced stages of development will be assessed using a
longitudinal model that examines trends in coalition development over time.
3. Modeling Substance Abuse Outcomes. Statistical models will also be used to identify
significant predictors of reduced substance abuse. For example, we will assess whether
being in an advanced stage of development (i.e., being a maturing or sustaining coalition)
is significantly related to a decrease in the proportion of youth who report using alcohol in
the last 30 days. This analysis will be conducted both for specific time points and using
longitudinal models so that trends over time can be assessed.
Trends in substance abuse outcomes in communities targeted by DFC coalitions will be
compared to the corresponding trends in communities that are not specifically targeted by
a DFC coalition indirectly using state and national level data. This analysis differs from an
approach where comparison communities are selected and matched to DFC coalition
communities and a direct comparison is conducted between the two different types of
communities. Employing a comparison‐community type approach was determined to be
iii
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
infeasible for this evaluation due to logistical considerations and resource constraints.
Therefore, the evaluation will rely on extracting indirect surrogates for substance abuse
outcomes for communities not targeted by a DFC coalition from existing national and state
surveys to serve as a comparison to DFC communities. Although this approach will have
the limitation that many of the explanatory factors will not be available for non‐DFC
communities, it has the decided advantage that it does not rely on a direct data collection
activity conducted as part of this evaluation effort.
4. Assessment of Additional Evaluation Questions and Hypotheses. There are a number of
additional evaluation questions that complement the three primary objectives of the
evaluation. Several of these questions are focused on assessing the relationship between
potential explanatory variables (e.g., coalition composition/collaboration, geographical
focus of the coalition, and effectiveness of environmental strategies) and substance abuse
outcomes. The other evaluation questions focus on coalition capacity and separate
analyses will be conducted to investigate each of these evaluation questions. For example,
the evaluation question “What evidence exists to demonstrate an increase in evidence‐
based programs, policies, and strategies in coalition communities?” will be addressed
through examining the number of environmental strategies employed by coalitions in their
first grant year to the number of environmental strategies employed by these same
coalitions in subsequent years.
Assessing the impact of ONDCP’s mentoring program is hindered by the relatively small
sample size associated with this program. However, the evaluation will include an attempt
to assess the impact of this program by grouping the support activities of the mentoring
coalitions to develop a scale related to the intensity of mentoring activities (i.e., a measure
of dosage). The relationship between these intensity categories and capacity outcomes of
the mentee coalitions (i.e. preparedness to implement Strategic Prevention Framework
(SPF): governing body, baseline measures, strategic planning activities, collaboration of key
sectors) will then be examined through exploratory analyses.
5. Analyses to Support GPRA Reporting. ONDCP is required to submit a Government
Performance and Results Act (GPRA) report to Congress annually regarding the DFC Grant
Program. Information on accomplishments towards these goals and objectives will
primarily be captured through the COMET. Two types of analyses will be conducted in
support of the GPRA reporting requirement. First, descriptive statistics or summaries for
every data element in the COMET will be developed. Second, an analysis will be conducted
to characterize progress toward reducing substance abuse in DFC communities through
longitudinal models (for trends over time) and through characterizing the percentage of
coalitions that have reported a positive change over time in a specific substance abuse
outcome. Data collected on behalf of ONDCP prior to the initiation of this evaluation effort
will be used to the extent possible as baseline information.
6. Analysis of External Data. There are a number of national surveys that collect information
about substance abuse outcomes of interest to ONDCP. Many DFC coalitions rely on these
iv
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
surveys for outcome measures that they then provide via their Semi‐Annual Progress
Report. Unfortunately, the public data sets associated with these efforts do not contain
community‐level data. Therefore, in January 2007, the evaluation team concluded a
Feasibility Assessment, conducted to determine if community‐level information could be
obtained from some of the established national surveys. The results of this assessment
follow.
•
•
•
•
Adequate data are not available from all states and therefore not all DFC coalitions
could be included in the same investigation using state‐collected data. The primary
design of the DFC National Evaluation and GPRA reporting require data on all DFC
program grantees and their communities. Therefore, under current conditions, the use
of state‐collected data or sources other than grantees is not feasible at this time.
Data from the National Youth Risk Behavior Survey (YRBS) is only available at the state
level. Data for the National YRBS are collected at the school level, but school identifiers
are removed when states send the data to the CDC. The data can be used to compare
states or region; otherwise, it has limited use to the DFC National Evaluation (i.e., as a
comparison for core measure trends). The National Evaluation could also consider
city/county comparisons using the data from the 23 YRBS participants in local
communities for secondary analysis purposes. However, the YRBS only adequately
includes two of the four core measures.
Some states administer different questionnaires to different populations in their states,
some with overlapping participants. The use of multiple questions and questionnaires
would require an extensive data coordination and cleaning task if state‐collected data
were to be used.
Frequency of data collection across and within states is so varied that only comparisons
of trends would be possible. Dates of data collection and the intervals between them are
inconsistent (e.g., two or three years apart). Having diverse data collection times limits
the opportunity to look at absolute changes (i.e., a net change between two times) in
local, state, and youth substance abuse. The National Evaluation would have to compare
trends instead, requiring the collection of multiple data points. Comparing trends would
allow the evaluation to indicate how many DFC coalitions report change, the average
change they report, and difference in trends between DFC communities and national,
state, or local communities. However, the evaluation would not be able to quantify the
actual net change that DFC coalitions have on their communities from 2004 to 2009.
Grantee‐provided data faces the same challenge.
A similar assessment will be conducted to investigate the feasibility of assessing the differences
between DFC communities and non‐DFC communities with regard to distal outcomes such as
drug‐related traffic fatalities, hospital discharges, etc.
v
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
The results of the statistical analyses conducted in these six key areas will address all three of
the primary objectives of the analysis, and will also answer virtually all of the additional
questions and hypotheses posed for this evaluation. Figure E4 summarizes how the analyses in
each of these areas come together to address the objectives of the evaluation.
Figure E4. Cross‐Link Between Evaluation Analysis and Objectives
vi
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
1.0 INTRODUCTION AND OBJECTIVES
The Office of National Drug Control Policy (ONDCP) funds the Drug‐Free Communities Support
Program (DFC) to build community capacity to prevent substance abuse among our nation’s
youth. The DFC program has two primary goals: (1) to reduce substance abuse among youth by
addressing local risk and protective factors to minimize the likelihood of subsequent substance
abuse in the community, and (2) to support community anti‐drug coalitions by establishing,
strengthening, and fostering collaboration among public and private nonprofit agencies, as well
as federal, state, local, and tribal governments to prevent and reduce substance abuse.
The DFC Support Program funded five cohorts (1998–2002) of community anti‐drug coalitions
in its first five‐year grant cycle, an additional five cohorts in its second five‐year cycle, covering
the period from FY 2003 through FY 2007, and is currently in its third five‐year grant cycle
covering the time period 2008 through 2012. Currently, 769 community anti‐drug coalitions are
receiving DFC grants. The DFC program anticipates awarding approximately 90 additional grants
each year through FY 2012. The focus of this document is the design of a national evaluation of
the DFC Support Program to assess the program’s effectiveness.
The national evaluation of the DFC Program represents a unique opportunity to collect
information and perform a comprehensive evaluation with a significant number of coalitions.
As such, there are many different issues and hypotheses that could be addressed by this
evaluation. However, the focus of this national evaluation of the DFC program will be to
examine and evaluate hypotheses related to the following three primary objectives:
Assess whether the DFC program has made an impact on reducing the substance
abuse outcomes at the community, state, and national level.
Determine if there are specific factors that can be identified that are related to
effective substance abuse prevention.
Assess whether the DFC program has increased the capacity and effectiveness of
substance abuse coalitions.
Within these broad primary objectives, there are a number of specific questions and
hypotheses of interest to the evaluation including questions and hypotheses related to
capacity:
Stage‐of‐development: What evidence exists to demonstrate whether DFC coalitions
achieve their performance targets and transition to a higher level of development?
What percentage move to a higher developmental level and what is the average
length of time needed to advance? What are the critical factors necessary in moving
to the next stage?
Increase in Evidence‐Based Programs, Policies, and Strategies: What evidence exists
to demonstrate an increase in evidence‐based programs, policies, and strategies in
coalition communities?
-1-
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
Sustainability: What evidence exists that demonstrates the sustainability of DFC
coalitions?
Increased National Capacity: To what extent has the number of communities with
established coalitions increased?
Successfulness of the DFC Mentoring Program: What evidence exists that
demonstrates a relationship between DFC mentee success and any specific mentor
characteristics or activities?
Impact of Technical Assistance on Data Collection, Application, and Implementation
of Environmental Strategies: What evidence exists that supports or negates an
association between the provision of technical assistance and increased data
collection and application and/or use of evidence‐based strategies by coalitions?
Does receiving technical assistance increase the likelihood that a new coalition will
subsequently obtain new DFC funding? Do these relationships vary with the source
of the technical assistance?
There are also several evaluation questions and hypotheses related to community outcomes
including:
Relationship Between Activities and Reduction in Substance Abuse Rates: What
evidence exists to demonstrate a relationship between DFC coalition activities and
reductions in substance abuse rates in their target communities for:
o 30‐day use (tobacco, alcohol, marijuana)
o Age of onset (tobacco, alcohol, marijuana)
o Other use measures
Relationship Between Activities and Improvements to Risk and Protective Factors:
What evidence exists to demonstrate a relationship between DFC coalition activities
and improvements in their target communities’ risk and protective factors, such as:
o Perception of risk (tobacco, alcohol, and marijuana)
o Perception of parental and/or peer disapproval (tobacco, alcohol, and
marijuana)
o Other factors
Composition/Collaboration: What mix of agencies and types of collaboration are
most associated with improvements in community substance use as well as risk and
protective factor outcomes?
Relationship Between Substance Abuse Outcomes and Explanatory Factors: What
evidence, if any, illustrates an association between differences in outcomes and such
factors as geographic location (urban/rural/suburban) or socio‐economic status?
Effectiveness of Strategies: What are the most effective strategies? What mix of
strategies led to positive community changes? Is there any relationship to type,
level, and coordination of outside funding streams?
-2-
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
This evaluation will largely rely on self‐reported information from coalitions as the core data
elements to address these questions. A combination of descriptive and more advanced
statistical modeling will be employed to address specific questions.
2.0 EVALUATION FRAMEWORK, TYPOLOGY, AND DESIGN OVERVIEW
An evaluation framework will be employed to guide the evaluation and to address the
objectives presented in Section 1.0. This framework is based on an extensive review of scientific
literature for conducting evaluations of substance abuse coalitions (Fawcett et al. 1997;
Francisco et al. 1996; Goodman et al. 1996; Mitchell et al. 1996; Stevenson et al. 1996), and will
be used to resolve the inherent difficulties and limitations associated with assessing changes in
substance abuse outcomes at the community level.
One such limitation has been the number of coalitions that have been simultaneously
examined. Typically, relatively few coalitions have been examined at the same time, reducing
the ability of statistical tests to identify significant improvements. The DFC program, however,
benefits from a relatively large number of participating coalitions with 769 current participants
and the addition of approximately 300 participants anticipated during the evaluation period.
However, most if not all of the information utilized in this evaluation will be self‐reported by
coalitions, including substance abuse outcomes and explanatory factors. One challenge of the
evaluation will be to ensure that the amount and quality of the reported data is sufficient to
facilitate reliable statistical analysis.
Previous evaluation efforts have also struggled to find significant impacts at the community
level. During the past twenty years, community coalitions have been typically evaluated against
ultimate substance abuse outcomes, such as the reduction in 30‐day use of tobacco by youth,
rather than against processes, capacity, and other immediate and intermediate outcomes
appropriate to their developmental stages; thus, only modest impacts of these community‐
based efforts were found (Merzel and D’Afflitti 2003; Berkowitz 2001). To address this
challenge, our evaluation framework views DFC coalitions as embedded in a developmental
process that can be tracked across certain dimensions and different developmental stages to
implement prevention interventions; to attain immediate outputs, intermediate outcomes, and
substance abuse outcomes; and to achieve sustainability and long‐term health and behavioral
impact. Further, the framework recognizes that the actions, activities, outcomes, and impacts
of DFC coalitions are conditioned by social, cultural, and environmental factors within
communities, including technical assistance, training, and mentoring provided by ONDCP and
others. DFC community coalitions at different stages of development likely use different
processes, have different capacities, and produce different outcomes. The degree of success of
a DFC community coalition should thus be evaluated against its ability to achieve targeted goals
relative to its stage of development (i.e., goal attainment).
In short, the framework underlying the national evaluation is a causal chain showing that DFC
coalition functions and activities lead to immediate, intermediate, substance use/abuse, and
long‐term outcomes as the coalition “matures” and develops its capacity (see Figure 1). Each
component of the framework represents a measurable hypothesis for the national evaluation
-3-
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
to assess. DFC coalitions are the catalysts that influence community dynamics, community‐level
collaborative activities, and community strategic planning. The composition/structure,
characteristics, and capacity of DFC coalitions determine coalition functions and activities, such
as comprehensive strategic planning, community‐wide collaboration, leveraging/redirection of
funds, system changes, and enhancement of policies, programs, and strategies. These coalition
functions and activities, in turn, are expected to produce immediate coalition outputs and
outcomes, such as increased prevention capacity and increased use of evidence‐based
prevention programs, activities, and strategies. The combination of coalition functions and
activities as well as the immediate outputs is expected to lead to intermediate outcomes, such
as decreased risk and increased protective factors at the community level. These intermediate
outcomes, in turn, are expected to lead to substance use/abuse outcomes, such as reduction in
30‐day use of alcohol, tobacco, and marijuana. These behavioral changes in substance
use/abuse are expected to lead to desired long‐term outcomes relating to health, crime, and
safety behaviors.
Figure 1. Evaluation Framework.
2.1 EVALUATION TYPOLOGY
The national evaluation of the Drug Free Communities Support Program is developing a
typology of community coalitions in order to better understand how they successfully reduce
substance abuse. A typology that is based on maturation or stages of development can
demonstrate that as a coalition develops their capacities to conduct internal functions needed
for its development and maintenance as well as the capacities for its external functions (those
needed to prevent substance abuse), they are more likely to reduce substance abuse. A stages
of development typology can also show how the support system (e.g. funding, technical
-4-
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
assistance and training, etc.) advances coalitions through the stages in order to enhance
community‐wide capacity to prevent substance abuse. The proposed coalition typology
framework is developed from the existing research literature and the experience of
practitioners. It merges three main themes in the literature: maturation (coalitions get better
over time); coalition processes (e.g. strategic prevention framework); and coalition capacities
(e.g. knowledge, skills, resources, and relationships needed to meet goals and achieve
functions).
2.1.1 Defining Coalition Stages of Development
Coalitions are dynamic entities embedded within dynamic contexts. Given the proper attention
and nurturance, mature and sustainable coalitions may develop. That is, they may
systematically progress from simpler to more complex durable forms of organization and
activities. This typology rests upon a conceptualization of coalitions moving through four stages
of development: (i) Establishing; (ii) Functioning; (iii) Maturing and (iv) Sustaining. Successful
movement through the stages of development is determined by the extent to which the
coalition has the capacity to perform requisite functions for each stage. Figure 2 displays a
complete list of capacities. Here they are briefly summarized for each stage of development.
Establishing coalitions face having to learn complex tasks in the establishing stage of coalition
development. They begin by recruiting a critical mass of active organizational members and
engaging representation from a broad spectrum of key community sectors and constituencies.
They must then establish an organizational structure and procedural operations which will
produce, from among actors from different community sectors, a collaborative team which is
both cohesive and task focused. After mobilization and initial structuring, the coalition must
build its capacity for action by ensuring that its members have sufficient knowledge and skills to
both participate (participation skills) and make informed decisions about substance abuse
prevention activities (content skills). And of course the coalition members must reach
consensus around the purpose or function(s) that the coalition will perform. Over time,
coalitions can “cycle” back through these tasks, especially those where capacities are weak.
Functioning coalitions turn their attention from an internal focus to an external focus and are
paying attention to the identification, coordination and integration of prevention programs and
services delivered by partners in the community. Here the coalition identifies community needs
and resources, prioritizes the needs and derives objectives and identifies an array of evidence‐
based prevention programs and services for partners to deliver to achieve objectives. Capacities
must be built depending upon the activities, programs, or strategies chosen and
implementation plans drawn up specifying responsibilities, timelines, and evaluation activities.
The coalition continues to learn and begins to develop proficiency in performing its intended
functions.
Maturing coalitions engage in a transformation of function. That is, these coalitions recognize,
either through its own internal evolution or external intervention, a mission or purpose broader
than being solely a coordinator of programs and services. Sometimes this involves the
incorporation of environmental strategies such as access, enforcement and policy change into
-5-
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
its prevention interventions. Coalitions soon recognize that they are ideal vehicles for
environmental strategies because, unlike the delivery or coordination of programs, it often
takes multi‐sector collaboration to accomplish environmental change. A coalition that
integrates programs, community‐wide activities, and environmental strategies for synergistic
impact is often called a “comprehensive community intervention”. Occasionally, coalitions
further transform their functions into becoming an “intermediary or community support
organization.” While not an inevitable result of maturation (that is, many coalitions remain
“comprehensive community interventions”), these few coalitions have developed mastery in a
broad array of capacities and are ready to teach others. The intermediary or community
support organization creates community change indirectly, impacting communities by building
the capacity of other local organizations and institutions, offering an array of training and
technical assistance services.
Sustaining coalitions have “institutionalized” themselves and their functions as an ongoing part
of community operations. They have become not a three‐year, grant‐funded entity, but part of
the fabric of the community. This can be as simple as having a community organization, such as
a school, adopt a prevention program initially sponsored by the coalition as a permanent part
of its curriculum, or as complex, having the coalition receive municipal funding for regular
strategic planning and evaluation of community efforts to address substance abuse problems.
The essential ingredient for the sustaining coalition is that its structure, procedures and
functions are validated and affirmed through resources (e.g., money, space, staff) and
recognition (e.g., it speaks with authority and is consulted by decision‐makers for the functions
it performs).
2.1.2 Operalization of Typology
The Coalition Classification Tool (CCT) is designed to describe and classify Drug Free
Communities (DFC) coalitions. The DFC coalitions and their communities are as diverse as the
U.S. in terms of cultural, sociopolitical, and historical context. Each coalition, furthermore, can
implement a variety of different kinds of prevention interventions. Capturing and
communicating this incredible diversity demands the development of commonly accepted
coalition dimensions. The Coalition Classification Tool collects data which produces a succinct
summary of important coalitional dimensions found throughout the literature: (i) Coalition
Development and Maintenance; (ii) Primary Intervention Foci and (iii) Capacities. Coalitions
develop as they increase their mastery of key functions needed to prevent substance abuse.
Key functions include: coalition development and management, coordinating preventing
programs /services, implementing environmental strategies, and serving as an intermediary
support organization to build the capacity of other organizations to do their part in preventing
substance abuse. Coalition development also entails mastery of capacities needed to perform
each of the five steps of SAMSHA’s Strategic Prevention Framework: conduct assessments,
mobilize and/or build capacity, develop a comprehensive plan, implement strategies, and
evaluate and plan for sustainability. In Figure 2, each cell of the figure describes functions
necessary for a coalition to engage in one of the five Strategic Prevention Framework (SPF)
Steps.
-6-
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
Figure 2. SPF Capacities Needed for Coalition Primary Functions
-7-
September, 2008
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
Data collected by the Coalition Classification Tool (CCT) will be used to estimate a coalition’s
knowledge and skill level (e.g., capacity) on a scale from one (Novice) to five (Mastery) and
provide a rating of coalition capacity for each function. The overall balance among capacities
and their particular configuration across intervention foci will be used to infer a coalition’s
stage of development. Stage of development is intended to capture a dynamic process in which
coalitions may systematically develop or progress from simpler to more complex forms of
organization and activities (described below). The CCT thus captures the common pathways and
the defining capacities associated with a growth trajectory that may be shared by otherwise
diverse coalitions. More specifically, the CCT will facilitate comparisons of progress and
effectiveness across coalitions, will assist the Coalition Institute in tailoring their training and
technical assistance to match each coalition’s stage of development, and will provide more
appropriate targeting of evaluation outcomes at both national and local levels based on stage
of development. A description of each dimension, theoretical rationale for inclusion and a brief
summary of supporting empirical evidence are provided below.
Coalition Development and Maintenance is the foundation upon which all other activities are
built. Coalitions must first learn the fundamentals: their own organizational development and
procedural management. Starting with varying levels of human and material resources (e.g.,
funding, technical assistance, and varied community representation), coalitions must develop
rules and procedures for working together. This is no small task for a coalition, by definition a
voluntary group of “equal” organizations that may have little history in working together.
Figure 2 presents the kinds of capacities necessary for coalition development and maintenance
at each of the five Strategic Prevention Framework steps. Notably, these capacities range from
collaboration skills (e.g., recruitment of appropriate member organizations, establishing regular
contact between coalition and community sectors) through leadership and participation skills
(e.g., creating consensus, facilitating discussions, addressing conflicts), to specialized knowledge
(e.g., developing cultural competence, establishing evaluation procedures). These are also not
tasks that can be accomplished once and then forgotten. As indicated in Figure 2, coalitions
may “cycle” back to previous steps (e.g., building content knowledge or participation skills
among new members) because a capacity was initially inadequate to the task and produces
later problems (e.g., proceeding to action without building a consensus on purpose leads to
conflict among members).
Empirically, some of these capacities have been associated with intermediate outcomes. Kegler,
Stecker, McLeroy and Malek (1998) found coalition factors such as communication, cohesion
and complexity related to the extent of implementation in ten local tobacco control coalitions.
Florin, Mitchell, Stevenson and Klein (2003), using data from thirty‐five municipal substance
abuse prevention coalitions, found that coalitions that “take care of business” in terms of
building their own capacity were more likely to be viewed as producing community change.
Specifically, coalitions that had done a better job in developing a task‐focused social climate, in
increasing their members’ perceived skills, and in making more extensive linkages with
community organizations, were more likely to be rated by community leaders as producing
effects (i.e., resources devoted to prevention, connections between organizations, community
attitudes concerning alcohol and other drugs; policies of organizations) one year later.
-8-
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
Primary Prevention Functions include the actual approaches to prevention coalitions adopt
after they have established their fundamental organizational structures and operating
procedures. Coalitions then develop further by focusing on the identification and coordination
of prevention programs and services or engaging in environment strategies (e.g., mobilizing
inter‐organizational collaboration for prevention policy, enforcement, media advocacy, etc.).
Coalitions may choose to focus on both functions. Other coalitions may evolve into an
“intermediary support organization” working indirectly on prevention by building the capacity
of other organizations. Each primary prevention function requires specific capacities within
each of the Strategic Prevention Framework steps that must be cultivated within the coalition
to attain mastery of the function and thus progress through the stages of development. Most
coalitions develop a combination of these functions. Not all coalitions will want to (or be
capable of) attaining mastery in all functional areas.
Prevention programs and services in a community are often designed for specific populations
(e.g., refusal skills for junior high students; parenting for single parents of elementary school
children) and intended to change perceptions, attitudes or skills. In any particular community
there might be one specific program or several prevention programs intended to produce
cumulative or synergistic immediate outcomes on individually focused risk and protective
factors. Empirical evidence for program effects continues to accumulate, as evidenced by
SAMSHA’s growing list of NREPP programs (May 2005). The capacities necessary for this focus
are displayed in Figure 2. Ineffectual and inadequately implemented programs not only waste
resources, but they may also cause disillusionment among implementers and policymakers who
see no impact. Therefore, interventions are necessary which help to influence the
dissemination and adoption as well as the fidelity of implementation of research‐based
prevention programs at the local community level (Rohrbach, Graham & Hansen, 1993). The
coordination, development, and integration of prevention services are a “natural” starting role
or function for most coalitions.
Increasingly, coalitions realize the greater and unique impact they can have by engaging in
environmental strategies—mobilizing inter‐organizational collaboration for prevention policy,
enforcement, and media advocacy. Such collaboration is seen as necessary because policy and
media advocacy initiatives are difficult to implement and often require multi‐sector efforts. For
example, Klitzner (1998) articulated a distinction between prevention strategies that attempt to
alter the environments in which individual children grow, learn, and mature and those that
attempt to alter the shared environments which influence all children. Three factors in the
shared environments that shape both positive (healthy) and negative (health‐compromising)
behavior are norms, availability, and regulations. Norms are basic orientations concerning the
acceptability of specific behaviors for a specific group of individuals. Availability is defined in
terms of the cost or difficulty of obtaining a commodity such as alcohol, marijuana and
cigarettes. Regulations are formalized laws or policies (of governments, public agencies or
private organizations) that control availability, codify norms and specify sanctions. Evidence has
been mounting that environmental strategies can impact consumption of alcohol, tobacco, and
other drugs and associated consequences (Birckmayer, Holder, Yacoubian and Friend, 2004).
Figure 2 lists the capacities necessary for a coalition to engage in such environmental strategies
and they are particularly suited to coalitions. In fact, there is growing agreement among many
-9-
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
prevention researchers and practitioners that coalitions will be most effective when they
include strategies that attempt to change both community conditions and norms as well as
those that focus only on changes in individual skills and competencies. “Comprehensive
community interventions” combine two rows in Figure 2: individual (program) and
environmental change strategies. Implemented across multiple settings, these interventions are
designed to prevent dysfunction and promote well‐being among population groups in a defined
local community.
Although theoretically compelling, positive outcomes have been produced only approximately
one‐third of the time (Roussos and Fawcett, 2000; Wandersman and Florin, 2003), perhaps
because of insufficient coalition capacity to undertake complex interventions. This has led some
coalitions to evolve into intermediary or community support organizations. Intermediary or
community support organizations support prevention interventions using a broad and
multilevel array of strategies including: training programs for skills development; telephone and
on‐site consultation; information and referral services; mechanisms for creating linkages among
coalitions; methods of recognizing group achievement; publications and other public education
materials (Chavis, Florin, & Felix, 1992; Florin, Mitchell, & Stevenson, 1993). Coalitions which
engage in this function focus on the conditions in which prevention programs are developed,
implemented, and evaluated, and works to build the capacity of other organizations and
institutions. Capacity building interventions have been advocated for many types of
community‐level interventions, from grassroots community coalitions to replications of
community trials for prevention interventions (Pentz, 2000; Roussos & Fawcett, 2000; Wolff,
2001).
Capacities to perform these primary functions need to improve in order for coalitions to
increase their impact. Coalition Capacities are the knowledge and skill sets necessary to
successfully engage in each of the five steps of the Strategic Prevention Framework. Capacity
must always be defined in terms of capacity “to do what?”. That is, capacities will vary by which
step of the Strategic Prevention Framework is being addressed, whether a coalition’s basic
foundations (structure and procedures) are in place and which primary focus (or foci) the
coalition is choosing to implement. Crisp, Swerissen & Duckett (2000) described several kinds of
professional, organizational, and systemic capacities. Capacities are important because
intervention programs are characteristically difficult to implement (Lipsey & Cordray, 2000),
and more so when they involve complex challenges in collaboration, organization, planning,
and coordinating multiple programs and policies (e.g., Florin, Mitchell & Stevenson, 1993;
Wandersman, Goodman & Butterfoss, 1997). Empirically, it has thus far been easier to
demonstrate that capacities have been developed in the short‐term, for example as the result
of a training curriculum, than it has been to demonstrate that intentional capacity development
delivered through training and technical assistance has produced improved outcomes (Florin et.
al, in press).
Measuring Stage of Development represents the overall balance among a coalition’s capacities
and their particular configuration across intervention foci. Coalitions are dynamic entities
embedded within dynamic contexts. As such, they may systematically develop or progress from
simpler to more complex forms of organization and activities. For example, as indicated in
- 10 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
Figure 2, a coalition may transform its primary focus from one devoted exclusively to the
identification, coordination, and integration of programs and services to a broader perspective
on environmental strategies. Alternatively, many coalitions begin with the broad perspective,
but lack the internal capacity to identify and implement environmental strategies. As coalitions
mature through the stages of development, they progress from learning internal and external
coalition capacities, to developing proficiency with them, to finally mastery. Coalitions are
learners when they recognize their needs for capacity development and mentoring and need
assistance finding resources. Proficient coalitions know how to implement environmental
strategies and how to access resources, but they may need additional guidance to do so. They
follow procedures and models mapped by others. A level of mastery is reached when coalitions
“instinctively” assess, build capacity, implement, and evaluate prevention strategies and are
able to design new and innovative ways of implementing environmental strategies that are
tailored for their community. Coalitions with mastery of prevention functions are able to
mentor others and have direct relations with the resources.
It is expected that establishing coalitions will primarily be at a learner level of capacity on
internal and external functions. Functioning coalitions begin to reach a level of proficiency on
most functions. Mature and sustaining coalitions have mastery of most functions. Sustaining
coalitions are also institutionalized and sustainable within the community (see Table 1).
Table 1. Prevention Coalition Stages of Development
Stage of
Development
Description
Level of
Competency to
Perform Functions
Establishing
Functioning
Maturing
Sustaining
Initial formation
with small
leadership core
working on
mobilization and
direction
Follows the completion
of initial activities, focus
on structure and more
long range
programming
Stabilized roles,
structures, and
functions;
Confronted with
conflicts to
transform and
“growing pains”
Established
organization and
operations, focus
on higher level
changes and
institutionalizing
efforts
Primarily learner
Achieving proficiency;
still learning and
developing mastery
Achieved mastery;
learning new
areas; proficient in
others
Mastery in primary
functions;
capacities in the
community are
sustainable and
institutionalized
The classification of each coalition into a specific stage of the typology is an important outcome
of the evaluation. This classification will initially be accomplished through an examination of the
factors and characteristics expected to be necessary for a coalition to qualify for a specific
stage. Following the first year of the evaluation and subsequent to the collection of detailed
information from coalitions, statistical analysis techniques will be employed to test and modify
the assumptions regarding the characteristics that are significant drivers for classifying a
coalition into a specific stage of development (see Section 4.2). Again, the typology will provide
a roadmap to guide the national evaluation in assessing progress toward established milestones
by comparing current values on the classification algorithm with those at baseline and during
previous years. The definitions of outcomes and measures for these outcomes in the typology
- 11 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
will help us to identify outcomes expected at each coalition stage, providing stage‐specific
criteria for measuring developmental progress. Similarly, the typology will facilitate evaluation
of capacity and processes at each developmental stage.
2.2 OVERVIEW OF EVALUATION DESIGN
Using the evaluation framework as a foundation, our evaluation will focus on both a direct and
indirect assessment of the success of coalitions at impacting substance abuse in their
communities. First, similar to previous evaluation efforts, we will examine whether the
activities, initiatives, strategies, etc. of DFC coalitions have an impact on substance abuse
outcome measures of interest, such as 30‐day use of tobacco. However, as coalitions develop,
we will focus the outcome analysis increasingly on Mature or Sustaining coalitions. The impact
of DFC coalitions on substance abuse prevention will also be examined using an indirect
approach because it may not be reasonable to evaluate a direct linkage for less advanced
coalitions. However, if the DFC program can be found to be an important factor for helping
coalitions mature into advanced coalitions (i.e., if characteristics related to advancement in
development can be identified as components of the DFC Program), and if the link between
advanced coalitions and substance abuse outcomes can be established, then it would be logical
and scientifically appropriate to conclude that the DFC program is effective in reducing
substance abuse outcomes; directly for advanced coalitions and indirectly by fostering an
environment where less advanced coalitions can become advanced coalitions.
Trends in substance abuse outcomes in communities targeted by DFC coalitions will be
compared to the corresponding trends in communities that are not specifically targeted by a
DFC coalition indirectly using state and national level data. This analysis differs from an
approach where comparison communities are selected and matched to DFC coalition
communities and a direct comparison is conducted between the two different types of
communities. Employing a comparison‐community type approach has significant challenges and
barriers to obtaining quality and comprehensive information. These challenges include the
need to identify appropriate comparison communities that can be determined to be “similar”
to a DFC coalition community, the need to identify key informants to provide community
information (a significant challenge in communities that do not have a coalition), convincing the
contact person to continue to provide this detailed information throughout the evaluation, and
the questionable quality of the outcome information that could be obtained. Due to these
limitations and resource constraints, our approach will rely on extracting indirect surrogates for
substance abuse outcomes for communities not targeted by a DFC coalition from existing
national and state surveys to serve as a comparison to DFC communities.
Although this approach will limit the number of explanatory factors available for advanced
modeling, it has the decided advantage in that it does not rely on a direct data collection
activity conducted as part of this evaluation effort. Unfortunately, publicly available information
is typically only available at a broad geographic level, which prohibits specific community‐to‐
community comparisons. However, as part of our approach, we intend to conduct a feasibility
assessment to determine if community‐level information could be obtained from some of the
national surveys. If available, this information could be used to refine the comparative analysis.
- 12 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
2.2.1 Limitations of the Study
As with any evaluation effort, there are competing constraints and resource needs that impact
the design and implementation of the evaluation. In this evaluation, one primary consideration
for the design is the ability to identify, recruit, and retain a suitable comparison community
during the evaluation period. To be as robust as possible, the DFC evaluation will require
information on substance abuse outcomes and covariates of interest from non‐DFC
communities as a basis of comparison. However, the acquisition of this data will be extremely
difficult for a number of reasons, including the limited resources available for the evaluation;
the nature of the outcomes in question requiring the examination of long‐term trends; and the
lack of a centralized substance abuse coalition infrastructure in comparison communities. As a
result, collecting information from non‐DFC coalitions or in communities that lack a substance
abuse coalition was determined to be impractical for this evaluation. Instead, this evaluation
will focus on using publicly available data sources such as the results of national surveys, and
more intensive statistical analysis techniques to define substance abuse outcomes for
comparison communities. This approach has the advantage in that outcome data will be
available for trend comparison but has the limitation that many of the factors that are being
collected through the COMET will not be available for the comparison analysis.
A second limitation of the study is that effecting positive changes in substance abuse at a
community level often requires significant effort over a sustained period of time that may
exceed the five year evaluation period. The evaluation will seek to overcome this challenge
through the use of historical progress reports, and an examination of the influence of the DFC
in helping coalitions become mature.
Finally, due to resource constraints, the evaluation is relying upon information that is self‐
reported by coalitions as the core information for analysis. This includes both explanatory and
substance abuse outcome information. As with any self‐reported information, the quality and
the potential for bias in this information represent limitations of the study. While it will not be
possible to completely verify the accuracy of the reported outcome information, the evaluation
will include procedures to assess the quality of the collected information.
3.0 DATA COLLECTION AND MANAGEMENT
3.1 DATA SOURCES
There are several sources of information that will be utilized as part of this evaluation.
However, the core information base for the evaluation will come from two primary sources—a
Coalition Classification Tool (CCT) and a Semi‐Annual Progress Report. All information for these
core instruments will be self‐reported by coalitions and will be completed by the director or
assistant director of the grantee organization. This information will be collected through an on‐
line computer system (see Section 3.2).
The Coalition Classification Tool (CCT). This survey is a data collection tool based on the
coalition typology developed for the national evaluation. It is used to collect information on the
- 13 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
coalition composition/structure, characteristics, capacity, functions, and activities for the
purpose of classifying coalitions into different stages of development to facilitate the evaluation
effort and to prioritize technical assistance and training activities. This CCT will be fielded once
annually beginning in January 2006.
The Semi‐annual Progress Report. The semi‐annual progress report is a data collection
instrument filled out by the director or evaluator of the grantee organization. Following a set of
standard data requirements, the semi‐annual progress report collects process, capacity, and
outcome data covering the six months being reported. The focus of this progress report is to
capture information that can be used to monitor and track the grantees, as included in their
grant requirements.
While integrated, these two data sources represent different data collection activities and will
therefore collect different data from coalitions. That is, coalitions will not need to provide the
same information across both of these data collection tools. Also, it is again important to note
that there are potential drawbacks to relying on this data as the core information for the
evaluation. Because of staff turnover, consistent and systematic reporting will not be
guaranteed. This can affect the reliability of reported data due to variability in understanding of
the reporting requirements, familiarity with the coalition, and variability in interpreting items
and measures. Despite these limitations, these sources will provide the most useful information
for assessing the performance of the DFC coalitions because they can be collected at the
coalition or community level, they are specific for a particular coalition, they can be readily
linked with other data for a specific coalitions, and they will be consistent across all of the DFC
coalitions for the duration of the evaluation. One critical component of the evaluation will be to
assess the quality of the collected, self‐reported data (see Section 3.3).
The self‐reported core data described above will be supplemented, if available and attainable,
with information from the coalitions grant applications, outcome information from prior years,
and resource and budget information. Grant applications will be reviewed, and relevant data
elements will be extracted to supplement the process, capacity, and outcome data collected.
The key data elements to be extracted from the grantee applications include information on
community baseline substance abuse, cultural and other contextual conditions, as well as
information on resources and capacity at the time the grantee applied for the DFC funds. The
DFC grant budget management systems currently managed and maintained by the Office of
Juvenile Justice Delinquency Program (OJJDP) and SAMHSA collects information on funding by
grantee and by year. We would like to be able to extract budget information from this system
and merge it with the core evaluation data to establish level of funding as an additional
covariate or factor of interest.
Information on substance abuse outcomes will be obtained from national and state‐specific
publicly available sources such as the National Household Survey on Drug Abuse (NHSDA), the
Youth Risk Behavior Survey (YRBS), and others, to the extent possible. Only data sources that
have consistent outcome measures to those collected directly from coalitions will be
considered. Again, this information will be used to extract a surrogate measure for substance
abuse outcomes in communities that are not targeted by a DFC coalition.
- 14 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
3.2 DATA COLLECTION APPROACH
The primary source of data for this evaluation will be semi‐annual progress reports submitted
by coalitions. This information will be collected through the use of a web‐based system,
referred to as the Coalition On‐line Management and Evaluation Tool (COMET). The COMET will
provide a central data acquisition and management system that systematically collects
information from all DFC grantees on an ongoing and annual basis. Battelle is responsible for
determining the functional requirements of this computer system including content, flow,
reporting and data extraction requirements, etc. However, the formal development and
implementation of this system is outside of the scope of this evaluation effort.
The COMET will capture information from coalitions on aspects related to structure and
characteristics, internal capacity, intended functions, immediate and intermediate outcomes,
and substance abuse outcomes in the coalition’s targeted community. Semi‐annual progress
reports will contain information such as coalition membership that may change on a semi‐
annual basis. An annual progress report will provide information on items that are expected to
change less frequently, such as intermediate outcomes. Finally, a separate module of the
COMET, the Coalition Classification Tool will capture information that can be used to classify
each coalition into a stage of development. Information from external sources will also be
collected and utilized as part of the evaluation, particularly to obtain explanatory variables such
as funding levels or additional outcome measures, such as substance abuse outcomes.
The COMET will be designed to be consistent with the strategic prevention framework (SPF)
and will have separate modules for the five elements of the SPF (assessment, capacity,
planning, implementation, and evaluation). Additionally, the COMET will have separate
modules for the typology instrument, administration, utilities, and reporting. Evaluation data as
well as information needed by ONDCP and SAMHSA to monitor and track DFC Grantees will be
captured within each of these modules. The COMET is envisioned as an interactive data system
whereby the data elements, such as goals, objectives, and activities are linked by the grantee as
they enter information.
Coalitions will have continuous access to the COMET and can enter information at any time.
However, each Coalition will be required to submit progress data on a semi‐annual basis and
additional information to support the evaluation on an annual basis (e.g., the CCT). COMET is
expected to have a number of features that will provide additional training and logistical
support to DFC coalitions. For example, the COMET will have the capability of capturing profiles
of coalition’s members, thus serving as a database repository for each coalition to manage their
mailing lists. Other features will include the ability to “stop‐and‐start” data reporting and
collection, as well as the ability to generate and view standardized reports for their coalition or
across all coalitions.
- 15 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
3.3 DATA ASSESSMENT AND MANAGEMENT
3.3.1 Data Quality Assessment
As discussed in Section 3.2, the primary data utilized for this evaluation will be provided as self‐
reported information by coalitions either through the semi‐annual progress reports or through
the COMET. As with any self‐reported data, it will be critical to assess the quality and reliability
of these data prior to its inclusion in statistical analysis. For example, some coalitions may
report outcome percentages that were not collected through a rigorous sampling design, have
sample sizes too small to be meaningful, or are otherwise questionable.
A thorough data quality assessment (DQA) will be performed following each data collection to
determine if there are any serious data quality issues that could impact the evaluation
conclusions and should be addressed before conducting statistical analysis of the data. One
focus of the DQA will be on the outcome measures as they represent response variables for the
study. As noted, the outcome measures will be community‐level statistics obtained from
surveys that will be conducted independently from this evaluation. It is the coalition’s
responsibility to identify the appropriate data source, locate the data corresponding to the
outcomes and strata requested in this evaluation, and enter them accurately into the data
collection instrument. This process may potentially lead to data that are below the minimum
data quality standards needed to conduct an unbiased evaluation. This can result from among
other things, significant amounts of missing or invalid data, evidence of inaccurate data, and
the use of unreliable methods by coalitions for collecting outcome measures.
Coalitions are asked to report outcome measures, but are not mandated as to how they obtain
the requisite information. That is, each coalition may choose to employ a different survey
technique to obtain this information. Therefore, there is the potential that some coalitions may
rely on techniques that are known to be biased. As part of the information collected from
coalitions, data on the instrument used for collecting outcome measures will be requested. For
example, coalitions will be asked to indicate the source of their outcome data—state survey,
established community survey, or custom survey, for example.
Outcome measures using an established state or community survey are more likely to yield
scientifically valid and representative results for the community. Outcome measures collected
using other methods (e.g. use of custom surveys) are more likely to be biased or
nonrepresentative, and additional information will be sought from coalitions that report using
these methods to evaluate the validity of the reported outcomes. If grantees indicate the use of
a custom survey, they must have the survey reviewed by the evaluation team and approved by
their Project Officer.
If data quality issues cannot be resolved by DFC Project Officers in conjunction with the
coalition, data from that coalition may be excluded from the statistical analyses, which will
reduce the effective sample sizes and resulting statistical power of hypothesis tests for the
evaluation.
- 16 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
An additional data assessment that will be conducted will be to verify that the outcomes
reported by coalitions represent their target community and not a larger or smaller geographic
area. This assessment will be conducted by comparing the responses among coalitions within a
particular State, comparing the responses of coalitions within a State to the YRBS state profiles,
and examining responses to the data element “Is the geographic area covered by this data
larger, smaller, or the same as your target area?” Coalitions that provide outcome data that is
not focused on their target area may not be included in the analysis.
3.3.2 Data Management
Data are downloaded from the COMET by the COMET vendor and sent to Battelle at the
conclusion of each semi‐annual report period. Battelle backs the data up to a separate data
system controlled by the evaluation team.
Data that is extracted from the COMET will be compiled into a SQL Server database, which will
be stored on a secure server without outside access. This SQL Server database will serve as the
main repository of data for the evaluation. As such, only permanent changes and corrections
will be made to this database. Data for specific analyses will be automatically extracted and
manipulated for statistical analysis using SAS®. Changes made to the data for a particular
analysis will not be made in the SQL data unless this change will be applied to every analysis.
Otherwise, data changes and modifications will be “soft‐coded” into the analysis SAS® code. The
Coalition’s grantee ID will be used as a unique identifier for all coalition records.
4.0 STATISTICAL ANALYSIS USING SELF‐REPORTED DATA
Under the evaluation framework discussed in Section 2.0, there will be five different statistical
analyses conducted as part of the core evaluation effort: 1) descriptive analyses, 2) assessment
of stage of development, 3) modeling substance abuse outcomes, 4) assessment of additional
evaluation questions and hypotheses, and 5) analyses to support GPRA reporting. The
combination of these five analyses frames will be used to determine whether the DFC program
is having an impact on reducing substance abuse. Modeling substance abuse outcomes will
provide a direct assessment of whether advanced coalitions are having an impact on substance
abuse outcomes. Evaluating progress in stage of development will provide an indirect
assessment of the likely positive impact of the DFC program through demonstrating a
relationship between coalition characteristics and progression in development. That is, if the
DFC program helps coalitions advance in their development, and coalitions who are advanced
in development are effective in reducing substance abuse, then it may be logical to conclude
that the DFC program is having an impact on reducing substance abuse, even if this evidence is
not directly apparent for every coalition. Investigating additional evaluation questions regarding
coalition capacity and conducting analyses to support GPRA reporting will help to illuminate
inherent relationships in how coalitions operate and change over time.
- 17 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
4.1 DESCRIPTIVE ANALYSIS
One of the implicit objectives of this evaluation is to provide information to ONDCP and others
regarding the status of the DFC Program and nature of coalitions that are participating with this
grant program. This information provides a useful context and background to the analysis and
in understanding the conclusions reached by the evaluation. Generally, this characterization will
be accomplished through descriptive statistics, graphical representations, etc. using
information reported by coalitions. This characterization will be performed and summarized
annually or on an ad‐hoc basis as requested by ONDCP. Many descriptive statistics and reports
will be available to ONDCP and coalitions as part of the COMET. The following describes the two
aspects of this characterization, status of the DFC grant program and characterization of
participating coalitions.
The DFC grant program is expected to grow and change over the course of the evaluation as
current grantees complete their grants and new grantees are added. In assessing the DFC grant
program, it is important to understand the state of the program at each assessment point. In
particular, each year’s grantees represent a natural cohort that can be followed and compared
over time; due to economic, political, and other conditions this cohort may or may not be
comparable to grantees in other cohorts. This component of the evaluation will describe the
status of the program through summarizing elements such as:
Number of coalitions
The current and historical classification of coalitions
Geographic representation of communities served
Total population served by DFC coalitions
As with the overall program, the features and characteristics of coalitions are expected to
change over time. Examining how these characteristics change will provide insight into testing
and refining the overall evaluation framework and stage‐of‐development typology. The
responses to each component of the four data collection instruments will be summarized using
frequency distributions and other descriptive statistics. To facilitate the characterization of the
relationship between coalition characteristics and outcomes, both cross‐sectional (single point‐
in‐time) and longitudinal (the same measure over time) summaries will be constructed. A cross‐
sectional descriptive analysis may be conducted annually following each phase of data
collection while the longitudinal analysis will begin in the second evaluation year. The analyses
presented here concentrate on summarizing the distribution of potential response variables,
identifying important subsets of data that should be considered for future analyses, and
identifying important explanatory variables. Simple descriptive summaries will be provided to
describe the outcome data and other potential explanatory variables among the population of
DFC coalitions that have met the minimum data requirements described earlier. As appropriate,
side‐by‐side box plots, bar charts, and line graphs as well as descriptive statistical summary
tables will be prepared to illustrate and summarize the distribution of substance abuse
outcome measures as a function of community type, stage of development, and grade of
school. The summary statistics provided in the tables will be sample size (number of
- 18 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
communities), mean, standard deviation, minimum, 10th percentile, 25th percentile, median,
75th percentile, 90th percentile, and maximum. For the outcome measures, these summary
statistics will be provided for the whole community, as well as for each community type, stage
of development, and school grade. Summary statistics will also be used to describe the
distribution of covariates by community type and stage of development.
One significant component of the exploratory analysis will be to examine those coalitions that
have the largest or most significant changes in substance abuse outcomes or the greatest
progression in maturation. These coalitions will be examined to determine if there are key
characteristics common to these coalitions that may be related to such a dramatic change in
outcomes. If these characteristics can be successfully identified, they may be candidates for
ONDCP to consider enhancing across all coalitions. Descriptive statistics such as the presence of
a specific characteristic, activity, function, etc. will be cross‐referenced with this group of
coalitions to determine if it is a shared characteristic of the group.
4.2 ASSESSMENT OF STAGE OF DEVELOPMENT
In this section we describe our approach to developing the coalition classification scheme as
well as validation. Our approach classifies the coalitions along a dimension of less mature to
more fully mature (see Table 1). The proposed coalition typology framework used by the study
team is developed from the existing research literature and the experience of practitioners. It
merges three main themes in the literature: maturation (coalitions get better over time);
coalition processes (e.g., SAMHSA’s Strategic Prevention Framework) and coalition capacities
(e.g., knowledge, skills, resources, and relationships needed to meet goals and achieve
functions). This typology rests upon a conceptualization of coalitions moving through four
“stages‐of‐development:” (1) Establishing; (2) Functioning; (3) Maturing; and, (4) Sustaining. As
shown, as coalitions move through these stages, they acquire greater sophistication with
respect to their organizational structure, capacity, and focus of efforts as well as in their levels
of competency to perform vital functions necessary to impact change.
Our model recognizes, however, that developmental progression may not be linear; coalitions
may progress and regress through developmental stages and change over the course of the
evaluation. For example, a Functioning coalition that loses a key coalition leader may regress to
an Establishing coalition while the coalition rebuilds, then become a Functioning coalition again
at a later date. Therefore, when assessing whether the DFC program has had an impact on the
stage of development for the grantee coalitions, it is important to assess the overall trend,
recognizing the often cyclical nature of coalition development.
Statistical approach. The Coalition Classification Tool (CCT) contains four six‐item scales
measuring coalition capacity and functions. Items are coded on a 5‐point scale as follows:
“Novice” (or score of 1 on the 5‐point scale) was defined that the coalition is still learning how
to perform the function in the various areas and could therefore benefit from assistance from
others; “Proficient” (or score of 3 on the 5‐point scale) indicates that the coalition thought they
were competent in performing the function; and, “Mastery” (or score of 5 on the 5‐point scale)
was indicated by those coalitions that believed they were at an expert level of performance in
- 19 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
the areas and could train or be of assistance to others in performing these functions. No labels
were associated with scores of 2 or 4, but the intention of the scales was that a 2 represent a
score between Novice and Proficient; and, 4 represent a score of between Proficient and
Mastery. Table 2 below gives examples of the items included in the scale of coalition
maturation.
Table 2. Examples of Questions Included on the CTT Measuring Stage‐of‐Development
Dimensions (Four Scale Items and 6 Sub‐Items within Each Scale)
Activity/
Functional
Areas (Sub‐
items)
Assessment
Scale (1‐5)
Capacity
(1‐5)
Coalition
Development and
Maintenance
Deciding which
skills and resources
will be needed,
assessing which
organization and/or
individuals to
recruit …
Building member
participation skills,
providing desired
training and
technical assistance
to develop coalition
structure..
Planning
(1‐5)
Building consensus
around coalition
mission, develop a
mission statement
and general goals.
Implementa‐
tion (1‐5)
Establishing the
coalition structure
and operating
procedures…
Stage‐of‐Development Dimension
(Item text has been abbreviated)
Question Labels Novice=1; 3=Proficient; 5=Mastery
Coordination of
Environmental
Intermediary or
Prevention
Strategies
Community Support
Program/Services
Organization
Compiling prevalence
and risk and
protective factors
data, prioritizing
needs…
Determining retail and
social sources of
substance availability to
underage youth,
knowledge of
community compliance
with local ordinances…
Build a solid
Developing a solid
knowledge base (e.g., knowledge base (e.g.,
familiarity with
definition, rationales,
evidence‐based
and evidence for
programs and
environmental
services) and required strategies)…
skills in program
design, activity
planning…
Analyzing and
Identifying a range of
selecting programs,
potential policy changes
services, and
and enforcement
activities that
activities, selection the
provides a best “fit”
best “fit with current
with community
community conditions…
conditions…
Arranging settings for Develop experience in
program delivery
carrying out a
(e.g., school, CBO),
sequenced social
creating public
marketing campaign..
awareness, recruiting
strategies…
- 20 -
Understanding current
knowledge and skills
among community
leaders, staff and
residents on
prevention strategies…
The capacity of other
organization,
community leaders,
and residents by
recruiting highly skilled
staff and consultants….
Designing learning
systems,
communications and
marketing plans,
integrated technical
assistance and training
plans…
Advertising, recruiting
and conducting a series
of workshops,
developing resource
centers or web site for
the distribution of
information, and
brokering resources
from state and national
resources.
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
Activity/
Functional
Areas (Sub‐
items)
Evaluation
(1‐5)
Planning for
Sustainability
(1‐5)
Coalition
Development and
Maintenance
September, 2008
Stage‐of‐Development Dimension
(Item text has been abbreviated)
Question Labels Novice=1; 3=Proficient; 5=Mastery
Coordination of
Environmental
Intermediary or
Prevention
Strategies
Community Support
Program/Services
Organization
Assessing member
satisfaction, skill
development….
Conducting process
and outcome
evaluations to refine
or eliminate
programs.
Monitoring
enforcement or
documenting changes
in social indicators to
measure policy change.
Planning for
changes in
leadership,
standardizing
operating
procedures….
“Institutionalization”
or incorporation of an
evidence‐based
program as part of
ongoing
organizational
operations in your
community.
Arranging for regular
prevention columns in
local newspapers or
securing line items in
organizations’ budgets
that institutionalize
prevention strategies.
(For example, training,
technical assistance/
consultation,
educational program,
and material etc.)
Monitoring satisfaction
and evaluating changes
in knowledge skills and
resources.
Planning for
sustainability of
capacity building
functions. Securing
ongoing funding or
institutionalizing
services into the
ongoing operations of
other community‐
based organizations.
Classification Methodology and Empirical Validation of Typology
Statistical approach. The statistical approach to this analysis involves two steps: (1) creating the
classification algorithm and (2) validation. To create the coalition typology, we will calculate a
mean score across each of the items in each dimension and overall for each coalition. In
addition, mean scores will be calculated for each of the three survey waves of the CCT.
Coalitions reporting average scores overall that were between 1 – 1.999 (novice average rating)
will be categorized as Establishing; Coalitions reporting average scores between 2 – 2.9999
(novice to proficient average rating) will be categorized as Functioning; Coalitions reporting
average scores between 3‐ 3.9999 (proficient average rating) will be categorized as Maturing;
and Coalitions reporting average scores between 4‐5 (highly proficient to mastery average
rating) will be categorized as Sustaining.
The typology will be validated using other items from the CCT and COMET. First, we will look for
internal consistency by assessing whether similar question items show consistent patterns of
response as those produced using the stage‐of‐development typology. For example, the CCT
asks coalitions about their capacity to perform key functions (Q27), how respondents best
describe your coalition (Q3), and whether the coalition has established a reputation for “being
able to get things done” related to the area of substance abuse prevention (Q16). Responses to
these items should map onto a coalitions’ status on the typology if the typology is valid. Second,
we will use external data to examine whether the leveraging of funding was related to the
proposed stage‐of‐development coalition typology. As coalitions become sustaining, they
- 21 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
should be less reliant on DFC funding (DFC funding should be a smaller percentage of the
coalition’s total funding).
4.2.1 Assessing Trends in Development
As previously described, one of the key questions of the evaluation will be to assess whether
DFC coalitions are progressing in development over time. It is important to note that this
progression may not be linear in nature and that coalitions may move forward and backwards
in development over the course of the evaluation. For example, a Functioning coalition who
loses a key coalition leader may change status to an Establishing coalition while they rebuild
only to once again become a Functioning coalition at a later date. Therefore, when assessing
whether the DFC program has had an impact on the stage of development for the grantee
coalitions, it is important to assess the overall trend in coalitions, recognizing the often cyclic
nature of coalition development.
In addition to descriptive analyses assessing these trends, polytomous logistic regression will be
used to assess the trends in development of the group of coalitions over time. Generalized
logits will be assumed for the model, which relaxes the assumption that stage of development
has to be an ordinal progression of growth with equal increases between stages. Under this
model, exponentiating the estimated regression coefficients will yield the estimated odds of
being in a particular stage of development versus a different stage of development. In statistical
notation, let Yikt = 1 if the ith coalition has the kth stage of development at time t. Then, let π ikt =
Probability (Yikt = 1/ xit, β ), where xit is a vector of covariates (explanatory variables) for the ith
coalition at time t and β is the parameters of interest estimated by the model. Then, the
generalized logit model fits the marginal probabilities of interest denoted by π ik , which
corresponds to the probability of coalition i being in stage of development k, for particular
values of the explanatory variables. The general form of this model is as follows:
π (x )
log k i = β k' xi , for k = 2, 3, 4 (the first stage‐of‐development serves as a reference).
π K ( xi )
In this model, candidates for explanatory variables would include time and the coalition’s initial
stage of development at baseline. Because there are multiple observations on the same
coalition over time, it is necessary to use specialized software that accounts for this inherent
clustering in the data. SUDAAN will be used to adapt this model for repeated measures using
generalized estimating equations (GEEs). The assessment of whether the DFC grantees are
progressing over time would be accomplished through examining the slope coefficient
associated with time in the model. A positive and statistically significant slope coefficient would
be an indication that DFC coalitions are advancing in development over time. A significant
negative slope coefficient would be indicative that DFC coalitions are decreasing in
development over time, while non‐statistical significance of this slope coefficient would be
indicative of no‐change, on average, over time. An illustration of this methodology is presented
in Figure 4, though not on a logit scale.
- 22 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
Logit
Scale
would
be used
for the
Model
Fitting
Stage of Development
5
Positive
Slope of
Fitted Trend
Line Would
Indicate
Significant
Maturation
of DFC
Coalitions
Over Time
4
3
2
1
0
0
1
2
3
4
5
6
Years as a DFC Coalition
Simulated Coalitions
A
B
C
D
E
F
G
H
Figure 4. Illustration of Modeling Stage of Development.
The logistic regression model will provide odds ratios that will be used to interpret the odds of a
coalition being in a higher stage of development compared to a lower stage of development as
a function of time and starting stage of development. Additionally, this model will enable
ONDCP to estimate the probability that a coalition who is in the program for X years and who
started at a particular stage‐of‐development will be in a particular stage‐of‐development.
4.3 MODELING SUBSTANCE ABUSE OUTCOMES
The modeling of substance abuse outcomes has two purposes for this evaluation. First, these
models will help to identify factors that are significant predictors of substance abuse outcomes.
For example, these models will be used to assess whether the stage of development, coalition
characteristics, capacity, activities, etc. are significantly related to substance abuse outcomes.
These models will primarily focus on analyses of coalitions that are part of the DFC program and
are described in Section 4.3.1. Another consideration of the evaluation is to evaluate trends in
substance abuse outcomes in communities targeted by DFC coalitions to the corresponding
trends in communities that are not specifically targeted by DFC coalitions (Section 4.3.2).
4.3.1 Identifying Factors Associated with Substance Abuse Outcomes among DFC Coalitions
The COMET represents a rich source of data on a number of potential covariates that may have
a relationship with reductions in substance abuse outcomes in DFC coalitions. One key
covariate is the stage of coalition development. However, other covariates include coalition
capacity, extent of activities, environmental strategies employed, and all other data elements
collected as part of the COMET. Regression models will be fit using a GEE approach to account
- 23 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
for the anticipated positive correlation in substance abuse outcome measures on the same
community over time. Specifically, the logistic regression models will express the proportion of
positive responses within a community as a function of a number of covariates.
Fixed and random effects inverse variance weighted regression using logit transformed
outcome proportions will be used to identify those covariates that are significantly associated
with substance abuse outcomes after adjusting for the presence of other covariates. Stage of
development and length of time that the coalition has existed will be retained in the model
regardless of their significance because these variables will be used to test for the significance
of any observed trends. The number of years of DFC funding will also be investigated as an
alternative to this time measure. The number of potential covariates included in the model may
need to be limited or reduced due to sample size considerations. Exploratory analysis may
reveal that two variables are highly correlated, in which case only one of the variables would be
included in the models at a time with preference in interpretation given to the more theory‐
defensible variable. Additionally, data reduction techniques such as principal components may
be used to reduce the dimensionality of the data. Importantly, the stage‐of‐development
classification described in Section 4.2 is expected to be an important predictor of substance
abuse outcomes. Moreover, the classification itself is related and based on a number of factors,
which essentially reduces the dimensionality of the data and could create collinear covariates if
included in a model with the explanatory variables used to create the classification. Therefore,
additional separate models will be fit with and without the classification to investigate the
impact of specific factors before and after adjusting for stage of coalition development.
Other forms of the models may need to be fit if information on the number of youth that each
percentage was based on are unavailable (i.e., sample size estimates for each outcome are not
provided by coalitions). In this case, two alternative models may be employed. First, the data
will be fit assuming that the percentages represent a continuous outcome. Second, an arc sine
transformation of the percentages will be conducted and used as the dependent variable in the
model. The arc sine transformation is a common technique used with percentages to stabilize
the variance and to ensure that the model residuals are reasonably normally distributed.
4.3.2 Comparison of DFC Communities to Non‐DFC Communities
The focus of this portion of the evaluation is to determine the impact of DFC coalitions on
lowering the prevalence of past 30‐day substance use in their communities when compared to
communities without DFC coalitions. Conceptually, this contrast can be reduced to a
comparison between two curves—one curve describing the outcome of interest among DFC
communities while a second describes the curve for the same outcome measure among non‐
DFCs communities for the same period of time.
To be as robust as possible the DFC evaluation will require information on substance abuse
outcomes and covariates of interest from non‐DFC communities as a basis of comparison.
However, the acquisition of this data will be extremely difficult for a number of reasons,
including:
- 24 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
The current DFC evaluation has limited resources to target, recruit, and retain
participation from these non‐DFC coalitions. Providing the substance abuse outcome
data will be a non‐trivial exercise for most coalitions—thus we will need to offer
these coalitions proper incentive for their participation.
Given that the evaluation focuses on long‐term trends in substance abuse outcome
measures, it will be important to include non‐DFC coalitions that will provide
information over the entire evaluation period (thus retention is a key component). A
plan which includes sampling non‐DFC coalitions as a basis of comparison will need
to accommodate reasonable estimates for the fraction of coalitions that will either
(1) not provide quality outcome data over the total evaluation period, or (2) join the
DFC program mid‐way through the evaluation (thereby minimizing their utility as a
basis of comparison).
Some non‐DFC communities will represent communities without a centralized
substance abuse coalition infrastructure. These communities are important to
include in the DFC evaluation (as part of the basis of comparison), but will be nearly
impossible to target and sample effectively.
As a result of these factors and other considerations, collecting information from non‐DFC
coalitions or in communities that lack a substance abuse coalition was determined to be
impractical for this evaluation. Instead, this evaluation will focus on using publicly available data
sources such as the results of national surveys, and more intensive statistical analysis
techniques to define substance abuse outcomes for comparison communities. More
specifically, outcome data for the DFC coalitions will be gathered via self reported data through
the planned semi‐annual progress reporting system (as implemented by the COMET).
Surrogates of outcome data for comparison communities (i.e., those that are not targeted by a
DFC coalition) will be constructed by extracting outcome estimates from the Youth Risk
Behavior Survey (YRBS) and other publicly available, and trusted, substance abuse data sources
(e.g. the National Survey on Drug Use and Health (NSDUH), Pride) using a mathematical
algorithm. Table 2 summarizes the substance abuse outcome information available from each
of these three national surveys. It should be noted, however, that geocoded information is
available only at the state or national level in these publicly available data. However, more
refined geocodes may become available during the evaluation.
Table 2. Summary of Substance Abuse Outcome Information Available from Three Large
Survey Efforts.
Outcome
30‐day use
Perception of Risk
Perception of Parental Disapproval
Age of Onset
Perceived Peer Approval
Perceived Norms and Beliefs
Perceived Availability
Availability of measures by grade
A
N
Y
Y
Y
Y
Y
Y
PRIDE
T
N
Y
Y
Y
Y
Y
Y
Yes
- 25 -
M
N
Y
Y
Y
Y
Y
Y
A
Y
N
N
Y
N
N
N
YRBS
T
Y
N
N
Y
N
N
N
Yes
M
Y
Y
N
Y
N
N
N
A
Y
Y
Y
Y
Y
Y
N
NHSDA
T
Y
Y
Y
Y
Y
Y
N
Yes
M
Y
Y
Y
Y
Y
Y
Y
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
Outcome
PRIDE
T M
Yes
8 to13+
14 to 18
4‐6 graders, 6‐
12 graders
Fee‐based
A
Availability of measures by gender
Respondents Age Range
Respondents Grade Range
Geographic Data Available
A
YRBS
T
M
Yes
September, 2008
A
NHSDA
T
Yes
M
12‐18
12+
9th‐12th
All
State
Conducted in All States
With this approach, we will combine data sources so that within each state a time‐series of
substance abuse outcomes for each DFC coalition over the evaluation period can be derived.
We will then construct a single time‐series of substance abuse outcomes for the areas of the
state that are not covered by the DFC program as a basis of comparison. Substance abuse
outcome information for non‐DFC communities will be constructed by subtracting substance
abuse outcomes reported in DFC communities from published state or national outcomes. A
high‐level example of this technique is illustrated in Figure 5.
30-Day UseAlcohol - Combined Grades for RI
•
RI – YRBS State Profile for 30-day Alcohol Use =
45%
•
1 Coalition, 30-day alcohol use = 10%
•
Suppose Community Targeted by DFC Coalition =
5% of population in RI, then
•
Non – DFC Comparison Community 30-day alcohol
use = 45% = 5% (0.1) + 95% * (X) Î X=46.8%
100
Percent
80
60
40
20
0
Coalition A
RI State
Profile
So, Non-DFC Communities in RI 30-day Use is 46.8%
Figure 5. Illustration of Calculating Non‐DFC Community Substance Abuse Outcomes
Using State Profiles.
To illustrate this concept in mathematical terms, we adopt the following statistical notation:
Within each State (i), let
Yijk = the substance abuse outcome measure for the jth DFC coalition, in year k
wijk = the estimated number of children represented in the service area of the jth DFC
program for year k (with respect to the outcome measure1)
1
Note that the estimated number of children will vary based on the specific substance abuse outcome measure that is
being investigated (i.e. these outcomes may be gender-, race-, or grade-specific)
- 26 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
Yi∙k = the substance abuse outcome measure for state (i), in year k – derived from the
analysis of existing data sources.
wi∙k = the estimated number of children in state (i) in year k (with respect to the
outcome measure1)
nik = the number of active DFC coalitions during year k in state (i)
We then derive estimates of the outcome measure and number of youth for areas within
state(i) that are not served by DFC coalitions as follows:
nik
⎡⎛
⎞⎤
⎜
⎢ ⎜ Yi⋅k ⋅ wi⋅k − ∑ Yijk ⋅ wijk ⎟⎟ ⎥
n
j =1
⎢⎝
⎠ ⎥ , and w = w − ik w
Yi 0 k = ⎢
∑
i0k
i ⋅k
ijk
⎥
nik
⎛
⎞
j =1
⎥
⎢
⎜ wi⋅k − ∑ wijk ⎟
⎜
⎟
⎥⎦
⎢⎣
j
=
1
⎝
⎠
In addition to the outcome variables and weights, we will also construct appropriate
explanatory variables (Xijk) to utilize in the formal statistical models. For example, we could
allow Xijk to be a categorical variable that represents the stage of typology (1‐5) for the jth DFC
coalition in state(i) during the kth year of the DFC evaluation. For the areas of the state that are
not participating in the DFC program (represented by the Yi0k outcome and wi0k weight), we
would create a sixth level of typology (level zero) that indicates an area that is not part of the
DFC program. We could also use this same 6th category for areas that join (or drop from) the
DFC program midway through the evaluation, or we could create additional levels of the
categorical variable to capture these particular areas differently (depending on the goals of the
particular evaluation model).
Due to the fact that our outcome variables represent specific spatial areas (i.e., communities),
we will also use the 2000 United States Census (and other spatial summary datasets) to create
explanatory variables to use in the evaluation. Thus, we can adjust the evaluation statistical
models for various community‐level summary demographic factors such as income, ethnicity,
education, population density, etc.
Note that there is an important difference between the two types of explanatory variables
discussed above. The stage of typology variable for DFC coalitions is considered a time‐varying
covariate for which we would expect the value to change over time within each DFC program
(to reflect the evolution and growth of each DFC coalition over the evaluation period). The
demographic factors summarized in the US Census represent time‐invariant covariates which
are generally assumed to stay stable over the evaluation time frame.
Once the outcome variables, covariates, and other factors have been created, extensive
statistical modeling with these variables will be accomplished to address the specific
hypotheses of interest to the evaluation. Several different models will be explored and assessed
as part of the evaluation, however, the following statistical model is an example of how the DFC
- 27 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
evaluation data will be combined to assess differences in trends over time in substance abuse
outcome measures as a function of stage of typology:
Yijk = β 0 Z ⋅ I ( X ijk = Z ) + β 1Z ⋅ I ( X ijk = Z ) ⋅ Year (k )
, where
+ γ 0ij + γ 1ij ⋅ Year (k ) + ε ijk
β0Z and β1Z represent separate fixed effect intercept and slope pairs for each stage of
typology (Z ranges from zero to five) describing a linear trend over time across the
nation for the substance abuse outcome measure Yijk;
I ( X ijk = Z ) is an indicator variable whose value is one if Xijk=Z, and zero otherwise
o Xijk indicates the stage of typology for the jth DFC coalition area within state (i) at
time k
o Xijk=0 when j=0 (for areas of the state not involved in the DFC program)
γ0ij and γ1ij represent random effect intercepts and slopes corresponding to each
geographic area, allowing each area to have its own linear trend in substance abuse
outcome measures. The actual γ0ij and γ1ij random effects represent deviations from the
overall national trend captured by the β0Z and β1Z fixed effect parameters for the jth area
in state(i).
εijk represents error not captured by the model (i.e. deviations from the area specific
linear trend in substance abuse outcomes).
In this model, it is assumed that δ0ij and δ1ij jointly follow a multivariate normal distribution with
⎡σ 112 σ 122 ⎤
mean zero and covariance matrix Σ = ⎢ 2
, and that εijk follows a normal distribution
2 ⎥
⎣σ 21 σ 22 ⎦
2
.
with mean zero and variance σ Error
This mixed models analysis of variance would be fit using the weights (wijk) described earlier so
that the results of the analysis can be generalized to the entire United States population of
adolescents. Note that the fixed effects parameter estimates βˆ0 Z and βˆ1Z that result from
fitting this model will allow us to compare the average trend across the nation in substance
abuse outcomes between coalitions in different stages of typology, as well as between each
stage of DFC typology and non‐DFC areas. In addition, we can easily collapse the 6‐level Xijk
stage‐of‐typology explanatory variable to a 2‐level variable that indicates whether the jth area
of state(i) is an active DFC coalition area in year(k) to allow for a global comparison of whether
trends in substance abuse outcomes are different between DFC and non‐DFC areas of the
country. This model can easily be expanded to allow for non‐linear trends over time in
substance abuse outcome measures by adding appropriate fixed effect and random effect
terms. This model can also accommodate the inclusion of other explanatory variables (including
both time‐varying factors and factors that stay stable over time) as long as these variables are
measured both for the DFC and non‐DFC communities.
(
- 28 -
)
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
There are a number of significant challenges with this approach that will be addressed in the
statistical analysis. One such challenge is that DFC coalitions may submit outcome data that
represent geographic areas that differ from their true sphere of influence. This is important
because the substance abuse outcome information for non‐DFC coalitions will essentially be
constructed through subtraction. Another related challenge will be situations where there are
multiple DFC coalitions targeting the same community. These and other anticipated challenges
to this analysis, along with our method for overcoming these issues are summarized in Table 3.
Table 3. Summary of Anticipated Analysis Issues and Methods to Overcome these Issues.
Description of Issue
Coalition reports outcome information that
covers a different geographical area than
that targeted by Coalition.
There are multiple DFC coalitions operating
within the same community, all reporting
the same substance abuse outcome data.
How this Issue Will Be
Identified
Included as a data element in
COMET.
Method for Addressing the Issue
Target community provided as
data elements in COMET.
There are multiple DFC coalitions operating
within the same community, all reporting
different substance abuse outcome data.
Comparison of outcome
information reported in
COMET.
- 29 -
Exclude these coalitions from
analysis.
Training/Technical Assistance by
PO to resolve.
Treat Coalitions as “super‐
coalition” and adjust covariates
appropriately (e.g., if either
coalition has a capacity, then the
“super‐coalition” does).
Include a covariate in the model to
indicate the number of DFC
coalitions having influence on the
outcome.
Identify portion of community
being focused on by each coalition
if possible, then consider this to be
the target community.
For overlapping portions of the
community, use the average of the
substance abuse outcomes, create
a “super‐coalition” for that portion
of the community.
Include a covariate in the model to
indicate the number of DFC
coalitions having influence on the
outcome.
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
Description of Issue
There are DFC and non‐DFC coalitions
targeting the same community.
How this Issue Will Be
Identified
Cross‐reference DFC
communities with CADCA, SIG,
and other known coalition
efforts.
September, 2008
Method for Addressing the Issue
Poor quality or missing data on substance
abuse outcomes reported by Coalitions.
Screening during reporting
period; comparison to State
profiles; comparison to other
Coalitions in State.
If there is a complete overlap, then
this would be considered a “DFC
community.” A variable indicating
the number of Non‐DFC coalitions
that affect the outcomes will be
included in the model.
If there is incomplete overlap, then
the portion of the community
being focused on by each coalition
will be identified if possible, then
consider this to be the target
community.
Substance abuse outcomes from
coalitions that are determined to
be unreliable will be excluded from
the analyses. Validation will be
performed using a subjective
assessment and a statistically‐
based outlier analysis.
Outcomes with missing
information such as associated
sample sizes will have sample sizes
imputed.
4.4 ASSESSMENT OF ADDITIONAL EVALUATION QUESTIONS AND HYPOTHESES
As discussed in Section 1.0, there are a number of specific evaluation questions in addition to
the three primary objectives of the evaluation. Three of these questions are focused on
assessing the relationship between potential explanatory variables and substance abuse
outcomes. However, the majority of these questions are focused on understanding and
evaluating the relationships between activities, strategies, and coalition capacity. Regardless, all
of these evaluation questions will be accessed through statistical modeling using data self‐
reported by coalitions.
For those three questions that are focused on the relationship between particular factors and
substance abuse outcomes, the primary approach will be to include these factors as
explanatory variables in the models described in Section 4.3. Unlike other potential explanatory
variables, however, these factors will be forced to remain in the models so that statistical
significance of the relationship between these factors and substance abuse outcomes can be
assessed. Table 4 summarizes the three evaluation questions that will be assessed with this
method along with proposed hypotheses that will be specifically assessed.
- 30 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
Table 4. Additional Evaluation Questions Regarding Relationships to Substance Abuse
Outcomes.
Evaluation Question
Study Hypotheses
Composition/Collaboration: What mix of
agencies and types of collaboration are most
associated with improvements in community
substance use as well as risk and protective
factor outcomes?
A. There are specific activities, and/or
collaborations that are associated with
substance abuse outcomes and
risk/protective factors.
Outcomes vs. Geography and Socio‐economic
status: What evidence, if any, illustrates an
association between differences in outcomes
and such factors as geographic location
(urban/rural/ suburban) or socio‐economic
status?
A. There are specific relationships between
geography and socioeconomic status and
substance abuse outcomes and
risk/protective factors.
Analysis
Method
Included as
Covariates in
Substance
Abuse Outcome
Models
Effectiveness of Strategies: What are the most A. There are specific relationships between
effective strategies? What mix of strategies led
use of specific strategies and substance
to positive community changes? Is there any
abuse outcomes and risk/protective
relationship to type, level, and coordination of
factors.
outside funding streams?
B. There are specific relationships between
type, level, and coordination of outside
funding and substance abuse outcomes
and risk/protective factors.
The other additional evaluation questions focused on coalition capacity will also be addressed
through statistical models, though a separate model will be used for each question. Generally,
these models will consist of the family of general linear models with repeated measures to
account for multiple observations from the same coalitions over time. The specific form of the
model will depend on the nature of the outcome variable but will include Poisson regression for
modeling outcomes that are counts, logistic and polytomous logistic regression models for
categorical outcomes, and linear regression models for continuous outcomes. All models will
employ GEE techniques to partition the variance components. To facilitate statistical analysis,
the equation questions have been translated into specific hypotheses of interest that can be
tested using a statistical model. Table 5 summarizes the specific modeling technique that is
anticipated for each evaluation question and related hypotheses.
The impact of ONDCP’s Mentoring program is hindered by the relatively small sample size
associated with this program (approximately thirty to forty coalitions participating in the
mentoring program). This small sample size may limit the ability to identify significant
relationships. However, we will attempt to quantitatively assess the impact of this program by
coding support activities of the mentoring coalitions to develop categories of activities (i.e., a
measure of dosage). Next, summary statistics and logistic regression models will be used to
examine the relationship between these categories or level of effort and capacity outcomes of
the mentee coalitions (i.e. preparedness to implement SPF: governing body, baseline measures,
strategic planning activities, collaboration of key sectors).
- 31 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
Table 5. Analysis Approach for Capacity Related Evaluation Questions.
Evaluation Question
Study Hypotheses
Increase in Evidence‐Based
Programs, Policies, and Strategies:
What evidence exists to demonstrate
an increase in evidence‐based
programs, policies, and strategies in
coalition communities?
A. DFC Grantees have increased their use of
evidence‐based programs.
B. DFC grantees have had an increase in their
impact on substance abuse policies.
C. DFC grantees have increased the use of
environmental strategies to reduce substance
abuse
A. DFC coalitions become (are) sustainable
Sustainability: What evidence exists
that demonstrates the sustainability
of DFC coalitions?
Increased National Capacity: To what
extent has the number of
communities with established
coalitions increased (a Healthy
People 2010 requirement)?
Impact of Technical Assistance on
Data Collection, Application, and
implementation of Environmental
Strategies: What evidence exists that
supports or negates an association
between the provision of technical
assistance and increased data
collection and application and/or use
of evidence‐based strategies by
coalitions? Does receiving technical
assistance increase the likelihood
that a new coalition will
subsequently obtain new DFC
funding? Do these relationships vary
with the source of the technical
assistance?
A. The DFC grant program has increased the
number of communities with established
coalitions.
B. DFC coalitions that have received funding
advance in development (i.e., become more
established)
A. Coalitions receipt of technical assistance is
positively correlated with stage‐of‐
development (i.e., more technical assistance
results in higher stages of development)
B. Some sources/types of technical assistance are
more effective than others
C. Technical assistance results in increased data
collection
D. Technical assistance results in increased use of
evidence‐based strategies
Analysis Method
Poisson Regression
and
Logistic Regression
Logistic Regression,
Linear Regression
Poisson Regression
Hypothesis B will be
examined through the
Stage‐of‐development
Analysis
Hypothesis A will be
examined through
Stage‐of‐development
analysis
Logistic Regression
4.5 ANALYSES TO SUPPORT GPRA REPORTING
ONDCP is required to submit a Government Performance and Results Act (GPRA) report to
Congress annually regarding the DFC grant program. The specific goals and objectives to be
addressed may vary somewhat from year‐to‐year. Table 6 summarizes ONDCP’s goals and
objectives for Fiscal Year 2005.
- 32 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
Table 6. ONDCP’s FY2008 GPRA Goals and Objectives.
GPRA Goals
Goal 1: Improve Coalition
Effectiveness
Goal 2: Strengthen technical
assistance to community coalitions
Primary Objectives
Enhance and strengthen infrastructure
Increase citizen participation in prevention efforts
Improve coalition capabilities
Increase intergovernmental and interagency collaboration in
coalitions
Ensure prevention efforts are more comprehensive and evidence‐
based and consistent with identified needs
Enhance prevention efforts
Strengthen coalitions in their prevention efforts to decrease risk
factors in the community
Strengthen coalitions in their prevention efforts to increase
protective factors
Strengthen coalitions in their prevention efforts to decrease
substance abuse indicators.
Implement and Assess Strategies of the National Community Anti‐Drug
Coalition Institute.
Implement and assess the efficacy of a mentoring coalition
demonstration program.
Statistical approach for establishing targets. Targets for annual GPRA performance were
established by calculating the upper 95% confidence interval for each baseline proportion and
for each successive year’s target value. That is, the upper 95% boundary value for each baseline
proportion became the second program year’s target value. The upper 95% boundary value for
the second year’s target, was selected as the third year’s target, and so on.
The 95% confidence interval was calculated using Logit transformed proportions (see Formula
1) with the standard error of the proportion calculated using Formula 2 and an assumed sample
size of 700 (the approximate number of coalitions in the first year of DFC‐funding). The actual
upper boundary of the confidence interval was calculated using Formula 3 and transformed
back into its associated proportion using Formula 4.
Calculating the logit transformed proportion
⎡ p ⎤
ES L = log e ⎢
Formula 1
⎥
⎣1 − p ⎦
Where loge = the natural log and p = the proportion of subjects in the category of interest (i.e.,
the baseline or current target).
Calculating the standard error
1
1
SE L =
+
Formula 2
np n(1 − p)
Where n = 700 and p = the proportion of subjects in the category of interest.
- 33 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
Calculating the upper 95% confidence interval
T = ES L + (1.96 * SE L )
Formula 3
Where T = the new target expressed as a logit, ESL is the logit transformed target, and SEL is the
standard error for the logit transformed target.
Transforming the logit target (T) back into its associated proportion
e ES L
p = ES L
Formula 4
e +1
Where e ES L is the base of the natural logarithm raised to the power of the logit transformed
proportion.
Because DFC outcomes are collected on alternating years, the target values calculated using
this method are ambitious in that they project statistically significant change in DFC coalition
performance between each anticipated wave of coalition reporting. Because the number of
funded DFC coalitions is expected to grow, the assumption of 700 coalitions contributing data is
conservative for estimating upper 95% confidence intervals.
Statistical approach for estimating GPRA actuals. Actual performance of the DFC coalitions on
the GPRA outcome measures is calculated by estimating the cumulative performance of
coalitions on each GPRA measure over baseline relative to the total number of coalitions for
which performance could be calculated. That is, only coalitions providing two or more years of
performance data for the same grade respondents are eligible to contribute to the estimate.
All calculations for GPRA performance measures follow the same basic logic: the number of
programs demonstrating success on a performance measure is divided by the total number of
DFC funded programs that provided a baseline estimate and some follow‐up estimate for each
measure. For the non‐past 30‐day use items (Age Of Onset, Parental Disapproval or Perception
of Risk) a coalition is counted as successful if there is any improvement from baseline (i.e., a
later measured outcome indicates improvement in performance over baseline). For past 30‐day
substance use measures the same basic logic is employed, except that it involves a calculation
in addition to the logical rule. If past 30‐day substance use decreases by 5% or more in two or
more grades, then a coalition is counted as successful.
As each coalition provides its own baseline, the 5% reduction necessary to be identified as
successful is calculated according to Formula 5, the value representing a 5% reduction in the
proportion of users over baseline for each substance and grade. A coalition is counted as
successful when two or more grades within the coalition provide subsequent past 30‐day use
proportions that are less than or equal to each grade and substance’s referent criterion
proportion.
- 34 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
Calculating each grade by substance 5% reduction criterion value
C A = p B * .95
Formula 5
Where CA is the criterion to be met or exceeded for success and pB is the baseline performance
for each grade and substance within a coalition.
The proportion of successful coalitions for each GPRA measure is calculated as the number of
successful coalitions divided by the total number of coalitions providing two or more waves of
data (see Formula 6).
Calculating the proportion of successful coalitions
S
pSC = C
Formula 6
TC
Where SC is the number of coalitions meeting or exceeding the performance criterion and TC is
the total number of coalitions providing two or more waves of data.
5.0 STATISTICAL ANALYSIS OF EXTERNAL DATA
Although this evaluation effort is in its initial phases, efforts to understand and collect
information on substance abuse in communities have been ongoing for decades. In some cases,
this information may be captured for the same communities that have been targeted by DFC
coalitions. This information may therefore provide a valuable source of additional outcome
information for understanding trends and progress of reducing substance abuse in DFC
communities. Generally, there are two types of outcome information that will be examined and
included as part of the statistical analysis if possible: (1) information on substance abuse
outcomes, and (2) information on distal outcomes. The following two sections discuss the
approach for assessing the feasibility of incorporating this data, and if feasible, our approach for
conducting statistical analysis.
5.1 MODELING SUBSTANCE ABUSE OUTCOME DATA COLLECTED THROUGH EXTERNAL
SOURCES
One of the drawbacks of the modeling approach described in Section 4.3 is that it can only
distinguish between DFC and non‐DFC communities for comparison of substance abuse
outcomes. Acquiring external substance abuse data at a more refined geographic level such as
counties, municipalities, etc. will enable a comparison of DFC coalitions to non‐DFC coalitions,
and to communities that do not have a coalition at all. However, because the collection of this
data is outside of the control of the evaluation, there are significant considerations that must
be addressed before attempting a meaningful statistical analysis. These considerations are
discussed in Section 5.1.1. The statistical methodology that would be employed if appropriate
data can be obtained is presented in Section 5.1.2.
- 35 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
5.1.1 Feasibility Assessment
Whenever external outcome data is used to evaluate a specific program there are a number of
questions that need to be addressed regarding access to the data and the validity of applying
the data to the current analysis. For this evaluation the primary questions regarding the use of
external substance abuse outcome information include (1) can the substance abuse outcome
data be acquired at a meaningful level of geography? (2) is the outcome data that is available
representative of DFC coalitions as a whole? and (3) will there be sufficient sample sizes and
power for statistical analysis? Addressing each of these questions forms the core of the
activities associated with the feasibility assessment. Failure to successfully address any one of
these three key questions would result in not moving forward with the statistical analysis.
Availability of Substance Abuse Outcomes from External Sources
As described in Table 1, there are at least four large surveys that collect information on
substance abuse outcomes of interest. These surveys include the National Household Survey on
Drug Use (NHSDU), both national and state‐specific components of the Youth Risk Behavior
Survey (YRBS), Monitoring the Future (MTF), and the National Parents Resources Information
for Drug Education (PRIDE) Survey. However, local geographical information such as county or
community‐level information is not typically included in the public release version of these files.
Obtaining access to the identity of the primary sampling units (PSUs) and segments from one or
more national surveys will enable the construction of substance abuse outcome measures for
specific geographic areas across the country (at the county, zip‐code, or census‐tract level,
depending on the design of each survey). If the survey sponsors (or contractors who perform
the surveys) are unwilling to provide the geographic identity of the PSUs and segments from
their surveys, we could alternatively request that they provide the aggregated substance abuse
outcome measures at the highest level of geographic specificity. This may be more feasible and
palatable to the survey sponsors, as there would be no possibility of identifying a single study
subject from this aggregated data. As part of the feasibility assessment, the evaluation team
will initiate contact with the various survey sponsors to discuss how the data will be used and
the rationale behind the request. This contact may also include a formal briefing to the survey
sponsors. ONDCP would also likely be required to initiate a formal interagency request for this
information.
A second thrust of the feasibility assessment regarding the availability of data will be to conduct
a literature and Internet review to determine the extent to which published substance abuse
outcome information is available over time.
Representativeness of the Communities where Data is Available to all DFC Communities
Assuming that data at a community level is available, the next consideration will be to assess
whether this data is representative of the DFC communities. Again, the evaluation team will not
have the ability to control the geographic locations that the data represent. National surveys
are typically conducted using a probability sampling scheme, with inclusion probabilities
- 36 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
weighted by population density or related factor. They may or may not include stratification to
ensure that the sample is distributed throughout all types of geographic entities (e.g., urban,
suburban, and rural communities). For example, because the focus of many of these surveys is
on calculating a national or state‐level estimate, it would be logical to oversample the large
population centers because this is where the largest segments of the population reside.
Therefore, it will be important to map the communities that are included in each national
survey or other data source to ensure that the communities included in this sample are
“similar” to DFC communities in terms of various demographic and socioeconomic
characteristics so that the results of analyses conducted with this data can be expanded to the
entire group of communities targeted by DFC coalitions.
Another consideration will be the “appropriateness” of the comparison communities for the
evaluation. That is, are the non‐DFC communities where data is available suitable to serve as
comparisons to DFC communities? For this evaluation the gender, race/ethnicity, age,
population density, and economic status for each community that data is available will be
collected and used to determine whether a community has a similar demographic and
socioeconomic composition as a DFC community. As an initial step, we will utilize an algorithm
to compute a “similarity” measure between DFC communities and non‐DFC communities.
Communities with roughly equivalent similarity scores will be considered to be candidate
communities for further investigation.
Evaluation of Sample Sizes and Statistical Power
Even if community‐level data are available from these sources, it is extremely unlikely that
representative outcome data will be available for every DFC community, especially for the same
communities over a five‐year period. Data will be much more likely to be available for some
DFC communities for some years and unavailable in others. A similar phenomenon could be
expected for non‐DFC communities. Therefore, if community‐level data are available, an
assessment will be conducted to determine if sufficient data is available for DFC and
comparison communities over time for the statistical analysis to be meaningful (i.e., have
sufficient power to detect differences). Simulation techniques will be used to establish
minimum sample size criteria needed in each year and across all years of the evaluation to
ensure a specific targeted power level.
5.1.2 Statistical Methods
There is a very important distinction between the modeling approach discussed in Section 4.3
and the modeling approach that will be conducted using external data sources. In Section 4.3,
the substance abuse outcome data will be obtained through actual repeated measures of the
same communities (via the self‐reported mechanism). With the external data, the substance
abuse outcome data will be constructed by integrating the results from a series of ongoing
cross‐sectional surveys. In some cases, a survey may repeatedly target the same geographic
area, but in many instances the survey will target different communities each year.
Nevertheless, the statistical methodology for external substance abuse outcome data is similar
- 37 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
to that employed for the self‐reported outcome data, although a simpler model would be
utilized:
Yijk = β 0 Z ⋅ I ( X ijk = Z ) + β 1Z ⋅ I ( X ijk = Z ) ⋅ Year (k ) + γ 0ij + ε ijk
This model is essentially the same as described in Section 4.3, with the following two important
distinctions:
In the models with the self‐reported outcome data, Z ranged from 0 to 5 providing
an index for the various different stages of typology captured by the categorical
explanatory Xijk variable. For this analysis, we anticipate expanding this range to
allow Xijk to capture additional information (e.g. different levels for each stage‐of‐
typology among DFC programs, different levels capturing similar pseudo‐typology
information for non‐DFC areas if available, and a single level corresponding to areas
with no known substance abuse coalition).
In the models for the self‐reported outcome data, both intercept and slope random
effects (δ0ij and δ1ij) for each coalition area were included because we anticipated
having at least two repeated measure substance abuse outcome measures for each
participating community, thereby allowing us to determine a linear trend for each
coalition area at a minimum. Because many geographic areas may only be surveyed
on one occasion over the evaluation period (among the limited number of targeted
surveys), the random effect slope term cannot be included in this model at the
current time. Removal of this term means that specific areas will be allowed to
deviate from the fixed‐effect national trends by an additive factor that stays stable
over time. The fixed effects parameter estimates (and associated standard errors)
should be robust to this subtle reduction in the model because the potential
correlation among repeated measures on a community will still be accounted for by
including the random intercept (δ0ij).
Similar to the model used for the self‐reported data, the above mixed models analysis of
variance will be fit using weights (wijk) so that the results of the analysis can be generalized to
the entire U.S. population. The fixed effects parameter estimates that result from fitting this
model will allow us to compare the average trend across the nation in substance abuse
outcomes between DFC coalitions in different stages of typology, between DFC coalitions and
non‐DFC coalitions in similar stages (based on any constructed pseudo‐typology explanatory
variables), and between areas served by DFC (or non‐DFC) coalitions and areas that are not
served by any known substance abuse coalition. This model could also be expanded (in the
fixed effects only) to explore non‐linear (quadratic or cubic) trends over time in substance
abuse outcome measures and to adjust for demographic factors that are available through the
United States Census or other spatial data sources.
- 38 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
5.2 EVALUATING DISTAL OUTCOMES
According to the hypothesized typology, some of the DFC coalitions, particularly the more
advanced coalitions, have the potential to be impacting long‐term or distal outcomes. These
distal outcomes include reductions in medical treatment for substance abuse related medical
conditions, reduced substance abuse related traffic incidents and accidents, a reduction in the
number of drug‐related crimes, reductions in mental illness, school failures, teen pregnancies,
etc. For this evaluation, if ONDCP so desires, we will focus on assessing the relationships
between DFC program participants and two distal outcomes, (1) hospital discharges for
treatment related to substance abuse, and (2) traffic‐related fatalities as a result of substance
use/abuse. Given the relatively low funding level of DFC coalitions, the length of time that the
DFC program has existed, the variable nature of distal outcomes, and the number of coalitions
that could be expected to have a measurable impact on these distal outcomes, we do not
anticipate that an analysis of distal outcomes will be a meaningful and significant portion of the
evaluation. However, we will conduct a feasibility study to determine the appropriateness of
the analysis and to support a decision of whether or not to include these outcomes as part of
the evaluation analysis.
The feasibility assessment related to distal outcomes will need to address many of the same
issues that were described for external substance abuse outcomes (Section 5.1.1). However,
there are two primary issues that need to be addressed in this feasibility analysis. First, what
sources of information will be used, and can this information be readily linked to a particular
DFC coalition’s targeted community? Second, are there a significant number of DFC coalitions
that are sufficiently advanced so that it would be reasonable to expect that these coalitions
may have had an impact on the distal outcomes?
Potential Data Sources
Hospital discharge information is available from a variety of different sources, including the
National Hospital Discharge Survey (NHDS), which is a well‐accepted source of reports for the
number of national hospital discharges for specific reasons. While the NHDS is a true probability
sample, its relatively small size implies that it may not provide useful information at the
community level for this evaluation. Another alternative is the Nationwide Inpatient Sample
(NIS), which contains the ICD‐9‐CM N‐codes corresponding to the diagnosis. The NIS consists of
all patients discharged in a single year from a stratified sample of about 1000 community
hospitals. The most recent NIS samples have been selected from state‐based hospital discharge
databases from about one‐half of the states, including nearly all the most populous ones. More
states are scheduled to join the HCUP in the next few years. The NIS will be the primary data
source investigated for this evaluation. All of the salient issues pertaining to matching this data
to DFC and non‐DFC comparison communities previously discussed will also need to be
addressed for this data.
Information on fatal accidents at a community level is available in the Fatal Accidents Reporting
System (FARS) created by the National Highway Traffic Safety Administration (NHTSA) in the
1970s. This database contains information for all crashes that involve a motor vehicle traveling
- 39 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
on a traffic way customarily open to the public and that results in the death of a person (either
an occupant of a vehicle or a non‐motorist). This information includes over 100 different data
elements related to the fatality and often includes information on the precise location of the
crash. However, the location information is not always available for geographic areas below the
county level.
Evaluation of Sample Sizes and Statistical Power
Under the hypothesized typology, the impacts to distal outcomes are not expected for
beginning, functioning, and even maturing coalitions. Therefore, only a subset of DFC coalitions
is expected to be sufficiently advanced enough to have influence on these distal outcomes.
Minimal sample size calculations together with a statistical power analysis will be conducted
using simulation techniques to determine if there are enough DFC coalitions that are expected
to have an impact on these distal outcomes, and the degree to which a change in the distal
outcomes can be reliably assessed. If sufficient numbers of coalitions cannot be identified to
facilitate a statistical analysis that can identify a meaningful change in the outcome with a high
degree of statistical power, then this analysis would not be conducted.
6.0 SUMMARY
As discussed in the previous sections of this document, there are many different activities and
assessments that will be conducted to accomplish the goals and objectives of this evaluation.
Many of these tasks will be completed during the first year of the evaluation, including the
formulation of the typology, data collection system, and the evaluation design. The core
components of the statistical analyses will be initiated following the collection of the first wave
of data from coalitions and will continue throughout the remainder of the evaluation.
Ultimately, the analyses described in Section 4.0 and Section 5.0 of this document will provide a
comprehensive examination of the DFC program and will provide answers to all of the
evaluation questions and hypotheses. Figure 6 summarizes how each of these analyses will be
combined to address the three key evaluation objectives.
- 40 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
Figure 6. Cross‐Link Between Evaluation Analysis and Objectives
- 41 -
September, 2008
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
7.0 REFERENCES
Birckmayer, J.D., Holder, H.D., Yacoubian, G.S., Friend, K.B. (2004). A general causal model to
guide alcohol, tobacco and illicit drug prevention: Assessing the research evidence. Journal of
Drug Education, 34(2), 121‐153.
Chavis, D.M., D.M., Florin, P. & Felix, M.R.J. (1992). Nurturing grassroots initiatives for
community development: The role of enabling systems. In T. Mizrahi & J. Morrison (Eds.),
Community Organization and Social Administration: Advances, Trends and Emerging Principles.
Binghamton, NY: Haworth Press.
Crisp, B.R., Swerissen, H., & Duckett, S.J. (2000). Four approaches to capacity building in health:
consequences for measurement and accountability. Health Promotion International. 15, (2), 99‐
107.
Florin, P., Celebucki, C., Stevenson, J., Mena, J., Salago, D., White, A., Harvey, B. & Dugall, M.
Cultivating Systemic Capacity: The Rhode Island Tobacco Control Enhancement Project.
American Journal of Community Psychology, in press.
Florin, P., Mitchell, R., & Stevenson, J. (1993). Identifying technical assistance needs in
community coalitions: A developmental approach. Health Education Research: Theory and
Practice, 8, 417‐432.
Florin, P., Mitchell, R., & Stevenson, J., Klein, I. (2000). Predicting intermediate outcomes for
prevention coalitions: A developmental perspective. Evaluation and Program Plannning. 23:
341‐346.
Kegler, M. C., Steckler, A., McLeroy, K., & Malek, S. H. (1998). Factors that contribute to
effective community health promotion coalitions: A study of 10 Project ASSIST coalitions in
North Carolina. Health Education and Behavior, 25(3), 338‐353.
Klitzner, M., Stewart, K., & Fisher, D.A. (1993). Reducing underage drinking and its
consequences. Alcohol Health and Research World, 17 (1), 12‐18.
Lipsey, M.W. & Cordray, D. S. (2002). Evaluation methods for social intervention. Annual review
of psychology, 51, 345‐377.
Pentz, M. A. (2000). Institutionalizing community‐based prevention through policy change.
Journal of Community Psychology, 28(3), 257‐270.
Rohrbach, L.A., Graham, J. W. and Hansen, W. B. (1993). Diffusion of a
school‐based substance abuse prevention program: predictors of program
implementation. Preventive Medicine, 22, 237‐260.
- 42 -
UPDATED EVALUATION DESIGN DOCUMENT
DRUG-FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
September, 2008
Roussos, S. T., & Fawcett, S. B. (2000). A review of collaborative partnerships as a strategy for
improving community health. Annual Review of Public Health, 21, 369‐402.
Wandersman, A., & Florin, P. (2003). Community Interventions and Effective Prevention.
American Psychologist, 58 (6/7), 441‐448.
Wandersman, A., Goodman, R., & Butterfoss, F. (1997). Understanding coalitions and how they
operate. In M. Minkler (Ed.). Community organizing and community building for health. New
Brunswick, NJ: Rutgers U. Press.
Wolff, T. J. (Ed.). (2001). Special Section: Contemporary Issues Series I Community Coalition
Building – Contemporary Practice and Research. American Journal of Community Psychology,
29 (2).
- 43 -
| File Type | application/pdf |
| File Title | Microsoft Word - Revised Evaluation Design Document 9 2008.doc |
| Author | treecem |
| File Modified | 2008-09-22 |
| File Created | 2008-09-22 |