Download:
pdf |
pdfOMB CLEARANCE PACKAGE
FOR
DRUG FREE COMMUNITIES SUPPORT
PROGRAM NATIONAL EVALUATION
Supported by:
Executive Office of the President
Office of Administration
Office of National Drug Control Policy
750 17th Street, NW
Washington, DC 20006‐4607
Government Project Officer:
Kenneth Shapiro
[email protected]
(202) 395‐4681 phone
(202) 395‐6711 fax
September 2008
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
Table of Contents
A. JUSTIFICATION ....................................................................................................................................... 1
A.1. Circumstances Making the Collection of Information Necessary....................................... 1
A.2. Purpose and Use of the Information .................................................................................. 3
A.3. Use of Information Technology and Burden Reduction ..................................................... 4
A.4. Efforts to Identify Duplication and Use of Similar Information.......................................... 7
A.5. Impact on Small Businesses or Other Small Entities........................................................... 8
A.6. Consequences of Collecting the Information Less Frequently ........................................... 8
A.7. Special Circumstances Relating to Guidelines of 5 CFR 1320.5.......................................... 8
A.8. Federal Register Notice and Consultation Outside the Agency ......................................... 9
A.9. Explanation of Any Payment or Gift to Respondents ....................................................... 10
A.10. Assurance of Confidentiality Provided to Respondents ................................................... 10
A.11. Justification for Sensitive Questions................................................................................. 10
A.12. Estimates of Hour Burden Including Annualized Hourly Costs......................................... 10
A.13. Estimate of Other Total Annual Cost Burden to Respondents or Recordkeepers ........... 12
A.14. Annualized Cost to the Federal Government ................................................................... 12
A.15. Explanation for Program Changes or Adjustments .......................................................... 12
A.16. Time Schedule, Analysis Plans, and Publication ............................................................... 13
A.17. Reason(s) Display of OMB Expiration Date is Inappropriate ............................................ 32
A.18. Exceptions to Certification for Paperwork Reduction Act Submissions ........................... 33
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS ............................................ 33
B.1. Respondent Universe and Sampling................................................................................. 33
B.2. Procedures for the Collection of Information .................................................................. 33
i
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
B.3.
Methods to Maximize Response Rates and Deal with Nonresponse............................... 36
B.4.
Test of Procedures or Methods to be Undertaken........................................................... 37
B.5.
Individuals Consulted on Statistical Aspects and Individuals Collecting and/or
Analyzing Data .................................................................................................................. 39
LIST OF EXHIBITS
Exhibit 1.
DFC Support Program National Evaluation – Evaluation Framework.................................. 3
Exhibit 2.
Estimates of Hour Burden .................................................................................................. 11
Exhibit 3.
Annualized Cost to Respondents........................................................................................ 12
Exhibit 4.
Project Time Schedule........................................................................................................ 13
Exhibit 5.
Cross‐link Between Evaluation Analysis Objectives ........................................................... 15
Exhibit 6.
Stages of Development for Prevention Coalitions ............................................................. 17
Exhibit 7.
Questions Included on the CCT Measuring Stage‐of‐Development Dimensions (Four
Scale Items and 6 Sub‐Items within Each Scale) ............................................................... 18
Exhibit 8.
Distribution of the DFC Coalitions across Survey Waves and by the 4 Dimensions .......... 20
Exhibit 9.
Percentage of DFC Coalitions responding to various CCT items by Proposed Stage‐
of‐ Development Typology (Wave 1 CCT) .......................................................................... 21
Exhibit 10. Percentage of DFC Coalitions responding to various CCT items by Proposed Stage‐
of‐Development Typology (Wave 1 CCT) ........................................................................... 21
Exhibit 11. Percentage of DFC Coalitions responding to various CCT items by Proposed Stage‐of‐
Development Typology (Wave 1 CCT)................................................................................ 22
Exhibit 12. Distribution of the DFC Coalitions by Proposed Stage‐of‐Development Typology by
CCT Waves* ........................................................................................................................ 22
Exhibit 13. Distribution of DFC Coalition Measures by Typology and by Year of Outcome Data
Collection—9th Graders ...................................................................................................... 23
Exhibit 14. Distribution of DFC Coalition Measures by Typology and by Year of Outcome Data
Collection—10th Graders .................................................................................................... 24
Exhibit 15. Distribution of DFC Coalition Measures by Typology and by Year of Outcome Data
Collection—11th Graders .................................................................................................... 24
Exhibit 16. Distribution of DFC Coalition Measures by Typology and by Year of Outcome Data
Collection—12th Graders .................................................................................................... 25
Exhibit 17. Summary of Substance Abuse Outcome Information Available from Three Large,
Ongoing Surveys ................................................................................................................. 27
Exhibit 18. Illustration of Calculating Non‐DFC Community Substance Abuse Outcomes Using
State Profiles....................................................................................................................... 28
Exhibit 19. Additional Evaluation Questions Regarding Relationships to Substance Abuse
Outcomes ........................................................................................................................... 29
Exhibit 20. Analysis Approach for Capacity‐Related Evaluation Questions ......................................... 29
Exhibit 21. ONDCP’s FY2008 GPRA Goals and Objectives.................................................................... 32
ii
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
LIST OF ATTACHMENTS:
Attachment 1. Drug Free Communities Authorizing and Re‐Authorizing Legislation & STOP Act
Legislation
Attachment 2. Evaluation Design Document
Attachment 3. Data Collection Instruments
(a) Coalition On‐line Management and Evaluation Tool (COMET) (screen shots)
(b) Coalition Classification Tool (CCT) (screen shots)
Attachment 4. Federal Register Notice
Attachment 5. List of Expert Review Group Members
Attachment 6. Waiver of Human Subjects Committee
Attachment 7. Feasibility Study
Attachment 8. Paperwork Reduction Act Submission
iii
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
A. Justification
A.1. Circumstances Making the Collection of Information Necessary
The Drug Free Communities Program (DFC) was created by the Drug Free Communities Act, (June)
1997 (Public Law 105‐20), reauthorized through the Drug Free Communities Reauthorization Act of
2001 (Public Law 107‐82) and reauthorized again through the Office of National Drug Control Policy
Reauthorization Act of 2006 (Public Law 109‐469). The latest reauthorization (see Attachment 1)
extended the program for an additional five years until 2012. The DFC‐authorizing statute (21 USC
§1521 – 1535) provides that community anti‐drug coalitions can receive federal grant funds and that
the amount of each DFC grant award shall not exceed $125,000 annually. This package is also intended
to cover the Substance Abuse and Mental Health Services Administration’s (SAMHSA) Sober Truth on
Preventing Underage Drinking (STOP Act) Program, which will fund current and past DFC grantees. The
STOP Act Program will be evaluated based on the same data already being collected for the Office of
National Drug Control Policy (ONDCP) DFC Program. STOP Act grants are authorized under the Public
Health Service (PHS) Act (42 U.S.C. 290bb‐25b), Section 519B.
As the lead agency for setting US drug control policy and strategy, the ONDCP funds the DFC to build
community capacity to prevent substance abuse among our nation’s youth. The DFC has two primary
goals. The first is to reduce substance abuse among youth by addressing local risk and protective
factors to minimize the likelihood of subsequent substance abuse in the community. The second is to
support community anti‐drug coalitions by establishing, strengthening, and fostering collaboration
among public and private nonprofit agencies, as well as federal, state, local, and tribal governments to
prevent and reduce substance abuse. An important objective of the DFC Program is to assist
community coalitions in becoming self‐sufficient.
The purpose of the STOP Act program is to prevent and reduce alcohol use among youth in
communities throughout the United States. The program was created to strengthen collaboration
among communities, the federal government, and state, local, and tribal governments; to enhance
intergovernmental cooperation and coordination on the issue of alcohol use among youth; to serve as
a catalyst for increased citizen participation and greater collaboration among all sectors and
organizations of a community that first demonstrates a long‐term commitment to reducing alcohol use
among youth; and to disseminate to communities timely information regarding state‐of‐the‐art
practices and initiatives that have proved to be effective in preventing and reducing alcohol use among
youth.
The National Evaluation of the DFC Program makes use of a single data collection instrument to gather
information. The Monitoring and Tracking Questionnaire (online tool) serves as a semi‐annual report to
support both the evaluation of the DFC Program and the STOP Act Program.
The DFC Support Program funded five cohorts (FY1998 – FY2002) of community anti‐drug coalitions in
its first five‐year grant cycle; funded an additional five cohorts (FY2003 – FY2007) in its second five‐year
grant cycle; and recently began its third five‐year cycle, covering the period from FY2008 through
FY2012. There are currently 769 community anti‐drug coalitions receiving DFC grants. The DFC
program anticipates awarding approximately 90 additional grants each year through FY2012.
‐ 1 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
In the reauthorization legislation (21 USC §1702), Congress specifically authorized the Administrator of
ONDCP to “(a)(3) assess and certify the adequacy of National Drug Control Programs; and (a)(4)
evaluate the effectiveness of the national drug control policy and the National Drug Control Program.”
A National Evaluation of the DFC Support Program commenced in September 2004. The evaluation
aims to assess the program’s implementation and effectiveness in reducing substance abuse among
youth and supporting community anti‐drug coalitions.
The cornerstone of the framework for the evaluation is a scheme for classifying community coalitions
along a continuum of organizational development and maturation. The four stages are: (i) establishing,
(ii) functioning, (iii) maturing, and (iv) sustaining coalitions. The typology recognizes that DFC coalitions
at different stages of development are characterized by different processes, capacities, and outcomes.
Successful movement through the stages of development is determined by the coalition’s capacity to
perform requisite functions for each stage. These functions are defined in detail in Attachment 2,
Evaluation Design Document. Rather than focus exclusively on changes in substance abuse outcomes,
the framework measures DFC coalitions against goals appropriate to their stage‐of‐development. For
example, only in later stages of development, are goals including changes in community‐level
substance abuse outcomes most appropriate, whereas goals in earlier stages focus on risk and
protective factors. Furthermore, the framework (Exhibit 1) specifically recognizes that DFC coalitions
are conditioned by a community context – specifically, those social, cultural, and environmental factors
affecting influence within communities. Environmental factors are of great importance to the DFC
evaluation as these factors include, among other things, technical assistance, training, and mentoring
of DFC coalitions as provided by ONDCP and its partners – SAMHSA, the National Community Anti‐Drug
Coalition Institute (the Coalition Institute), and the National Institute on Drug Abuse (NIDA).
The development and appropriateness of the typology are described in Attachment 2, Evaluation
Design Document.
‐ 2 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
Exhibit 1. DFC Support Program National Evaluation – Evaluation Framework
At the time of the original submission of this OMB package in June 2005, ONDCP grantee coalitions
were providing routine, quarterly progress reports that were used to monitor and track progress on
their grants. While providing useful and necessary information, these progress reports were not
sufficient to support a National Evaluation effort. Therefore, two additional requests for information
were proposed (and have since been implemented) to be provided by ONDCP grantees. First, data
obtained via a web‐based questionnaire completed by the director of the coalition are used to provide
information necessary to classify coalitions into a particular stage‐of‐development. This tool is referred
to as the Coalition Classification Tool or CCT. Second, supplemental progress reporting information to
support the evaluation is being requested along with other required grantee progress report
information on a semi‐annual basis.
To reduce the reporting burden on coalitions, to enhance the quality of the data, and to facilitate the
monitoring and tracking of grantee progress, a new web‐based system was developed (and continues
to undergo enhancements) that integrates all requests for information from grantees into a single
system, the Coalition On‐line Management and Evaluation Tool (COMET). This system also provides a
mechanism for providing training and technical assistance to ONDCP grantees.
A.2. Purpose and Use of the Information
The DFC Support Program – an integral component of the National Drug Control Strategy and a
requirement of Healthy People 2010 – supports the President’s pledge to reduce America’s drug use by
25 percent in five years. The collection of this information is necessary to conduct a rigorous evaluation
of the overall progress and effectiveness of the DFC Support Program in meeting its primary goals.
Although the Federal Code language authorizing the DFC program does not specifically mandate an
evaluation of its effectiveness, the specificity of the code language authorizing an evaluation leaves
‐ 3 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
little doubt about Congressional intent. This information is also necessary to inform and support future
improvements or initiatives to strengthen the program.
The overall goal of the National Evaluation is to assess the DFC program’s implementation and
effectiveness. Three primary objectives of the evaluation are to (1) assess whether the DFC program
has reduced substance abuse and/or related risk factors, attitudes, and beliefs among persons at risk
for substance abuse (referred to as “substance abuse outcomes”) at the community, state, and
national levels; (2) identify specific factors that are associated with increased efficacy of substance
abuse prevention efforts; and (3) assess whether the DFC program has increased the capacity and
effectiveness of substance abuse coalitions. Within these broad objectives, the evaluation addresses a
series of specific questions and hypotheses which are discussed in Section A.16, Time Schedule,
Analysis Plans, and Publication, and in Attachment 2, Evaluation Design Document.
The evaluation examines both direct and indirect linkages between measures of DFC Program
effectiveness and changes in drug abuse outcomes in coalition communities. First, the strategies,
initiatives, and activities of DFC coalitions are examined to determine whether they have a direct
impact on substance abuse outcome measures of interest, such as the proportion of youth who report
using tobacco in the last 30 days. However, the evaluation focuses this outcome analysis primarily on
mature or sustaining coalitions where impacts to substance abuse outcomes would be expected.
Second, the indirect impact of the DFC program on enhancing the capacity of grantee coalitions to
influence change in the community is assessed by evaluating the degree to which coalitions
participating in the DFC Program mature (defined operationally as progressing to “maturing” and
“sustaining” coalitions ). If coalitions that participate in the DFC program advance to the status of
mature coalitions, and if mature coalitions favorably alter substance abuse outcomes, this finding
would support the hypothesis that the DFC program is effective in reducing substance abuse
outcomes:
• directly among mature and sustaining coalitions and
•
indirectly among less mature coalitions, by fostering their advancement to the status of mature
coalitions.
A.3. Use of Information Technology and Burden Reduction
A web‐based information system referred to as the Coalition On‐line Management and Evaluation Tool
(COMET) is being used to collect all data for this project. This system, described further in Section B.2,
is software‐independent, requiring only that users have a web browser and access to the Internet.
Users are able to enter data freely, closing and opening the system as they wish, with data being saved
and maintained by the system. For example, a user can begin entering data on a computer at work,
close the application, and return to the application on their home computer.
COMET is an integrated grant management and evaluation information system consisting of three
primary components. Information for the evaluation is being requested from all DFC coalitions through
the Coalition Classification Tool or CCT (see Attachment 3b) and the on‐line Semi‐Annual Progress
Report (see Attachment 3a). Information collected for the purpose of managing DFC grants is also used
in the evaluation.
‐ 4 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
Compared with the paper reports previously submitted by grantees, use of COMET as the data
collection mechanism for this evaluation has significantly reduced the reporting burden on
respondents. Advantages of using COMET include:
1. Adaptation from existing software already being used by many coalitions. The platform from
which COMET was developed was the Performance Management and Measurement System
(PMMS) – a commercially available application that has been extensively tested and
streamlined as a stand‐alone for management reporting for DFC coalitions in Florida, Maine,
Nashville, Pennsylvania, Rhode Island, South Carolina, South Dakota, Virginia, and Washington.
The commercial package was too expensive for all DFC coalitions to purchase directly.
Nevertheless, it does provide a tested and streamlined foundation for implementation with a
national program and was therefore chosen as the model platform for development of the
COMET system.
2. Integrated data collection. Once respondents input information into COMET, information can
then be propagated to appropriate fields throughout the system.
SAMHSA’s Prevention Platform (a voluntary web‐based tool for coalitions –
https://www.preventionplatform.SAMHSA.gov ) has undergone significant re‐development
since the original OMB package was submitted in 2005. With re‐development complete,
COMET and the Prevention Platform have been integrated, such that grantees can easily access
the COMET system via the Prevention Platform. This eliminates the burden of having to keep
track of a separate entry portal in order to gain entry to COMET.
Integrating the data collection activities required by the evaluation and those needed for grant
management reduces the overall burden on grantees by providing a one‐stop‐shopping source
of information. This integrated system represents a forum where government project officers
and grantees can view identical information in real time and proactively answer questions of
and about grantees.
The majority of coalitions have Internet access and are therefore able to access and utilize
COMET. However, for coalitions without Internet access, the SAMHSA project officer is able to
generate a blank “report” version to capture the progress reporting and evaluation information
in hard copy and then enter the information into the system on behalf of the coalition.
3. Reducing reporting burden for grant management. Currently, DFC grantees are required to
submit semi‐annual reports to ONDCP. Originally, grantees were required to submit a paper
report to their Project Officers, which was time consuming and required on a quarterly basis.
COMET captures and retains semi‐annual report information automatically. Coalitions are able
to enter or edit information and report accomplishments to ONDCP throughout the year, thus
spreading the reporting burden over a longer period. Additionally, much of this information is
retained in the system and propagated from reporting period to reporting period, thus
eliminating the need to re‐enter information that has not changed during a given reporting
period. Finally, because the information can be captured throughout the year (i.e., proactively)
by coalitions and immediately shared with Project Officers as needed, the official reporting
‐ 5 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
requirements for grant monitoring and tracking were changed to a semi‐annual rather than a
quarterly schedule.
4. Enhancing a coalition’s ability to succeed. COMET is intended to be a training tool as well as a
data collection mechanism. For example, data collection is organized around the Strategic
Prevention Framework (SPF) steps of Assessment, Capacity, Planning, Implementation, and
Evaluation. Coalitions are naturally led through the SPF as they enter data. The SPF was
initiated by SAMHSA in April 2004, as a science‐based mechanism for increasing the
effectiveness of substance abuse prevention in the United States. The five‐step SPF process was
designed to increase the effectiveness of substance abuse prevention on both the state and
local level through collaborative interagency planning, youth development, reduction in risk‐
taking behaviors, building upon community assets, and preventing problem behaviors. This
process is grounded in six underlying principles that are, in turn, based upon scientific research:
1) Prevention is an ordered set of steps along a continuum to promote individual, family,
and community health, prevent mental and behavioral disorders, support resilience and
recovery, and prevent relapse.
2) The common components of effective prevention for the individual, family, or
community within a public health model are the same regardless of the focus of the
prevention.
3) Common risk and protective factors exist for many mental health and substance use
problems.
4) Resilience is built by developing assets in individuals, families, and communities through
evidenced‐based health promotion and prevention strategies.
5) Systems of prevention services work better than service silos.
6) Baseline data, common assessment tools, and outcomes shared across service systems
can promote accountability and effective prevention efforts.
Through requiring that coalitions report grant progress in terms of the five‐step SPF process,
ONDCP reinforces SAMHSA’s message to coalitions on the processes and capabilities needed to
succeed as a coalition and be effective in substance abuse prevention.
Also, COMET will provide coalitions with an effective tool for promoting the work of the
coalition through the ability to generate real‐time progress reports from the information that is
contained in the system for their coalition. This immediate reporting and feedback mechanism
will provide coalitions with the ability to quickly summarize their progress and report
accomplishments to stakeholders, community leaders, and coalition members.
‐ 6 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
A.4. Efforts to Identify Duplication and Use of Similar Information
The evaluation team conducted a feasibility assessment (see Attachment 7) to determine if consistent
and reliable community and school‐level data on the four core measures could be collected at the state
and/or national level. Several data sources exist, but none of these sources meets all the specific needs
of the proposed evaluation nor do they encompass all of the DFC coalitions. The following sources
include some of the required information:
• SAMHSA’s Center for Substance Abuse Prevention (CSAP) supports the Prevention Platform, a
web‐based online resource that collects some of the same data elements. The Prevention
Platform is being revised so that data elements that are also required for COMET will not have
to be entered a second time. Commitments previously made concerning the confidentiality of
organizational data voluntarily maintained in the Prevention Platform preclude the use of these
data in connection with this evaluation. Currently, SAMHSA does not attempt to register or
track users of the Prevention Platform. That is, there is no way to link information to a specific
coalition. In the future, DFC grantees will be given the option to import their data from the
Prevention Platform into COMET.
• State programs. Several states and individual coalitions employ a similar reporting system,
including the commercially available system upon which COMET is based. However, the
majority of the 769 coalitions are not using these systems, and these systems do not collect the
data necessary to appropriately classify a coalition into a particular stage‐of‐development.
• National surveys. Several large‐scale surveys collect outcome information that is relevant for
the evaluation. These surveys are:
o National Survey on Drug Use and Health (NSDUH)
o Youth Risk Behavior Survey (YRBS)
o Parents’ Resource Institute for Drug Education (PRIDE)
o Monitoring the Future (MTF)
No new community‐level outcome data is being collected in connection with the National Evaluation.
Rather, data from these surveys is used extensively in one of the five principal analyses for the National
Evaluation (i.e., comparing outcomes in DFC communities to outcomes in non‐DFC communities).
However, these national surveys generally do not collect sufficient information that can be associated
with a specific community targeted by a coalition. Moreover, these surveys do not collect the detailed
information needed to classify coalitions into a stage‐of‐development. Thus, the design for the
National Evaluation requires additional data collection in these two areas.
In summary, no other source or combination of sources includes all the data needed to evaluate the
DFC Program.
‐ 7 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
A.5 Impact on Small Businesses or Other Small Entities
Data for the CCT and the Semi‐Annual Progress Report are collected from funded DFC coalitions, some
of which may be small entities, as defined by the statute. Compared with early paper‐based reporting
requirements, the web‐based data collection system (COMET) streamlines access to and submission of
coalition data, and thus reduces the paperwork burden on these small entities (see Section A.3 for a
full discussion of use of information technology and burden reduction). COMET includes detailed
instructions and a Help function in the event that technical difficulties are encountered. Finally, a
Helpline number is provided should respondents need additional assistance in using the system or
completing the instruments. No significant impact on small entities is expected.
A.6. Consequences of Collecting the Information Less Frequently
The proposed data collection supports multiple purposes as described in Section A.2, including grant
progress reporting, support of developmental progress, and evaluation.
Grantee progress report data are collected to assess a coalition’s performance in meeting program
goals and objectives and to ensure coalitions receive the technical assistance they need to meet goals
and objectives. ONDCP and SAMHSA policies require that these data now be collected semi‐annually, a
requirement clearly spelled out in the terms and conditions of the grant award and in the request for
proposals. Thus, the fact that this schedule is being reduced from quarterly to semi‐annual reporting in
connection with the National Evaluation actually represents a reduction in required reporting.
While grantee progress report data is captured semi‐annually, the CCT is completed annually. These
two components of COMET cannot be completed less frequently without adversely affecting the
quality and reliability of evaluation data. Anecdotal evidence indicates that coalition capabilities and
capacities can develop and expand substantially over the period of a year. Administering the CCT less
frequently than annually risks missing these developmentally important changes. Similarly, data from
the Semi‐Annual Progress Report cannot be collected less frequently without impairing ONDCP’s
capacity to assess the effectiveness of the program as a whole.
A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
This information collection is consistent with the provisions 5 CFR 1320.5(d)(2). Specifically:
•
Grantees are not required to report information to the agency more often than quarterly.
Progress reports are due semi‐annually and the CCT is due annually.
•
Grantees are not required to prepare a written response to the collection of information in
fewer than 30 days.
•
Grantees are not required to submit more than one original and two copies of any document.
Grantee’s report submissions are done on‐line so there is rarely a need to generate hard copy
reports.
•
Grantees are not required to retain records for more than 3 years.
•
All information collection has been designed to produce valid and reliable results that can be
generalized to the universe of the study.
‐ 8 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
•
Statistical data classification will not occur in the absence of review and approval by OMB.
•
Information collection will not be conducted in a manner that includes a pledge of
confidentiality that is not supported by authority established in statute or regulation, that is not
supported by disclosure and data security policies that are consistent with the pledge, or which
unnecessarily impedes sharing of data with other agencies for compatible confidential use.
•
Grantees are not required to submit proprietary trade secrets or other confidential information
unless the agency can demonstrate that it has instituted procedures to protect the
information’s confidentiality to the extent permitted by law.
A.8. Federal Register Notice and Consultation Outside the Agency
The notice required in 5 CFR 1320.8(d) was published in the Federal Register on Friday, July 18, 2008
(FR vol. 73, no.139, pp. 41382 ‐ 41384, see Attachment 4). No comments were received in response to
the notice.
The evaluation design for the National DFC Support Program – including the classification typology,
data collection tools, and data analytic strategies – was initially reviewed by the DFC Expert Review
Group (ERG) on October 7, 2004. Since this original review group was convened, the evaluation’s
analytic strategies and data collection tools have undergone ongoing scrutiny by the ERG. The ERG was
formally convened in Washington, DC, on January 13 and 14, 2005, and again on February 1 and 2,
2007. A special session of the ERG was convened on May 31, 2007, to review and provide feedback on
the subtraction methodology proposed as a means of comparing DFC to non‐DFC coalitions. A final,
formal gathering of the ERG will take place in the final year of the current evaluation contract and is
anticipated to be scheduled early in 2009. Also, the members of the ERG are often consulted,
depending on the need and the ERG member’s expertise, on an ad hoc basis to review and comment
on specific aspects of the evaluation plan as they unfold.
The ERG comprises representatives of federal agencies and others with expertise in the field of
substance abuse prevention research and evaluation. Information resulting from the discussion among
the participants has been included in final drafts of the evaluation design and analysis plans (see
Attachment 5 for a full list of current ERG members).
ONDCP consulted with the Office of Juvenile Justice and Delinquency Prevention (OJJDP), the
Substance Abuse and Mental Health Services Administration, and the National Institute on Drug Abuse.
OJJDP provided substantial financial resources to support the first year of the evaluation contract.
SAMHSA is providing both financial and technical support for development of the data collection
system (COMET), while NIDA detailed an employee with evaluation expertise to assist in development
of the evaluation design and implementation.1
1
Shakeh Kaftarian, Ph.D.
Health Scientist Administrator, Division of Epidemiology, Services and Prevention Research
National Institute on Drug Abuse, National Institutes of Health
Bethesda, MD 20892-9589
e-mail: [email protected], telephone: (301) 443-8892
‐ 9 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
A.9. Explanation of Any Payment or Gift to Respondents
No payments are made to respondents.
A.10. Assurance of Confidentiality Provided to Respondents
The information to be collected pertains solely to DFC coalitions (e.g., their characteristics, activities,
functions, and community‐level outcomes). No outcome data for any individual persons (e.g., youth
who participate in DFC coalition activities) are sought. Names and contact information for individual
persons in their official capacities as coalition leaders or representatives of organizations comprising
the coalition are captured. This information is routinely provided in the course of grant application and
administration. All personal contact information is treated in a confidential manner. No narratives
gathered as part of the information collection activities will be attributed to a specific individual in any
reports. Below we describe the handling and reporting procedures employed in order to maintain the
privacy of individuals who provide data in their capacities as coalition representatives.
• Every DFC coalition participating in the National Evaluation is assigned a unique identification
number by the Battelle evaluation team. This ID number is used to monitor the DFC coalition’s
status throughout the evaluation.
•
Access to identifying information is limited to the Battelle evaluation team and ONDCP staff
who need to review survey response and data quality.
•
Coding documents and computer files of survey data refer to DFC coalitions by their ID
numbers. No name or institutional identifiers other than ID numbers appear on computer
forms.
•
Individual data bases and computer files are protected by passwords or other techniques to
restrict access to staff involved in data analysis.
•
No data will be reported by ONDCP or the Battelle team in any form that can be traced back to
individual DFC coalitions.
•
Coalitions are not asked to provide individual‐level outcome information or any outcome
information for subgroups that could be used to identify responses of individuals.
Battelle has sought and received internal clearance for the protection of human subjects (see
Attachment 6) in order to comply with 45 CFR 46.
A.11. Justification for Sensitive Questions
No questions are asked that are of a sensitive nature.
A.12. Estimates of Hour Burden Including Annualized Hourly Costs
The time required by coalition leaders to complete the CCT was estimated through the use of a pilot
test (see Section B.4). Nine coalition leaders completed the pilot questionnaire in an average of 55
minutes. Four coalition leaders indicated that it took 60 to 90 minutes to complete the pilot
‐ 10 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
questionnaire. Two leaders indicated that it took 45 to 60 minutes to complete the pilot questionnaire.
Two leaders indicated that it took 30 to 45 minutes to complete the pilot questionnaire, and one
leader completed it in 20 minutes. For the purposes of estimating burden, an average of 60 minutes
per response was assumed.
The Semi‐Annual Progress Report was not formally pilot tested with a group of coalition members.
Rather, the COMET development team and the evaluation team along with representatives from
ONDCP and SAMHSA tested and provided feedback on the content of the Semi‐Annual Progress Report
to ensure that only those items essential to progress reporting and supporting the evaluation were
included as required fields in the system. Further, the design of the COMET system is intended to allow
for daily access and data entry on the part of the coalition and is also intended to serve as a
management tool to help lessen the level of effort associated with day‐to‐day coalition oversight
activities. Therefore, it is anticipated that except for a coalition’s initial access to COMET, time will be
spent in brief spurts throughout the reporting periods for a total of no more than 3 hours in any semi‐
annual report period devoted to COMET reporting.
Exhibit 2. Estimates of Hour Burden
Type of
Number of
Frequency of
Respondents
Respondents
Response
Instrument – Coalition Classification Tool
DFC Grantee*
769
1
Program Directors
Instrument – Semi‐Annual Progress Report
DFC Grantee*
769
2
Program Directors
STOP Act (Prior DFC)
16
2
Grantee Program
Directors**
Total
Average Time
per Response
1
769
3
4,614
3
96
5,479
* includes approximately 64 STOP act grantees who are also DFC grantees
** includes approximately 16 STOP act grantees who were prior DFC grantees
‐ 11 ‐
Total Annual
Burden (in
hours)
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
Exhibit 3. Annualized Cost to Respondents
Type of
Number of
Frequency of
Respondents
Respondents
Response
Instrument – Coalition Classification Tool
DFC
769
1
Grantee*Program
Directors
Instrument: Semi‐Annual Progress Report
DFC Grantee*
769
2
Program Directors
STOP Act (Prior
16
2
DFC) Grantee
Program
Directors**
Total
Hourly Wage
Rate
Respondent
Cost
$24.00
$18,456
$24.00
$36,912
$24.00
$768
$56,136
* includes approximately 64 STOP act grantees who are also DFC grantees
** includes approximately 16 STOP act grantees who were prior DFC grantees
A.13. Estimate of Other Total Annual Cost Burden to Respondents or Record keepers
There are no capital/start‐up or operational/maintenance of services costs to the respondents
associated with this evaluation.
A.14. Annualized Cost to the Federal Government
The annualized contract cost for development of the data collection system and instruments, data
collection, data processing, and analysis is $1.3M. In addition, one federal employee will be involved for
approximately 25% and two additional employees (including a Senior Executive Service staff person)
for approximately 5% of their time over the five years of the project. Annual costs to the government
for federal staff to oversee and support this project are $30,000 for each year, resulting in a total cost
of approximately $150,000 for five years.
A.15. Explanation for Program Changes or Adjustments
The collection of information to support grant management (Semi‐Annual Progress Reports) is an
existing data collection activity. However, the CCT represents an additional data collection activity, as
does the integration of the progress report and the CCT into a web‐based reporting system. Again,
these additional data collection activities are needed to support the National Evaluation. The CCT
captures additional information that will be used to identify the appropriate stage‐of‐development for
a coalition. The Semi‐Annual Progress Report captures additional information to support the
evaluation and the reporting of Government Performance and Results Act (GPRA) measures.
‐ 12 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
A.16. Time Schedule, Analysis Plans and Publication
Time Schedule
The data collection for the evaluation will occur throughout the five‐year period of the contract. A
specific time schedule is provided in Exhibit 4 and has been and will continue to be dependent on the
evaluation team’s access to COMET data following each of the semi‐annual report cycles.
Exhibit 4. Project Time Schedule
Activity
Letters sent to respondents
Time Schedule
2 weeks after OMB approval
Semi‐Annual Progress Reporting Protocol
Ongoing across the five years (semi‐annually)
Coalition Classification Tool Protocol
1 month after OMB approval then annually for five
years
Data analysis: Includes classification, data quality
assessment, and cross‐sectional analyses
6 months after OMB approval, then ongoing for
five years
Submit Annual Report
12 months after OMB approval then ongoing across
the five years (annually)
Conduct longitudinal data analysis
Ongoing across the five years
Submit final report
52 months after OMB approval
Publish national findings on evaluation
52 months after OMB approval
Analysis Plans and Publication
During the past 20 years, community coalitions have typically been evaluated against ultimate
substance abuse outcomes (such as the reduction in 30‐day use of tobacco by youth), rather than
against processes, capacity, and other immediate and intermediate outcomes appropriate to their
developmental stages; thus, only modest impacts of these community‐based efforts were found
(Merzel and D’Afflitti 2003; Berkowitz 2001). To address this challenge, the evaluation framework
views DFC coalitions as embedded in a developmental process that can be tracked across certain
dimensions and different developmental stages to implement prevention interventions; to attain
immediate outputs, intermediate outcomes, and substance abuse outcomes; and to achieve
sustainability and long‐term health and behavioral impact. Further, the framework recognizes that the
actions, activities, outcomes, and impacts of DFC coalitions are conditioned by social, cultural, and
environmental factors within communities, including technical assistance, training, and mentoring
provided by ONDCP and others. DFC community coalitions at different stages of development produce
different processes, capacities, and outcomes. The degree of success of a DFC community coalition
should thus be evaluated against its ability to achieve targeted goals relative to its stage‐of‐
development (i.e., goal attainment).
Similar to previous evaluation efforts, the activities, initiatives, strategies, etc. of DFC coalitions that
have an impact on substance abuse outcome measures of interest (such as 30‐day use of tobacco) will
be examined. However, the greatest warrant for estimating the efficacy of the program will come from
an outcomes analysis focused on those mature and sustaining coalitions (i.e., those coalitions that are
‐ 13 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
expected to have impacted substance abuse outcomes). The impact of DFC coalitions on substance
abuse prevention will also be examined, but using an indirect approach because it may not be
reasonable to evaluate a direct linkage for less advanced coalitions that have not sufficiently organized,
built capacity, and had sufficient time to impact their community to the extent that it can be
successfully measured through substance abuse outcomes. However, if the DFC program can be found
to be an important factor in helping coalitions mature into advanced coalitions (i.e., if characteristics
related to advancement in development can be identified as components of the DFC Program), and if
the link between advanced coalitions and substance abuse outcomes can be established, then it would
be logical and scientifically appropriate to conclude that the DFC program is effective in reducing
substance abuse outcomes; directly for advanced coalitions and indirectly by fostering an environment
where less advanced coalitions can become advanced coalitions.
Trends in substance abuse outcomes in communities targeted by DFC coalitions will be compared to
the corresponding trends in communities that are not specifically targeted by a DFC coalition, and this
will be done indirectly using state‐ and national‐level data. This analysis differs from an approach
where comparison communities are selected and matched to DFC coalition communities and a direct
comparison is conducted between the two different types of communities. Employing a comparison‐
community type approach has significant challenges and barriers to obtaining quality and
comprehensive information. These include the need to identify appropriate comparison communities
that can be determined to be “similar” to a DFC coalition community, the need to identify key
informants to provide community information (a significant challenge in communities that do not have
a coalition), convincing local contacts to continue to provide this detailed information throughout the
evaluation, and the questionable quality of the outcome information that could be obtained. Due to
these limitations and resource constraints, the current approach relies on extracting indirect
surrogates for substance abuse outcomes for communities not targeted by a DFC coalition from
existing national and state surveys to serve as a comparison to DFC communities.
Although, this approach will limit the number of explanatory factors available for advanced modeling,
it has a decided advantage in that it does not rely on a direct data collection activity conducted as part
of this evaluation effort. Unfortunately, publicly available information is typically only available at a
broad geographic level, which prohibits specific community‐to‐community comparisons.
In January, 2007 the evaluation team concluded a Feasibility Assessment (see Attachment 7),
conducted to determine if community‐level information could be obtained from some of the
established national surveys. The results of this assessment follow.
• Adequate data are not available from all states and therefore not all DFC coalitions could be
included in the same investigation using state‐collected data. The primary design of the DFC
National Evaluation and GPRA reporting require data on all DFC program grantees and their
communities. Therefore, under current conditions, the use of state‐collected data or sources
other than grantees is not feasible at this time.
• Data from the National YRBS is only available at the state level. Data for the National YRBS are
collected at the school level, but school identifiers are removed when states send the data to
the CDC. The data can be used to compare states or region; otherwise, it has limited use to the
‐ 14 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
DFC National Evaluation (i.e., as a comparison for core measure trends. The National Evaluation
could also consider city/county comparisons using the data from the 23 YRBS participants in
local communities for secondary analysis purposes. However, the YRBS only adequately
includes two of the four core measures.
•
•
Some states administer different questionnaires to different populations in their states, some
with overlapping participants. The use of multiple questions and questionnaires would require
an extensive data coordination and cleaning task if state‐collected data were to be used.
Frequency of data collection across and within states is so varied that only comparisons of
trends would be possible. Dates of data collection and the intervals between them are
inconsistent (e.g., two or three years apart). Having diverse data collection times limits the
opportunity to look at absolute changes (i.e., a net change between two times) in local, state,
and youth substance abuse. The National Evaluation would have to compare trends instead,
requiring the collection of multiple data points. Comparing trends would allow the evaluation
to indicate how many DFC coalitions report change, the average change they report, and
difference in trends between DFC communities and national, state, or local communities.
However, the evaluation would not be able to quantify the actual net change that DFC
coalitions have on their communities from 2004 to 2009. Grantee‐provided data faces the same
challenge.
Within the evaluation framework, six different statistical analyses will be conducted as part of the core
evaluation effort: (1) preliminary analyses, (2) assessment of stage‐of‐development, (3) modeling
substance abuse outcomes, (4) assessment of additional evaluation questions and hypotheses,
(5) analyses to support GPRA reporting, and (6) Analysis of External Data to Enhance Substance
Abuse Outcome Information. The combination of these five analyses will be used to assess whether
the DFC program is, directly or indirectly, reducing substance abuse in the communities where it
operates. Exhibit 5 illustrates the relationships between the proposed analyses and the evaluation
objectives.
Exhibit 5. Cross‐link Between Evaluation Analysis Objectives
‐ 15 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
This section describes our proposed statistical approach for each analytical task in relatively broad
terms. Additional detail, including statistical formulae, is provided in Attachment 2, Evaluation Design
Document.
1. Preliminary analysis. One implicit objective of this evaluation is to provide information regarding the
status of the DFC program and the characteristics of the participating coalitions. This information will
represent a useful context and background for the remaining analyses. It will also provide an initial
overview of how the coalitions’ characteristics (e.g., size, structure, degree of formalization, etc.)
change over time. This information is important for refining the overall evaluation framework and
stage‐of‐development typology.
Statistical approach. This component of the evaluation will use descriptive statistics to describe the
coalitions participating in the DFC Program by summarizing such elements as:
• Number of coalitions
•
Current and historical classification of coalitions
•
Geographic representation of communities served
•
Total population served by DFC coalitions
•
Ratio of number of DFC coalitions to number of all substance abuse coalitions
•
Describe basic characteristics (i.e. capacities, context, etc.) of the coalition
As with the overall program, the features and characteristics of coalitions are expected to change over
time. The responses to each component of the data collection instruments will be summarized using
frequency distributions and other descriptive statistics. To facilitate characterization of the relationship
between coalition characteristics and outcomes, both cross‐sectional (single point in time) and
longitudinal (measures across time for the same coalitions) summaries will be constructed.
Cross‐sectional and longitudinal analyses will be conducted annually following each phase of data
collection. Simple descriptive summaries will describe the outcome data and other potential
explanatory variables among the population of DFC coalitions. Side‐by‐side box plots, bar and line
charts and simple descriptive statistical summary tables will be prepared to illustrate and summarize
the distribution of substance abuse outcome measures as a function of community type (urban/inner
city, suburban, rural), stage‐of‐development, and grade of school, gender, and race. As appropriate,
the summary statistics provided in the tables will include sample size (number of communities), mean,
standard deviation, minimum, 10th percentile, 25th percentile, median, 75th percentile, 90th percentile,
and maximum. For the outcome measures, summary statistics may be provided for the whole
community, as well as for each community type, stage‐of‐development, school grade, gender, and
race. Summary statistics will also be used to describe the distribution of covariates by community type
and stage‐of‐development.
2. Assessing stage‐of‐development. After the initial analyses to understand the basis capacities and
characteristics were conducted, the evaluation team set out to develop a typology for coalitions’ Stage
‐ 16 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
of Development, or maturation over time. Using the information collected through the CCT modules of
COMET, DFC coalitions have been classified into one of four stages of development (Exhibit 6). This
section describes the methodology used to develop the coalition classification scheme and the
validation of the approach selected.
Coalitions are classified along a dimension of less mature to fully mature and sustaining. The coalition
typology framework used by the evaluation team was developed from the existing research literature
and the experience of practitioners. It merges three main themes in the literature: maturation
(coalitions get better over time); coalition processes (e.g., SAMHSA’s Strategic Prevention Framework)
and coalition capacities (e.g., knowledge, skills, resources, and relationships needed to meet goals and
achieve functions). This typology rests upon a conceptualization of coalitions moving through four
stages of development: (1) Establishing, (2) Functioning, (3) Maturing, and (4) Sustaining. As shown in
Exhibit 6, as coalitions move through these stages, they acquire greater sophistication with respect to
their organizational structure, capacity, and focus of efforts as well as in their levels of competency to
perform vital functions necessary to impact change.
The model recognizes, however, that developmental progression may not be linear; coalitions may
progress and regress through developmental stages and change over the course of the evaluation. For
example, a Functioning coalition that loses a key coalition leader may regress to an Establishing
coalition while the coalition rebuilds, then become a Functioning coalition again at a later date.
Therefore, when assessing whether the DFC Program has had an impact on the stage‐of‐development
for the grantee coalitions, it is important to assess the overall trend, recognizing the often cyclical
nature of coalition development.
Exhibit 6. Stages of Development for Prevention Coalitions
Stage of
Development
Description
Level of
Competency to
Perform Functions
Establishing (i)
Functioning (ii)
Initial formation Follows the completion
with small
of initial activities, focus
leadership core on structure and more
working on
long‐range
mobilization and
programming
direction
Primarily learner
Achieving proficiency;
still learning and
developing mastery
Maturing (iii)
Sustaining (iv)
Stabilized roles,
structures, and
functions.
Confronted with
conflicts to
transform and
“growing pains.”
Achieved mastery;
learning new
areas; proficient in
others
Established
organization and
operations, focus
on higher level
changes and
institutionalizing
efforts
Mastery in primary
functions;
capacities in the
community are
sustainable and
institutionalized
Statistical approach. The CCT contains four six‐item scales measuring coalition capacity and functions
(see Attachment 3b, questions 2, 20, 22 and 25). Items are coded on a 5‐point scale as follows:
“Novice” (or a score of 1 on the 5‐point scale) indicates that the respondent feels the coalition is still
learning how to perform the functions in various areas and could therefore benefit from assistance
from others; “Proficient” (or a score of 3 on the 5‐point scale) indicates that the respondent feels the
‐ 17 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
coalition is competent in performing the function; and, “Mastery” (or a score of 5 on the 5‐point scale)
indicates that the respondent believes the coalition is at an expert level of performance in the areas
and could train or be of assistance to others in performing these functions. No labels are associated
with scores of 2 or 4, but the intention of the scales was that a 2 represents a score between Novice
and Proficient and a 4 represents a score between Proficient and Mastery. Exhibit 7 shows each of the
items included in the scale of coalition maturation. See Attachment 3b for the full text on each
dimension.
Exhibit 7. Questions Included on the CCT Measuring Stage‐of‐Development Dimensions (Four Scale
Items and 6 Sub‐Items within Each Scale)
Stage‐of‐Development Dimension
(Item text has been abbreviated)
Question Labels Novice=1; 3=Proficient; 5=Mastery
Activity/
Functional Areas
(Sub‐items)
Assessment
Scale (1‐5)
Capacity
(1‐5)
Planning
(1‐5)
Implementation
(1‐5)
Coalition
Development
and
Maintenance
Deciding which
skills and
resources will be
needed,
assessing which
organizations
and/or
individuals to
recruit
Building member
participation
skills, providing
desired training
and technical
assistance to
develop coalition
structure
Building
consensus
around coalition
mission,
developing a
mission
statement and
general goals
Establishing the
coalition
structure and
operating
procedures…
Coordination of
Prevention
Program/Services
Compiling prevalence
and risk and protective
factor data, prioritizing
needs
Environmental
Strategies
Determining retail and
social sources of
substance availability to
underage youth,
knowledge of
community compliance
with local ordinances
Intermediary or
Community Support
Organization
Understanding current
knowledge and skills
among community
leaders, staff, and
residents on
prevention strategies
Building a solid
knowledge base (e.g.,
familiarity with
evidence‐based
programs and services)
and required skills in
program design, activity
planning
Analyzing and selecting
programs, services, and
activities that provide a
best “fit” with
community conditions
Developing a solid
knowledge base (e.g.,
definition, rationales,
and evidence for
environmental
strategies)
Leveraging the capacity
of other organizations,
community leaders,
and residents by
recruiting highly skilled
staff and consultants
Identifying a range of
potential policy changes
and enforcement
activities, selecting the
best “fit” with current
community conditions
Designing learning
systems,
communications and
marketing plans,
integrated technical
assistance and training
plans
Arranging settings for
program delivery (e.g.,
school, community‐
based organization),
creating public
awareness, recruiting
strategies…
Develop experience in
carrying out a
sequenced social
marketing campaign..
Advertising, recruiting
for, and conducting a
series of workshops,
developing resource
centers or web site for
the distribution of
information, and
brokering resources
from state and national
resources
‐ 18 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
Stage‐of‐Development Dimension
(Item text has been abbreviated)
Question Labels Novice=1; 3=Proficient; 5=Mastery
Activity/
Functional Areas
(Sub‐items)
Evaluation
(1‐5)
Planning for
Sustainability
(1‐5)
Coalition
Development
and
Maintenance
Assessing
member
satisfaction, skill
development….
Planning for
changes in
leadership,
standardizing
operating
procedures….
Coordination of
Prevention
Program/Services
Conducting process and
outcome evaluations to
refine or eliminate
programs
“Institutionalization” or
incorporation of an
evidence‐based
program as part of
ongoing organizational
operations in your
community
Environmental
Strategies
Monitoring
enforcement or
documenting changes
in social indicators to
measure policy change
Arranging for regular
prevention columns in
local newspapers or
securing line items in
organizations’ budgets
that institutionalize
prevention strategies
Intermediary or
Community Support
Organization
(For example, training,
technical
assistance/consul‐
tation, educational
program, and material,
etc.) Monitoring
satisfaction and
evaluating changes in
knowledge skills and
resources.
Planning for
sustainability of
capacity building
functions. Securing
ongoing funding or
institutionalizing
services into the
ongoing operations of
other community‐
based organizations.
Classification Methodology and Empirical Validation of Typology
A. Classification Methodology Statistical approach. The statistical approach to this analysis involves two
steps: (a) creating the classification algorithm and (b) validation. To create the coalition typology, we
calculated a mean score across each of the items in each dimension and overall for each coalition. In
addition, mean scores were calculated for each of the three survey waves of the CCT. Coalitions
reporting average overall scores that were between 1 – 1.999 (novice average rating) were categorized
as Establishing; coalitions reporting average scores between 2 – 2.9999 (novice to proficient average
rating) were categorized as Functioning; coalitions reporting average scores between 3 – 3.9999
(proficient average rating) were categorized as Maturing; and coalitions reporting average scores
between 4 – 5 (highly proficient to mastery average rating) were categorized as Sustaining.
Results
The distribution (percentages and means) of the coalitions as a function of the stage‐of‐development
typology for each of the CCT waves is shown below in Exhibit 8. As shown in Wave 1, 11.5% of the DFC
coalitions are categorized as Establishing, 43.3% as Functioning, 38.7% as Maturing, and 6.5% as
Sustaining. The mean score across all 26 items for the establishing coalitions was 1.7; for functioning
coalitions the mean score is 2.6; for maturing coalitions the mean score is 3.4; and for sustaining
coalitions the mean score is 4.3. Percentages for each of the subdomains are also shown in Exhibit 8.
These results show that according to this method of categorizing the coalitions, the vast majority are
categorized as either Functioning or Maturing; and many fewer as Sustaining or Establishing. Similar
‐ 19 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
results are shown for the other two survey waves. It does appear that over time fewer coalitions are
categorized as Establishing and more as Sustaining, indicating some level of maturation over time.
However, there are different coalitions in each of the three waves so the results over time are not
directly comparable. (In a later section of this package, the sample is restricted to only those coalitions
that were in all three survey waves.)
Exhibit 8. Distribution of DFC Coalitions across Survey Waves and by the 4 Dimensions
Wave 1 (N=719)
Percentage
Mean
Subdomains:
Coalition Development
and Maintenance
Coordination of
Prevention
Program/Services
Environmental Strategies
Intermediary or
Community Support
Organization
Wave 2 (N=729)
Percentage
Mean
Coalition Development
and Maintenance
Coordination of
Prevention
Program/Services
Environmental Strategies
Intermediary or
Community Support
Organization
Wave 3 (N=745)
Percentage
Mean
Coalition Development
and Maintenance
Coordination of
Prevention
Program/Services
Environmental Strategies
Intermediary or
Community Support
Organization
Establishing
11.5%
1.7
5.29%
Proposed Typology
Functioning
Maturing
43.3%
38.7%
2.6
3.4
36.86%
43.67%
Sustaining
6.5%
4.3
14.19
7.23%
30.88%
44.37%
17.52
17.66%
16.83%
41.59%
44.09%
31.71%
31.02%
9.04%
8.07%
7.1%
1.7
4.66%
42.4%
2.6
31.28%
40.1%
3.4
48.70%
10.4%
4.4
15.36%
3.57%
32.37%
43.21%
20.85%
12.62%
12.21%
41.15%
44.03%
31.96%
30.59%
14.27%
13.17%
5.9%
1.6
4.16%
40.3%
2.6
31.28%
41.5%
3.4
46.58%
12.4%
4.3
17.99%
4.17%
28.40%
43.34%
24.09%
10.51%
10.78%
38.81%
41.37%
37.06%
34.77%
13.61%
13.03%
B. Empirical validation of typology. To validate the typology, the evaluation team used available data
from other items on the CCT and data collected by COMET. First, they looked for internal consistency
with the additional items on the CCT. If the CCT is a robust tool for categorizing the coalitions, then
other similar question items on the survey should be consistent with the stage‐of‐development
‐ 20 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
typology. Second, they used data from an external source (COMET) to examine whether the leveraging
of funding, an indicator of coalition maturation, was related to the proposed stage‐of‐development
coalition typology. Below are the results of each of these analyses.
Exhibits 9, 10, and 11 below show the stage‐of‐development typology across the three CCT survey
waves for four validating items on the CCT (see Attachment 3b, questions 27, 3, 16, and 18). These four
items were selected because they represent other “global” measures of the coalitions’ stage‐of‐
development and could be used as a means to check the reliability of the new typology measure with
other related items on the survey. Results indicated that these additional measures of global
competency were positively correlated with the typology, resulting in correlation coefficients ranging
from .55 to .71 (p<.0001).
Exhibit 9. Percentage of DFC Coalitions responding to various CCT items by Stage‐of‐Development
Typology (Wave 1 CCT).
Rate your coalition on its capacity to perform key functions (Q27)
Proposed
Stage of
Development
Typology
Establishing
Functioning
Mature
Sustaining
Coalition is learning
how to perform key
functions
75.9%
24.2%
2.5%
0%
Coalition achieving
proficiency
24.1%
68.2%
53.4%
14.9%
Coalition achieved
mastery in most but
cannot sustain
0%
7.1%
39.7%
68.1%
Coalition achieved
mastery and is
sustainable/
institutionalized
0%
.6%
4.3%
17.0%
Exhibit 10. Percentage of DFC Coalitions responding to various CCT items by Stage‐of‐Development
Typology (Wave 1 CCT).
How would you best describe your coalition? (Q.3)
Proposed Stage
of Development
Typology
Establishing
Functioning
Mature
Sustaining
A highly formal
arrangement with
most
A semi‐formal
Formal group of
organizations
group of
organizations who
organizations who
plan and act
having a clear role
Loosely organized
have begun to
together to
in the planning
group whose main work together on
implement
and
goal is
implementation
prevention
prevention
information
of community
programs and
programs and
sharing
strategies
strategies
wide strategies
8.4%
65.1%
24.1%
2.4%
.3%
39.9%
57.2%
2.6%
0%
12.2%
75.9%
11.9%
0%
2.1%
55.3%
42.5%
‐ 21 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
Exhibit 11. Percentage of DFC Coalitions responding to various CCT items by Stage‐of‐ Development
Typology (Wave 1 CCT).
Proposed Stage
of Development
Typology
Establishing
Functioning
Mature
Sustaining
Your coalition has established a
reputation for “being able to get
things done” related to the area of
substance abuse prevention (Q16a)
(percent agree/strongly agree)
38.5%
68.5%
92.4%
97.9%
Do you think your coalition will be able
to sustain itself in the community for the
next ten years? (Q17)
(percent yes)
55.4%
60.0%
76.9%
84.8%
COMET Data – Percentage of DFC Funding
When we examined the relationship between the percentage of DFC funding (collected in COMET) and
the stage‐of‐development typology, we found that the more likely the coalition is to rely on DFC
funding only, the less likely they are to be in a “higher” category in the typology (e.g., sustaining)
because more mature/sustaining coalitions are better able to leverage other resources. This finding
supports the validation of the typology since the relationship was in the expected direction and was
statistically significant (‐.23 and p<.0001).
Maturation of the Coalitions over Time
To investigate whether the typology would show that the DFC coalitions were “maturing” over time,
we re‐ran the analysis for each of the three CCT survey waves (see Table 3 above) by restricting the
sample to only those coalitions that participated in all three waves. As shown in Exhibit 12, the
percentage of coalitions classified as Establishing decreased from 12.3% in Wave 1 to 3.5 % in Wave 3,
while the percentage of coalitions classified as Sustaining increased from 6.3% in Wave 1 to 14.6% in
Wave 3. When we investigated coalition advancement between Wave 1 and Wave 3, with the same
subset of coalitions, we found that 85% of the Establishing coalitions at Wave 1 advanced to
Functioning or above by Wave 3; 51% of the Functioning coalitions advanced to Maturing or above
(46% remained the same); 21% of the Maturing coalitions became Sustaining coalitions (58% remained
the same); and 66% of the Sustaining coalitions remained Sustaining in Wave 3.
Exhibit 12. Distribution of the DFC Coalitions by Stage‐of‐Development Typology by CCT Waves*
CCT
Wave 1
Wave 2
Wave 3
Establishing
12.3%
4.7%
3.5%
Proposed Typology
Functioning
Maturing
43.7%
37.7%
42.2%
42.0%
36.1%
45.9%
Sustaining
6.3%
11.1%
14.6%
* Sample restricted to only those coalitions participating in all three waves; N= 602.
Descriptive analyses were then conducted on the typology as a function of grade level and outcome,
without any imputation procedures. The results of these analyses show that between 2006 and 2007
past 30‐day use rates declined for all three drugs and for all four grades. Furthermore, when examining
‐ 22 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
the outcomes as a function of maturity (i.e., by collapsing Maturing with Sustaining and Establishing
with Functioning), Maturing/ Sustaining coalitions had lower use rates compared to Establishing/
Functioning coalitions for all three drugs and for all four grades in 2007 (see Exhibits 13 through 16).
Exhibit 13. Distribution of DFC Coalition Measures by Typology and by Year of Outcome Data
Collection—9th Graders
30%
4.
6%
6.
7%
6.
4%
2.
1%
3.
1%
6.
3%
6.
5%
10%
9.
6%
9.
6%
10
.5
%
15
.4
%
16
.3
%
20%
0%
2006
2007
Alcohol
2006
2007
2006
Tobacco
Establishing/ Functioning
‐ 23 ‐
2007
Marijuana
Maturing/ Sustaining
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
Exhibit 14. Distribution of DFC Coalition Measures by Typology and by Year of Outcome Data
Collection—10th Graders
13
.6
%
9.
2%
7.
2%
10%
8.
3%
9.
6%
13
.6
%
16
.0
%
17
.1
%
20%
16
.2
%
19
.0
%
27
.0
%
30%
29
.6
%
40%
0%
2006
2007
2006
Alcohol
2007
2006
Tobacco
2007
Marijuana
Establishing/ Functioning
Maturing/ Sustaining
Exhibit 15. Distribution of DFC Coalition Measures by Typology and by Year of Outcome Data
Collection—11th Graders
40%
37
.4
%
41
.0
%
50%
10%
8.
0%
7.
6%
11
.5
%
13
.4
%
18
.5
%
19
.1
%
22
.1
%
16
.1
%
20%
21
.5
%
23
.6
%
30%
0%
2006
2007
Alcohol
2006
2007
Tobacco
Establishing/ Functioning
‐ 24 ‐
Maturing/ Sustaining
2006
2007
Marijuana
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
Exhibit 16. Distribution of DFC Coalition Measures by Typology and by Year of Outcome Data
Collection—12th Graders
12
.6
%
9.
5%
11
.2
%
14
.1
%
20%
17
.5
%
16
.6
%
22
.0
%
21
.4
%
25
.7
%
30%
21
.7
%
36
.7
%
40%
38
.1
%
50%
10%
0%
2006
2007
Alcohol
2006
2007
Tobacco
Establishing/ Functioning
2006
2007
Marijuana
Maturing/ Sustaining
In 2006, there were few differences between coalition type and use rates for marijuana and tobacco.
However, past 30‐day use rates of alcohol were lower for alcohol in both 2006 and 2007 for
Maturing/Sustaining DFC coalitions compared to Establishing/ Functioning coalitions.
Overall, the results suggest that among DFC coalitions, trends of decreasing use rates have been found
over time and that when examined as a function of grade and coalition maturity over time,
Maturing/Sustaining coalitions report lower use rates than Establishing/Functioning coalitions.
Although there were no statistical tests of significance conducted due to limited sample sizes across
the years, grades, and drugs, more sophisticated analyses such as these will be conducted in the
future, as additional data are collected.
These results demonstrate that the coalition typology is robust and provides a theory‐supported tool
for categorizing coalitions. Further, when mapped against past 30‐day coalition substance use rates,
these findings provide support for the theory that maturing and sustaining coalitions will be more
effective in reducing past 30‐day substance use than establishing and functioning coalitions. The
coalition typology provides a valuable tool for tracking coalition development over time and provides
empirical support for a developing theory of the determinants of coalition effectiveness. As more
coalitions progress through the typology, the DFC National Evaluation can expect to see increasing
coalition effectiveness in reducing youth substance use.
‐ 25 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
3. Modeling substance abuse outcomes. Since the typology has been established as a valid and
reliable indicator of maturation, it is important to examine outcome measures as they relate to stage‐
of‐development. Significant predictors of favorable changes in substance abuse outcomes will be
identified by statistical modeling as our first step in the analysis. For example, we will assess whether
the presence of a DFC coalition classified as Maturing or Sustaining is associated with a decrease in the
proportion of youth who report using alcohol in the last 30 days. This analysis will be conducted both
for specific time points and using longitudinal models so that trends over time can be assessed.
In addition to comparing trends directly, trends in substance abuse outcomes in communities targeted
by DFC coalitions will be indirectly compared—using state‐ and national‐level data—to the
corresponding trends in communities that are not specifically targeted by a DFC coalition. For this
evaluation, communities with DFC coalitions cannot be compared directly to matched communities
that do not have DFC coalitions. As a result, the evaluation will rely on surrogate measures of
substance abuse outcomes in communities not targeted by DFC coalitions as the benchmark or index
against which DFC communities will be compared. These surrogate measures will be derived from
large‐scale surveys that are designed to estimate outcome and risk‐factor prevalence at the state and
national levels. A significant limitation of this approach is that data for many explanatory factors will
not be available for non‐DFC communities. On the other hand, derivation of these indirect measures
does not require that data for comparison communities be collected as part of this evaluation effort.
This matched comparison approach was considered but was determined to be infeasible for this
evaluation because of logistical considerations, concerns about whether an appropriate contact person
could be identified, and concerns about recruitment and retention of comparison communities.
Statistical approach. The association between key explanatory covariates and substance abuse
outcomes in DFC communities will be modeled using fixed and random effects inverse variance
weighted regression using natural log transformed proportions. Key covariates will likely include the
coalition stage‐of‐development, coalition capacity, extent of activities, and environmental strategies
employed. Longitudinal regression models will be fitted using a GEE approach to account for the
anticipated positive correlation in substance abuse outcome measures on the same community over
time. Specifically, the regression models will express the proportion of beneficial responses within a
community as a function of a number of covariates. Forced entry regression will be used to identify
covariates that are significantly associated with substance abuse outcomes after adjusting for the
presence of other covariates. Stage of development and length of time that the coalition has existed
will be retained in the model regardless of their statistical significance. If a preliminary analysis
indicates that two variables are highly correlated, only the most theory‐defensible variable will be
tested first and retained if significant. Additionally, data reduction techniques such as principal
components may be used to reduce the dimensionality of the data.
In addition to examining substance abuse trends among DFC coalitions, we propose using surrogates
for national benchmarks against which to judge DFC findings. Several large national surveys collect
substance abuse outcome data for states and for some large subdivisions of states (counties,
metropolitan areas) (see Exhibit 17). For the purposes of the evaluation, states (or other geopolitical
subdivisions) can be seen as comprising two kinds of communities—those that have DFC coalitions
(DFC communities) and those that do not (non‐DFC communities). At the state level, the prevalence of
substance abuse outcomes at any point in time is the weighted average of the outcome prevalence
across both types. By extension, trends in the prevalence of these outcome measures over a follow‐up
‐ 26 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
period (say, five years) form two sets of curves, one for DFC communities and one for non‐DFC
communities. For the state, the trend in prevalence of the outcome measures is the sum of the curves
for DFC and non‐DFC communities. Assuming that appropriate weights are available, the prevalence of
substance abuse outcomes in non‐DFC communities can be estimated if outcome prevalence for DFC
communities and the state are known. Exhibit 18 illustrates the use of prevalence estimates for
substance abuse outcomes in states and in DFC communities to evaluate the impact of the DFC
program over a follow‐up period. The statistical methods for the comparison are given in detail in
Attachment 2, Evaluation Design Document. Separate analyses will be conducted to investigate the
other four evaluation questions that focus on coalition capacity.
4. Assessment of additional evaluation questions and hypotheses. Seven additional evaluation
questions that complement the primary objectives of the evaluation are laid out in the evaluation plan
(see Section A.2 and Attachment 2, Evaluation Design Document). Three of these questions focus on
the relationship between potential explanatory variables (coalition composition and degree of
collaboration, geographical focus of coalition, and effectiveness of environmental strategies) and past
30‐day substance abuse outcomes (i.e., percent of youth who report using of alcohol, tobacco, and
marijuana in the previous 30 days). The remaining four questions focus on facets of coalition capacity
(use of evidence‐based strategies, sustainability, national capacity and impact of technical assistance)
and how these factors relate to substance use outcomes. Separate analyses on these relationships will
be examined.
Exhibit 17. Summary of Substance Abuse Outcome Information Available from Three Large,
Ongoing Surveys
Outcome
30‐day Use
Perception of Risk
Perception of Parental Disapproval
Age of Onset
Perceived Peer Approval
Perceived Norms and Beliefs
Perceived Availability
Availability of Measures by Grade
Availability of Measures by Gender
Respondents’ Age Range
Respondents’ Grade Range
Geographic Data Available
1
PRIDE1
A* T# M§
N
N N
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Yes
Yes
8 to13+
14 to 18
4‐6 graders, 6‐
12 graders
Fee‐based
2
A
Y
N
N
Y
N
N
N
YRBS2
T
Y
N
N
Y
N
N
N
Yes
Yes
M
Y
Y
N
Y
N
N
N
NSDUH3
T
Y
Y
Y
Y
Y
Y
N
Yes
Yes
A
Y
Y
Y
Y
Y
Y
N
12‐18
12+
9th‐12th
All
State
Conducted in All States
Parents Resource Institute for Drug Education survey; Youth Risk Behavior Survey;
National Survey on Drug Abuse and Health (formerly the National Household Survey of Drug Abuse).
*A = alcohol; #T = tobacco; §M = marijuana.
3
‐ 27 ‐
M
Y
Y
Y
Y
Y
Y
Y
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
Exhibit 18. Illustration of Calculating Non‐DFC Community Substance Abuse Outcomes Using State
Profiles; Rhode Island Used as Model
30-Day UseAlcohol - Combined Grades for RI
100
80
60
40
20
0
•
RI – YRBS State Profile for 30‐day Alcohol Use =
45%
• 1 Coalition, 30‐day alcohol use = 10%
•
Suppose Community Targeted by DFC Coalition =
5% of population in RI, then
•
Non‐DFC Comparison Community 30‐day alcohol
use =
45% = 5% (0.1) + 95% * (X) Î X = 46.8%
So, Non‐DFC Communities in RI 30‐day Use is
46.8%
An additional evaluation question concerns the effectiveness of ONDCP’s Mentoring program, which is
relatively small—31 of the 769 funded coalitions participate. This sample size is too small to identify
significant relationships between mentoring or support activities and changes in capacity or level of
effort among the mentored coalitions. Nonetheless, we will attempt to assess the impact of this
program quantitatively by coding support activities of the mentoring coalitions to develop categories
of activities (i.e., a measure of mentoring “exposure”).
Statistical approach. The three evaluation questions that are related to the role of explanatory
variables in substance abuse outcomes will be addressed by forcing these measures or indicator
variables as covariates into the substance abuse outcome models. Separate models will be developed
for each of the remaining four evaluation questions focused on coalition capacity. Generally, these
models will consist of a family of general linear models with repeated measures to account for multiple
observations from the same coalitions over time. As appropriate, analyses of outcome data (e.g., past
30‐day substance use) will be weighted by confidence in the finding as represented by the inverse
variance weight of the community outcome estimate. The specific form of each model will depend on
the nature of the outcome variable—Poisson regression for modeling outcomes that are counts;
logistic and polytomous logistic regression models for categorical outcomes; and linear regression
models for continuous outcomes. The evaluation questions have been translated into specific
hypotheses of interest that can be tested using statistical models. Exhibit 19 and Exhibit 20
summarizes the specific modeling technique that is anticipated for each evaluation question and
related hypotheses.
For the coalitions in the Mentoring program, summary statistics and logistic regression models will be
used to examine the relationship between “mentoring exposure” or effort and coalition capacity
outcomes.
‐ 28 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
Exhibit 19. Analysis Approach for Additional Evaluation Questions Regarding Relationships to
Substance Abuse Outcomes
Evaluation Question
Study Hypotheses
Analysis Method
Composition/collaboration. What mix of agencies A. There are specific activities and/or
and types of collaboration are most associated
collaborations that are associated with
with improvements in community substance use
substance abuse outcomes and
as well as risk and protective factor outcomes?
risk/protective factors.
Outcomes vs. geography and socio‐economic
A. There are specific relationships between
status. What evidence, if any, illustrates an
geography and socioeconomic status and
association between differences in outcomes and
substance abuse outcomes and
such factors as geographic location (urban/rural/
risk/protective factors.
suburban) or socioeconomic status?
Effectiveness of strategies. What are the most
A. There are specific relationships between
effective strategies? What mix of strategies led to
use of specific strategies and substance
positive community changes? Is there any
abuse outcomes and risk/protective factors.
relationship to type, level, and coordination of
B. There are specific relationships between
type, level, and coordination of outside
outside funding streams?
funding and substance abuse outcomes and
risk/protective factors.
Variables Included
as covariates in
substance abuse
outcome models
Exhibit 20. Analysis Approach for Capacity‐Related Evaluation Questions.
Evaluation Question
Study Hypotheses
Increase in evidence‐based programs, A. DFC Grantees have increased their use of
policies, and strategies. What evidence
evidence‐based programs.
exists to demonstrate an increase in
B. DFC grantees have had an increase in their
evidence‐based programs, policies, and
impact on substance abuse policies.
strategies in coalition communities?
C. DFC grantees have increased the use of
environmental strategies to reduce substance
abuse.
Sustainability. What evidence exists
A. DFC coalitions become (are) sustainable.
that demonstrates the sustainability of
DFC coalitions?
Increased national capacity. To what
A. The DFC grant program has increased the
extent has the number of communities
number of communities with established
with established coalitions increased (a
coalitions.
Healthy People 2010 objective)?
B. DFC coalitions that have received funding
advance in development (i.e., become more
established).
‐ 29 ‐
Analysis Method
Poisson regression,
logistic regression
Logistic regression,
linear regression
Poisson regression
Hypothesis B will be
examined through the
stage‐of‐development
analysis
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
Evaluation Question
Study Hypotheses
Analysis Method
Impact of technical assistance on data
collection, application, and
implementation of environmental
strategies. What evidence exists that
supports or negates an association
between the provision of technical
assistance and increased data
collection and application and/or use of
evidence‐based strategies by
coalitions? Does receiving technical
assistance increase the likelihood that a
new coalition will subsequently obtain
new DFC funding? Do these
relationships vary with the source of
the technical assistance?
A. Coalitions’ receipt of technical assistance is
positively correlated with stage‐of‐development
(i.e., more technical assistance results in higher
stages of development).
B. Some sources/types of technical assistance are
more effective than others.
C. Technical assistance results in increased data
collection.
D. Technical assistance results in increased use of
evidence‐based strategies.
Hypothesis A will be
examined through
stage‐of‐development
analysis
Logistic regression
5. Analyses to support GPRA reporting. ONDCP is required to submit a Government Performance and
Results Act report to Congress annually regarding the DFC Grant Program. Exhibit 21 lists the 2008
GPRA goals and objectives. Information on accomplishments toward these goals and objectives will
primarily be captured through COMET.
Statistical approach for establishing targets. Targets for annual GPRA performance were established by
calculating the upper 95% confidence interval for each baseline proportion and for each successive
year’s target value. That is, the upper 95% boundary value for each baseline proportion became the
second program year’s target value. The upper 95% boundary value for the second year’s target, was
selected as the third year’s target, and so on.
The 95% confidence interval was calculated using Logit transformed proportions (see Formula 1) with
the standard error of the proportion calculated using Formula 2 and an assumed sample size of 700 (the
approximate number of coalitions in the first year of DFC-funding). The actual upper boundary of the
confidence interval was calculated using Formula 3 and transformed back into its associated proportion
using Formula 4.
Calculating the logit transformed proportion
⎡ p ⎤
ES L = log e ⎢
⎥
⎣1 − p ⎦
Formula 1
Where loge = the natural log and p = the proportion of subjects in the category of interest (i.e., the
baseline or current target).
Calculating the standard error
1
1
SE L =
+
np n(1 − p)
Formula 2
Where n = 700 and p = the proportion of subjects in the category of interest.
Calculating the upper 95% confidence interval
‐ 30 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
T = ES L + (1.96 * SE L )
Formula 3
Where T = the new target expressed as a logit, ESL is the logit transformed target, and SEL is the
standard error for the logit transformed target.
Transforming the logit target (T) back into its associated proportion
e ES L
p = ES L
e +1
Formula 4
Where e ES L is the base of the natural logarithm raised to the power of the logit transformed proportion.
Because DFC outcomes are collected on alternating years, the target values calculated using this method
are ambitious in that they project statistically significant change in DFC coalition performance between
each anticipated wave of coalition reporting. Because the number of funded DFC coalitions is expected
to grow, the assumption of 700 coalitions contributing data is conservative for estimating upper 95%
confidence intervals.
Statistical approach for estimating GPRA actuals. Actual performance of the DFC coalitions on the
GPRA outcome measures is calculated by estimating the cumulative performance of coalitions on each
GPRA measure over baseline relative to the total number of coalitions for which performance could be
calculated. That is, only coalitions providing two or more years of performance data for the same grade
respondents are eligible to contribute to the estimate.
All calculations for GPRA performance measures follow the same basic logic: the number of programs
demonstrating success on a performance measure is divided by the total number of DFC funded
programs that provided a baseline estimate and some follow-up estimate for each measure. For the nonpast 30-day use items (Age Of Onset, Parental Disapproval or Perception of Risk) a coalition is counted
as successful if there is any improvement from baseline (i.e., a later measured outcome indicates
improvement in performance over baseline). For past 30-day substance use measures the same basic
logic is employed, except that it involves a calculation in addition to the logical rule. If past 30-day
substance use decreases by 5% or more in two or more grades, then a coalition is counted as successful.
As each coalition provides its own baseline, the 5% reduction necessary to be identified as successful is
calculated according to Formula 5, the value representing a 5% reduction in the proportion of users over
baseline for each substance and grade. A coalition is counted as successful when two or more grades
within the coalition provide subsequent past 30-day use proportions that are less than or equal to each
grade and substance’s referent criterion proportion.
Calculating each grade by substance 5% reduction criterion value
C A = p B * .95
Formula 5
Where CA is the criterion to be met or exceeded for success and pB is the baseline performance for each
grade and substance within a coalition.
The proportion of successful coalitions for each GPRA measure is calculated as the number of
successful coalitions divided by the total number of coalitions providing two or more waves of data (see
Formula 6).
‐ 31 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
Calculating the proportion of successful coalitions
S
pSC = C
TC
Formula 6
Where SC is the number of coalitions meeting or exceeding the performance criterion and TC is the total
number of coalitions providing two or more waves of data.
Exhibit 21. ONDCP’s FY2008 GPRA Goals and Objectives
GPRA Goals
Goal 1 – Improve Coalition
Effectiveness
Goal 2 – Strengthen technical
assistance to community coalitions
Primary Objectives
Enhance and strengthen infrastructure
Increase citizen participation in prevention efforts
Improve coalition capabilities
Increase intergovernmental and interagency collaboration in
coalitions
Ensure prevention efforts are more comprehensive and evidence‐
based and consistent with identified needs
Enhance prevention efforts
Strengthen coalitions in their prevention efforts to decrease risk
factors in the community
Strengthen coalitions in their prevention efforts to increase
protective factors
Strengthen coalitions in their prevention efforts to decrease
substance abuse indicators
Implement and assess strategies of the National Community Anti‐Drug
Coalition Institute
Implement and assess the efficacy of a mentoring coalition
demonstration program
6. Analysis of External Data to Enhance Substance Abuse Outcome Information
Several national surveys collect information about substance abuse outcomes of interest to ONDCP
(see Exhibit 17). Many DFC coalitions rely on these surveys for outcome measures that they then
provide via their Semi‐Annual Progress Report. Unfortunately, the public data sets associated with
these efforts do not contain community level data. Therefore, the evaluation team assessed the
feasibility of obtaining this community‐level information directly from the organizations that conduct
the national surveys in an effort to enhance the internal consistency and reliability of the substance
abuse outcome information. See Section A.16 of this report for a summary of findings from the
Feasibility Assessment. A detailed report on the Feasibility Assessment is included in Attachment 7.
Reporting results
During the five‐year period, the evaluation team will develop and submit a total of five annual reports
and one final report. In addition, upon request from ONDCP, ad hoc analyses and reports will be
prepared during the five‐year period of the National Evaluation.
A.17. Reason(s) Display of OMB Expiration Date is Inappropriate
No exemption from displaying the expiration date is requested.
‐ 32 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
A.18. Exceptions to Certification for Paperwork Reduction Act Submissions
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act
Submissions (see Attachment 8).
B. Collections of Information Employing Statistical Methods
B.1. Respondent Universe and Sampling
Respondents
DFC grantees. The respondent universe consists of all funded DFC coalitions beginning in FY2000
through FY2008 (n=769) with approximately 90 additional grantees per year being added throughout
the evaluation. Information will be used to estimate substance use rates among youth in DFC
communities and compare trends in these rates with trends in non‐DFC communities. Data for non‐
DFC communities will be abstracted from national‐ or state‐level surveys (i.e., without the need for
additional primary data collection or sampling). Unfortunately, community‐level information is not
available from the sponsors of these surveys. Even if it were, this information would not be expected
to provide sufficient longitudinal coverage for DFC‐funded communities as separate probability‐based
cross‐section samples are selected with each survey wave. However, a surrogate outcome measure of
past 30‐day substance use for non‐DFC communities will be extracted by assuming that the state
average can be considered to be a weighted average of the outcomes by population size of all
communities in the state and solving the equation for the unknown rates of substance use in non‐DFC
communities.
B.2. Procedures for the Collection of Information
The formal process of data collection began with a letter from the Deputy Director of the Office of
National Drug Control Policy to all DFC coalition directors in February, 2005, to provide information
about the evaluation. Specifically, this letter introduced the new web‐based COMET (referred to as
PMMS at the time this communication was made) interface; provided a password and instructions to
access the system; and outlined a timetable for submission of data. After receiving the initial letter,
coalitions were notified electronically and through the DFC listserv of dates to access the system to
submit data for the instruments. In all subsequent years, the Terms and Conditions that accompany
grantees’ Notice of Awards at the beginning of the grant award years have contained specific
information about the reporting requirements. This information is also posted on the Drug Free
Communities Program’s website.
• The DFC CCT, a data collection module in COMET based on the four‐stage coalition typology
developed for the National Evaluation and outlined in the framework introduced above.2 This
instrument is used to collect information on coalition composition/structure, characteristics,
2
The four stages are Establishing Coalition (I), Functioning Coalition (II), Maturing Coalition (III), and Sustaining Coalition
(IV).
‐ 33 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
capacity, functions, and activities. These data are being used to classify coalitions into different
stages of development for the evaluation. The typology can also be used to prioritize technical
assistance and training activities. The director or assistant director of the grantee organization
completes and submits the CCT information online to ONDCP annually (see Attachment 3a).
•
The Semi‐Annual Progress Report serves as a progress report for the DFC grantees. Intended
primarily as a management tool for SAMHSA Project Officers, the data elements include
information pertaining to progress, accomplishments, capacity, planning, implementation, and
self‐evaluation. The director, assistant director, or staff delegates of the grantee organization
submit the information online to SAMHSA semi‐annually (see Attachment 3b) through COMET,
although information can be entered into the system as often as desired.
Information obtained for progress reporting will be used by SAMHSA and ONDCP to:
o Assess grantees’ progress in meeting program objectives.
o Identify needs and target resources for technical assistance and training.
o Satisfy federal reporting requirements such as those mandated by the Government
Performance and Results Act.
The Semi‐Annual Progress Report has replaced the interim quarterly report previously required
of grantees and collects data for management and oversight of grantees by SAMHSA. The Semi‐
Annual Progress Report is less time‐consuming and more flexible for coalitions and more
effective as a management tool for SAMHSA.
Data quality is a particular concern on this project because the primary data for the evaluation are self‐
reported, and the DFC coalitions are responsible for identifying and reporting community‐level
outcome data. Thus, community‐level outcome measures may be affected by reporting and selection
biases. This issue is inherent in all self‐reported data. Data quality will be improved through the use of
vetted survey items and training and technical assistance in responding to survey items. In addition, at
each stage in data collection, data quality assessment (DQA) will be undertaken prior to conducting any
statistical analysis.
One focus of the DQA will be on the outcome measures, because they represent response variables for
the study. As noted, the outcome measures will be community‐level statistics obtained from surveys
that are conducted independently of this evaluation. Coalitions will be responsible for identifying the
appropriate data source, locating the data corresponding to the outcomes and strata requested in this
evaluation, and entering them accurately into the data collection instrument. This process may
potentially lead to data that are below the minimum data quality standards needed to conduct an
unbiased evaluation. Deficiencies may be indicated by, among other things, (1) significant amounts of
missing or invalid data, (2) evidence of inaccurate data, and (3) the use of unreliable methods by
coalitions for collecting outcome measures.
Evidence of potentially inaccurate data will be identified using a number of quality checks, including:
• Identical responses across multiple categories within the same strata and year.
‐ 34 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
•
Identical responses in large numbers of cells over time.
•
Discrepancies in sample sizes or estimated proportions across different levels of aggregation
(e.g., the total number of respondents summed across grade levels does not equal the total
number of respondents summed across gender; or the number of respondents changes
radically from year to year).
•
Performing a formal, statistically based outlier analysis for reported outcomes.
•
Establishing criteria that may be indicative of potentially invalid or inaccurate responses, such
as the reporting of 100% or 0% of youth for a particular outcome.
Coalitions are asked to report outcome measures, but are not mandated as to how they obtain the
requisite information. That is, each coalition may choose to employ a different survey technique to
obtain this information. Therefore, there is the potential that some coalitions may rely on techniques
that are known to be biased. As part of the information collected from coalitions, data on the
instrument used for collecting outcome measures will be requested. For example, coalitions will be
asked to indicate the source of their outcome data – state survey, established community survey, or
custom survey, for example.
Outcome measures using an established state or community survey are more likely to yield
scientifically valid and representative results for the community. Outcome measures collected using
other methods (e.g. use of custom surveys) are more likely to be biased or nonrepresentative, and
additional information will be sought from coalitions that report using these methods to evaluate the
validity of the reported outcomes. If grantees indicate the use of a custom survey, they must have the
survey reviewed by the evaluation team and approved by their Project Officer.
If data quality issues cannot be resolved by DFC Project Officers in conjunction with the coalition, data
from that coalition may be excluded from the statistical analyses, which will reduce the effective
sample sizes and resulting statistical power of hypothesis tests for the evaluation.
Data Management
Data will be routinely downloaded from COMET, sent to the evaluation contractor, and backed up to a
separate data system controlled by the evaluation team. In addition to regularly scheduled downloads,
data will be downloaded after submission of each Semi‐Annual Progress Report and each submission of
the CCT.
Data extracted from COMET will be compiled into a SQL Server database by the evaluation team, which
will be stored on a secure server without outside access. This SQL Server database will serve as the
main repository of data for the evaluation. As such, only permanent changes and corrections will be
made to this database. Data for specific analyses will be automatically extracted and manipulated for
statistical analysis using SAS®. Changes made to the data for a particular analysis will not be made in
the SQL data unless this change will be applied to every analysis. Otherwise, data changes and
modifications will be “soft‐coded” into the analysis SAS® code. Each coalition’s grantee ID will be used
as a unique identifier for all coalition records.
‐ 35 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
B.3. Methods to Maximize Response Rates and Deal with Nonresponse
A 100% response rate among the DFC coalitions is expected for both the CCT and the Semi‐Annual
Progress Report, since this is considered a reporting requirement of the grant award.
In addition, our anticipation of a high response rate among DFC coalitions is based on benefits
garnered by coalitions in using the COMET platform. COMET will allow coalitions to generate analytic
information to assess their own coalition’s performance and target efforts to improve sustainability.
For example, a critical function of the CCT will be to assist coalitions in identifying areas where
additional technical assistance and training may be required to further improve the performance of the
coalition.
Non‐respondents will be contacted by their project officers and encouraged to complete either the
semi‐annual report or the CCT. Incentives in the form of “Quick Facts” and other ad‐hoc analyses/
summaries of the collected information will be provided to participating coalitions. ONDCP’s partners,
particularly the Coalition Institute, will be asked to emphasize the benefits of understanding the stage‐
of‐development of a coalition and will begin to organize training and technical assistance around this
issue.
Non‐respondent grantees will be made aware that they face an escalating response from their
government project officer. There are three progressive discipline actions that can be taken that are in
accordance with DFC Statutes.
(1) High‐risk status;
(2) Suspension;
(3) Termination.
1. High‐Risk Process
When a DFC grant recipient is placed on high risk status, they are notified of additional requirements
that are being placed upon them and what actions are required of them to no longer be on High Risk.
They are also informed that failure to respond to the corrective action detailed in the high‐risk
notification letter may result in suspension or termination of the grant. The notification letter will
include the SAMHSA DFC appeals process for suspension and termination. The SAMHSA DFC Grant
Specialist and Project Officer provides guidance to the grant recipient in making corrections.
2. Suspension Process
Suspension is the second step in the progressive discipline process. Suspension is an action that
temporarily withdraws the agency’s financial assistance under an award, pending corrective action by
the recipient or pending a decision to terminate the award” (45 CFR 74.2).
‐ 36 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
3. Termination Process
The third step in the progressive discipline process is grant termination. Termination is “the
cancellation of awarding agency sponsorship, in whole or in part, under an agreement at any time prior
to the date of completion” (45 CFR 74.2).
B.4. Test of Procedures or Methods to be Undertaken
The core COMET platform is built around an existing system that has been successfully used by a
number of coalitions. Significant internal testing and review of COMET took place and continues to
occur on a daily basis as part of the software development process and as grantees use the system and
come across issues/improvement suggestions.
A stratified sample of 20 DFC coalitions was selected to participate in a pilot test of the CCT. Coalitions
were stratified based upon a subjective assessment of their approximate stage‐of‐development: (1)
Establishing, (2) Functioning, (3) Maturing, (4) Sustaining. A random sample of five coalitions was
selected in each stratum. Establishing and Functioning coalitions were only considered for selection
into the sample if they had previously reported data for two outcome measures. Maturing and
Sustaining coalitions were considered to be candidates for selection if they had reported data for all
outcome measures. Finally, although not part of the formal stratification, diversity was sought in the
geographic areas targeted by the coalitions in the sample to enhance representativeness of the
population of grantees. The selected coalitions included eight coalitions targeting urban areas, five
coalitions targeting suburban areas, and seven coalitions targeting rural areas.
All 20 DFC coalitions were contacted and asked to participate in the pilot test. Eight of the coalition
leaders contacted were willing and able to participate in the pilot test within the allotted time frame. A
ninth coalition leader volunteered for the pilot through the ONDCP Community Prevention listserv. The
coalition leaders were asked to review the CCT and present their feedback during a follow‐up
telephone call. Coalition leaders willing and able to participate in the pilot were scheduled for a follow‐
up telephone call and sent an electronic copy of the CCT as well as a copy of the discussion questions.
Follow‐up telephone conversations with the coalition leaders participating in the pilot were conducted
over a three‐day period. One coalition leader also provided handwritten comments on the CCT.
The follow‐up questions included:
1. What is your overall impression of the Coalition Classification Tool’s clarity and organization?
a. Does the description of coalition development on page 1 make sense to you? Why or
why not?
b. Do the coalition functional areas listed on page 1 make sense to you? Why or why not?
c. Do the General Rating questions make sense to you? Why or why not?
2. What is your overall impression of the Coalition Classification Tool?
3. How long did it take you to complete the Tool?
4. How do you think the Coalition Classification Tool can be useful for coalitions?
a. In what ways can the Coalition Classification Tool be made more useful?
‐ 37 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
b. How would you use the Coalition Classification Tool internally, within your coalition?
What kinds of reports/information would you want to know about your coalition?
5. What additional information about coalitions would you like to see included in the Coalition
Classification Tool?
6. Do you have any other comments about the Coalition Classification Tool?
Pilot participants’ feedback on the CCT was generally positive. The CCT’s overall clarity and
organization was described as “good,” “understandable,” “usable,” and “comprehensive.” Most
participants affirmed that the description of coalition development, the coalition functional areas, and
the general rating questions made sense to them. Regarding the CCT’s use, pilot participants felt it
would be a useful management tool for coalitions—“to gauge progress and set goals,” use it “as part of
our strategic plan,” and use it “as a needs assessment.” Suggested improvements included adding
better definitions for the terminology used in the CCT, using more consistent terminology throughout
the CCT, and adding examples of the CCT’s utility to coalitions on the first page so that respondents are
convinced immediately of the importance of the instrument. Pilot participants further suggested
creating a glossary of terms for easy reference while completing the CCT as well as a manual of
resources for coalitions interested in making needed improvements in areas of development. One
participant also recommended incorporating questions about sustainability and cultural competence
throughout the CCT.
The CCT was revised to incorporate the feedback of pilot participants. To clarify terminology associated
with the general rating questions, definitions of the rating scale response options (mastery, proficient,
and novice) and the term global rating were incorporated in the descriptive narrative of the CCT. The
CCT was reviewed and revised to ensure consistent use of terminology. An area for participant
comments was added to allow participants to explain their responses and provide feedback about their
experiences and concerns with the CCT. Efforts were also made to include the ideas of sustainability
and cultural competence in the first four sections of the CCT and to clarify the purpose of the tool by
adding additional narrative to the first page of the questionnaire. There were several question‐by‐
question comments and efforts were made to address commonly identified areas for improvement.
‐ 38 ‐
DRUG FREE COMMUNITIES SUPPORT PROGRAM NATIONAL EVALUATION
OMB CLEARANCE PACKAGE
B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
James Derzon
Battelle
2101 Wilson Blvd, Suite 800
Arlington, VA 22201
Phone: 703‐248‐1640
Fax: 703‐527‐5640
Email: [email protected]
Fred Dong
Battelle
1100 Dexter Avenue North, Suite 400
Seattle, WA 98109
Phone: 206‐528‐3120
Fax: 206‐528‐3550
Email: [email protected]
Jennifer Malson
Battelle
6115 Falls Road, Suite 200
Baltimore, MD 21204
Phone: 410‐372‐2724
Fax: 410‐377‐6802
Email: [email protected]
Jeanine Christian
Battelle
6115 Falls Road, Suite 200
Baltimore, MD 21204
Phone: 410‐372‐2751
Fax: 410‐377‐6802
Email: [email protected]
Michael Ponder, Ph.D., Consultant
Applied Social Research and Evaluation
11098 SW Lynnvale Drive
Portland, OR 97225
Phone: 503‐644‐8371
Fax: 503‐526‐9260
Email: [email protected]
Paul Florin, Ph.D., Professor
Community Research & Services Team
University of Rhode Island
Providence Campus, Room 236
80 Washington Street
Providence, RI 02903
Phone: 401‐277‐5302
Fax: 401‐277‐5486
Email: [email protected]
David Chavis, Principal Investigator
Association for the Development of Community
312 South Frederick Avenue
Gaithersburg, MD 20877
Phone: 301‐519‐0722
Fax: 301‐519‐0724
Email: [email protected]
Ben Pierce, M.S., Senior Analyst
Battelle
505 King Avenue
Columbus, Ohio 43201
Phone: 614‐424‐3905
Fax: 614‐458‐3905
Email: [email protected]
Warren Strauss, Sc.M., Task Leader
Battelle
505 King Avenue
Columbus, Ohio 43201
Phone: 614‐424‐4275
Fax: 614‐458‐4275
Email: [email protected]
Mary Kay Dugan
Battelle
1100 Dexter Avenue North, Suite 400
Seattle, WA 98109
Phone: 206‐528‐3142
Fax: 206‐528‐3550
Email: [email protected]
‐ 39 ‐
| File Type | application/pdf |
| File Title | Microsoft Word - FINAL OMB Package for ONDCP 9 23 2008.doc |
| Author | treecem |
| File Modified | 2008-09-23 |
| File Created | 2008-09-23 |