Systems Measures and Tools

As ONDCP's evaluation contractor for the Drug Free Communities Support Program (DFC) National Evaluation, ICF has been asked to determine appropriate systems, measures, and tools for the new five-year project period. Much of this assessment focused on the Coalition Online Management and Evaluation Tool (COMET), which is the major data collection system for the national evaluation. Grantees report data every six months in COMET to meet eligibility and programmatic requirements pertaining to their day-to-day activities and operating context. This system is also used to collect core outcomes on youth substance use every other year, as well as an annual survey of coalition operations called the Coalition Classification Tool (CCT).

The multipurpose nature of this online data system has led to confusion among DFC grantees on exactly what data is to be collected and for what purpose – and most significantly, it has posed substantial burden. As part of our assessment to determine the most appropriate measures, systems, and tools, ICF formulated a number of research questions to guide our efforts in enhancing the data system to better meet ONDCP and SAMHSA needs, improve the quality of evaluation data collected, and reduce burden for grantee communities. The guiding research questions are listed below.

Research Questions

To what extent is the present COMET system meeting the needs and expectations of DFC end-users?
To what extent is the present COMET system meeting the needs and expectations of ONDCP?
To what extent is the present COMET system meeting the needs and expectations of SAMHSA Project Officers?

To what extent does the new evaluation plan require changes in the data presently being collected by COMET?
What options exist for dealing with the distinct data needs of the DFC National Evaluation and SAMHSA’s grants management/reporting oversight?

5a. If a new data system were designed and developed to replace COMET, what are the opportunities for increasing its usability and decreasing the reporting burden of grantees?

5b. If a new data system were designed and developed, what are the opportunities for increasing its usability for ONDCP and for SAMHSA Project Officers?

5c. If a new data system were designed and developed, what are the opportunities for increasing its usability and efficiency for ONDCP's evaluation contractor?

Methods

ICF conducted the following evaluation activities, which are described below, to assess the usefulness and effectiveness of the COMET data system for ONDCP, SAMHSA personnel (i.e., project officers and managers), ONDCP personnel, and DFC grantees.

Drug Free Communities National Evaluation’s Stakeholder Meeting. In June 2010, ONDCP convened a group of stakeholders, which included ONDCP staff, SAMHSA project officers, Kit Solutions (the developer of the COMET system), HSRI (SAMHSA’s DITIC contractor), and the DFC national evaluation team to discuss potential changes to COMET. During this meeting, ICF presented findings from our initial review of COMET (and CCT), queried attendees on current strengths and challenges with the COMET system, and asked for guidance on how COMET could be improved as a programmatic tool and an evaluation tool in a manner that would reduce grantee burden.

Focus Groups. Focus groups were conducted separately for SAMHSA managers and SAMHSA project officers in June 2010. These focus groups were designed to obtain each stakeholder group’s perspectives regarding the current system and how it could be improved. Focus group questions, tailored to SAMHSA personnel’s specific roles in the study, asked participants to reflect on and discuss:^¹

Experience with the COMET system

Pros and cons of the COMET system

COMET as a tool to support and ensure needed information

Suggestions for improvements to the COMET system

Vision for the “ideal” data system

A total of twenty participants took part in two single-session focus groups. This included a session for thirteen project officers and another session that included seven managers. Participants represented DFC programs from urban, rural, and suburban locations nationwide.

DFC Grantee Feedback via a Social Media Website. To obtain real-time feedback from DFC grantees regarding their experiences with COMET, ICF established a social media website. Respondents were asked to provide information about their general experience(s) with COMET, both pro and con; suggestions for specific areas that could be improved; and what their vision would be, as a DFC grantee, for an “ideal” system. In total, 384 responses were received during the two-week period that the social media website was open.

Focus Groups at the CADCA Mid-Year Training Institute. Two focus groups with DFC grantees were held at the CADCA Mid-Year Training Institute in July 2010. One focus group was primarily devoted to understanding how grantees use the COMET system and what changes were needed. The other focus group was devoted to receiving grantee input on the new logic model and theory of change, which allowed the national evaluation team to understand what additional measures need to be included in the COMET system in order to accurately capture both processes and outcomes of DFC grantees.

Logic Model Workgroup Meetings. Based on extensive feedback during the initial TAG meeting, ICF formed a workgroup to inform the development of a new DFC National Evaluation Logic Model. In addition to the revision the National DFC Evaluation Logic Model (external model), workgroup members recommended the development of a more specified logic model (internal model). The development of the internal model was necessary to ensure that enhancements to the data system were aligned with the new evaluation plan. ICF’s evaluation team also comprehensively mapped the data source for every variable and corresponding item identified in the internal model. Finally, the National Evaluation team conducted a brief literature review documenting the empirical rationale for each of the major constructs and variables in the internal logic model.

COMET and CCT Review and Recommendations. The National Evaluation team systematically reviewed every item in both COMET and the CCT in order to recommend which items should be retained, which items should be retained but modified, which items should be added, and which items should be deleted to increase the collection of quality data and simultaneously reduce grantee burden. Based on feedback from the initial TAG meeting, and to align data collection with our external and internal logic models, the National Evaluation Team re-reviewed COMET and CCT data to make certain that we are collecting the appropriate measures to assess our hypothesized relationships regarding the importance of coalition structures, processes, strategies and activities, and how these are linked to differential coalition and community outputs and outcomes. During this systematic re-review of items within COMET and CCT, content analysis was conducted on sections within COMET that (1) either already had dropdown menus, or (2) could benefit by the inclusion of dropdown menus.

Question One: "To what extent is the present COMET system meeting the needs and expectations of DFC end-users?"

Although a formal DFC user survey of COMET usage and focus groups are desirable, they require OMB approval in advance, which takes several months to accomplish. In order to collect data in real-time and inform both refinements to COMET and the evaluation design, ICF capitalized on a recent OMB decision to allow the the use of social media technologies for "general solicitations" that pose a series of specific questions designed to elicit relevant public feedback. Specifically, "agencies may offer the public opportunities to provide general comments on discussion topics through . . . social media websites, blogs, microblogs, audio, photo, or video sharing websites, or online message boards (whether hosted on a .gov domain or by a third-party provider)."

In response, ICF developed a social media site to which DFC grantees were directed to provide feedback on the current data system. As mentioned, this feedback was purposely aligned with feedback obtained via focus groups with SAMHSA managers and project officers, in order to obtain multiple perspectives from those who directly or indirectly interface with the COMET system. This includes the pros and cons of the current COMET systems, suggested refinements and improvements, and what would be included in an “ideal” on-line data system. Below is a listing of major themes identified by DFC grantees, including specific quotes from the social media website.

Using the Strategic Prevention Framework (SPF) to Organize Data Entry – Familiar & Reinforces Coalition Work but not User-Friendly

COMET requires grantees to enter data according to the SPF framework and its five major steps: assessment, capacity, planning, implementation, and evaluation. Grantees’ reflections on their experiences using this framework to guide data entry resulted in the identification of a number of pros and cons.

Familiarity and Reinforcement of DFC Coalition Framework/Model. Many of the participants, who have extensive experience and exposure to the SPF, liked the fact that the COMET system is organized using this framework/model because familiarity and experience with SPF allowed for an easier understanding of COMET.

I like how the COMET system follows the SPF/SIG model. It really is the ONLY reporting tool that is established for coalition/prevention work. It’s much better than many other systems I have to use, which mainly count numbers, is based on classes or treatment. So, I like the logic model it follows. [Grantee]

I like that the system [COMET] has categories based on the SPF. I like that the reporting system is cumulative since it helps us be able to see in one report the progression of our work. [Grantee]

Furthermore, this data system reinforces the importance of the SPF not only in meeting program requirements, but also in reducing youth substance use through evidence-based structures, processes, and practices. It also provides grantees an opportunity to reflect on the complex work coalitions undertake to plan and implement community-based prevention and intervention activities.

Keeps us organized and places a data collection, check and balances system with our coalition and its members. Bi-annually the entire aspect of the coalition is reviewed and discussed for the purpose of the data entry process and follow-up, in advance of our annual strategic planning meetings in May. Reminds me how thankful I am for the laptop and technology affiliated with this grant, because this data collection tool while challenging, does stir thought into planning, reporting, and marketing the information to the community it serves. The technology eases the complexity of producing the outcome reports. [Grantee]

Not User-Friendly. However, while this familiarity and reinforcement of the major steps involved in coalition efforts to reduce youth drug use are beneficial, the utilization of this framework for data entry entails much repetition, redundancy, and is not user-friendly. In particular, the direct linking of goals to objectives and then to specific activities and risk and protective factors leads to substantial burden, and often, confusion. Grant communities also reported that some of the same strategies and activities address multiple objectives, leading to considerable duplication of effort. This is particularly the case since grant communities are already required to link goals, objectives, strategies, and activities in their strategic and action plans which are uploaded into COMET; thus, this format essentially duplicates this effort.

[COMET] is repetitive—some of our strategies are linked to multiple objectives and we find ourselves entering duplicate information into multiple places throughout the reporting system, this is very time consuming and frustrating. [Grantee]

Don’t like the repetitive process. The worst part for me is where we place our activities under each goal. There has to be an easier way to do this. I feel like I do not do that section justice because of the repetitive and difficult process to insert each activity. [Grantee]

It is not user friendly—a few examples of this: Items are not organized or alphabetized (such as coalition lists); items are difficult to revise (IF we are even allowed to do so, such risk and protective factors, that have changes from the first 5 years to the next 5 years); … Having to enter “Active; Inactive; Planned; Complete; Discontinued” each time requires us to go back to things previously entered as well as update what needs to be entered. [Grantee]

The largest threat posed by this structure is the duplicate entry of activities to fulfill multiple SPF objectives. This not only increases response burden; it also results in the over-counting of certain activities which does not provide an accurate representation of DFC grantees’ efforts. This situation compromises the DFC evaluation team’s ability to establish causal linkages between processes and outcomes.

Substantial Burden

While certainly related to the redundancy issues noted above, burden itself emerged as a major theme from social media entries. Also, according to technical advisory group meetings, this has been one of the major issues noted since the inception of COMET.

It takes me about 4 days to complete this report, and the only way that I am able to do so is to first create a Microsoft Word document that follows COMET’s structure, and figure out how I am going to put our coalition’s accomplishments into the different parts. (And when I do a new report, I can easily see what it was that I reported the last time.) I think my reports are reasonably good because I do it this way. I would go insane if I tried to just plug our results in “live,” going through COMET screen by screen. [Grantee]

It takes me a good 2 weeks to enter all the information and activities our coalition participates in within a reporting period. It would be nice if we could enter the activity in once and have radio buttons or a drop down box where we could select the various categories the activity falls into (implementation, evaluation, etc.). That would save me a lot of time. [Grantee]

Utilization of Dropdown Categories. The utilization of dropdown menus for grantee data collection are often commonplace as this substantially reduces burden and also provides aggregated data that is helpful in evaluation activities. Grantee submissions to the social media website, similar to comments regarding the SPF framework, included both the benefits and costs of this type of data entry. Interestingly, one of the main con themes centered on the appropriateness or relevance of the current dropdown categories.

The drop-down menus are so limiting and counter-intuitive; I finally did a print-screen of every drop down menu to try and figure it all out! I don't mind using someone else's verbiage to describe what we are doing, but I need to see the bigger picture to figure out how to make an imposed system link up with our actual work. [Grantee]

In summary, the social media website proved to be an effective way to reach out to DFC grant communities and learn in real-time their likes and dislikes of COMET, as well as their suggestions for improvement. ICF learned a great deal from grantees during the two week timeframe the social media website was open for submission. In fact, the sheer number of submissions, along with the topics of their submissions, indicates that grantees have long struggled with this system and desire major enhancements, if not a complete re-design. While a complete re-design is likely not possible in the short-term, ICF believes a number of enhancements can be implemented to improve the quality of both programmatic and evaluation data collected through the system.

Question Two: "To what extent is the present COMET system meeting the needs and expectations of ONDCP?"

ICF has engaged in extensive discussions with our ONDCP contacts about their experiences with COMET. ICF may conduct a structured group interview with ONDCP staff that have had direct experience with DFC use of COMET, or with ONDCP use of COMET data. Discussion questions would include (1) what does ONDCP need and want from a DFC data collection and reporting system, and (2) how has COMET met or failed to meet these needs?

Question Three: "To what extent is the present COMET system meeting the needs and expectations of SAMHSA Project Officers and SAMHSA managers?"

ICF conducted focus groups with SAMHSA personnel regarding their experiences with the current COMET data system. Given their different roles and responsibilities, ICF conducted separate focus groups for SAMHSA managers and for SAMHSA project officers. As highlighted earlier, questions focused on pros and cons of the current system, ability of COMET to be used to collect and access needed information, suggestions for refinement and improvement, and what elements would constitute their ideal system.

It should first be noted that many of the same themes reported by grantees were also reported by SAMHSA staff. For instance, they recognized both the costs and benefits of utilizing the SPF framework and dropdown categories for grantees to select from to reduce burden. They also acknowledged and verbalized the extensive time-consuming nature of the current COMET system. This section focuses on the extent to which COMET is meeting the needs and expectations of SAMHSA project officers and managers. In terms of what information SAMHSA most needs from grantees and how COMET can better support these needs, three major themes emerged.

Compliance and Technical Data Needs

Both project officers and managers reported very specific needs for compliance and program data, which included core outcome data, compliance data (e.g., sector information and use of environmental strategies), and appropriate contact information.

The compliance information is very important; program evaluation and the program evaluation encompasses all of that information about what are they doing, what is their focus, what activities are they doing—that implementation information that we need. We also need organizational information and we need impact information...we need to get the technical needs information, but we also need to get information that allows us to identify issues that are happening in communities. [Manager]

[W]e have to have the core measures data every other year so we definitely check for that…We need the 12 sector data and then we need to make sure they have environmental strategies. Those are the three key monitoring pieces from the terms and conditions perspective that we need. [Project Officer]

A lot of times the grantees, the coalition members come and go, so to have an accurate picture of who the 12 active representatives are, those they have MOUs [memorandums of understanding] with, and also any other members that are representing those sectors. So whenever they report, they can accurately report who is a part of the coalition and who’s not. [Project Officer]

Contextual Data – Grantee Activities, Accomplishments, Challenges, & Needs

Project officers and managers reported that, many times, they find the most useful information in the narrative text. For instance, selection of an environmental strategy provides little no information as to whether it was useful, how it was used or implemented, or whether it was adapted to achieve success. A reality of DFC work is that each community is unique and communities are encouraged to tailor proven strategies and activities to their local context. Project officers and managers acknowledged that they would like to have more contextual information to understand the day-to-day work in each of their grant communities.

The other pieces I need are the details when we ask them to write descriptions, when we ask them to describe their activities, those descriptions are the key pieces for me to understanding what they’re doing. [Project Officer]

One of the things we don’t have that’s critical to the day-to-day understanding on the part of the project officer is narrative information, simple and brief as it can be, but narrative stuff, and I bulleted some things. What’s the general progress per the action plan? The action plan is the heart and soul of the grant so are you making—can you tell us in two paragraphs what are you doing generally? [Manager]

Being able to query for all the information needs we have. Demographics, everything. [Manager]

While both project officers and managers reflected this need for more contextual and idiosyncratic information, there was a detectable difference. Project officers seemed to want to know more about the here and now while managers wanted better organization of COMET to capture the developmental nature of progress over the five-year grant period in terms of strategies employed, activities, and successes and challenges encountered. As stated by one participant, one way to meet both of their needs, and increase user-friendliness for grantees, was the potential to be able to archive information from prior report periods.

We’ve not had an opportunity to get the type of information, the developmental progress that the grantees are making over a 5-year period. One of the things that we do within the program is we focus on grantees in years 1 through 3, and we put a lot of our attention to those. But we are not getting direct feedback from the data that lets us know how successful we’re being, what types of strategies we need to implement to help the grantees be more successful.[Manager]

What I want is an understanding of what happened during the last 6 months. I want to understand that and I want it to be clear, I want it to be concise, and I want it to be in such a format that I can pick up the phone and talk to somebody that put it together and can stand by that because when I go on a site visit that’s what I do, I take the COMET with me and I sit down with them and I spend unfortunate amounts of time trying to sort through to get them to actually tell me what happened. [Project Officer]

I would really like to have COMET be that one place to have a repository of a lot of different documents that we would have handy because as the government goes greener and greener, we’re getting more and more away from the paper files. …and I’d like to see us be able to look at a snapshot of where is the project officer with their portfolio in site visits, in reviewing the progress reports, in a number of different categories, and then also aggregated by teams and branches so that you’ve got a good picture of what’s going on and where people are in the progress of things, whatever the task might be. [Manager]

Need for more Innovation, Integration, & Interaction

Given the multipurpose nature of the COMET data system, future refinements and improvements need to focus on facilitating interaction among users (grantees, project officers, managers, ONDCP), integrating the various data elements where appropriate, and applying innovative methodologies to make the data not only accessible, but facilitate real-time analyses that will help inform grantee strategies and activities. This could include implementing some canned queries that provide information in real-time for SAMHSA and ONDCP staff, including how communities compare to each other on national and/or regional averages for core outcomes, coalition functioning scores derived from the CCT, as well as strategies and activities employed (e.g., numbers, hours of contact with target population) to reduce substance use and the associated challenges and accomplishments of such approaches.

We don’t have the capability of pulling that data up and doing a very simple report or doing some analysis that will allow us to structure or paint a picture of what are the issues out there that communities are focusing on. We also don’t have the ability to summarize where the grantees are having problems at, and then structure a strategy or some planning to help them. If grantees are having, if the majority of grantees are having issues in their capacity development phase of the grant, we can’t pull that information … We have no abilities like that. So we’re a bit at a disadvantage in terms of being effective to even provide support and advice to grantees. [Manager]

If we could just get this system to create a contact person and a name of a coalition and their telephone number and their email address, we would be 20 years ahead of where she said we were in 1993. We’d be back to where we were in 1993. That’s sort of how bad it is. [Manager]

[T]he key pieces of the trend information is important if we could see the graphs. However, they report every 6 months, a lot of times the data information is only reported every 2 years. So you’re not going to see a trend from 6 months to 6 months until they hit that second-year evaluation review, and that’s where probably looking at the strategies to try and find out—or other assessment tools—to find out if they’re actually moving the needle, is going to be important. [Project Officer]

In summary, a balanced approach needs to be employed in order to obtain the technical, programmatic, contextual, and evaluation information that ONDCP and SAMHSA need, while also providing beneficial evaluation data to grantees. Similarly, COMET can be a more supportive system to the data collection process by being more innovative, integrated, and interactive. Such modifications would ensure that the system is both responsive grantees’ concerns and accurately captures and reflects their experiences. Specific changes and enhancements to the current COMET data system (and CCT and core measures), informed by focus group data, social media website data, the Technical Advisory Group, and our own internal review will be highlighted later in the report.

Question Four: "To what extent does the new DFC National Evaluation plan require changes in the data presently being collected by COMET?"

To answer this question, ICF formed a workgroup to review and revise the DFC National Evaluation Logic Model (external model), created a more specified internal logic model linking theoretical constructs to COMET and CCT items, and reviewed the literature to ensure empirical support for all processes, structures, and variables incorporated into the national evaluation. Finally, ICF conducted a qualitative review of historical data to identify more appropriate dropdown categories. This activity was conducted in direct response to grantee feedback and was intended to result in reduced grantee burden and increased quality of data collected; specifically, the dropdowns were focused on accurately capturing the amount and type(s) of strategies and activities employed to address the two main goals of DFC, which include (1) to establish and strengthen collaboration, and (2) to reduce substance use among youth and adults.

To address the fourth guiding research question, we will first discuss what activities and outputs were created by the Logic Model Workgroup. This includes a focus on the National DFC Evaluation Logic Model (external model), with supporting information provided through the development of a more specified internal logic model. This will be followed by a brief overview of the COMET, core measures, and CCT reviews, including overall or major recommendations for system enhancements along with some specifics of possible enhancements and changes by data source (e.g., CCT). Please see Appendix A for specified indicators of each construct and a brief literature review.

DFC National Evaluation Logic Model

At its first meeting, the DFC National Evaluation Technical Advisory Group (TAG) identified the need for revision of the “legacy” logic model prepared by the previous evaluator. A Logic Model Workgroup was established and charged with producing a revised model that provides a concise depiction of the coalition characteristics and outcomes that will be measured and tested in the national evaluation. The TAG directed the Workgroup to develop a model that communicates well with grantees, and provides a context for explicating evaluation procedures and purposes.

The Workgroup held its first meeting by telephone conference in July 2010. In the following two months, the committee (1) developed a draft model, (2) reviewed literature and other documents, (3) mapped model elements against proposed national evaluation data, (4) obtained feedback from grantees through focus groups at the CADCA mid-year training in Phoenix, (5) developed and revised several iterations of the model, and (6) produced the recommended logic model shown in Figure 1. The National Evaluation logic model has six major features, described below, that define the broad coalition intent, capacity, and rationale that will be described and analyzed in the National Evaluation.

Theory of Change. The DFC National Evaluation Logic Model begins with a broad theory of change that focuses the evaluation on clarifying those capacities that define well functioning coalitions. This theory of change is intended to provide a shared vision of the overarching questions the National Evaluation will address, and the kinds of lessons it will produce.

Community Context & History. The ability to understand and build on particular community needs and capacities is fundamental to the effectiveness of community coalitions. The National Evaluation will assess the influence of context in identifying problems and objectives, building capacity, selecting and implementing interventions, and achieving success.

Coalition Structure & Processes. Existing research and practice highlights the importance of coalition structures and processes for building and maintaining organizational capacity. The National Evaluation will describe and test variation in DFC coalition structures and processes, and how these influence capacity to achieve outcomes. The logic model specifies three categories of structure and process for inclusion in evaluation description and analysis:

Member Capacity. Coalition members include both organizations and individuals. Selecting and supporting individual and organizational competencies are central issues in building capacity. The National Evaluation will identify how coalitions support and maintain specific competencies, and which competencies contribute most to capacity in the experience of DFC coalitions.

Coalition Structure. Coalitions differ in organizational structures such as degree of emphasis on sectoral agency or grassroots membership, leadership and committee structures, and formalization. The logic model guides identification of major structural differences or typologies in DFC coalitions, and assessment of their differential contributions to capacity and effectiveness.

Coalition Processes. Existing research and practice has placed significant attention on the importance of procedures for developing coalition capacity (e.g., implementation of SAMHSA’s Strategic Prevention Framework). Identifying how coalitions differ in these processes, and how that affects capacity, effectiveness, and sustainability is important to understanding how to strengthen coalition functioning.

Coalition Strategies & Activities. One of the strengths of coalitions is that they can focus on mobilizing multiple community sectors for comprehensive strategies aimed at community-wide change. The logic model identifies the role of the National Evaluation in describing and assessing different types and mixes of strategies and activities across coalitions. As depicted in the model, this evaluation task will include at least the following categories of strategies and activities.

Information & Support. Coalition efforts to educate the community, build awareness, and strengthen support are a foundation for action. Identifying how coalitions do this, and the degree to which different approaches are successful, is an important evaluation activity.

Enhancing Skills. This includes activities such as workshops and other programs (mentoring programs, conflict management training, programs to improve communication and decision making) designed to develop skills and competencies among youth, parents, teachers, and/or families to prevent substance use.

Policies / Environmental Change. Environmental change strategies include policies designed to reduce access; increase enforcement of laws; change physical design to reduce risk or enhance protection; mobilize neighborhoods and parents to change social norms and practices concerning substance use; and support policies that promote opportunities and access for positive youth activity and support. Understanding the different emphases coalitions adopt, and the ways in which they impact community conditions and outcomes, are important to understanding coalition success.

Programs & Services. Coalitions also may promote and support programs and services that help community members strengthen families through improved parenting; that provide increased opportunity and access to protective experiences for youth; and that strengthen community capacity to meet the needs of youth at high risk for substance use and related consequences.

Community & Population-Level Outcomes. The ultimate goals of DFC coalitions are to reduce population-level rates of substance use in the community, particularly among youth; to reduce related consequences; and to improve community health and well-being. The National Evaluation logic model represents the intended outcomes of coalitions in two major clusters: (1) core measures data, which are gathered by local coalitions, and (2) archival data (UCR, FARS), which synthesized by the National Evaluation team. These data will be utilized to assess the impact of DFC activities on the community environment and substance use and related behaviors.

Community Environment. Coalition strategies often focus on changing local community conditions that needs assessment and community knowledge identify as root causes of community substance use and related consequences. These community conditions may include population awareness, norms and attitudes; system capacity and policies; or the presence of sustainable opportunities and accomplishments that protect against substance use and other negative behaviors.

Behavioral Consequences. Coalition strategies are also intended to change population-level indicators of behavior, and substance use prevalence in particular. Coalition strategies are also expected to produce improvements in educational involvement and attainment; improvements in health and well-being; improvements in social consequences related to substance use; and reductions in criminal activity associated with substance use.

Line Logic. The National Evaluation Logic Model includes arrows representing the anticipated sequence of influence in the model. If changes occur in an indicator before the arrow, the model represents that this will influence change in the model component after the arrow. For the National Evaluation Logic Model, the arrows represent expected relations to be tested and understood. How strong is the influence? Under what conditions does it occur?

In summary, the National Evaluation Logic Model is intended to summarize the coalition characteristics that will be measured and assessed by the National Evaluation team. The model depicts characteristics of coalitions that will be described as they present themselves, not prescriptive recommendations for assessing coalition performance. This model is intended to guide an evaluation process through which we can learn from the grounded experience of the DFC coalitions who know their communities best. The model uses past research and coalition experience to provide focus on those coalition characteristics that we believe are important to well functioning and successful coalitions. The data we gather will tell us how actual community coalitions implement these characteristics, what works for them, and under what conditions. In this sense the model is an evolving tool -- building on the past to improve learning from the present and to create evidence-based lessons for coalitions in the future.

Figure 1. DFC National Evaluation Logic Model

Theory of Change: Well functioning community coalitions can stage and sustain a comprehensive set of interventions that mitigate the local conditions that make substance abuse more likely.

For more detail, please see Appendix A which contains the National Evaluation’s internal logic model, a complete listing of all variables and items in COMET, CCT, and Core measures data and how they map to the internal model, and a brief literature review that provides empirical support and the National Evaluation rationale for inclusion of new variables.

COMET, Core Measures, and CCT Reviews and Recommendations

To fully address how the new DFC National Evaluation plan requires changes in current data collection efforts (Research Question #4), in addition to the extensive work conducted by our TAG, an intensive review was conducted by ICF for each data collection system and source. This included an initial review to bring recommendations for discussion at the initial TAG, as well as an intensive re-review after obtaining feedback on the current system from ONDCP, SAMHSA project officers and managers, and grantees. Due to length and specificity of material, this information is laid out in a number of appendices. However, below we present a top line summary of potential major enhancements that could be made to data systems and sources that would increase usability and decrease burden. This is followed by an overview of our intensive review of COMET, core measures data, and the CCT. For detailed results, please see respective appendices:

Appendix B includes an item-by-item review of COMET, along with recommendations on which items to keep, modify, or delete. We also provide potential items to include that are pertinent to understanding coalition effectiveness as represented in both our external and internal logic models.

Appendix C includes a brief description of our content analysis of COMET data, which was conducted to identify where dropdowns could be modified or included to reduce response burden.

Appendix D presents our review of core measures data, which includes a survey template for collecting core measures data that are NOMS compliant.

Appendix E presents an item-by-item review of the CCT, along with recommendations of which items to keep, modify, delete, or add.

Major Recommendations for System Enhancements

Overall, there are a number of major recommendations that could be implemented to improve both process and outcome evaluation data, while also reducing burden. First, based on all research activities conducted since the first TAG meeting, ICF recommends continuing to utilize the SPF as an organizing framework for COMET data collection, with some clarifications such as defining each SPF element (e.g., assessment) and listing what constitutes assessment activities, challenges, and accomplishments. This recommendation for clarifications is based on the fact that our qualitative review discovered many grantees were inputting incorrect information for each SPF element. For example, implementation activities were being listed in planning and vice versa; there was also little understanding of the difference between assessment and evaluation. Second, the explicit linking of objectives to strategies, activities, core outcomes, targeted substance, grade, and gender creates considerable burden, with relatively little benefit, particularly for the National Evaluation. We suggest de-linking this extensive chain to substantially reduce burden and increase quantity and quality of usable data. Third, rather than focusing on how all activities are linked to objectives, we recommend focusing on dosage and information related to type(s) of strategies, and activities. This is the information needed to assess how coalition processes (strategies, activities, dosage, penetration) impact the targeted community and can be linked to changes in outcomes. Fourth, ICF believes there needs to be a greater understanding of quality of collaboration and how it is associated with higher functioning coalitions that produce more positive community and systems changes. To attain this goal, we have recommended the inclusion of new variables (e.g., interorganizational coordination, leadership, organizational impact) and innovative methodologies, such as a social network analysis within our case study sites. While these are the main recommendations regarding potential enhancements, please see appendices for specific details by data source (i.e., COMET, Core Measures, CCT). For instance, we recommend changing the layout of the CCT where Likert-type items are listed together, instead of by section, which will allow for a much faster completion of the survey even with addition of new variables.

COMET: Review of Data Collection Items and Burden

COMET collects information about the five steps in SAMHSA’s Strategic Prevention Framework (SPF): assessment, capacity, planning, implementation, and evaluation. COMET also covers general information about coalitions, such as membership, leadership, and catchment areas. In the matrix presented in Appendix B, we propose several changes to enhance the utility of the system for evaluation purposes. This matrix represents our thinking about how COMET can be enhanced to reduce burden and also obtain better process, output, and outcome data.

COMET imposes a large amount of data entry burden on respondents primarily for three reasons. First, as previously highlighted, the explicit linking and specificity for every activity is time-consuming. Based on feedback from grantees, this requires extensive burden and repetition while providing little programmatic information that is not captured in strategic and action plans. We also want to focus more on dosage and present an enhancement that could obtain these data in the planning and implementation sections (see Appendix B for details). Second, there are many “enter all” format questions. DFC coalition staff, for example, are asked to enter information about all events they hosted during each of the reporting periods. The “enter all” questions include questions on risk and protective factors, as well as on achievements and challenge/barriers. To minimize respondents’ burden, we propose to replace this format of inquiry with a standard survey check list format with four or five response categories. For example, we recommend having respondents rate the degree to which a dropdown was a (1) challenge to (5) success. This will provide the evaluation team with much more meaningful data to analyze as “check all” questions are notoriously unhelpful in evaluations (beyond simple descriptives, such as X% of coalitions checked that financial resources was a challenge). Third, many COMET questions are open-ended, which provides rich detail, but it also involves substantial response burden and extensive post-data collection coding of responses which does not facilitate timely analyses. We propose that, based on COMET’s historical data and policy relevance, we should derive a broad classification system of response choices. For example, when asking DFC coalition staff what assessment activities they conducted, they should be given a list of broad activity types (see appendix C), so respondents can choose what category best fits their activity in place of providing text and an open-ended description. This will provide more standardized data that can be aggregated and used in analyses to understand different activities, accomplishments, and challenges and how these are related to coalition and community outputs and impacts. However, based on focus group and social media website data, we also understand that rich detail is important to capture data that cannot be categorized easily and to provide SAMHSA project officers information they need to effectively monitor coalitions’ progress. We will therefore provide an open-ended text field for respondents to provide further description for project officers (for each section of the SPF). We will also provide “other” fields in case coalition activities are not accurately captured by any of the dropdowns that are based on historical data from grantees. In this manner, we can obtain more quantitative and useful data for evaluation purposes but also allow for the rich narrative that these text fields provide.

In Appendix B, we present tables that follow the COMET’s SPF structure, including sections on assessment, capacity, planning, implementation, and evaluation. For each item, the matrix includes its field name/category, explicit description of data, and our rationale for proposed changes. Ultimately, we believe that these suggested changes will significantly reduce reporting burden, which will in turn improve the quality of the data being submitted by grantees. Specifically, the columns are organized as follows:

Column A – Field name/category
Column B – Data type (text, yes/no, Likert scale, etc.)
Column C – Description of every data element presently collected by COMET
Column D – Indicates whether question should be dropped, kept, modified, or whether a new question is suggested
Column E – Rationale for keeping/adding/dropping data elements.

Core Measures Survey

Appendix D presents a standard survey created by ICF that can be used to effectively gather data on the current four core measures, as well as additional proposed core substances – including as prescription drugs.^²

At last count, DFC grantees are using more than 135 different surveys to report their core measures. While these surveys are subject to detailed review and approval by the National Evaluation team, there are nonetheless differences in how data are being collected and reported across sites. While the universal use of a standardized survey is likely not feasible at the present time, we would nonetheless like to standardize data collection among as many DFC grantees as possible. Given the challenges that DFC grantees have faced in standardizing their survey data, this simple one-page (double sided) survey can simplify data collection efforts and can ensure that standard data collection metrics are being used by as many grantees as possible.

In his FY 2010 Budget Request, President Obama recommended eliminating the State Grants portion of the Safe and Drug Free Schools and Communities (SDFSC) program. A number of DFC grantees used funds from this grants program to collect and report their core measures data. With the elimination of Safe and Drug Free Schools funding, many grantees will need to retool their core measures data collection, and this survey will ease that transition.

The movement to the NOMS represents several major changes in the core measures:

30-Day Use: Although the NOMS capture the number of days each substance was used in the past 30 days, we have received assurance from SAMHSA that a simple yes/no framework to estimate community prevalence rates is NOMS compliant.
Perception of Disapproval: The NOMS focuses on friends’ disapproval of substance use instead of parental disapproval. Because parental disapproval rates are a strong predictor of outcomes, we will also keep this metric. By including both parental and friends’ disapproval, we can simultaneously study what are arguably the two biggest influences on a youth’s decision to use ATOD.
Perception of Harm: The NOMS question on perception of harm from alcohol focuses on binge drinking (drinking five or more drinks once or twice a week). The current core measure for alcohol asks about regular use of alcohol (one or two drinks per day for five or more days per week), which has been publicized recently as being good for one’s health.^³ STOP Act grantees will continue to report on regular use; all other DFC grantees will transition to the measure of binge drinking.

Age of first use will be dropped because it does not produce reliable estimates. Although these changes in core measures will reduce burden for SPF/SIG grantees and other grantees using state surveys that are already NOMS compliant, there may be substantial burden in the transition process to these new core measures. The GPRA measures for STOP Act will continue to be based on the current core measures as they apply to alcohol (30-day prevalence, parental disapproval, and perception of risk from regular drinking).

CCT: Review of Data Collection Items and Burden

The first two objectives of our National Evaluation plan focus on (1) strengthening the measurement of process data and (2) developing new metrics for coalition operations. In order to meet these objectives, in addition to analyzing COMET, a thorough review of CCT data is required. Appendix E provides a detailed listing of variables and items that comprise the CCT, as well as a listing of the variables and items that could potentially be added to the instrument. This appendix lists our recommendations for how possible adjustments and enhancements could be made to the CCT, which would provide the National Evaluation team and program staff with higher quality process data on day-to-day coalition activities.

As part of our work to validate the CCT, the National Evaluation team examined the reliability of existing CCT scales and individual item correlations. In general, we found that CCT scales had acceptable reliability levels and most items were sufficiently correlated with their respective scales. Moreover, an initial review of the CCT found that there are currently one hundred and forty-three items for the coalition leader to answer regarding their coalition’s functioning and dynamics.^⁴ Twenty-four of these items comprise the typology assessment (e.g., establishing, functioning, maturing, sustaining), and we consider these items to be central to our efforts to track coalition development across time and to preserve meaningful historical data. We have included these 24 items as core questions in the instrument. However, adjustments could be made to other variables and items within the existing CCT, and these adjustments would provide us with an opportunity to study coalition processes in greater depth and with the ability to link quality processes with improved outcomes. Typically, only one member per coalition fills out the CCT (a couple of members may fill out parts of the same survey depending on their roles). However, generally peer reviewed journals frown upon a single respondent for a group or organization as this can lead to substantial bias, particularly if the leader fills out the survey. Data can be further questioned since it is not anonymous and will be reviewed by Federal Project Officers. While there is no exact criterion, usually coalition research uses data when at least fifty percent of its members participate in the survey. Since there are over seven hundred DFC coalitions, requiring participation by a majority of coalition participants is not feasible and would substantially increase burden. Nevertheless, we recommend collecting data from more than one individual to understand coalition functioning beyond the leader’s perspective. We plan to work with ONDCP to arrive at a reasonable estimate that is defendable in the peer reviewed literature. There are a number of options, including (1) requiring at least three coalition members to fill out surveys, two of whom must not be the leader of the coalition, or (2) targeting a subset of coalitions (stratified by region, size, and urbanicity) and attempting to collect data from all members. For our case study sites, we plan to collect CCT data from 100% of participants and include a separate section for social network analysis questions.

While additions to the CCT may lengthen the current instrument, there are a number of improvements in formatting and programming that could be made to reduce the burden on grantees. Currently, the CCT is laid out according to the four overarching functional areas that inform the typology rating, with a fifth section on synergy and collective self-efficacy. This layout provides a good conceptual framework but considerably slows down respondents by going back and forth between typology assessment ratings, which are the most complex and time-consuming, to Likert-type questions and related items. Survey response time could be substantially improved by placing all the typology questions together and then arranging all Likert-type items together as much as possible, given response scales and occasional yes/no or checklist items. In this manner, adding additional Likert-type items does not heavily impact response burden, since these items can be grouped together and quickly completed by respondents. Items from different scales can also be interspersed, reducing some of the potential bias of respondents, as opposed to the current format, in which all items for specific variables are listed together.

Additional Variables and Items and Summary of Deleted Items

A thorough review of the literature was conducted to identify potential variables that could be added to capture data on indicators of the quality of collaboration and associated outputs and outcomes. To provide some context, some of the additional variables target the quality of coalition processes (e.g., leadership), outputs and outcomes (e.g., interorganizational coordination, perceived effectiveness), and local community context (e.g., community readiness for change, community social organization). While we recommended that a number of items should be deleted, some items are retained so that historical comparisons can be made with past data sets. However, it should be noted that little analysis has been conducted on CCT variables outside of the 24 items that comprise the typology assessment. While we retained the 24 core items from the CCT used in the classification system, there were more than one hundred other items that were not used for such purposes. Some of the general items/scales recommended for deletion include items around conflict management, sustainability, and collective self-efficacy. Based on recommendations from ONDCP, this pool of potential items and scales can be expanded or reduced.

Coalition Classification Tool Data Dictionary Description

All variables and items in the CCT are listed in Appendix E. From left to right, columns indicate the variable name (e.g., leadership), data type (e.g., Likert scale, checklist), a brief description of introductory text and item stem (along with the exact item content), a recommendation for keeping, deleting, or adding in a new variable, reliability information (i.e., the standardized correlation of the item with all other items in the scale; this is presented for scales with greater than three response categories), and a rationale for the decision to keep, delete, or add variables and items so deletion will not have a large impact on future analyses. Specifically, the columns are organized as follows:

Column A – Variable name
Column B – Data type (text, yes/no, Likert scale, etc.)
Column C – Description of every data element presently collected by CCT
Column D – Indicates whether question should be dropped, kept, modified, or whether a new question is suggested
Column E – Reliability information (if relevant)
Column F – Rationale for keeping/adding/dropping data elements.

Question Five: What options exist for dealing with the distinct data needs of the DFC National Evaluation and SAMHSA’s grants management/reporting oversight?

There are a number of options available for dealing with the distinct data needs of the DFC National Evaluation and grants management and oversight. At a minimum, colored fields could be used to distinguish what information pertains to the National Evaluation and what pertains to grant management and oversight. Delivery of improved and intensive evaluation technical assistance from the DFC national evaluation team’s Technical Assistance group will also help grantees and project staff better understand which components of COMET pertain to the National Evaluation and which pertain to SAMHSA. Moreover, evaluation technical assistance can provide information and assistance to grant communities in correctly understanding and distinguishing which data to enter into which SPF category or element. These efforts are already underway through the “Ask an Evaluator” webinar series. Such outreach and technical assistance will be critical as changes are made to the National Evaluation. There are other options, such as completely separating these data; however, we feel the easiest manner to address this issue is to keep data collection across all elements of the SPF, but perhaps highlight or distinguish which data pertain to the National Evaluation and which to SAMHSA’s grant management and oversight.

One option that ONDCP and SAMHSA may want to consider is completely separating core measures data collection to stand on its own rather than be part of the COMET SPF framework. Since core measures data is collected every other year, this may decrease confusion among grantees about when submissions are required. Moreover, system enhancements could walk grantees through the system, step-by-step, with scroll over on-screen help, FAQs, and other assistance to help grantees submit higher quality and more standardized data.

Question 5a: "If a new data system were designed and developed, what are the opportunities for increasing its usability and decreasing the reporting burdens of grantees?"

ICF will conduct an in-depth analysis of COMET, using trained specialists in Web usability and designers experienced in the design of grantee data systems. Additionally, ICF will enlist a small group of two to three new grantee coalition staff members who have not previously used COMET and bring them to ICF’s computer laboratory for an observed initial use of COMET. Each grantee will be assigned a trained usability specialist who would oversee a simulated data entry session by the new grantee. The specialist would only answer questions and provide assistance if the grantee were truly "stumped" and unable to proceed. A preliminary report of findings will be shared with the DFC Technical Advisory Group. A findings report will detail the opportunities identified along with an explanation of each. This current report lists a number of enhancements that could be implemented prior to a major re-design to increase usability and decrease burden; the vetting process for a new or an enhanced system are essentially the same.

Question 5b: "If a new data system were designed and developed, what are the opportunities for increasing its usability for ONDCP and for SAMHSA Project Officers?"

ICF probed for responses to this question as part of the process adopted for Question Two (in the case of ONDCP), and for Question Three (in the case of SAMHSA Project Officers). This information was listed in these previous sections, although the main focus was on enhancements that could be implemented relatively quickly in order to increase the quality of data collected for both ONDCP and SAMHSA project officers.

Question 5c: "If a new data system were to be designed and developed, what are the opportunities for increasing its usability and efficiency for ONDCP's evaluation contractor?"

The present arrangement of having separate contractors responsible for the DFC data system and the DFC National Evaluation is not optimal. As past experience has shown, such a division of responsibility can lead to errors. ICF is familiar with the present data collection arrangement, and will study and document the potential benefits of creating a unified system. A "unified system" is a central location and staffing for all data operations including survey response submissions, data validations, data cleaning, and data reporting. The central database would be the data of record, and all analytic work would be performed against these data.

Next Steps

ICF would like to move ahead rapidly to submit an OMB package based on feedback from ONDCP. Based on feedback from ONDCP, ICF will revise this report and collaborate with ONDCP on any final changes to COMET, CCT, and core data; and include such information in our OMB package. Moreover, we intend to obtain the input of SAMHSA project officers to determine how these modifications will affect Stop Act data collection requirements. We expect the OMB package to be developed early in Year 2 of the evaluation, with final OMB approval in Year 2. Our current target is to roll out a modified system for the November 2011 COMET data collection cycle.

REFERENCES

Allen, N. (2005). A multilevel analysis of community coordinating councils. American Journal of Community Psychology, 35(1/2), 49-63.

Ansari, W.E., & Weiss, E.S. (2006). Quality of research on community partnerships: Developing the evidence base. Health Education Research, 21(2), 175-180.

Arthur, M.W., & Blitz, C. (2000). Bridging the gap between science and practice in drug abuse prevention through needs assessment and strategic community planning. Journal of Community Psychology, 28(3), 241-255.

Arthur, M.W., Hawkins, D.J., Brown, E.C., Briney, J.S., & Oesterle, S. & Abbott, R.D. (2010). Implementation of the communities that care prevention system by coalitions in the community youth development study. Journal of Community Psychology, 38(2), 245-258.

Bartunek, J. M., Foster-Fishman, P. G., & Keys, C. B. (1996). Using collaborative advocacy to foster intergroup cooperation: A joint insider-outsider investigation. Human Relations, 49(6), 701-731.

Beebe, T.J., Harrison, P.A., Sharma, A., & Hedger, S. (2001). The community readiness survey: Development and initial validation. Evaluation Review, 25(1), 55-71.

Berg, M., Coman, E., Schensul, J.J. (2009) Youth action research for prevention: A multi-level intervention designed to increase efficacy and empowerment among urban youth. American journal of Community Psychology, 43, 345-359.

Beyers, J. M., Bates, J. E., Pettit, G. S., & Dodge, K. A. (2003). Neighborhood structure, parenting processes, and the development of youths’ externalizing behaviors: A multilevel analysis.

American Journal of Community Psychology, 31(1/2), 35–53.

Bowen, D. J., Kinne, S., & Urban, N. (1997). Analyzing community readiness for change. American Journal of Health Behavior, 21(4), 289-299.

Brooks-Gunn, J., Duncan, G. J., Klebanov, P. K., & Sealand, N. (1993). Do neighborhoods influence child and adolescent development? American Journal of Sociology, 99(2), 353–395.

Bursik, R. J., Jr., & Grasmick, H. G. (1993). Neighborhoods and crime: The dimensions of effective community control. New York: Lexington Books.

Butterfoss, F.D., & Francisco, V.T. (2004). Evaluating community partnerships and coalitions with practitioners in mind. Health promotion and Practice, 5(2), 108-114.

Butterfoss, F. D., Goodman, R. M., & Wandersman, A. (1993). Community coalitions for prevention and health promotion. Health Education Research 8(3), 315-330.

Cantillon, D. (2006). Community social organization, parents, and peers as mediators of perceived neighborhood block characteristics on delinquent and prosocial activities. American Journal of Community Psychology, 37(1/2), 111-127.

Catalano, R.F., Hawkins, D.J., Berglund, L., Pollard, J.A., Arthur, M.W. (2002). Prevention science and positive youth development: Competitive or cooperative frameworks? Journal of Adolescent Health, 31(Supp16), 230-239.

Chavis, D. M. (2001). The paradoxes and promise of community coalitions. American journal of Community Psychology, 29(2), 309-320.

Colder, C. R., Mott, J., Levy, S., & Flay, B. (2000). The relation of perceived neighborhood danger to childhood aggression: A test of mediatingmechanisms. American Journal of Community

Psychology, 28(1), 83–94.

Coulton, C. J., Korbin, J. E., Su, M., & Chow, J. (1995). Community level factors and child maltreatment rates. Child Development, 66, 1262-1276.

Deacon, Z., Foster-Fishman, P., Mahaffey, M. & Archer, G. (2009). Moving from preconditions for action to developing a cycle of continued social change: Tapping the potential of mini-grant programs. American Journal of Community Psychology, 37(2), 148-155.

Durlak, J.A., DuPre, E.P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American journal of Community Psychology, 41, 327-350.

Elliott, D. S., Wilson, W. J., Huizinga, D., Sampson, R. J., Elliott, A., &Rankin, B. (1996). The effects of neighborhood disadvantage on adolescent development. Journal of Research in Crime and Delinquency, 33(4), 389–426.

Feinberg, M.E., Greenberg, M.T., Osgood, W.D., Anderson, A., & Babinski, L. (2002). The effects of training community leaders in prevention science: Communities that Care in Pennsylvania. Evaluation and Program Planning, 25, 245-259.

Florin, P., Mitchell, R., Stevenson, J., & Klein, I. (2000). Predicting intermediate outcomes for prevention coalitions: A developmental perspective. Evaluation and Program Planning, 23, 341-346.

Florin, P., Mitchell, R. E., Stevenson, J. F. (1993). Identifying training and technical assistance needs in community coalitions: A developmental approach. Health Education Research, 8(3), 417-432.

Foster-Fishman, P.G., & Behren, T.R. (2007). Systems change reborn: Rethinking our theories, methods, and efforts in human services reform and community-based change. American journal of Community Psychology, 39, 191-196.

Foster-Fishman, P. G., Berkowitz, S. L., Lounsbury, D. W., Jacobson, S., & Allen, N. A. (2001). Building collaborative capacity in Community Coalitions: A review and integrative framework. American Journal of Community Psychology, 29(2), 241-261.

Foster-Fishman, P.G., Cantillon, D., Van Egeren. L.A., Pierce, S.J. (2006). Building an active citizenry: The role of neighborhood problems, readiness, and capacity for change. American Journal of Community Psychology, 39, 91-106.

Friedman, S.R., Reynolds, J., Quan, M.A., Call, S., Crusto, C.A., & Kaufman, J.S. (2007). Measuring changes in interagency collaboration: An examination of the Bridgeport Safe Start Initiative. Evaluation and Program Planning, 30, 294-306.

Granner, M. L., & Sharpe, P. A. (2004). Evaluating community coalition characteristics and functioning: A summary of measurement tools. Health Education Research, 19(5), 514-532.

Hawkins, D.J., Catalano, R.F., Miller, J.Y. (1992). Risk and protective factors for alcohol and other drug problems in adolescence and early adulthood: Implications for substance abuse prevention. Psychological Bulletin, 112(1), 64-105.

Israel, B.A., Schulz, A.J., Parker, E.A., Becker, A.B. (1998). Review of community-based research: Assessing partnership approaches to improve public health. American Review of Public Health, 19, 173-202.

Julian, D.A. (1997). The utilization of the logic model as a system level planning and evaluation device. Evaluation and Planning, 20(3), pp.251-257.

Julian, D.A., Ross, M., & Partridge, C. (2008). Challenges in supporting community implementation of science-based programs: A critical review of local partnerships for success plans. American Journal of Community Psychology,41,351-360.

Kreger, M., Brindis, C. D., Manuel, D. M., & Sassoubre, L. (2007). Lessons learned in systems change initiatives: Benchmarks and indicators. American Journal of Community Psychology, 39, 301-320.

Levanthal, T., & Brooks-Gunn, J. (2000). The neighborhoods they live in: The effects of neighborhood residence on child and adolescent outcomes. Psychological Bulletin, 126(2), 309–337.

Levy et al. (2005). Using multilevel, multisource needs assessment data for planning community interventions. Health Promotion Practice, 5(1), 59-68.

Manger, T.H., Hawkins, D.J., Haggerty, K.P., Catalano, R.F. (1992). Mobilizing communities to reduce risks for drug abuse: Lessons on using research to guide prevention practice. The Journal of Primary Prevention, 13(1), 3-22.

McKnight, J.L. (1996). A twenty first century map for healthy communities and families. Evanston, IL: Institute for Policy Research.

McMillan, B., Florin, P., Stevenson, J., Kerman, B., & Mitchell, R. E. (1995). Empowerment praxis in community coalitions. American Journal of Community Psychology, 23(5), 699-726.

Nowell, B. (2009). Profiling capacity for coordination and systems change: The relative contribution of stakeholder relationships in interorganizational collaboratives. American Journal of Community Psychology, 44, 196-212.

Roussos, S. T., & Fawcett, S. B. (2000). A review of collaborative partnerships as a strategy for improving community health. Annual Review of Public Health, 21, 369-402.

Sampson, R.J., & Groves, W.B. (1989). Community structure and crime: Testing social disorganization theory. American Journal of Sociology, 94(4), 774-802.

Sampson, R.J., Morenoff, J.D., & Gannon-Rowley, T. (2002). Assessing “neighborhood effects”: social processes and new directions for research. Annual Review of Sociology, 28, 443-478.

Sampson, R. J., Raudenbush, S. W., & Earls, F. (1997). Neighborhoods and violent crime: A multilevel study of collective efficacy. Science, 277, 918–924.

Saul, J., Wandersman, A., Flaspohler, P., Duffy, J., Lubell, K., Noonan, R. (2008). Research and action for bridging science and practice. American journal of Community Psychology, 41, 165-170.

Thompson, A. M., Perry, J.L., Miller, T.K. (2007). Conceptualizing and measuring collaboration. Journal of Public Administration Research, 19, 23-56.

Trickett, E.J., Schensul, J.J. (2009). Summary comments: Multi-Level community based culturally situated interventions. American Journal of Community Psychology, 433, 377-381.

Wandersman, A. (2009). Four keys to success (theory, implementation, evaluation, and resource/system support): high hopes and challenges in participation. American Journal of Community Psychology, 43, 3-21.

Watson-Thompson, J., Fawcett, S. B., & Schultz, J.A. (2008). Differential effects of strategic planning on community change in two urban neighborhood coalitions. American Journal of Community Psychology, 42(1/2), 25-38.

Weeks, M. R., Convey, M., Dickson-Gomez, J., Li, J., Radda, K., Martinez, M., & Robles, E. (2009). American Journal of Community Psychology, 43, 330-344.

Zakocs, R. C., & Edwards, E. M. (2006).What explains community coalition effectiveness? A review of the literature. American Journal of Preventive Medicine, 30(4), 351-361.

Appendix A: Internal Logic Model, Measurement Mapping, & Brief literature Review

Overview of Internal Logic Model, Measurement Maps, & Literature Review

This appendix presents and describes the various constructs and variables targeted by the DFC National Evaluation. First, we present the internal National Evaluation logic model, which was developed by the technical advisory group (TAG) and members of ICF’s National Evaluation team. Next, we present a series of tables that lists the exact items used to measure each variable and construct listed in the internal model (i.e., measurement maps). Finally, this appendix ends with a brief literature review. This review provides the empirical base for all selected constructs and variables, and provides a high-level description on how current and proposed variables are being measured in the evaluation and will be used in analyses. For more detailed literature reviews, please see some recent comprehensive reviews that have focused on coalition dynamics and effectiveness across a number of research areas (Ansari & Wells, 2006; Foster-Fishman et al., 2001; Granner & Sharpe, 2004; Israel, Schulz, Parker, & Becker, 1998; Roussos & Fawcett, 2000; Thomson, Perry, & Miller, 2009; Zakocs & Edwards, 2006).

Background

While the logic model working group was initially tasked with revising the National DFC logic model, it quickly became apparent that a much more specified logic model was needed to guide our work and ensure the collection of appropriate data. This internal model was designed to highlight the details that were not included in the external DFC National Evaluation logic model (see p. 15). For instance, the external model identifies the importance of coalition structure and processes as a major component of DFC and links this to the implementation of strategies and activities, which is then hypothesized to produce positive community and population-level outcomes. However, little to no detail regarding how we will measure these major components is provided. This was purposeful as the intent of the external model is to produce an overarching vision and road map on how DFC coalitions produce positive change in their communities. While such a top level perspective is needed for communicating with more than seven-hundred and fifty coalitions and their partners, the guiding internal National Evaluation logic model mandates much more specificity (see Figure 2 below).

Figure 2. Internal DFC National Evaluation Logic Model

CONTEXT	Coalition Structure and Processes	Strategies and Activities	COALITION CAPACITY	COALITION EFFECTIVENESS
Community Sense of community / Community social organization Community readiness for change ATOD needs ATOD prevention resources Related initiatives Socio-economic context Critical events Stability Coalition Organizational history Past collaboration involvement Funding history Past accomplishment Institutional / grassroots focus Critical Events	Member Competency Member coalition involvement Member relationships Skills and knowledge Structure Formalized structures Degree of centralization Membership breadth and diversity Leadership structure Institution / grassroots mix Workgroup/committee organization Processes Formalized procedures Task and role clarity Supports for member involvement Leadership style Communication Conflict resolution Shared decision making Information / expertise input and use Strategic planning framework (SPF) Use data to inform decisions External relations / resource access	Coalition Role Membership and Focus Direct implementation Advocacy Initiation / support Member contributions External partnering Programmatic Capacity Clear, realistic objectives Unique, innovative services Need driven Cultural competence Use of evidence-based practice Strategy / Activity Mix Information / Education Coordination Environmental policy / enforcement Environmental assets and opportunities Direct service programs Coalition & Community Outputs Coalition outputs Population targets/ penetration Volume (dose) Comprehensiveness	Coherence Shared vision Shared solutions Inclusiveness Broad involvement Commitment /Satisfaction Coalition Climate Open Trusting Values diversity Shared input Positive External Relations Non-member agencies Activity focus Access community resources Community role / recognition Capacity Building Effort Outreach / recruitment Access to Training / TA Build member, coalition, and programmatic capacity Continuous Improvement Data based & Systematic Explicit decision points (e.g., annual action plans) Responsive / adaptive	Community/ System Outcomes
				Norms & Awareness ATOD risk perception / acceptance Media Systems & Policy Change Collaboration Service integration Public / business /education policies Sustainable Accomplishments Systems Services
				Community Behavioral Outcomes
				Substance Use Prevalence Core use measures (revised) Contributors & Consequences Educational performance Health & wellness Crime Family Social Economic

The internal model is designed to fill this gap by breaking down each of the major DFC components or domains in the external model (i.e., context, coalition structures and processes, strategies and activities, coalition capacity, and coalition effectiveness) and lists the exact constructs and variables that will be used to assess these components. To provide an example, within the domain of context, the first column indicates that there are two overarching constructs: (1) community context, and (2) coalition context. We have selected eight variables to assess community context and six variables to assess coalition context. These variables were identified in the research and practice literature as critical to the success of community change initiatives targeting a multitude of community and population-level outcomes. For example, the variables for community context include:

Sense of community / Community social organization
Community readiness for change
ATOD needs
ATOD prevention resources
Related Initiatives
Socio-economic context
Critical events
Stability

Measurement Maps and Data Sources

While the internal model presents some more specificity, what is still needed is the exact item content and data source for each domain, construct, and variable in the internal model. This ensures that the National Evaluation is collecting item-level data that is needed to test the hypothesized relations in both the internal and external models. Thus, what follows is a series of measurement maps providing this linkage (see Tables 1-5).

For DFC, there are three primary sources for the DFC National Evaluation – the Coalition Online Management and Evaluation Tool (COMET), the Coalition Classification Tool (CCT), and core outcome data (core). COMET data is collected every six months online and is typically entered by the coalition chair and/or other leaders who are able to answer questions regarding planning and implementation activities, including budget-related and other administration information. The CCT is collected one time a year and represents a more typical coalition survey that asks members about their perceptions regarding coalition dynamics, functioning, outputs, and impacts. Core outcome data is submitted every other year via the COMET data system. Other data sources include process and outcome data that will be collected during annual case study site visits and publicly available data (e.g., Census data, UCR data, FARS data). Please see the following tables below, which provide linkage down to the item level. This is followed by a brief literature review which provides the empirical rationale for both the external and internal models and briefly describes both current and proposed variables, data source, and how they will be used in analyses.

Table 1. Variables and Items of Context

VARIABLE		ITEM		DATA SOURCE
Community Context
Sense of Community / Community Social Organization	Community Social Organization. New proposed variable that aligns with risk and protective framework.		CCT
	Assessment Section. Community Needs Assessment. Target community name.		COMET
	Assessment Section. Community Needs Assessment. Target geographic area (urban, suburban, rural).		COMET
	Assessment Section. Community Needs Assessment. Target specific geographic area.		COMET
	Assessment Section. Community Needs Assessment. Target zip codes.		COMET
	Assessment Section. Community Needs Assessment. Text (Please further describe the geographic area(s) selected above).		COMET
Community readiness for change	Community Readiness. New proposed scale to address an important and integral variable for community change efforts – readiness at the community level.		CCT
	Assessment Section. Assessment Progress. Accomplishment or barrier (new drop downs).		COMET
	Assessment Section. Assessment Progress. Descriptive Text.		COMET
	Assessment Section. Community Needs Assessment. Text (Assessment summary and key findings).		COMET
ATOD Needs	Assessment Section. Community Needs Assessment. Substance of issue in the community.		COMET
	Assessment Section. Risk and protective factor framework. ID of assets and risks.		COMET
	Assessment Section. Community Needs Assessment. Text (Assessment summary and key findings).		COMET
ATOD prevention resources	Readiness scales above address some of these resources (e.g., leadership, financial resources).		CCT
	Assessment Section. Community Needs Assessment. Text (Assessment summary and key findings).		COMET
	Assessment Section. Risk and protective factor framework. ID of assets and risks.		COMET
	Assessment Section. Risk and protective factor framework. New proposed item. “To what extent does your coalition use a risk/protective factor approach in your prevention efforts?”		COMET
	Capacity Section. Funding Streams. Dollar amount.		COMET
	Capacity Section. Funding Streams. Dollar amount by source.		COMET
Related initiatives	Assessment Section. Community Needs Assessment. Modified variable. “Approximately, how many groups, excluding your own organization, are working on the prevention of substance abuse in your community?”		COMET
Related initiatives	Assessment Section. Community Needs Assessment. Modified variable. “In the past 6 months, how many of these groups (excluding your own DFC partners/members) have you worked with to reduce substance abuse in your community through shared activities and events, such as co-sponsoring substance abuse prevention activities?”		COMET
SES Context	Assessment Section. Community Needs assessment. Text (Please further describe the geographic area(s) selected above).		COMET
	Assessment Section. Text (Assessment summary and key findings).		COMET
	Census. Poverty Rate.		Census
Critical Events	Assessment Section. Assessment Progress. Accomplishment or barrier (new drop downs). Descriptive Text.		COMET
	Capacity Section. Capacity Progress. Accomplishment or barrier (new drop downs). Descriptive Text.		COMET
	Planning Section. Planning Progress. Accomplishment or barrier (new drop downs). Descriptive Text.		COMET
	Implementation Section. Implementation Progress. Accomplishment or barrier (new drop downs). Descriptive Text.		COMET
	^⁵Evaluation Section. Evaluation Progress. Accomplishment or barrier (new drop downs). Descriptive Text		COMET
	Protocol/Item to be developed for case study sites.		Case Study
Stability	Census. Rates of residential stability.		Census
Other	There will be text fields for each section in order to provide ample opportunity for coalitions that want to further provide more context and detail regarding their needs assessment process, activities, accomplishments, challenges, etc.		COMET CCT
Coalition Context
Organizational History	Assessment Section. Assessment Activity. Activity Name (new drop downs) – Descriptive text.		COMET
Organizational History	All SPF Sections. Accomplishments/Barriers.		COMET
Past Collaborative Involvement	Case Study Sites. Social network analysis.		Case Study
Past Collaborative Involvement	Case Study Sites. Qualitative Data. Collaborative representatives, neighborhood leaders, etc.		Case Study
Funding History & Current Additional Sources	Capacity Section. Funding Streams. [What is your coalition’s total annual operating budget (with a number of categories to list money amounts under, including DFC grant, in-kind contributions, etc.)].		COMET
Coalition Readiness for Change	Coalition Readiness. New proposed scale to address an important and integral variable for community change efforts – readiness at the coalition level.		COMET
Past Accomplishments	Assessment Section. Assessment Progress. Accomplishment or barrier (new drop downs). Descriptive text.		COMET
	All SPF Sections. Accomplishments.		COMET
	Protocol/Item to be developed for case study sites.		Case Study
Institutional / Grassroots Focus	Capacity Section. Coalition Membership.		COMET
Institutional / Grassroots Focus	Protocol/Item to be developed for case study sites.		Case Study
Critical Events	Administration Section. Changes in Leadership.		COMET
Critical Events	All SPF Sections. Accomplishments/Barriers.		COMET
Other	There will be text fields for each section in order to provide ample opportunity for coalitions that want to further provide more context and detail regarding their activities, accomplishments, challenges, etc.		COMET

Table 2. Variables and Items for Coalition Structure and Process

VARIABLE	ITEM	DATA SOURCE
Member Competency
Member Coalition Involvement	Capacity Section - New Proposed Item. “How many active members are there in your coalition from each of the 12 sectors (Active members means an individual has attended at least one meeting in the last 6 months).	COMET
Member Coalition Involvement	Capacity Section. New Proposed Item. In addition to the number of active members, another column asks to rate the average level of sector involvement from (1) Low to (5) High.	COMET
Member Relationships	Communication and Conflict Management. New proposed scale to address an important coalition dynamic.	CCT
	Social Network Analysis. Case Study Sites.	COMET
	Protocol/Item to be developed for case study sites (interviews, focus groups)	COMET
Skills & Knowledge	Assessment Section. Community Needs Assessment	COMET
VARIABLE	ITEM	DATA SOURCE
Coalition Structure
Formalized Structures	(2) Coalition Type - Which of the following best describes your coalition? Loosely organized to a highly formal arrangement.	CCT
Formalized Structures	(3) Organizational Structure - How would you describe your coalition’s organizational status? Reports to a larger organization to fully independent	CCT
Degree of Centralization	(5a) We have a board or governing body that sets the direction of the Coalition? (A board or governing body is defined as a formal group that makes decisions for the coalition).	CCT
Membership Breadth & Diversity	Capacity Section - New proposed item “How many active members are there in your coalition from each of the 12 sectors. (Active members means an individual has attended at least one meeting in the last 6 months).	COMET
Membership Breadth & Diversity	Cultural Diversity Scale. Individual items can be use from this scale (e.g., 12a, 12e, 12f, 12g).	CCT
Leadership Structure	(2) Coalition Type - Which of the following best describes your coalition? (e.g., Loosely organized to a Highly Formal Arrangement).	COMET
Leadership Structure	(4) Confidence in Task Completion Scale. Individual items, such as 4k.	CCT
Institution / Grassroots Mix	(6) Identification of Community Leaders Scale.	CCT
Institution / Grassroots Mix	(7) Community Leadership in Coalition Efforts Scale.	CCT
Workgroup / Committee Organization	(5b) – Formalized Procedures; Established subcommittees	CCT
VARIABLE	ITEM	DATA SOURCE
Coalition Processes
Formalized Procedures	(5b) Written Procedures – E.G., policy for leadership rotation, written expectations for member participation, leadership selection, written description for decision making) & Formalized Procedures – E.G., hold regularly scheduled meetings, written agenda, prepare and distribute minutes, organizational chart, established subcommittees, paid or in-kind staff.	CCT
	Planning Section. Logic Model.	COMET
	Planning Section. Strategic Plan.	COMET
	Planning Section. Action Plan(s).	COMET
Task & Role Clarity	Planning Section. Action Plan(s).	COMET
Supports for Member Involvement (Member Satisfaction)	Member Satisfaction Scale. Proposed new scale with three items.	CCT
Leadership Style	(6) Identification of Community Leaders Scale.	CCT
	(7) Community Leadership in Coalition Efforts Scale.	CCT
	Coalition Leadership Scale. New Proposed Scale.	CCT
	Committee Leadership Scale.	CCT
	Coalition Coordinator Leadership Scale.	CCT
Communication	(8) Collaborative Decision Making Scale.	COMET
Conflict Resolution	(9) Frequency of Conflicts. Propose deleting.	COMET
	(10) Cause of Conflicts. Propose deleting.	COMET
	(11) Overall Impact of Conflicts. Propose deleting.	COMET
Shared Decision Making	(8) Collaborative Decision Making Scale.	CCT
Information / Expertise Input & Use	Capacity Section. Assistance Needed by Coalition.	COMET
	Capacity Section. Proposed Assistance Provided by Coalition.	COMET
	Capacity Building Capacity Scale.	CCT
Strategic Planning Process (SPF Framework)	Planning Section. Strategic Plan.	COMET
	Planning Section. Action Plan.	COMET
	Planning Section. New Proposed Item. “To what extent does your coalition use the SPF framework?	COMET
	Planning Section. New Proposed Item “To what extent does your coalition incorporate other planning frameworks (e.g., CCT)?	COMET
Data Use (e.g., needs, feedback, evaluation)	(13) Performance Evaluation – Yes/No items that tap into program monitoring (system for monitoring and tracking coalition activities) and performance and general evaluation activities (collect and analyze data coalition activities and community indicators).	CCT
External Relations / Resource Access	(13) One specific item of yes/no. “Use relations with local Universities or others to get assistance on evaluating coalition efforts.”	CCT
External Relations / Resource Access	Protocol/Item to b developed for case study sites	Case Study

Table 3. Variables and Items for Strategies and Activities

VARIABLE		ITEM	DATA SOURCE
Coalition Role
Membership and Focus		Capacity Section. Coalition Membership.	COMET
		(17) Coalition Focus 1. Exclusively coordination to exclusively environmental.	CCT
		(22) Coalition Focus 2. Directly in the community or indirectly by building partner agencies’ capacity.	CCT
Direct Implementation		Planning Section. Strategies to achieve objective.	COMET
		Implementation Section. Implementation Activity.	COMET
		Implementation Section. Implementation Progress. Accomplishment/Barrier and text field.	COMET
		(22) Coalition Focus 2. Directly in the community or indirectly by building partner agencies’ capacity.	CCT
Advocacy		Planning Section. Strategies to achieve objective.	COMET
		Implementation Section. Implementation Activity.	COMET
		Implementation Section. Implementation Progress.	COMET
Initiation / Support		Planning Section. Strategies to achieve objective.	COMET
		Implementation Section. Implementation Activity.	COMET
		Implementation Section. Implementation Progress.	COMET
Member Contributions		Capacity Section. Basic Collaborative Activity. Is collaboration among members of your coalition (Increasing, Decreasing, Staying the Same)?	COMET
Member Contributions		Capacity Section. Basic Collaborative Activity. Rating of degree of involvement by sector.	COMET
External Partnering		Case Study. Social Network Analysis.	Case Study
Programmatic Capacity
Clear & Realistic Objectives		Planning Section. Logic Model.	COMET
		Planning Section. Strategic Plan.	COMET
		Planning Section. Action Plan(s).	COMET
		Planning Section. Objective(s). Consistent with other sites (dropdowns)?	COMET
Unique & Innovative Services		Planning Section. Action Plan(s).	COMET
		Protocol/Item to be developed for case study sites.	Case Study
		Case Study. Document Review.	Case Study
		Implementation Section. Implementation Progress. Accomplishment/Challenges and text field.	COMET
Need Driven		Assessment Section. Community needs assessment (all subsections).	COMET
Cultural Competence		(12) Cultural Diversity – Representative staff members, culturally appropriate materials, accessible meeting times, staff trained to be culturally sensitive in interactions with target population, etc..	CCT
Cultural Competence		Case Study – Review of all Documents, including social marketing materials.	Case Study
Use of Evidence-Based Practices		Planning Section. Strategic Plan.	COMET
		Planning Section. Action Plan(s).	COMET
		Implementation Section. Activities.	COMET
		Protocol/Item to be developed for case study sites.	COMET
VARIABLE		ITEM	DATA SOURCE
Strategy/Activity Mix
Information / Education		(17) Coalition Focus 1 – exclusively helps coordinate programs and services, split between coordinating and environmental/ policy change, to exclusively environmental or policy change.	CCT
		Implementation Section. Strategy.	COMET
		Implementation Section. Implementation Activities.	COMET
		Implementation Section. Implementation Progress. Accomplishments/Barriers with text field.	COMET
Coordination		(17) Coalition Focus 1 - Exclusively helps coordinate programs and services, split between coordinating and environmental/ policy change, to exclusively environmental or policy change.	CCT
		Implementation Section. Strategy.	COMET
		Implementation Section. Activity.	COMET
		Implementation Section. Implementation Progress. Accomplishments/Barriers with text field.
Environmental Policy / Enforcement		(17) Coalition Focus 1 – Exclusively helps coordinate programs and services, split between coordinating and environmental/ policy change, to exclusively environmental or policy change.	CCT
		(21) Environmental Strategies – nine items assessing the degree to which the coalition believe in and have the ability to address environmental change strategies.	CCT
		Planning Section. Goals and Objectives. Strategies to achieve objective. (e.g., modifying/changing policies, physical design)	COMET
		Implementation Section. Strategy.	COMET
		Implementation Section. Activity.	COMET
		Implementation Section. Implementation Progress. Accomplishments/Barriers with text field.
Environmental Assets & Opportunities		Implementation Section. Strategy.	COMET
		Implementation Section. Activity.	COMET
		Implementation Section. Implementation Progress. Accomplishments/Barriers with text field.
Direct Service Programs		(22) Coalition Focus 2 – One item that asks the degree to which the coalition is engaged in directly in the community (coordinating programs and pursuing policy changes), indirectly in the community (building the capacity of other organizations), or a mix of both of these approaches.	CCT
		Implementation Section. Implementation Activity.	COMET
		Implementation Section. Implementation Progress. Accomplishments/Barriers with text field.	COMET
VARIABLE		ITEM	DATA SOURCE
Coalition and Community Outputs
Coalition Outputs	Planning Section. Objective (new list of dropdowns). To what extent has the objective been achieved (from Not at all to exceed our objective)?
	Implementation Section. Implementation Progress. Accomplishments.		COMET
	Case Study sites. Document Review.		Case Study
Population Targets / Penetration	Implementation Activity. New categories/columns which includes penetration.		COMET
	Case Study sites. Document Review.		Case Study
	Protocol/Item to be developed for case study sites.		Case Study
Volume /Dosage	Planning Section. Objective (new list of dropdowns). To what extent has the objective been achieved (from Not at all to exceed our objective)?		COMET
	Implementation Section. Implementation Activities. New categories/columns which includes dosage.		COMET
	Implementation Section. Implementation Accomplishments/Challenges.		COMET
Comprehensiveness	Planning Section. Goals and Objectives. Strategies to achieve objective.		COMET
	Implementation Section. Implementation Activities.		COMET
	Implementation Section. Implementation Accomplishments/Challenges.		COMET
	New Proposed Variable. Implementation Section. Implementation Strategy. Count total number of activities within each strategy (and collect related dosage information).		COMET

Table 4. Variables and Items for Coalition Capacity

VARIABLE	ITEM	DATA SOURCE
Coherence
Shared Vision	Confidence in Task Completion Scale. Provide direction and vision for the Coalition through its leadership	CCT
	Planning Section. Strategic Plan	COMET
	Planning Section. Logic Model	COMET
	Planning Section. Action Plan(s) – Years 2-5	COMET
Shared Solutions	(25) Synergy Scale	CCT
Inclusiveness	Shared decision making items can be used for this variable.	CCT
	Planning Section. Strategic Plan	COMET
	Planning Section. Action Plan(s) – Years 2-5	COMET
Broad Involvement	(8) Shared decision making	CCT
Broad Involvement	Capacity Section. Coalition Membership.	COMET
Commitment / Satisfaction	(25) Synergy – One item specifically asks about commitment of partners.	CCT
Commitment / Satisfaction	Capacity Section. Coalition Membership. Rate the average level of involvement for each represented sector. Yes/No for representation and a Likert scale from (1) Low to (5) High.	COMET
Coalition Climate
VARIABLE	ITEM	DATA SOURCE
Open	Protocol/Item to be developed for case study sites.	Case Study
Trusting	Social Network Analysis.	Case Study
Trusting	Capacity Section. Challenges and Accomplishments.	COMET
Value Diversity	(12) Cultural Diversity Scale.	CCT
Shared Input	Shared Decision Making Scale. Shared decision making items can be used for this variable.	CCT
Positive External Relations
VARIABLE	ITEM	DATA SOURCE
Non-Member Agencies	Capacity Section. Coalition Membership.	COMET
Non-Member Agencies	Capacity Section. Challenges	COMET
Activity Focus	Capacity Section. Basic Collaborative Activity. This section now includes # people who attended event, along with a rating of degree of success of event (None to A Great Deal)	CCT
	Capacity Section. Assistance Provided by Coalitions. This new variable asks for a text description of any training and technical assistance provided by the coalition.	COMET
Access Community Resources	Capacity Section. Funding Stream. In the next 12 months do you expect your coalition’s funding level to (Increase, Decrease, Stay about the Same)	COMET
	Capacity Section. Funding Stream. Fundraising/private donations. Dollar Amount.	COMET
	Capacity Section. Funding Stream. Foundations/non-profit foundations. Dollar Amount.	COMET
	Capacity Section. Funding Stream. City/county government. Dollar Amount.	COMET
	Protocol/Item to be developed for case study sites.	Case Study
Community Role & Recognition	(26) Collective Efficacy Scale – One item specifically asks about this variable. “Our coalition has the support of other organizations and influential community leaders.”	CCT
Capacity Building Effort
VARIABLE	ITEM	DATA SOURCE
Outreach/ Recruitment	Capacity Section. Coalition Membership.	COMET
Outreach/ Recruitment	Protocol/item to be developed for case study site.	Case Study
Access to Training & Technical Assistance (T?TA)	(24) Capacity Building Capacity – Ability to produce education al materials, resources to send members to workshops and conferences, information on research based prevention programs and strategies, etc.	CCT
	Capacity Section. Assistance needed by coalition. Listing of 15 core competencies for coalitions and they rate themselves on the degree to which they could benefit from training and technical assistance from Not at All to A Great Deal.	COMET
	Capacity Section. Assistance needed by coalition. Descriptive text field.	COMET
	Capacity Section. Assistance needed by coalition. Mode of TA	COMET
	Capacity Section. Assistance needed by coalition. Source of TA	COMET
Build Member , Coalition, and Programmatic Capacity	Capacity Section. Assistance needed by coalition. Listing of 15 core competencies for coalitions and they rate themselves on the degree to which they could benefit from training and technical assistance from Not at All to A Great Deal.	COMET
	Protocol/Item to be developed for case study site.	Case Study
VARIABLE	ITEM	DATA SOURCE
Continuous Improvement
Data-Based	(24) Capacity Building Capacity. Two of the items specifically reference research or evidence-based programs.	CCT
	(13) Performance Evaluation Scale.	CCT
	(19) Action Plan Activities Scale.	CCT
Systematic
Explicit Decision Points	Planning Section. Strategic Plan.	COMET
	Planning Section. Action Plan.	COMET
	(19) Action Plan Activities Scale. Specific items can be used from this scale (e.g., Created a realistic timeline for completing activities).	CCT
Responsive & Adaptive	(27) Coalition Self-Assessment Scale.	CCT

Table 5. Variables and Items for Coalition Effectiveness

VARIABLE	ITEM	DATA SOURCE
Norms & Awareness
ATOD Risk Perception / Awareness	Evaluation Section. Core Outcome Measure. Peer Disapproval.	CORE
	Evaluation Section. Core Outcome Measure. Perception of Risk/Harm.	CORE
	Assessment Section. Risk/Protective Factor. Trend is (Increasing, Decreasing, Staying the Same).	COMET
Media Other	Implementation Section. Accomplishments.	COMET
	Implementation Section. Challenges.	COMET
	Evaluation Section. Accomplishments.	COMET
	Evaluation Section. Challenges.	COMET
	Protocol/item to be developed for case study sites.	Case Study
VARIABLE	ITEM	DATA SOURCE
Systems & Policy Change
Collaboration	Organizational Impact Scale. This proposed scale has been included to better document and assess how the quality of collaboration matters in coalition efforts.	CCT
	Interorganizational Coordination Scale. Proposed new scale with four items	CCT
	Capacity Section. Basic Collaborative Activity. Is collaboration among members in your coalition (Increasing, Decreasing, Staying the Same)	COMET
Service Integration	Organizational Impact. Proposed new eleven item scale.	CCT
Service Integration	Protocol/Item to be developed for case study sites.	Case Study
Policy Change – Public / Business / Education	Implementation Section. Strategy.	COMET
VARIABLE	ITEM	DATA SOURCE
Sustainable Accomplishments
Systems	(14) Coalition Sustainability Scale. This scale has been modified by including more items.	CCT
	(26) Perceived Effectiveness Scale. This is the collective efficacy scale re-named with additional proposed items to assess the degree and extent of successful coalition activities and outcomes.	CCT
	Planning Section. Planning Objectives. To what extent has the Objective been achieved? (From not at all to exceeded our objective).	COMET
	Implementation Section. Implementation Accomplishments/Challenges.	COMET
	Protocol/Item to be developed for case study sites.	Case Study
Services	Planning Section. Planning Objectives. To what extent has the Objective been achieved? (From not at all to exceeded our objective).	COMET
	Implementation Section. Implementation Accomplishments/Challenges.	COMET
	Protocol/item to be developed for case study sites.	Case Study
VARIABLE	ITEM	DATA SOURCE
Substance Use Prevalence
Core Measures - Revised	Evaluation Section. Core Outcome Section. Average Age of Onset. Alcohol, Tobacco, Marijuana	CORE
Core Measures - Revised	Evaluation Section. Core Outcome Section. Past 30-Day Use. Alcohol, Tobacco, Marijuana	CORE
Contributors & Consequences
Educational Performance	Local Data if Available.	Local
Educational Performance	Archival Data if Available.	Educational District
Health & Wellness	Local Data if Available.	Local
	Community Readiness and Capacity. New proposed scale that assesses a community’s readiness and capacity to implement a community-based substance use prevention program (DFC).	CCT
	Community Social Organization. Proposed new variable that assesses the degree of interaction with neighbors, attachment to the neighborhood, and includes a perceived safety in the neighborhood scale.	CCT
Crime	Local Data if Available.	Local
	Archival Data if Available.	UCR; FARS
	Community Social Organization. Proposed new variable that assesses the degree of interaction with neighbors, attachment to the neighborhood, and includes a perceived safety in the neighborhood scale.	CCT
Family	Local Data if Available.	Local
Social	Local Data if Available.	Local
Social	Archival Data if Available.	Census; Other
Economic	Local Data if Available.	Local
Economic	Archival Data if Available.	Census; Other

LITERATURE REVIEW

As highlighted throughout this appendix, this literature review is intended to provide empirical support and rationale for the re-design of the National DFC Evaluation Logic Model and the development of our internal model, which guided decisions for which variables to keep in COMET and other data sources, which to delete, what to add, and what additional analyses are needed to enhance our understanding of how coalitions can create community and population-level changes. The literature review is organized around the major components of the National DFC Evaluation logic model and includes: (1) context, (2) coalition structures and processes, (3) strategies and activities, (4) coalition capacity, and (5) coalition effectiveness. It provides a high level review of the State of current research for each major component and associated constructs and variables. We then briefly explain on how we conceptualize using it in our evaluation of DFC, including possible additional variables and analyses.

Context

Community

The local community context and history are important contextual factors that must be captured to adequately understand why each coalition choose to select specific objectives and strategies, address specific subpopulations of youth, and how they involved youth and community members and leaders in their efforts. In fact, we believe the community context impacts all facets of coalition operations as reflected in our external logic model. For our internal model, we broadened the importance of contextual variables to also include the context of the coalition, which includes organizational and collaborative history, coalition readiness for change, and type of change approach (institutional or grassroots focus).

In terms of community context, constructs included in the internal model include community social organization, community readiness for change, strengths/weaknesses of the community ATOD prevention resource system, number of related initiatives, socioeconomic context, critical events, and community stability. Since the prior evaluation did not include many of these contextual variables, we have selected a number of new measures to include in the evaluation. For instance, one aspect of the community context that has received extensive attention and empirical support in differentiating successful community-based efforts from unsuccessful ones is the concept of community readiness for change (Arthur & Blitz, 2000; Bowen, Kinne, & Urban, 1997). Community readiness has been defined as “the relative level of acceptance of a program, action, or other form of decision-making activity that is locality-based” (Edwards, Oeting, & Littlethunder, 1997). There are many different conceptualizations of community readiness for change and there is also substantial conceptual and measurement overlap with related constructs, such as community capacity for change. Given the state of the research, it is important for researchers to task themselves “capacity and readiness for what and where?” (Foster-Fishman, Cantillon, Van Egeren, & Pierce, 2006, p. 3,). Thus, given DFC’s main goal is to reduce substance use among youth, we created a community readiness and capacity for change scale specifically for this initiative^⁶. This scale queries coalition members about their community’s knowledge of and support for DFC goals, leadership, resources, and history of collaboration to address community concerns and issues.

Measures that assess community identify and community social organization have also been found to be important contextual variables that can help facilitate or impede the implementation of community-based programs and services designed to promote community health. Over the last couple of decades, research has re-established that community matters in terms of a number of outcomes, from educational success to violence and drug use (Bursik & Grasmick, 1993; Leventhal & Brooks-Gunn, 2000; Sampson, Morenoff, Gannon-Rowley, 2002. Given the overarching importance of community and its influence on every major component in our external model, one of the main goals of the evaluation is to increase the measurement and understanding of how community contextual conditions can impact the degree of implementation of community policies and practices targeting substance use reduction. For instance, in line with the protective and risk factor approach (Hawkins & Catalano, & Miller, 1992), we propose using a measure of community social organization that assesses the general safety of the neighborhood and the degree to which community members feel they are connected to their neighbors and are able to act and create change in the community (Cantillon, 2006).

Research on neighborhood effects has established that the degree of poverty within a community, particularly concentrated poverty, and the amount of residential turnover or stability has important and long-standing impacts on community rates of violence and drug use (Colder, Mott, Levy, & Flay, 2000; Sampson & Groves, 1989; Sampson, Raudenbush, & Earls, 1997). For our evaluation, we plan on using census data, along with some quantitative scales to create a neighborhood typology and assess whether DFC activities are more successful in some types of neighborhoods compared to others, or if different neighborhood types utilize different community change strategies. These types of analyses will provide a much deeper understanding of the influence of the neighborhood environment and context compared to the current urban, suburban, and rural typology. We also plan to use qualitative data collected from neighborhood leaders at our case study sites to augment this quantitative analysis to obtain a richer perspective on how concentrated poverty and neighborhood instability impact the implementation of DFC.

Community readiness and social organization or sense of community are broad indicators of a community’s infrastructure and capacity for creating positive change. More specifically, communities need to have the resources (ATOD prevention resources) to successfully implement a comprehensive and community-wide initiative designed to reduce substance use and promote community well-being (Wandersman, 2009). This includes the financial resources to implement evidence-based practices, reimburse key participants, and access to high-quality coordinated training and technical assistance resources to support effective implementation practice. In fact, research has demonstrated that many community-based programs fail to be adequately implemented because they are lacking this infrastructure, which has been also been labeled the prevention support system (PSS) (Saul et al., 2008). In a related manner, the existence of prior or ongoing initiatives facilitates the development and implementation of community-based interventions. Such initiatives often force communities to address the typical challenges encountered when designing and implementing social change programs and initiatives, such as the amount of time it takes to produce noticeable change and degree of collaboration required to improve the delivery of appropriate services and improve youth outcomes.

Coalition

Since coalitions are comprised of a subset of community members, many of whom are community leaders or leaders of community-based agencies, we also created a coalition readiness for change scale, which includes the same items in community readiness but also asks about prior positive collaboration among partner agencies, available financial resources, and availability of formal and informal supports and services. Our prior experience has supported differentiating broader community readiness and capacity from the coalition itself, as they often differ, and such measures provide an understanding of the various levels involved in comprehensive community change initiatives (Kreger, Brindis, Manuel, & Sassoubre, 2007; Levy et al., 2005).

As mentioned under community context, another important variable in differentiating coalitions’ ability to create positive community change is simply their past experience in collaborative efforts, including key organizations or agencies’ history or being involved in these activities and efforts. We intend to gauge how prior collaborative work impact DFC through qualitative data obtained during site visits aas well as a social network analysis, which will include an item specific to prior collaborative work and relations. Also, while each DFC community receives the same amount of grant funding, some DFC communities are able to broker additional funding from foundations, local governments, or obtain in-kind funding from key participating agencies and organizations. Finally, coalitions can differ depending upon the degree to which their constituents are from State and typical neighborhood agencies compared to those that are formed by neighborhood residents and described as grassroots in nature. It is important to understand these differences as they often lead to different goals, objectives, and strategies for change (Chavis, 2001).

Coalition Structures and Processes

Member Competency

Coalitions work to produce community change by connecting neighborhood residents, organizations, and institutions together in a manner that their collective efforts are much greater than individuals or organizations working in isolation. In other words, a coalition’s effectiveness is heavily dependent upon individuals’ and organizations’ skills, knowledge, assets, and resources and coalition members are regarded as its primary asset (Butterfoss et al., 1993). While this capacity can be enhanced through education, experience, coaching, and training and technical assistance (Florin, Mitchell, Stevenson, & Klein, 2000; Foster-Fishman et al., 2001), it nonetheless represents a coalition’s baseline or starting point in terms of its ability to foster and create community and systems change. Initial assessment of members’ skills and knowledge is often part of the needs assessment process, along with a broader assessment of the larger community. In fact, a comprehensive needs assessment process often increases members’ skills and knowledge as they learn firsthand about prevalence of youth drug use and ATOD needs and prevention resources. It is also the first step in the strategic planning framework (SPF) process, which is the community planning process used by all DFC coalitions to plan, implement, and evaluate their activities and efforts toward substance use reduction.

As the research and practice literature emphasize, participating on a coalition often means dealing with conflict among members and organizations due to differing priorities and perspectives. A healthy coalition requires that individual members make compromises for the benefit of the coalition’s overarching vision, mission and goals. In fact, one of the major review articles used to inform our selection of constructs and variables identified active member involvement as one of six factors that were associated with coalition effectiveness (Zakocs & Edwards, 2006). Past experience with working on coalitions, degree of member involvement, and positive member relationships provide individuals with experience in negotiating these difficult decisions and processes for the greater good. Currently, COMET queries coalition leaders to report on the number of active members in the coalition by the required twelve sectors. We propose adding another column and simply asking the coalition leader to report on the degree of active involvement by sector. This simple addition to COMET provides a much more accurate assessment of involvement, not just member attendance at meetings. The addition of this item also allows for the possibility of assessing a typology of active member involvement. In other words, is there a right mix of players or do certain sectors need to be actively involved for healthy coalition functioning, and is this associated with impacts on systems and community change? Given the focus on at least twelve sectors, this addition would allow us to investigate if sectoral representation of all sectors is needed, how much, and at what levels; or, if other dynamics like leadership are the key to effective coalition functioning.

While nobody denies the importance of member relationships in coalition functioning, this variable is difficult to operationalize and measure, outside of more in-depth qualitative and case study methodologies. We plan on identifying the importance of member relationships by conducting social network analyses in each of our 36 case study sites over the course of the initiative. The utilization of this innovative methodology in coalition research has led to the identification of the relations required to bring about community change (Nowell, 2009). This analysis also provides us a means to delve greater into quality of collaboration, a measure often assessed by counts of MOUs and other methodologies that do not truly assess this important variable. For instance, we plan on asking members to rate each other on (1) communication frequency, (2) shared philosophy, (3) responsiveness to concerns, (4) legitimacy, (5) trust in follow-through, and analyze whether higher degrees of true collaboration are associated with higher coalition functioning and positive community changes. Also, as mentioned in the prior section, given the importance of past relationships and participation in related initiatives, we plan on including this as a question in the network analysis to better understand how community and coalition context affect member relationships.

Coalition Structure

An important component of any coalition to act effectively in the community is having an array of diverse stakeholders that represent all constituencies and members of the community. In fact, to ensure this adequate representation, DFC requires participation from twelve community sectors, including education, law enforcement, youth, and community leaders among others. This requirement is strongly supported by the literature as membership breadth and diversity provides members with access to a wide array of resources to build collaborative capacity and enhances credibility with the local community (Butterfoss et al., 1993; Florin et al., 1993). Recruitment of a diverse membership base is one of the first significant tasks coalitions encounter, and, often requires considerable attention and resources to maintain and sustain throughout the lifecourse of coalition. This is particularly the case with youth, the target population of coalition efforts, and their engagement and sustainability in efforts is truly needed to bring about community change. Tracking the diversity of membership can also assist in identification of possible technical assistance and training needs to ensure coalition membership is representative of, and credible with, the targeted community.

As coalitions grow and mature, they often institute a number of formalized structures and processes to increase their functioning and capacity for community change (Zakocs & Edwards, 2006). This column or component of our internal model reflects this increasing capacity and formalization. In terms of coalition structure, this includes the implementation of a leadership structure and often a number of workgroups/committees and other formalized structures to be responsible for completing work outside of coalition meetings. This formalization is often guided by the completion of strategic and action plans that specify goals, objectives, activities and coalition members responsible for ensuring the successful completion of coalition work. Implementation of such governance and procedural structures has also been cited in collaborative literature reviews as one of the main factors in coalition effectiveness (Roussos & Fawcett, 2000; Zakocs & Edwards, 2006). The degree of formalization also often depends on the local community context and history of collaborative efforts in the community. For instance, while coalitions with formalized structures have been associated with improved functioning and outcomes, some communities and coalition types (grassroots versus institutional or agency-based) may feel this degree of formalization is unwarranted and possibly even an unhealthy approach to community change efforts. It is these types of community and contextual elements that must be understood when implementing, and evaluating, community change efforts.

Coalition Processes

As highlighted in reviews of the field, collaborative structure and process variables are the most heavily researched and understood components of coalition effectiveness. Reviews have indicated the importance of establishing effective coalition processes, such as: (1) having a clear vision and mission, (2) leadership, (3) role clarity, (4) group cohesion, and (5) formal governance procedures among others (Israel, Schulz, Parker, & Becker, 1998; Roussos & Fawcett, 2000). However, at the same time, one of the most valid criticisms of coalition research is the degree to which collaborative processes and dynamics (shared vision, leadership, conflict management, shared decision making) are tied to coalition functioning rather than the actual linking these mechanisms to community changes (Zakocs & Edwards, 2006).

Our current measurement and analysis plan focuses heavily on better understanding these variables by creating overarching metrics that move beyond analysis of individual processes and tying these larger constructs to community and systems changes. For instance, Allen (2005) created an overarching climate construct that included (1) effective conflict resolution, (2) presence of a shared mission, (3) shared decision making, and (4) task oriented and inclusive leadership. Additionally, given the large number of variables, aggregation to higher-level constructs is the only meaningful way to analyze data in a parsimonious manner that can actually lead to implications for policy and practice for DFC. Finally, it should be noted that where we identified important missing structure and process variables (e.g., leadership), we have included them in our instruments, in particular the CCT.

An integral component under coalition processes includes the utilization of the strategic planning framework (SPF), which focuses coalition activities into five distinct components (i.e., assessment, capacity, planning, implementation, and evaluation), while also acknowledging the importance of sustainability planning and cultural competence in each phase or framework component. As part of our evaluation, we plan on attempting to better understanding how the SPF is linked to coalition functioning and community change. This is one of the reasons we recommend to continue to include data entry into COMET based on the components of the SPF, despite some evidence from our qualitative analysis of dropdown boxes that coalition members did not completely understand the differences between all components of the framework. This finding from our qualitative analysis is also the reason we propose highlighting definitions to a greater extent and providing examples of what constitutes SPF activities prior to data entry. This will increase the quality of data collected and also serve as a continuous education process for the SFP framework and essential coalition processes. We feel an evidence-informed community change framework is needed to educate coalition members regarding the iterative processes of community and systems change. Meanwhile, the role of evaluation is to attempt to unpack how SPF processes lead to change. For instance, we can utilize an interrupted time-series design to examine the effects of each SPF component (e.g., assessment) on the type and scope of community changes (e.g., increased awareness of ATOD use). This type of innovative analysis has been recently used to better understand the strategic planning process and how differential outcomes are achieved depending upon community and coalition context, and degree and intensity of community planning (Watson-Thompson, Fawcett, & Schulz, 2008).

Strategies and Activities

The strategies and activities column is perhaps the most straightforward of the internal model as it focuses on the explicit strategies and activities implemented in the community to increase collaboration and to decrease youth substance use. While there has been an exponential increase in evidence-based “clearinghouses” across a number of fields in recent years (e.g., SAMHSA’s National Registry of Evidence-Based Programs and Practices, DOE’s What Works Clearinghouse, OJJDP’s model programs), broader dissemination and implementation of these certified programs and practices often do not meet with the same success as initial results, which were often established in a small number of settings and with much higher levels of funding. In fact, effective implementation of evidence-based programs and practices has been limited across all fields, in particular community-based programs and services (Arthur et al., 2010; Durlak & Dupre, 2008; Julian, Ross, & Partridge, 2008).

One of the main goals of ICF’s evaluation is to strengthen attribution between process (i.e., what coalitions do and how they do it) and outcomes (i.e., results of their efforts). This is the reason such effort went into revisiting and revising the National Evaluation logic model, and perhaps more importantly, development of a specified internal model that required an item-by-item intensive review of every data source and variable. Another major related goal of the National Evaluation is to deconstruct DFC programs and activities to identify best practices – what works, why it works, and in what situations it works. In essence, while there has been tremendous growth in identifying effective prevention programs, there needs to be an equal amount of effort in identifying how to effectively implement such programs across a diverse array of community settings. We intend to fill this void by focusing on documenting the different types of strategies and dosage of activities required to result in community-level changes in youth substance use rates. By building the evidence base between process and outcomes, we intend to provide communities with the information needed to bridge science to effective community-based practice.

Coalition Role

As previously discussed under coalition context, coalitions often have more of an institutional or agency-based focus or are more grassroots in nature and led by local community leaders or those most affected by the issue the coalition is addressing. The different types of coalition members and make-up are important because each brings a different perspective toward the best way to focus coalition efforts toward reducing substance use (Chavis, 2001). DFC purposely combines an agency and grassroots approach by requiring the membership of twelve sectors, including a mix of agencies and institutions (schools, law enforcement) and local parents, youth, and youth-serving organizations. While DFC coalitions often engage in a number of roles in their planning, implementation, and evaluation of substance use reduction efforts, they generally focus more intensely on some roles than others. For instance, some coalitions focus more on coordinating community-based efforts of prevention programs or services while others focus more on environmental or policy change. Other roles identified in the internal model include direct implementation, advocacy, initiation/support, member contributions, and external partnering. Currently, the CCT collects information pertaining to coalition focus and roles and this data can be used to assess if certain types of coalition roles work more effectively than others, or employ a different mix and strategies and activities. We also plan to utilize the case study component to delve deeper into the relationship between membership and coalition focus, role, selection of strategies and activities, and what relationship, if any, this has to community changes and outcomes.

Programmatic Capacity and Strategy/Activity Mix

Programmatic capacity generally refers to the processes and outcomes of the assessment, capacity building, and planning stages; that is, the development high-quality strategic and action plans that outline the key goals, objectives, and activities to achieve identified objectives in a realistic timeframe. If coalitions adequately plan and prepare for the development and implementation of evidence-based programs and practices, they will be much more effective in this stage. For instance, the major reviews of coalition research have demonstrated the importance of action planning, establishing clear and realistic objectives, as well as infusing cultural competence in all prevention and intervention activities (Roussos & Fawcett, 2000; Zakocs & Edwards, 2006).

DFC coalitions are guided in their efforts by seven overarching strategies to achieve community change, which include: (1) providing information, (2) enhancing skills, (3) providing support, (4) modifying/changing policies, (5) changing consequences, (6) enhancing access/reducing barriers, and (7) physical design^⁷. Another broad category, strengthening coalitions, captures how the coalition develops its own capacity and capability to accomplish the seven community change strategies. These change strategies operate along a continuum and target varying levels of intervention from more general information sharing and education to targeting changes at higher levels, such as changing consequences (e.g., increasing compliance and enforcement of tobacco sales), changes in physical designs of communities (e.g., outlet density, lighting), and changes to policies and practices at multiple levels (city, county, State). Given the multi-causal nature of ATOD use and related problem behaviors, community coalitions need to utilize a mix of multilevel strategies in order to effectively combat the various influences and pressures that can negatively affect our youth (Butterfoss & Francisco, 2004; Foster-Fishman & Behrens, 2007; Kreger, Brindis, Manuel, & Sassoubre, 2007). Thus, the type(s) and intensity or dosage of prevention and intervention activities are important mediators between effective planning and positive community outcomes (Beebe, Harrison, Sharma, & Hedger, 2001).

These strategies are currently assessed in the COMET database system and we intend on keeping all categories but modifying data entry to decrease burden and increase the collection of information related to strategy mix and activity type, scope and dosage. As discussed throughout the main text, the current COMET system is extremely burdensome because of the extensive linking involved, all of which should be already captured in strategic and action plans. For instance, when adding a new objective, COMET requires linking this objective to: (1) strategies to achieve objectives, (2) percent coalition efforts and resources into achieving objective, (3) date objective established, (4) link objective to targeted risk factor, (5) link objective to targeted protective factor, (6) link objective to DFC core outcome measure, (7) link objective to targeted substance, (8) link objective to targeted grade, (9) link objective to targeted gender, (10) targeted date for achieving objective, (11) rating to what extent objective was achieved. Additional details are requested at the activity level, such as status, date started, and date complete for each activity further increasing burden. We propose de-linking this extensive chain and focusing more on what is important: number of activities (by strategy), who is involved, at what level, total number of coalition hours involved, scope of activity (# served), dosage of activity, and impact of activity. Please see Appendix C for more details.

Coalition and Community Outputs

We have included coalition and community outputs in this column to emphasize the importance of celebrating the small wins that occur while coalitions are working to change larger community and macrolevel variables. Our experience has taught us the importance of taking seriously the documentation of all activities and other community outputs, and even graphically displaying the number of activities and other accomplishments the coalition has engaged in over time. This is because one of the biggest challenges people cite when working on coalitions to address larger social issues is simply time – the amount of time it takes to produce visible change at the community and individual levels^⁸. In our prior work, while logic modeling helps visualize the bigger picture, it is also helpful to break down longer-term changes by creating activity models, which visually display, at a more concrete and discrete level, how each activity builds upon another and may ultimately produce community and population-level change. For DFC, examples could be producing activity models around each of the core outcomes, or for each of the identified risk and protective factors. Of course, this variable also includes more traditionally conceived outputs such as the quality of the strategic and action plan, social marketing materials, development of curricula, etc. Particularly for coalitions with local evaluators, documenting these activities and showing members all the work they are doing will keep them motivated and participating over the long-term change process involved in coalition work.

Coalition Capacity

Coherence and Coalition Climate

The coalition capacity column of our internal logic model reflects the need to build on the most important assets of coalitions – the voluntary membership of individuals, organizations, institutions, agencies, and community members. The five constructs of coalition capacity include: (1) coherence, (2) coalition climate, (3) positive external relations, (4) capacity building effort, and (5) continuous improvement. First, while logic models are almost always displayed in a linear fashion, it should be recognized the often considerable overlap of stages, which are represented by columns in our model (Florin, Mitchell, Stevenson, & Klein, 2000). For instance, there is considerable overlap between coalition structures and processes and coalition capacity. This is due to the fact that this column identifies how individual member capacity and coalition capacity, which includes most of the structures and processes already discussed, need to be developed to increase the effectiveness of coalition activities and efforts in the community. In essence, the coalition structures and processes column describes where communities are at the beginning or baseline of their efforts, while the coalition capacity column attempts to capture what has been accomplished in building on the coalition’s initial functioning.

The importance of establishing a shared vision and mission for coalitions has been identified by the majority of collaborative research, particularly by research syntheses and reviews (e.g., Foster-Fishman et al., 2001). Indeed, in terms of strategic planning, one of the major first steps is to use needs assessment data to inform the development of a shared vision and mission of the coalition. Our own prior research on community-based coalitions has indicated that a shared vision keeps members committed during challenging or difficult times and also facilitates the integration of new members. This is vital given the inevitably of member turnover during the course of a community change initiative. As discussed, in order to be an effective change agent, coalitions need broad and diverse membership from all community sectors. This not only establishes credibility but access to a larger number of resources and perspectives. Unifying these different perspectives is required and the first step toward a coordinated group effort is buy-in to the overarching goals and vision of the coalition.

Commitment by coalition members is also vital for successful community change efforts. Research suggests that targeted community level changes often take 3-10 years to become detectable; broader population-level change can take even longer (Roussos & Fawcett, 2000). Thus, commitment and satisfaction are vital to engage and sustain members in such long-term change efforts. Perhaps even more vital is assessing commitment and satisfaction during coalition efforts so that adjustments can be made and technical assistance sought if there is a drop in this important variable – before members “vote with their feet” by leaving the coalition. Thus, we have proposed including a short measure of member satisfaction in the CCT. Inclusiveness is an indicator of coherence and represents how inclusive and diverse coalition membership is throughout the course of coalition change efforts, which has been identified as a major factor explaining coalition effectiveness (Zakocs & Edwards, 2006). For DFC, this includes substantial community representation, in particular youth, and the rationale for us including an extra item regarding youth on the pre-existing cultural diversity scale. We also plan on combining a number of coalition climate variables (e.g., trusting, shared input) and assessing how such a metalevel construct is associated with coalition functioning and effectiveness.

Positive External Relations

For a coalition-driven community change effort, environmental changes will not occur unless members and organizations establish credibility and enjoy consistently positive relations with the larger community, including the target population (e.g., youth) (Berg, Coman, & Schensul, 2009). This is particularly important at the beginning of community change initiatives, and research has suggested the importance in establishing “small or quick wins” (i.e., small scale projects that produce quick and visible results) to help promote community-buy in and jump-start larger activities that will involve greater access to community resources (Deacon, Foster-Fishman, Mahaffey, & Archer, 2009).

Despite the fact that researchers, and, especially practitioners recognize the importance of external partnerships, little attention has been paid to this variable in evaluation efforts, except possibly for case studies. While we plan on developing protocols to capture this in the case study component (e.g., community leader interviews, community focus groups), we also have included an interorganizational coordination measure in the CCT. This variable asked members to rate how coalition efforts have increased coordination of their efforts and how organizations now view themselves as a larger system addressing the issue of drug use in the community instead of separate silos addressing specific domains (e.g., enforcement, treatment). Currently, there are a number of areas within COMET that obtain data on the number of activities in the community, and we have proposed including more ratings of whom in the community was targeted by each activity (scope), the number of people who participated (penetration), and the degree of success and impact of the activity. This type of data can accurately gauge coalition activity focus and participation of non-coalition member agencies, and assess if this is associated with differential outputs and outcomes.

Capacity Building Effort & Continuous Improvement

In a major effort and review the collaborative literature, Foster-Fishman et al. (2001) synthesized the core competencies and processes of coalitions into an overarching capacity building framework. According to the framework, there are four distinct levels of capacity to build for sustainable community change efforts – member capacity, relational capacity, organizational capacity, and programmatic capacity. Member capacity includes the core skills, knowledge, and attitudes that individual members bring with them to the table. For DFC, this would be an individual’s awareness and knowledge of substance use in the targeted community, availability of ATOD in the community, and consequences of ATOD use. This level also includes an individual’s or organization’s ability to work collaboratively with members of the community and different organizations, ability to create and guide and implement evidence-based programs for substance youth prevention, and the ability to build an effective coalition infrastructure for change.

Relational capacity involves the development of a positive internal working climate and positive external climate. As previously discussed, a positive coalition climate includes the development of an open and honest environment where people feel free to share information and input and conflict is handled in a transparent and fair manner. Such a positive climate facilitates the development of common understanding of community assets and problems, promotes the development of a shared vision, mission, and solutions. The importance of comprehensive planning and establishment of a shared vision to describe a coalition’s overarching goals is vital to keeping members focused on the ultimate goals of their collaborative activities and efforts (Bartunek et al., 19996; Roussos & Fawcett, 2000). While coalitions need active involvement by its key members during meetings and other coalition work, participation by non-member agencies is critical if community-based activities and interventions are going to successful (Roussos & Fawcett, 2000). In fact, the involvement of all community sectors and members is so critical to coalition success that we created a separate construct entitled Positive External Relations in our internal model as discussed above.

Organizational capacity includes coalition processes that are more formalized in nature, such as detailed action plans, establishment of clear roles and responsibility, strong leadership, and effective internal and external communication. Many of these variables and constructs were highlighted earlier under coalition processes, procedures, and structures and this column recognizes the need to build on these constructs. The importance of leadership in producing an internal and external climate cannot be overstated as research has indicated its importance to coalition effectiveness (Allen, 2005; Zakocs & Edwards, 2006). While community leadership was well measured by the CCT, we noticed that coalition leadership measures were lacking. Given the central importance of leadership, we have included three short measures that assess the multiple levels of leadership common in coalition efforts (i.e., coalition leadership, committee leadership, and coordinator leadership). The leadership scales were originally adapted from Butterfoss’s (1998) coalition effectiveness self-inventory tool and have been used to study coalitions that work to reduce and prevent domestic violence (Allen, 2005). It should also be noted that there are individual leadership items spread out in other CCT scales, such as confidence in task completion that can be used to augment these proposed scales or create a larger leadership metric. Finally, also discussed earlier under strategies and activities, programmatic capacity involves clear and realistic programmatic objectives, ecologically valid prevention and intervention strategies, and the utilization of data to inform programming and any necessary adaptations to achieve desired results.

In sum, research has demonstrated the importance of building capacity at all these levels, and how such capacity building efforts can lead to more effective coalitions and community change (Florin, Mitchell, Stevenson, 1993; Florin, Mitchell, Stevenson, & Klein, 2000; McMillan et al., 1995). In fact, in a thorough review of the literature, many of the indicators of capacity (e.g., shared vision, leadership, resources, technical assistance and support) were found to be linked to enhanced partnerships and improved population-level health outcomes (Roussos & Fawcett, 2000). Thus, capacity building efforts must be continuous and must target multiple levels (member, coalition, organizational, and programmatic) if coalition efforts and activities are going to successfully address and community conditions and outcomes.

Coalition Effectiveness

Community & System Outcomes

The coalition effectiveness column represents the major outputs, impacts, and outcomes of coalition efforts to increase collaboration and decrease youth substance use. These outcomes are listed separately at both the system level and at the behavioral level to visually display the multiple levels at which DFC coalitions operates and intends to achieve outcomes. The fact that DFC targets multiple levels (e.g., community, school, individual) to reduce youth substance use is in accordance with current prevention science research that suggests multilevel interventions are not only more successful, but also more sustainable (Trickett & Schensul, 2009; Weeks et al., 2009).

DFC utilizes a risk and protective factor targeted approach in their effort to reduce substance use. This approach helps stakeholders understand that it is integral to target and build assets of youth, as well as, target the more traditional risk factors. As the research literature highlights, negative behaviors such as drug and alcohol use are usually not found in isolation, but are bundled with other negative behaviors such as poor educational outcomes and delinquency (Catalano et al., 2002; Coulton, Korbin, SU, & Chow, 1995). This approach also encourages communities to collect data (i.e., assessment) to find out current drug use problems and then tailor prevention activities to the local community context of risk and protection (Hawkins et al., 2008). This is extremely important as every community is unique and even evidence-based programs need to be adaptable and tailored to community conditions if they are going to be successful outside of well-funded demonstration grants and trials. It is also in line with research on asset-based community development and environmental strategies for change, which indicate one of the best ways to mobilize the community is to focus on assets and not solely on problems (McKnight, J.L. (1996). Given the research findings that have indicated positive impacts on precursors to and delay of negative youth behaviors, particularly when provided with training and technical assistance (Feinberg et al., 2002; Hawkins et al., 2008; Manger, Hawkins, Haggerty, & Catalano, 1992), we suggest continuing to use these measures (and framework). However, we have suggested an enhancement to the on-line system (COMET) that goes beyond a check-all approach. For instance, with needs assessment data, even if communities are targeting a host of risk and protective factors, they are certainly targeting some more than others. This is why we suggest, at a minimum, ranking in order the top five protective and top five risk factors. In this manner, analyses can group communities by these targeted factors and correlate what strategies, activities, and dosage are effective in reducing risk or promoting protection, and ultimately, reducing rates of youth substance use. Currently, there is little discoverable information with a check-all approach to data entry.

Increasing collaboration is one of the main goals of DFC, yet past research has not included enough measures that truly evaluated this overarching goal of the program. In response, we have included a number of measures to better address this goal, and, moreover, to understand the link between collaborative efforts and community changes and outcomes (e.g., decreased youth substance use rates). As previously discussed, one of ways we address this gap is to utilize social network analysis at our case study sites. This innovative methodology has been increasingly used to understand coalition functioning and how it is associated with various degrees of collaboration (Friedman et al., 2007; Nowell, 2009). Social network analysis will also allow us to study whether, for instance, the density of networks among collaborative members leads to increased coordination of programs and implementation of environmental strategies that target the various levels of risk and protective factors (individual, family, school, and community). Another measure we propose including is interorganizational coordination. This measure assesses the degree to which coalition members are increasingly working together to address youth substance use and has been found to be an outcome of high functioning coalitions (Allen, 2005). Another proposed measure, organizational impact, assesses how coalition efforts are creating changes within the various child-serving organizations and agencies. That is, this measure asks members to report on how participating in coalition activities has assisted them in changing policy, procedure, and practice within their own organizations. Taken together, these measures go well beyond typical assessments of collaboration, and will provide DFC with information on how the overall quality of collaboration matters.

In terms of community behavioral outcomes, we have not made any changes to the Core outcomes to ensure comparability with historical data. We do realize that some of these indicators may change or additional indicators included, depending on what public use data can be gathered that assesses population-level outcomes beyond substance abuse. As proposed in our logic models, there are likely to be impacts on these behaviors as broad-based community change initiatives are theorized to have an impact on correlated behaviors and outcomes. An emphasis on this type of data also helps DFC communities realize that the goals of program are much broader than reducing drug use and also includes the prevention of other problem behaviors, as well as the promotion of community health and well-being. It is with this latter category that we believe DFC may have, as of yet, unrealized impacts outside of anecdotal information. For example, in addition to being a measure of context, we also look at community social organization and community readiness and capacity for change as potential indicators of community health. That is, do DFC activities, over time, help increase the social organization, cohesion, and health of communities? This is serious area to assess since social organization and related variables (sense of community, social capital, informal social control, and collective efficacy) have been found to be associated with lower rates of violence, delinquency, and drug use (Beyers, Bates, Pettit, & Dodge, 2003; Brooks-Gunn, Duncan, Klebanov, & Sealand, 1993; Elliott et al., 1996; Sampson, Raudenbush, & Earls, 1997). Or, as discussed under context, does the initial level of community readiness and capacity influence the degree of implementation of DFC and lead to better outcomes. That is, do those communities who start out higher in terms of readiness and capacity end up with better outcomes than their less “ready” peers? To close, with the proposed enhancements to the data system, proposed new variables, metrics, and analyses, we believe we can help move prevention science forward to better understanding what are the key strategies and activities to implementing effective community-based drug prevention initiatives.

Appendix B: COMET Data Collection Plan

ASSESSMENT SECTION
Field Name	Data Type	Description	Keep (Keep, Delete, Modify, or New)	Rationale
		Each section will begin with a definition of the SPF component (e.g., assessment) along with a list of what typically involves assessment activities, challenges, and accomplishments (and maybe examples of what should not be included in the section). Each section will also end with a text box for respondents to enter any final thoughts or innovative practices that have implemented that were not captured in COMET.	New	A thorough review of COMET established that many respondents are entering information in the wrong sections. For instance, listing evaluation activities in assessment or highlighting planning accomplishments in assessment.
		Date Updated	Keep
Community Needs Assessment^⁹	Text	Target Community Name	Keep	Basic coalition information
	Check one or more	Target Geographic Area (Urban, Suburban, Rural)	Keep	Catchment areas are very important for aligning data from other public use data files.
	Check one or more	Target Specific Geographic Area Multiple School Districts Single School District Native American/American Indian/Alaskan Native Reservation Neighborhood Multiple Neighborhoods City Multiple Cities Town Multiple Towns County Region or other subsection of a State	Keep	Provides contextual information about catchment areas.
	Enter numeric values (Add as many as apply)	Target Zip Codes	Keep
	Check one or more	Target Gender	Keep
	Check one or more	Target Grade	Keep	Multiple grades should be more easily selected. Consider the use of check boxes.
	Check all that apply	Substance of Issue in the Community CORE OUTCOMES Alcohol Tobacco Marijuana Prescription Drugs (Ritalin, Adderall, Xanax, etc.) OTHER TYPES NOT LISTED HERE (Allow the entry of other substance names by free text format --- up to three items) OTHER DRUGS Ketamine (Super K, Special K) Amphetamines (stimulants, Speed, etc.) Cocaine [including crack] Heroin Hallucinogens (other than LSD; Mescaline, Peyote, Mushrooms or “shrooms” etc.) LSD (acid) Narcotics other than heroin Tranquilizers Nitrites PCP Rohypnol (date rape drug, Roofies, ruffies, etc.) Barbiturates Methamphetamine (including Ice, Crystal, Crank) MDMA (Ecstasy) GHB (Liquid G) or GBH Over-the-counter (OTC) drugs Inhalants (including Amylnitrite, glue, cleaning fluid, gasoline etc.) Steroids OTHER TYPES NOT LSITED HERE (Allow the entry of other substance names by free text format --- up to three items)	Modify	We classified drug types into a smaller number of groups to reduce respondents’ burden. We referenced other drug surveys, such as PRIDE Surveys and ADAS. Instruction: Users will click on the broad substance categories and the list of specific names will expand for users to click/check. Within broad substance categories, drugs will be listed alphabetically.
	Text (500 Character Limit)	Please further describe the Geographic Area(s) selected above (e.g., names of cities, counties, etc.)	Keep	Although entering target community name and zip codes should be sufficient, we need as specific catchment information as possible. This will be important for developing a sampling plan for each grantee and for providing guidance to grantees on sampling.
	Choose one	Have there been other substance abuse prevention coalitions or groups targeting some of the same communities and risk and protective factors that your coalition is targeting? Yes No Unknown	Delete. Combine with the next question.	Combine the two questions to reduce respondents’ burden.
	Choose one	How many other non-DFC coalitions are working to prevent substance abuse or addressing the same risk and protective factors in the targeted community? N/A 1 other coalition 2-3 other coalitions 4-5 coalitions more than 5 coalitions	Delete. Combine with the next question.	Combine the two questions to reduce respondents’ burden.
	Enter a numeric value	Approximately, how many groups, excluding your own organization, are working on the prevention of substance abuse or addressing the same risk and protective factors in your community?	New	These questions help us evaluate whether the effect of DFC on outcomes is different in the areas served by DFC alone and the areas served by a mix of other prevention groups. These would also provide us with social network analysis variables. For example, the items here help measure the density of supportive collaboration and shared activities among groups that share the same areas of service. By DFC regulation, coalitions are not generally expected to include overlapping service areas and thus asking if there are other groups “in your community” may appear to contradict the regulation. However, this question is important for analysis and evaluation purposes as it helps differentiate the areas served only by a coalition and the areas served also by other entities. We propose the word “in your community” be kept, but focus on non-DFC coalitions.
	Enter a numeric value	In the past 6 months, how many of these groups (excluding your own DFC partners/members) have you worked with to reduce substance abuse in your community through shared activities and events, such as co-sponsoring substance abuse prevention activities?	New
	Text (no limit)	Assessment Summary and Key Findings	Keep	Although this will be difficult to analyze, (i.e., data are not categorized) we need to determine whether, realistically, these data are updated beyond what is in the grant application. Categorization of assessment text data revealed confusion among grantees of what is an assessment activity. Specifically, there is confusion between assessment and evaluation activities. Definition of “assessment” should be available in this section as a pop-up menu as noted in the first row along with examples of what constitutes assessment activities and what does not..
	Text	Overall, is there anything else you would like to include regarding your community needs assessment process?	New
Risk and Protective Factors (Users can add as many factors they choose to)	Check Yes or No	Is this the main model that you use?	Modify: To what extent does your DFC coalition use a risk/ protective factor approach in your prevention efforts? (5-point Likert scale (Never, Little, Somewhat, Much, A Great Deal)	This new question captures: (a) whether the DFC coalition uses a risk- and protective-factor-based model, and (b) how important the model is for the coalitions. It will also help us assess the association between the use of risk/protective factors approach and successful reduction in substance use.
	Check one	Type of risk and protective factors Risk Protective	Delete	Respondent burden is heavy. The historical DFC data shows that most coalitions have provided information; however, their data may not be complete. There is no way of knowing what really existed if the coalitions did not enter full information. Also judgment required here is highly subjective. See new proposed items below. Simplified Risk/Protective Factors and Order Them By Domain. Note: the factors were based on Hawkins & Catalano’s classification system.
	Check one	Factor to Target (For RISK factors): Community - Availability of substances that can be abused Community - Perceived acceptability (or disapproval) of substance abuse Community - Poverty Community - Transitions and mobility Community - Racism and discrimination Community - Poor family management Community - High family conflict Family - Family trauma/stress Family - Mobility of family Family - Abuse and neglect Family - Family history of antisocial behavior Family - Parental attitudes favorable to antisocial behavior	Delete
		Factor to Target (For Protective Factors): Community - Laws and policies Community - Level of community organization (e.g., less crime, less visible drug dealing) Community - Advertising and other promotion of ATOD Community - Enforcement of laws and regulations Family - Family economic resources Family - Acculturation Family - Parental monitoring and supervision Family - Family bonding Family - Opportunities for pro-social community involvement Family - Rewards for pro-social community involvement Family - Opportunities for pro-social family involvement Family - Rewards for pro-social family involvement Family - Family history of successful socialization	Delete
	Text (no limit)	Description	Delete
	Check one	Trend is (Improving, Staying the Same, Worsening, No Trend Data)	Modify and combine with the new item below.	Integrate this with the new item below. Make “No trend data” as first response (to avoid users to make a guess that is not based on trend data). Then, above improving, stay the same, worsening; we could have a header "Trend Data Indicates".
	Check all that apply	What are the major risk factors that your coalition is targeting for improvement for youth in your community? Check all that apply and indicate which are your coalition’s priorities (‘Top 5’ Items). For the ones you chose, indicate whether information to indicate the trend for the factors is available and, if so, whether the trend is (Improving, Staying the Same, Worsening). Community - Availability of substances that can be abused Community - Perceived acceptability (or disapproval) of substance abuse Community - Poverty Community - Transitions and mobility Community - Racism and discrimination Community - Poor family management Community - High family conflict Family - Family trauma/stress Family - Mobility of family Family - Abuse and neglect Family - Family history of antisocial behavior Family - Parental attitudes favorable to antisocial behavior Individual - Alienation and rebelliousness Individual - Friends who engage in the problem behavior Individual - Favorable attitudes towards the problem behavior Individual - Early initiation of the problem behavior School - Early and persistent antisocial behavior School - Academic failure beginning in elementary school School - Low commitment to school	New	This style of question will reduce respondents’ burden. It also allows users to identify priority items. We added family and individual factors to be more comprehensive than the original set of factors. Individual- and school-level factors (identified by Hawkins & Catalano) are now added.
	Text (1000 Character limit)	Is there any additional detail that you would like to provide so we can better understand your targeted risk factor approach?	New
	Check all that apply	What are the major protective factors that your coalition is targeting for improvement for youth in your community? Check all that apply and indicate which are your coalition’s top priorities (‘top 5 Items’). Community - Laws and policies Community - Level of community organization (e.g., less crime, less visible drug dealing) Community - Advertising and other promotion of ATOD Community - Enforcement of laws and regulations Family - Family economic resources Family - Acculturation Family - Parental monitoring and supervision Family - Family bonding Family - Opportunities for pro-social community involvement Family - Rewards for pro-social community involvement Family - Opportunities for pro-social family involvement Family - Rewards for pro-social family involvement Family - Family history of successful socialization Individual - Meaningful opportunities to contribute to the peer group Individual - Skills to successfully take advantage of those opportunities Individual - Recognition/acknowledgement of efforts School - Meaningful opportunities to contribute to the school community School - Skills to successfully take advantage of those opportunities School - Recognition/ acknowledgement of efforts	New
	Text (1000 Character limit)	Is there any additional detail that you would like to provide so we can better understand your targeted protective factor approach?	New
Assessment Activity	Text	Activity Name	Delete	Too much burden on respondents.
	Date	Date started	Delete	Too much burden on respondents.
	Date	Date completed	Delete	Too much burden on respondents. We want to ask grantees about the degree or intensity level of DFC efforts on specific assessment activities. Dropdowns with "groupings" and have them rate how often they engaged in these activities using a Likert scale. Based on qualitative analysis of randomly selected pre-existing data, dropdown categories were created.
	Check one	Activity Status Active Inactive Planned Completed Discontinued	Delete
	Text (1,000 character limit)	Description of activity	Delete
	Check one	How often does your coalition engage in the following assessment-related activities?	New
Assessment Progress	Text (200 Character Limit)	Progress Name	Delete	Too much burden on respondents. No substantive purpose for the evaluation.
	Choose reporting period	Reporting Period Identified (2005S1, etc).	Delete	Too much burden on respondents. No substantive purpose for the evaluation. The current system lets users choose relevant reporting periods for which data are entered. The availability of this option could be confusing as respondents reach this page mostly to enter information for a current period. To avoid confusion, users should not be given an option to revisit the data entered in the earlier reporting periods. Dropdowns with "groupings" and have them rate the ‘degree to which each item was a (1) Significant Challenge to (5) Significant Accomplishment. Based on qualitative analysis of randomly selected pre-existing data, dropdown categories were created and are listed in another appendix. This will provide much more useful information for the evaluation of DFC.
	Choose one	Type Accomplishment Challenge/Barrier	Delete
	Text (3,000 Character Limit)	Description of Progress	Delete
			New
	Check all that apply	What accomplishments has your coalition achieved during the past 6 months while performing activities related to assessment?	New	Dropdowns with "groupings" and have them rate the ‘degree to which each item was a (1) Significant Challenge to (5) Significant Accomplishment. Based on qualitative analysis of randomly selected pre-existing data, dropdown categories were created and are listed in another appendix. This will provide much more useful information for the evaluation of DFC.
Specific Examples of Challenges	Text (3,000 Character Limit)	Please report specific examples of challenges that your coalition has experienced during the past 6 months.	New	Choose-from option style questions may miss specific incidences of achievement or challenges that a coalition may want to report.
Specific Examples of Accomplishment	Text (3,000 Character Limit)	Please report specific examples of accomplishments that your coalition has experienced during the past 6 months.	New	Choose-from option style questions may miss specific incidences of achievement or challenges that a coalition may want to report. Question may be particularly valuable for grant monitoring.

CAPACITY SECTION

Field Name

Data Type

Description

Keep (Keep, Delete, or New)

Rationale

Coalition Membership

Choose One

Member Type

Individual
Organization

Delete

Entering each coalition member poses too much burden on respondents (see new proposed items below).

Project officers need to know if coalitions are compliant with DFC regulations by including members from all 12 sectors.
We propose that coalitions report the number of active members from the twelve sectors and then rate an average degree of involvement by each sector.
This will provide us with quick numbers for quantity of representatives by sector and quality of involvement/

participation.

Text (2,000 Character Limit)

Member Name, Either Organizational or Individual

Delete

Choose One

12 Required Sector Representatives:

Parent
Youth

Business community
Civic and volunteer group
Healthcare professionals
Law Enforcement agency
Media
Religious or fraternal organization
School
State, local, and/or tribal governmental agency
Youth-serving organization
Other organization involved in reducing substance abuse

Delete

Choose One

Membership Status (defined as attending one meeting in the past year)

Active
Inactive

Delete

Numeric

Total Number of Current Representatives

Delete

Numeric

Number of Representatives Active in Coalition Meetings, Activities, and Tasks

Delete

Numeric & Rating scale

How many active members are there in your coalition from each of the 12 sectors? (Active members attended at least one meeting in the past six months; Please use other for representatives outside the 12 key DFC sectors).

Also, please rate the average level of involvement for each represented sector. Yes/No for representation and a Likert scale from (1) Low to (5) High. See example below.

	# of Active members	Level of Involvement
Parent
Youth
Business community
Civic and volunteer group
Healthcare professionals
Law Enforcement agency
Media
Religious or fraternal organization
School
State, local, and/or tribal governmental agency
Youth-serving organization
Other organizations involved in reducing substance abuse (Report by organization and specify organization names)

New

Simply asking the number of active members by sector reduces data entry burden.

Social media data also indicated difficulties in deleting old members.
Also, we are interested in members’ level of involvement and not simple representation/attendance at meetings (e.g., they work on subcommittees and DFC activities outside of meetings).
Sectors will be rows and columns include 1) # active and 2) level of involvement.

Choose One

Is Collaboration Among Members of Your Coalition

Increasing
Decreasing
Staying the Same?

Keep

Basic Collaborative Activity

Choose One

Activity Type

Hearing on drug problems
Combined public and private funding for substance abuse prevention initiatives
Ad hoc task forces successfully expanding the community effort for substance abuse prevention
A media plan to draw attention to new drug threats
Coalition meetings involving multiple coalition members
Planning or conducting a community event involving multiple coalition members
Research or evaluation

Modify

Keeping because part of GPRA (see below).

Text (1,000 Character Limit)

Description

Keep

GPRA questions so keeping but also enhancing.

Date

Date Started

Delete

Date

Date Completed

Delete

Choose One

Activity Status (Active, Inactive, Planned, Completed, Discontinued)

Delete

Choose One

In the past 6 months, how often did your coalition engage in the following collaborative activities? Also include columns with number of people who attended the event, along with a rating of degree of success (None, Little, Somewhat, Much, A great deal) and magnitude of impact (None, Little, Somewhat, Much, A great deal).

Activity Type

Hearing on drug problems
Combined public and private funding for substance abuse prevention initiatives
Ad hoc task forces successfully expanding the community effort for substance abuse prevention
A media plan to draw attention to new drug threats
Coalition meetings involving multiple coalition members
Planning or conducting a community event involving multiple coalition members
Research or evaluation
Added in items that correspond to the core competencies of the SPF.

Modified

Date

Date Needed

Keep

Essential item for grant monitoring.

Assistance needed by coalitions

Check all that apply

In what area do you need assistance to improve the capacity of your coalition?

Create coalitions and partnerships
Maintain coalitions and partnerships
Assess community needs and resources
Analyze problems and goals
Develop a framework or model of change
Increase participation and membership
Build leadership
Enhance cultural competence
Improve organizational management and Develop strategic action plans
Develop interventions
Advocate for change
Influence policy development
Write grant applications for funding
Evaluate initiatives
Sustain projects and initiatives
Other (Specify)

Modification:

On a scale from Not at All (0) to A Great Deal (5), please rate the degree to which your coalition could benefit from training and technical assistance in the following areas.

Modify

Note: These are the 15 Core Competencies identified by Dr. Renee Boothroyd through a literature review. They are also used in the National Coalition Institute evaluation.

Text (1,000 Character Limit)

Description of Technical Assistance or Training Needed

Keep

This will probably suffice for most requests.

Check all that apply

Delivery Mode

Distance learning
Technical assistance by telephone
On-site/In-person technical assistance
Technical assistance by email
In-person class, conference, or workshop not specific to your community
In-person class, conference, or workshop specific to your community
Tele-conference or telephone-based training
Written materials

Keep

Check One

Desired Source

CADCA's National Coalition Institute
DFC Project Officer (SAMHSA)
State Agency
My Coalition

Delete

Check One

Status

Needed
Received

Closed

Delete

Assistance provided by coalitions

Please provide a brief description of any training and technical assistance your coalition provided to other community groups or organizations.

New

To capture community outputs/impacts.

Funding Streams

Choose One

In the next 12 months do you expect your coalition's funding level to

Increase
Decrease

Stay about the Same

Keep

Numeric

What is your Coalition's total annual operating Budget?

Keep

Dollar amount

Please indicate the dollar amount of your total budget for sources that support your coalition and its strategies

Keep

Include, for each item, when funding started, and when funding runs out. This will allow us to track sustainability and funding streams much more carefully.

Dollar amount

DFC Grant

Keep

Dollar amount

STOP Act Grant

Keep

While this section is burdensome, most of the items need to be kept for ONDCP related purposes. Some analyses may require the identification of coalitions funded by DFC, STOP, SPF-SIG, etc.

Dollar amount

SPF-SIG Funding

Keep

Dollar amount (all combined)

Other drug abuse prevention grants

Keep

Dollar amount

Fundraising/private donations

Keep

Dollar amount

In-kind contributions

Keep

Dollar amount

Foundations/non-profit organizations

Keep

Dollar amount

City/county government

Keep

Dollar amount

State government

Keep

Dollar amount

Federal government

Keep

Dollar amount

Other (please list)

Keep

Text

Add Accomplishment or Challenge/Barrier. Progress Name.

Delete

Too much burden on respondents. Regarding choose reporting period:

If users need to enter historical information, they should enter the survey interface that is specific to reporting periods.

Progress in the capacity build-up of coalitions

Choose one

Reporting Period Identified (Reporting periods)

Delete

Dropdowns with "groupings" and have them rate the ‘degree to which each item was a (1) Significant Challenge to (5) Significant Accomplishment.

Based on qualitative analysis of randomly selected pre-existing data, dropdown categories were created and are listed in another appendix.
This will provide much more useful information for the evaluation of DFC.

Choose one

Type

Accomplishment
Challenge/Barrier

Delete

Text (3,000 Character Limit)

Description

Delete

Check all that apply

What challenges has your coalition faced during the past 6 months while performing activities related to coalition capacity?

New

Check all that apply

What accomplishments has your coalition achieved during the past 6 months while performing activities related to coalition capacity?

New

Dropdowns with "groupings" and have them rate the ‘degree to which each item was a (1) Significant Challenge to (5) Significant Accomplishment.

Based on qualitative analysis of randomly selected pre-existing data, dropdown categories were created and are listed in another appendix.
This will provide much more useful information for the evaluation of DFC.

Specific Examples of Challenge

Text (3,000 Character Limit)

Please report specific examples of challenges that your coalition has experienced during the past 6 months.

New

Choose-from option style questions may miss specific incidences of achievement or challenges that a coalition may want to report.

Specific Examples of Accomplishment

Please report specific examples of accomplishments that your coalition has experienced during the past 6 months.

PLANNING SECTION
Field Name	Data Type	Description	Keep (Keep, Delete, or New)	Rationale
Strategic Plan	Choose one	Plan Status Plan was complete and/or updated but not used for operations Plan already exists that is guiding our operations Plan in progress We intend to develop a plan within the next 6 months No intentions to develop a plan within the next 6 months	Keep
	Choose one	Does your coalition re-visit its strategic plan on an annual basis to make modifications? Yes No	New	This item informs how important the plan is for the coalitions. This item could be included in CCT instead.
	Text (no limit)	Year Plan Was First Developed	Modify: Allow users to enter month and year.	For ease of data entry. The data type will be changed to DATE. The use of the SAMHSA’s strategic prevention framework is mandated by DFC RFA; however, there may be other planning frameworks (e.g., CCT) used in addition to the SPF. We will work with ONDCP and SAMHSA to identify response categories for this new question.
	Text (no limit)	Year Plan Was Last Updated	Modify: Allow users to enter month and year.
	Choose one	Plan Used SAMHSA's Strategic Prevention Framework Yes No	Keep
	Radio buttons/multiple response	To what extent does your coalition use the SFP framework (Likert scale)?	New
	Text (1,000 Character Limit)	Description	Keep
		To what extent does your coalition incorporate other planning frameworks (e.g., CCT)?	New
	Text (1,000 Character Limit)	Description	New
	Browse (find a file to upload)	Upload the following two documents: Long-term strategic plan 12-month action plan	Modify	DFC coalitions are taught to do 2 products related to strategic planning: a long-term Strategic Plan and a 12-month Action Plan. The Strategic Plan is longer, includes outlook for 3-5 years and outcomes related to the timeframe. The Action Plan is more specific and forces a coalition to name who is responsible for certain actions, short-term outcomes, etc. An example of an Action Plan is in the NCI Primer on Planning.
	Text (1,000 Character Limit)	Note	Keep
Logic Model	Browse (find a file to upload)	File	Keep
Logic Model	Text (1,000 Character Limit)	Briefly describe any changes that the coalition made to the Logic Model.	Keep (directions slightly modified to ask to note changes to model)	When Logic Models are updated, have them note changes. This information describes the changes of coalitions.
Goals and Objectives	Choose one	Goal Reduce substance abuse among youths and adults Establish and strengthen collaboration	Delete	Too much burden on respondents. There are two goals “Reduce substance abuse among youths and adults” and “Establish and strengthen collaboration” are specific to DFC requirements. The current data include objectives as free text information. Based on the historical data, we created dropdowns of objectives that are both frequently reported and important in light of the goals of the DFC coalitions. Objectives are currently linked to goals and strategies are linked to objectives. Risk/Protective factors are also linked to objectives. Objective is linked to core outcome measure variable. Linked to grade. Linked to gender. Linked to date. “De-link” this association given feedback on COMET – focus group and social media website indicated this not only caused substantial confusion but repetition and duplication of data entry (e.g., one strategy could be linked to multiple objectives).
	Text (200 Character Limit)	Objective Name	Delete
	Check all that apply	Strategies to Achieve Objective Providing information (e.g., community education, increasing knowledge, raising awareness) Enhancing skills (e.g., building skills and competencies) Providing support (e.g., increasing involvement in drug-free/healthy alternative activities) Enhancing access/reducing barriers (e.g., improving access, availability, and use of systems and service Changing consequences (e.g., incentives/disincentives, increasing attention to enforcement and compliance) Physical design (e.g., environmental and structural) Modifying/changing policies (e.g., changing institutional or government policies)	Delete
	Text (1,000 Character Limit)	Strategy Description	Keep
	Choose one	Percent of Overall Coalition Effort and Resources that went into Achieving the following objectives? (0 to 100% by 10% increment)	Keep
	Date	Date Objective Established	Delete
	Check all that apply	Link Objective to Targeted Risk Factor (Risk factors are selected earlier by respondent in the assessment section) For example: Family – Parental substance abuse Community – Cultural norms etc.	Delete
	Check all that apply	Link Objective to Targeted Protective Factor (Risk factors are selected earlier by respondent in the assessment section) For example: Community – Community attachment Community – Enforcement of laws and regulations	Delete
	Check all that apply	Link Objective to DFC Core Outcome Measure Average Age of Onset Past 30-day Use Perception of Parental Disapproval Perception of Risk or Harm	Delete
	Check all that apply	Targeted Substance Alcohol Tobacco Marijuana Amphetamines Barbiturates Cocaine (including crack) Crystal Methamphetamine MDMA (Ecstasy) GHB (liquid G) Hallucinogens (other than LSD) Heroin Hydrocodone Inhalants (including amylnitrite, cleaning fluid, gasoline, paint, and glue) Ketamine (super K) LSD Non-medical use of other prescription drugs (other than Hydrocodone, Oxycodone, Barbiturates, Amphetamines) Methamphetamine (including Ice) Narcotics other than heroin Nitrites Other Over-the-counter drugs Oxycodone PCPRohypnol (date rape drug) Steroids Tranquilizers	Delete
	Check all that apply	Targeted Grade K-5 6 7 8 9 10 11 12 College Post-College/Young Adult Adult	Delete	Dropdowns with "groupings" and have them rate the ‘top 3 . Respondents will also answer: This objective is included in our strategic and/or action plan? Has progress been made achieving this objective over the last six months/ Rate to what extent this objective has been achieved. (Not At All – Exceeded Our Objective) Based on qualitative analysis of randomly selected pre-existing data, dropdown categories were created and are listed in another appendix.
	Choose one	Targeted Gender Both Genders Female Male	Delete
	Date	Targeted Date for Achieving Objective	Delete
	Choose one	Choose objectives of your coalition for each of the goals: Goal 1: Reduce substance abuse among youths and adults DROPDOWN MENU Goal 2: Establish and strengthen collaboration DROPDOWN MENU The following is asked for each of the objectives specified above. The new page is generated with a list of the selected objectives and the following questions are asked for each of the objectives.> What were the strategies to achieve objectives of your coalition during the current reporting period? Check all from the following list and identify top three strategies used: Use the same list from an earlier entry of the strategies-to-achieve objectives list (seven choices). For example: Providing information (e.g., community education, increasing knowledge, raising awareness) Enhancing skills (e.g., building skills and competencies) Providing support (e.g., increasing involvement in drug-free/healthy alternative activities) Enhancing access/reducing barriers (e.g., improving access, availability, and use of systems and service Changing consequences (e.g., incentives/disincentives, increasing attention to enforcement and compliance) Physical design (e.g., environmental and structural) Modifying/changing policies (e.g., changing institutional or government policies) To what extent has the Objective been achieved? Not at all Some Halfway Mostly Completed Exceeded our objective	Keep/Modify. Add the dropdown list of objectives. Objectives are linked with strategies.
	Text (200 Character Limit)	Progress Name	Keep
	Choose one	Reporting Period Identified	Delete	Too much burden on respondents; no practical use for the evaluation. We have to assume that information entered is relevant for the current reporting period.
Planning Progress	Choose one	Type Accomplishment Challenge/Barrier	Delete
	Text (3,000 Character Limit)	Description	Delete	Dropdowns with "groupings" and have them rate the ‘degree to which each item was a (1) Significant Challenge to (5) Significant Accomplishment. Based on qualitative analysis of randomly selected pre-existing data, dropdown categories were created and are listed in another appendix. This will provide much more useful information for the evaluation of DFC.
	Check all that apply	What challenges has your coalition faced during the past 6 months while performing activities related to planning?	New
	Check all that apply	What accomplishments has the coalition achieved during the past 6 months while performing activities related to planning?	New
Specific Examples of Challenge	Text (3,000 Character Limit)	Please report specific examples of challenges that your coalition has experienced during the past 6 months.	New	Choose-from option style questions may miss specific incidences of achievement or challenges that a coalition may want to report.
Specific Examples	Text (3,000 Character Limit)	Please report specific examples of accomplishments that your coalition has experienced during the past 6 months.	New

IMPLEMENTATION SECTION
Field Name	Data Type	Description	Keep (Keep, Delete, or New)	Rationale
Implementation Activity	Already filled	Goal	Delete	Too much burden for respondents.
	Already filled	Objective	Delete
	Choose one	Strategy (Identified earlier by respondent)	Delete
	Choose one	Activity Type. The choice depends on specific strategies taken. For example: Media campaigns Information Dissemination, Direct, face to face communication, etc.	Delete
	Text (no limit)	Brief Description of Activity	Delete
	Date	Date Started	Delete
	Date	Date Completed	Delete
	Choose one	Activity Status (Active, etc.)	Delete
	Choose one	Activity Leader (Names of Coalition Staff)	Delete
	Choose one	Scope/Reach of Activity (25% increment of target community)	Delete
	Check all that apply	Please see Appendix C for more explicit directions and an example graphic. Instruction for interface design: Users will enter data in seven separate screens, each representing one of the seven coalition strategies. On each screen/strategy, you will see a list of relevant activities. For each of the activities, responses are a given a set of questions, such as number of activities and number of staff involved. See the rationale section for details. Instruction for respondents: Please provide necessary information requested for each of the activities associated with the < specific to a screen > strategy. Instruction for interface design: Seven Coalition Strategies (one strategy on a screen at a time) and lists of activities. PROVIDING INFORMATION: 1) Media Campaigns, 2) Information dissemination, 3) Direct, face-to-face information sessions, 4) Special events to highten awareness, 5) Clearinngouse for ATOD information, 6) Other 1 ~ 3ENHANCING SKILLS: 1) Drug refusal skills program component, 2) Communication/decision-making program component, 3) Role-modeling program component (e.g., mentoring),4) Parenting Skills Program, 5) Conflict management skills trainings, 6) Training program for teachers or trainers, 7) Other 1~3 PROVIDING SUPPORT: 1) Sponsoring drug-free events (e.g., drug-free dances), 2) Teen drop-in centers or clubs, 3) Support youth athletic leagues, 4) Youth support groups, 5) Sponsoring healthy “risky” activities (e.g., rock climbing, ropes courses, etc.), 5) Other 1~3 MODIFYING/CHANGING POLICIES: 1) Alcohol and cigarette advertising restrictions in public areas, 2) Restrictions on alcohol and cigarette use at community events, 3) Limitation and restrictions of location and density of alcohol outlets, 4) Responsible beverage service training (voluntary or mandatory, 5) Efforts to increase tax on alcohol or tobacco, 6) Restrictions on Methamphetamine pre-cursor access, 7) Efforts to require treatment for nonviolent drug offenders, 8) Instituting drug testing, 9) Other 1~3 CHANGING CONSEQUENCES: 1) Increase enforcement of underage drinking laws, 2) Increase enforcement of illicit drug laws, 3) Increase surveillance of areas known for illegal drug sales, 4) "Shoulder-tap" enforcement program, 5) Increased enforcement of impaired-driving laws, 6) Compliance checks for alcohol or tobacco sales to minors, 7) Recognition program for merchants who pass compliance checks, 8) Prescription drug abuse tracking, 9) Other 1~2 ENHANCING ACCESS/REDUCING BARRIERS: 1) Improve access to healthcare services, 2) Improve quality and availability of childcare,3) Improve access to transportation, 4) Improve access to housing, 5) Improve safety and justice in the community, 6) Improve quality and availability of education 7) Improve access for people with special needs 8) Improve cultural language sensitivity 9) Improve access to employee assistance programs 10) Other 1~3 PHYSICAL DESIGN: 1) Improve parks and other physical landscapes (e.g., neighborhood clean-ups), 2) Improve signage, 3) Improve lighting, 4) Reduce the density of alcohol outlets, 5) Invoke nuisance laws to rehabilitate dangerous rental housing, 6) Other 1~3	New	The new format may still appear labor-intensive, but requires a lot less effort than the old system. Data entry will occur in a matrix format. Information requested for each activity: Number of activities/initiatives Number of collaborative staff hours involved Number of people served Percent of coalition effort Total # of partners involved in implementation activity Primary partner(s) (pull-down menu of sector names, e.g., law enforcement, parent, etc. and choose up to 3) Degree of success of activity (Non – A Great Deal) Magnitude of impact (None – A Great Deal) To what extent was objective achieved (Not at All – Exceeded our Objective To reduce data entry burden, we are not asking respondents to specify target group or target substance. We assume that the target group/substance information entered in the assessment section remain the same for information collected in this section and should be intuitive enough to derive from type of activity and strategy. SAMHSA project officers have indicated that they would like to be able to link activities to specific goals and objectives for grants monitoring purposes. Based on the the review of user complaints, the current level of linking is not sustainable. The proposed allows a “soft” linking by users linking their objectives with strategies in the planning section and linking strategies with a set of activities and their quantitative information (e.g., Number of initiatives) in the implementation section. SAMHSA project officers also wanted to know when specific activities began and were completed. This will provide information about how many activities are currently underway and whether efforts are currently underway in specific aspects of the strategic/action plan.
Implementation Progress	Add Accomplishment or Challenge/Barrier
	Already filled	Goal	Delete
	Already filled	Objective	Delete
	Text (200 Character Limit)	Progress Name	Delete
	Choose one	Reporting Period Identified	Delete	If users need to enter historical information, they should enter the survey interface that is specific to reporting periods.
		Type Accomplishment Challenge/Barrier	Delete
	Text (3,000 Character Limit)	Description	Delete
	Check all that apply	What challenges has your coalition faced during the past 6 months related to implementation? DROPDOWN MENU	New	Dropdowns with "groupings" and have them rate the ‘degree to which each item was a (1) Significant Challenge to (5) Significant Accomplishment. Based on qualitative analysis of randomly selected pre-existing data, dropdown categories were created and are listed in another appendix. This will provide much more useful information for the evaluation of DFC.
	Check all that apply	What accomplishments has your coalition achieved during the past 6 months related to implementation? DROPDOWN MENU	New
Specific Examples of Challenges	Text (3,000 Character Limit)	Please report specific examples of challenges that your coalitions have experienced.	New	Choose-from option style questions may miss specific incidences of achievement or challenges that a coalition may want to report.
Specific Examples of Accomplishment	Text (3,000 Character Limit)	Please report specific examples of accomplishments that your coalitions have experienced.	New

EVALUATION SECTION
Field Name	Data Type	Description	Keep (Keep, Delete, or New)	Rationale
DFC Core Outcome Measures	Choose one	Outcome Category this Data Applies to: Average Age of Onset Past 30-Day Use Perception of Parental Disapproval Perception of Risk <Other outcomes specific to a coalition>	Keep and modify	Clearer guidance is necessary. For example, provide a policy on rounding and define terms, such as “%.” “% of people who used,” “Sample size,” “Overall sample size.” Also need specific guidance on sampling. Allow coalitions to add the outcomes of their choice that are not available in the list. We will also consider asking DFC grantees for baseline data that they submitted in their grant applications. Consider adding a section on intermediate outcomes that categorizes some of the more common accomplishments. This will be excellent fodder for DFC reports, and it provides concrete examples of progress: Number of parents who received information/training Number of students trained/provided information Number of drug houses closed Number of laws passed Number of billboards Number of parks/other hangouts modified to be less conducive to drug dealing/use Alternatively, these outputs could be standard responses to questions about the logic model or strategic plans. Or, could be captured in implementation section of COMET.
	Choose one	Source for this Data	Keep
	Choose one (Month and Year)	Month and Year data was collected	Keep
	Choose one (Month and Year)	Compared to the Target Area (Locust Point), the Geographical Area covered by this data is Large Smaller Same	Keep
	Numeric	Enter Data by Grade	Keep and modify
	Numeric	Enter Data by Gender	Keep
Long-Term Health Outcomes	Text	Outcome Category (School Drop Out, etc.)	Delete	Since this is part of the Logic Model, we should keep or find a better/easier way for them to report anything they may have.
	Date	Date Collected	Delete
	Choose one	Did you coalition use this data for program planning purposes? Yes No	Delete
	Text (1,000 Character Limit)	Description	Delete
Evaluation Activity	Text	Activity Name	Delete	Too much burden to enter multiple entries. See proposed new item below.
	Choose one	Type Data collection Evaluation Presentation of findings Recommendations for improvements	Delete
	Date	Date Started	Delete
	Date	Date Completed	Delete
	Choose one	Activity Status (Active, Inactive, etc.)	Delete
	Choose one	Activity Leader (A list of names)	Delete
	Choose one	How often has the coalition conducted the following evaluation activities during the past < > months? Data collection Evaluation Presentation of findings Recommendations for Improvements	New	This would reduce data entry burden.
Evaluation Progress	Add Accomplishment or Challenge/Barrier
	Text (200 Character Limit)	Progress Name	Delete
	Choose one	Reporting Period Identified	Delete
	Choose one	Type Accomplishment Challenge/Barrier	Delete
	Text (3000 Characters Limit)	Description	Delete
	Check all that apply	What challenges has your coalition faced during the past 6 months related to evaluation? DROPDOWN MENU	New	Dropdowns with "groupings" and have them rate the ‘degree to which each item was a (1) Significant Challenge to (5) Significant Accomplishment. Based on qualitative analysis of randomly selected pre-existing data, dropdown categories were created and are listed in another appendix. This will provide much more useful information for the evaluation of DFC.
	Check all that apply	What accomplishment has your coalition faced during the past 6 months related to evaluation? DROPDOWN MENU	New
Specific Examples of Challenges	Text (3,000 Character Limit)	Please report specific examples of challenges that your coalition has experienced during the past 6 months. Your description may include both core outcomes, as well as other outcomes, such as education, health and wellness, crime, family, and social and economic issues.	New	Choose-from option style questions may miss specific incidences of achievement or challenges that a coalition may want to report. Question may be particularly valuable for grant monitoring.
Specific Examples of Accomplishment	Text (3,000 Character Limit)	Please report specific examples of accomplishments that your coalition has experienced during the past 6 months. Your description may include both core outcomes, as well as other outcomes, such as education, health and wellness, crime, family, and social and economic issues.	New
Cultural Competence and Sustainability	Text	Questions and text field.	New	Add a new section on cultural competence and sustainability. Although these concepts are supposed to permeate all steps in the SPF, it would be conceptually easier to measure them across all efforts. For sustainability section, we reviewed CADCA’s guidance on 6 steps to sustainability planning and developed questions that correspond to these major steps. They are currently included in the sustainability section of the CCT. We may be able to gather some questions on cultural competence from SPF/SIG data collection efforts.
Cultural Competence	Text	To what extent has your coalition reached out to organizations representing the diversity of your community?	New
Cultural Competence	Text	To what extent has your coalition successfully engaged organizations representing the diversity of your community?	New
Cultural Competence	Text	To what extent does your coalition have formal, written policies and practices in place that address cultural competence?	New
Cultural Competence	Text	To what extent is compliance with cultural competence policies and practices monitored within your coalition?	New
Cultural Competence	Text	To what extent do you believe the environmental change strategies your coalition implemented were culturally competent?	New
Cultural Competence	Text	To what extent do you believe the drug abuse prevention practices (e.g., curricula) you implemented were culturally competent?	New

ADMINISTRATION SECTION
Field Name	Data Type	Description	Keep (Keep, Delete, or New)	Rationale
Grantee Information	Text	Grantee Name	Keep
	Text	Alternative Grantee Name	Keep
	Numeric	Award Number	Keep
	Numeric	Year of First Award	Keep
	Numeric	Award Number (Mentoring)	Keep and modify	Provide a definition of “mentoring.” Also state that this is only for grantees with a DFC mentoring grant.
	Prefilled	Status (Active)	Delete	All coalitions entering information are active.
	Date	Date Coalition Was First Established	Keep
	Text	Contact Name	Keep
	Text	Contact Email	Keep
	Text	Address	Keep
	Text	City	Keep
	Text	State	Keep
	Numeric	Zip Code	Keep
	Numeric	Phone	Keep
	Numeric	Fax	Keep
	Numeric	Phone	Keep
	Numeric	Extension	Keep
	Text	Email	Keep
Staff Registration (Add Individual Members)		Users need to contact KIT solutions to modify staff list.	Keep
Key Leadership	Text	Leader 1	Improve	Add a function to specify the names of coalition leaders to up three most high ranking personnel and their titles (e.g., director, coalition coordinator), and to record the changes of Coalition leadership. This informs the level of organizational stability of the coalitions. We will need to provide concrete definitions of each position, and may want to ask about the background of the coalition leader – this may be a key determinant of coalition success. Based on feedback from SAMHSA project staff, there was a big need to for more contact information. Also ask each leader how long they have been in the prevention field, and determine whether coalition chairs are paid staff.
Key Leadership	Text	Leader 2

Appendix C: COMET Qualitative Analysis and Possible Dropdown Lists by SPF Category

Based on our review of the COMET data system, and the results from focus groups and the social media website, we felt there could be a number of modifications and enhancements that could improve the collection of high-quality programmatic and evaluation data. This appendix provides a brief description of how COMET data was analyzed to develop relevant sets of dropdowns for each section (e.g., assessment) and lists all potential dropdown categories by each section.

Qualitative Coding and Development of Dropdown Categories

To obtain relevant categories based on historical DFC data, we randomly selected two hundred cases from all of the DFC reporting periods. This procedure increased our confidence that we were obtaining data on both new and older grant communities, as well as, coalitions along the entire range of functioning (high/low). Responses were then coded and major categories were retained. Another coder then used these dropdowns to categorize an additional one hundred randomly selected responses. This quality assurance check was conducted to ensure that all major categories were included within each SPF section. Then, reviewers edited the categories to remove redundancy among categories, to make sure those core competencies that facilitate implementation of SAMHSA’s Strategic Prevention Framework (SPF) were included^¹⁰, to make sure identification of best processes was included^¹¹, and to eliminate categories that did not fit the SPF section.

Finally, based on feedback from both grant communities and Federal personnel, we included three additional other (specify) categories to provide flexibility for communities who may be conducting unique activities. While we initially thought of using this list of dropdowns with rankings (e.g., Please rank your top three challenges/accomplishments), we can also have respondents rate each of these items on a scale, for instance from (1) Significant Challenge to (5) Significant Accomplishment. Since there are typically approximately 10 items for each section, we feel this would not add much burden. For each item, we could also leave in a text field for the respondent to list more detail. This change to the data system would improve the evaluation by providing extremely useful data on DFC coalition activities, challenges, and accomplishments. Currently, with the check all that apply format and no corresponding Likert value, there is little quantitative resonance of the data.

Assessment Activities

How often does your coalition engage in the following assessment-related activities? Propose using a six-point scale (1) Never, Very Rarely, Rarely, Occasionally, Frequently, (6) Very Frequently.

(Assessment is the systematic gathering and analysis of data about the population your coalition serves in order to identify the current problem(s) and related conditions that require intervention).

1. Planning and preparation for assessment activities

2. Data collection (e.g., surveys, interviews, focus group Interviews, community forums, listening sessions) to identify needs and resources in the community, ATOD norms, substance use prevalence, etc.

3. Research/study of pre-existing data (e.g., archival data collection, census, police, county statistics, educational data) to identify needs and resources in the community, ATOD norms, substance use prevalence, etc.

4. Complete “S.W.O.T.” (strengths, weaknesses opportunities, and threats) analysis of current issues, specifically drug-related issues

5. Resource/risk identification (e.g., community mapping)

6. Analysis of assessment data

7. Create and maintain coalitions and partnerships (include key collaborators for the initiative; establish meeting and decision making processes that connect individuals and build trust; facilitate brainstorming and encourage consensus)

8. Analyze problems and goals (facilitate group problem analysis, incorporate experience and expertise of coalition members; apply a risk and protective factor framework; facilitate prioritization of problems and related actions; identify and make use of targets and agents of change)

9. Develop a framework or model of change (facilitate group logic modeling processes; facilitate group critique and ongoing improvement of the logic model based on new information; ensure consensus for the logic model)

10. Other (specify)

11. Other (specify)

12. Other (specify)

Assessment Accomplishments and Challenges

What accomplishments and challenges has your coalition faced during the past 6 months while performing activities related to assessment? Please rate each item below from (1) Significant Challenge to (5) Significant Accomplishment.

A mock-up of how this section would be presented in COMET is also presented below the items.

Response rates to assessment activities
Attaining buy-in from key stakeholders (e.g., school districts, police departments)
Issues with the survey instrument(s) (e.g., resulting from changes in survey formats, comprehensiveness of questions, assessment tool concerns)
Issues administering the survey (e.g., inconsistent implementation, technical issues, challenges identifying comparison groups)
Understanding how to use the data
Consent and related issues (e.g., Institutional Review Board, parent consent, youth consent)
Changes in leadership and/or staff
Delay in program operations
Time required for assessment results/reports
Time requirements in general (e.g., to gain buy-in, develop instruments, administer survey)
Competing priorities
Financial resources to support assessment activities
Understanding of local issues and conditions
Used assessment results to inform strategic planning, including the development of common vision, goals, and objects
Findings were used to inform Drug Free Community activities (e.g., development of new program, creation of materials, implementation of new trainings)
Data was used to educate/inform community members/key stakeholders
Obtained new data sources (e.g., hard to reach populations, students in additional grades)
Obtained baseline information on which progress can be measured (e.g., acceptance of substance use, prevalence of substance use)
Other (Specify)
Other (specify)
Other (Specify)

Figure C1. Assessment Accomplishments and Challenges COMET Mock-Up

Assessment Accomplishments & Challenges ~ Assessment is the systematic gathering and analysis of data about the geographic area your coalition serves in order to identify current assets, problems, and related conditions that require intervention. What accomplishments and challenges has your coalition faced during the past 6 months while performing activities related to assessment? Please rate each item below from (1) Significant Challenge to (5) Significant Accomplishment.
	Significant Challenge 1	2	Neither 3	4	Significant Accomplishment 5
Response rates to assessment activities.	1	2	3	4	5
Attaining buy-in from key stakeholders (e.g., school districts, police departments).	1	2	3	4	5
Issues with the survey instrument(s) (e.g., resulting from changes in survey formats, comprehensiveness of questions, assessment tool concerns).	1	2	3	4	5
Issues administering the survey (e.g., inconsistent implementation, technical issues, challenges identifying comparison groups).	1	2	3	4	5
Understanding how to use the data.	1	2	3	4	5
.........................................................	1	2	3	4	5

Capacity Activity

Please report on each of the following activities (GPRA measures). In the past 6 months, how many activities did your coalition conduct over the listed categories of capacity-related activity? How many people attended the events planned, what was the degree of success of the events (None, Little, Somewhat, Much, A Great Deal), and what was the magnitude of impact for all activities conducted in each of the categories (None, Little, Somewhat, Much, A Great Deal)?

(Capacity refers to the types (such as skills or technology) and levels (such as individual or organizational) of resources that a coalition has at its disposal to meet its aims).

Mock-Up of Dosage Information

Capacity Activity	# Activities	# People Attended	Degree of Success	Magnitude of Impact
1-14	…	….	…	…

1. A media plan to draw attention to new drug threats

2. Ad Hoc task forces successfully expanding the community effort for substance abuse prevention

3. Combined public and private funding for substance abuse prevention initiatives

4. Hearing on drug problems

5. Coalition meetings involving multiple coalition members

6. Planning or conducting a community event involving multiple coalition members/partners

7. Research or evaluation

8. Increase participation and membership (conduct outreach and build relationships; assess current involvement levels; communicate memorable messages; recognize talents and use membership base to maximize the coalition’s success)

9. Build leadership (know the varied attributes and forms of leadership, including collaborative and servant leadership; understand the context to enable individuals to tailor their leadership style appropriately; create group leadership development plans and nurture future leaders for the organization).

10. Enhance cultural competence (assess the level of cultural competence in the coalition’s processes; communicate the importance of diversity and cultural competence in community work; identify the steps required to promote cultural sensitivity; ally with multiple cultural groups).

11. Improve organizational management and development (develop governance and management structures; manage and enhance human resources; create sound financial operations; and ensure effective internal and external communication).

12. Other (Specify)

13. Other (specify)

14. Other (specify)

Capacity Accomplishments and Challenges

What accomplishments and challenges has your coalition faced during the past 6 months while performing activities related to coalition capacity building? Please rate each item below from (1) Significant Challenge to (5) Significant Accomplishment.

Recruitment/engagement of target population(s) (e.g., youth, parents)
Recruitment/engagement of key stakeholders (e.g., faith-based community, schools)
Interagency collaboration (e.g., disagreements, lack of clear roles/responsibilities)
Local political climate
Attendance at meetings/events
Logistical issues (e.g., time and location of meetings, technical issues)
Financial resources
Changes in leadership/key positions (e.g., in the administering agencies, on the coalition)
Provided training (to law enforcement, service providers, coalition members)
Received training and technical assistance to improve coalition functioning and effectiveness
Developed new sub-committees, coalitions, etc.
Other (Specify)
Other (Specify)
Other (Specify)

Planning section: Objectives

Choose objectives that your coalition aims to achieve for each of the two overarching DFC goals. Objectives are based on historical data like other dropdowns in this appendix and will allow for the evaluation to provide more information regarding standardized objectives across a diverse array of coalition bodies and community contexts. Users will be also asked to select what strategies they use for the selected objectives (including identifying the top 3) and to indicate the extent to which “objective has been achieved.”

Below, we first present a graphic to highlight the key steps in this process. This is followed by the more typical presentation of text. Also, see Appendix B for more detail.

(Planning is a process of developing a logical sequence of steps that lead from individual actions to community-level drug outcomes and achievement of the coalition’s vision for a healthier community).

Establish and Strengthen Collaboration

Establish a coalition
Increase the number of target community participants (e.g., youth, parents)
Increase the number of key-stakeholders participating in the collaborative (e.g., partner agencies, schools)
Engage in strategic planning (development of an initial strategic plan and update action plans annually; logic/activity modeling; identify needed action steps and outline the actors, timelines, and required support for successful completion).
Enhance collaboration between different sectors of the community (e.g., government, non-profit, faith-based)
Enhance community/citizen participation in the collaborative
Enhance the infrastructure/capacity of the collaborative (e.g., provide training to collaborative members, develop policies, subcommittees or workgroups)
Ensure the sustainability of the collaborative and/or its activities
Engaged in activities to educate/inform community members/key-stakeholder of local issues and/or DFC activities
Other (Specify)
Other (Specify)
Other (Specify)

Reduce Substance Abuse among Youth and Adults

Engaged in activities to educate/inform community members/key-stakeholder of local issues and/or DFC activities
Increase perception that drug/alcohol/tobacco use is harmful and not used/accepted by peers
Enhance communication strategies to increase public awareness
Reduce reported drug use
Increase the average age of onset of substance use
Establish/enhance community norms that promote prevention activities
Increase parental awareness
Increase the number of available prevention activities/programs
Increase the number and/strength of protective factors (individual/family/school/community e.g., offer life skills)
Decrease the number and/strength of risk factors (individual/family/school/community e.g., anti-poverty programs)
Enhance enforcement of current anti-drug laws and policies
Reduce access to drugs/alcohol/tobacco (e.g., reduce access to over-the-counter prescriptions)
Other (Specify)
Other (Specify)
Other (Specify)

Planning Accomplishments and Challenges

What accomplishments and challenges has your coalition faced during the past 6 months while performing activities related to planning? Please rate each item below from (1) Significant Challenge to (5) Significant Accomplishment.

Logistical issues (e.g., time and location of meetings, technical issues)
Interagency collaboration (e.g., breakdown in communication, different visions/agendas)
Strategic planning
Attendance at coalition meetings
Consideration for new programs/activities/policies among key stakeholders
Time commitment required
Competing priorities
Other (Specify)
Other (Specify)
Other (Specify)

Implementation Activity

In this section, users are asked to list activities by the 7 major strategies for community change (and strengthening coalitions) and report on a) number of activities, b) number of collaborative staff hours involved, c) number of people served, d) percent of coalition effort, e) number of partner agencies, e) primary partners (top 3), and f) degree of success, g) magnitude of impact. This will allow us to better understand the types and mix of strategies employed, along with estimates of dosage received by communities. The list of activities and strategies come from the current COMET system .

Below, we first present a graphic to highlight the key steps in this process. This is followed by the more typical presentation of text. Also, see Appendix B for more detail.

(Implementation puts into motion the activities identified in the planning process).

Implementation Accomplishments and Challenges

What accomplishments and challenges has your coalition faced during the past 6 months while performing activities related to implementation? Please rate each item below from (1) Significant Challenge to (5) Significant Accomplishment.

Recruitment/engagement of target population (e.g., youth, parents)
Recruitment/engagement of key stakeholders (e.g., local businesses, schools, law enforcement)
Competing priorities
Community norms regarding drug and alcohol use
Time commitment required
Identifying/developing interventions (identify, adapt, and implement evidence-based programs and practices; develop unique and local responses that employ multiple strategies; prioritize needed community and systems changes; develop an intervention map based on the coalition’s logic model).
Advocating for change (conduct advocacy research; identify potential allies; design an advocacy campaign; build community capacity and willingness to effect change).
Influence policy development and enforcement (conduct policy research; identify needed resources and assets; ensure community voice; support effective policy implementation).
Write grant applications for funding (create relationships with potential funders; align grant opportunities with intervention plans; maintain positive working relationships and communication with funders).
Local laws/ordinances and/or lack thereof
Other (Specify)
Other (Specify)
Other (Specify)

Evaluation Activities

In the past 6 months, how often did your coalition engage in the following evaluation and evaluation-related activities? Propose using a six-point scale (1) Never, Very Rarely, Rarely, Occasionally, Frequently, (6) Very Frequently.

(Evaluation measures the quality and outcomes or coalition work. Evaluation enables the improvement of interventions and coalition practices).

Evaluation planning (e.g., evaluation survey design, identifying evaluators)
Data collection
Analysis of evaluation data (e.g., core outcomes, risk/protective factors)
Recommendations for evaluation improvements
Presentation of findings
Using evaluation data to inform/adapt/improve coalition activities and programs
Sustain projects and initiatives (evaluate how to continue the work; use a business planning process to sustain work; match identified needs with the appropriate resource development strategy; use full membership participation in implementation of the sustainability plan).
Other (Specify)
Other (Specify)
Other (Specify)

Evaluation Accomplishments and Challenges

What accomplishment and challenges has your coalition faced during the past 6 months related to evaluation? Please rate each item below from (1) Significant Challenge to (5) Significant Accomplishment.

Engaging the buy-in and participation of all relevant schools
Current survey instrument does not include important measures (e.g., information on DFC core measures, breakdown by substance used, demographic information – age, sex, grade level)
Conducted a program evaluation (primarily through pre/post tests)
Response rate
Time it takes to receive data
Time it takes to analyze and report data
Obtaining data from key-stakeholders (e.g., crime data, hospital statistics)
Identified/engaged an external, experienced evaluator
Developed the evaluation survey
Developed an evaluation plan
Conducted a community behavior survey to identify changes since the initial assessment
Conducted an evaluation of the collaborative’s effectiveness
Conducted a process evaluation
Established an evaluation sub-committee
Turnover in leadership (including turnover of evaluators)
Stakeholders’ understanding the data
Getting stakeholders to understand and appreciate the importance of evaluation
Provided evaluation training for key stakeholders
Sustain projects and initiatives (evaluate how to continue the work; use a business planning process to sustain work; match identified needs with the appropriate resource development strategy; use full membership participation in implementation of the sustainability plan).
Other (Specify)
Other (Specify)
Other (Specify)

Appendix D: Core Measures Survey

Drug Free Communities (DFC) Support Program Evaluation of Core Measures Survey

I. General Information

II. 30-Day Use

1. Sex:

 Male

 Female

2. Grade:

 6

 7

 8

 9

 10

 11

 12

3. Age

 10 years old or less

 11 years old

 12 years old

 13 years old

 14 years old

 15 years old

 16 years old

 17 years old

 18 years old

 19 years old or more

Yes

Have you had alcoholic beverages (beer, wine, cocktails, hard liquor, etc.) to drink – more than just a few sips – during the past 30 days?



Have you smoked cigarettes during the past 30 days?



Have you used marijuana during the past 30 days?



Have you used prescription drugs not prescribed to you during the past 30 days?



III. Perception of Risk
	No Risk	Slight Risk	Moderate Risk	Great Risk
How much do you think people risk harming themselves (physically or in other ways) if they take five or more drinks of an alcoholic beverage (beer, wine, liquor) once or twice a week?				
Required for STOP Act Grantees: How much do you think people risk harming themselves (physically or in other ways) if they take one or two drinks of an alcoholic beverage (beer, wine, liquor) nearly every day?				
How much do you think people risk harming themselves (physically or in other ways) if they smoke one or more packs of cigarettes per day?				
How much do you think people risk harming themselves (physically or in other ways) if they smoke marijuana once or twice a week?				
How much do you think people risk harming themselves (physically or in other ways) if they use prescription drugs that are not prescribed to them?				

IV. Perception of Parental Disapproval
	Not at all wrong	A little bit wrong	Wrong	Very wrong
How wrong do your parents feel it would be for you to have one or two drinks of an alcoholic beverage nearly every day?				
How wrong do your parents feel it would be for you to smoke one or more packs of cigarettes a day?				
How wrong do your parents feel it would be for you to try marijuana once or twice a week?				
How wrong do your parents feel it would be for you to use prescription drugs not prescribed to you?				

V. Perception of Peer Disapproval
	Neither Approve Nor Disapprove	Somewhat Disapprove	Strongly Disapprove	Don’t Know/ Can’t Say
How do you feel about someone your age having one or two drinks of an alcoholic beverage nearly every day?				
How do you feel about someone your age smoking one or more packs of cigarettes a day?				
How do you feel about someone your age using marijuana once a month or more?				
How do you feel about someone your age using prescription drugs not prescribed to them?				

Appendix E: CCT Data Collection Plan

CCT Table of Measures^¹²

Field Name	Data Type	Description	Keep (Keep, Delete, or New)	Std. Corr.	Rationale
CCT – Coalition Development & Management Global Rating	Likert	Section I: Coalition development and management involves designing and implementing your coalition’s organizational structure and operating procedures. These elements are fundamental and create the delivery mechanism for the array of prevention strategies that will be undertaken.	Keep		This is one of four function areas that comprise the coalition typology. Past evaluation reports have extensively used the typology to classify/categorize coalitions (i.e., Establishing, Functioning, Maturing, & Sustaining) and assess if they achieve different results on core outcomes. The typology should be retained in order to compare future findings to historical data. Additional typologies or other methods to categorize coalitions in developmental stages will also be investigated in future analyses. Overall reliability = .88.
		Assessing needs*	Keep	.72
		Mobilizing and building capacity in the coalition*	Keep	.73
		Developing a comprehensive plan*	Keep	.74
		Implementing*	Keep	.70
		Evaluating*	Keep	.65
		Planning for sustainability*	Keep	.63
Coalition Type	Check one	Item stem - Which of the following best describes your coalition?	Keep		This is basic and necessary descriptive information which should be retained.
		A loosely organized group whose main goal is information sharing.	Keep
		A semi-formal group of organizations who have begun to work together on prevention programs and strategies.	Keep
		A formal group or organization who plan and act together to implement prevention programs and strategies.	Keep
		A highly formal arrangement with most organizations having a clear role in the planning and implementation of community wide prevention strategies.	Keep
Organizational Structure	Check One	How would you describe your coalition’s organizational status?	Keep		This is another basic and necessary item to obtain some descriptive information about coalition structure and functioning.
		Our coalition reports to a larger organization on programmatic and fiscal matters.	Keep
		Our coalition reports to another organization on fiscal matters only, but we are able to operate otherwise independently, including choosing our own staff.	Keep
		Our coalition is a totally independent organization. We make decisions regarding finances, personnel, and programming.	Keep
	Text	Other, please specify:	Keep
Confidence in Task Completion	Likert	Item stem - How confident are you that your coalition can…?	Keep		This is a reliable scale that asks members if they are confident that their coalition can complete a number of tasks, from agenda setting to developing new leaders. This scale can be used to create a measure of collaborative formalization and could possibly be re-worded along strongly disagree to strongly agree anchors, reducing the need for a separate section and decreasing response burden. Factor Analysis – Good factor loadings (all >.50, except “develop agendas” which cross-loads with another factor). Overall reliability = .94.
		Develop agendas and stick to them in meetings.	Keep	.48
		Recruit new members who have the ability to take action in the community	Keep	.67
		Recruit new members who are accountable for reporting to their organization or constituency.	Keep	.69
		Follow up on decisions made at meetings.	Keep	.66
		Recruit members from the different sectors needed to address your Coalition’s goals.	Keep	.68
		Provide direction and vision for the Coalition through its leadership.	Keep	.77
		Maintain support of Coalition members through its leadership.	Keep	.78
		Share leadership among Coalition members.	Keep	.73
		Maintain stable leadership.	Keep	.75
		Develop new leaders.	Keep	.75
		Delegate responsibilities to committees.	Keep	.64
		Engage members of target populations (e.g. youth) and diverse cultural groups as active members and leaders.	Keep	.61
		Set and achieve annual goals.	Keep	.70
		Recruit “champions” to act on Coalition’s behalf.	Keep	.68
		Hold each other accountable.	Keep	.73
Coalition Governing Body	Yes/No	Item stem - We have a board or governing body that sets the direction of the coalition (A board or governing body is defined as a formal group or body that makes decisions for the coalition)?	Keep		This item precedes all the questions regarding written procedures and provides some basic descriptive information which is needed. This item should be retained.
Written Procedures	Yes/No Likert ** 3-point	Item stem - Please Indicate your agreement by selecting “Yes” or disagreement by selecting “No” with each of the following statements as they apply to your coalition.	Keep		Yes/No items on collaborative procedures are pretty common and standard items. There are currently eleven items regarding if there are written procedures around coalition activities and other items meant to tap into the “formal” nature of the collaboration. These items can be used to create scales (using a cut point to identify formalized coalitions versus not formalized) or looked at individually. There are some disadvantages to this method though, such as giving equal weighting to each activity and possible lack of variance across coalitions.
		We have a governing body that sets the direction of the coalition.	Keep
		We have a written policy for leadership rotation.	Keep
		We have written expectations for member participation (e.g., policy on missed meetings).	Keep
		We have a written description of procedures for leader selection.	Keep
		We have a written description of the procedures for decision making (e.g., majority rule, etc.).	Keep
		We hold regularly scheduled meetings (i.e., on a specific date/time).	Keep
		We prepare a written agenda for each Coalition meeting.	Keep
		We prepare and distribute written minutes of Coalition meetings.	Keep
		We have a current organizational chart showing Coalition structure and relationships.	Keep
		We have established subcommittees.	Keep
		We have paid staff members or in-kind staff (in-kind staff are staff who are paid by someone else to work at your coalition).	Keep
Identification of Community Leaders	Likert	Item stem - Please indicate how much you agree or disagree with each of the following statements.	Keep		Adequate factor loadings. This scale has adequate reliability and seems to tap into the grassroots and community-based nature of DFC. Thus, we recommend retaining this scale. Overall reliability = .70.
		Our coalition has identified all of the community leaders that can help in the coalition’s activities.	Keep	.45
		Our coalition regularly works with the necessary community leaders and they work on coalition activities or actively support strategies.	Keep	.56
		The most important community leaders are members of the coalition and work on coalition committees and activities.	Keep	.53
Community Leadership in Coalition Efforts	Likert ** 3-point	Please indicate the degree to which each statement below describes the role of the community leadership (not coalition staff) of the coalition. The coalition community leadership…	Keep		This scale follows-up on the previous scale and asks about the extent of community leaders’ involvement in collaborative activities.
		Takes responsibility for coordinating the coalition.	Keep
		Sets the agenda for coalition meetings.	Keep
		Keeps coalition members accountable to tasks.	Keep
		Promotes cohesiveness and team spirit.	Keep
		Creates an environment where opinions can be voiced.	Keep
		Are recognized leaders in the larger community on issues related to substance abuse prevention and related issues.	Keep
		Are very involved in regional, state or national prevention organizations or coalitions.	Keep
Coalition Leadership	Likert	Item stem - Coalition leadership refers to the official chair or co-chairs of your coalition. Below are several statements about leadership within coalitions.	New		In the current CCT, there are two good scales around community leadership – but there is no specific scale for coalition leadership. This may be because one of the co-chairs fills out the survey. If this is often the case, we propose at least including short scales on committee leadership and coalition coordinator leadership. Practitioners stress the importance of leadership and empirical research has demonstrated the strong links between leadership and collaborative effectiveness. These scales were originally adapted from Butterfoss (1998). Overall reliability is .97.
		Coalition leadership is committed to the coalition’s mission.	New
		Coalition leader(s) provide leadership and guidance in maintaining the coalition.	New
		Coalition leader(s) have appropriate time to devote to the coalition.	New
		Coalition(s) leaders promote equality and collaboration among members.	New
Committee Leadership	Likert	Item stem - Committee leadership refers to the chair or co-chairs of subcommittees.	New
		Committee leader(s) plan meetings effectively and efficiently.	New
		Committee leader(s) are flexible in accepting different viewpoints.	New
		Committee leader(s) is/are adept in organizational and communication skills.	New
		Committee leader(s) are adept in obtaining resources.	New
Coalition Coordinator Leadership	Likert	Item stem - Coalition coordinator refers to the paid staff on your coalition.	New
		Does funding from DFC support a part-time or full-time coalition coordinator?	Yes/No
		Does funding from local or another source support a part-time or full-time coalition coordinator?
		Our coalition coordinator plays a vital role in organizing coalition activities and efforts.	New
		The coalition coordinator facilitates communication across coalition participants.	New
		The coalition coordinator supports the coalition’s goals.	New
Collaborative Decision Making	Likert ** 3-item	Item stem - We would like to understand the effectiveness of your coalition in making decision. Please indicate the degree to which you either agree or disagree with each of the following statements describing how your coalition makes decisions. If you have trouble deciding, please select the answer that describes your feelings most of the time.	Keep		Decision making processes are key to coalition functioning and member satisfaction. This scale has adequate reliability if the reverse coded item is deleted. Reliability is somewhat low because a 3-point scale is used and it seems there are two issues being asked about: (1) if decisions are made and (2) if the decision making process is democratic and participatory. It might be beneficial to add a few items from pre-existing scales to increase reliability or use a 5-point agreement response set.
		The coalition makes decisions when they are needed.	Keep
		The general membership has real decision-making control over the policies and actions of the coalition.	Keep
		Decisions are made by a small group.	Keep
		There is formal process for making decisions (e.g., a voting system).	Keep
		Decisions on the allocation of coalition resources are made in an open and participatory manner.	Keep
		We effectively use the coalition process to plan and make decisions	Keep
Shared Vision & Cohesion	Likert	Item Stem - Please indicate the extent to which you agree with each statement from (1) Strongly Disagree to (5) Strongly Agree.	New		According to reviews of collaborative research, shared vision is one of the key dynamics that explains variability of coalition functioning and effectiveness. 5 items with .87 reliability.
		The coalition has a feeling of cohesiveness and team spirit.	New
		There is a shared vision for desired outcomes of DFC coalition work (e.g., reduce use, increase protective factors).	New
		Coalition members feel valued and important.	New
		There are clearly defined, attainable goals for the initiative.	New
		There is a shared vision of what the coalition should accomplish.	New
Frequency of Conflicts	Likert	How frequently do conflicts such as major disagreements and tensions among individual and groups occur in your coalition?	Delete		The items around frequency, cause, and impact of conflicts seem to provide less information than other items and they are recommended for deletion due to the need to include additional variables and items. For instance, check all that apply items are difficult to analyze in a meaningful fashion. If retained, CCT should be altered so communities could at least rank the top 2 or 3 causes for conflict.
Cause of Conflicts	Check all that apply	What are the causes of these conflicts? (Please check all that apply?)	Delete
		Different perspectives or philosophies	Delete
		Personality differences	Delete
		Turf battles	Delete
		Issues in the larger community	Delete
		Arguments over allocation of coalition resources	Delete
		Arguments over among or burden of work	Delete
	Text	Other, please specify	Delete
Impact of Conflicts	Check one	Overall, what has been the impact of these conflicts on the coalition? (Please check only one).	Delete
		The coalition is a lot worse off/Conflicts have hurt the coalition’s progress.	Delete
		The coalition is somewhat worse off.	Delete
		These conflicts have not made a difference.	Delete
		The coalition is somewhat better off because of the conflict.	Delete
		The coalition is a lot better off/very positive results.	Delete
Communication and Conflict Management	Likert	Item Stem - Please indicate the extent to which you agree with each statement from (1) Strongly Disagree to (5) Strongly Agree.	New		This was added because conflict management is an important coalition dynamic, but prior questions were not obtaining the necessary data. This is an abbreviated scale that we have used in the past which has established reliability. It can be used as one scale or broken out into communication management and conflict management (depending on results of factor analysis).
		There is a formal process for resolving conflicts among participating organizations.	New
		The communication procedures are clearly understood among collaborative members.	New
		Conflicts arise frequently among participating organizations in the collaborative.	New
		Communication between member organizations is closed and guarded.	New
		Differences among collaborative members are recognized and worked through.	New
Cultural Diversity	Likert	The following statements cover issues regarding how your coalition addresses cultural diversity and cultural competence. Please read each statement carefully and check the one response that comes closest to your belief. With regard to cultural diversity, your Coalition has....?	Keep		Good factor loadings (>.5). This scale has seven items and good reliability. Also, given the defining role of cultural competence or diversity in DFC, we recommend retaining this scale. Overall reliability = .85.
		Staff members that are representative of the demographic and cultural diversity in your community.	Keep	.54
		Culturally and language appropriate materials relevant to the target population.	Keep	.62
		Materials examined by diversity experts or target population members.	Keep	.56
		Developed a culturally appropriate outreach action plan.	Keep	.61
		Activities and decision-making processes that are designed to be inclusive.	Keep	.61
		Meetings and activities scheduled at times that are convenient and at locations that are accessible to the target population	Keep	.57
		Coalition members that are representative of the demographic and cultural diversity in your community.	Keep	.69
		Targeted youth are involved in coalition meetings and activities.	New		This was added because of the importance of engaging and involving the target community of DFC efforts.
Performance Evaluation	Yes/No	Item stem - Please respond Yes or No for the following statements. Does your coalition …?	Keep		This Yes/No list of performance evaluation and regular evaluation items should be retained given the need for GPRA data and substance use data. Items in the CCT can only serve to reinforce the importance of data collection for coalition leaders. Possibly create a Likert scale to provide a reliability estimate for the scale.
		Record coalition decisions in minutes or otherwise keep track of decisions.	Keep
		Have a system for monitoring and tracking coalition activities and other efforts.	Keep
		Monitor on a regular (annual or more frequently) basis community indicators of substance abuse or related information.	Keep
		Have someone on staff at the coalition or a member organization that assists the coalition with collecting and analyzing data on coalition activities and community indicators.	Keep
		Use relations with local universities or others to get assistance on evaluating coalition efforts.	Keep
		Typically hold meetings to reflect on the result of monitoring or evaluation activities in order to make adjustments to implementation.	Keep
		Collect school survey data that matches the coalition’s geographic boundaries at least once every two years.	Keep
Coalition Sustainability	Likert	There are many factors that may contribute to the developmental of a sustainable coalition. A sustainable coalition is one that can continue to work in the community without DFC funding and share ownership and capacity to collaboratively prevent substance abuse. We would like your assessment of the extent to which your coalition has accomplished each of the following. Your coalition has ….	Keep		Sustainability is a major focus for DFC and is even factored in the typology ratings assigned to coalitions, yet we believe it is not being fully assessed in the current version of the CCT. Of the thirteen sustainability items, we propose retaining six and adding an additional five items for an overall reduction in the number of items but expansion in assessing the quality of coalitions’ sustainability efforts. In addition to simply asking if the coalition itself will keep meeting, the items tap into other mechanisms through which DFC has been embedded/ institutionalized in the community (via MOUs, procedures for sharing relevant information, strategies that combine agency resources, such as blended funding, etc.). Factor analysis is OK with a couple items below .5. Good overall reliability (.77). The last six items utilize CADCA’s
		Established a reputation for ‘being able to get things done’ related to at least one initiative or practice.	Keep	.57
		Adopted an entrepreneurial spirit in seeking additional support.	Keep	.54
		Established on-going jobs in your organization.	Keep	.47
		A mature and stable lead organization or has all functions (501c3 status, etc.) to operate independently.	Keep	.45
		Aligned the coalition’s goals and priorities well with past work done by the coalition.	Keep	.57
		A plan for continued leadership.	Keep	.52
		The DFC community coalition plans on continuing meeting after federal grant funding ends.	New
		Youth-serving organizations have begun to or are already taking on coalition efforts in order to institutionalize them and provide long –term sustainability	New
		The coalition has developed strategies to continue to combine agency resources to better serve youth and families (e.g., blended funding, identification of alternative funding, etc.).	New
		The coalition has secured funding to continue its substance abuse reduction efforts when federal funding ends.	New
		Procedures for continuing to share relevant information across agencies have been established.	New
These items correspond to CADCA’s 6 steps of sustainability planning		Identified what must be sustained to continue our community-based efforts toward drug abuse prevention.	New
		Identified the resources that are required to continue our community-based efforts toward drug abuse prevention.	New
		Created case statements that explained our need to continue community-based efforts toward drug abuse prevention efforts.	New
		Determined funding strategies to continue our community-based efforts toward drug abuse prevention.	New
		Identified potential partners to continue our community-based efforts toward drug abuse prevention.	New
		Created an action plan to contact and present to potential partners in our efforts to continue community-based drug abuse prevention.	New
Sustainability	Choose one (Yes/No/Maybe or Don’t Know)	Item stem - Do you think your coalition will be able to sustain itself in your community for the next ten years?	Delete
	Check all that apply	Why don’t you think your coalition will be able to sustain itself in your community for the next ten years?	Delete
		Changes in circumstances, staff, and resources.	Delete
		DFC is no longer a good “fit” in our community.	Delete
		Desired outcomes were not achieved.	Delete
		Changes in population, place, or policy of interest reduced the need for DFC.	Delete
		Funding and resources are not available to continue	Delete
	Text	Other, please specify.	Delete
Coalition Focus 1	Check one	Some coalitions focus on coordinating programs or services delivered by their partners (e.g., curriculum for youth or parents, provision of after school programs); others focus on changing aspects of their community environment (e.g., availability, policy, enforcement). Which of the following do you think best describes the current status of your coalition?	Keep		Keep for identification of degree of coalition focus on direct services and prevention programming compared to environmental focus.
		Our coalition focuses exclusively on coordinating the provision of prevention programs or services.	Keep
		Out coalition focuses primarily on coordinating the provision of prevention programs/services, though occasionally attention is paid to environmental or policy change.	Keep
		Our coalition’s focus is about evenly split between coordinating prevention programs/services and environmental or policy change.	Keep
		Our coalition focuses primarily on environmental or policy change, though occasionally attention is paid to coordinating prevention programs and services.	Keep
		Our coalition focuses exclusively on environmental or policy change.	Keep
CCT – Coordination of Prevention Programs/Services	Likert	Section II: Coordination of Prevention Program/Services. A community uses community data and lists of evidence-based programs and services to identify and coordination an array of “best fit” prevention programs and services to be delivered by its partners. A coalition then aligns and coordinates the integration of these programs and services across the community and evaluates their impact.	Keep		This is one of four functional areas that comprise the Coalition Classification Tool. Classifications have been extensively used in data analyses and should be kept for historical comparative purposes. Overall reliability = .91.
		Assessing needs*	Keep	.73
		Mobilizing and building capacity of partners*	Keep	.80
		Developing comprehensive Strategic and Action Plans*	Keep	.78
		Implementation of Strategic and Action Plan*	Keep	.74
		Conducting Process & Outcome Evaluations*	Keep	.72
		Planning for sustainability*	Keep	.69
Action Plan Activities	Likert	Please choose one response for each of the following statements to characterize your coalition. Your coalition has ……	Keep		Factor Analysis - All greater than .50 except 1 (evidence-based, which approaches .50 and loads on another factor). This is a reliable scale which obtains good information on the degree of follow-up action planning. Overall reliability = .87.
		Identified specific strategies and activities to reach its goals.	Keep	.61
		Created a realistic timeline for completing activities.	Keep	.71
		Identified responsible person(s)/agencies for each activity.	Keep	.71
		Developed a strategy to recruit participants for activities or events.	Keep	.61
		Developed a budget that outlines the funding required for each coalition related activity, training, or event.	Keep	.65
		Identified other resources needed for each activity.	Keep	.69
		Planned to use primarily evidence-based strategies.	Keep	.48
		Plans to evaluate strategies and activities.	Keep	.56
		Revisited and updated your action plan.	New		Added to track a DFC requirement to update action plans annually (after creating the strategic plan in the first year).
CCT - Environmental Strategies	Likert	Section III: Environmental strategies (e.g., reducing access, policy change, increasing enforcement) involve the coalition assessing the community environment to select and implement an array of “best fit” environmental initiatives. The coalition then aligns and integrates them across the community and evaluates their impact.	Keep		This is one of four functional areas that comprise the coalition typology. Classifications have been extensively used in data analyses and should be kept for historical comparative purposes. Overall reliability = .93.
		Assessing needs*	Keep	.79
		Mobilizing and building capacity*	Keep	.83
		Developing a comprehensive plan*	Keep	.85
		Implementing strategies*	Keep	.83
		Evaluating environmental strategies*	Keep	.77
		Planning for sustainability*	Keep	.64
Environmental Strategies	Likert	Please indicate how much you agree or disagree with each of the following statements about your coalition.	Keep		Factor analysis – there is some cross-loading for this construct, mainly on two constructs but a few on other constructs as well. Given the importance of this variable for DFC, it may need to be modified. Overall reliability = .84.
		Our members are just learning the importance of environmental strategies.	Keep	.46
		We have had past success implementing policy and other environmental strategies.	Keep	.67
		The majority of our members are supportive of the coalition pursuing environmental strategies.	Keep	.59
		The majority of our members participate in implementing environmental strategies.	Keep	.61
		Our coalition does not have sufficient ability to advocate for policy changes.	Keep	.54
		Our coalition does not currently have the ability to implement an effective media campaign.	Keep	.45
		We have identified the policy and environmental changes relevant to our community.	Keep	.59
		We have the appropriate members to launch the needed policy change and environmental strategies.	Keep	.58
		We belong to other coalitions and networks in order to have a larger impact on policy and environmental strategies.	Keep	.43
		Our coalition uses an appropriate mix of environmental change strategies from providing information to changing consequences and physical design.	New		Added in to obtain information regarding using a mix of strategies targeting multiple levels.
Coalition Focus 2	Check one	Whatever your coalition’s focus, some coalitions act directly in the community (e.g., coordinating prevention programs and services or advocating for an environmental/policy change). Other coalitions act indirectly by building capacity of other organizations (through convening, incubating, training, or technical assistance). Which of the following do you think best describes the current status of your coalition?	Keep		This is another item that is attempting to assess the coalition’s specific focus – direct or indirect. Recommend keeping item.
		Our coalition exclusively acts directly (coordinates programs/services and pursues policy change ourselves).	Keep
		Our coalition primarily acts directly (coordinates programs/services and pursues policy change), though occasional attention is paid to building the capacity of other organizations.	Keep
		Our coalition is evenly split between acting directly and building the capacity of others to act (acting indirectly).	Keep
		Our coalition primarily acts indirectly (building the capacity of other organizations) though occasional attention is paid to acting directly to coordinate programs/services or pursue policy change.	Keep
		Our coalition exclusively acts indirectly (we build the capacity of other organizations to act).	Keep
CCT – Intermediary or Community support Org	Likert	Section IV: Some coalitions evolve into “intermediary or community support organizations.” The coalitions act mainly indirectly in the community by building the capacity of other organizations. They assess training and technical assistance needs and resources, build capacity where needed, develop a capacity building plan, provide an array of support services and evaluate impact on recipient organizations and the community.	Keep		This is one of four functional areas that comprise the coalition typology. Classifications have been extensively used in data analyses and should be kept for historical comparative purposes. Overall reliability = .92.
		Assessing needs*	Keep	.77
		Mobilizing and building capacity of other organizations*	Keep	.80
		Developing a comprehensive plan for capacity building*	Keep	.80
		Implementing or integrating capacity building strategies*	Keep	.78
		Evaluating capacity building efforts*	Keep	.75
		Planning for sustainability*	Keep	.71
Capacity Building Capacity	Check All That Apply	Please check all of the following resources your coalition has available for capacity building?	Modify		This is a decent checklist on some of the technical assistance and capacity building that coalitions could be providing to community-based organizations; however, need to put on a Likert Scale.
		Information on research-based prevention programs and strategies.	Keep
		Staff and consultants who are knowledgeable and skilled in capacity building methods (e.g., adult education, consulting).	Keep
		Staff and consultants are knowledgeable of evidence-based prevention programs, services and environmental strategies undertaken by the coalition.	Keep
		Ability to produce educational materials.	Keep
		Resources to send coalition members to workshops, conferences, or to visit other communities.	Keep
		Ability to distribute materials and other educational resources.	Keep
Synergy	Likert	What makes coalitions unique is their potential to combine the different perspectives, knowledge, and skills of a group of people and organizations. This unique combining power has been called synergy. We would like to know the extent to which you feel synergy is being created in your coalition. How much do you agree or disagree with each of the following statements.	Keep		Inspection of individual item content suggests that some of the items may be capturing related constructs (common language and common goals seems to be getting at shared vision). Depending on stakeholder feedback, this variable may be able to be deleted or modified. Factor analysis – good results (all items >.50). Overall alpha = .84.
		Our coalition has developed a common language for communication among diverse partners.	Keep	.62
		Our coalition has developed common goals that are understood and supported by all partners.	Keep	.63
		Our coalition is better able to carry out its work because of the contributions of diverse partners.	Keep	.63
		Our coalition has clearly communicated how its action will address problems that are important to people in the community.	Keep	.65
		Our coalition has committed the perspectives, resources and skills of partners.	Keep	.68
Perceived Effectiveness	Likert	Collective self-efficacy refers to a group’s shared belief in its joint capabilities to attain goals and accomplish desired tasks. We are interested in your assessment of the collective self-efficacy of the coalition. How much do you agree or disagree with each of the following statements.	Keep		In the CCT, this scale is called collective self-efficacy. Two items, which reflect connections to other organizations and larger entities, are proposed to be kept while three other more general items (e.g., “can work together as a group”) are proposed to be cut. Seven additional items, specific to the goals of the DFC community change initiative, are proposed to be included. Factor Analysis – Poor results, majority of items want to load on synergy, but display poor loadings.
		Our coalition has the support of other organizations and influential community leaders.	Keep
		Our coalition participates in larger coalitions and groups that can affect change at a higher level (e.g., State).	Keep
		Our coalition has a sufficient combination of skills within its membership to achieve the coalition’s goals.	Delete
		Our coalition can work together as a group and take collective action to achieve the coalition’s goals.	Delete
		Whenever our coalition has to move forward towards a goal, almost everyone is willing to join in and do their share of the work.	Delete
		Coalition activities and efforts have been successful in reducing substance abuse rates among youth in the community.	New
		Coalition activities and efforts have been successful in increasing community awareness of substance abuse issues in the community.	New
		The coalition has established ‘credibility’ in the community by their on-going efforts to reduce substance use rates among youth in the community.	New
		Coalition activities and efforts have been successful in increasing community-based approaches to reducing substance use.	New
		Coalition activities and efforts have been successful in increasing interagency collaborative efforts to address substance abuse issues.	New
		Coalition activities and efforts have been successful in increasing protective factors among youth in the community.	New
		Coalition activities and efforts have been successful in decreasing risk factors among youth in the community.	New
Coalition Self-Assessment	Check one	Coalitions develop as they increase their mastery of key functions needed to prevent substance abuse… For the work you need to do in order to be a successful coalition, how would you rate your coalition’s capacity to perform the key functions presented above?	Keep		Overall assessment of coalition functioning – keep.
		My coalition is learning how to perform most of these key functions.	Keep
		My coalition is achieving proficiency (competency) in performing most of the key functions to prevent substance abuse, though some we may be learning or have mastery (expertise).	Keep
		My coalition has achieved mastery (expertise) in performing most of the key functions needed to prevent substance abuse, however we could not sustain our coalition sufficiently without DFC funding	Keep
		My coalition has achieved mastery in most of the key functions we perform in the community. The capacities are sustainable and institutionalized (i.e., they can be supported without DFC funding and the capacities and mission are shared among many organizations).	Keep
	Text	Additional Comments	Keep
Member Satisfaction	Likert	Please indicate how much you agree or disagree with each of the following statements about your coalition.	New		This new variable is proposed to obtain an idea of how member satisfaction contributes to coalition functioning and could even be modeled as a basic output or outcome variable as well. Could be strengthened even further if we move to a multiple-informant data collection format.
		Members are satisfied with the coalition’s progress in addressing youth substance abuse issues.	New
		Members are satisfied with the degree of collaboration that has occurred as a result of the coalition’s activities and efforts.	New
		Members are satisfied with the progress made toward coalition goals and objectives.	New
Interorganizational Coordination	Likert	Please indicate how much you agree or disagree with each of the following questions about your coalition.	New		Given that one of the major goals of DFC is to increase collaboration, this variable is proposed to be included so we can capture whether or not organizations and agencies are indeed coordinating and collaborating to a greater extent.
		To what extent has the efforts of your coalition resulted in organizations and agencies working together more efficiently?	New
		To what extent has the efforts of your coalition resulted in members seeing their organization/agency as part of a broader system responding to youth drug abuse?	New
		To what extent has the efforts of your coalition increased the ability of organizations/agencies to coordinate their efforts?	New
		To what extent have the efforts of your coalition increased members’ knowledge of the strengths, as well as limitations, of each other’s organizations and agencies?	New
Organizational Impact	Likert	For my organization/agency, participation in the coalition has led to:	New		This variable assesses the degree of impact of coalition efforts and activities on member organizations. This could be a good “systems and community change” outcome variable – how has DFC efforts impacted the system -- that is, changed organizational/agency/community work for the benefit of youth and families?
		The generation of new ideas for improving our practices and/or services.	New
		The acquisition of useful knowledge about services, programs, or people in the community.	New
		A decrease in the number or severity of barriers we face in accomplishing our mission.	New
		A greater ability to identify the source of problems we encounter in order to come up with more effective solutions.	New
		A heightened public profile for my organization/agency.	New
		Increased our ability to affect public policy.	New
		Increased access to tools, best practices, and/or other information that has informed the work of my organization.	New
		Greater knowledge about how the system works and how organizations and agencies affect one another.	New
		An increase in our ability to find the answers to questions or problems that arise.	New
		An improvement in our ability to compete for grants and/or other funding opportunities.	New
		Increased utilizations of my organizations/agency’s expertise or services.	New
Community Social Organization	Likert	Please indicate how much you agree or disagree with each of the following statements about your coalition.	New		This variable is proposed in order to capture more about the community context and is an indicator of a community asset (social organization). Past research has demonstrated how this variable transmits neighborhood disadvantage to outcomes like substance use and delinquency.
		People in this community know each other.	New
		People in this community participate in social activities.	New
		People in this community feel connected to each other.	New
		People who live here feel they are part of a community.	New
		People who live here never do things to improve the community.	New
		People talk to each other about community problems.	New
		People in this community have a voice regarding important issues.	New
		Together, people in this community can persuade the city to respond to their needs and concerns.	New
		It is fairly safe to walk in this community at night.	New
		People in this community don’t trust each other.	New
		Residents don’t care about the community’s future.	New
		People in this community make it a safer place to live.	New
Collective Efficacy	Likert	Please indicate how much you agree or disagree with each of the following statements about your coalition.	New		Similar to community social organization, this variable, made famous by the project of human development in Chicago neighborhoods, obtains data on whether or not people feel comfortable in the neighborhood to intervene when kids are misbehaving and has been linked to lower levels of neighborhood violence.
		Children are skipping school and hanging out on a street corner.	New
		Children are spray-painting graffiti on a local building.	New
		Children are showing disrespect to an adult.	New
		Children are fighting in front of your house.	New
		Fire station closest to your home was threatened with budget cuts.	New
		People around here are willing to help their neighbors.	New
		This is a close knit neighborhood.	New
		People in this neighborhood can be trusted.	New
		People in this neighborhood generally don’t get along with each other.	New
		People in this neighborhood don’t share the same values.	New
Coalition Readiness	Likert	Please indicate how much you agree or disagree with each of the following statements about your coalition.	New		As discussed in the analysis plan, this variable obtains data on the degree to which coalition participants are ready and able for a comprehensive community change initiative like DFC.
		There is widespread knowledge about preventing drug abuse among participating agencies, organizations, and individuals.	New
		There is widespread support for the prevention of drug abuse among participating agencies, organizations, and individuals.	New
		There is a history of productive interaction among the stakeholders involved in designing and implementing the coalition’s drug abuse prevention efforts.	New
		Leaders of participating organizations are willing to commit resources, including staff time, for the coalition’s drug abuse prevention efforts.	New
		Financial resources (in addition to grant funds) are readily available to support the coalition’s drug abuse prevention efforts.	New
		Services and supports – formal and informal – are readily available in the community to support the coalition’s drug abuse prevention efforts.	New
Community Readiness	Likert	Please indicate how much you agree or disagree with each of the following statements about your coalition.	New		As discussed in the analysis plan, this variable obtains data on the degree to which community residents are ready and able for a comprehensive community change initiative that seeks to reduce substance use rates.
		There is widespread knowledge about the coalition’s drug abuse prevention efforts in the community.	New
		There is widespread support for the coalition’s drug abuse prevention efforts in the community.	New
		Community leaders are concerned about reducing substance abuse rates in the community.	New
		Community members and stakeholders are concerned about reducing substance abuse rates in the community.	New
		Existing programs within the community are conducive to developing interagency collaborative relationships.	New

1 For a more detailed description of focus group protocols, findings, and the utilization of a social media website to obtain information from DFC grantees, please refer to ICF’s Coalition Online Management and Evaluation Tool (COMET) Data Systems Assessment: Findings from Focus Groups and Social Media Website.

2 The GPRA measures for STOP Act will continue to be based on the current core measures as they apply to alcohol (30-day prevalence, parental disapproval, and perception of risk from regular drinking). For other DFC grantees, the NOMS calls for the addition of a peer disapproval measure. We will also drop age of first use due to the lack of reliability this metric produces.

3 For a discussion of recent research related to the health benefits of moderate alcohol use, please see: http://www2.potsdam.edu/hansondj/AlcoholAndHealth.html. Much of this research does not take into consideration potential confounds, such as social characteristics of moderate drinkers. For example, moderate drinking may be indicative of an active social life, which is more likely among younger (i.e., more healthy) people, people who do not have acute mental or physical disabilities, and people who have a good sense of self-control (which may be correlated with other healthy behaviors).

4 This total is based on the Coalition Classification Tool survey document received from the previous DFC National Evaluation vendor, Battelle Memorial Institute. Each response for ‘check all that apply’ items was included in this total.

5 It should be noted that accomplishment or barrier section can be used across many topics; however, currently it is text only. In the future, we have proposed adding dropdowns based on historical DFC data (and still keep a text field) and this will allow for more quantitative and nuanced analyses.

6 ICF created and used a community readiness scale for a child welfare-led community initiative designed to improve safety, permanency, and well-being. This scale was created after a thorough review of the research on readiness and capacity for change. This scale proved to be successful in differentiating communities ready for comprehensive change compared to their peers, and also allowed us to track readiness over the course of the initiative.

7 These strategies were developed by CADCA’s National Coalition Institute and have been slightly adapted over time during DFC program operations.

8 Please see ICF’s “Systems and Organizational Change Resulting from the Implementation of Systems of Care” report for the Children’s Bureau.

9 In this and all subsequent sections, we recommend that the end-user be given an option of “no data to report.” Because not all COMET items are required during each reporting period, grantees should be given the option to justify the reason why no data were entered. Based on feedback from individual grantees and project officers from SAMHSA, we have also included an “Other” option where appropriate.

11 Identification of best collaborative processes was identified by a Kansas University workgroup who conducted a thorough review of the literature.

12 Items marked with a “*” were used in the development of the typology of coalition maturity (i.e., Establishing, Functioning, Maturing, Sustaining). These items are being preserved in order to facilitate historical comparisons.

File Type	application/msword
Author	ICF
Last Modified By	15150
File Modified	2011-08-05
File Created	2011-04-04