Attachment 5 Coalition Classification Tool (CCT) Proposed Revisions

Attachment 5:

Coalition Classification Tool (CCT) Proposed Revisions

Recommended CCT Revisions

The Coalition Classification Tool (CCT) was developed in 2005 to provide information on the structure of DFC community coalitions and the actions they take to prevent and reduce youth substance abuse. In 2010, the CCT was revised to include items that could capture current issues from the prevention science literature. The new 2010 items—which assessed community and coalition context, coalition effectiveness, and coalition capacity—coupled with the retention of old items to maintain continuity for longitudinal analysis resulted in a longer instrument and increased burden to participants. Additional items were added to the CCT in 2015 to highlight work on and the priority of 32 important community prevention coalition objectives; each objective required responses to 3 questions. The inclusion of 45 Community Assets items increased the total number of items on the 2015 version of the CCT to 336 and the estimated completion time to 3 hours.

In 2017, ICF began a more comprehensive assessment of the 2015 CCT survey with the goal of producing a streamlined instrument that would support stronger evaluation results and reduce burden to respondents. After a careful reexamination of the 2015 version of the CCT and the way the data had been used to date, the review team conducted the following analyses: 1) examined of the wording of each item for clarity, conciseness, answerability (i.e., face validity); 2) assessed the internal consistency and reliability of each item and multiple-item measure (MIM); and 3) determined the extent to which the 2015 CCT reflected the evolving, evidence-based knowledge concerning effective community prevention efforts, and the extent to which it fills in gaps in information provided by the semi-annual progress reports (i.e., content validity).

Face Validity

The first step in the review process was to reexamine each item for clarity, conciseness, and apparent conceptual fit. This part of the review included but was not limited to making sure each item assesses only one issue. For example, the following item inadvertently asked respondents to indicate their level of agreement with two similar but separate aspects of coalition structure and leadership: Our coalition has developed common goals that are understood and supported by all partners. This item could have been improved by splitting it into two separate items to reflect the partners’ understanding of common goals and their support of goals: however, since one of the goals of the review process was to streamline the survey, this item was ultimately recommended for deletion because of inadequate variance among the response options. Other items such as, People in the community know each other, were eliminated for being too vague to prevent multiple interpretations. Still other items were eliminated because they did not appear to be conceptually appropriate to assess the constructs they were designed to measure. For example, Children in this community are spray-painting graffiti on a local building, was deemed inappropriate for assessing youth substance abuse prevention efforts.

Psychometric Review

Item Variance

The response patterns of each item were analyzed to determine how well they performed based on the extent to which respondents endorsed each response option. Patterns in responses led reviewers to believe response set bias (i.e., the tendency to give socially desirable responses) may be a threat to subsequent findings. To address this problem, it was determined that items using the “agreement” scale would be slated for elimination if fewer than 5% of respondents endorsed both “strongly disagree” or “disagree” or “agree” and “strongly agree”. Another set of items used a different scale: “Not applicable (not in place in your community)”, “In place before DFC grant started”, “In place as a result of DFC Coalition efforts”, and “New accomplishment within the past year”. For these items, those in which more than half of the coalitions selected “Not applicable” were recommended for elimination.

Multiple Item Measures Development (First Round).

The evaluation team further analyzed response patterns and inter-correlations of sets of items with the same response format to assess their individual robustness and fit for inclusion in multiple-item measures (MIMs). Bivariate Pearson correlations for continuous response options, and tetrachoric correlations for dichotomous (yes/no) response options were then run for each set of those items that met the variance criteria. Items that were not significantly correlated with any or very few other items within the set to they had been assigned were not used to create multiple item measures. Conversely, redundant items, those that were too highly correlated with other items, were not included in MIMs.

Principal components analyses (PCA) was used to identify and eliminate items that did not contribute to the measure of an underlying trait (i.e., load on two or more items in the set). Specifically, a principal axis factoring extraction method and direct oblimin rotation, were conducted on two pools of items that met the variance and redundancy criteria. Twenty-five items were included in a pool of 5-point Likert items anchored by “Strongly agree” and “Strongly disagree” and 45 items were included in a pool of 4-point Likert items anchored by “To a great extent” and “Not at all.” PCA identified four components in the “agreement” pool, and eight components in the “extent” pool that were candidates for building MIMs based on purely empirical criteria. To further help determine which items needed to be eliminated, Cronbach’s alpha reliability coefficients were calculated as a measure of the internal consistency. Items that decreased the overall reliability of the MIM to less than .50 were eliminated.

Most of the items (84%) loaded strongly (> .400) on one of seven components. Fifteen items were identified as meeting criteria for four potential measures. The success rate for the “extent” items was substantially higher than that of the “agreement” items. This may be because the “agreement” items tended to ask for perceptions and judgments, and thus were more appropriate as measures of individual attitudes and perceptions rather than reports of collective organizational behavior, which likely contributed to a stronger positive response bias. The “extent” items tended to ask for reports of factual attributes of coalition, collective member, or organizational attributes or accomplishments. Half (45) of the 89 “extent” items met the variance, redundancy, and measurement criteria. Thirty-eight of these items loaded strongly (> .400) on 1 of 9 components accounting for 58% of shared variance. Thirty-two of those items met the criteria for potential measures.

Multiple Item Measures Refinement (Second Round)

The evaluation team went through an iterative process of consensus agreement on exact item wording, the inclusion of additional items, and the use of single lead-in statement and a single item response format on 90 items and 17 MIMs. The same level of review was used with the 45 items in the Community Assets section. The final round of reviews resulted in 16 more items being recommended for deletion from the main body of 90 items and the number of items in Community Assets section was reduced from 45 to 33, which includes up to 10 optional items. The optional items are intended to allow ONDCP and the DFC National Evaluation Team to identify potential community assets over time that may be of interest.

Final 2019 CCT Revisions

Following is the final revised 2019 CCT. Without any optional items, there are a total of 96 items on the CCT. If up to 10 optional community assets were included, this would increase to 106 items. As noted, this is a significant decrease from the 336 items on the 2015 version of the CCT. The estimated time needed to complete this more streamlined version of the CCT is one hour.

Coalition Classification Tool

These items ask you to reflect on your coalition over the past year. For each item, please indicate the extent to which your coalition engaged in or achieved the activity described. For each item, think about the question in line with substance use prevention work as it relates to DFC goals. Not applicable should be selected only in those cases where the coalition never engaged in the given activity.
During the past year of working on DFC goals, our coalition:	To a Great Extent	To a Moderate Extent	To a Slight Extent	Not at all	Not Applicable
Built Capacity / Strengthened Collaboration / Collaborative Coalition Environment
Increased members’ knowledge of the work (e.g., services or programs offered) of other sector member organizations.
Facilitated opportunities for members to collaborate with one another in new ways.
Developed shared understanding across sectors that promoted innovative strategy implementation by our coalition.
Made decisions on the allocation of coalition resources in an open and participatory manner.
Had a strong feeling of cohesiveness across sectors.
Recruited new sector members who have the ability to take action in the community.
Relied upon multiple sectors to reduce barriers to planning strategies.
Increased the likelihood of a cross-system/sector approach in strategies to address emerging drug issues in our community.
Increased community perception of our coalition as the go to resource for addressing youth substance use.
Increased availability of tools, best practices, and/or other information that has informed the work of individual organizations/agencies.

These items ask you to reflect on your coalition over the past year. For each item, please indicate the extent to which your coalition engaged in or achieved the activity described. For each item, think about the question in line with substance use prevention work as it relates to DFC goals. Not applicable should be selected only in those cases where the coalition never engaged in the given activity.
During the past year of working on DFC goals, our coalition:	To a Great Extent	To a Moderate Extent	To a Slight Extent	Not at all	Not Applicable
Coalition Formalization
Maintained a current organizational chart showing coalition structures and relationships.
Followed our written description of procedures for leader selection.
Followed our written description of procedures for decision-making.
Followed our written expectations for member participation (e.g., policy on missed meetings).
Followed our written description of procedures for resolving conflicts among members.
Utilized a structure that primarily relied on subcommittees/work groups (as compared to the coalition as a whole) to complete the work of the coalition.
Utilized a structure that primarily relied on the coalition as a whole (as compared to subcommittees/work groups reporting to the coalition) to complete the work of the coalition.
Strategic Prevention Framework Utilization
Followed a systematic process for assessing community needs.
Engaged in focus groups/interviews with key stakeholders to inform assessment of community needs.
Relied on the findings of our ongoing needs assessment to guide our action plan.
Followed a plan to address identified gaps in capacity.
Emphasized practices supported by research in our action plan.
Referred to our action plan to make decisions about activities.
Completed the activities stated in our action plan.
Sought feedback on the quality of implementation of activities.
Used feedback on the quality of implementation of activities to make improvements.
These items ask you to reflect on your coalition over the past year. For each item, please indicate the extent to which your coalition engaged in or achieved the activity described. For each item, think about the question in line with substance use prevention work as it relates to DFC goals. Not applicable should be selected only in those cases where the coalition never engaged in the given activity.
During the past year of working on DFC goals, our coalition:	To a Great Extent	To a Moderate Extent	To a Slight Extent	Not at all	Not Applicable
Data, Evaluation, and Outcomes Utilization
Collected a range of outcomes data to track progress towards coalition goals.
Regularly used evaluation results to inform the community about coalition efforts.
Updated its action plans based on evaluation results.
Identified data needs to inform future program planning.
Collaborated across sectors to share data in a timely manner.
Increased awareness of substance use (e.g., prevalence, types of substances) in the community.
Increased awareness of harmful consequences associated with substance use by youth.
Decreased prevalence of specific youth use of at least one substance other than the core measures (e.g., meth, cocaine, inhalants).
Decreased prevalence of substance use in at least one specific target population (e.g., minority youth).
Decreased incidence of at least one specific risk factor for youth substance use in our community.
Increased incidence of at least one specific protective factor against youth substance use in our community.
Successfully shifted youth social norms related to youth use of at least one substance.
Successfully shifted adult social norms related to youth use of at least one substance.
Resource Acquisition Effort
Identified community organizations or members that provided facilities supporting coalition activities.
Identified community organizations or members that provided support services for coalition activities.
Developed effective strategies to recruit adult participants for coalition activities and events.

These items ask you to reflect on your coalition over the past year. For each item, please indicate the extent to which your coalition engaged in or achieved the activity described. For each item, think about the question in line with substance use prevention work as it relates to DFC goals. Not applicable should be selected only in those cases where the coalition never engaged in the given activity.
During the past year of working on DFC goals, our coalition:	To a Great Extent	To a Moderate Extent	To a Slight Extent	Not at all	Not Applicable
Building Sustainability
Developed strategies that coalition sectors will continue to support after DFC funding ends.
Established plans to continue meeting after DFC funding ends.
Established procedures for continuing to share information across agencies after DFC funding ends.
Secured funding to continue prevention efforts after DFC funding ends.
Transitioned responsibility for at least one coalition activity to a specific sector.
Improved sector members’ willingness to collaborate on new funding opportunities.
Coalition Cultural Competence
Identified the demographic composition of the coalition’s service area (from recent census data, local planning documents, statement of need, etc.) including, but not limited to, ethnicity, race, and primary language spoken as reported by the individuals.
Created a coalition cultural competence outreach plan to address cultural diversity from demographics to economic class, religion, customs, and beliefs.
Had a workgroup/subcommittee/task force dedicated to monitoring progress on the coalition cultural competence plan.
Involved sector members of targeted cultural groups in developing coalition materials for their community.
Arranged to provide materials (e.g., brochures, billboards) in the home language(s) of English language learners in the community.
Arranged to provide services/activities (e.g., training, town halls) in the home language(s) of English language learners in the community.
Considered the cultural makeup of the community when planning and implementing a strategy.

These items ask you to reflect on your coalition over the past year. For each item, please indicate the extent to which your coalition engaged in or achieved the activity described. For each item, think about the question in line with substance use prevention work as it relates to DFC goals. Not applicable should be selected only in those cases where the coalition never engaged in the given activity.
During the past year of working on DFC goals, our coalition:	To a Great Extent	To a Moderate Extent	To a Slight Extent	Not at all	Not Applicable
Community Leadership Engagement
Had community leaders actively involved in coalition committees.
Had community leaders present at coalition events.
Youth Involvement
Successfully increased youth participation in coalition activities.
Had organized youth members who planned many of the coalition activities.
Had organized youth members who implemented many of the coalition activities.
Had youth members who played a key role in developing our action plan.
Had youth members who shared the coalition’s message with the community.
Member Empowerment
Placed the responsibility for setting the agenda for coalition meetings on members.
Placed the responsibility for implementing coalition activities on members.
Placed the responsibility for what activities to implement on members.

Coalition Structure

Please indicate who is PRIMARILY responsible for carrying out each of the following activities by circling a number on the scale in which 1 suggests Primarily Coalition Staff are responsible and 5 indicates that Primarily Coalition Members are responsible.
Activity	Primarily Staff Members		Staff and Coalition Members Equally	Primarily Coalition Members
	1	2	3	4	5
66. Making budget and expenditure decisions	1	2	3	4	5
67. Identifying and recruiting new coalition members	1	2	3	4	5
68. Organizing committees and work groups	1	2	3	4	5
69 Leading committees and work groups	1	2	3	4	5
70. Developing the coalition action plan	1	2	3	4	5
71. Planning coalition activities	1	2	3	4	5
72.. Implementing coalition activities	1	2	3	4	5
73. Developing communications sent to coalition members	1	2	3	4	5
74. Developing communications sent to community partners	1	2	3	4	5

For the following community assets, please indicate what was in place before your coalition started, what is in place as a result of your coalition’s efforts (since your DFC grant started), and which assets are new accomplishments within the past year
Community Asset	In Place Before DFC Grant Started		In Place as a Result of DFC Coalition Efforts (Prior to the Past Year)		New Accomplishments in Place as a Result of DFC Coalition Efforts Within the Past Year		Not Applicable (Not in Place in Community)
Town hall meetings on substance use and prevention within the community								
Culturally competent materials that educate the public about issues related to substance use								
Social norms campaigns								
Prescription drug disposal programs								
Substance use warning posters								
Recognition programs for businesses that comply with local ordinances								
Recognition programs for drug-free youth								
Vendor/retailer compliance training								
Billboards warning youth about/against substance use								
Drugged driving prevention initiatives								
Compliance checks: Tobacco								
Compliance checks: Alcohol								
Compliance checks: Marijuana								
Formalized school substance use policies								
Responsible beverage server training								
Prescription monitoring program								
Social host laws								
Media literacy training								
Alcohol restrictions at community events								
Party patrols								
Secret shopper programs for alcohol outlets								
Ordinances on teen parties								
Optional Items TBD (up to 10 annually)
Optional 1: (e.g., Quick Response teams to connect opioid overdose survivors with treatment options; partnering with law enforcement and health professionals)								
to 32. Optional items 2 through 10								