Social Media Feedback on COMET

The Office of National Drug Control Policy contracted with ICF to conduct an assessment of the Coalition of Management Evaluation Tool (COMET) data system. The purpose of the assessment is to gain insight into the usefulness and effectiveness of the COMET data system by Substance Abuse and Mental Health Services Administration (SAMHSA) project officers, managers, and Drug Free Communities’ (DFC) grantees.

Methodology

ICF employed focus groups and the use of a social media Web site as methodologies^¹ to allow sufficient exploration of the most critical issues for the assessment of the COMET data system. The focus groups were conducted separately with SAMHSA project officers and managers. The social media Web site was established to garner direct feedback from SAMHSA grantees. This mixed methodological approach allowed for multiple perspectives to be heard from persons who directly and indirectly interface with the COMET system. Key topics for discussion included pros and cons of the COMET system, COMET as a tool to support and ensure needed information, suggestions for improvements to the COMET system, and envisioning an ideal system. Presented below are the key study findings^²:

Key Findings

Theory vs. Practice. The COMET system can be generally characterized as good in theory but not necessarily in practice. The end-users cited the conceptual value of the system and the familiarity with the Strategic Prevention Framework (SPF) as pros. However, the restrictive, redundant, time-consuming and non-user friendly nature of COMET reflects the fundamental difficulties the end-users have in actually using the system.
A Balanced Approach. A balanced approach will obtain both the technical and contextual information SAMHSA needs and the end-users want to provide. Similarly, COMET can be a more supportive system to the data collection process by being more innovative, integrated and interactive. Such modifications would ensure that the system is both responsive and reflective of the end-user experience.
COMET Usefulness. Project officers are primarily concerned with quick turnaround on key information; essentially they want to be able to report out quick facts in response to external requests. Additionally, they would like for access to be made available to external individuals who have need and desire to monitor grantee status. Managers need COMET to be more internally organized so that it can more accurately reflect the grant’s progress and grantee experience.

In summary, participants articulated that an ideal DFC data system would, first, accurately reflect what grantees are actually doing in the field. The majority feel that the current system does not allow them to report on the good impact their organizations are doing on a local/community level. Many feel confined by drop-down menus and limited space for detailed text. Second, the ideal system would be better organized and more easily navigable. Many expressed that the system should have the ability to archive old or outdated information; doing so would make their online experience less confusing and cumbersome. Third, the ideal system should be more interactive, specifically between project officers and grantees. These participants saw a gap between what is needed and/or expected in the report and what is actually received. A system that would allow for interaction during the reporting process (e.g., prompts when information is missing) would go a long way to ensuring that the reporting and reviewing process is complete and collaborative.

I. Purpose and Description of Project

The Office of National Drug Control Policy (ONDCP) contracted with ICF to conduct an assessment of the Coalition of Management Evaluation Tool (COMET) data system. The purpose of the assessment is to gain insight into the usefulness and effectiveness of the COMET data system by Substance Abuse and Mental Health Association (SAMHSA) project officers, managers and Drug Free Communities’ (DFC) grantees.

The project incorporated the following overall assessment approach to:

Provide information about your experience with the Coalition Online Management and Evaluation Tool (COMET) data system, both pro and con.
Provide suggestions for specific improvements. Also, envision the ideal data system for the DFC data collection and describe its features.
Think of how the DFC data system could provide more value to your coalition and list the ways it might do that.

II. Methodology

ICF employed focus groups and the use of a social media Web site as methodologies to allow sufficient exploration of the most critical issues for the assessment of the COMET data system. This mixed methodological approach allowed for multiple perspectives to be heard from persons who directly and indirectly interface with the COMET system. Topics for discussion included pros and cons of the COMET system, identification of key information needed, COMET as a tool to support and ensure needed information, monitoring/reviewing COMET submissions, ascertaining training and technical assistance needs, and suggestions for improvements to the COMET system.

Single-Profession Focus Groups

Two in-person focus groups were conducted separately with SAMHSA project officers and managers in June 2010. A total of 20 participants took part in the two single-profession focus groups: 13 project officers and 7 managers. The focus groups were planned for 120 and 90 minutes respectively. The participants represented SAMHSA DFC programs from urban, rural, and suburban locations nationwide.

The moderator guide for the single-profession focus groups were designed in accordance with recognized standards in the field of qualitative research by including a comprehensive introduction, primary and probing questions around each topic, and a proper thank you and closure to the group experience.

During each of the focus groups, participants were asked to discuss their experience with the COMET system, the pros and cons of the system, information needs, how COMET could better support SAMHSA needs, and their vision for an ideal data reporting system. For the most part, the same questions were asked to the project officers and the managers with a couple of exceptions (see Appendix A). The questions that differed were specific to the project officer’ or managers’ role and how they individually or uniquely interface with the COMET data system.

Recruitment of Focus Group Participants

SAMHSA, as the lead agency, extended an open invitation to all DFC project officers and managers to participate in the focus group. Attendees responded favorably and were both available and willing to participate in the focus group discussion.

Conduct of Focus Groups

A professionally trained moderator facilitated both of the single-profession focus groups to ensure consistency in how questions were asked. A second project staff member served as a note-taker during the groups.

All focus group participants received the following guidelines at the beginning of the discussion:

Audio recordings will produce transcripts. Personal identifiers will be removed.
Written input is welcomed.
One person speaks at a time after being recognized by the moderator.
No consensus is needed or expected. Express your own views and feel free to disagree.
The moderator must keep things moving in order to cover the needed questions.
There is no right or wrong answer to any question.
The moderator will be moderating, not giving opinions or providing information.

Social Media Web Site

In August 2010, ICF established a social media Web site^³ to obtain a real-time response from DFC grantees about the COMET data system. Similar to the focus group participants, Web site respondents were asked to provide information about their experience with COMET, both pro and con; provide suggestions for specific improvements and; envision the ideal data system for DFC data collection (see Appendix B).

The social media Web site was open to participants for a 2-week period. A total of 384 responses were received through the social media site. Fifty individuals registered to use the system, but did not leave feedback. Given that the responses were anonymous by design, there are minimal data or characteristics reported for the respondents.

Recruitment of Social Media Web Site Respondents

During the availability period of the Web site, ICF sent three mailings to the distribution list provided by ONDCP. The mailings consisted of an initial invitation, a first reminder approximately 1 week later, and a final reminder at the end of August. In total, 379 individuals registered to use the site: 225 from the original distribution list; 57 from individuals requesting the invitation via a technical assistance contact; and 97 users were likely forwarded the invitation by a colleague.

Conduct of the Social Media Web Site

A unique feature of the social media Web site was that respondents could review earlier postings as well as submit their own response. In an effort to maintain anonymity, the system generated an individual identification number for all respondents. The site created an open, honest, and self-contained environment in which respondents could share their experiences and simultaneously created instant and useful feedback on COMET.

III. Findings

The findings presented here are based on a thorough review and analysis of the focus group and social media Web site data. A grounded theory approach was used to identify key themes that emerged from the data and are supported by the participants’/respondents’ own words.

Feedback on the COMET Data System

In order to obtain feedback on the COMET data system, project officers, managers, and grantees were all asked: What is your general assessment of the COMET data system, both pro and con? Their input and perspectives, both pro and con, are presented as themes and supported by direct quotes from the data. (NOTE: Some of the themes presented are in response to an individual’s specific role and use of COMET.)

The Pros:

Online Reporting Viewed as Valuable. Many of the participants and respondents acknowledged the value of COMET as an online reporting system. More specifically, managers see COMET as a tool to gain important and useful insight from grantees about their experiences and progress in the field. The mere fact that COMET is an online system was viewed by a few of the project officers as positive. Some grantees view COMET as a good way to organize their data collection efforts and stimulate future planning/marketing strategies.

The data system has given us an opportunity to get some very important and useful information from grantees in terms of their accomplishments. Specifically what they’re doing, what activity they’re engaged in, and it gives us an opportunity to—particularly for the project officers, it gives the project officers an opportunity to do an assessment of how effective they are in terms of implementing the grant and accomplishing their mission to a certain degree. [Manager]

Keeps us organized and places a data collection, check and balances system with our coalition and its members. Bi-annually the entire aspect of the coalition is reviewed and discussed for the purpose of the data entry process and follow-up, in advance of our annual strategic planning meetings in May. Reminds me how thankful I am for the laptop and technology affiliated with this grant, because this data collection tool while challenging, does stir thought into planning, reporting, and marketing the information to the community it serves. The technology eases the complexity of producing the outcome reports. [Grantee]

One of the things that it has felt to me like they were trying to achieve when COMET was first established was it could do everything, and it could be a management decision tool and a way for grantees to look at their own progress, a way for project officers to look at what they’re doing and how it has improved things, a way to capture outcomes. [Manager]

I have entered Data into Comet for the last 4 years. I think the system is for the most part user friendly and obtains valuable and relevant information. [Grantee]

I found that it was very understandable and also a good way of keeping up with progress made and helps keeps us in perspective of things that we need to do for the future. [Grantee]

We’ve had some informal discussions in our team groups and we really like it that it’s online. [Project Officers]

Familiarity With SPF Favors COMET. Many of the participants, who had experience and exposure to the SPF, liked the fact that the COMET system is organized using this framework/model because familiarity and experience with SPF allowed for an easier understanding of COMET.

[F]ollowing the SPF, the Strategic Prevention Framework, areas is a good thing; it kind of maintains the language and allows us to communicate our concerns pretty effectively. [Project Officer]

I like how the COMET system follows the SPF/SIG model. It really is the ONLY reporting tool that is established for coalition/prevention work. It’s much better than many other systems I have to use, which mainly count numbers, is based on classes or treatment. So, I like the logic model it follows. [Grantee]

I think it [COMET] was systematically a good system in the sense that it did follow the Strategic Prevention Framework. I think the grantees like that, and it was organized such that it helped them to organize their coalition and that kind of thing. [Manager]

Also another comment that one of my coalitions made is that the different sections do link up so we do like the SPF SIG framework because we’re all familiar with that model and that framework. [Project Officer]

I like that the system [COMET] has categories based on the SPF. I like that the reporting system is cumulative since it helps us be able to see in one report the progression of our work. [Grantee]

Mixed Reviews for the Drop-Down Format. Some respondents expressed that COMET’s drop-down format enables them to “query information quickly.” Other respondents stated that the drop-down menus are “limiting” and do not “accurately reflect” their specific planning and implementation strategies and the system does NOT allow them to change these aspects.

One dynamic we’ve had throughout the years with the COMET system has been some of this debate here, and the debate is giving grantees the freedom to do narratives so that things can be pretty idiosyncratic or unique to that grantee and you can look and see what it is, versus the desire to be able to query systems quickly, be able to have checklists and one-word answers that are in drop-down menus that we can then aggregate and that kind of thing. So that’s been the tension, and right now we’ve erred on the side or gone on the side of checking boxes and standardizing, and having drop-down menus. [Manager]

The drop-down menus are so limiting and counter-intuitive; I finally did a print-screen of every drop down menu to try and figure it all out! I don't mind using someone else's verbiage to describe what we are doing, but I need to see the bigger picture to figure out how to make an imposed system link up with our actual work. [Grantee]

[T]he system currently has drop menus that forces them to make choices—but they [grantees] can’t fit in the information, the qualitative data. They want to enter into the report because it doesn’t fit where they think it belongs. [Project Officer]

The Cons:

Redundant, Repetitive, and Restrictive. The majority of grantees were frustrated by the amount of redundancy and repetitiveness within the COMET system. Questions prompted by COMET’s format means having to input the same information in more than one place within the report. Additionally, grantees expressed that COMET is “restrictive” and “limiting” by the categories provided. Some project officers, based on feedback from grantees, implied that there might be a technical glitch in the system whereby data entered in one section of the report is “sometimes” carried over and repeated other sections.

[COMET] is repetitive—some of our strategies are linked to multiple objectives and we find ourselves entering duplicate information into multiple places throughout the reporting system, this is very time consuming and frustrating. [Grantee]

Don’t like the repetitive process. The worst part for me is where we place our activities under each goal. There has to be an easier way to do this. I feel like I do not do that section justice because of the repetitive and difficult process to insert each activity. [Grantee]

It is a difficult system. It is hard to decide where information should go and what information is wanted. The information that you want to report does not always match with the choices available. It is extremely time consuming, confusing to work with, redundant, and limiting. [Grantee]

Once I am in the system, there seems to be a lot of repetition of information. I also feel as if I am restricted to using certain categories to identify tasks and accomplishments of my coalition. For example, I am unable to name specific events and projects from the action plan, but instead I am asked to assign the event or project a generic title. It is very confusing. [Grantee]

COMET is picking up certain information and carrying it from one section, I guess assessment, to planning, to implementation, and repeating it; so all of that data becomes kind of redundant. One of my grantees said that “sometimes the data shows up and sometimes it doesn’t show up.” Sometimes in some reporting periods, the data has made it from different sections to other sections, and in other reporting periods it hasn’t. So it’s not very consistent, the system itself, in other words. [Project Officer]

Not User Friendly. The majority of the participants cited COMET as NOT being user friendly. Specifically, the current structure/format of COMET does not allow them to make changes (e.g., add/delete old information once reports are submitted). Grantees specifically have to add any new information instead of editing or replacing existing information; this makes the reporting process “cumbersome,” “confusing,” and “frustrating” for the writer as well as the reviewer.

It is not user friendly—a few examples of this: Items are not organized or alphabetized (such as coalition lists); items are difficult to revise (IF we are even allowed to do so, such risk and protective factors, that have changes from the first 5 years to the next 5 years); there is no separation of DFC and STOP Act initiatives. Some options from the drop-downs lists aren’t a good fit of choices. Having to enter “Active; Inactive; Planned; Complete; Discontinued” each time requires us to go back to things previously entered as well as update what needs to be entered. [Grantee]

From a grantee’s perspective, a lot of my grantees have stated that they don’t find it to be very user friendly. When they go and put things in, they put their information in the first time, and then they go back to put something else in, in a lot of cases they’re talking about wanting to delete what they had in previously to put something in new, and if they put something in they’re not sure exactly what to do. So they find it not to be very user friendly, as well as in a lot of cases when they try to contact the support team, in a lot of cases they don’t get the support. [Project Officer]

COMET has many inputting features that frustrate the user and lead to simplistic entries. The SPF framework is excellent, but the information requested in each section is so redundant that you wonder why you are wasting your time on it. The feedback is minimal and often not relevant. The end result is that it gives the Federal Government the core measure data that is critical for accountability but totally misses the coalition's story for the past 6 months. The structure needs to be changed to avoid this redundancy. [Grantee]

I have to say, as a nonuser of COMET, I hear mostly negative comments on it. I haven’t heard actually any positives...a lot of complaints about not being user-friendly. Project officers and grantees not being able to see the same data or see the same screens. Not really meeting our monitoring purposes, and having a lot of gaps in what we would like to see. And from a management perspective, we don’t have—to be able to pull reports from it and be able to make it work for us. [Manager]

I have used COMET for 2 years and I do not think it is a good system. My problems with it are: 1) You can never erase old data (i.e. old members) 2) You can't make edits to certain sections so if your action plan changes, all you can do is add to COMET, you can't delete activities that you are no longer doing 3 - It doesn't appear to be useful, once you dump the information into COMET, then what? We do not receive any helpful feedback. [Grantee]

Time-Sucker. Many of the participants stated that reporting on the COMET system is a time-consuming process. The initial setup of the report format takes a great deal of time by the grantee; similarly the time for project officers to review reports is equally as long. Project officers often spend much of the time trying to find specific information that may in fact be in the report, but because of the structure it is not intuitively or logically placed.

It takes me about 4 days to complete this report, and the only way that I am able to do so is to first create a Microsoft Word document that follows COMET’s structure, and figure out how I am going to put our coalition’s accomplishments into the different parts. (And when I do a new report, I can easily see what it was that I reported the last time.) I think my reports are reasonably good because I do it this way. I would go insane if I tried to just plug our results in “live,” going through COMET screen by screen. [Grantee]

[O]ne of my coalitions expressed that initially, to set up her report it took her initially 20 hours, and then there is some pro and con because you do input some information and then it stays there, and it’s nice because when you do further reporting in a different quarter or in a second 6-month period, it’s there. But at the same, then you can’t get rid of some things, and how to tweak it is not very clear. And she said that right now it takes her about 10–12 hours, which, to me, it just seems like too much time. [Project Officer]

It takes me a good 2 weeks to enter all the information and activities our coalition participates in within a reporting period. It would be nice if we could enter the activity in once and have radio buttons or a drop down box where we could select the various categories the activity falls into (implementation, evaluation, etc.). That would save me a lot of time. [Grantee]

On our end, in being the COMET liaison, it was difficult to maintain. Every time we went through a cycle not all the grants were showing up, or grantees were showing up that weren’t theirs, some grants may have been missing or grants were there that had closed out and shouldn’t be there. So we’re constantly having to correct what’s supposed to be presented to the project officer in terms of that cycle for their review, so we spend a lot of time every cycle cleaning up, so it’s just cumbersome. [Manager]

What I want is an understanding of what happened during the last 6 months. I want to understand that and I want it to be clear, I want it to be concise, and I want it to be in such a format that I can pick up the phone and talk to somebody that put it together and can stand by that because when I go on a site visit that’s what I do, I take the COMET with me and I sit down with them and I spend unfortunate amounts of time trying to sort through to get them to actually tell me what happened. [Project Officer]

In summary, the COMET system can be generally characterized as good in theory but not necessarily in practice. The end-users cited the conceptual value of system and the familiarity with the SPF framework as pros. However, the restrictive, redundant, and time-consuming nature of COMET reflects the fundamental difficulties the end-users have in actually using the system.

Information Needs and System Support for SAMHSA

Given SAMHSA’s role as the lead agency, project officers and managers were asked: What information does SAMHSA most need from grantees? How can COMET better support SAMHSA’s information needs? Their input, including grantees (where appropriate), is presented as themes and supported by direct quotes from the data.

The What:

Need Technical Data. Both the project officers and managers expressed the need for “technical data” from grantees specifically, core measures data including, program evaluation, compliance, eligibility, and contact information.

The compliance information is very important; program evaluation and the program evaluation encompasses all of that information about what are they doing, what is their focus, what activities are they doing—that implementation information that we need. We also need organizational information and we need impact information...we need to get the technical needs information, but we also need to get information that allows us to identify issues that are happening in communities. [Manager]

[W]e have to have the core measures data every other year so we definitely check for that…We need the 12 sector data and then we need to make sure they have environmental strategies. Those are the three key monitoring pieces from the terms and conditions perspective that we need. [Project Officer]

I think it should be a place where things are standard, even with eligibility. We look to see if they have 12 sectors and then we see if they’re active. I mean it’s information there but it’s information—as a project officer I would call and say: “I see you have an inactive parent. Can you tell me a little bit about that?” We’re still going to help them with that item that’s there, but what if we could put in all that contact information. [Manager]

A lot of times the grantees, the coalition members come and go, so to have an accurate picture of who the 12 active representatives are, those they have MOUs[memorandums of understanding] with, and also any other members that are representing those sectors. So whenever they report, they can accurately report who is a part of the coalition and who’s not. [Project Officer]

Need Contextual Data. Of equal importance to project officers and managers is the need for more contextual and narrative information, such as the grantees’ environmental strategies and narratives that describe their day-to-day realities. This type of information would give the managers and project officers a better understanding and more comprehensive view of the grantees’ experience and ultimately enable them to better assist them. Grantees expressed their desire to provide contextual data that reflects their on-the-ground experiences; it is perceived as both an accurate and essential part of how they mark progress and change in the field.

The other pieces I need are the details when we ask them to write descriptions, when we ask them to describe their activities, those descriptions are the key pieces for me to understanding what they’re doing. [Project Officer]

I would like to say, however, that my greatest disappointment with the system is its inability to accurately reflect the work of the coalition and its impact on the community served. It becomes extremely frustrating when you have to try and describe the activities your coalition is implementing under goals and objectives that have changed as the coalition has changed. [Grantee]

One of the things we don’t have that’s critical to the day-to-day understanding on the part of the project officer is narrative information, simple and brief as it can be, but narrative stuff, and I bulleted some things. What’s the general progress per the action plan? The action plan is the heart and soul of the grant so are you making—can you tell us in two paragraphs what are you doing generally? [Manager]

Being able to query for all the information needs we have. Demographics, everything. [Manager]

The How:

Be More Innovative. From the project officer and manager perspective, COMET could better support SAMHSA information needs by being more innovative. More specifically, some respondents questioned COMET’s ability to map the core measure data from multiple grantees; this would be useful toward understanding regional trends and variations. Also, COMET should have the capability to query specific information so that managers and project officers are able to quickly paint a picture of a particular community’s issues and how they can address issues and be more effective to grantees. From the grantee perspective, they seem to suggest that the current structure of COMET does not permit them to highlight the innovative work that is occurring on the community level; that type of information simply does NOT fit in the box.

I’d also love to be able to map several grantees’ core measures data on top of each other, so I can see regionally what the trends look like, if there are any outliers. [Project Officer]

We don’t have the capability of pulling that data up and doing a very simple report or doing some analysis that will allow us to structure or paint a picture of what are the issues out there that communities are focusing on. We also don’t have the ability to summarize where the grantees are having problems at, and then structure a strategy or some planning to help them. If grantees are having, if the majority of grantees are having issues in their capacity development phase of the grant, we can’t pull that information together and say: “Okay guys, we need to develop a true strategy and put some resources into addressing that because we’re seeing the majority of the grantees are experiencing that.” We have no abilities like that. So we’re a bit at a disadvantage in terms of being effective to even provide support and advice to grantees. [Manager]

Create a reporting opportunity that would allow qualitative entries to complement quantitative entries to more accurately reflect the story of each coalition instead of going through the motion in reporting and documentation. [Grantee]

[T]he key pieces of the trend information is important if we could see the graphs. However, they report every 6 months, a lot of times the data information is only reported every 2 years. So you’re not going to see a trend from 6 months to 6 months until they hit that second-year evaluation review, and that’s where probably looking at the strategies to try and find out—or other assessment tools—to find out if they’re actually moving the needle, is going to be important. [Project Officer]

So just at a minimum to have it house a lot of information, I think it should be one stop for eligibility, have everything in there. And then we get into the more advanced things that we really want to look at in terms of outcomes and impact. But we need a system that does multiple things. [Manager]

Be More Integrated. Others implied the need for COMET to be more thoroughly integrated with respect to grantees’ member lists and separate databases. Additionally, there appears to be lack of integration of among various key systems such as SGEMS, IMPAC, Workstation, and COMET—all of which contain pockets of information.

[I]f there’s any potential for building an upload so that we could upload member lists from an Excel document into COMET, just to make that whole process a little smoother for them, so a lot of our grantees keep separate databases for their members and they’re very frustrated at having to enter all of their members separately in the COMET. [Project Officer]

Almost what is [being done] right now with the Workstation—where we want all the sectors to be involved, we would have that information. It just kind of seems like right now we go to four different documents to complete a spreadsheet. [Manager]

If we could just get this system to create a contact person and a name of a coalition and their telephone number and their email address, we would be 20 years ahead of where she said we were in 1993. We’d be back to where we were in 1993. That’s sort of how bad it is. [Manager]

Be More Interactive. Project officers and grantees stated that COMET would be more helpful if it were more interactive. They suggested that prompts be put into the system to ensure critical information is captured and keeps the grantee engaged in reporting process.

[I]f some of the fields were targeted that we absolutely need you to put something in that blank, or you aren’t able to submit, so when they hit the submit button and it goes back and there’s a little red comment saying you didn’t fill in this assessment section, you have to put something in there. [Project Officer]

The COMET report is primarily used by our coalition when we are trying to determine what efforts we have tried and who was responsible for activities; it is not a document that leads us anywhere. If I had the dream program, it would include a way to see what other coalitions were doing to cover the same areas of concern, or a way to connect strategies. For example, when I order books on Amazon, I receive a list of other books ordered by other customers who ordered the same book. This would cause me to be more interactive with the COMET system throughout the year as the coalition looks at ways to improve our actions. [Grantee]

[A] better COMET would be a truer picture of what they’re actually doing and that the sections interact with each other so that if they’re putting something in that doesn’t make sense, there’s some kind of flag to them and us that there’s a mismatch. [Project Officer]

In summary, a balanced approach needs to be employed in order to obtain both the technical and contextual information SAMHSA needs and that the end-users want to provide. Similarly, COMET can be a more supportive system to the data collection process by being more innovative, integrated and interactive. Such modifications would ensure that the system is both responsive and reflective of the end-user experience.

Project Officers’ Perspectives: Grantees’ Strategic
and Action Plans

During the focus group, project officers were asked specifically to give their perspectives on the following: In reviewing grantee data submissions from COMET, does SAMHSA routinely investigate whether grantees are pursuing activities that were part of their strategic and action plans? If so, is there an easier way to monitor this? The bulleted themes and supporting quotations capture their response to those questions.

A Manual Method. Some project officers stated that an examination of the grantees’ COMET submission for strategic and action plan activities is done manually. However, a larger issue raised was the timing and information needed for submission of the COMET report and the continuation application; this conflict creates “confusion” for the grantees.

I think we do so manually. [T]he way it is now they can pretty much put in whatever fits and then it’s up to us to actually try and make sense. And the data is only due every 2 years, so again it has to be less than 2 years old, so from period to period, which kind of links to the idea of doing the COMET less often. It’s really hard to do that at this point unless you really dig into it yourself. And then also, they’re trained and you’ve trained them to understand what you’re asking. [Project Officer]

[A]t a minimum they have an action plan that they update yearly because that’s required in the continuation application, but they do their continuation application for the following year and then they give us their COMET submission based on the action plan for the current year, the last 6 months. So they’re confused by that because they’ve just spent all this time writing goals and objectives for the next year, so they don’t know whether to put those in COMET or the old ones in COMET, and what exactly needs to happen. So I think we do try to keep in mind what they planned when we look at what is in COMET, but we have to understand that it’s a little confusing for them at this point about exactly which objectives, strategies, activities they’re supposed to have listed in there. [Project Officer]

COMET as a Cure. In terms of making the monitoring process easier, some project officers expressed that COMET could in fact be modified to assist with this task. The modifications ranged from simple-- providing a “mockup of an ideal COMET report”-- to the more complex – making COMET a more “intuitive” system.

We need better directions for the grantees. It would be great to have a mockup of an ideal COMET submission that they could refer to while they’re going through and completing it. That was a suggestion by one of my grantees that used to be a teacher, because she said: “We need something to reference when we’re trying to do this the right way.”[Project Officer]

[T]here would be a way to build COMET so that it helped us do that where if there was a disjoint, for instance, they can write an objective and relate it to all four core measures, which doesn’t make a whole lot of sense, but they can do it because there’s nothing in the system to stop them. Unless we come back and say: “Tell me how this objective relates to this particular outcome,” they can keep doing it. But an intuitive system might ask them for more details and help us dig deeper and hone in on problems. [Project Officer]

Project Officers’ Perspectives: COMET Trainings Needs

Project officers were also asked to give their input on COMET training needs for grantees and themselves. The key suggestions are listed below along with supporting quotations:

Training Together. Some project officers recommended that the COMET training be conducted at the same time as the grantees. The benefit of this approach would be that both parties would hear the same information; any questions could be addressed from both the grantee and project officer perspective.

I really recommend that we do it all together so that we’re all on the same page in terms of who’s being told what to do. So the training, whatever new COMET iteration is, it really is important that we’re all in it together, emphasizing those things that are the most important things. [Project Officer]

Refresher Courses and Webinars Work. Many of the project officers stated that refresher courses and Webinars work really well, specifically those that break the COMET sections down into individual components.

[W]e should always have like a little refresher course even on the sections that haven’t changed, to walk through the COMET and have maybe a COMET report next to us. We learn well from each other on systems of how people review and what they do, whether they have a running email going to the grantee while they’re reviewing the COMET report and they say in the assessment section this, and just how we methodically go about it. But I think a refresher course and getting yourself in the mindset, not too far out from when we start doing the reviews, would be helpful to me as a project officer, just to get myself back in the mind frame of doing it. [Project Officer]

For us to also be seeing what the grantees are filling out, that would be idea…a Webinar or some kind of [with] each section clearly laid out. I think some real attention and time should be spent in terms of the training. What I would like is something that was comprehensive enough so that we could refer people to, they could go online, they could look at it, they could get their questions answered, and when we amended things, when we changed things, we changed the module. We reconvene or figure out a way to communicate those changes so we’re in good shape. [Project Officer]

Project Officers’ and Managers’ Perspectives:
Increasing COMET’s Usefulness

During the focus group, project officers were asked specifically to give their perspectives on the following: How could COMET be more useful for you in monitoring your grantees? A similar yet slightly different question was asked to managers: How could information from COMET be more useful to support DFC team leaders and managers? The bulleted points highlight key areas that COMET could be more useful to project officers and managers.

The project officers recounted the need to use COMET for “quick” and more “practical” forms of grantee monitoring such as:

Quicker turnaround on regional, State-by-State, and national indicators
Allowing external access to supervisors and other agencies that can see the present status and progress of grants if project officer is unavailable.
Being able to conduct quick queries of the COMET system that respond directly to management requests.
Quick technical fixes that simplify the review process (e.g., change the font color, to distinguish old grantee information from the new).

The following quotes further elaborate the points stated above:

A point was made here to look at some of these indicators regionally, certainly State by State, certainly nationally in terms of quicker turnaround so that when we go do site visits, we have something valuable to put up in front of folks to help them gauge where they’re at in relationship to other communities, maybe even communities of like size, population, and characteristics. [Project Officer]

It would be nice for our supervisors and other agencies to be able to look and see which grants—for instance, if we go out, if we deploy, if we have babies, and we are out of the loop, that someone can step in our place and literally be able to almost see where the grants are going. And COMET would be a vehicle to do it if COMET was built in a way that it could assist us with this. And currently, it takes us reading these reports like they’re a book report to get that kind of data, and then, really, we only know ourselves unless we put it in some kind of written guidance to the grantees or to our co-workers so they understand what we understand about our own caseloads. {Project Officer]

I think it’s also important for instances where we need to know how many grantees are doing such and such, and how many grantees are doing such and such, where those drop-downs—that can be very important as opposed to me going through all of my 40 grantees trying to figure out who’s doing compliance checks or whatever…using COMET as a way to do quick turnaround requests for management. [Project Officer]

Alternatively, managers suggest the need for better organization of the COMET system, so that it may be more useful to them. The ideal system would:

Identify and prioritize the important information to be gleaned from COMET.
Be able to capture the success of grantees.
Maintain individual snapshots of grantees that include up-to-date information on key personnel and progress and serve as a repository of information.
Extract and examine the developmental process of grantees over the entire 5-year grant period.
Use COMET to establish grantee-based assessments, so that they themselves can better address the needs of their communities.

The following quotes further explain their ideas on increasing COMET’s usefulness:

To me it feels like maybe a better strategy this time around when you’re trying to fix it is to prioritize. What are the most important things we need it to do? And let’s make sure we get at least the first one or two right because probably it’s not going to be able to do everything, and maybe there’s a better way to do some of those other things. So one of the things I think it would be very useful for this group to do is to prioritize what we need the system to do, and let’s do the most important ones first, and I think that’s where [participant] was going. [Manager]

I feel like the Drug Free Communities program has been shooting themselves in the foot because you have wonderful teams doing wonderful things with people in the communities, and there hasn’t yet been to my knowledge an easy way of capturing how much success they’re having and how important they are in the communities where they’re working, and how much they’ve been changing substance abuse and drug abuse in their areas. [Manager]

I would really like to have COMET be that one place to have a repository of a lot of different documents that we would have handy because as the government goes greener and greener, we’re getting more and more away from the paper files. But we’ve got to have somewhere, especially when you’re going to transition and you’re going to have different project officers coming behind, there’s got to be somewhere where you’re going to keep and archive things. So I would like to see COMET do that, and I’d like to see us be able to look at a snapshot of where is the project officer with their portfolio in site visits, in reviewing the progress reports, in a number of different categories, and then also aggregated by teams and branches so that you’ve got a good picture of what’s going on and where people are in the progress of things, whatever the task might be. [Manager]

We’ve not had an opportunity to get the type of information, the developmental progress that the grantees are making over a 5-year period. One of the things that we do within the program is we focus on grantees in years 1 through 3, and we put a lot of our attention to those. But we are not getting direct feedback from the data that lets us know how successful we’re being, what types of strategies we need to implement to help the grantees be more successful. We do know that working with them in the first 3 years of the grant is a critical time so, yeah, that’s a critical need for us. [Manager]

One of the things the system should be doing for the grantees is allowing them to make assessments so they can make changes and be better prepared to address the needs in their communities, and help the people in their communities. That’s a big shortfall we’ve had with the system, so let’s roll up our sleeves and, yeah, get to work. [Manager]

In summary, the perspectives of the project officers and managers vis-à-vis the usefulness of COMET reflect their individual roles and how they interface with the system. More directly, project officers are primarily concerned with quick turnaround on key information; essentially they want to be able to report out quick facts in response to external requests. Additionally, they would like for access to be made available to external individuals who have the need and desire to monitor grantee status. Managers need COMET to be more internally organized so that it can more accurately reflect the grant’s progress and grantee experience.

Specific Issues Raised About Evaluation

The issue of evaluation content and its connection to COMET were implicitly and explicitly discussed by the project officers and managers in their respective focus groups. The participants expressed the following:

Purpose of the COMET as a system is unclear and warrants clarity in terms of its relationship to an evaluation.
Tension in the design of COMET: narrative vs. drop-down format. Evaluators preferred drop-down format to enable them to aggregate data more easily.
Grantee reports using COMET sometimes contain outlier information that could affect the national evaluation.
Need to know polling information for national evaluation in order to communicate to grantees.

I think we ought to clarify the purpose of this [COMET reporting]. Is it a 6-month report? It has always felt to me like it has many purposes. Whatever they are, I think it would help us as project officers to understand what they are. It has a purpose, I believe, for evaluation and parts of the information that’s gained from this equal something that you’re evaluating or used to be evaluated. It would help us an awful lot to have the connecting points. In other words, this section is very important to this piece of an evaluation, as opposed to not knowing, I’m not sure we emphasize the right things in terms of completion of the COMET. So whatever it is that this is intended to be, would be nice, it would be wonderful if we understood what that was. I think that would be useful to us. [Project Officer]

As we developed this COMET system initially, I think it never was totally clear, at least to most project officers, what was being done in the design and methodology of the evaluation of the DFC. So I do know that in the tension that occurred between narrative stuff and trying to get that very specific information and the drop-down boxes and that kind of thing, evaluators tended to go to the drop-down boxes because they could aggregate the data easier and that kind of thing. And most of the time, what we did was, we exceeded to the evaluators, so what happened I think was this: As we exceeded more and more to drop-down boxes and one little word or one little sentence thing, it got less and less relevant to you for a lot of stuff. So I don’t think there was ever an intention of doing that kind of thing but I think the effect of it was just what [participant] said, was COMET became less and less relevant to us as evaluators wanted to aggregate data more and more and make it easier to do that. [Manager]

I found outlying data in my grantees’ reports and have sent it back. They’ve gone back to their evaluator and fixed it. But that’s stuff that would have been messing up the national evaluation data had I not been able to identify it, and I don’t think we as a general population of project officers know how to do that, so we are not being as helpful as we could be to make sure the data is solid. And I think we also need some clear guidance from top down about: “You must send the reports back if these pieces are missing.” I think we have different standards based on the kinds of things that we want to see from our grantees, and then when they get transferred from project officer to project officer. [Project Officer]

We know you’re polling the core measures data so we stress to our grantees how important that is. But other than that, we’re not sure what’s being polled for things other than our grant monitoring purposes, and I think if we knew the pieces that were being polled for the national evaluation, we would stress that more with our grantees. We would make sure that they understood what kinds of information needed to go in. [Project Officer]

IV. Participant Suggestions for an Ideal DFC Data System

Better Organization, Categorization, and Navigation

It would be very useful as a project officer if, somewhere in there, they would have to confirm their current project coordinator, project director, and contact information before the application, before their progress report is able to be submitted. That way we will always have the most relevant information. [Project Officer]

Maybe there could be a program that we use more often so that we could update and add to it constantly and not have to go through so many channels to get to where we need to go to input things. I would much rather be able to input it similar to the program I have to input data at our State level. That way, I don’t have to do it more than once on two different systems. I know there is data I am not getting into the system, and so we don’t get accurate numbers. [Grantee]

[I]f we need to enter all of the people in our coalition into the system, it would be nice to be able to pull that information up to use as a mailing list, way to keep track of how many people from each sector participate, etc. Now, we have to enter all of that into COMET and keep a separate mailing list, donor list, etc. [Grantee]

I would also like to smoothly move from function to function without having to go back to the top of each page, scroll down or select from the pull-down boxes that often are not appropriate answers. It would be so much easier to use text boxes to put in new info. [Grantee]

Use other programming or databases to facilitate speedier process: user-friendly step-by-step drop-downs to get one started, and at end, walk through of steps to submit; rather than waiting to hear from PO re submission, send an email to those on administrative list (PD [project director], evaluator or COMET lead) that submission has been successful; include error-checking and send message if any problems. [Grantee]

COMET Needs To Reflect a Truer Picture for What Grantees
Are Actually Doing

I think one of the philosophies of doing this work is meeting the community where it is, and I think you were getting there when you were talking about knowing the demographics and that piece. I don’t know that I have a means of really looking at a good solid community assessment. We offer them leverage to put that in however they put it in. But if I could see in a community assessment that this community lacks a general awareness of the issues and the problems, they’re on square one. It’s tough for me to come in and start yelling at them or telling them: “You need to do environmental strategies,” and run in there and change the policies, because that’s not going to fit. [Project Officer]

[F]or the most part, people say: “It [COMET] doesn’t do what we need. We collect a lot of information but you can’t really use it very much.” It would be great if we could use it better to have a good communication tool between the project officers to try and help provide technical assistance when it is needed, to use it as a way of picking up where technical assistance is needed and to answer questions that do come up. [Manager]

[W]hen a new coalition comes in, there’s the front page of, when you open up COMET, it’s a template that tells you about that coalition, and to have it be important information. Not just that they tick everything, because really what’s in their action plan maybe, of what they’re really working on, so that you have a snapshot right off the bat of where you started with them and what this coalition is focused on. [Manager]

In addition, the capacity/membership section is restrictive in not being allowed to delete members or change their sectors. People’s lives change as do their purpose they might serve in our coalitions, for example, you have a business sector who stops working and stays home with their children and then changes to a parent sector in the capacity…it would be helpful to have a template to gather data from your coalitions. I coordinate a regional coalition which includes the 10 county coalitions which are utilizing the DFC funds. So quarterly, I am reviewing 10 separate narrative reports to gather the data for the DFC COMET. [Grantee]

To the extent that we could gather good demographic information on the coalitions, I think that would be a good starting point. I think the effectiveness of the organizations themselves by some key measures could be tremendously useful, both to kind of get some research going and get some movement going that supports the effectiveness of these organizations in community, particularly in the development of community prevention vision. So it’s kind of like the strength of the organization, the quality of their assessment capabilities, how they find problems, how they develop their strategies, the infrastructure that they use to do that, and then the outcomes. What needles have they been able to move in their respective communities? I think those three things—best practice, if you will, IT capabilities, and infrastructure—together the three of those evaluated from this system somehow could have a very major impact on our ability to seek, if you will, support for the movement nationally. [Project Officer]

Ability To Archive

If there were a method to archive completed activities, rather than having to scroll through everything each time, that would be helpful. The archived events would need to be easily accessible for future reference from the site. Another helpful feature would be if after each entry a selection (possibly multiple selections) could be checked at one time to identify what type of activity (assessment, implementation, etc.) it is, what goal it fits under, etc(to reduce repetitive entries). [Grantee]

I think a future component where you can archive the data would be very helpful. That would help us, as opposed to having 900 page reports, they can archive all the data. Also that would help us compare what you did this year and what you’re doing now. [Project Officer]

Archive data should be tabbed with an archive section under each data collection section, as if scrolling through your files on your computer. There are ample times when it is desirable to look back to the previous year’s report and check where you located an activity, or what risk and protective factors were selected and where you stand with that in this report. However, the data currently being entered should be the only info on the screens that you look at for this fiscal year's report. Long-term grantees end up having data that is immense with membership names and activities/accomplishments. [Grantee]

Appendix A

Focus Group Moderator Guides

DFC Data System Focus Groups

Wednesday, June 30, 2010

Rock Creek Conference Room – SAMHSA Headquarters

Project Officers – 9:00am to 11:00am

Management – 11:30am to 1:00pm

Research Questions:

Staff is welcome to provide written input on these questions if they are unable to attend, or if they have detailed suggestions to provide that time might not allow time for. This input should be provided in electronic format. Focus group sessions will be recorded and transcripted. The transcriptions will remove personal identifiers and will be available to all interested parties after the focus groups. Any written submissions will be appended.

Questions for the Project Officer Focus Group:

What is your general assessment of the COMET data system, both pro and con?
What information does SAMHSA most need from grantees? How can COMET better support SAMHSA’s information needs?
In reviewing grantee data submissions from COMET, does SAMHSA routinely investigate whether grantees are pursuing activities that were part of their strategic and action plans? If so, is there an easier way to monitor this?
What is the COMET training need for grantees and Project Officers?
How could COMET be more useful for you in monitoring your grantees?
If we had the most ideal data system possible, what would be its features? How, specifically, would it be an improvement over COMET?

Questions for the Management Focus Group:

What is your general assessment of the COMET data system, both pro and con?
What information does SAMHSA most need from grantees? How can COMET better support SAMHSA’s information needs?
How could information from COMET be more useful to support DFC team leaders and managers?
What reports from COMET do you need in order to optimally perform your duties?
If we had the most ideal data system possible, what would be its features? How, specifically, would it be an improvement over the current COMET system?

Please provide information about your experience with the Coalition Online Management and Evaluation Tool (COMET) data system, both pro and con. Provide suggestions for specific improvements. Also, envision the ideal data system for the DFC data collection and describe its features.

Think of how the DFC data system could provide more value to your coalition and list the ways it might do that. You can provide as much information as you like. Your posts can be viewed by others, commented on by others and in turn, you can do the same. However the system will substitute an anonymous identifier (like “user123”) for all responses. Thank you very much for your participation in this important feedback process.

Appendix B

Social Media Web Site Invitation

Appendix C1

Focus Group Transcript: SAMHSA Project Officers

VERBATIM TRANSCRIPT

DFC DATA SYSTEM

FOCUS GROUP #1

SAMHSA Drug Free Communities Project Officers

JUNE 2010

PROCEEDINGS

All right, just quickly, the background…(moderator covers Background and Guidelines).

Our first question is – What is your general assessment, your general assessment, of the COMET data system, and your remarks could be pro or con, whatever you’d like to contribute as your assessment. So who would like to be first with a general assessment of the COMET data system?

We kind of had very informal discussions among our groups. We’ve had some informal discussions in our team groups and we really like it that it’s online. There are some features of having the data, this reporting system, we like that it’s online, we also like that it follows the SPF system, we like that. Those are some of the things we’ve come up with.

(moderator explains use of microphone system)

I think on the plus side, it being online and accessible is a good thing. I think the way it’s organized, initially I think for me was kind of confusing, and as time went by I had to kind of go back and forth. There’s a lot of redundancy in it to other things that we’re doing. I guess I’d say it isn’t as clear and concise as I would like it to be. But following the SPF, the Strategic Prevention Framework, areas, I think, is a good thing, it kind of maintains the language and allows us to, I think, communicate our concerns pretty effectively.

Thanks. Other general assessments?

I’ll say something. From a grantee’s perspective, a lot of my grantees have stated that they don’t find it to be very user friendly. When they go and put things in, they put their information in the first time, and then they go back to put something else in, in a lot of cases they’re talking about wanting to delete what they had in previously to put something in new, and if they put something in they’re not sure exactly what to do. So they find it not to be very user friendly, as well as in a lot of cases when they try to contact the support team, in a lot of cases they don’t get the support. First, they can’t get a hold of anyone when they contact someone for support, so they find themselves calling the (unclear) in a lot of cases to try to get the assistance, and we’re limited to what we can do when it comes to it actually being a COMET issue. So from that perspective, they say it’s not very user friendly, plus they don’t get the support they need and deserve in order to be able to report the data they need to report in a timely manner.

Have others gotten the same or different impressions or feedback from your grantees?

This is my first time really being involved with this system and I totally agree, I’ve had the same experience that (participant’s) had with grantees contacting me and saying there’s technical things they can’t do, and sometimes I feel like I’m passing the buck because I can make suggestions, but I can’t really help them. And also, there was something else – well maybe that was it for now. But it’s exactly the same thing that (participant) said.

Others?

I think one of the things that (participant) was getting at is partly the cumulative nature of COMET, that our grantees put stuff in and once we accept the reports they can never, they can’t change it, they can’t delete it, it just stays there. So grantees get frustrated because they want to maybe change a couple of the words in one of their objective, but instead of being able to do that, they have to write an entire new objective to take the place of the first one. The first one is still there. So for us the system becomes really cumbersome because what should be a 50-60-75 page report is now 200 pages because old information is just always present, there’s no way to get rid of it or archive it. And that’s really difficult for the grantees because it confuses them when they go into the system about what they’re supposed to be updating, what objectives to write information under, just because it always lingers. So a lot of times we just have to tell them to ignore it, and that can be difficult for someone who – I have a couple of really Type A grantees that that’s not a sufficient answer for them.

Yes, along those lines, one of my coalitions expressed that initially, to set up her report it took her initially 20 hours, and then there is some pro and con because you do input some information and then it stays there, and it’s nice because when you do further reporting in a different quarter or in a second six-month period, it’s there. But at the same, then you can’t get rid of some things, and how to tweak it is not very clear. And she said that right now it takes her about 10-12 hours, which to me it just seems too much time. I think also from our end, the review process is very – it takes us a very long time; some of these documents like Olivia is saying are 200 pages if they’ve had 10 years of funding. So there’s something here that is not quite working for us on the review end. We cannot tell what’s new or old. There’s no highlighted areas. We’ve kind of talked about that before. So we also have a very short amount of time to review these. Our window of reviewing them, it’s like a 30- day turnaround. I have a portfolio of about 40-45 grants. So all of these things, we’re kind of working against each other in terms of them submitting really meaningful information that we can process, and from our end too seeing really meaningful information that we can assimilate. Another very personal opinion that I have is that the reporting really should be once a year. Given all these dimensions, I know that it may change over time, maybe this will be more short and concise and time period specific information, but right now it’s just too much information. I do also have a comment about the data towards the end of the last section on evaluation. All of those numbers are irrelevant because they’re not graphed. We don’t really see any trends. Unless we crunch those numbers ourselves independent of them submitting their own graphs, which they do submit to us and we’re able to see the data more hands-on when we do site visits. So unless we really see it from that perspective, the data part is totally irrelevant, personally speaking for me. Also another comment that one of my coalitions made is that the different sections do link up so we do like the SPF SIG framework because we’re all familiar with that model and that framework. But at the same time it seems that COMET is picking up certain information and carrying it from one section, I guess assessment, to planning, to implementation, and repeating it. So all of that data becomes kind of redundant. One of my grantees, she said that sometimes the data shows up and sometimes it doesn’t show up, so she’s confused herself. She told me - “I don’t know if I’m dreaming this.” I go: “It could be in your dreams.” That sometimes in some reporting periods the data has made it from different sections to other sections, and in other reporting periods it hasn’t. So it’s not very consistent, the system itself, in other words.

Thank you. Now the remarks that I’m hearing about too much data, the problem of old versus new commingled and hard to tell the difference; And I also heard a remark about lack of analytics as far as the data is displayed. Those things, as we get further into our questions, especially 5 and 6, remember if you have specific suggestions for how to deal with issues like those and others that you’re raising here, we’re very eager to hear those and have those on the record as well. So we’re not just interested in identifying what you may consider to be problematic issues, but also to get suggestions for how to deal with those in the future.

I just would say this one overriding thing. I think we ought to clarify the purpose of this. Is it a six-month report? It has always felt to me like it has many purposes. Whatever they are, I think it would help us as project officers to understand what they are. It has a purpose, I believe, for evaluation and parts of the information that’s gained from this equal something that you’re evaluating or used to be evaluated. It would help us an awful lot to have the connecting points. In other words, this section is very important to this piece of an evaluation, as opposed to not knowing, because I don’t think we emphasize – I’m not sure we emphasize the right things in terms of completion of the COMET. So whatever it is that this is intended to be, would be nice, it would be wonderful if we understood what that was. I think that would be useful to us. I don’t know if I’m speaking for everybody.

I understand your question. Do people feel that COMET is serving – is intended to serve one purpose or multiple purposes?

Multiple purposes.

For me it’s not very clear. I know for one purpose is for the data collection. I do know that for sure, but for the other purposes, I’m unclear. And to just reiterate the redundancy, my question would be – we just finished our continuation applications which covers exactly what they’re planning to do, and then a month after that, they’re submitting the same information in the COMET report. So that makes me question, OK, what is the information, other than the data that they’re collecting; what is the information used for such as whether they receive TA or not, or whether they’re requesting TA, because many times that section is blank. And some of the other sections may be blank, for instance when they first submitted or created their logic model, sometimes that initial date is blank. So some of the information, and when I go back to ask my grantees for the information, I’m not sure if they’re telling the truth or not, but they’re saying: “Oh, I put that in there and somehow it fell off.” So I can’t go in and see if they put it in or not. There’s no way to track that information, if it actually was put in, if they didn’t hit save. So I’m kind of lumping many thoughts together.

Okay, we’ve strayed a little bit from a general assessment, but I think the question that you’ve raised about the purpose of the system. Why does this system exist? Why are grantees asked to go to this effort? Why are you asked to monitor this, and what are the objectives and the end result uses of the data? They are certainly important questions. Before we leave that topic and perhaps question 1, are there any other impressions that you have or answers that you would give if a grantee asked you as their project officer “Why are we doing this? What is my data that I’m submitting being used for?” How would you answer that with a grantee who inquired?

Well, understand that it’s part of a national evaluation, so in some way there’s been an intent to document that the work that the DFC coalitions do has an impact in terms of reducing. So we’ve always kind of framed it along those lines, and we also understand there’s been an evolution of this report. It’s evolved over time as well. Some of our grantees who have been with us for a long time, they understand that. New ones, they just want to meet the requirements from their end, but that’s essentially one of the main things that we tell them. And then obviously, yes, we’re able to just make sure that they’re doing strategic planning, that they have submitted the data core measures. So essentially there’s 5 or 6 things that we look for when we review this plan, just to make sure they’re on the right track. So I would say that those are essentially the 2 areas that we are telling them that that is part of the point of why we’re asking them to submit these reports.

I would echo that, in that mainly we’re trying to find out where they started and see if they’re moving the needle. And then in looking at what they do, we can recommend strategies that may move the needle faster. For my part, I guess that’s how I explain it to them. And the other thing is to review their strategies to see a.) if they need technical assistance to find out what strategies might work, and if we know the communities well enough we might be able to recommend strategies as well. But I was going to make a comment on the general assessment. The biggest thing is for us to get into it and for them to get into it, because it was down over a weekend, that was a very crucial weekend, or a couple of weekends, for us and it has a chain reaction, because a lot of the people here telecommute so that they can stay somewhere and just focus on the COMET for an entire day. And if the system is down and you can’t get into it, you’ve taken the day off, you’ve made no arrangements to come to work, and you get up in your slippers in the morning and you’re ready to do those COMET reports and all of a sudden, you can’t get in. That’s very crucial. And I’m not sure whether the grantees can get in on an ongoing basis. So this should be something they should wait 6 months and be sitting down and starting to enter in. I encourage mine to enter them in as they do them. A lot of them do it on a separate system maybe and download it all in 6 months. But they can’t sit back and not record anything for 6 months and then expect to be able to do it in a half hour. The key pieces for me are we have to be able to get into it, and this is more specific, but we have to be able to open up those work plans and logic models, because I think onetime when we were doing it we couldn’t open up the work plan so we had to run back to the file and open up and dig through the file to look for the work plan on paper. So those are key issues as to whether we can get into it.

Have others heard access issues from grantees?

Yes.

Huge.

Yeah, they can’t get in.

I’m seeing most of you saying this.

Yeah, either they can’t get in, they’ve gone through – especially I had a lot of issues with my Year One grantees -- that they went through the training, they got their password, and the password and login didn’t work. So we had to go back and forth trying to figure out what the issue was, why they couldn’t log in. So I had a lot of problems with my year ones. Oh, and my Year Sixes, the ones that had their renewed – went through the competing process, and so when it came back as a Year Six, I had a lot of issues with my Year Sixes with their passwords and logins.

OK. Thank you. We need to move on to Question 2.

Can I – okay it’s in question 2. I’m sorry to interrupt.

Question 2 implies one of the answers to the discussion that we were just doing about the purpose of the system. So Question 2 is – What information does SAMHSA need from grantees, and how can COMET better support SAMHSA’s information needs? SAMHSA being represented by yourselves. (late entrant arrives) There’s a place up here, if you don’t mind coming, it’s closer to a microphone. Could you give us your name?

(Participant) project officer (voice trails off)…

Do you have a note from your mom?

(Participant), glad to have you. (laughter)

With SAMHSA -- meaning me, I’ll be SAMHSA in this case -- what I want is an understanding of what happened during the last 6 months. I want to understand that and I want it to be clear, I want it to be concise, and I want it to be in such a format that I can pick up the phone and talk to somebody that put it together and can stand by that. Because when I go on a site visit that’s what I do, I take the COMET with me and I sit down with them and I spend unfortunate amounts of time trying to sort through to get them to actually tell me what happened.

I think what (participant) mentioned about the key pieces of the trend information is important if we could see the graphs. However, they report every 6 months, a lot of times the data information is only reported every 2 years. So you’re not going to see a trend from6 months to 6 months until they hit that second- year evaluation review, and that’s where probably looking at the strategies to try and find out – or other assessment tools – to find out if they’re actually moving the needle, is going to be important.

I would agree with that, that a better COMET would be a truer picture of what they’re actually doing and that the sections interact with each other so that if they’re putting something in that doesn’t make sense, there’s some kind of flag to them and us that there’s a mismatch. Because if you have a good writer, this report can look wonderful, and then when you site visit them, the lights are on and no one’s home because they could write but it didn’t – also the system currently has drop menus that forces them to make choices -- but they can’t fit in the information, the qualitative data. They want to enter into the report because it doesn’t fit where they think it belongs. And part of that is finesse on our part and theirs, but also the system, in an attempt to kind of wean down the size and for us to get the information we’re looking for, we’re limiting what they can put in there, which they find very frustrating.

I think this maybe goes back to a general comment. Is you have the submit button, you know when you go to order something and you hit the submit button and they’ll say you forgot your address or you forgot your phone number – if some of the fields were targeted that we absolutely need you to put something in that blank, or you aren’t able to submit, so when they hit the submit button and it goes back and there’s a little red comment saying you didn’t fill in this assessment section, you have to put something in there.

(Sounds of agreement.)

That might be helpful, but that’s back to the general comment.

That’s also a suggestion for improvement, so thank you. Other comments on the information that you need from grantees as a SAMHSA representative to better do your job?

Well, just from a terms and conditions monitoring piece, we have to have the core measures data every other year so we definitely check for that. Under this question about training, we need some training on how to better check for that, and I do think the charting issue is going to help because we can see outlying data. I’d also love to be able to map several grantees’ core measures data on top of each other so I can see regionally what the trends look like, if there are any outliers. We need the 12 sector data, so a lot of our grantees keep separate databases for their members and they’re very frustrated at having to enter all of their members separately in the COMET. So I don’t know if there’s any potential for building an upload so that we could upload member lists from an Excel document into COMET, just to make that whole process a little smoother for them. And then we need to make sure they have environmental strategies. Those are the three key monitoring pieces from the terms and conditions perspective that we need. The other pieces I need are the details when we ask them to write descriptions, when we ask them to describe their activities, those descriptions are the key pieces for me to understanding what they’re doing. So if those pieces aren’t there, I send the report back because it doesn’t make any sense to me. So for a thing like (participant) was talking about, red flags for things that are missing before they finally are able to submit, that’s one of the pieces that’s really important for me is those progress sections and the description sections so that I can get a full picture of what’s happening, because otherwise it’s lots of little facts that may not stitch together to tell a complete story.

The comment you made about uploading contact information goes to the question of burden. What is the burden to grantees and to grant budgets and to their time and effort of providing what we’re asking for? Are there others that feel that being able to upload contact data and to do things like this so that they don’t do double entry, locally and then into the system, would be useful? Is that a suggestion that others have found?

That probably would be helpful to be more accurate. To be more accurate, one of the things in the system, because a lot of times the grantees, the coalition members come and go, so to have an accurate picture of who the 12 active representatives are, those they have MOUs with, and also any other members that are representing those sectors. So whenever they report, they can accurately report who is a part of the coalition and who’s not.

A couple of the other, when you talk about uploading documents, one of the things that we talked about was what’s currently helpful is the ability for the grantees to upload their logic model, but the ability to upload more than one logic model to be consistent with their training, that if they identify one problem at a time, that they can upload more than one logic model into the COMET so we can see each logic – but right now it’s one, they can only upload one.

Okay, do others think that’s a good suggestion, being able to upload multiple logic models?

Yes.

Yes, that makes a lot of sense.

I’m hearing some comments about the timing, the frequency of reports. Could you elaborate on that?

Yes, I have a comment, and I kind of want to turn this question around because I feel our job is to make these coalitions get smarter, like using catch line, make them get smarter in a quicker time. Right? So when I come out, I definitely want to make sure that the coalition has a strategic planning process going on, and ideally the strategic planning process should be an annual process. Maybe they have a 3-year or 5- year plan, hey, that’s great, but on an annual basis you really should be evaluating whether the interventions, the strategies that you currently have in your work plan, are working. And so for me that’s kind of one of the things that I really encourage grantees to do. Now I think we have – they have so much requirements going on for this very small grant that I do not think we’re doing service to them. They have two COMET submittals, they have a continuation submittal that is due probably 6 months – it’s due6 months before their new fiscal year. So all of these things take a lot of time, and I don’t think we give them sufficient time for them to really have a planning, a strategic planning evaluation process that works for them. So my suggestion is why don’t we have only one COMET that’s due once a year and that they use that reporting to really drive their strategic planning process that will inform their continuation grant. So right now the timing of COMET is not working because they submit in November, then submit their continuation grant application in March, and then they submit a second COMET report in May, April-May. So it’s too much, and unless they’re just submitting the button and repeating themselves because they lack really a sufficient amount of time to really get this done in a proper way – and there are those spectacular grants that are doing the strategic planning and they’re doing a great job with all of that, but I would say about 60 percent of the grantees really do not have sufficient time to really focus on a meaningful strategic and evaluation process internally. So one of my personal goals as a project officer is to really encourage them – you’ve got to do it and you’ve got to do it right and you’ve got to wrap it around these COMET reportings. But I do think we have too many competing things. We’re asking too much from these grantees in my opinion. Like I said, this one grantee that is a stellar grantee, she’s just absolutely wonderful, but it takes her 12 hours to get this COMET report done, and I’m not sure someone who’s not so up on their game probably it takes them 20, probably it takes them a week to get it done.

I’m not sure that there should be 2 COMETs. I am sure that I’d like to know what took place in the last 6 months, and maybe it’s the continuation that doesn’t make sense, maybe asking people to put detailed activities that are going to happen 8 months from now in these very fluid organizations isn’t making a lot of sense and confusing the issue. And I’m not wanting to go there, but the thing of it all I’m saying these processes have a tendency to overlap each other and I think you’re right, I think they’re confusing in the minds of many of our coalitions. Our best coalitions I think do handle it, but (voice trails off)…

Are there other, especially from those of you who haven’t spoken, any other – any reply, agreement, or differing experiences?

I have a question for the PO’s. I guess my question is – do you feel there is going to be a lot of change within the6-months period of their strategic plan, their logic model, their evaluation information that they have to submit from one 6- month period to the other, or if that would be better accomplished in 1year at a time? After 6 months are we looking at really a lot of new information compared to what we’d be looking at after a full year?

I think that’s a good question, (participant_. No, I don’t. I don’t see them changing anything. I don’t see any changes occurring that much within the 6 months.

I was going to say the same thing, and that was a good question. When I’m reading the reports 6 months apart and the continuation in the middle of that, I’m not seeing any change. Actually sometimes I wonder if they’re just copying and pasting the same information with each report.They are, the only thing maybe I may see the change are the 12 sectors, if they lose a sector member or they gain a sector member, or maybe some sectors will step up and begin to take leadership roles. But other than that, I really don’t see much change in the reports.

OK, yes?

I wanted to add something to that. I think for me, doing the continuation application prior to doing the COMET, it’s somewhat confusing because I feel like if I did the COMET first, I can then provide feedback to the grantees to help them then work on their action plan for the upcoming year. But by them turning in their application first, it doesn’t make sense because then I’m giving them feedback 3 months later, and they’re still having to go back and redo their action plan based on my feedback for their progress 6 months before they turned in the continuation application. So I feel like if they turned in their COMETs first, I would then be able to give them feedback and then they can use that feedback and it could potentially change their whole action plan, because I’d say 60 percent of my portfolio, they’re not getting their objectives accomplished within 6 months. Some of them aren’t getting it done within a year, so their action plans aren’t even changing. So I think, I mean if we did COMET once a year, I think it would give the grantees more time to actually accomplish their objectives, and it would give us time to actually give them feedback on their continuation application. So maybe once a year or changing when things are due might be beneficial on both ends.

Okay. It’s time we moved on to Question 3, and actually you have set the stage for Question 3 in your remarks. This question came from a colleague who wants to know – In reviewing grantee data submissions from COMET, does SAMHSA routinely investigate whether grantees are pursuing activities that were part of their strategic and action plans? If so, is there an easier way to monitor this? This is our longest question, I hope it’s clear. Can you comment on the question about the strategic and action plans?

I think we do so manually, but there would be a way to build COMET so that it helped us do that where if there was a disjoint, for instance, they can write an objective and relate it to all four core measures, which doesn’t make a whole lot of sense, but they can do it because there’s nothing in the system to stop them. Unless we come back and say: “Tell me how this objective relates to this particular outcome,” they can keep doing it. But an intuitive system might ask them for more details and help us dig deeper and hone in on problems. But the way it is now they can pretty much put in whatever fits and then it’s up to us to actually try and make sense. And the data is only due every 2 years, so again it has to be less than 2 years old, so from period to period which kind of links to the idea of doing the COMET less often. It’s really hard to do that at this point unless you really dig into it yourself. And then also they’re trained and you’ve trained them to understand what you’re asking.

You used an interesting term there that intrigues me – “intuitive system.” What do you mean by “intuitive system?”

If they’re putting in information in different sections that doesn’t link up, if the strategic plan and their activities don’t match, there could be a way where COMET would tell them and us: “This doesn’t make a whole lot of sense, provide us more information or contact your project officer before you continue,” which would build a way of them, giving them reason to contact us more often other than us contacting them asking them questions about their reports.

OK. Yes?

Well, I don’t know if the system could actually be built to check on that, but it could at least have directions that say: “These objectives should match the objectives that you’ve listed in your action plan; please refer to your action plan when inputting your objectives; objectives should be measurable,” blah-blah-blah-blah-blah. There’s no directions. We need better directions for the grantees. It would be great to have a mockup of an ideal COMET submission that they could refer to while they’re going through and completing it. That was a suggestion by one of my grantees that used to be a teacher, because she said: “We need something to reference when we’re trying to do this the right way.” But I think the issue doesn’t match the strategic and action plans. Right now we don’t even – the requirement for a strategic plan is a little ambiguous. It used to be that we required it with RFA, and now it’s not in the RFA, we don’t have a due date for it. So at a minimum they have an action plan that they update yearly because that’s required in the continuation application, but as you’ve just heard, they do their continuation application for the following year and then they give us their COMET submission based on the action plan for the current year, the last 6 months. So they’re confused by that, because they’ve just spent all this time writing goals and objectives for the next year, so they don’t know whether to put those in COMET or the old ones in COMET, and what exactly needs to happen. So I think we do try to keep in mind what they planned when we look at what is in COMET, but we have to understand that it’s a little confusing for them at this point about exactly which objectives, strategies, activities they’re supposed to have listed in there.

And since we teach them clearly that there are 2 goals and 7 strategies, they can take many shortcuts to kind of giving us what we need, as opposed to really what is going on. This “what is going on” question is the needle- moving question, and where’s the evidence answer, somehow it doesn’t come through in this report. Now it does when we go out and visit them. , That’s where we find out and make sense of it all, so this becomes somewhat useful a tool. “Oh, I see you said this, but this is what you’re doing.” That happens frequently. There’s a disconnect.

I agree. I hate to keep harboring on the point but I really feel the issue is the reporting cycle, because I feel we’re encouraging them to make stuff up because we’re saying we want to see improvements but this whole 6- month cycle, they really don’t have enough time to report on what they had in the original action plan.

Yes?

Let me get my thoughts together. I spent on the phone with2 grantees yesterday, probably 2 hours going through their COMETs page by page, and I think the biggest issue I’ve seen is that – I think someone said something about this-- but they don’t know how to write objectives, and I had to clearly say to them: “Out of 7 or 8 objectives that were put in, these are not objectives. These are activities.” So if they’re not starting off with clear objectives, the information that they’re putting in there makes no sense to me. And I think they’re also unclear as to what objectives, I think (participant) said, that should be put into COMET because some people don’t have a strategic plan. So if their action plans are changing every year, then their objectives, some of them, are changing every year, but they’re not putting that into COMET. So they have to, I guess, be trained to even write an objective before they can get down to listing strategies and activities because, to be honest, some of them will have to start all over again because these are not objectives. So I mean, their data, I don’t even know if that’s clear to them how it links, because their objectives are all activities. So I guess it’s kind of beyond COMET. They have to be trained to even know what these things are before they can even use COMET, and maybe COMET can be the source that we put something in there explaining what these things are. But it’s beyond COMET with some of this information because, again, I can’t use it so I have to then say by email: Send me something else, but then in your next COMET please make sure you have accurate objectives. But again they’ve already turned in their action plan for next year, some of them with those same objectives that aren’t objectives. So it’s a process that I think, you know, is beyond COMET at this point.

I heard you use the word “training” and that’s Question 4, our next question is about training so I’m sure we’ll have some information about that, some opinions. Other comments on Question 3?

One component I use of COMET that is useful I guess in investigating to see why or why not an objective has not been accomplished or completed is the challenges section. If they do fill that section out-- again, sometimes it’s blank and sometimes they complete that section -- it gives me an understanding. It helps me to understand why the needle appears not to be moving. And so that section has been helpful, I guess, in my investigation or monitoring of their progress, or the lack thereof.

I do have another comment regarding this. It seems like in the assessment part they give a laundry list of all the drug problems they have in their communities, but they don’t really define what are the drug areas they’re going to target with our funding. And so when we get to the work plan it’s very general again. So there’s something we can improve that part by asking: “What are the priority drug problems in your community and what are you going to target,” and then look for those, and then we can evaluate and recommend in terms of their strategies or how to define. I do agree with the comment in terms of objectives, that they may not know how to write those up, but they also are lacking focus. They don’t really know. Our best grantees are really focusing on specific drugs and they’re using environmental strategies to reduce those drugs in the community access. So in our worst grantees, they lack focus completely in terms of the drug problems in their community, and their strategies are very general.

I think too, I don’t know where – without a COMET in front of me, you’ve got them right there. The barriers and challenges section where the question is listed after the fact, I’ve been talking to grantees saying “You didn’t fill this part out” or “You didn’t write it down here,” and they can verbally tell me up in the section up above where they identified, showed that things were a barrier or a challenge, and it’s not written under that question. So I don’t know about the format there, if they need to reiterate it down at the bottom. But I have a general email that I start with and there’s a paragraph in there about: “Please fill out the challenges and barriers,” and this gal pointed out to me that it was all there, but it wasn’t where I was looking for it. And so I don’t know if that question has to be – it’s just this teeny little line that runs across there and I think they maybe even don’t, I don’t know if they see it the way we see it. But that’s really an important part for us in trying to do the TA and the recommendation of strategies and recommendations of training for them is where the challenges and barriers are. But I don’t know the format.

I’ve heard several times references to asking grantees things, emailing them for things, in other words getting information outside of the COMET system. I’ve heard that a number of times. I think that one of the purposes of our focus groups are to try to identify ways that you would not have to go around the system, that you could get that information that you need through the system, not that that would necessarily always be possible, but that’s something that we’re definitely interested in.

You could do that with COMET by having some feature where we can go in, almost like track changes, and say: “This section, please look at it, please rewrite it,” where we can highlight it, add a comment, and send it back. We do have one general square where when we release the report back for revisions we can tell them what we want them to look at, but then they have to go back, and their side looks different than what we get as well, so if we say the sector membership, the leader section is generally zero all the way down, but I follow- up with the grantee, they tell me; “No, we have leaders, we just can’t put it in properly.” But being able to actually go and highlight saying: “Please look at this section and make your objectives more specific, make them attainable, explain to me how you can measure this objective,” and we can’t do that except by either going in the little box or separate emails to them explaining what we want.

That’s a good suggestion, exactly the kind of ideas for improvements that we’re looking for.

So maybe when we talk about looking at the timing of the COMET, if there was a longer period of time between the time that they submitted the report and the final time when the data is going to be analyzed, so there’s a cutoff date when they can’t submit any more because the data is going to be analyzed. So if there was a longer period of time, if we look at the timing of the COMET, then we could have them go back and revise the report. If there are critical items that are missing, they could submit those documents so they could be included within COMET. And also improving, really using COMET as a teaching tool for the grantees, that, if they’re not Year One and they’re not coming to the new grantee meeting and they’re not going to some of the other trainings, that there are better definitions and better tools, like (participant) said, and examples of what they should, especially regarding their action plan.

Good. Yes?

I don’t want (participant’s) point to get lost and I looked on this list and I don’t know where it falls, so I just wanted to throw this out there. I think the issue that some of our grantees have is they explain things so well in the activity section, that when they get to the progress section they don’t put anything because they already explained it in the activity section. So I don’t know if better directions would help or if, when we’re thinking about how to redo COMET, if we could mash those into one thing and there’s a checkbox after they describe it to mark it as a challenge barrier. I’m not sure how we can retool that, but I think that’s the real issue. They don’t know where to describe it. Do I describe it under activity? Do I describe it under progress? Do I have to put it in both places? And that makes the report longer for us and more frustrating for them. And I really like (participant’s) idea about maybe giving us 2 months instead of 45 days to review the COMET so that we have more opportunities to send them back, because if we’re doing a thorough review, it takes 2-3 hours just to get through one report, so then sending it back and then re-reviewing it, we need some significant chunk of time to do all of that.

We’re ready to move on to training, which you’ve mentioned previously. What is the COMET training need for grantees and even for yourselves?

You know, whatever it is, I really recommend that we do it all together so that we’re all on the same page in terms of who’s being told what to do. You know, one of the hardest things, when I first took this job, was trying to understand what this thing was, and then immediately have to explain it to a bunch of people. So the training, whatever new COMET iteration is, it really is important that we’re all in it together, emphasizing those things that are the most important things. So there has to be, if you change anything, there has to be training.

Sure.

You mean for us, right?

Yes, for us.

Everybody.

For us and them together, hopefully.

I had previously heard training needed on how to write objectives and how to understand objectives.

I said that. Yes, I think it’s something that the grantees need and possibly some of the project officers. I’m not sure if, you know … because everyone’s not on that same level. So even if we’ve had training on it before, we might need it again. On our aspects, of what an objective is, a strategy, an activity – how they’re all linked. Because we do have new project officers so it’s important that we do get that training prior to doing COMET because it definitely helps me to kind of link the COMET to the action plans, because they should match, but they don’t always match. So then I have to go back and tell the grantee that, just like I said before, it needs to be redone. Again, I really don’t have time to have them do it in that time period before COMET is due for us, so then I have to have them email it to me, but then say: “But in your next report, please make sure you put this information in.” And that’s not always done because it’s 6 months later, but we have deadlines and I have to turn it in, so they don’t have time to then go back to their board and say we have to redo the action plan or redo our objectives. And the same thing with the continuation application, we just don’t have that time to sit down with them and go through it and then have them turn it back in before things are due on our end. So I definitely think we need training for the grantees and the project officers on objectives and…

Well, let’s just imagine that, based on your feedback and the feedback that we’re going to be getting from grantees, and the feedback that ONDCP will provide, let’s just for purposes of discussion imagine that there were quite significant changes in COMET. So how would you see training most effectively being done in that situation? How would you like to be trained, how would you suggest that the grantees be trained, how would we proceed with that in your opinion?

I think I’ve never been in a training where I haven’t learned something, and I think going into COMET season, especially if we’re only going to do it once a year, like with continuation sometimes I have trouble remembering, “What were we doing?” And I think about a week out from when, not necessarily when they’re – maybe when they’re due, and we’re going to start reviewing them, we should always have like a little refresher course even on the sections that haven’t changed, to walk through the COMET and have maybe a COMET report next to us. We learn well from each other on systems of how people review and what they do, whether they have a running email going to the grantee while they’re reviewing the COMET report and they say in the assessment section this, and just how we methodically go about it. But I think a refresher course and getting yourself in the mindset, not too far out from when we start doing the reviews, would be helpful to me as a project officer, just to get myself back in the mind frame of doing it. I did that with continuations this year, they were training new people and I jumped in and I was sitting there going: “Oh my gosh, I think I missed that on the continuations before; didn’t think to look for this.” And so I think something always raises your awareness, I don’t care how long you’ve been doing it. But I think the time frame close to when we start reviewing the COMETs if we did it now after we just finished them all, we wouldn’t remember by the time we got there. So even if the COMET changes come out next month, if we’re not going to do the COMET reports until later, we should wait until later to start the training. Does that make sense?

Yes?

Maybe it’s just me, but my understanding is from the grantees that I work with, as well as myself, there is no training at all in the first place. Grantees are given an ID and a password and say, go for what you know. Project officers are given their IDs-- when I first started being a project officer you’re given your ID -- and someone will sit down with you and say; “Okay, go here, click on this, print the report.” That’s the only training I received, and my understanding from the grantees I’ve talked with, that’s the only type of – they may do a Webinar, but they haven’t received any type of training at all. And if you’re going to put something new out there, because a lot of times the grantees when they see something new that’s been put out there, the only way they find out is when they go in there to start putting their information in and they see, “Oh, this is different. I’m used to putting this in this way. Now it’s something totally different.” So if you’re going to change something or make something different, you need to prepare them to let them know. Send something out. Everything is going now to Webinars. Put a Webinar out there and say: “You need to go on and you need to sit down at this point to see what the changes are or how you need to implement what you’re going to be doing when you put your data in.” Don’t wait until the time when they get ready to go in there and “Oh well, the screen is this and it’s different.” When they call us we don’t get a chance to see the screens that they see, we tell them to call the COMET support people and they say: “If they can get on, it’s not our problem,” and send it back to us. So we’re constantly – I don’t believe in finger pointing back and forth, but we’re constantly throwing things back to this one because they’ve been thrown back to us. So in my understanding there is no training at all. We have people that we go to that are the direct contacts with the COMET support or the COMET people because they sit in on the meetings, but other than that, we go to them and if we can’t get what we need, and you try calling COMET, if you get a hold tof them, you better have a specific question if you’re going to get an answer, because if you let them disconnect you might not get a hold of them again for a couple of days. So there needs to be some serious training, real training, not just give me an ID and a password and say: “Go out there.” That’s the blind leading the blind, that’s just what it looks like, in my opinion.

There actually is online training, I don’t know if it’s new. Maybe (participant) made me take it. It’s not bad, but as far as us being trained – I guess this wouldn’t work with the grantees – I personally like this kind of setup. If we could have the COMET screen up there and somebody going through it – I guess that’s because I learn maybe more visually. And ideally, for us to also be seeing what the grantees are filling out, that would be ideal, and it would probably take forever, but maybe what Fred said, we all did it together, but I don’t know how that would work. A Webinar or…

Some kind of Webinar. And the sections, each section clearly laid out. I think some real attention and time should be spent in terms of the training. What I would like is something that was comprehensive enough so that we could refer people to, they could go online, they could look at it, they could get their questions answered, and when we amended things, when we changed things, we changed the module. We reconvene or figure out a way to communicate those changes so we’re in good shape.

Going back to (participants’s) remarks about grantees being, I guess, lost or unprepared. Is there help built into COMET that’s effective, that’s clear how to use, that if you are stuck you can get help through the system, or submit a question and get answers back? Is anything like that in place that’s serving its purpose?

Actually that’s basically what (participant)was saying, and (participant), I think all of us, we should be simultaneously trained with the grantees. I, to be honest with you, I don’t know what’s in COMET because we weren’t trained on it, and the training for the grantees is, in my opinion, very generic. I’ve been following my last 2 Year Ones and between their planning and implementation section, both years have done the same thing. They have environmental strategies what they put in planning but it’s not showing up in their implementation. In both years they’re telling me the same thing. They don’t know what the problem is, and I have called the prior evaluator and no one could seem to answer that question. So it goes back to, I don’t know what could be done in COMET. I would like to be trained to see what they’re seeing, and to this point that hasn’t been done.

I think we need to say there is required training for the first- year grantees, and also there was training I think for the past2 years from the SAMHSA end for the grantees as far as the content. So maybe there needs to be more information about when these trainings are taking place, we need to make sure that these grantees are taking the training because there is training for it, so maybe they didn’t take those trainings.

I told mine they couldn’t get their password without taking the training. Maybe I lied.

Yes, that’s true.

But that online stuff is not (overlapping comments), that online stuff is not giving the support they need, online is not giving them the support they need. Maybe some can look at something, look at it and do it, and get what they need, but a large majority cannot get what they need, they can’t do that.

But I think the first one isn’t facilitated as well as the (overlapping comments)…

They’re still not giving them the support in which they need. That’s one of the biggest problems.

I see your point, (participant), but it’s not getting there. They’re taking those trainings and they’re not getting it.

Agree.

I developed a separate content based training for COMET because after my first review of COMET reports, they were atrocious, I couldn’t follow what was happening, the information I needed wasn’t there. So I developed something for my grantees and then we kind of rolled it out for the rest of the grantees the following year. But I feel like it’s much better if we have one training. So if we’re developing this new system, I would love for there to be some way for project officers to be involved in it somehow, at least to help guide the content, because currently the (unclear) solutions training is “This goes here, this goes here,” but there’s no real explanation of “This is the kind of information you would put here; this is how you tell the story through the system.” So then I have to do a follow-up training with my people so their reports are understandable. So I think if we do a new training to roll out the new system, “Here’s what it looks like, here’s what’s going on.” We should make sure that we infuse those content pieces into it. “So don’t forget, these are the kinds of information you would put here; this is the stuff that you want to make sure you carry through,” instead of it being just a technical kind of training. And in terms of project officers, I agree that we need general training, but specifically for me, I think we need training on how to review the core measures data, because I found outlying data in my grantees’ reports and have sent it back. They’ve gone back to their evaluator and fixed it. But that’s stuff that would have been messing up the national evaluation data had I not been able to identify it, and I don’t think we as a general population of project officers, know how to do that, so we are not being as helpful as we could be to make sure the data is solid. And I think we also need some clear guidance from top down about: “You must send the reports back if these pieces are missing.” I think we have different standards based on the kinds of things that we want to see from our grantees, and then when they get transferred from project officer to project officer. It’s really confusing. “Well my last PO didn’t send it back because of this.” So if there’s some clear guidance about that, it would be useful for us.

I’ve heard the phrase a couple of times about seeing what the grantees see. Could you explain that a bit more to me and tell me what you would like there that you’re not able to do, and how it would work?

It’s my understanding that there are two different screens. The screen that they input the information in is different from the screen that we see when we read the reports, and so when the grantee calls with technical issues and they say: “I put it here and here,” it may be the assessment section or the section they put some information in may not be in the same section or the same number when we read the report. That is just my understanding, so it’s hard for me to follow where they put the information in or what they’re seeing based on what I see. So it would be helpful if I could see their – I don’t know if it’s possible, but it would be good to see their screen as well as the final report.

Gotcha.

Thanks.

I do have one point about that. When we get it, we get this great chart and everything is in these nice columns and it looks really pretty, and when I have sent that back to grantees to say: “Here’s what I’m talking about, here’s the section,” they’re like: “Oh I wish I could put it in in this format.” Because when they’re putting in the information, it’s separate screens and there’s prompts and they have to hit submit. They don’t see it all stitched together. So when we say it doesn’t make sense, they don’t understand why because they’ve entered it in all these little separate pieces, and when we see it it’s laid out in this really nice grid. So if there was a user interface on their end that looked like what we get for the report, that might be really helpful, just based on the few grantees that I’ve sent my version of the report back to them, they’re always: “Oh, this makes more sense now.”

And they’re not able to produce that themselves?

They could but they don’t know how. There’s a way to produce the PDF at the very end, after you enter everything, right before you submit it, there’s an option to make a PDF. But then they would have to produce it and re-read all of it to see what we’re going to see, as opposed to them being able to enter it into a format that kind of already simulates it.

Yes, they’re not clear on how to do that. It’s actually, they just have to go under the administration tab the same way we do and they should be able to generate their own reports, but they lack that awareness. So apparently that was not part of the training, their original training, because a lot of people – some get it and then some don’t. So it’s very uneven.

So that would be a training need?

Hm-hmm. . Or it would be more helpful, if we’re looking at a section that doesn’t make sense, we can double-click and pull up what they’re populating, where we can literally go: “Oh, well, they’re not doing it.” Or that seamless where we can tell what they’re putting in and why our charts are blank, like under sector membership.

Okay, thank you, that’s very useful. We’re doing very well on time. I want to congratulate the group on staying on question, and we’re doing quite well. Anything else on training before we do our last two questions, which are very important because they’re sort of the positive suggestions for improvement? We’ve touched on some of these already I’m sure, but anything else on Question 4 before we go to 5 and 6 with our remaining time?

(silence)

Okay, general question, and you’ve touched on some of this already but use this time to summarize and to amplify on how COMET can be more useful to you in monitoring your grantees. I’d like to just mention, I have not heard any mention of the coalition classification tool thus far.

We don’t see it. (several)

You don’t see it and you don’t have anything to do with it?

(overlapping faint voices)

We just monitor whether they have submitted it or not, that’s it.

We don’t even know if they did it until we get notified that they didn’t.

(overlapping voices)

Well no, actually when they submit it, when they submit their COMET report, it’ll say “COMET” and it’ll say “CCT.” So that’s how…

I thought it was separate.

No, the last one, this time around, on that submissions sheet, the notification that you receive online, if they have submitted their CCT at the same time when they submitted their COMET report, it’ll be on that same notification.

So am I hearing you don’t make any use of the CCT data?

No.

It’s never been shared.

So what do you understand as the purpose of the CCT?

I think it’s to help with the first goal of establishing the coalition and seeing if the coalition is working well together as an infrastructure That was my understanding of it, and how they’re progressing and going from conversation to coordination to collaboration and moving the needle on how well the infrastructure functions. Is that what it is?

(Laughter)

Did I just say something really stupid or was that just a hopeful comment on my part?

I don’t answer questions.

(laughter)

Was that just a hopeful comment on my part as to what I was telling them?

So leaving the CCT aside, (laughter), how can the data collected in COMET be more useful to you in monitoring your grantees? And that’s sort of where the rubber meets the road here.

I think in just a ton of ways. If we can decide what data we want a little bit and if we can talk about how we cut it up. A point was made here to look at some of these indicators regionally, certainly state by state, certainly nationally in terms of quicker turnaround so that when we go do site visits, we have something valuable to put up in front of folks to help them gauge where they’re at in relationship to other communities, maybe even communities of like size, population and characteristics. So there’s many, many ways of taking that data, I think, given where it’s coming from, and using it in a much more valuable way. Up to now I don’t think we’ve – I would say we’ve had just a few reports on the information nationally, and they were taken back. We were told we couldn’t use them -- after we’d passed them out to everybody.

We won’t go there but…

I just thought I’d bring it to your attention.

Yes?

We recently looked at a monitoring system within SAMHSA where there was actually a drop-down menu of all the grants in your caseload with a dashboard, where you could go to your grant, click on it, and then do a statewide comparison of the data that they were turning in, and then look at errors and submission of data so that you could say they’re making on average 12 mistakes, and the mistakes were minor, but you could literally say they’re falling out, they’re different than the rest of the grantees for the following reasons, which gave you a way to follow up with them and address specifically what they were doing, which would be nice because we do ask for environmental strategies. Well, it’s not just the numbers and the core measurement data, but also looking at, what are their environmental strategies? Are they out of scope of their neighbors? And what are they doing with them? Again, we look at the reports and read it and hopefully are coming up with something in that neighborhood right now on our own, but it would be nice for our supervisors and other agencies to be able to look and see which grants – for instance, if we go out, if we deploy, if we have babies, and we are out of the loop, that someone can step in our place and literally be able to almost see where the grants are going. And COMET would be a vehicle to do it if COMET was built in a way that it could assist us with this. And currently it takes us reading these reports like they’re a book report to get that kind of data, and then really, we only know ourselves unless we put it in some kind of written guidance to the grantees or to our co-workers so they understand what we understand about our own caseloads.

Is the data that’s useful to you – correct me if I’m wrong. I’m hearing that it’s more the narrative data where people write as opposed to where they make selections, yes and no, select all that apply, the more quantitative.

It’s qualitative.

It’s the qualitative data that is mainly useful to you?

Well, that’s what we get.

I think it’s two things. I guess for the purposes of the reports, the qualitative part is important, but I think it’s also important for instances where we need to know how many grantees are doing such and such, and how many grantees are doing such and such, where those drop-downs – I don’t know how you would pull that in a report but that can be very important, as opposed to me going through all of my 40 grantees trying to figure out who’s doing compliance checks or whatever. And that may be in their action plan or in another section, but I’d still have to read them all as opposed to being able to – like if they clicked, if that was an option for them to click, maybe being able to pull that together quickly versus going through every COMET. So I don’t really know how that could be done or if it could be done in COMET, but possibly using COMET as a way to do quick turnaround requests for management.

Data mining.

Just to add on to that, for example, grantees that are working with prescription drugs and take-backs, just categorizing more information so the system can be queried quickly when there are quick turnaround requests. That would be (voice trails off)…

So I’m hearing that a combination of qualitative and statistical type data that can rapidly be extracted is probably desirable for monitoring purposes?

I think the essential question really is what does good work look like, and I think (participant). you made a comment that a good submission might be helpful, maybe it’s internally for training purposes, for the PO’s maybe it’s for – I’m not sure how much of it is used for training for grantees because grantees we’ve been told they copy good plans or good whatever reports. But for me personally, what something really good looks like, whether it’s a comment, whether it’s a site visit or whether it’s in their continuation application, is that they are targeting – I made this point already by the way – that they’re really targeting specific drugs, that they have significant community involvement from the sectors, because that’s really what’s going to drive policy change and it’s going to help to implement environmental strategies, and obviously that their work plan is very comprehensive in that it addresses those specific drugs, and that they’re heavily focused both on media campaigns that build the support in the community to implement these environmental strategies. So essentially that’s what a good grantee looks like to me, and sometimes even a good grantee cannot convey that in COMET, so that’s why for me personally COMET is just another tool, it’s not the “it” one. It’s nice but hopefully it can become the “it” so that it really gives me a complete picture, but right now it doesn’t, and yes, I look for all these other things to complement that perspective that COMET is definitely not “it” for me, independent of everything else.

The 4 core measures are very important for us to look at from a national perspective, but there are many things that these coalitions accomplish and do and set out to do, and the tying together of their planning system to these outcomes, to these products or to these success stories becomes extremely important in terms of the future of this movement. I think ONDCP knows this and I think SAMHSA knows this as well or is learning this, particularly as we move into health reform and the importance of prevention, period, what that word really means and how it ought to be organized in communities. So the question becomes, what else would be useful to help portray the effectiveness of the movement? I’m asking you to think about that. I don’t have an immediate answer for you.

Yes?

I think just on a practical note too, we were talking about how the new information sometimes gets lost with all the old information. Something as simple as having that appear for project officers in a different font color, so everything else is in black text for the old information, and the new information is in blue, so we can very quickly scan and identify the new pieces of information would simplify things a lot for us. I would imagine that wouldn’t be too difficult in terms of coding. And this whole thing with being able to graph the data and the things that Fred was suggesting in terms of being able to even graph it against state and national, I think that would be incredibly useful for us for feedback for our grantees.

And also wouldn’t it make sense if, for instance, like we get a lot of queries from ONDCP like:“Which coalitions are serving the need of Americans” or: “Which are focused working with the Latino community” or: “Which ones are doing prescription drugs take-back?” Wouldn’t it make sense to use this database to plugin and request that to do a query and then have it pop out from the COMET system itself?

Does it pop out from the journals?

It doesn’t pop out from anywhere. It pops out (overlapping comments / laughter)…

So you don’t have the ability to do any live query of the data?

Nothing.

Is that correct?

Not that I know of.

Not that we’re aware of.

Do the grantees – you’re talking about graphing and so forth – what is your understanding of the reporting that’s available to the grantees on their data or on aggregate data?

There hasn’t been – was there ever aggregate data given to them? Other than the reports I mentioned…

So they would wait for the national evaluation type reports?

Right.

There was one I think three years ago, there was a national, there was one that was done, and then there was an abbreviated couple of reports that were passed out I think last year. So they haven’t received…

There’s been no feedback to grantees that I’m aware of.

Would you imagine that it would be useful to grantees if they were able to do immediate reporting from the system?

Yes.

We’re merging here into Question 5 which is good – I mean Question 6. It’s time for it anyway. Let’s just expand into a general discussion where we imagine a world where we have no limits here, we have plenty of funding, we have great intelligence, we have creativity, we have the ability to work out, have dialogue, work out requirements, and to have the most ideal data system possible. What would be its features and how would it differ from the present system? That’s our final question.

I think starting with the characteristics of the coalitions becomes very, very important. Population size and characteristics of community have a great deal to do with the integration of sectors and their ability to accomplish things. To the extent that we could gather good demographic information on the coalitions, I think that would be a good starting point. I think the effectiveness of the organizations themselves by some key measures could be tremendously useful, both to kind of get some research going and get some movement going that supports the effectiveness of these organizations in community, particularly in the development of community prevention vision. So it’s kind of like the strength of the organization, the quality of their assessment capabilities, how they find problems, how they develop their strategies, the infrastructure that they use to do that, and then the outcomes. What needles have they been able to move in their respective communities? I think those 3 things – best practice, if you will, IT capabilities, and infrastructure – together the 3 of those evaluated from this system somehow could have a very major impact on our ability to seek, if you will, support for the movement nationally.

Let me just probe to ask. Do you see any – and I’m not saying that I’ve sensed this at all, I’m just asking this as a question – do you see any disconnect between the grant monitoring information needs that you have and SAMHSA has and the national evaluation needs on the other hand that ONDCP has and that our contract is also connected with? Or are those simply two sides of the same coin and a common kind of thing, and if you’re interested in one, you’re interested in the other, and they should be together, not separated? If I’m making that question clear enough.

I don’t think we’re clear on all the things that you’re polling for the national evaluation. We know the CCT is for that and we don’t get that information, so we know there’s information about our coalition’s development progress that we don’t know. We know you’re polling the core measures data so we stress to our grantees how important that is. But other than that, we’re not sure what’s being polled for things other than our grant monitoring purposes, and I think if we knew the pieces that were being polled for the national evaluation, we would stress that more with our grantees. We would make sure that they understood what kinds of information needed to go in. If it was entered incorrectly, we would have them clean it up. So in my mind it should all be one report, they shouldn’t have two separate reports or two separate processes to get information that you need versus we need. But I think we want to make sure that we understand what you need and how we can help you get it in the best way possible, because we know the national evaluation is what keeps our money coming in terms of funding our grantees, and they care about that, they want to be able to provide the best information they can to keep Congress supporting this.

Do others feel that you need more information and an understanding of the national evaluation?

Yes.

What about the grantees, do you feel that the grantees need more understanding and information?

Yes, and they frequently ask.

I kind of see the progress report, trying to measure their progress as kind of short-term goals, and the national evaluation pieces or the core measure pieces at the end. The complete progress of the CCT information as being more of the long- range end of things, because we are only measuring that every 2 years, and you probably aren’t going to see a 5 percent decrease in something in 6 months. So that national evaluation piece seems to me as more of a long- range plan. If you were looking at a 5- year plan from your grantee, that’s what you would hope to see on the end outcome, but in the meantime we need to monitor the progress of what they’re doing in the middle.

Yes, but I think it kind of boils down to having an accurate picture of what is going on, and also research and analysis that really has not been shared. If it was done, it wasn’t really openly shared with program officers or with grantees themselves, so we understand that something was done, and I’ve been here for about 6 years so in the first couple of years, there was some conversation going on between the different DFC partners. That was not sustained, and in some way we don’t have a clear picture. I mean what I essentially try to do at my level is what is going on with my state, what are the rates, are they going up in the specific coalitions, in the specific communities that they serve? Are these rates up, down? Do they have a good handle on this themselves? Because sometimes it’s very difficult for even a coalition to evaluate their own data. This requires more sophistication, and some of the non-profits may not have an in- house evaluator. But I guess now that I understand my state, well how about if I then, if you’re working next door to California, what’s going on in Oregon? We’re not going beyond that so we’re not really seeing a regional picture, we’re not seeing a national picture, unless we do the work ourselves on an individual basis, we’re not having this bigger discussion, and I think we should be having this bigger discussion. That way we can have a handle on what’s going on. I do support that we’re kind of having this forum to actually bring up those ideas so we can hopefully move to that model where we’re looking at things from a national perspective and we’re having that level of discussion. It’s very important for the movement, like you said.

Okay, we’re continuing here to vision an ideal system, and wanting your thoughts about what that would look like both from the standpoint of content and policy, but also from the standpoint of the Web-based system itself and the process. So we’re looking both at the process and the outcomes.

I’ve been rolling stuff around in my head and it seems like it’s grinding, maybe because it feels like it’s a Monday morning. I think one of the philosophies of doing this work is meeting the community where it is, and I think (participant), you were getting there when you were talking about knowing the demographics and that piece. I don’t know that I have a means of really looking at a good solid community assessment. We offer them leverage to put that in however they put it in. But if I could see in a community assessment that this community lacks a general awareness of the issues and the problems, they’re on square one. It’s tough for me to come in and start yelling at them or telling them: “You need to do environmental strategies” and run in there and change the policies, because that’s not going to fit. But if there was some way to look at the community assessment maybe on a graph thing, maybe on a bar graph thing, and say the majority of the people don’t think underage drinking is an issue, and then I can go down and look at the plan and the strategies – and maybe this is what we’ve been saying, I’ve been hearing altogether – is that looking at their plan and the strategies that they’re going to implement, I’m hoping the majority of their strategies are going to work on the community awareness piece because that’s where they are. Maybe we feel they weren’t ready to be awarded, but that’s beyond the point because they have been awarded, and if they’re just at that base community level of awareness we can’t be saying to them: “You need to do strategies 4, 5, 6 or 6, 7 and 8,” because you’re not going to walk in there and get that to fly. So maybe some way of – am I making any sense? Somebody help me out because I’ve been sitting here trying to figure out (overlapping comments) how to put it in there.

There’s going to be many levels of a coalition and our job is to help them wherever they’re at so they can move down that continuum.

So they may start out with the community does not have a high level of awareness so a lot of their activities are here. Hopefully after 2-3 years they’re more up on the end of where the community is and now the strategies are shifting in that direction. But since we don’t have them do a 5- year plan anymore, I’m not sure how we move them there. The other thing is, in talking about any type of evaluation, it’s kind of be careful what you ask for, because we could have a community that we reviewed that has an enforcing underage drinking laws grant, that has a SPF SIG grant, that has a DFC grant, that has a state grant, that has…

Tobacco.

…everything. And they can move the needle 5 percent. You can go over to the community next door who’s had a tragic accident, lost four of the brightest and best kids that they had, and their reduction is 10 percent. So it’s really hard for us to say the DFC money accomplishes this. It may contribute to the reduction, but I don’t know that if you can say dollar for dollar that it attributed to the reduction. So I think that’s something we struggle with as well. When we’ve talked about it with the STOP Act, it’s the same thing – some community can get every source of funding that’s out there and not move the needle as much as one that just enforced a curfew that they didn’t have before. So I don’t know how we measure that and I don’t know how that comes out in the national evaluation. We’d love to say the DFC communities – and I think we can make general statements, we’ve tried before – the DFC communities are doing better than the non-DFC communities. But there’s lots of contributing factors there, so I think it makes a contribution to the overall decrease that may not be attributable.

This may be very minute in comparison to what (participant) just said. But we’re constantly being asked for the grantees’ updated contact information, the project coordinator, the project director, that information. It gets lost in our current system which is SJEMS (spelled phonetically). I know this has nothing to do with the national evaluation, but it would be very useful as a project officer if somewhere in there, they would have to confirm their current project coordinator, project director, and contact information before the application, before their progress report is able to be submitted. That way we will always have the most relevant information. I mean something as small as just their relevant contact information, email, because many times on the cover sheet, this sheet that we get here, the project coordinator that started when the grant was first started, it may be the project director or the project coordinator, but it’s changed 3 times since then and for whatever reason they haven’t changed it in their system. So this information on the cover sheet many times is inaccurate. So if there’s some way that we can get just a confirmation of accurate information – although they’re supposed to submit any key personnel changes, many times they don’t. But somewhere before the report can be submitted, they have to check their current information, that would be helpful, that would be ideal for me.

I’m glad you made that comment because I wanted in just the short time we have left to make sure that I prompt you for specifically comments, suggestions about the data system itself. What is COMET doing that you wish it wouldn’t do? What is COMET not doing that you would like for a system to do? And what would our ideal system, what ought it to be? Are there any other specific comments on those topics?

I think a future component where you can archive the data would be very helpful. That would help us, as opposed to having 900- page reports, they can archive all the data. Also that would help us compare what you did this year and what you’re doing now.

Yes, and we’ve heard that before, so the archiving of data and the separating of old and new. And I guess that includes the question of to what extent are grantees able to go back and change previous data and would bring up the issue of data locking? Should there be a data locking function that after some period of time you wouldn’t be able to change data that you’d previously submitted? Anyway, other suggestions about the COMET data system itself?

It would be nice, one of the functions of reading COMET is to recommend TA, and it would be nice if COMET communicated directly with the work station at CADCA where we could, instead of saying “Please contact CADCA, or we’re contacting CADCA for you,” that you click and it starts linking, and that the system sends out reminders to the project officers and the grantees when it’s due, what’s missing. It does send, if you return a report to the grantee to change it, it’ll send them a letter that says it’s been released with a note from you saying hopefully why. But making that more seamless and giving us reminders of when things are due, and if the grantee has a question. Some of the TA with COMET it is our responsibility and if they do have questions, they can literally be in the middle of a page and send us something saying: I’m lost, please contact me,” and the system remembers where they were. Because some of our grantees really do just have trouble getting back to where they were lost sometimes in the beginning, and starting to talk to them about that. But having a system that just communicates where we’re not going in and pulling up an entire 60- page report, and something that’s searchable on our end where if we’re looking for something specific. Most of the time if you’ve read 6 COMET seasons, you kind of know where you’re going, unless the grantee was really all over the place on where they put the information in. So that would be nice.

Okay, thank you, you’re on a roll. Others?

An additional piece as far as evaluation feature that may be helpful would be the ability to upload additional evaluation reports. Some of the grantees do have evaluation reports that are beyond the 4 core measures or may be, for example, some of my grantees have done informative evaluation, they have reports, so that could be a part of the documented piece for COMET instead of them sending it separately. That might be helpful.

Other suggestions, ideas, brainstorms?

In talking about uploads, it would be great if they could upload their sustainability plan also because we collect that separately, so it would be nice to have that in one place, and maybe even to give them a non-directed optional upload item, because sometimes they just have something they really want you to see, and it would be good to be able to capture them all in one place.

It would be kind of nice, following up on what we said on the research, that once you receive this data ICF, the way you interpret the data and the analysis that you do, that if on an annual basis that your analysis is shared with us and the grantees, that would be really helpful. I do think it’s very valued by the grantees when they hear back in terms of feedback in general, whether it’s at that coalition level or whether it’s at a state level, it’s very important to them. And I specifically want to say that one of my coalitions, recently I was talking to her and there was a change in the mayors in this city, and apparently they went and somehow the new mayor visited ONDCP, and ONDCP over there told them that they’re doing good work, so now the mayor is supporting her coalition. So it does mean a whole lot to these people when they’re recognized about their accomplishments and the work they do. It makes a huge difference.

I want to echo that. Information that notes progress and effectiveness of coalitions is oftentimes used immediately by the coalition to really build capacity and have derivative side effects that can be very, very valuable to the community. So you might want to keep that in mind.

We have a couple of minutes so I want to conclude and I really want to thank you for taking this time and for sharing these thoughts…(closing comments) (End.)

Appendix C2

Focus Group Transcript: SAMHSA Managers

VERBATIM TRANSCRIPT

DFC DATA SYSTEM

FOCUS GROUP #2

MANAGERS

JUNE 2010

Note: As standard transcription procedure, names have been withheld for confidentiality.

PROCEEDINGS

If we could do brief introductions—the previous session were all project officers so perhaps if you just give your name but also indicate your role in regard to the DFC effort.

I’m [participant], and I’m the ONDCP liaison for the Drug Free Communities program.

I’m [participant]. I’m a team leader but I used to be a community partnership grantee for many years, a DFC grantee starting in 1990. I also served as the COMET liaison for our branch.

So you attended some of those community partnership trainings back then.

Oh yes.

And here is [participant], the man.

[Participant], my main man. I haven’t seen you since we went on a long bicycle ride.

Are you a participant?

Yes.

Would you mind sitting over here please?

(overlapping comments)

I’m [participant]. I’m the acting branch chief, and I too used to have a DFC grant in Chicago.

[participant], team leader.

Hi, I’m [participant]. You just took me down memory lane. I used to be the subcontractor (overlapping comments) way back when that started, back in the late ’80s, early ’90s. So he just did a rendition of the last 2 years.

Oh, I forgot the national training system.

Yes, I was (inaudible/faint). Anyway, I’m currently serving as branch chief, and next Tuesday I’ll start as the acting division director.

I’m [participant]. I’m the team lead with this. Did you and I go to Pennsylvania once to look at this system?

Yes, we did.

It just dawned on me. You and I went once too, didn’t we?

Yes.

So we go way back with all this stuff.

We won’t get started reminiscing but…

There’s an interesting circle here.

Yes, there is. OK, enough of that. I’ve got to put on my moderator hat now… [moderator covers Background and Guidelines].

Question 1 is the same question that we began with the project officers, just a general assessment from your perspective of the COMET data system. Who’d like to begin?

I’ll begin only because I’ve written some stuff, and I can give you a copy of this. Although the third word there is a typo, although it may be subliminal. But what I said here in terms of my general assessment in the pros, I actually said it was— I started to say ‘good’ but I had ‘goof’ there. But it was a good idea to have one tool, so my general assessment is it was a good idea in the beginning, and I remember this clearly why, to have a mechanism that could actually capture compliance and progress via national evaluation tool, and really be a place where there was a repository of information that could be accessed. And I think in the beginning, at least the beginning that I’m aware of, that was kind of the general idea, the general theory, that COMET can serve all these different communities, that is project officers and (unclear) be a national cross-site for data, and be a repository for this typology, blah-blah-blah. And those are all good things, I think, and it was a grand idea. From my point of view, COMET never lived up to its billing. It may have done bits and pieces of that stuff over time, but it never really became the thing that it was billed as becoming. And it may have some elements that are positive like the ability to have national cross-site data in it, but the con is how good is the data relative to X number of survey answers. So my general comment is that COMET was a good idea in the beginning, that the ability to have those kinds of tools available on different populations in one place and be user-friendly is great, but I don’t know that it ever became user- friendly. I don’t know that it ever really actualized any one of those things in an exemplary way, and I’m not even sure how functional it was as a data collection effort over 5 years.

Thank you, [participant]. Other general assessments?

I would second a lot of what [participant] said in the sense that I think it was systematically a good system in the sense that it did follow the Strategic Prevention Framework. I think the grantees like that, and it was organized such that it helped them to organize their coalition and that kind of thing. The main problem that I saw out of COMET that never really realized its potential was I don’t think I ever saw an aggregated report that aggregated the data, that compared the data coalition to coalition, or any longitudinal data where we could look and see whether we were making a difference longitudinally. So without aggregated reports, we were always on an ad hoc basis trying to put together data when ONDCP or somebody else would ask for something. We were always were scurrying around, calling grantees, and trying to aggregate data that this system was supposed to do for us, and that never happened.

Yes?

I was just going to say, you know, it captures some data but not everything that we need to assess maybe where a grantee is at. The other part that I didn’t like about it was I felt like it wasn’t grantee-friendly. We said it was going to build each time, and grantees put information in, and then they say: “I can’t take this out and it’s supposed to build, but we’re not even doing this in our action plan anymore.” So you just had a multitude of information in there, but it really made no sense. It wasn’t even project officer-friendly, to me. It doesn’t tell me a lot. I think we get more information if we do our continuation or we ask narrative questions, we usually get more information. That’s just my opinion.

As I said, we are going to ask the grantees for feedback and it will be interesting to see what feedback we get. Do others of you, have you heard grantee feedback pro or con, or in your position do you get that? I guess you would probably get that more from the project officers.

I have to say, as a nonuser of COMET, I hear mostly negative comments on it. I haven’t heard actually any positives until [participant] just made a few. A lot of complaints about not being user-friendly. Project officers and grantees not being able to see the same data or see the same screens. Not really meeting our monitoring purposes, and having a lot of gaps in what we would like to see. And from a management perspective, we don’t have—to be able to pull reports from it and be able to make it work for us. So I’d say 100 complaints to 1 positive.

I was a project officer until recently, and also the team leaders have grantees that we work with, so we use COMET. I just had a few things— I agree with [participant] when he was talking about the original concept of it being an all-in-one product. It reminds me of those all-in-one printers-fax-scanners, and when they first came out they couldn’t do any one of them really good, but they could do them all. So I don’t know if that was unrealistic, trying to accomplish multiple goals and some of them maybe didn’t mesh very well. So I kind of agree with that, and I think that kind of led to a system that’s not user-friendly. That’s the biggest complaint that we got, and I think you hear that from project officers as well as the grantees, but certainly from the grantees. And on our end in being the COMET liaison, it was difficult to maintain. Every time we went through a cycle not all the grants were showing up, or grantees were showing up that weren’t theirs, some grants may have been missing or grants were there that had closed out and shouldn’t be there. So we’re constantly having to correct what’s supposed to be presented to the project officer in terms of that cycle for their review, so we spend a lot of time every cycle cleaning up, so it’s just cumbersome. And that’s just one example. There’s others—the constant updates and problems with getting updates to being accurate and everything. And then for me, having been a grantee and understanding what we’re trying to achieve ultimately, it seemed to me it was primarily focused on capturing activities. So when I became a project officer having been a grantee it’s like, I don’t care how much you’re doing or all these activities, I want to know what changes have you made in your community that are resulting ultimately in reducing problems related to drug abuse. And then I saw CADCA had a system that was capturing changes, and I was kind of jealous. I would love to have something like that because that’s what I want to see. And not only did it capture changes, they were instructed and taught what a change is, because a lot of people I think when they started using that system were reporting activities when they weren’t really changes. So I would love a system that trains grantees on what a community change actually is and a system that captures it, so we can start connecting those changes to the results and the evaluations. Is there a connection from these changes to reductions in problems related to drug use or prevalence rates? And make those connections.

Thank you. When we get to our question about the most ideal system possible, perhaps you can reflect on what you saw in the CADCA system that particularly appeals to you that you’d like to see in a future data system for DFC. We’ve been joined by a new participant. Would you introduce yourself?

Sure. My name is [participant]. I’m acting division director for the Division of Community Programs for a short period of time. Unfortunately I’ll be transitioning from the Drug Free Committee program, but I have worked on it for the last 5 years so I wanted to come down and certainly provide some perspective to you as you guys are going to be working on improving the system, data support system.

Thank you, [participant]. We just really have gotten started. The ground rules are standard things for focus groups, and I did mention that we are getting input directly from the grantees through another mechanism, and we’re interviewing ONDCP, and we’re just getting started on Question 1. Would you like to comment on your general assessment of the COMET data system, pro or con?

I’ll make some pros as well as some cons. The data system has given us an opportunity to get some very important and useful information from grantees in terms of their accomplishments. Specifically what they’re doing, what activity they’re engaged in, and it gives us an opportunity to—particularly for the project officers, it gives the project officers an opportunity to do an assessment of how effective they are in terms of implementing the grant and accomplishing their mission to a certain degree. The con is that for the grantees it was a bit cumbersome, a bit complicated. I think that in all honesty. I think there were probably major revisions done to the system every year for the 5 years that I’ve been here, and it was always done in an effort to make it more effective and easier for the end-user. I don’t know how effective it has been; we still get comments back from grantees indicating it’s a bit cumbersome. And ultimately it didn’t provide either the program or the grantee or the evaluation contractor all of the information that they really wanted to be able to glean from the program.

I think [participant] and [participant] sort of hit on it, and I agree. There was never any incremental resolution to ongoing issues to speak of. So each year there would crop up this opportunity, or periodically there would crop up these opportunities, to fix something or to change something or to get feedback and then we’d keep going forward, but going forward in a way that you took care of whatever it was. And it seems like with this—[participant] says 5 years—it seems like with this thing, it never seemed to arrive at any real place of betterment over the period of time, at least in my opinion. And that’s daunting actually when you stop and think about it, because it wasn’t like issues weren’t identified and problems weren’t known, but for whatever reason, and I don’t know the reason, they never got resolved in any meaningful way, thus making the thing at the end of 5 years much better than it was in the first year. And so it was kind of a weird thing, and I think there’s some explanations for some of that, like the gap between the SGEMS (spelled phonetically) and COMET and who owns the data and all that kind of stuff early on, but it never really seemed to grow. And in a place where with contracts in, particularly where you should have interactive, substantive discussion and identification of issues and resolution, it never got done it seems, at least that’s my opinion. And if it got better, it didn’t get fixed to be better and to deliver on what it was supposed to deliver on. I mean, so it leaves one going: “Wow, what do we have with COMET now?” We have lots of stuff, but the stuff isn’t manageable in any way that we can do anything with it. So it’s not like saying there’s nothing there…

It’s just there.

Do you mind if I jump in and make a couple of comments?

Sure.

I feel like maybe I shouldn’t say anything because I’m so new to the system. But as a newbie coming in and seeing what you’re trying to achieve with COMET and what you’re trying to do with the national evaluation, and I particularly enjoyed your comments, [participant], because you actually captured exactly where my thinking has taken me. One of the things that it has felt to me like they were trying to achieve when COMET was first established was it could do everything, and it could be a management decision tool and a way for grantees to look at their own progress, a way for project officers to look at what they’re doing and how it has improved things, a way to capture outcomes. And to me it feels like maybe a better strategy this time around when you’re trying to fix it is to prioritize. What are the most important things we need it to do? And let’s make sure we get at least the first one or two right, because probably it’s not going to be able to do everything, and maybe there’s a better way to do some of those other things. So one of the things I think it would be very useful for this group to do is to prioritize what we need the system to do, and let’s do the most important ones first, and I think that’s where [participant] was going. And another thing I think is really important—please forgive me for saying this but I feel like the Drug Free Communities program has been shooting themselves in the foot because you have wonderful teams doing wonderful things with people in the communities, and there hasn’t yet been to my knowledge an easy way of capturing how much success they’re having and how important they are in the communities where they’re working, and how much they’ve been changing substance abuse and drug abuse in their areas. And it would be great in my opinion if this system could somehow help to capture that information, because it would be useful to ONDCP and SAMHSA to be able to say: “This is a great program. These community coalitions are doing great work and are having great success, and here’s some evidence of that.” I think it would be really great if we could build that into this system, a way of getting some of those success stories from the coalitions.

I guess my question would be—when we’re talking about the system I’m not quite sure I’m convinced that we’re talking about the COMET system, fixing it, or replacing the COMET system. Because let me ask you, because you’ve made some recommendations. So what are we talking about?

I don’t really have that information but I think Question 6 implies the answer. We’re asking—if we had the most ideal system possible, what would be its features? So the fact that that’s one of our…

Which one is Question 6?

I’m sorry. It’s your number 5. The final and probably most important question around the table today is Question 5, which we’ll get to soon. And so we are wanting to envision the most ideal system possible, and so I think that implies that on the table is fixing up what is, if that’s appropriate for replacing with a new ideal system.

So it’s a combination of using multiple sources. I think someone has indicated that perhaps part of our quandary was that we were using SGEMS (spelled phonetically) and COMET; now there’s another data gathering application that’s in play, or could be in play, and that’s the Workstation that has come to pass. So how do you effectively use all of the resources that you have available to get what you want? [Participant] made some very good points—one part of our history was that the priority when COMET first came in play was that there may have been a higher priority in certain areas and less of a priority in others, whether it was a program or whether it was being able to provide data back to the grantees for the grantees to do their own analysis, whether it was to provide the information that was necessary for the (unclear) reporting. And so we came at it in many different directions, without having an opportunity to sit down and establish the priorities as you’ve identified. And so we have an opportunity at this junction to do that with all three of the end-users, having an opportunity to identify what are the one or two areas that are most important to them, and so this creates a good opportunity for us. But we’ve attempted to make adjustments in the past, that was not a commitment perhaps in terms of resource, there was not a willingness on the part to take a look at all of the applications that were available and make some good decisions about exactly what is it that you want COMET to do. Do you want it to do everything or do you just want it to do 20 percent or 30 percent of the data that’s needed, and then utilize the other systems that we utilize more effectively to make the entire package work?

And we’re here to ask you that question and get your take on those questions. So [participant]’s question about repair or replace, that’s what we want to hear from you about, and priorities. What should be the priorities? What should be the scope? Those are all on the table and you’re the source today of that information. That’s what we would like to get from this session. So anything and everything is possible here in this time. But I have no information about the federal budget or anything programmatically, but we’re here to brainstorm and to get your information that you have and your suggestions, and they’ve been really good so far. Perhaps we’re ready to move on from general assessment question and go to a second question, which we’ve touched on to some extent. What information do you need—does SAMHSA need—from grantees and how can COMET better support SAMHSA’s information needs? Comments on that question?

Don’t be bashful, guys.

It’s a big question. I think at a minimum, what first struck me is at a minimum it’s just compliance information. But then on second thought, I don’t even know because you brought up a good point—there are other ways for getting information. One of the basic things are the 12 sectors and the basic compliance issue. Well, we ask for almost like a notary certified form, whatever—I’m being sarcastic—but where they have to sign off every year to affirm that they have these 12 sectors plus other compliance issues. I’m not sure what else is on there. So I think one thing we need to look at, we duplicate a lot of things. We put them through hoops multiple times in one year. And I’m thinking that maybe we need to look at, Do we get the information elsewhere, does it all have to be loaded into COMET or the progress system? Is that the place for it or do we get that elsewhere? So I’m not even positive that my first thought is correct, that at a minimum it just needs to collect compliance information, because we get some of that elsewhere.

Let me ask [participant]. When you talk about program compliance information, are we getting compliance information now?

Well, they enter their membership, but usually what we’re doing when we look at the membership is we’re looking to see if 12 sectors—we have a guide and so that’s what we’re doing, but we do that on site visits, we do that…

In the eligibility checklist…

…checklist and then we’re doing it in the progress reports, so we do that at least three times. What other requirements?

I was thinking more when you said that, terms and conditions, so if you think of compliance things, I think of terms and conditions. If that’s the information that we need, I don’t know whether it goes in COMET, whether it goes in the DFC Workstation. I don’t know. We have now, as you said, three other places to put data. But what if we just create one system for everything? I don’t get it. But it just seems like we’re doing—we do repeat, we collect things four different times a year, in the continuation, in the COMET report…

Site visits …

…site visits, yeah. So when you say that, it is a loaded question. What information do we need? If we want to talk about the successes, as Dr. (last name) said, that’s one thing.

I want to add on. We just need so many different pieces of information, and to be able to house it in one place would be just an unbelievable advancement for us. For example, we’re doing a quarterly contact update. It seems like a simple thing. It’s a nightmare. Why can’t we just have one system that the grantee puts in their contact information and keeps it up to date? That is very simple. When you brought up the partnership days—we’re talking 20 years ago, not that long, 17 years ago—we had a contractor that got all the report information on a disk, they put it into the computer, and we queried it for everything. We had to go through them to get it, but I feel like we were further ahead 17 years ago than we are now. Yes, it is very, very scary. So just at a minimum to have it house a lot of information, I think it should be one stop for eligibility, have everything in there. And then we get into the more advanced things that we really want to look at in terms of outcomes and impact. But we need a system that does multiple things.

So you’re envisioning like a management information system, grants management…

Reporting tool. Being able to query for all the information needs we have. Demographics, everything.

Maybe if I can clarify what I meant by my comment about prioritizing needs, because I agree, we need a way to capture the information we need. But my question is, does it all have to be part of COMET or is it better to design a system that answers the most critical needs with one system, and maybe some of these other things by another system that’s dedicated to that purpose? It may be too difficult to build one system that does everything, but you can still get all the things you need. I’m wondering if, by trying to put everything in COMET, it’s made it so complex for coalitions to use.

No, COMET hasn’t done that. We haven’t even started to try. It doesn’t do any of what you just said [laughing while speaking]. It doesn’t do that.

Well, maybe that’s my point then. What we need is a system, maybe pieces of systems, that do different things, that do what we need, and not try to build it all into one big glob.

I’d start with the big glob.

Yes, I’m sure that we could overload it.

Because you’ve already said you get some of the same information a bunch of different ways.

We do. Some of it is just on paper.

But then we still have to put it into a spreadsheet, a form, into another system. Grants management uses IMPAC, we use SGEMS. We don’t have the same information, so [participant] could be on a continuation for 3 days, putting information in10 different places. I guess I’m for the glob [garbled] at this point, only because if people are working 10 times to put things in 1 place—and I know what you’re saying, Dr. (last name), about maybe it should only be for a few priority items. I get that part, but I just kind of feel like if the system is, as [participant] said, you guys went to go visit and it was supposed to do these other things, it still hasn’t lived up to whatever it was supposed to be, is all I’m saying. And I don’t want to overload it either, I don’t want grantees to think: “Oh my gosh, COMET now takes 2 weeks to complete.” I mean maybe it does take two weeks. Now it takes 3 weeks. I’m not asking for that. I mean they already do this twice a year, but I think it should be a place where things are standard, even with eligibility. We look to see if they have 12 sectors and then we see if they’re active. I mean, you know, it’s information there but it’s information—as a project officer I would call and say: “I see you have an inactive parent. Can you tell me a little bit about that?” We’re still going to help them with that item that’s there, but what if, as we were saying, we could put in all that contact information—and it’s almost what [participant] is doing right now with the Workstation—where we want all the sectors to be involved, we would have that information. It just kind of seems like right now we go to four different documents to complete a spreadsheet. I don’t know.

So let me just probe for these other channels of information, these other pockets and these other ways that go around COMET or don’t go through COMET. What are those? Let’s just list those.

Let me give you a list of a couple of them. We deal with what’s called an SGEM system, which is actually an HHS management information system, so that one is internal to HHS. Then our finance system uses a system called IMPAC; I don’t know a lot about that.

That’s from NIH.

Okay, and that sort of pays the bills and works the finances. Then we have COMET system, and then we have the new Workstation. So those are four right there that we currently, not all of us collectively, but parts of us or all of us in some cases are working on. So those are the big systems that I see.

And then we have our favorite, Excel.

Well, see what happens is…

Spreadsheets.

…what we end up doing is we end up trying to create spreadsheets of contacts with emails and that kind of thing which should be, like [participant] said, so simple and yet it is a nightmare, and everybody is frustrated and angry and pointing fingers, and goodness knows what all, and it’s been that way since this thing came to HHS. So I’m with [participant]. If we could just get this system to create a contact person and a name of a coalition and their telephone number and their email address, we would be 20 years ahead of where she said we were in 1993. We’d be back to where we were in 1993. That’s sort of how bad it is.

Now these systems though, the grantees don’t interact—the four systems as you started out, the grantees don’t interact with any of those?

They do with the Workstation

Workstation.

The DFC Workstation.

All right.

[Participant], I was around in 1973 and I’ve learned that the most important thing we can say is “Senator, I had nothing to do with most of this.” [laughter] When I read your question I responded principally from the notion of project officers and CSAT and what we need to do on a daily basis, and that’s what I’ve written down there in terms of what I think. But I think there’s a bigger question here that probably, as evidenced by these examples, SGEMS and all these other things. SAMHSA, if you want to look at it in those terms, and I don’t sit at that table, but SAMHSA has some other needs that probably should be considered in all of this. For example, their NDAC (spelled phonetically) and data system. To [participant]’s point, if you could tell these stories in multiple places, ONDCP being one, SAMHSA’s data system being another, you could get this story out. So I think there’s another subgroup of focus discussion that could be important, which is some of the other people in SAMHSA who do things with data. And our dilemma here is that DFC data, and just what [participant] inferred— the good stories, the progress, the absolute community based change that’s going on—can’t be reported by SAMHSA because SAMHSA has no access to the data in any meaningful way beyond project officers and COMET. So that’s just one comment. But what I’ve written down here, and I should parenthetically say, “Not only do you and I go back once upon a time, but do you remember the Decision Support System?”

Oh yes.

The Decision Support System which was supposed to be an interactive computer-based blah-blah-blah. If I’ve learned anything in the last 15 years I’ve learned that every time there’s a new opportunity, someone has a grand idea for an electronic something that’s going to do all these wonderful things, and the reality is, rarely do they come into fruition. So I think we should take some care here, it sort of goes to [participant]’s point, to be very selective about what you prioritize, what you say you’re going to do, and then deliver on something, like she said, a couple of things so that we don’t just spin more wheels. But my point’s that you have here, compliance certainly is an important thing. One of the things we don’t have that’s critical to the day-to-day understanding on the part of the project officer is narrative information, simple and brief as it can be, but narrative stuff, and I bulleted some things. What’s the general progress per the action plan? The action plan is the heart and soul of the grant so are you making – can you tell us in two paragraphs what are you doing generally? Accomplishments to date? We don’t have any way to discern hardly, and somebody said the other day (unclear), maybe each time they report, they could change the color of the font and you could figure out what’s new information. But we don’t have any way to understand the accomplishments to date on any given date of reporting. Problems, challenges and barriers encountered, and comments on what’s going to be done to address them. Simple, narrative points that, when I’m a project officer, I could sit every 6 months and read something I can grab onto and call them and say: “Hey, you say here that you’re having these three problems.” So that level of stuff would be very helpful, and again I’ve just read a few for you, I won’t go on. But that kind of concrete, simple, and I guess the thing I would say to you for your tape for later is, these things don’t have to be extensive. What they have to be is substantive and brief points that can be reported on so that in 6 months, we can see it twice, and in 3 years, we can see it six times and we can actually be able to know what we’ve read each time and show change. It’s simple, I mean it sounds like far more but it can be simple.

You mentioned the three and the six. Is the reporting—is the 6 months reporting, is that comfortable, appropriate from your perspective? We had a discussion about this earlier with the project officers that I won’t reflect on, but just to throw that out here. Is the 6-month reporting, is that about right from your perspectives?

Yes, I think it’s about right. Having an opportunity to get data at various stages of a project, these are 5- year project periods, and being able to get information at that interval versus yearly—because we don’t have options to get it more often. In the grant program, semiannual is the standard, unless the grantees are actually experiencing problems, and then you can ask them for the data more often if there are problems to monitor their efforts to address it. But semiannually is a good interval because that gives you an opportunity to get information and make some adjustments sooner than later. You don’t want to fund a grantee for a year and just find out that they’re having problems and they’re not really being able to implement the program.

If I can reflect on your question here and what information do we need. Everyone has reflection on what we need as a program, and to summarize it, the compliance information is very important; program evaluation, and the program evaluation encompasses all of that information about what are they doing, what is their focus, what activities are they doing—that implementation information that we need. We also need organizational information and we need impact information. As Linda indicated, we need to get the technical needs information, but we also need to get information that allows us to identify issues that are happening in communities. When we get asked how many programs are focusing on tobacco, well, all of them are supposed to. But then we get a question—what unique things are people doing in the community around any particular drug? We don’t have the capability of pulling that data up and doing a very simple report or doing some analysis that will allow us to structure or paint a picture of what are the issues out there that communities are focusing on. We also don’t have the ability to summarize where the grantees are having problems at, and then structure a strategy or some planning to help them. If grantees are having, if the majority of grantees are having issues in their capacity development phase of the grant, we can’t pull that information together and say: “Okay guys, we need to develop a true strategy and put some resources into addressing that because we’re seeing the majority of the grantees are experiencing that.” We have no abilities like that. So we’re a bit at a disadvantage in terms of being effective to even provide support and advice to grantees.

So the information you’re interested in you would like to be able to query the system in real time and get information directly from the system whenever you need it or want it?

Yes, that would be very helpful.

And you do not have that ability presently?

No. (several)

Grantees might type in like, what drugs are you currently working on. For some of my former grantees they type in 10 drugs, and that’s not helpful because I have to call and say: “What did your assessment data tell you? What are your…,” and I have to focus them back down. This is my dream world, I would like for us to have like prescription drugs, inhalants, meth—I mean, I would like them all to be checked off, then if we have to do a query, we could see who’s working on inhalants, for example, and then all those SP numbers in the grant and the contact information comes up, and we would be able to have a report “Five hundred are working on inhalants.” I would like that to come up, but it doesn’t. That would require—a person like myself who used to almost hold 50 grants—I’d have to go back through each of the COMET reports and see who’s doing inhalants. I mean usually our turnarounds are really quick and I’m not shying away from work, that’s not it. We’re going to get it done regardless. But it would be great to have a system that was efficient, I guess is what I’m saying—it would be nice.

Yes, but the problem with that in a system like this, and we’re already seeing it, is when you put something like that up there, the grantee thinks: “Well, I get more credit, the more I can show.” So if they’re doing anything, even if minimal where they have a partner, “Oh yes, so and so is doing that. We’ll check it off and make sure we get credit for it.” It’s not meaningful information…

Maybe the assessment batches. You know what I’m saying? It should query things, I guess is what I’m saying. I just kind of feel like we go back through and we do…

Well, I think the reason why this comes up is that we get questions. Someone needs to know: “How many of your grantees are involved in doing such and such?” And I thought that was the purpose of the ONDCP Workstation, that when something specific came up there could be a specific inquiry made to the field. But to have a crystal ball and try to anticipate what information we’re going to need to know in the next couple of years, I don’t know how we can do that.

Those are my dreams. You said tell us what are our dreams.

Yes, absolutely, that’s…

You said we could disagree.

(overlapping laughter)

Practical suggestions we’re very interested in capturing.

Standardizing…

…and standardizing, having drop-down menus. The result of that has been, I think [participant] was very good in stating what we don’t get because of that.

I think we have verged into Question 3. How could information from COMET be more useful to support DFC team leaders and managers? And to some extent, I mean we have also verged into Question 4 which is fine. Which reports from COMET do you need in order to optimally perform your duties? So before with our last 30 minutes we just open up the final question, which is the ideal system, are there other comments from an agency perspective how COMET is or is not fulfilling those information needs, and how a data system could better support the duties and tasks and roles that SAMHSA has to play in this program?

Well, I want to delve into a little bit off territory. A couple of years ago, we started to do a site visit letter process improvement, and we looked at some other systems that were currently being used in SAMHSA, one in particular at CMHS, and they had a management information system for a division there where you could house a number of products in there. So you could put in—you could go out and do a site visit and you could put in all of your information, so you could pull up by project officer and see where they’ve been, where they’re planning to go, you could have site visit letters in there, you could actually do the letters right in the system. It was kind of a one-stop shopping thing. So it really got me thinking that I would really like to have COMET be that one place to have a repository of a lot of different documents that we would have handy, because as the government goes greener and greener, we’re getting more and more away from the paper files. But we’ve got to have somewhere, especially when you’re going to transition and you’re going to have different project officers coming behind, there’s got to be somewhere where you’re going to keep and archive things. So I would like to see COMET do that, and I’d like to see us be able to look at a snapshot of where is the project officer with their portfolio in site visits, in reviewing the progress reports, in a number of different categories, and then also aggregated by teams and branches so that you’ve got a good picture of what’s going on and where people are in the progress of things, whatever the task might be. So it would be going around the grant cycle, in my opinion, starting off the first of the year with the orientation. When we orient the grantees there’s a number of different things we go through, information we provide. That would be the perfect place a management information system could house that they received this and they’ve gone through all these different things. I’m thinking broadly.

Just at a far more simple level, one of the things that I think would help project officers a lot would be access to the CCT information, if in fact it’s going to continue to be used. That notion of that developmental typology had in terms of technical assistance and training and monitoring the structure, you know, the things it touches on has been not available to project officers. And as I understood it some time ago, the notion was well, they don’t want to dirty up the pre/post kind of national cross-site in terms of having project officers have that information. That’s what I heard. So I think the availability of the CCT or whatever, if anything, continues to be used as an analytical tool for a project officer would be a good thing, and to have a project officer be able to click into that, look at where the snapshot is at whatever points in time have been determined to be important, and be able to see some kind of—look at it with the notion of seeing some here and now, here’s where it is, and then over time see, here’s where it was and here’s what’s changed. Because technical assistance and training as part of the grant project officers’ mandate is a big deal in terms of assisting day-to-day these grantees, unless the system is going to change to put the onus of that on someone else beyond the project officer. And I don’t know what the intention is with the Workstation, but it seems like that’s a possibility that there’s a vision somewhere that others would be managing that growth as compared to a project officer. If that’s the case, then I think a couple of things—I think then what I just said may or may not have any relevance because the project officer is taken out of that primary role, number one. And number two, if that’s the intention, I think that just adds another layer of possible confounding of and confusion of who’s responsible for the ultimate orientation of these grants. Is it the government or is it whoever the Workstation is? And I think that that’s a question for the future but I think those things are tied together in terms of what you’re doing, or should be tied together at least thoughtfully, so they work together as opposed to just becoming again—talk about silos—become another series or kinds of silos that are a little bit different from the traditional governmental silo that people are happy to talk about.

I’m glad you brought up the classification tool. Are there other opinions in the room or experience with, or how much contact have you had with the coalition classification tool?

My grantee told me about it when I was at the site visit. I saw some of her; she requested it from someone. We never see it over here so that was how I learned about it, and she asked me: “What do I do with this now,” and I said—of course I came up with an answer after I studied it for a minute—but it would have been nice to have seen that before I got out there. I don’t know what that report is for. Yes, I’ve read all the reports that it’s supposed to do, but I don’t know who it’s benefiting. I really don’t know.

We’ve not had an opportunity to get the type of information, the developmental progress that the grantees are making over a 5-year period. One of the things that we do within the program is we focus on grantees in years 1 through 3, and we put a lot of our attention to those. But we are not getting direct feedback from the data that lets us know how successful we’re being, what types of strategies we need to implement to help the grantees be more successful. We do know that working with them in the first three years of the grant is a critical time so yeah, that’s a critical need for us.

Okay, let’s click the mouse here and go to our last question, which is sort of overarching and goes back to the point earlier about—are we talking about fixing up or replacing, we’re talking about the possibility of both. And I’m sure that many or most or all of you have contact with other data systems, you’ve seen them, used them. Perhaps there are features of other data systems that you would love to see implemented in the data system for the DFC. Or maybe your thinking is more to economize and to simply make changes in the present COMET system. However you’re coming at it, if we had the most ideal system possible, what would it do, what would it look like, what would it be like? [Participant] has already articulated this broader view and we’ve sort of discussed that back and forth. So let’s just proceed with this as far as ideas and especially, if you can, practical ideas for an ideal data system for the DFC program.

One thing to started this off, and I think [participant] started us off very well, and was picked up later by some of the rest of us , I think maybe one of the ways of designing the ideal system, given the fact that we have these multiple information systems now, is maybe listing on a left-hand column the things that you want to get out of the program, and then going across the column and, in the right-hand side, saying which data system currently captures that, given the fact that we probably can’t get rid of these systems that we have, or we have a limited amount of flexibility in terms of what we can and can’t get rid of. So that would be one thing that I would do. The second thing would be priorities. Some of the priorities that I have, Jaimie’s query system definitely, but we need a system that minimally is going to be able to get that basic data of names, addresses, telephone numbers, to get that off our backs so that particular thing isn’t the nightmare it usually is. Then we do need compliance data—I don’t know who should pick that up but we do need that. We need process data, we need outcome data, and we need technical assistance data, and where grantees are having their problems so we can aggregate that. Those would be my areas, and then figuring out which system is going to collect that. Somebody said, or maybe a few of us said, we shouldn’t have this thing do everything, and I totally agree with that because I think COMET did try to do everything and it didn’t help us out much over the long term. So that would be my ideal.

I’m thinking even broader than what I said previously. When I came from Justice, we had in our program office of 45 people we had over 10,000 open grants, and we used a grants management system there which was not ideal in any way, shape or form, but at least all the applications were in that system. I mean we haven’t even touched upon the processes we go through for new awards and for continuation awards. At least we didn’t have to touch a lot of paper in the GMS system. It moved from project officer to team leader to branch chief through the different system, all the way up actually through the final signoff at the top level. And you could do a lot of things, you could do your Congressional notifications and all that kind of stuff within the system. So if I was thinking an ideal system, I would have that built-in feature as well.

I agree with [participant] but I think that’s an internal problem in our agency. Our agency is really lagging far behind in terms of technology, and we still have paper file systems so we have duplicate paper file systems. So for me that’s an internal issue. I don’t know if that’s what we want this to fix.

Well, let’s be realistic though. SAMHSA is a much smaller agency than Justice and can’t afford probably the all bells-and-whistles management system that would be the ideal. So what we have to do is be realistic with the resources that we have and where we are in making our decisions going forward, I think. I guess where I was going when I first said about prioritizing is that I think the reality is for the grants management aspect of things, we’re forced to use grants.gov, and the grants management team here at SAMHSA seems committed to continuing to use SGEMS and some of the things that have been a frustration. But I guess where I was going is, if you’re going to revise COMET, maybe this is horrible, but make it work for the drug-free communities first, for the coalitions, for the project officers who are tracking what they’re trying to achieve, to track the action plans, to help them make assessments when technical assistance is needed, to track grantee information, and kind of leave the grants management to grants management, and make sure that this system meets drug-free community needs in particular to answer the questions you have every day for managing these programs.

Dr. (last name), when you say grants management, are you talking about the fiscal people or are you talking about us?

The fiscal people.

Okay, because they use IMPAC.

Yeah, I know. I realize you’re going to have to liaise your system with them periodically, because another thing I’m saying was built into COMET was this thing of using it as a management decision tool, and I wonder if that can be done a different way. It seemed to me that COMET just had so much built into it and it was trying to do so much, that it wasn’t possible to be really successful at all of those things. So let’s try in revising this to make sure the most important things are actually in there, and that we can get them done, and we can get there. Because when you hear people talk about COMET, I mean I know [participant] had some pros and some of the others of you have pros, but for the most part people say: “It doesn’t do what we need. We collect a lot of information but you can’t really use it very much.” It would be great if we could use it better to have a good communication tool between the project officers to try and help provide technical assistance when it is needed, to use it as a way of picking up where technical assistance is needed, and to answer questions that do come up. Another idea I had was maybe when a new coalition comes in, there’s the front page of, when you open up COMET, it’s a template that tells you about that coalition, and to have it be important information. Not just that they tick everything, because really what’s in their action plan maybe, of what they’re really working on, so that you have a snapshot right off the bat of where you started with them and what this coalition is focused on. You probably already know that, but sometimes when you’re given queries and people are saying: “How many coalitions do you have that do methamphetamines?” You could use that template to then answer, “It’s 30 percent of them,” or whatever it is, “and here’s a printout of the coalitions that do that.”

That would be great.

Those are the types of things we need.

In terms of this ideal (overlapping comments)…

Go ahead, [participant].

In terms of what might be important elements, I think the 12-month action plan, being able to monitor that and any new information that goes in during the reporting dates, might be a good idea. I think that the targeted dates for completion somehow need to be able to be easily changed or somehow changed in tandem with the project officer, particularly if it’s changing an objective. But that needs to be sort of changeable to the extent that we can see it and that we can see it, so the action plan. I think what I said about the CCT as part of this as a diagnostic tool should be considered. I think the ability of team leads to look administratively across project officers’ case loads on their teams would be important, and to be able to access that like we access our own. So that would be important. I think as [participant] is suggesting, the extent to which we can finagle the demographics or target populations or environmental strategies and look at them individually by state or by project officers as team leads would be a good thing because we could actually produce something internally for ourselves. So I think those—and I’ve written some other stuff that you can refer to—and user-friendliness is important. The final thing I would say, and I hadn’t thought about it until just a few minutes ago, you know, [participant] says biannually or annually. Well historically, we have had quarterly, biannual, twice a year or annual. And I’m not suggesting quarterly, but what I thought of was, we have this continuation process which is another point in time where we get information. So whatever you do, that opportunity for information should be kind of considered in this somehow, so you can take off the table some other needs because they get reported then. So whatever you do relative to reporting should work in tandem with the continuation information in particular, because there may well be—and [participant] or somebody said this—there may well be a notion of compliance that gets satisfied at that time, and once satisfied as a compliance issue, it may or may not be another issue for 12 months. Now it could if things change relative to sectors and that kind of thing, but then that’s something we could factor in. So I think the short of all that is there’s several points in time that we’re now collecting information, and more importantly diagnostically, we’re collecting a lot of stuff we don’t get back. And we can’t manipulate anything. Nothing.

I just have one last thing. There’s some crap in there we can just take out, I mean there’s stuff that we don’t do anything with it, and I mean maybe that’s just for the cross-site evaluation, but that would be great, maybe leave that for the cross-site evaluation people. We don’t need to see some of this—it’s just there, it doesn’t tell us anything. We still refer usually back to our continuation or the action plan, because that’s where the real meat and potatoes is. And sometimes they just, because the drop-down boxes are there, you’re right, they just type some of them in. I say: “Did you guys work on this,” and they say: “You know, when we were running COMET, we were going to do this,” and I’m thinking, but it’s not in your action plan. So sometimes I think they see something and get excited maybe, but they’re not using the data that’s there. They may say: “Yes, I’m doing all of the seven strategies,” and I’m like (inaudible/faint) to be doing all of the seven strategies, but they’ll check every single one and then I’ll say, “Tell me a little bit about the strategies and the activities you’re going to use.” I still don’t get that information until I call. You know what I mean? The system isn’t telling me that. I don’t know if I said it.

Did I hear you perhaps suggesting that the system separates the evaluation function from the monitoring and management? Should those be separate or…?

I wouldn’t know that. I would ask Dr. (last name) because I don’t know enough about the evaluation piece. I know there are pieces there that we may put them on that checklist when we do the review, but I know we don’t do anything with the information, we just kind of go say “Oh, it’s there.” But then maybe Dr. (last name) is working on a project where he’s trying to aggregate data (inaudible/faint)…

As we developed this COMET system initially I think it never was totally clear, at least to most project officers, what was being done in the design and methodology of the evaluation of the DFC. So I do know that in the tension that occurred between narrative stuff and trying to get that very specific information and the drop-down boxes and that kind of thing, evaluators tended to go to the drop-down boxes because they could aggregate the data easier and that kind of thing. And most of the time what we did was, we exceeded to the evaluators, so what happened I think [participant] was this: As we exceeded more and more to drop-down boxes and one little word or one little sentence thing, it got less and less relevant to you for a lot of stuff. So I don’t think there was ever an intention of doing that kind of thing but I think the effect of it was just what [participant] said, was COMET became less and less relevant to us as evaluators wanted to aggregate data more and more and make it easier to do that. Would you agree with that, [participant]? Because you were in the middle of that.Yes, [participant], you were responsible for this.

(overlapping comments)

You were like I was. You and I were sort of at the main door of some of that.

That’s true, and [participant]’s observation that over time we—as I indicated earlier—we had projects where we were really trying to adopt the system to be more responsive to GIPRIM (spelled phonetically) and the national evaluation. Those were the priorities, and over time the information that we were able to get out of it that would be really beneficial to the program was spotty. It was spotty. The thing that we know about COMET is that COMET, over time, we’ve asked the grantees to provide quite a bit of information, and they have. There’s a lot of information there. It’s our only repository that we have available right now for data associated with implementing the SPF and identifying what grantees are actually accomplishing, what they’re focusing on. By default, it’s a sole-source instrument for a lot of information. There’s some ancillary systems that we have available, we talked about them, SGEMS, but COMET is the instrument that we’ve used primarily to identify impact and to make assessments on what the grantees are doing. Our efforts to make it better are very important. The decision to either put new tires on the car or to go out and buy a new car, it’s going to be a difficult decision for the individuals that make that decision. And contrary to what [participant] says, at no time have I ever made that decision.

Senator, he wasn’t involved.

(Overlapping comments and laughter)

I was just part of the process and we’ve all tried to make it work.

Sounds like (overlapping comments).

And we’ve done a good job to a certain extent. I am confident that working with you guys now and having us at the table, having grants management at the table to some extent—but I don’t think grants management really uses the system quite as much as we do—that we have a reasonable opportunity to make the system work for everybody involved, and that includes the grantees. The grantees have asked to provide a lot of information, and over the 5 years they’ve not had an opportunity to go in there, pull information out, do their own analysis. One of the things the system should be doing for the grantees is allowing them to make assessments so they can make changes and be better prepared to address the needs in their communities, and help the people in their communities. That’s a big shortfall we’ve had with the system, so let’s roll up our sleeves and yeah, get to work.

I want to add one more thing to my ideal, big system as it is. I’d like to see a technical assistance component enhanced, rather than just a question or two. Something where they can house all their needs, but also what they’ve received, whether it be from coaching, from different sources, and so it can kind of be tracked over time as well, so that kind of paints a picture over the course of the grant. What the needs are and how they were addressed and how they built upon that. Because that’s a big void for us.

Just a follow-up to that though [participant]. If you design that as far as the technical assistance and what they might need and what they might have gotten, that kind of thing, one thing I’ve found with many of my grantees was for some reason reluctance to put down anything around technical assistance.

They never answer that box.

And as a consequence, we never got that kind of critical data and somehow or other, when we design the systems, if we could somehow give people some level of feeling that they have some security if they ask for some help so that we actually get some. I always concluded I think they were scared to ask for (overlapping comments) because they were thinking: “Oh my gosh, if I tell the project officer that I need this, they’ll think I’m a stupid idiot.”

Well, except on two questions for technical assistance, at least in some of the Vermont grants they’re more than happy to say: “We got technical assistance from this place and this place and this place and this place.” And they’ll say “We got it from the State. We got it from here. We got it from there.” They don’t often ask because I think they’re getting their needs met, which I think is another issue around the whole notion of silos doing different things for these grantees. So then the question becomes—is there a connectedness from the institute, for example, into this tool in a more direct way that actually monitors the nature of the assistance they’re getting by virtue of another mechanism that is there but isn’t interconnected to this? Right? So that if the reporting is coming in per grant, you know, you’re getting a COMET and it gets this grant number and this grant, then the question is if you’ve got another informational system that’s say technical assistance and stuff, why can’t somebody connect those two dots so that we can actually see what they’re getting without having to burden them? Which I think is part of hopefully your notions of reducing burden, right? And making these systems kind of interconnected and not necessarily putting the onus on the grantee for everything, given the fact they’re probably already reporting to someone or connecting to someone. Right? So the system should be able to do some interfacing.

Got it. Okay, we’re almost out of time. [Participant], you’ve done this—would you send that to me as an email attachment? And can I say to all of you, if you have great thoughts or if you wish we had more time or if you have other conversations, just send to me via email and we’ll make that a part of this. I want to thank you for your participation, for your commitment. I have been extremely impressed both by this group and by the previous group, the interest, the passion, and commitment to this project is just extraordinary. The focus group is ended.

(End.)

Appendix D

Social Media Web Site Responses: Grantees

User13

I am the newly appointed Interim Executive Director of the Shelby County Drug Free Coalition, I have had no experience in entering data into this system. I just recently received my username and password for COMET, I am sorry that I have no feedback for you. Thank you.

User14

I have entered Data into Comet for the last 4 years. I think the system is for the most part user friendly and obtains valuable and relevant information.

User16

In the beginning, it was very cumbersome to enter all the information into the system. Once it was in, the reporting was much easier. I do feel that the way we are asked to enter the information is redundant. The membership section is tricky in that you cannot remove people who are no longer on the coalition.

User17

If there were a method to archive completed activities, rather than having to scroll through everything each time, that would be helpful. The archived events would need to be easily accessible for future reference from the site. Another helpful feature would be if after each entry a selection (possibly multiple selections) could be checked at one time to identify what type of activity(assessment, implementation, etc) it is, what goal it fits under,etc. (To reduce repetitive entries). Overall it is a great system! Thank you for the consideration!

User15

I am the coordinator of the DFC project, Creating Lasting Connections. I first utilized the COMET tool in Sept. 2006. I was new on board and initially was a little intimidated with the data tool. However, I have since then became very acquainted with COMET and find it user friendly and informative. I had to familiarize myself to using the new coalition strategies list and still sometimes find it difficult to decide which strategy to list my data information under, if any of them.

User15

Quoted: - I am the coordinator of the DFC project, Creating Lasting Connections. I first utilized the COMET tool in Sept. 2006. I was new on board and initially was a little intimidated with the data tool. However, I have since then became very acquainted with COMET and find it user friendly and informative. I had to familiarize myself to using the new coalition strategies list and still sometimes find it difficult to decide which strategy to list my data information under, if any of them.

User21

I have only been at this for a year this week. So I have had to use the COMET report and I think it is overwhelming. Maybe there could be a program that we use more often so that we could update and add to it constantly and not have to go through so many channels to get to where we need to go to input things. I would much rather be able to input it similar to the program I have to input data at our state level. That way I don't have to do it more than once on two different systems. I know there is data I am not getting into the system and so we don't get accurate numbers.

User22

I have been using COMET for just over a year. COMET is not easy to use because the drop down boxes take time to load after each selection. There should be a way to list coalition members in alphabetical order. It does not allow you to enter events for multiple goals unless you want to type the whole thing over and over. The way this system is set up is not user friendly, its time consuming and less information winds up getting entered. Just my opinion.

User24

I agree with several of the other comments. The first time entering all the information was in a word, tedious. However, on later occasions, once everything that been initially entered, it was much easier. Some pros that I did like was the ability to attach already put together excel documents for reporting. I don't remember anything that was too unreasonable, it was pretty user-friendly. I hope that helps?

User29

I like the system. However, I wish they would have cleard out the report after yr. 1-5. SO out 6-10 data would be seperate as our outcomes changed.

User28

There is a lot of redundancy in the system. Many things need to be entered more than once and it is not easy to cut and paste anything in. Also, if we need to enter all of the people in our Coalition into the system, it would be nice to be able to pull that information up to use as a mailing list, way to keep track of how many people from each sector participate, etc. Now, we have to enter all of that into COMET and keep a seperate mailing list, donor list, etc. Also, it is really confusing to have all the other postings from other reports on the pages. I would like to have a way to access that information if i needed it. And, when you look under the objective, it would be blank space to fill in for that reporting period. Finally, an easier link that takes you right to the reporting. I feel like I sign in two or three times to get to my report.

User31

This is the only reporting system that I have used. I think that it has been tweaked a little bit every time I get on to report a new time period, but all of those have been improvements over the last version. It would be nice to be able to delete activities from 4 years ago, but everytime I try to do that I get an error message and I cannot delete. After several years you get a lot of activities in there and it makes it tedius to go back through all of them to make sure they have been closed and don't have any remaining items to enter.

User26

I have been the Prevention Grants Manager/Project Coordinator since November 2009 so I am relatively new. When first accessing the COMET reporting system I felt very overwhelmed. I am getting more use to the system however; I feel the system is extremely redundant with respect to entering in the planning and implementation activities. I guess my question is why do you need to know what is planned? I would think that the implementation and outcomes would be the information that needs to be known. Another helpful feature might be to offer multiple selections that could be checked at one time to identify the type of activity (Implementations, Planning, etc) and what goal it would fit under. Also, being able to select only CURRENT activities (archiving older activities) would be nice so that you don’t have to scroll through 3 years of activities. In addition the capacity/membership section is restrictive in not being allowed to delete members or change their sectors. People’s lives change as do their purpose they might serve in our coalitions, for example you have a business sector who stops working and stays home with their children and then changes to a parent sector in the capacity. Being able to sort alphabetically would be helpful as well. Finally it would be helpful to have a template to gather data from your coalitions. I coordinate a regional coalition which includes the 10 county coalitions which are utilizing the DFC funds. So quarterly I am reviewing 10 separate narrative reports to gather the data for the DFC COMET. I appreciate you asking for our input and look forward to great changes to make the system more efficient and useful.

User30

I am the Project Director for the Bay Shore/Brightwaters COMPASS Coalition. We are in our 10th year. The COMET is long and tedius. It would be great to just have each objective with space for inputing its current status. I think the Coalition members should just be those actively involved.

User34

I think the system is cumbersome, and parts are redundant. You cannot change items put in years ago, although there may be a better understanding, new systems/strategies, etc. It doesn’t need to have so many layers. Simpler communication can be better.

User36

Quoted: - If there were a method to archive completed activities, rather than having to scroll through everything each time, that would be helpful. The archived events would need to be easily accessible for future reference from the site. Another helpful feature would be if after each entry a selection (possibly multiple selections) could be checked at one time to identify what type of activity(assessment, implementation, etc) it is, what goal it fits under,etc. (To reduce repetitive entries). Overall it is a great system! Thank you for the consideration!
I agree that being able to archive completed strategies would help.

User35

I have only had the opportunity to utilize this system once, since we are Year 1 awardees. However, initial thoughts were that it was very straight forward and laid out nicely compared to other information systems that I've needed to enter data on. The only constructive feedback, based on the limited exposure and familiarity would be to minimize the redundancies. It seemed like we repeated our narrative sections several times throughout the reporting. Thank you for the opportunity to provide comments. Perhaps in the future, and after a more recent entering of data, another feedback process could be generated to allow for folks to be mindful of ideas to share then. Thank you.

User36

Overall, COMET is a much friendlier reporting system than other ones used in our state. However, information to be entered is sometimes redundant, as many people have said. For instance, we enter activities related to capacity building and then the next screen is to enter accomplishments related to capacity building. Can we just enter activities and specify whether they are an accomplishment or a challenge or a process? Additionally, it would be very helpful to be able to archive old goals or projects. We revised some of our objectives but are not able to reflect this in the system. It just appears that we have twice as many objectives. The area I find least helpful is the membership section. I would like to be able to remove members who are no longer with the group at all, instead of just letting the list of inactive organizations get unmanageable. Also I would like to be able to specify the representatives in each agency. I put 5 representatives from the police department, but next time I log in, I don't know who those five people were. Overall the system is not a bad one, and could be improved with some minor adjustments to eliminate duplication and allow the archiving of completed or eliminated data.

User37

I have had trouble with username and passwords which have had to be reset numerous times. I am a little discouraged by the fact that you cannot take old directors or past coalition coordinators information out and update. Otherwise, I find the system user friendly. I would like to see a better format for entering into statistical data. Especially YRBS and PNA data for our youth.

User38

We have been a DFC Recipient since 2003 and have used COMET since it's inception. It is overwhelming to input information and extremely redundant in many locations. I feel like I input the same info over and over. I would also like to smoothly move from function to function without having to go back to the top of each page, scroll down or select from the pull down boxes that often are not appropriate answers. It would be so much easier to use text boxes to put in new info. I have never found this tool to be very helpful and honestly, I get so tired and brain dead during reporting that often I just quit so I am almost sure you do not have an accurate or honests picture of our community coalition work, process or current progress. We are the Calloway County Alliance for Substance Abuse Prevention in Murray, KY. Thank you for allowing us the opportunity to provide feedback to make this a better, more user friendly and useful site for everyone. We would be glad to assist in any way possible so this works for all DFC community coalitions.

User39

I am an evaluator (Bridging Resources in Communities--Ward 8 Drug Free Coalition)--like the ability to print a pdf of the report sections (life saving when system crashed last Spring); easy to input info, but uite qprocess to gather what's needed beforehand and then enter in format COMET wants; below, dittoing comments of several others re problems and recommended solutions: 1. Problem: system is extremely redundant with respect to entering in the planning and implementation activities. Recommend: feature for multiple selections that could be checked at one time to identify the type of activity (Implementations, Planning, etc.) and what goal it would fit under. Recommend: Select only CURRENT activities (archiving older activities) 2. Problem: Capacity/membership section does not allow one to delete members or change their sectors. Recommend: way to change sectors, delete names, sort names alphabetically, upload information Recommend: electronic template to gather data from coaltiion members 3. Problem: Cumbersome, slow movement through each screen; should be way to print out or just upload all information into a another program then transfer to COMET, feel tethered to the screen Recommend: use other programming or databases to facilitate sppedier process 4. format for data entry in evalaution section cumbersome; not clear even with checking back and forth between manual and online explanations what is wanted for "sample size"; why not just ask for it in a way that it is generated in SPSS, EXCEL, STATA, SAS that is convenient for garntee rather than statistical staff of contractor; very tedious and I suspect many errors in what is actually entered there Problem: figuring out how to start and how to submit Recommend: user friendly step-by-step drop downs to get one started, and at end, walk through of steps to submit; rathern than waiting to hear from PO re submission, send an email to those on administrative list (PD, Evaluator or COMET Lead) that submission has been successful; include error mechecking and send message if any problems

User40

I took over our DFC grant in the grant's 3rd year (2007); I had no one to help or train me locally and I found COMET to be very difficult as I am a somewhat "Technology challenged" person. I feel DFC grantees should be trained not only in COMET during new grantee training butalso in federal financial reporting during the new grantee training. I often feel I am "guessing" what is wanted and where shpecific information should be placed/reported. I also wish old information for years 1-5 and years 6-10 where seperated as our 2nd cycle DFC has some different strategies. We also are STOP grantees and we never know how to report the DFC and STOP grants to indicated differences; luckily much or our activities are similiar so we intermingle their reporting. I too wish inactive Coalition could either be deleted or more simply be designated inactive so they show up seperately. I feel since 2007 the COMET report is more easily manuvered thru; I used to be logged off while still working on it - if I got an interruption when I'd go back to it it was closed/shut down, so there have been huge changes for the good but I still find it difficult and feel with out better training and explanation I am just "guessin" how and what to report. Since I have now been around for almost 4 years I see the huge transition in grantees and that not everyone is a computer wiz kid so I suspect there are more "technologically chanllenged" grantees like me who need more help and more in dept training. Having a COMET training session during the various CADCA conferences (Forum, Wash., DC and Mid-Year) would be excellent opportunities if not during "New Grantee" training. That way if there are personnel changes during the 5 years of the DFC new personell can be offered continually chances for COMET trained. I find TECH assistance is very good possibly additional personnel can be scheduled toward the last 2-3 weeks before the reports are due or possibly different due dates can be given geographically so there is only 1/4 or 1/5 of the "crunch" at the same time so COMET Technical assistance is available more timely. Thanks for the support you do offer.

User42

I feel that a lot of the information that we enter is not organized in an efficient manner. Last year was my first time using COMET but we were in the 7th year of the grant. Due to staff changes, many people had entered information in COMET before me and it is still very hard to sort through the initiatives they wrote about. I think it would be easier if we entered all the information about each project together. That way, we would be able to see everything about a project in one place instead of five. Also, if we had a way to sort out the old information, that would be helpful (like deleting coalition members who are not active and haven't been for 5 years).

User41

Quoted: - There is a lot of redundancy in the system. Many things need to be entered more than once and it is not easy to cut and paste anything in. Also, if we need to enter all of the people in our Coalition into the system, it would be nice to be able to pull that information up to use as a mailing list, way to keep track of how many people from each sector participate, etc. Now, we have to enter all of that into COMET and keep a seperate mailing list, donor list, etc. Also, it is really confusing to have all the other postings from other reports on the pages. I would like to have a way to access that information if i needed it. And, when you look under the objective, it would be blank space to fill in for that reporting period. Finally, an easier link that takes you right to the reporting. I feel like I sign in two or three times to get to my report.
My input regarding COMET seems to be closely aligned to the majority of points made in this entry. I appreciate the opportunity to provide feedback, and while COMET is clearly a valuable data collection/reporting tool, as in most such formats, areas for improvement do exist.

User32

COMET is overall an user friendly system; however, I find frustrating the fact that we can't delete coalition members that are not working with us anymore, so when I go to the list of members I have a long list of names that are not relevant to the projects anymore. Also, I find it confusing sometimes when the system list activities and projects from years ago that are not longer active.

User43

We are a Coalition in our 7th year. I had not used the COMET system until this past reporting period. Here are my issues: 1- Information entered can often times be tedious and redundant. If we didn't have to report in separate areas for the SPF components and could choose a category for entered information, that would be beneficial. 2-After information has been entered, it is not organized systematically or logically, therefore requiring me to write down what I have entered in order to more easily keep track of what I have entered, being sure to cover all bases. 3-When entering an activity, for example, it becomes frustrating to have to abandon the entry to return to say the capacity section and enter the coalition member name who was active in the activity. 4-It becomes frustrating when previously entered information is not archived, causing one to have to scroll through the oodles of information to get to the place they need to be. 5-Having to identify the members involved for every activity is time consuming and seems as though it would be tedious for review purposes on behalf of the reviewer.

User46

I have used Comet for two years and I do not think it is a good system. My problems with it are: 1 - You can never erase old data - i.e. old members 2 - You can't make edits to certain sections so if your action plan changes, all you can do is add to Comet, you can't delete activities that you are no longer doing 3 - It doesn't appear to be useful, once you dump the information into COMET, then what? We do not receive any helpful feedback.

User47

Quoted: - In the beginning, it was very cumbersome to enter all the information into the system. Once it was in, the reporting was much easier. I do feel that the way we are asked to enter the information is redundant. The membership section is tricky in that you cannot remove people who are no longer on the coalition.
I agree that it would be helpful to be able to remove individuals and organizations who are no longer involved in the coalition from the membership section.

User46

It would be helpful if COMET could read from other programs. For example, if you are using an evaluator and entering data into that evaluator's system, it would be nice if that could be exported into COMET so you don't have to retype the same information.

User47

I would like to be able to remove individuals and organizations from the member section who are no longer involved in the coalition. This would help in adding new members who come on board. Also, our coalition has been entering our data into COMET since 2002 or 2003. The way the goals are written has changed since then and new coalition staff have organized objectives and strategies differently as the coalition has updated it's strategic plan and now has a new strategic plan. Being unable to change the goals and our objectives has created some challenges for reporting our activities. It would also be helpful to be able to print reports in different ways to use COMET to report back to the coalition. As it stands, I have to enter information into COMET and create a seperate document for the coalition.

User44

I have been entering data into COMET for the past 5 years. I agree with many of the opinions stated here. There are sections that are redundant. I'm never quite sure I've got everything in where it's supposed to be. We have two mobilizers who also enter data, but there's no way for me to pull a report for just the current period, so I can check for accuracy and completeness. I do appreciate that this reporting system is more interested in the process and accomplishing goals, and not how many brochures were handed out, or how many ads were run. Thankfully, COMET is not about bean-counting! I have two wishes for this system - 1) to be able to pull data (or at least certain sections of data from the system by select-able time periods, and 2) archive old goals/objectives, so staff do not incorrectly continue to put data into old goals. Thank you.

User45

COMET has its advantages and its disadvantages. After the initial data entering (which should somehow be linked to an on line application form), there is a robust template to report on many coalition successes and challenges. The report is easy, but also tedious. The process is systematic, which, unfortunately, leads to unnecessary compartmentalization. The process of adding or removing outcomes, names, interim progress, etc., is in some cases, unclear or unavailable, and in other cases, is too time-consuming to complete. I would like to go in weekly and add updates to COMET, but because the process is so laborious, I usually wait until the week the report is due. I appreciate the work that went into it, but hope that there are folks are continuing to work toward improving it for both the grantee and the grantor. Thank you for this opportunity to share.

User49

Quoted: - I am an evaluator (Bridging Resources in Communities--Ward 8 Drug Free Coalition)--like the ability to print a pdf of the report sections (life saving when system crashed last Spring); easy to input info, but uite qprocess to gather what's needed beforehand and then enter in format COMET wants; below, dittoing comments of several others re problems and recommended solutions: 1. Problem: system is extremely redundant with respect to entering in the planning and implementation activities. Recommend: feature for multiple selections that could be checked at one time to identify the type of activity (Implementations, Planning, etc.) and what goal it would fit under. Recommend: Select only CURRENT activities (archiving older activities) 2. Problem: Capacity/membership section does not allow one to delete members or change their sectors. Recommend: way to change sectors, delete names, sort names alphabetically, upload information Recommend: electronic template to gather data from coaltiion members 3. Problem: Cumbersome, slow movement through each screen; should be way to print out or just upload all information into a another program then transfer to COMET, feel tethered to the screen Recommend: use other programming or databases to facilitate sppedier process 4. format for data entry in evalaution section cumbersome; not clear even with checking back and forth between manual and online explanations what is wanted for "sample size"; why not just ask for it in a way that it is generated in SPSS, EXCEL, STATA, SAS that is convenient for garntee rather than statistical staff of contractor; very tedious and I suspect many errors in what is actually entered there Problem: figuring out how to start and how to submit Recommend: user friendly step-by-step drop downs to get one started, and at end, walk through of steps to submit; rathern than waiting to hear from PO re submission, send an email to those on administrative list (PD, Evaluator or COMET Lead) that submission has been successful; include error mechecking and send message if any problems

My comments about the COMET system are closely aligned with those posted by User 39. We are a 7th year grantee and have seen improvements in the system, but it is a cumbersome data entry tool. I strongly agree that old information should be archived, that we should be able to delete inactive coalition members, and that we should have drop-downs for selecting which categories our activities fit into (e.g., assessment, implementation, planning, etc.)and a place to describe which goals/objectives they correlate to. Thanks for providing us with a mechanism for input.

User51

I was not with the coalition in the beginning, so I interited some information in COMET that is no longer relevant. When I go into change something or delete an initiative that the group is no longer doing, it will not allow it. Also, you can't delete people or organizations that are no longer participating if they are tied to an initiative. And in most cases those are initiatives I would also like to delete. Our group has become much more organized and pro-active in our approach. And it was very easy to include this new information into COMET and use the tool to track our progress.

User51

It would be nice to be able to delete old members or initiatives that are no longer active. The coalition has changed dramatically in the last 4 years, and to have to scroll through the data that isn't relevant is burdensome. On a positive note, COMET has been very easy to use to input our new initiatives and progress.

User50

I have only posted 3 times. Redundancies should be eliminated. Some menu options are too limited ie; strategies and activities descriptions/menu should reflect the full range (of those) being used as examples in the CADCA strategizers and at trainings or allow for individualization. As I am trying to integrate these on a systemic basis, responses would be more reflective of reality and training. I did two different COMET trainings prior to first reporting and think that more adequate examples in the training modules are necessary. The webinar that I did prior to this past report was the most understandable because it had examples. The page used to update financials is vague.

User54

I am a Project Coordinator for a Drug Free Community and I used Cmet the first time this year adn found that it was very understandable and also a good way of keeping up with progress made and helps keeps us in perspective of things that we need to do for the future.

User56

I feel that the COMET system is not user friendly. First, the information for volunteers is hard to sort to see if someone is entered, also it would be helpful to be able to have a "go to search" feature for the volunteers to see if Joe Smith is entered or Joseph Smith or Mr. Smith. It is also difficult to enter the volunters when you have to start over each new person. Other systems I have worked with would let you go from one form to another without having to start completely over each time. I have also found that entering the information is not in a user friendly format, there is a lot of redundancy, and probably a lot of missed information entered because it is hard to see where it falls in the COMET reporting. It is also difficult entering information when the volunteers must be entered and made active before you get to the event or you have to go back. For me, COMET is frustrating and inaccurate.

User58

I used the COMET system for the first time this spring and found it very frustrating, labor-intensive, and stressful; and when I finished, I was not sure that it acurately reflected what we have been doing (or that I accurately gave the info it wanted, and there was no feedback on process). The membership section did not allow you to accurately reflect status of some individuals (i.e. students who were not "active" but participated in a particular activity...they were either "active" members or I could not add them in an activity). Too much redundance, particularly in membership info. Inability to delete or archive old information that is not accurate/current. Too many layers and compartmentalization. I felt like I could not see "the big picture" when completed. Perhaps a format similar to the Action Plan that we had to submit. Menu options limited and did not always reflect what I wanted leading me to leave out info or just pick the closest. I completed the webinar training, but it showed how to fill out the form but was not always clear about what type of information was requested in a certain area. The financial part was vague and I wasn't sure if I completed it correctly.

User61

The COMET Tool can be initially intimidating. The more you use it, the easier it is to navigate. The key to not being overwhelmed is to enter the data in stages so that one is not rushing to meet the deadline date.

User60

COMET is a necessary tool for evaluation, but could be improved. Several suggestions have already been made relative to ability to edit members, drop down menus for SPF framework, etc. For the implementation section, some of the strategy drop downs are too specific and are not necessarily a direct match with a coalitions activity. I would recommend a drop down that just correlates the activity with one of the 7 core strategies. The description or activity name would provide the details. On the report end, it is good to print so you know what you put in the system, but it is cumbersome to read. The challenges/strengths narrative is what I use to really summarize highlights in review. From an operational standpoint, we use a spread sheet to track meetings, training or capacity, events, information dissemination & PR, special projects, e.g. a social marketing campaign, youth activities, evaluation and administrative functions, by date, and volunteer hours. It is only when we enter COMET that we transfer our day to day to the SPF model and ask: what is the primary function of this activity. The section on community change could be enhanced. COMET tracks activities by SPF, and outcomes by youth risk data, but does not always capture in cumulative summary coalition or community change progress.

User62

Quoted: - COMET is a necessary tool for evaluation, but could be improved. Several suggestions have already been made relative to ability to edit members, drop down menus for SPF framework, etc. For the implementation section, some of the strategy drop downs are too specific and are not necessarily a direct match with a coalitions activity. I would recommend a drop down that just correlates the activity with one of the 7 core strategies. The description or activity name would provide the details. On the report end, it is good to print so you know what you put in the system, but it is cumbersome to read. The challenges/strengths narrative is what I use to really summarize highlights in review. From an operational standpoint, we use a spread sheet to track meetings, training or capacity, events, information dissemination & PR, special projects, e.g. a social marketing campaign, youth activities, evaluation and administrative functions, by date, and volunteer hours. It is only when we enter COMET that we transfer our day to day to the SPF model and ask: what is the primary function of this activity. The section on community change could be enhanced. COMET tracks activities by SPF, and outcomes by youth risk data, but does not always capture in cumulative summary coalition or community change progress.
Our organization was asked to take over the responsiblities of the Drug Free Grant. Previously another individual was responsible for completing the report. After reviewing previous submissions, and then preparing our own submission - we felt the system would not allow us to accurately reflect what was happening with the grant TODAY. As mentioned, it was cumberson, inadequately reflective of current status of members, etc. and feedback was short - and generic. This system was not helpful.

User63

I find the COMET system to be easy to use - but I wish I had a monthly activity tracking system that correlated directly to it. In other words - I have my internal system that I use on a daily/weekly/monthly basis and then I have COMET - I feel like I am writing and reporting everything twice and it is unnecessary. Could you give us a monthly form based on the 7 strategies that could just "dump" into the COMET report when it was time? Overall, it's a good system - but I think there is a lot it's not capturing.

User64

Although COMET is easy to use, there are alot of repeations in the answers. I feel like I am typing the same answers over and over. If you could consolidate some areas it would be great. Also I like the idea of a monthly feed to put our information into. Most coalitions keep track of their strategies monthly and this would be helpful. Thank you for asking for our input.

User66

As another user has stated, much of the information being entered is repetive. It would be helpful if there were an autofill feature or a drop-down menu for such entries.

User67

I think COMET is fairly easy to use, however it is VERY repetitive. It takes me a good 2 weeks to enter all the information and activities our Coalition participates in within a reporting period. It would be nice if we could enter the activity in once and have radio buttons or a drop down box where we could select the various categories the activity falls into (Implementation, Evaluation, etc.). That would save me a lot of time. From what I have read, it appears others would like to see the same improvement! That is really my only suggestion. That and I wish we could delete past members who are no longer active. Thank you!

User68

COMET is fairly easy to use, but it is very repetitive, and it does not accurately reflect the work of our Coalition. It is difficult to accurately record activities as they never seem to fit into any of the categories correctly. Since you cannot change anything entered previously, before the strategies were changed, it is terribly difficult to categorize items. I wish I could utilize COMET records as a reflection of our work, but I truly cannot. We use another tracking system to accurately present our work.

User65

I have found the COMET format to be cumbersome, especially in the ability to delete older information. We are in Year 6 of our grant program and have also changed staff in the process, so the ability to create a natural flow and progression of current data is very difficult to do and manage under this system. Data that has been previously entered is not formated in a way that is easy to follow and make clear connections as to what year and reporting cycle it pertained to. As mentioned by other users, it would be nice for us to have a collection system/tool that would allow for the easy transference of data that corresponds to the questions in the COMET system. A system that is user friendly and easy to train for new staff members would be ideal, especially if used for a multiyear cullumilation of data for the DFC grantees. A system that allows for easier yearly updates/changes that accurately reflect the changes to objectives/strategies on a year to year basis.

User69

COMET is a necessary tool, but as others have reported it is very repetative and doesn't necessarily reflect everything we do. Sometimes it is hard to determine which area of the report something is to go into and so you put it in all locations. We should be able to put an activity, goal, etc in once and hit radio buttons in relation to what area it concerns.

User70

I have not found COMET difficult to use -- just cumbersome. I agree with other comments on this posting. It is repetitive, it does not accurately reflect the work involved in reaching outcomes (meeting with city commission members is an important part of buy in that translates (hopefully) to action that in turn leads to results), and it requires us to manually upload data. Some thoughts:

-when entering under any category in COMET, populate that information to other categories and prompt with questions like "You entered XYZ under assessment. Do you have planning (implementation, evaluation) input to enter?" The process should be intuitive rather than iterative.

-if the majority of coalitions are doing surveys with approved survey companies, find a way to allow us to electronically upload survey dara without having to re-key it.

-allow users to view their survey data entries over time -- a snapshot of changes. perhaps it could generate a "dashboard" from the data input. this might also be helpful with visually charting the progress of activities.

-create more functionality for recording changes to the coalition -- like being able to delete, move, edit or otherwise tweak the information that's already in the system. I cannot delete previous coalition members who are no longer active. Perhaps there's a good reason to keep them in the database, but I have to fish through all those names to confirm that everyone who is part of my coalition is listed. Better to be able to archive those names somehow and keep current members separate.

-allow for the reporting of coalition activities outside the four core measures. the measures are important, but they are the outcome of a set of actions that may or may not be reportable under COMET categories as they currently exist (see example above). I realize that changes in use data are important, but those changes need context -- particularly when the numbers aren't going the right way, as in the case of marijuana use right now.

-the categories where you enter the types of activities are limiting and often don't fit what you're doing.

-a nit-pick: right now, when the inputs are saved and you view them, you can't read what they are without clicking on each one to figure out whether it is current or a previous entry. as entries are saved, the system should "archive" the previous posts by listing current posts at the top of the category and/or you should be able to view the full description or a reporting timeframe to determine whether an entry is current.

User71

COMET isn't hard to use, just difficult. Simple changes to the format could not only improve the system but also the data collected. Some suggestions would be:
- List events in date order. Its very difficult to find something once its entered in the system

- When logging in, have a list of goals that are marked as ongoing so that we can check to see what has and has not been completed.

- Its often difficult to make activities, especially prevention ones, fit into the predetermined categories

-Member lists should be in alphabetical order, by the person's last name.

-Items that fit under both capacity and implementation should have a way to simply copy and paste the item or to tag it under each category. This would elimanate the need to re-type the same event in both categories.

User58

I used the COMET system for the first time this spring and found it very frustrating, labor-intensive, and stressful; and when I finished, I was not sure that it acurately reflected what we have been doing (or that I accurately gave the info it wanted, and there was no feedback on process). The membership section did not allow you to accurately reflect status of some individuals (i.e. students who were not "active" but participated in a particular activity...they were either "active" members or I could not add them in an activity). Too much redundancy, particularly in membership info. Inability to delete or archive old information that is not accurate/current. Too many layers and compartmentalization. I felt like I could not see "the big picture" when completed. Perhaps a format similar to the Action Plan that we had to submit. Menu options limited and did not always reflect what I wanted leading me to leave out info or just pick the closest. I completed the webinar training, but it showed how to fill out the form but was not always clear about what type of information was requested in a certain area. The financial part was vague and I wasn't sure if I completed it correctly.

User72

I have COMET easy to use in general. But I have a few suggestions:

In the Risk and Protective Factors section, the only options are Community and Family Risk and Protective factors. Our coalition works extensively on school risk and protective factors, yet we cannot target them according to COMET.

The logging in process could be simpler. I would suggest a direct link to COMET off the Prevention Management Reporting and Training System page....why make us guess that we should click "data submissions?" And do we really need to change passwords so often? It becomes confusing for staff and discourages us from making periodic updates.

For those of us in rural areas, the speed of our internet connections can slow the reporting process down tremendously because of how COMET processes and saves data at each step. Data entry that would take 1-2 hours with a fast connection can take 4 times as long. Is there anything that can be done to accommodate sites that do not have a high-speed connection?

Thank you for asking for input.

User75

Quoted: -

What is the other report you use to better reflect the work of your coalition?

User76

After using COMET for six years or so I find that it no longer reflects the changes we have made in our strategic plan. We cannot add items, cannot delete or archive old activities and it has become cumbersome to use. When we try to add a strategy it often will not accept it under the risk and protective factors or objectives as you have them. Not as we have them. As a result I do not feel that our bi-annual reports truly reflect all the good work we are doing.

User76

after reading some of the other comments I would like to agree with the one about being able to edit members. I tried to delete a member who has died and it would not let me. the list does not reflect what is truly happening, who is at the table etc. it is hard to keep it straight after while. i would love to be able to go in and as i said before archive objectives or activities that we are no longer doing. Every time I have to do the report I feel like I want to clean it up. also, some activities end up not being in the report due to clicking off that they were completed during that half year or somehow. I am not always sure when to close something out so many things I keep open because I don't want them to disappear in the report.

User76

Quoted: -

In addition after thinking more about this i would like to add that I would like to be able to use the report from Comet with my coalition but the way it reads it is not helpful to them. I end up doing a second report for the coalition and Key Leaders using the info i have inputed. I know you are collecting data to show that we are successful so that funding can continue. So what data is really important for you? Reports on Compliance Checks? I find no where to enter that data. Number of merchants trained? Again, no where to enter that. Number of youth involved in coalition activities? No good place to put that. I have spoken to your staff at various times about this and hope that with the feedback from all of us that the system can be retooled to better reflect our successes.

User62

User78

As a new user it would be useful to have this feedback forum sooner after I've worked on COMET and tried to hit a deadline. I'm far enough away from using it that I can't speak to the details of my frustrations, or make good recommendations.

User79

I have been using COMET for two years. From the beginning I was disappointed with the framework around how it was built and the laborious nature of submitting the reports. Below I have some bullet notes on what I would do to make it better:

COMET could/should be used as enhancing the DFC work, not adding to it. We should be able to generate reports from it, so we can use it with our strategic planning. I understand it was created to give data to the powers that be, but now that they have a way of populating data for their needs, ours should be met as well. To that end I'd like to see; graphs comparing all of our CORE OUTCOMES from previous reporting sessions, compare and contrast what was previously reported on a split screen with what we are entering in now, ability to print reporting in sections, such as "what have we done as a coalition towards implementing our evaluation piece, or lets look at what we've done around planning", etc.,
If we cannot create graphs or be able to populate reporting tools that would help us, at least make the way we report our stuff easier, such as importable AND exportable from word or excel (this is especially important for the core outcome numbers) Because I don't know about other coalitions but I also have my core outcomes in spreadsheets so I generate graphs for our community assesments and powerpoints etc.
Allow us to manipulate/alter coalition members while still retaining the info from past reports. ie. I had a member who participated for 2 years, but is now leaving. I'd like to see that she at one time existed in the coalition with the reporting years next to her name, but also "resigned" or "no longer member" with the new reporting date. Jane Smith 2008, 2009, resigned 2010 or something similar.
Autosave everything anytime anybody opens it. Clicking to save is important, but it would be nice if it autosaved every few minutes or so.
Actually come to think of it, if the report was populated in a screen that looked more like a PDF form, where we could cut and paste, etc. and you retained the back end of it like a database it would probably be much easier for most people to understand. As it currently is created, it's not very user friendly.

If you need coalition directors on your team in evaluating/creating/altering the current and or a new system, I would love to be a beta tester. I'm going to go back in to the system and play around with it some more because I know I'm forgetting things.

User81

I think the ideal system would provide feedback on a weekly basis. I know that would be time consuming by it would be extremely helpful especially when quarterly reports are due. I also think it would be easier if the password would not change so often. I understand that feature is for secruity but it would make life easier if the password could stay the same.

User81

Quoted: -

I agree having feedback sooner would be very helpful.

User83

I have been using the system for 6years and I still don't comprehend the overall layout of the system. I need to be able to see the entire system to know where to best enter certain data. I feel like I am being redundant and unclear because the system feels circular, and when I asked at a TA opportunity, I was told that I had to enter the same data multiple times. This is non-sensical. Assessment, planning, implementation all require the same data over and over to get from the assessment to the program description. The system should build on itself and take the data entered and build the case thru to the implementation.

The arbitrary categories that you want things classified as make no connection to the programs that are happening here. I cna't even remember all the exact phrasing, but we end up just picking one if it is not a clear match, because we can't proceed without clicking a box.

Finally, You have to give us a way to remove coalition members who are not active rather than just making them inactive. I have a person who has been dead for 3 years. I guess he is inactive!! I also have people who have changed jobs and are now in another coalitions' jurisdiction and we still list the person, albeit as inactive.

I would be happy to discuss this with a person, preferably as I am using the system and can better remember all the frustraions.

User84

Quoted: -

I would be happy to discuss this with a person, preferably as I am using the system and can better remember all the frustraions.

I have been using the system since the beginning, was one of the first trained on it! I agree with the comments from the person who posted above. You have to keep putting the same info in every area!! Please that is busy work!!! It is very frustrating to do the same thing over and over! The system should be able to connect the areas. It is cumbersome and I have to make sure I remember to put the same info in each area. That is ridiculous! It takes a lot of time to input your information. I spend a least a week putting in the info that does not include gathering it up and organizing it!

We are in year 5 therefore we have information in the comet system that is outdated and should be able to be deleted, but no you can't do that. The reports are so large no one wants to read them!! I am not sure what the purpose of a report is that cannot be used by the community! When I print the report the size is overwhelming!!.

I also would like to know how the comet report is used by DFC? I can't imagine how much trouble and work it is for my project officer to read and digist 50 to 100 of these comet reports!

User84

Quoted: -

I agree whole heartedly!!! What we are doing is not reflected in the report.

User85

What I appreciate about COMET:

1. It has made us think about how we count everything we do, either as a process variable/output, or an outcome. 2.It has forced us to put all of our activities into the goal/obj/strategy framework that SAMHSA, CSAP and CADCA want us all to use. 3.It has given us a national vocabulary about prevention strategies and activities.

What I do not appreciate about COMET:

1. The vocabulary and categories are so limited that they do not reflect our reality

2. I would like to see a matrix or taxonomy of how the risk and protective factors, strategies and activities all connect. The drop down menus are so limiting and counter-intuitive; I finally did a print-screen of every drop down menu to try and figure it all out! I don't mind using someone else's verbiage to describe what we are doing, but I need to see the bigger picture to figure out how to make an imposed system link up with our actual work.

3. The site itself is cumbersome. Since we are an older coalition, we have pages and pages of activity information. When I enter data, and need to move onto the next activity to enter, I am sent back to the very beginning of the section rather than going right back where I was. It is VERY frustrating to have to page down and down and down to get back to my place... (and difficult to put into words what is actually going on, functionally).

4. If the site was more user friendly, and could generate a report, I could see our agency using it to track every activity we conduct. We do an activity sheet for everything we do, so if the input screen was more like an activity sheet, with embedded data capturing fields, we could just fill out the form, print it for our records, and COMET could have the data. Of course, I would want compiled reports at the end of the quarter...I recognize the challenges here, with the vast diversity of our programs across the Nation, but there are still only a handful of R/P factors, and 7 recognized strategies that we all have to use. We could then have individualized other data collection fields on the form for our own use. That way we have national data capturing and local data capturing that reduces our workload.

User90

I have used COMET and found it frustrating. I am uncomfortable with two particular items. Since we put information in every six months I forget exactly how to do it. Can it be used regularly to put information in?

I have difficulty changing the coalition makeup. I would like to delete the coalition member from the list when they are no longer active, or have left the area. The coalition list is years long!!! And when a person was the chair of a certain initiave, that initiative has to retire to change the chair.

I would like to see an opportunity to "restart" the information in the data base so it actually reflects what we are doing. The first logic model we started with in 2002 is what is in the data base. Even though I attach different logic models as I learn more, and as our initiatives change, I am unable to retire the old.

Actual outcomes of our coalition are not shared. Just process. And I would like to learn how to include outcomes. More like the community tol box.

User73

As I review the comments made, I feel the issues I would have brought to the forum have been expressed succinctly.

User92

I find the membership data section difficult to use. I would like to be able to add and remove names, sort by name or organization, etc. but have to scroll through the whole list to see if their name is listed yet or to make changes. It's a waste of time.

We are also required to use the KIT system, which is made by the same developers - why can't the two "talk" with each other? We are doing lots of duplicate reporting and it really seems to be senseless.

User96

As a new DFC grantee, we feel that COMET is quite confusing especially the first time using it. I understand the need for an uniform reporting system that everyone needs to use. However, I would like to see a few screens where you could just add stuff in in whatever format you want. This would allow for your DFC project officer to see exactly what you are doing and then he/she could make recommendations on where the extra stuff you put in would fall in your COMET report. I found it very confusing to know what to put where even after reading the CADCA strategizers, asking other grantees for assistance and speaking with my project officer. I would like to see an example of a good COMET report so I have a baseline to know what I am hoping to achieve.

User97

I have been using COMET for two years now, although our coalition is entering its 8th year. I agree with many of the responses from older coalitions who noted that some of the information does not apply any longer and it is still contained in the database. When I first began to use the COMET system for reporting, this was extremely confusing, especially where we report coalition members. It makes it seem that we only need to report new members.

I htink that a reporting system would be most beneficial if coalitions could use the information to market themselves in the community and also to evaluate their activities. I feel that our COMET reporting is not an accurate reflection of all the work that we do in the community. The best way for us to reflect that is to use the outputs and indicators that our coalition has agreed upon, uploading that into COMET and then being able to use that to generate reports to the coalition about how we are doing. It would also be nice if the information could be used to create graphs and other data to recruit new membership or to report to the community about how the coalition is doing.

User98

We have been using COMET since its inception and found it less user friendly as we went along since we were not able to toggle between "planning and implementation". We also found that we had to list the year of the event when it was one we hold annually and had to search for it since it was no longer listed under the event name but under a generic category. We are also not able to delete members when they move on or pass away - we can only deactivate them. We are completing our 10th year as a DFC grantee so won't be using COMET in the futue but hope that it can be improved to enable coalitions to utilize the information to help build their capacity in addition to providing information for continued DFC funding.

User94

The membership section is useless in it's present state, it does not reflect the current status of the Coalition. Members have moved from the area and even died and they can't be removed.

The system does not allow for an accurrate report/status of our Coalition. The data that is entered is regurgitated several times in the SPF components.

Being an older coalition we have matured and changed, yet it is difficult to articulate those changes with the current system. It is not the most user friendly tool.

User102

There are many parts of the COMET that are not user friendly to the person completing the report. The membership section is very inadequate and does not reflect the membership of the coalition, specifically the way it is designed. The technical assitance section could also be improved by allowing the user to enter the date as to which they received assistance. The current state appears to be a request almost rather than a place to enter what has already taken place.

The system could be improved by archiving "OLD ITEMS". When it is time for you to print the report is uses way too much ink a paper. I would also like to see the system have the ability to spell check. This would be very helpful. Currently I use firefox when completing the report so that I will have this option.

I would highly recommend that any changes to the system be made in advance. There have been times when I would plan to submit my report, but have been able to do so, due to changes in the system.

User99

I am a coalition coordinator. I have used the COMET report system for over two years. I am just as confused as I was two years ago. I find myself spending a whole day trying to re-learn the reporting system each time I log on to update or submit reports. My concerns are as follows: (1) Redundancy throughout the whole system is time consuming (i.e. I need to type in one (same) activities summary in three separate locations), (2) Need to have a archive capability to sort the previous activities reported (i.e. implementation activities are like a big laundry list, confusing, difficult to refer back to, and tedious trying to find old activities).

User104

It has been a good tool for us to use. no issues at ths time.

User101

Having used COMET for the last three years, I have found it harder to use. It is not a user friendly site. I have struggled with the implementation portion. We are not able to update and add as much as we would like to. We update our progress every 6 months and it seems to change everytime we need to update or add information.

Simply logging in seems to be a challenge at times.

Something that would be useful would be to make is user friendly and not change it so much, so that one is able to become familiar with it.

User105

Of course, there is good and bad about COMET, but mostly is have been very difficult using this system as I have for about 7 plus years. I believe we used a different system when we were still with OJJDP, 3 years prior to the DFC grant, but I am not sure.

I do agree with another comment that said, "It has made us think about how we count everything we do, either as a process variable/output, or an outcome. 2.It has forced us to put all of our activities into the goal/obj/strategy framework that SAMHSA, CSAP and CADCA want us all to use. 3.It has given us a national vocabulary about prevention strategies and activities.":

The limitations of COMET are many. I find that the categories that are there as options are very limiting. You are forced to make a choice. Sometimes you are not able to view the whole phase fo a particular choice in order to make a good valid choice.

It would be helpful if the prior reports could be hidden, as it makes each report very lengthy to sort through.

I belive the site is very cumbersome as well. Thank goodness for the copy and paste feature that works in COMET. Since I also worked with an older coalition, we had pages and pages of activity information. When I entered data, and needed to move onto the next activity to enter, I was sent back to the very beginning of the section rather than going right back where I was. It is VERY frustrating to have to page down and down and down to get back to my place... (and difficult to put into words what is actually going on, functionally). Not only does all that information get in the way, it is time consuming and very difficult on the eyes. I have learned not to enter in all the coalition's activities, just general entries that would reflect dates/times noted within. So I would not enter into COMET anymore, each time I was on the radio, I would just enter, Radio Campaign and just enter dates with topics in the description area. All in all, how do the DFC Officers even evaluate our reports, how difficult and frustrating to be able to sort through all of it for each coalition, not to mention, how many other coalitions work that they have to go through.

I would very much like to see COMET be able to generate a report. I could see our agency using it to track every activity we conduct. We do an activity sheet for everything we do, so if the input screen was more like an activity sheet, with embedded data capturing fields, we could just fill out the form, print it for our records, and COMET could have the data. Of course, I would want compiled reports at the end of the quarter...I recognize the challenges here, with the vast diversity of our programs across the Nation, but there are still only a handful of R/P factors, and 7 recognized strategies of change that we all have to use. We could then have individualized other data collection fields on the form for our own use. That way we have national data capturing and local data capturing that reduces our workload. As is stands now, the hours that it takes to work on the COMET report, there doesn't seem to be much a reason for us as a coalition to benefit from the report. Usable Feedback once our data is entered, would be nice to have, right now, we have nothing to show for, other than pages and pages of information wasting our valuable tree resources on paper that I would be embarrased to show our stakeholders, community or even coalition for that matter. Just like data, it needs to be compiled so that it can be usuable.

I know this has been a long road for the COMET tool, and it has a long way to go, may God grant you all patience in your work.

User107

I have used the COMET system for reporting coalition progress for many years. I have not found it to be terribley diffiult to use, but somewhat aggravating. The ability to archive data in chronological order so that it ican be easily reviewed by the individaul inputting the data and/or coalition members would be extremely helpful. Submitting data can be a completly exasperating task. Several times I have not been able to submit the required report, eventhough I have tried days before the due date. This is very curious: Is the system not able to manage the amount of data created by the DFC sites? I do not know the answer to that question. However, with its trials and tribualtions the reporting of information in designated categories is quite beneficial. Utilizing established goals, objectives and strategies is not only an organized mechanism for data input, but is a tool that can be continually used to reinforce the SPF process with the coalition and it's members-established and new.

User108

Unfortunately, as most of the posts I have seen here, my comments tend to be more on the "things that need improvement" than on the "things I like" about COMET. I have been entering data for about 2 years and still have not been able to truly feel that I reported my coalition's current position in the best way.

1) The membership area within the Capacity section is extremely hard to maneuver. As has been stated, there is no way to delete entries or clearly classify individuals/agencies. As the list stands now, no matter how much time I spend on it, does not 100% accurately reflect our coalition. If I could suggest a possible solution: divide the section into the twelve sectors and allow us to enter the information in per sector. Allow us to have a bit more control as to delete/edit/add functions. Also, can't the new Drug Free Communities Workstation speak the same language as COMET and populate itself with the member contacts. We are all working on limited resources, not only financial, but also time, and it takes A LOT of time to provide reporting into one system, let alone several.

Also, and this may just be a personal issue with it, but on the membership section, when I click to order the list by sector or alpha order, when I go back to it after editting an entry, it goes back to the randomly organized list - again, very frustrating!

2) There is a lot of repetition and regurgitation that goes into completing each of the sections, especially within capacity building, planning and implementation since many of our initiatives/strategies may pass through each of those stages within a 6 month or 1-year period. In my perfect "reporting universe", wouldn't it be possible to enter an initiative in and then fill out the sections within that initiative that correspond to activities/challenges/barriers for each of the SPF stages? Example: Enter in the initiative for a pharmaceutical take back program. Within this initiative, have a sub-section for assessment (why was this chosen as a strategy), capacity building (who was brought to the table), planning (what are you going to do), implementation (how did you do it), evaluation. This is a way for coalitions to neatly report on their progress based on our workplans. If I were to show our workplan and our COMET report to a Board Member or interested individual, it is very hard to parallel the two and track progress. How our project officers do it is beyond me.

User112

I have a hard time manuevering around the system sometimes, it would be nice if we could go back and forth between the sections easier. I feel like I am duplicating information I input many times. And the membership section is really tough because it does not reflect our current membership since we cannot remove people.

The one positive thing is it has remained the same the last few years. Prior to COMET I felt like the forms changed everyt reporting period.

User111

I have found COMET to be user unfriendly and lacking the ability to gather the true picture of what is happening in your coalition. The 2 biggest examples of this I find are the inflexibility of the program with regard to goals and objectives, and the constant retyping of events/programs that meet multiple strategies, goals, or objectives.

We are called to constantly assess the needs of our service area and evaluate the success of what we are doing with the goal of adapting our work to meet the changing needs of the community. However, once you have entered those goals and objectives, you are stuck with them for 10 years! You can’t even change them when you rewrite for the second round of funding. It is difficult to match our activities to objectives that may have changed with ones that were written years before. Nor does this offer the information I can use to report progress or challenges.

Secondly, I believe there should be a way of entering program or event information only once. Then simply link it to all appropriate goals, objectives, strategies, etc through drop down menus. The constant rewriting is annoying and a waste of time. It also discourages giving a complete account due to time constraints for those of us who are a staff of one.

For me, positive changes to these two things would help to make COMET more user friendly.

User82

COMET is a valuable tool for documenting our coalition and project work. We have been a DFC recipient since 2005 and we are grateful for all the resources and support we have received through the DFC grant. I have been utilizing COMET since its inception. Thank you so much for requesting our input. I appreciate the work that has gone into creating the system and the focus on continually working to make the system better. The time and effort put into creating the current feedback collection website is significant, as is the investment in personnel hours to review and categorize the feedback for improvement measures. The structure of COMET sections as the parts of the Strategic Prevention Framework is very logical and helpful and enables me to track our activities in a meaningful way. In general, COMET is an effective tool, but I have found some things that I think would make it even more effective for data collection. It would be helpful if COMET. . .

only showed the current six month period’s entries during entry, exporting and/or printing the report. Once the items were complete and submitted, it could then collate that data into the comprehensive multi-year report in date order. The prior report’s membership list could be available with check boxes to select which members were involved during the current six months in addition to being able to add new members. These changes would enable grantees to more easily track and edit their current progress report data and assist federal employees in reviewing and approving the data, since the former data has already been reviewed and approved. To facilitate a comparison to prior years, there could be a link to the archived, previously submitted reports and a link for the comprehensive multi-year report that has been combined with the current report.
defaulted to date order for all entries, especially when exporting and/or printing the entire report. This would make it easier to find items that have been entered in the system. I appreciate COMET’s currently functionality in sorting data by columns. This would just be a small tweak to make the entering and reviewing process work a little more efficiently.
had a link on the homepage that would list all the “planned” and “active” items in a grid noting also the section in which each item is entered so that grantees can make sure to report on the updated status of each item.
had an “other” option to choose in the drop down menus for items that don’t fit into the predetermined categories. Users who select “other” could then type a name/description for that entry.
listed members alphabetically by the organization’s/person's last name. Additionally, it would be helpful to have the option to enter member names under organizations that have multiple members participating. Grantees would not have to utilize this function if they opted not to (i.e. the information would save without each individual’s name being entered).
enabled users to enter an item once and then select if we would like to duplicate that entry into another category (Implementation, Evaluation, etc.) via a drop down box or radio buttons. Then in the final report, it could note in parenthesis behind each item which other sections that information appears in: “(see also Capacity and Implementation)” I also like the suggestion of another user who noted that COMET could perhaps prompt us with a question like "You entered XYZ under assessment. Do you have planning (implementation, evaluation) input to enter?"
had a link to a page that took survey data entries from the same survey over time and charted the results into some kind of graph. . . or at least listed the results side by side for subsequent years in a table format. This would be helpful in quickly seeing the results and progress of our activities.
allowed users in the Basic Collaborative Activity, Assistance, Implementation Activity and Local Long Term Outcomes sections to type in the name of the activity in addition to selecting a “Type” and entering a “Description.” Once the inputs are saved, it would be helpful to add a column in the summary between “Type” and “Description” that would list the name of the activity so that users viewing the page can determine what the entries are without clicking on each one and opening a new screen to read them.
would allow users to add an organization/individual member while in another section of the report. For example, if I’m entering an implementation activity and I discover that one of the partners isn’t listed in the drop down provided by the membership list I’ve already entered, then COMET would allow me to enter the person/organization there and automatically add it to the membership section as well. In addition, if I am adding an activity in which all members participated, it would be nice to have a selection in the drop down bar that says “all members.” This would also save space in the printed report.
had a search function that allowed me to pull up all items with that word in it labeled with the section in which it appears so we could see everything about each project together and be able to locate it within the larger report if it needs updated.

Thank you for the opportunity to provide feedback. It is our hope that the data you collect in this process will enable COMET to be made into an even better tool not only for grantees, but also for federal officers. If additional input or clarification is needed, we would be glad to assist. We look forward to great changes and more efficient reporting in the future!

User113

The most frustrating aspect of COMET is the entry of strategies in implementation and linking them to goals and objectives. Most of our strategies are linked to multiple objectives requiring us to enter the same strategy multiple times. IT would be best if we had the capability to enter a strategy or activity and allow us to select the relevant goal and objectives.

Over the years we have changed, improved objectives and outcome measures and it is very difficult to change unless you retire then reenter. If that is the process so be it, just archive the retired objectives. I don't want to see them anymore and certainly don't need them on the report from here to eternity.

Also please archive closed or completed activities in all sections.

I agree with the other comments the coalition representatives/members are difficult to enter, or edit. Again archive the inactive.

User116

I have read through many of the posts already on this website. Many of the frustrations mirror what I have been told by those individuals who have been responsible for inputting information for our coalition over the past five years. I myself have not worked on Comet. However, due to recent personnel changes, I will gain this experience in the near future.

Two years ago our coalition graduated from the Coalition Institute. I was our Coalition Team Leader and attended all the sessions. During our training, we understood that the ODSS system available to all Coalition graduates would eventually be useful in completing the COMET reports. I would like to know the current status of ODSS in relationship to COMET.

Thank you for setting up this Feedback Forum. I find the reading fascinating and hope that many of the suggestions will be inplemented in the near future. Clearly, most of the writers haveinvested considerable time and effort to offer well-reasoned ideas for your consideration..

User120

COMET is not particularly difficult to use but there is much duplication of information. It is imossible to delete old entries and it would be nice if reports from previous years could be collated. All in all it is easier to use than some of the other online grant reportying systems.

User122

I reviewed the many posts and comments regarding COMET and must echo many of those same sentiments. COMET is not very user-friendly. It is difficult to manuever between sections in a way that makes the information flow. There is definitely a duplication of entering strategies and interventions. In adding to implementations already entered, it is very difficult to find them in the lists by the way that they are displayed.

User121

I agree with several of the previously made comments and feel it is extremely helpful to be able to voice our opinions. Thank you for the opportunity!

I too, have inherited old information from previous coalition staff members and find it hard to sort through. I believe it would be beneficial to have the opportunity to remove/edit inactive members/organizations. It would be very convenient if other reporting systems could "speak" to one another to avoid duplicating entering information.

Being able to print reports and cutting down on the repetitiveness of imputing the same information into duplicate sections would also both be helpful.

Thank you!

User123

I have been an COMET user since 2006, I too become frustrated while inputting Coalition activities and progresses. While I believe the system works well overall, as I read the comments posted by other users, I feel my anxiety build. I, too, share many of the concerns already posted. The redundancy of several sections, the membership area does not reflect actual activity, the inability to archive sections--just to name a few. ---Any updates made would be greatly appreciated.

User127

I have recently replaced the long time grant coordinator. I did not find COMET confusing or difficult to use. I have only used it twice. I thought the pull down menus were eay to understand. My only concern is that I did not feel COMET actually told the story of our coalition.

User125

I have been entering information since 2005. It seems that in order to provide information of how the steps of the Strategic Prevention Framework are used to guide our work the same activity is entered into most categories to show the full process. It would seem logical to enter the activity once and describe the steps that were taken for each activity on one screen. It would also create a logical report to follow for the future especially if there was a turnover in leadership. It would create a blueprint for strategies to give to other groups and have a guide to replicate successful strategies.

User128

I would like to echo the comments of my fellow DFC grantees.

The tools we are given to report out really have hindred the reporting of what the actual work we are doing in the field. Capturing this data is essential to not only justifying the grant application but also celebrating the very work we do ourselves.

Thank you for your continuing efforts in helping with that reporting system. We want you to know what we are doing too.

User23

I agree with many of the posts. My coalition started out under the Office of Juvenile Justice. I found that process much easier, more comprehensive with what exactly the work plan we were working with. I could take the deliverables, place them in a table and report on that deliverable. It was easy! There were no searching for the planning tab, implementation tab, re-entering over and over again the objectives and activities!!! I can't tell you how frustrated I am when I am working in COMET especially when November and May roll around. It is much better than some of the state of Connecticut reporting system I have worked under. I would just like the reporting to be easy and user friendly. Just allow the coalition to input the objectives and the activities once! It is confusing and hard to navigate. THanks for allowing me to finally tell you how I really feel about COMET!

User133

Comet is a suitable system for tracking coalition work - and could be more effective by:

allowing for variations to the standard drop-downs

as a place to document work-plans and progress within workplans

place to access tools to accomplish SPF steps - (actually use the tools on-line as a reporting process and usefulness for coalition work)

opportunities for on-going feedback via the systems - not 2x per year.

User130

I have not found Comet to be user friendly. Suggestions for reformatting include;

eliminate duplication of information

display entries to correlate with measureable oucomes

include a system to provide constructive feedback to the person who is entering the data ie, assess progress towards attaining goals and objectives, utlilization of the SPF, and the tieing in of the progams logic model

allowing data to be deleted or saved or stored .

shortening the length of the report

User131

The most cumbersome part of COMET is the redundancy. I know many people have named this as an issue, but it really must be stated again. Also linking the strategies and goals and objectives needs to be set up differently to give us the option to choose several different goals and objectives that the strategy may fit into. The idea of COMET is very good, it is the just the actual execution of it that causes headaches at times. I wish that there was a way to archive information instead of it showing back up on the report. But, like I said the redundancy- if I could only choose one thing to change- it would definitely be the redundancy!

User132

It is very difficult to make changes in Comet especially when there is a change in the strategic plan. We are in our 2nd cycle, year 6-10 and although I had to enter my new plan, my old one is still there. It is difficult to put in activities and make them fit with the strategies. So sometimes, I do not enter activities we have done because the system will not let me do it. And the parameters that are there are the old ones and do not fit with the new strategies. It is frustrating to try to enter an activity we have completed and the system not take it.

The list of coalition members section has some difficulties as well. If a member has left the coalition, there seems to be no way to delete them from the list.

It also seems that we repeat ourselves when we enter data. If we could just enter the activity one time it would be great, but it seems many times we enter some things more than once.

User129

I have used the COMET system for about 5 years now. COMET is a good source for reporting your coalition's activities and progress. However, I do have a few issues to address. Redundancy for some sections. I will be nice to cut down on the repetitiveness and having to input the same info. The strategies in the implementation section and the linking them to the goals and objectives needs some work. It will be easier if we could just enter a strategy or activity in and then be allowed to select a relevant goal and objective. It will also be nice to be able to delete old entries and be able to print reports or save them to our computer or flash drive.

User138

It is very difficult to change prior entries in the COMET system, even when they are incorrect. I also feel that many sections are asking for the same information.

User137

The COMET reporting system is too long and too cumbersome. Coalitions should be able to articulate how well they've met their process and outcome measures using a much simpler format. All reporting should be linked to the logic model/strategic plan, with just one place to note successes/barriers for that period. I have to agree with other folks who said that they liked the original reporting process through OJJDP better. Thanks for listening!

User136

The present system is redundant in many ways. I feel like I am reporting the same thing over and over again. I feel that the present definitions allow us to use our perceptions which means that everyone is reporting data differently. The system needs to be more concrete for all of us so that we can report data in the proper place (we all need to be reporting the same type of data in the same places). I know that I am unsure that I am submitting it correctly and I am sure others feel the same way. It would be nice if we could capture data in such a way that it can show our accomplishments. I realize this is not a simple - for everyone nationally to collect data and that it be the same data. I believe that all of us work hard and want to feel that we are accomplishing what we need to accomplish. I am not always sure that the data reflects this. Yet, I also know from being part of other national data collection that it is difficult. I do believe that DFC does a better job than other agencies/organizations in collecting data.

The redundancy of the data collection may cause those entering the data to actually forget to enter pertinent data because they are tired of the time that it takes to enter it. I hope my thoughts are clear for you to understand. I know that I realize the importance of data and want to make sure that I am entering it in a way that makes sense to me, coalition members, and to DFC. Apparently, there is a tool that some DFC grantees have utilized for evaluation/data collection. I do not remember the name of it but I have heard good things about it. Apparently, it really helps grantees to ensure they are collecting the right data and the most useful data.

User134

I think that overall COMET is simple to use, but very repetitive and time consuming. As many people have already expressed there are activities and events that may address several areas of our logic model and we used to enter them multiple times, and now we only enter them where we think their primary focus was aimed. I think this reduces the overall image of our efforts.

The COMET report is primarily used by our coalition when we are trying to determine what efforts we have tried and who was responsible for activities, it is not a document that leads us anywhere. If I had the dream program it would include a way to see what other coalitions were doing to cover the same areas of concern, or a way to connect strategies. For example, when I order books on Amazon I receive a list of other books ordered by other customers who ordered the same book. This would cause me to be more interactive with the COMET system throughout the year as the coalition looks at ways to improve our actions.

User141

Quoted: -

The list of coalition members section has some difficulties as well. If a member has left the coalition, there seems to be no way to delete them from the list.

It also seems that we repeat ourselves when we enter data. If we could just enter the activity one time it would be great, but it seems many times we enter some things more than once.

The comments from this user are exactly what I would share. We are yr. 6-10 and it is not a continuation of yr. 1-5 yet the system works like we are. We cannot add new information; we cannot actuually report on what we're doing because of the lack of accepting the information. I feel like no one will really understand the fabulous work that is happening becasue it is either too difficult to get to--or isn't there!

I very much like online reporting yet, as stated time and again--redunancy is a problem. Deleting informaiton would be very helpful when it is no longer applicable and certainly deleting members who have moved away, just no longer involved would increase accuracy of who is involved.

OUr goal, of course, is to make sure that ONDCP has the data that is required to justify this wonderful grant. I"m afraid that right now COMET probably isn't getting that to them.

So glad you are asking the questions and getting feedback....hooray!

User144

First, I think this format for gathering opinions is not the best that could be done. I would have preferred a survey format.

COMET is cumbersome and slow and provides really no way for coalitions to access the information they input in a way that would be simply and efficient (not time consuming).

Ideally, COMET would provide:

1. a way to duplicate activities so we do not have to enter the same thing manually more than once

2. a way to delete partners and other things that change over time

3. be a faster

4. not ask us to estimate the scope of reach...i think that information is rather useless given that it is just an estimate

User143

Redundancy-> As previously stated the system requires you to enter the same data in multiple places.

Rigidity-> There are some options in the current COMET system that I would like to be able to change. For instance, some of the drop down menus contain options that are really outdated for our coalition and don't fully reflect what we are trying to report. The ability to change some of the drop down menus and insert current terms and wording would be nice.

Suggestions:

I would recommend a way to track coalition progress by using a system similar to the Community Toolbox. Our coalition participated in some projects with one of their work groups and the evaluation tool was very user friendly and even allowed for the output of graphs and charts to display progress on our strategic plan.
I would also recommend clearer guidelines to what needs to be completed in each section. For the past couple of times we have put something in each section, even if it was "no change".

User146

Quoted: -

User142

Dido on the redundancy of entering information. Some of the drop down menu options are difficult to match with the actual work of the coaltion. Very time consuming.

User145

I have similar responses to many other users, COMET is redundant and not necessarily useful as a working document for us - the coalition. It is an okay reporting tool but it doesn't provide us with a usable document to use as a metric on our progress.

I like the category breakdown so it is clear as to what we are reporting.

However, as others have mentioned, we under report in certain categories and thus don't reflect all our work because we have already reported the same information in a different category. Although our activities may have a stronger impact in several categories, by the time we have to repeat the same information for the second or even third time, reporting fatigue sets in and the information reported is diluted.

I believe the report is user-friendly enough and captures all the right information on activities, outputs, outcomes, and other progress. It could be made shorter by reducing redundancy and by having a print-out tool with which we could see our own status.

User135

Thank you for creating this forum for feedback on the COMET system.

Although it is easy enough to maneuver through the system, we feel that the COMET system:

* Is repetitive - some of our strategies are linked to multiple objectives and we find ourselves entering duplicate information into multiple places throughout the reporting system, this is very time consuming and frustrating.

* Should archive previous grant years - It is very difficult to enter current information when last year's grant information is still present. Also, having all the previous years stored in the system, must affect how it functions as well. I say this because the system has kicked us out several times.

* Should allow more commenting space - Since old information is still present there is limited space for current comments, in turn the information entered doesn't reflect the amount of work that was put into a project.

* Every time we enter COMET a new password has to be selected, not twice a year, every time!

Also, in the future will it be necessary to enter our coalition members by name/agency, since we are now doing this through the DFC Workstation? Would be great if the two systems could be linked somehow or the membership could be downloaded from the DFC Workstation to COMET.

Again, Thank you for this forum, very supportive!

User150

The repetitiveness is really frustrating and really adds unneccesary time to the entire process. The drop down menus are also difficult to manage at times.

User148

Working with you action plan and reporting initiatives is too cumbersome to be useful. I understand needing to have some structure for evaluation purposes, but COMET really limits you in being able to tell your story effectively. Information shouldn't be collected just for the sake of collecting it. If a particular level of information is not needed, don't take more of the coalition's time (we don't have enough as it is) to report information that is unused or unimportant.

User149

I find the site to be unclear and very redundant to say the less. If the information that is entered is not understood by project officers and viewers, what good does the site do for the agency entering the information. The site is not practical for report submission.

User146

Quoted: -

I have to agree with this comment. It is very redundant and I feel like I lose the quality of what I'm trying to report on with each similar entry. The process is lengthy and takes too much time. However, I do like how the final report is processed and integrates our data.

User151

I wish that COMET would be more tied to the evaluation strategies that we are learning in CADCA conferences and to the 7 Environmental Prevention Strategies for Community Change.

It is tedious to got through all the years of comments that have been made. I think if a strategy is completed, it should be hidden from the list.

I do like the accomplishments and barriers section.

User139

We are a first year DFC Grantees and I have only entered information into COMET one time (May 2010). While the system was fairly self explanatory (with drop down boxes, etc.), I found the information required to be redundant and there by more time consuming. A few suggestions:

Eliminate the redundancy – Allow coalitions to input activities and all information pertaining to each specific activity at one time.

Have a feature that will allow the user to periodically save the information that has been input.

Allow variations to the standard dropdown answers. Include an “other” response and then a space that allows the person to elaborate. This might help give a more accurate reflection of the coalition’s actual activity instead of the “best response.”

Shorten the length of the report.

User153

comet is a pain. We cannot highlight our program and activities through this sort of report. It is simple and easy unlike the past reports where I had to write up a 5 page summary.

You cannot delete past coaltion members, you cannot delete anything. It is quite fustrating when you no longer have that objective since it was tied with a past years submission.

I often duplicate the same project in several areas which is counterproductive. it would be nice if you can upload your action plan and then bascailly have a check off sheet which tells the outcomes from your events..etc.

User153

Quoted: -

Thank you for creating this forum for feedback on the COMET system.

Although it is easy enough to maneuver through the system, we feel that the COMET system:

* Every time we enter COMET a new password has to be selected, not twice a year, every time!

Again, Thank you for this forum, very supportive!

I agree with all posters, but this person summed it all up for us.

User152

Thank you for this oppportunity to share my thoughts.

Sometimes the drop-down menu does not reflect what our coalition has accomplished. It might be helpful to have an option to create my own drop-down list reflective of our coalition's work.

Ditto on several comments before me...It is time consuming and tedious to write duplicate entries as I work through the reporting steps.

I like having all of my coalition accomplishments, challenges, barriers in one place.

User154

Please make the COMET system less repetitive. We are reporting on the same things in several sections. For most coalitions, several of the activities that we do tie into many areas of the reporting process.

Also, it would be nice if we could delete inactive coalition members and archive data from the past. I have to surf through many old things on a page to enter our goals and objectives for the current year.

Allow us to be more specific when we list activities and not have to use what is on the drop-down menus. Sometimes the drop-downs are very vague and also make it hard to find the activity again later to add or edit if needed.

Thank you for allowing us to have input on this!!

User154

Quoted: -

Thank you for creating this forum for feedback on the COMET system.

Although it is easy enough to maneuver through the system, we feel that the COMET system:

* Every time we enter COMET a new password has to be selected, not twice a year, every time!

Again, Thank you for this forum, very supportive!

Ditto!

User155

I agree with almost all of the comments already made. Comet is a very redundant reporting system, and having to search through all of the previous years reported information is EXTREMELY time consuming and frustrating. Since nothing can be deleted once you report on it, it would be helpful if it could be stored according to the year it was reported. That way the information could be stored and easily accessed if grantees needed to refer to it on subsequent reports.

I also agree that it would be a good idea to set up a report template based on the work plans of each of the coalitions. I assume that this would be a bit more work, but that way the progress could be reported on through the coalition activities, and be tied directly to our goals and objectives. There should also be a section for additional information, i.e. if a project is completed that is not listed in the work plan. I do agree that the barriers/accomplishments section is helpful, please continue to utilize those sections for the DFC report.

thanks for taking the time to check in with the grantees regarding reporting!

User93

Under capacity; trying to find out what you entered before is hard to locate. Foe example if you or another employee entered something and you wanted to know if it was entered it is hard to find, sometimes causing double entry. Titles do not always fit. Under training if you already received it but did not request it you have to entered it requested and then you it becomes double entry. You can not spell check so you have to cut and paste into another document to check for spelling errors.

User156

Quoted: -

OUr goal, of course, is to make sure that ONDCP has the data that is required to justify this wonderful grant. I"m afraid that right now COMET probably isn't getting that to them.

So glad you are asking the questions and getting feedback....hooray!

I feel the same as many who have commented. The COMET system is repetitive and very, very time consuming. We are going into the 10 th year of funding and the system has saved ALL of it! Many members, especially youth, have left the coalition during this time but there is no way to delete them. Many strategies have changed and there is no way to delete them either.

I spend hours completing the information because I have to spend time on each section finding the current strategies since they are all named things such as"Planning or conducting a community event involving multiple coalition members/partners" and "Special events to heighten awareness." It would be nice if you could have the grant year dates on the report.

Thanks for asking for our input!

User158

I get frustrated every time that I have to enter information/data into the comet system. As noted in other comments, it is very redunant, and I seem to enter the same information over and over again throughout the entire report. I also agree with previous comments that there is no opportunity to enter additional information which may highlight significant programs that our coalition has participated in.

The baseline data section should also enable us to put in the dates that our data is collected, and then to be able to indicate the next data collection date, i.e. we use the Montana Prevention Needs Assessement. This survey instrument is administered in even numbered years, it takes 6 months for the new data results to be analyzed and made available to us. In the meantime, trying to complete a semi-annual performance report I am always asked for updated baseline data. When I don't have any new data to enter, the report continues to indicate that the baseline data section needs to be completed. The report does not look like it is finished if I don't put new data in. There needs to be a way that we can certify that the most current data available to usis already in the system, and for us to be able to indicate the date that new data will be available.

User159

I agree with the comments about redundancy, and feel there needs to be an update in the system. I feel it is very difficult to work with, mostly due to the layering of each of the reporting categories. I think one of the problems is that "collaborative activities" under Capacity are often the same as implementing activities. And we often do the same things-meet with the same people for the same purpose to create the same project each year--but we have to input it over and over again--there is no way to add another date without recreating the entire activity or collaboration "story" each time.

The system needs to be updated for us to better tell our individual communities' stories. Although all of us use the SPF and the 7 strategies for community change, our communities have been taught different ways/ different models to define risk and protective factors and to implement these strategies. Its silly that the strategies were retired, as I use them every day. Un-retire them or update them to allow us to make our vision clearer.

User161

Although COMET takes some time to get used to, as well as readjust to for each reporting period (its limitations and repetitiveness cause confusion throughout every entry cycle), I recognize the reporting system/process could be substantially worse. I do agree with user 135. I find myself entering the same info. in several places. I also find the system's space limitations prevent me from entering important details of the work we are doing, especially when work takes a big turn in a new or different direction because of the climate, culture and politics of the community. There is a lack of space and alignment between categories to provide full status updates and tell the full story. It's also difficult to track what's already been submitted on a piece of work the coalition's done because the story is entered in several places. For example, an activity implemented can strengthen capacity and lead to new planning as well as offer new assessment info.

It might work better to structure reporting off each community's work plan. This would allow the SPF framework to be maintained but also allow people to tell more of their full story about all the things happening and work being done.

I would also like to suggest that this feedback forum be open and available during and/or directly after a COMET reporting cycle - it may provide DFC with more/deeper details and suggestions for improvement. In all honesty, it's difficult for me to provide specific feedback when the last time I entered anything into the system was 4 months ago.

User164

Reporter 135 sums it up for me. Thank you for allowing us to post our feedback.

User163

Quoted: -

I agree with the comment. It is often difficult to use the document to track my reporting on specific projects due to the repetitiveness of having it in various places. I also feel that it often does not allow a more complete picture of our program accomplishments, weaknesses, and changes to meet the changing circumstances of the political and social life of the community.

User166

I agree with all the comments everyone has made so far.

Too repetitive!!!
Need to archive previous years
Some reporting areas are too rigid
No area to address new ideas your coalition may work toward due to assessments etc.

Thank you for allowing our comments.

User167

I agree with the redundancy comments. It also seems as though it would be easier to discuss each activity and then the SPF steps under each activity instead as this would reduce the amount of time spent going back and forth between each SPF steps in order to make sure the thread/ information for each activitiy was carried through.

User165

Like others, I believe entering information into the system is quite repetitive and could be condensed. A big issue with me is when I login to enter data into the COMET system, click the implementation tab and then implementation activity. The screen that comes up is everything I have entered into the system. It would be a lot easier if only the entry points were listed (1.1, 1.2, etc.), that way I wouldn't have to scroll through pages just to get to 2.1. I understand that people need to be able to go back and make corrections to previous entries, but it seems this should be accessed another way. Additionally, a user should be able to choose more than one entry point for one activity, as some activities do more than just one service. This is where it gets really time consuming, because some of our activities qualify for 4 or 5 of the entry points.

On the COMET system, under implementation and implementation activity, 1.4 is listed as "lower age of onset for alcohol." To me that reads that we are trying to lower the age kids begin using alcohol. I understand that we want to lower the number of kids using alcohol at a young age, but I believe it should be "raise age of onset for alcohol".

User170

In the 8 years we have had the DFC grant we have had four different Coordinator's responsible to record in the COMET. Although, this should not make a difference since Coalition goals do not change throughout the changes of Coordinators, it may make since to record who the author was for that entry, so someone from the outside can see that. In my case there were two different Coordinator's input and it would have been helpful who authored, so I could go back and ask that person more details about that entry.

On another note, the way COMET is set up, it does not tell the, "Coalition Story". The things we learned at Coalition Academy and how much of an impact the Coalition has had on community change, there is not an avenue to truly show this. Everything is in a section and so a person does not get the ripple effect of this change. Having a section similar to this Feedback Forum where a Coordinator could write a summary or narrative titled the "Coalition Story", would be helpful. Our Project Officers would be able to read it and have a better understanding about our entries and what we have truly accomplished, above and beyond the four core measures.

User157

I completely agree with all the comments so far.

It would be especially useful to be able to enter all the information on an activity at one time. Many times we've found ourselves getting to the last screen and realizing that the choices in the drop down boxes don't fit, so we have to start over.

It's frustrating to spend so much time with redundency and using a cumbersome system for reporting instead of being able to spend that time working on our programs and activities.

User168

I agree with the previous posts, especially #135 and #139. The repition, layering, uncertiainty about where and how to list some things and categories that don't fit are very cumbersome. Archiving the old reports and shortening the lenght would also be very helpfu.

Some other issues:

The inability to read the full content in the drop-down boxes
Need spell check and the ability to correct the spelling of individual words without having to rewrite the whole entry

As a first time user this year, a sample COMET report to review before our first submisssion would have been very helpful and may have cleared up some of the intitial questions and confusion about what to report, where it should go and how it should be listed.

User163

Quoted: -

thanks for taking the time to check in with the grantees regarding reporting!

I agree with User 155.

User60

Quoted: -

First, I think this format for gathering opinions is not the best that could be done. I would have preferred a survey format.

COMET is cumbersome and slow and provides really no way for coalitions to access the information they input in a way that would be simply and efficient (not time consuming).

Ideally, COMET would provide:

1. a way to duplicate activities so we do not have to enter the same thing manually more than once

2. a way to delete partners and other things that change over time

3. be a faster

4. not ask us to estimate the scope of reach...i think that information is rather useless given that it is just an estimate

I would start by asking what is the main informtion needed at the federal level to justify success of DFC coalitions, and also, who and how is necessary evaluative information being used? For example, is it to ensure progress of coalition work against funding objectives and used an oversight tool by program managers? Is it to measure both long-term behaviorial oucomes as well as community change? Is it to use the data as a learning tool in which the "most common" core strategies are summarized and compared across coalitions nationally so there is a databank of effective/innovative strategies? e.g. able to see at a glance which coalitions are working on social media, policy related to advertising or taxes, etc. Is it to be used at all on a local level as a real time reporting system? (My guess, although coalition may want this, it may not be a realistic goal.)

It is hard to give accurate feedback without understanding a few key objectives of the primary intent(s) of the tool. Primary, secondary, tertiatary purpose. Also, one tool may not do it all. Clear objectives would reflect a better, targeted system--possibly lead to more useable sub-sets of ways to evaluate and collect data or separate sections. For example the CADCA coalition survey is a primary tool to evaluate coalition/organizational development against the SPF model, but we never receive feedback or summary on our data input year to year, and this is an important evaluation tool as well--not connected to COMET, and not as useful, when reports or summaries of progress year to year are not available.

One direct suggestion, somtimes an activity has more than one function. An annual planning meeting for example, has planning a primary function, but when done right inclused elements of evaluation of progress and assessment o new data and directions, as well as elements of organizational development and community mobilization, depending on how it is done. So....a drop down menu asking for PRIMARY purpose of activity, and then corresponding description would only be one one time. Then a check all that apply option would avoid some of the reduncancy. In other reverse the SPF matrix- by core strategy. Each activity is part of SPF in some way.

User174

Quoted: -

I agree with all the comments everyone has made so far.

Too repetitive!!!
Need to archive previous years
Some reporting areas are too rigid
No area to address new ideas your coalition may work toward due to assessments etc.

Thank you for allowing our comments.

I agree with other comments:

Comet is too rigid

Old data needs to be archived

We need to be able to track progress

Too much redundancy

Difficult to find activities to add progress because we have to use Comet titles which often don't fit/are not accurate.

User173

Thank you for allowing us this opportunity. For years some of us have wondered about the benefits of COMET.

On a positive note, we realize there needs to be a way to report activities and are glad to have such a system.

However, COMET does not seem to be very user friendly. We are in agreement with most of the comments previously stated about the repetition and time consuming tasks as well as the inability to input correct data. Some of the fields are a "you have to choose one of these" when being able to add just a little would more correctly clarify the statements we are entering. While Coalitions perform the same type of work, each community is different and this system seems to pigeon hole everyone.

The most frustrating field seems to be implementation. It is difficult to enter in information when you cannot read the full content.

Our fear is that people get tired of entering data that they feel is going nowhere and that they won't be able to use in the future and that this will effect our ability to receive funds because all of that Coalition's story is not told. Coalitions are probably doing much more than they are entering, but they don't know how to input the data so it will be reported accurately and effectively and they see no worth to the system. We think COMET might be something people feel they "have" to do and not something they "want" to do so the information provided might not be helpful to anyone. It would be wonderful to have a reporting system that the Coalition can print out in report form to review against their goals and objectives to see their accomplishments and areas that need work, we are just not sure that COMET is helpful in that area.

An ideal data system as we see it would be one that:

1) is easy to access and navigate

2) can be updated easily by the user (for example: deleting or making members inactive)

3) one that can be customized for the reporting Coalition (be able to add activities or goals while in the field you are updating)

4) provides a way to print out just that reporting dates entries (either by year or 6 months)

Thank you for considering the suggestions submitted by all these Coalitions.

User57

In my opinion the questions were very repetitive and the information asked was overwhelming.

User25

I agree with most all of what has been said regarding the way the COMET system is currently set up. Yes it is to cumbersome, yes it has too much repetition, yes it is cluttered with old documentation which makes it even more confusing to navigate through. I find that I (along with other DFC grantees I have spoken with) become very concerned about making sure we are hitting our target areas so we add information to just be adding information..... I can't believe this makes our project managers want to read our reports throughly. The way COMET is set up now does not give an accurate reflection of the work we are doing in the community and the collaboration and capacity building that is actually going on. The COMET doesn't allow us to link environmental strategies.

It takes way to long to enter data!!!!! I'm not sure we are entering the correct information in the correct spot. If there is a way to streamline the reporting it would be greatly appreciated.

User25

There is no way we can use this COMET report to show our goals and progress to other members of our community or possible funders. Is there a way we could get a report back for this purpose?

User177

1. The information we are required to enter is too repetative. It would be nice if we did not have to re-enter information.

2. The forms seem overwhelming at times with the amount of information we have to enter in.

3. Some of the data is multiple choice. It seems that sometimes, the information I enter in does not fit well into any of the catergories I am forced to choose from.

4. It would be nice to have a spell checker available :)

User179

The system is hard to navigate due to the fact that old information stays active. It would be very helpful if once you submit a report for the period, the majority of that information moves to some saved location that is not visual when you start reporting on the next period. Maybe just things like the needs assessment, goals and objectives, core measure reporting remained.

User176

I understand the need for reporting and appreciate doing it.

The COMET makes reporting and telling our coalition story too confusing.

The drop down bars are too rigid and do not allow for flexibility and change.

I would like to be able to report the strategy once and then maybe choose all of the goals and outcomes that the strategy helped to meet, the barriers and the successes....... This may simplify the reporting and cut down on duplications. If we could put the strategy in when it is implemented and then easily go back to it to add success and outcomes that would be helpful. Strategies have multiple outcomes and are chosen or multiple purposes so it would be helpful to be able to choose more then one if there is a drop down bar.

What has already been reported and accomplished is not easy to see or go back to.

Too much repition.

Coalition members that participate would be easier to report as sectors that participated. ex. youth, law enforcement....

User182

I found the Comet trainings to be extremely helpful. I loved the idea of working along side the PRC's. The continued support is beneficial to be able to sustain the programs and move along with policy changes.

As policies change and new programs are implemented it is important to have the guidance that Comet can supply. It give us a "go to" person/group. The idea that the different Coalitions can communicate and share information is also important. Sometimes there is no sense reinventing the wheel if another coalition has already done so.

I hope the communication and support continues.

Thank you.

Debbie Virga

Commack Coalition of Caring - Coordinator

User181

I haven't had much experience with the system, as I am fairly new. I have only completed one report. I do think that the system is cumbersome. I don't think that the right questions are being asked either. Too many of them are not applicable.

One thing that I think which would be really nice would be for those who have more than one grant (we are DFC and STOP) for the system to identify between the grants. Currently they are in the same form and it can be difficult to distinguish.

Also, it would be good for the reports in COMET to have a distinguishment from the first grant period and the second grant period. A grantee should be able to basically develop a new form when they begin the second five-year period.

User184

Thank you for looking for our input on COMET. I am a person that generally likes to do reporting to show what we have been doing in our community and in turn be able to tell all the great things that we are doing! However, I don't feel that what I am reporting into COMET it is a good representation of what our Coalition has accomplished. This could be for a couple of reasons. Number one, I don't feel that I have been given enough direction on how to report. I was given the training on how to use the drop down menus, but not the why's and how's to enter the data. Because of this, it is not very user friendly and honestly , I am not sure if I put things in the correct spots and often times the choices that are given don't represent what I am trying to report. It takes too much time to figure this out and then I feel the need to do alot of repetition to report accuratly.

I personally would rather have a system that I could enter information that can tell a better story of what our Coalition is doing.. It would be great to have a reporting system that would allow easy ways to update a report and ease of knowing where things should go. Another thing that would be great is to be able to use COMET as a way to run reports to show our Coalition and Community of our process.

User27

I cannot argue with any of the posts from my DFC colleagues. You have done a thorough job in addressing the concerns I have had. There was a bit more flexibility initially (5 years ago), but that has diminished. I understand the challenge for the Feds to quantify our accumulated data as they need to show the effectiveness of our combined prevention efforts. While they need statistics, I also believe they need our stories. We know how powerful and compelling those stories can be.

The other group that you might seek feedback from is our project officers who read these- are they getting the info they need from us? Are they getting the full story about what we are doing? If we struggle in how to report accurately, maybe those who read our reports have ideas on a format that flows for them as well. Just a thought- and yes I think very highly of our project officer....

User89

I have found the COMET system to be very repetitive. Also, the drop down menus are very limiting and often don't allow us to report what the coalition is really doing. It is time consuming to have to go through all the past entries to add what has been done during the current reporting period. We have several coaltion members who have moved out of the area and there is no way to delete them from our list. A plus is that tech support has always been very helpful when we needed to call them.

User187

Thank you for seeking feedback on the COMET system. Overall, it is a decent system. However, it never seems to capture the true amount of work completed by the coalition, not does it measure the impact of the coalition.

The old data needs to be archived to make the current input more meaningful.

There is not a capacity to erase errors with out origninal obljectives and goals, hence we have ended up with a very congested and less than meaningful pull down menu. It seems to skew our data and make it less than logical.

Many thanks.

User190

The COMET site is not flexible enough to make changes to the work we are doing. If objectives and activities change within our SOW because they are complete or data collection indicates that it is not longer an issue in our community, we cannot revise our activities easily in COMET. The initially uploaded information is there and it gets difficult to maneuver through all the information. There is also no area to provide information on new emerging issues or activities that are not part of our workplan.

The evaluation section is the most straight forward section.

User191

I understand the need for reporting. I would agree with most if not all of the comments already posted.

COMET can be confusing, drop down bars do not always allow good options to choose from. It seems that things are duplicated and repeated many times.

What has already been reported and accomplished is not easy to see or go back to.

User192

I feel the same way as many other respondents. As a first time user, I has confused because our coalitions accomplishments weren't always ab;e tp be represented correctly. The categories and drop down menus were very rigid and I couldn't always figure out where things fit in. I think this may lead to work being reported in different categories in different reports.

It was extremely time consuming. Was that because it was our first time reporting or is it that way every time?

User75

I appreciate being asked for feedback. The Comet report takes an enormous amount of time and there is such repetition and frustration in getting it done. I agree with so many others - it is not clear if those reading this report fully appreciate what the coalition is accomplishing or its challenges (how can we receive meaningful feedback or generate some dialogue about the report and work we are doing?) We would appreciate help keeping the report current as we go along so we don't have so much work just before the report is due. Making the report useful in reporting out work/accomplishments to our coalition partners would also be useful. Is it possible for smaller regional groups of Coalition Coordinators to get together in person to talk about this report and brainstorm together about how to make this report useful to Washington and at the same time helpful to us in the field - it is good for us to step back and think about our work in the big picture so a COMET type report is useful if it could be less cumbersome and less repetitive. What is hard is going back into COMET and updating projects.

Even though a written summary of coalition projects, progress, frustrations, challenges may take time to read, it is often the best way for us to express where we are in our own words (it would have to be limited to 2 pages). Combining COMET with this more personal assessment might make many of us feel better about communicating the work we are doing. COMET is one report but it seems that it needs to serve many end users to be really valued by us all. How do the Feds use this report? How is it valuable to Coalitions or could be more valuable? How does it help CADCA? Others?

The report particularly Coalition members are hard to update so we have current membership information.

I am a big visual person and would appreciate seeing the COMET report with its various threads...so I could see what I put in in Year 1 and how it has been updated over the years? How assessment ties into capacity, into implementation, into evaluation etc.

Someone mentioned using COMET as a basis for dialogue with our project officer is a good idea and a way to stay in touch. This may be difficult as I am not familiar with individual workloads.

That's it for now.

User195

I agree with many of the posts. The need for reporting is useful and valuable. However that being said, here is my feedback:

The drop down menus are far too restrictive and force you to choose from items that may not accurately reflect what you are doing.

I think the information being entered should somehow be made available for coalitions to use with community stakeholders and other funders.

The areas of input seem extremely repetitive.

Spell check should be included.

Old data should be archived as it is very difficult to differentiate between old and current strategies.

This type of feedback survey should be done shortly after Comet has been submitted. Honestly, I forget many of the difficulties I have utilizing the system, but know there are more.

Good Luck.

User198

Here are some of the challenges posed by the existing COMET reporting system:

Although DFC does not prescribe strategies, activities, etc., the system is set up in such a way that there are finite responses allowable for many of the required fields (which often do not reflect the work of the coalition). There is not an option to accurately describe strategies and activities related to specific coalition objectives.
There is not an option to delete old members, objectives, strategies, etc. which makes it extremely cumbersome to attempt to track progress.
This system does not provide value in the way of utility for coalition planning, evaluation, etc.
Most of the sections require duplicate responses. Would be helpful if there was a feature to populate other sections as appropriate (those that require the same information).
The system interface could be more useful in terms of linking all sections together.
It would also be helpful if users received feedback on their data entry and how this information is being used.

Because the system is so cumbersome, it is nice that we only have to enter data twice per year. Our hope is that the new system will not be as time consuming and much easier to enter and analze data and yield results that could be shared with coalition membership and other stakeholders.

One of the features we would like to see is the ability for the new reporting system to provide a way to track actual coalition progress along the SPF steps.

User193

I am also a new user of COMET, having completed one report to date. I was not particularly intimidated by it, but did find it restrictive, repetitive and cumbersome. I also agree with respondents who noted the following:

1. The inflexibility of the system as a whole and different sections in particular;

2. The inability to correct errors without taking the whole segment out and starting over;

3. The system is very repeititve and the drop down menus do not reflect choices for what we actually do;

4. Particular inflexibility of stating/changing goals and objectives.

Overall, COMET gathers qantitative information, albeit in a cumbersome and repetitive but essentially competent manner; its inflexibility makes gathering qualitative information difficult and it is the qualitiative information that tells the story of each of our coalitions. And actually, although I noted those four areas of agreement, I agreed with the others who responded before me with most of their comments.

Finally, in spite of frustrations and concerns about reporting with COMET, tech support is amazing and patient and if they haven't got an answer right that minute, they get it and get back to me.

User197

I've read a few of the comments posted so far - would have to agree with the masses that it is VERY repetitive. It is also difficult when you have to scroll through lists and lists of implemented activities, collaborative activities, etc. to find what you're looking for.

Is there a way we could catalog the activities by year - to be able to click open a year & see what you entered that year, freeing the viewable lists for new entries?

Also, a lot of the drop down menus often do not have a long enough window to see & groups that are repetitive are difficult to determine which one you're choosing.

User199

Regarding the entering of GPRA data, the ideal would be that the data collected from our State sponsored youth survey would be communicated to the Feds without our need to retrace the data to our local area and then submit it piece-meal by each DFC or STOP community. If a State has a State-sponsored youth behavior survey, doesn't the State's data already get sent to the Feds? Are we not duplicating this in the COMET? If it doesn't get parsed to DFC communities, perhaps there could be a parsing system at the State level that would enable the State to distinguish which data sets relate to the various federal funding streams? I'm in favor of submitting this data ONCE to the State and Federal level and having it routed to the appropriate entities, or allowing those entities to search a database and pull zip code detail or another community identification number to get the information. Again, in the ideal world, once the local surveys are conducted, the data would be quickly accessible by an EASY web-based search for any community member or prevention professional. Aren't we all about sharing our local data with our community in order to focus our work and celebrate our successes? Our state's website is particularly difficult to navigate when trying to obtain state-level or county-level data. Seems to me it should be as easy as finding the federal per diem rates per community.

Regarding the progress in SPF focus areas, the repetition in COMET is a killer. Since coalition work by design is supposed to follow the process of the SPF, would it not be easier to list an initiative/project and then indicate a progress marker as the various components of the SPF are completed or in-progress? The challenge here is that the SPF is not always linear - sometimes a coalition is still doing assessment and capacity building while it is also implementing and evaluating the same project. Perhaps a simple group of 5 check boxes per project description would suffice and multiple responses are available?

The project descriptions have been helpful as we have had staff turn-over and a COMET provided drop-down title would not adequately describe a local project. I can see where the variety of narrative descriptions make summarizing data among coalitions nearly impossible, so I imagine that some uniform way to quantify progress and outcomes must be necessary to share nationwide progress.

Since the GPRA data captures the long-term outcomes of the projects, it seems that we only need capture the progress and barriers occurring prior to seeing changes with the community-level behavior. So, what are the short-term outcomes that coalitions yield? Increased community readiness to address the issue - increased community capacity and engagement (implementation/intervention) around the issue - sustainable coalition infrastructure & funding - environmental changes designed to impact community-level behavior - improved assessment & evaluation strategies. Perhaps the system could allow coalitions to set their goals in relation to contract timelines (perhaps in an uploadable Excel file or something they could also use when NOT in COMET?) that would be the same document where they record their progress per 6 month report. Then coalitions would simply identify to what percentage of attainment are they per goal per 6 month report. The ability to retire a goal or timeline would be necessary to account for the inevitable changes encountered during a 5 year project.

A narrative box could be provided to explain why a milemarker wasn't reached as proposed, or why the entire goal was reached in an unpredictedly short amount of time. This would create a summary report that reflects progress in community readiness, capacity building, engagement, sustainability, assessment and evaluation that could be communicated in conjunction with the GPRA long-term data. Successes and challenges might then be sorted and shared in SPF categories so that coalitions having difficulty in one area could benefit from the notes from other communities, paired with contact information, the system could be an opportunity to network with coalitions around the nation.

User206

I was thrilled when the COMET system came on line because I thought it would be a reflective tool for our coalition over the years. It turned out to be very difficult to navigate the COMET as a reflective tool becuae it did not condense, systematically categorize by date or category and it did not offer any sort of prompt to go bcak and change the status of a particular portion of the SPF. It also split the SPF and asked the same questions in each category. I found that I was telling the whole "story" in planning that should have been segmented into planning then implementation. I had to go back and make changes to place things in the right place. It was not efficient.

I feel that we should have been able to put the all the stages for each identified area from assessment-evaluation in one area and marked the changes from each step as we acheived them. I also think that the Comet system could have been much more interactive. It seemed archaic.

I think I got the hang of it after a while but I felt that I could not always show how wonderful a job we had done because I thought if it was hard to report in it was also hard to gather the true progress of each coalition in this hard to foloow report.

User208

As a first year grantee, the prospect of completing the report was daunting. However, as I completed the survey, I recognized the value of all the data and documentation that I had accrued throughout the year. All of the information that our mentoring agency provided to us was vital in completing the survey.

In terms of the COMET System itself, many of the other posts summarize the short-comings of the system. It is cumbersome, repetitive, and limiting in the responses from the drop-down menus. It was difficult to fit some of our activities into the categories or descriptions provided.

User210

We found COMET useful but at times redundant. In general it was a helpful way to organize the coalition's work within SPF

User209

The COMET report is an opportunity to review the progress each quarter. This is good because it assures documentation of requirements.

However, the format of the questions are often labor intensive. Many of the areas ask for data that was in another area.

Many times we fill in information that was just requested in another report - for the state or national.

I have only completed 2 reports, so I am still learning how to best complete areas.

User185

Every time I complete and submit a COMET report I feel frustrated that it does not accurately reflect our progress and accomplishments. I think the program is in desperate need of an overhaul. The categories are too restrictive and narrow, the drop down menus too limiting. It's not always clear where information should be placed within COMET. For example, if one of my strategies is to build membership in my coalition, and I have included in my plan the steps I'm going to take to do so, where do I report my progress in this area? The overall strategy is about "capacity", but I am "implementing" my plan.

The evaluators need to decide if the report is about how closely grantees are following the SPF process, how closely grantees are following their plans, how closely they are following DFC objectives, or . . . ? COMET is structured according to SPF, but we are reporting on the progress of our own strategies, which don't always fit neatly into single SPF categories.

If the purpose of the evaluation is to report on the progress of our own plans, then the strategies should be included on the evaluation with a way to clearly narrate progress on each.

Thank you for providing this forum for feedback! I think the open narrative style is fabulous!

User199

Regarding coalition membership, I think it would be much less time intensive for a coalition to simply upload a membership list. Perhaps that list is designed to identify stakeholder representation & member engagement level (supporter, implementer, leader) or whatever. We have specifically been uncomfortable entering youth names into COMET, so we've been using initials instead. It has been challenging to figure out the definition of "member". Providing a valued and appropriate spectrum of involvement is more realistic for coalition work. Our community may have a local champion who will provide impactful influence occasionally, but who doesn't get involved in the administrative maintenance of the coalition, or drug-free supporters who wish to stay informed, but don't have time to commit to the cause. At the other end of spectrum are key coalition leaders who conduct planning, evaluation and administrative duties on behalf of the coalition. In our communities, this entire spectrum of involvement is valued but we struggle to communicate these varying roles in our membership list. The term "membership" can often imply a continuous and high-engagement with an organization, whereas we might have a small number of continuous and highly engaged individuals who mobilize the other sectors of the community for specific projects in a less continuous fashion.

Regarding coalition functioning, I think a growing and improving coalition needs to be tracking its leadership development, expertise in implementing prevention interventions, and ability to sustain progress made in drug-free community goals. COMET does not currently include a self-evaluation from coalition members regarding their input on the items above. Ideally, a standard coalition member assessment (with an ability to indicate member engagement level as related to above comment) could be conducted via a short online survey (distributed to individual coalition members) which would aggregate at the local, state and federal levels so that each entity can see what technical assistance might be needed to help coalitions increase effectiveness and sustainability. Ideally, data would be tracked over time to show increased capacity and self-reported effectiveness of coalitions. While it might be scary to have local coalition member survey data available to state and federal stakeholders, data entry would only occur once, by each individual coalition member, without the need for local or state workers to re-enter the data for a DFC or STOP report. Ideally, this survey would be conducted annually, NOT in the summer, and results would be available in real-time, so that local coalition leaders could gauge participation, completion and results without waiting 6-9 months for data to trickle back down to the local level. Maybe a partnership with Zoomerang or Survey Monkey could be arranged?

User189

We have been using Comet for a long time and have encountered the same challenges that other reporters have cited:

the inability to change who is leading an activity when someone relocates and is therefore no longer part of the coalition
inability to delete 'clutter' that is no longer in use because it was tied to something else used more than 5 years prior. For example as our Coalition has progressed we have refined or replaced some of our original objectives; we can not delete the original nor can we delete risk and protective factors that were associated with them. We therefore have a lot of clutter to go through to complete new reports.
we would like to see more flexibility in reporting the percent of effort on various objectives - i.e. 15% or 25% - not simply in 10% integrals.
the options available from the pull down menus do not always seem appropriately responsive in subsequent reports.
repetitive from section to section of the report - must be difficult for SAMHSA to read - are we communicating what we intend to.

Other responders noted the inability to tell 'the rest of the story'. We have been trying to make use of the Accomplishments and Challenges/Barriers sections to attempt to tell some of the rest of the story.

User207

Compared to the state tool that our coalition is required to use, I give COMET high ratings. I appreciate the overall picture that COMET addresses regarding overall coalition, capacity building and the opportunity for specific outcome measurement. Difficulties that I have discovered with COMET is that there is a lot of repetition in some of the reporting elements. I would like to be able to completely drop coalition members that have moved. Would be interested in having the COMET archived data separate as opposed to everything being showed. COMET provides the opportunity to see the big picture but at the same time that "big picture" data collection is somewhat confusing. My biggest difficulty has been that our external evaluator has had issues in accessing the data in order to incorporate COMET information into our coalition's overall evaluation/assessment.

User205

Overall, I don't think COMET is as terrible as everyone makes it out to be. However, there is definitely room for improvement. I agree with what many others have already said, with my main concerns being:

1. The drop down menus are inflexible and do not always provide us with a choice that allows us to report our efforts accurately.

2. The system is extremely repetitive. It would be fabulous if we could enter the information or a description in only one time instead of several times.

3. It would be nice if the information entered in for a previous reporting period was accessible but not "live." I really only want to see the information I'm entering for the current reporting period. I don't want to have to scroll through pages of old information.

4. It would be helpful if we could generate reports from the information we enter into COMET. For example, if I could print a report showing the projects our coalition implemented in May 2009 or a report showing the percentage of growth in our coalition membership over the past two years.

I see that many other grantees have similar concerns. If the report could be adjusted just a little to allow us to report our efforts in a more accurate and streamlined manner, I would be very happy.

User204

Wow, so many people who are sharing my thoughts and experiences!

I've been doing the COMET reports for five years now. I remember when I first set our coalition up in it I thought to myself "wow! this is great! This could be a really useful tool for us to measure our progress, if we keep it current."

Would that this had proved to be the case! Images from "A Christmas Carol" come to mind, as Ebenezer Scrooge is confronted by Jacob Marley, who clanks the heavy chains that are coiled about him. My entries into COMET - 70 pages long at this point - form my chain. There are many accomplishments in there to celebrate, but they are so hard to find! All I can see is coils of chain.

In any event, there are more pages of comments and recommendations in here than I can read (and I'm a reader). I especially agree w/User 174, User 125, User 98, User 27, User 198, User 195.

It takes me about 4 days to complete this report, and the only way that I am able to do so is to first create a Microsoft Word document that follows COMET's structure, and figure out how I am going to put our coalition's accomplishments into the different parts. (and when I do a new report, I can easily see what it was that I reported the last time) I think my reports are reasonably good because I do it this way. I would go insane if I tried to just plug our results in "live," going through COMET screen by screen.

Most annoying is the fact that I can't get rid of the coalition members who have long since moved on, and pages of projects that were completed years ago. There is a link between the people & the projects that keeps you from deleting one or the other, I can't remember which. The system used to be more flexible (you could add your own drop down option in some of the screens)-that was better.

I doubt that comments like "more flexible" and "more user friendly" are particularly helpful. I am sure that there is project management software used in the private sector; I recommend you study 'what else is out there' as you strive to make improvements to the DFC reporting system.

Thanks for asking.

User211

I honestly think that COMET needs some major work! Not only is it VERY repetitive and time consuming, it does not allow you to change items once they have been submitted. Over the course of our grant, we have changed our projects to address environmental strategies. Once a planned project or activity is listed, you are pretty much stuck with it forever! There needs to be an option of removing an planned project from your report if that project is not completed or has been changed. The way the system is currently, it lists items that are no longer key to our progress. We had another person initially entering information in to the COMET system and much of it was put in to the wrong catagories. Now, we cannot change anything and it makes our final report VERY messy! I just think the reporting system could be much easier to use without the repitition. We should be able to enter the progress made on each of our Action items and Work Plan without all the other 'stuff'. I hope the system is revamped to make it much more user friendly in the future. As it is now, we have to spend way too much time with it when we could be out doing what we do best..............make community change!

User203

I agree with all of the feed back given. After reporting several times, I've gotten more comfortable with it, but it remains a daunting task that I have to psych myself up for every time I go back into it. It's a bit of a labyrinth that requires finding the path again each time you enter. Most of the time technical assistance help is forthcoming fairly quickly, but the problem lies more in knowing what help to ask for.

For instance, I asked several times for a list of examples of each of the activity types from the drop down box under capacity, like hearing on drug problems, combined public and private funding for...etc. No one has been able to get me such a list. Another frustration is that when clicking on the drop down box, it won't display the entire line, like "combined public and private funding for"... and that's all that shows. That's especially frustrating the first few times using COMET

Also, it's been advised to have more than one person trained for reporting in COMET. I have no administrative assistant and it is so time consuming for one person to learn and report in COMET, that only sheer necessity would provide enough incentive to train and have more than one person reporting consistently.

I've tried to "be friends" with COMET, and I have learned much using it, however, it is not user friendly and does not flow well.

User213

I read a few of the comments online and agree with those of 189 and 205. I don't think the report is necessarily hard to fill out.

User217

Quoted: -

I agree with this comment. COMET is very time consuming, not user frienldy and truly a bit difficult to make sense of any of the information. So if I am challenged to understand my own communities history through the system I can't imagine how this is being used on the other end for legislation, data tracking, etc. Tracking data is very important but when we are asked to do it for multiple funding streams on similar measures it becomes time consuming and frustrating. It would be really great to have a more systemic approach to this through counties, states and the federal system. Accountability is one thing and data tracking is another. Please make some changes....thank you!

User215

COMET was great the first time I used it, but now I have to scroll through everything we have ever done in our coalition when adding something new. Sometimes I forget if I added something already and since you can't really name your effort, I have to scroll through all of them that were done around the same date to make sure I'm not submitting something twice.

The drop down choices don't always offer a category that I feel our project fits under.

When I asked about the question "what percentage of your targeted population were you able to reach" I got two answers. One said my target population is always my entire community and the other said if you are implementing something at your high school then your high school is your target population. It is hard to answer that question because I'm getting mixed messages about it.

It is hard to explain what you are working toward. For example, we worked toward a social host ordinance that didn't get passed. So now we are doing a lot of education things and promotion things to help build support for a social host ordinance, but COMET doesn't see those things as building toward an environmental strategy it just prints the report out like we are only doing information dissemination.

It would be better if I could just see things that are for this reporting period and if I had more ways of expressing what we are doing instead of a drop down menu.

User219

It is a difficult system. It is hard to decide where information should go and what information is wanted. The information that you want to report does not always match with the choices available. Like others have stated, it is extremely time consuming, confusing to work with, redundant, and limiting. As I work with Comet it is a becoming a little easier.

User220

I have used this program in a number of different capacities: As a coalition coordinator/director, an evaluator and as a TA provider. However, I have always felt that it is frustrating, redundant and not practical. Activities often times to be added in all five sections and sometimes multiple times within those sections. The editing capability is poor. The word number caps that are enforced in each section do not cut you off as you are typing but when you to print the report they are cut off. Often times a compnent of an program ends but if there is still an active part in another section it does not let you change it to inactive or complete. Also, if a member becomes inactive but they are still attched to an active activity you can not change there status. Overall, COMET is not a tool in the work that we all do and causes more of a headache than anything. I am all for meaningful reporting and reflection but it is not either one.

User220

Quoted: -

Also, the categories and options in the drop downs do not fit with many of the strategies etc and they keep changing.

User124

As a coalition who is in year 6, there are things in Comet that have been eliminated for reporting but still show up in our report. Could these be taken off our report?

In all areas of Comet, once something is completed, it should be moved to an archived area not be listed along with current activity. It is very cumbersome to have to scroll through these.

Also, by design, we took a year off DFC between years 5 and 6. When I went to submit my report, it had me submitting reports from the year I missed. Luckily my PO was able to help out with this but it was a cliche in the system.

I don't feel that reporting through Comet is an accurate reflection of the scope and depth of the work being completed in our community. If something does not fit into a predetermined catagory, it may not get reported.

thanks for asking for our feedback.

User221

I have done COMET twice since taking over as the project coordinator. For a smaller coalition the pages came to 70. I had alot of trouble with the Evaluation section on the 4 Core measures. For a report to be so specific I felt like this section needed to be that much more specific in directions.

The training video does not let you move on until you type in everything as is. For myself trying to implement programs in the community and keep up with on-line training can be labor intensive. I would like to see the training for such be a little less time consuming. If we typed in everything as requested the training would truly take up 5-8 days. I ended up skipping around and figuring out what I could which does compromise the program at best.

User218

Overall, I think that COMET is straightforward to use. However, I feel like many of the drop-down categories/selections do not reflect our coalition’s events or activities. This does not promote accurate reporting. In addition, I had the most problems in the Evaluation section. Calculating the percentages and inputting the data were confusing and problematic. Specifically, when I entered an “inadequate” amount of participants, a response box would pop up “data is insufficient.” There were no explanations, just the vague response. If an explanation immediately followed, it might of proved helpful. I had to troubleshoot most of these problems myself.

Pro: accessibility (online)

Con: more options and specific categories/selections in drop-down menu

Suggestions:

Post specific questions that DFC/ONDCP would like to know and have ample space for narrative responses. A narrative option could provide supplemental and necessary information to substantiate responses.
If DFC does not need certain information, then do not ask for it. Reporting useless information takes time away from answering important and pertinent ones.
Have a running blog with Q&As to help with troubleshooting.
Having rollover bubbles that help you while you are reporting might be helpful. For example, when you rollover on a question about assessment, a pop-up bubble will appear asking, “How did you collect the data” or “How many people participated in your event.”
Like the SPF process, COMET should be mandated and included in DFC training.

User196

I could not agree more with what has been said in the other posts. I have been working as our Project Director and reporting into COMET since 2005. The system has no way, as some have mentioned, to clean itself of old data, including activities that may have changed, sector partners that have left the coalition, and so on. I would typically apologize to our Project Officer that there was such old data in there and she had to look through much of it.

Additionally, it is repetitive and still does not allow for the coalition to report on all the activities. I think that coalitions by nature are entities that change and do not remain static, if they do, something is wrong. By the same token, the tool used to report needs to also be flexible and fluid.

Finally, as one person said, the same report is used for STOP and DFC grantees making the report even more difficult to sort through and read.

That being said, I use other reporting systems, namely the state's and COMET is easier and more encompasing that the states but still needs work to be done to improve upon the system. Thank you for seeking our input! Hopefully positive changes can be made.

User223

I agree that the lack of having a feature that allows coalitions to clear old data and add new data in other areas aside from using the same box is annoying. Also many of the drop-down selections are not pertinent to the work/projects some of us do. It's difficult to try to make some of our coalitions activites fit the definition/labels in the drop-down categories. A pro is that we have the ability to look back at old data; I just wish it was more of a timeline view.

User180

I certainly echo what has been said thus far on the message boards, the system is incredibly frustrating to use. I consider myself to be pretty tech savvy and I have a very difficult time navigating from section to section, while being able to maintain the thought process to report about the initiative concisely, without compromising the material in order to obey the systems options (i.e., trying to fit a square peg into a round hole…)

Another area of frustration is the fact that the drop down menu does not match much of the risk/protective factors from the SPF/CTC model that we have all been trained in.
Drop down options are too limiting and then you end up taking so much room to explain yourself, that it's not really a media initiative, but kind of...
In terms of updating projects that were planned/in progress from the last reporting session, most of the time I run out of room reporting where we currently stand. I then have to add a "new activity" even though it isn't new at all.
The questions/sections are extremely repetitive and I find myself entering the same information over and over
The display spaces for the text from the drop down menu are too small, and you have to click into each just in order to read what risk/protective/type of activity you originally selected.
Spell check out be GREAT!
OVERALL, just not very user friendly - just in order to track things for myself I copy and paste everything to a word document, changing the color of the font each time I update every 6 months. As the coordinator, I have to send my COMET report to my project director so they are able to check the final product, before we send it in. If I didn't give her the word document, I think it would be pretty impossible for her to navigate through the system and be able to tell what is new and what is old information.

User91

First of all reporting should be a coalition’s best friend. Beyond the importance and respective points of having and using data. Reporting is the coalition success story. Great reporting should be a clear tool to the progress you have made and where you need to go next. You should be able to utilize the report as a foundational tool to indicate your achievements. This would be beneficial to acquiring key leadership of your community and to potential funders. I’m not sure if the COMET system and what we are entering into it are achieving what we need it to. I feel frustrated, tired and confused by the end of the COMET report asking myself did I place all of our entries into the right cataogies because they don’t seem to match your pull down options. As much time as we allocate to complete the COMET report it would be nice to actually have something that you could utilize to sustain and in achieving your coalitions efforts.

User214

Like many others that have expressed their feelings prior to me, I would have to say DITTO. I feel that comments specifically stated by 219, 215, 211, and 205 all summarize my feelings about the COMET system.

User91

Quoted: -

I just read 180 comments and I think they read our minds in total agreement

User225

I enjoy having somewhere to go to add the goals and summit accomplishment of the coalition efforts. However it isn't as user friendly as I would like it to be. It gathers all the information with no feed back of what could be more helpful when working in the community and what doesn't work. I had trouble at first, but have grown to understand the process.

User224

* Comet should be included in DFC training. The "virtual" training videos and webinars are not very helpful and do not go into enough detail, or answer questions.

* The drop down choices/strategies under the coaltions projects/activities a lot of times do not fit our coalitions projects or hoped for outcome. As some have already posted, it is difficult sometimes to fit the activity with the definitions or choices given.

* You have to complete the entire page/sector before COMET will save your data. As a consequence, you have to devote large blocks of time to accomplish this.

* The word number caps are very limiting when you are trying, as a first year grantee, to describe your community assessment, etc., and then you try to "save" the data and you have gone over the word number caps and you have to go back and take 3/4 of your data out.

* Activities/strategies have to be added a number of times under different headings. This becomes too repetitive and time consuming.

* COMET, in my opinion, is not user friendly and I spent a large block of time repeating data I had already put in.

* It is good to see all of the activities and the progress the coalition has accomplished when you read the finished report.

* Thank you for asking for the coalition's input and I hope some positive changes can be made as a result of the information you receive.

User222

Glancing through many of the comments that have previously been posted, I actually was relieved to find that I was not the only one that felt that the COMET is very repetitive and highly labor intensive. My last printable progress report was 150 pages long.

Since our coalition has so many activities to report on and so many sections ask for the same information, I find that I need to work on it in the evenings and weekends in order for me to complete it and still carry on the important day-to-day activites of our coalition. While I really appreciate the ability to access the system from other computers, I wanted to bring up the most annoying factor of the COMET that I ran into the last time that I completed the report ...The system requires a change in password at certain intervals (which I understand for security purposes), but because I had to work on the COMET after work hours in order to do it "justice", I did not have my most recent password change with me and my workplace is located some distance away. Further, more importantly, I did not know that I could only have 3 strikes to remember the password and that I would be thrown out of the system with a big red warning box to contact the system managers (or DITIC support) or I would have driven the distance to my office before I tried inserting my password 3 times! It was early on Saturday morning that I was barred from the system and I had planned to work on it off and on all weekend so that I could be well ahead of the deadline, but I could not reach DITC support until Monday so I lost all the time I had set aside for the purpose of working on the report. Other systems that I have worked with allow a person to ask for their password, and provided certain criteria are met, the password is emailed to them. I would suggest DITIC sipport set up a way for people to retrieve their password if they need to access the system and their password is not readily available or someone forgets to write it down when it is changed, particularly since it does take some time to get a response from the DITIC staff as they are busy too, regardless of when the password is needed. Similarly, please do not shut the system down on weekends before the report is due to work on it. If it does need maintenance, it would be more optimal to wait until after the due date if at all possible.

I have found that the printable report that our Project Officer receives does not show that we have a number of leaders heading up our activities which may be because many of our activities are implemented by our numerous committees and not just by one individual. I am not exactly sure why this is happening, but we only have one leader listed for all of our activities and we have innumerable leaders on all of these committees that should be counted! Thus, I agree with the comments that have been made that the report does not accurately reflect our coaliton's work. I have found the drop downs to be inflexible and inappropriate, especially when they do not allow for multiple members/partners in one or more sections. I particularly have run into a problem with objectives that are ongoing. I would like to say that these objectives are completed each year of the grant, but when I did this the first year of the report, I could not change them back to an active or ongoing status and now I am stuck trying to explain that these objectives are not really completed, but are ongoing every time I complete the report. In addition, in the Capacity Section where we list training needed/received or assistance needed, it is a real challenge to enter this information as one has to go down to the last pull down menu and click "received in order to even activate the top boxes to record this training or assistance needed. Isn't there an easier way to complete this section?

Lest it appears that I am just complaining, I do want to point out that i have been glad to have the information in the COMET when I need to find out how many times we have offered a Town Hall Meeting and what we presented as well as other useful historical information, but I do think that it would be easier to find this information if we could somehow access this information by topic and/or year instead of wading through so much information to find what we are seeking. Thus, achiving the information by year and/or topic would be very helpful.

Thank you for the opportunity to give input into the COMET process. I hope it is helpful.

User226

I have only been working with the COMET system for the last 2 years, but so far so good. It is nice to have a place to update and see our past accomplishments as well as future goals listed. However, I do find this process a little repetitive and time consuming. Most of the directors are probably on a part time basis and there is just not enough time to finish these reports. It does become a race to finish entering on time and then you have to make sure you didn't skip one of the many pages you just spent 2-3 hours going through.

We have a great support system locally with Prevention LI and thankfully they are right here every step of the way helping us transition from year 9 to year 10. We are transitioning into sustainability and our focus is really on our foundation and getting the board set to move forward as an entity in our community for years to come.

Hopefully the feedback you receive here will help to reform the the process and enable all of us to report with ease every quarter.

User227

We agree with much of the feedback already provided by others. Some other thoughts we have are….

It would be helpful if the report could be modified so that it is not so cumbersome to work with and time consuming to complete. Could the previous year’s data be archived in some way so that we could do a printout of the current work plan only? Could the print out be more reader friendly and in layman terms so that community members can comprehend? A lot of $ (time) is invested into this for the benefit of the Project Officer; it would be beneficial if we could utilize this info also. It is not a tool that we find easy to work with, nor helpful to the coalition or rural communities we support.

COMET would be more of an asset if we were able to create visual reports from the data entered

that could be shared with community members to help them see the value of the work they are

doing, hence the opportunity for more community buy-in.

On another note, it would be helpful if all reporting requirements, program and financial, for the

DFC and STOP grants could be addressed at one website instead of multiple websites.

Thank you for this opportunity for us to share our feedback.

User228

The feedback we give has not been reflective of what we do. Data that we have sometimes is better gotten in a qualitative format versus quantitative. Someone stated it would be nice to be able to use this forum for creating reports of the impact we have that we can share with the community.

User231

I agree with many of the posting that is labor intensive and is very repetitive. I have been using the system for 2 years and I think that it should have a better walk through for training. I learned a lot from the webinar but think it should have been included during year 1 training. I would have had a better idea and where things should be going. I also wish that there was a way that you could add pictures or articles that have been done about your coalition.

User230

Thank you for seeking our feedback about COMET. I’ve been using the system for approximately two years now, and I have identified three drawbacks. As others have mentioned, the drop-down choices/strategies often do not fit our coalition’s projects. I must then select one choice that most closely matches and explain how it is/is not like what I’ve selected from the drop-down menu.

In addition, as others have stated, it would be helpful to clear old data between reports, including activities that may have changed and sector partners that have left the coalition or changed roles.

Finally, it would be helpful if the character limit for responses were increased.

On the positive side, it is very helpful that the report is web-based. I also appreciate that I can access every report that we’ve already submitted. Thanks again for seeking our feedback.

User232

As a first year grantee, our coalition has been attending CADCA's National Coalition Academy for training throughout the year. Our last session focused on evaluation and the trainers requested that coaltions use 4 types of logs - event, media coverage, resources generated, and services provided (based on the model created at the University of Kansas). If all coalitions use these logs for evaluation then it only makes sense that the COMET reporting system mirrors that format. Obviously the 4 core measures are vital in reporting as well and should also be incorporated. The COMET system as it is now is quite cumbersome and is not user friendly. The drop-down menus limit responses that coalitions are able to select, sometimes leading to choosing the best option instead of the perfect/correct one. Developing a COMET system to mirror coalitions' local evaluation efforts will be very efficient and cut down on the time staff spend completing multiple recording/data collection systems. Also, it will allow coalitions to be more time focused on completing the 5 steps of the SPF process.

Thank you for allowing coalitions to provide feedback!

User229

The below comments reflect my opinion:

Comet should be included in DFC training. Although the webinar was helpful, it doesn't answer many questions that come up when you are actually using the program. This could be addressed with hands on training.

The drop down choices/strategies under the coalition's projects/activities often does not fit our coalition's projects. I agree with the other posts - it is difficult sometimes to fit the activity with the definitions or choices given. It would be helpful if we could modify these choices to fit our program.

Certain areas have a maximum number of letters and we often go over those numbers in our entries. We have to rewrite some of our data to fit in the allotted number. This is labor intensive and unnecessary. It also meant that one of our requirements was not included in a timely manner which resulted in a notification saying we were missing required data.

It is impossible to erase some of the data entered in error. We needed to contact COMET, the person helping us was unable to correct the problem and it took several weeks for them to get back to us. This was extremely frustrating.

I like the accomplishment/barrier section. It allows us to recap the things that worked and the challenges we need to address.

User235

While I realize the importance of getting all of the information to the DFC regarding our grant and what we are doing, I find that the information we must enter is quite repetitive. I am entering the same information in about three different areas of COMET and it takes a lot longer than I feel is necessary.

When I am entering information and hit save, the program takes me back to the top of the list on the previous page, which requires my scrolling down to where I was. This is aggravating. If we could go back to the exact section we were on, it would be less time consuming.

It would be wonderful if there were someway to hide the objectives from previous grant years so there is not quite so much to weed through while I am entering the current information/data.

Spell check would also be a good option to have.

Some of the choices in the dropdown menus are not reflective of what we are doing, for example, we do party patrols. This is not on the drop down menu for changing consequences.

User238

I'd love to be able to print off the data entered that would be useful to our coalition. Currently, I have only been able to print off the data as it was entered and submitted. But it would be cool to be able to manipulate the data to find out the number of new parterships and/or members that have increased or decreased over time. I'd like to see a diagram that showed the link between local conditions and interventions to have a visual to help us understand if we are addressing our efforts evenly or focusing on one objective.

Current entries take a long time and sometime feel repetitious. Sometimes I wonder if I had already entered something. It would be great if you could provide a search around key words so entries could be found easily.

User237

I have only had one experience with COMET being a first year grantee. I felt the COMET "virtual" training videos and were more about navigating the technology than effectively inputting your data. As others have stated the repetition takes time and seems unnecessary. As some have already posted, it is difficult sometimes to fit the activity with the definitions and word/character limits are frustrating. Saving information was cumbersome as I had to take a phone call or answer an email and couldn’t save until that section was complete. It ended up in loss of data and re entry several times.

Using COMET is fairly easy, as I knew how to enter the data however I spent a lot of time repeating data I had already put in and shortening descriptions to fit character limits.

The final report is helpful to celebrate successes and determine progress of the coalition.

I appreciate the chance to offer feedback and hope to see changes that make COMET a more efficient tool for coalition reporting.

User236

I agree with other posts. The system does not allow for cleaning of old data, activities, sector partners/graduating teens that leave the community .

Data needs to be entered repetitively.

Drop-downs do not reflect our current work.

Thank you for asking for our input. I am looking forward to improvements.

User239

Quoted: -

Thanks for asking for feedback from users! Comments are pretty consistent and I agree with almost all...

hands-on training is critical - and lacking
drop down choices are not appropriate for our coalition work
past reporting data should be archived, but available
I should be able to generate reports meaningful to the community by allowing me to choose which data items I want to focus on (including archived items)
data items are repetitive and definitions are very unclear

online access is WONDERFUL

User234

I have used this system regularly since it was established. I have not used the COMET tool for anything other than reporting to the Project Officer. Ideally this report should reflect the progress of our coalition and perhaps it does in some way. I just dont get it! I spend way too much time doing it and then copy the 168 page document in fear that it may get lost somewhere and file it away. Our coalition members do not read the report but are only interested if the P.O. has approved and accepted it. So, COMET is not used as a coalition tool just a report card.

Perhaps it would work better to do away with the SPF categories for reporting. I think that is one of the reasons so many things have to be re-entered. We are all doing the SPF work but struggling to report clearly. For example coalition trainings are listed under capacity along with multiple other activities and then many are relisted in implementation. Many times I would like to have one area to list coalition/staff trainings received just for our tracking and reporting. Media Campaign activity could have its own area, we all know you have to assess, build capacity, plan, implement, and evaluate. But what was the real outcome. How many media outlets did you utilize and how many people did you reach? A media campaign activity area would be something I could share with the coalition to show accomplishments, this could even be used to help track in-kind donations with our media partners. Another example would be collaboration meetings. Some of them we plan and implement. Others we just attend. Both can be building capacity and planning. Is the P.O. really getting a true picture? or Are they just interested in our knowledge of how it fits into the SPF.

The ideal tool would be something we could use regularly for reporting at our regular meetings and tracking. Graphs for our youth data could be useful. Perhaps an area to track in-kind. This would certainly provide a snapshot of activities and prevent duplication.

Previous grantee comments are ALL true and I totally agree!

User241

As a new Project Coordinator, I found COMET to serve as a training tool. It did help synthesize the project more fully for me.

However, I did not find it to be extremely user friendly. Additionally, the expectations seemed somewhat vague and up for interpretaton. Much of the areas seem to be quite repetitive.

Thank you for valuing my opinion.

User233

I found this comment below and agree 100%. Along with really needing custom reports.

It is a difficult system. It is hard to decide where information should go and what information is wanted. The information that you want to report does not always match with the choices available. Like others have stated, it is extremely time consuming, confusing to work with, redundant, and limiting.

User243

The COMET report gives our coalition a comprehensive compilation of valuable information compiled from many of the surveys we conduct with youth and community members.

I personally find the COMET report to be of significant use in meeting objectives and goals.

Many thanks for your support of our coalition initiatives.

User245

COMET is limited in its usefulness as a tool for our work, though it probably generates what the project officer needs for summary reporting. I find the drop down choices inadequate, repetitive and not reflective of our coalition's work . Furthermore, as a third year grantee, the sheer number of past entries makes the checking of new entries' information cumbersome and time consuming. Thirdly, COMET does not make it easy to print a hard copy report in an easy-to-read format.

I do appreciate the ease of access online and benefitted from the training videos about the system, but COMET still very much feels like a work-in-progress (duh, since you're asking for feedback), so any streamlining would be appreciated.

Thank you for giving us a chance to comment.

User246

Things that we would like to see improved with the COMET system are:

1- on a lot of the drop down boxes -the options provided don't always fit - it would be nice and more accurate if we could add our own titles if the ones that are provided aren't accurate - it would be simplier if we could just write what we did instead of picking one of the options on the drop downs.

2- on several of the narrative boxes there isn't enough room to tell you what we have done. It feels like we have to cut out so much that we aren't giving you an accurate picture because there is only room for so many characters.

3- it would be nice to be able to remove the deceased or coalition members who have moved away from our list.

4-it would be nice to see the data organized by date or quarterly report - it is very frustrating to have to look through the past several years to find things. It is too much information to sort through each quarter.

5-It seems it could be a lot simplier - for example: uploading the workplan and then being able to just write a narrative for what the coaition did from needs assessment to evaluation.

6-Spell checker would also be great :)

The pros are that it is a bi-annual report - some of it is quick and simple. The steps are also nice to be set up like SPF but it does seem like we repeat ourselves some.

User244

You should only have to log in one time with one password, not two times with two passwords.

When you are working in the COMET document, the data "home" page of whatever segment you are working in, should always stay in date order or the order you prefer. Once you enter something, when it takes you back to the "home" page you have to put it back in date order or subject order to review what you just entered. This is too time consuming.

You should be able to review the report, in report format, in the section you are working on, when you are entering your data. You should be able to look at any previous reporting time period without affecting how the current reporting period will actually print out.

You should be able to access any section of the report, in any reporting period, pull it up and if changes are necessary, you should be able to make them right there and be able to print the report.

You should be able to access the evaluation section of the report, in any reporting period, see it and then be able to make changes to it. For example, if there was an error on a previous survey that you conducted, as the report stands now, you cannot go back and make that correction. You cannot even make notes regarding the correction for future reference. The evaluation sector should be a useful reference tool and we should be able to use it and make reports from it. You should be able to use your historical data.

Referring to the "activity" sections and the "progress" sections - the report should populate the data where it needs to go instead of having to enter the data in both the activity section and the progress section. This is too time consuming and the results don't often print as you think they would anyway.

Referring to the "implementation" section. The current objectives should be the ones that pull up. There is no need to show old objectives - you can't do anything with them anyway. Again, it would be helpful to be able to pull up the old objective, if needed, without affecting the current report. Again, it is too time consuming to scroll through them all to get to the current reporting period.

Planning objectives should be able to be changed over time without re-entering all your objectives every year. Things change over the 5 year period, but yet we are unable to go in and make modifications. You should be able to make notes and change things in the current reporting period and be able to review it. The strategies and activity types should also be able to be changed with your objectives every year and the "drop down boxes" should be readable. You should also be able to change the strategies and activities, when you are working on a report, when, in special cases, they do not apply. Risk Factors are another thing that should be able to be changed.

DFC will make changes to the report, however, we cannot see those changes nor do we know how it will impact the data already entered or what we are about to enter. Therefore, how you are entering something today may not truly reflect the correct progression or how you were previously reporting it because of the change. Once again, because you cannot control the report or easily access segments you need, it proves to be ineffective. You only see the entire report once you are ready to submit it and print it.

Your due date for the report should be modified, perhaps the last day of the six months rather than two months later. Because of the data cut-off in the current reporting period - you must be very careful as to not affect the data should you want to go in and look at something or go ahead and start the next reporting period. You may affect the report without even knowing it and then things may not coincide as they should. Therefore, when you look at a current report, old data is still showing up from previous periods even though that information is old and does not apply to the current period and then current information may not show up correctly or repeats itself in the next report. It is very confusing when you look at a final report to be printed because it does not truly reflect what you have entered for the current period. The reporting period access needs to be greatly improved to avoid this problem in the future.

It would be nice if numbers would accumulate on certain projects. For example, if you are required to do so many compliance checks over a time period, it would be nice if the report would show the "cumulative" number to-date.

User249

The existing system requires some repetition as many other people have commented but, in particular, I've had trouble inputting my data on youth substance use - an area of the sight that I feel should be very simple and very clear because the data is so important.

User248

In the Implementation section, reporting activities, there are not enough choices in the drop-down menus for different strategies. For instance, under Enhancing Skills, in the CADCA primer on planning, the examples provided for enhancing skills are not even choices. The same with Providing Support. Usually, I have to select activities that are not even remotely related to what was done because nothing comes close. Many of the choices reflect your typical "individual change" prevention activities/strategies, not community-level change ones.

Our coalition has literally hundreds of people involved in committees, work groups, councils, etc. They are not members of the coalition. In the Implementation section, we have to identify what groups/individuals (MEMBERS of the coalition) were involved in implementing different activities/strategies. Often, I can't identify the proper groups because they're not registered in the COMET system as coalition members. Perhaps if there was a section in which we could keep adding partners, not just a place to list coalition members.

I've always been unclear about how many activities I should be listing. We do a lot...our project officer has suggested that we list too many things, but how much is too much?

User247

I appreciate the opportunity to provide feedback on the COMET system. Our coalition is finishing our first year and therefore I have completed one COMET report thus far. I heard so much about COMET during many breakout sessions at the new grantee meeting in December that I was apprehensive to start the process. The process was cumbersome but not nearly as overwhelming as I anticipated.

The message we took home from the meeting in D.C was that the system is being carefully reviewed for improvements and will be enhanced during the next year. I have read through all of the detailed feedback expressed here and realize that most people share similiar opinions on the limitations of the system yet most also agree on the need and value of a concise reporting system for our work. I share these overarching opinions as well and look forward to the system improvements that will be made after careful review of this feedback forum.

User251

 Far too much duplication on some of the sections, specially the Basic Collaboration

 The Assistance needed / received needs to be re-sequenced

 It would make the reporting process easier if we were able to print the current report without having to print previous ones

 It would also make it easy to report if we were able to do entries as we go rather than waiting until we are giving the green light to do it

 The options in most of the sections are hard to match with what you are reporting on

 The drop down boxes do not allow us to fully see our report when it is printed, please make these boxes bigger

User252

There are not alot of choices to choose from in the drop boxes. It is alot of repeated information that has to be posted. It is very time consuming to do a quarter's worth of information and still sometimes when you pick a category to put something in for example assessment and you have a lot of information about an event that category (and it could possilbly be another one) will not let you enter the whole documentation only the title of the event. Alot of time the tools that we use for reporting we can put them in every category, it doesnt just fit in one, is there a place for everything in one, instead of putting this in every category. Also, there are members that are not with the coalition any more, students have graduated and not involved, people have moved, etc how do we delete them off of our membership. Cadca/DFC should provide us with a specific tool, instead of us making up our own to cover, all the questions that are in comment to ensure that we are putting the correct information in the right place and also when it is time to complete the report there is no figuring it out, the questions have been answered all we have to do is enter. With the hand made tool, it doesnt cover everything that is in the comet report therefore sometimes I feel all information is not being considered or reported.

I have not been happy with COMET from the beginning it really doesnt seem effective or efficient in the reporting process...

User253

Some of the following improvements we feel are needed:

1. One thing that comes to mind is the need to "cross-populate" data. In other words when an event or activity is entered under planning, it should automatically be entered in progress, challenges etc. Then when you go to that section all you do is add details...the name, place, who etc would already be there.

2. The membership list that is created and continuously updated in the capacity section, should be allowed to be truly edited. People die, people discontinue membership, people change names through divorce and marriage, people change agencies...etc. I would think it is more accurate and less cumbersome to be able to edit the list more fluidly.

3. In the planning and implementation and even capacity sections, the drop down lists provided to us enables us to choose a certain category. These menus are often incomplete and too narrow for what we do. They also are discontinued too frequently. So, it makes it more confusing and less accurate than if we had a more consistent and broad drop-down menu of choices. I feel that DFC coalitions need to be engaged in this process to better represent the actual things that coalitions do, inside and outside of the grant requirements.

4. When entering activities as it relates to goals in the implementation section...the types of information that needs to be given is often duplicated over several of the goals/objectives. This particular process needs to be shortened because the activities and progress often can be connected to several different goals/objectives at a time. It would be nice to be able to choose the goals/objectives that it relates to all at once. So that we don't have to continue entering redundant information for each single goal...rather we should be able to enter relevant information and choose maybe from a dropdown window all of the objective and goals that it directly or indirectly addresses in one small and swift keystroke.

5. I like the ease and accessibility of using an online reporting system. Much of it is helpful, but it should be improved upon to make it more userfriendly, consistent, and that accurately addresses a broader range of relevant coalition activities.

User106

No one enjoys the part of the job that requires reporting and data entry. However, I have found over the COMET system for data reporting is by far the most complicated and unfriendly system I have ever used. Trying to log-in alone can be immensely complicated, following a revamp of the system just over a year ago that directed you to a new page to log-in with the direction to create a new password. Now, if you enter your password (requiring letters, upper and lower case, along with numbers and now a symbol) wrong (more than twice I believe) you get locked out of the system and must wait for technical support (that is only open during office hours) to ‘set you free’. And that is just step one!

The entry system itself is a road map of picking and choosing what you hope is the correct option for activity and data input versus trying to scroll down through years of already input materials. The obstacle course of inputting your entries at least three times to ensure you have it in the right spot is only made easier by the fact you can cut and paste your items from your own personal word file. Scrolling down through years of previously entered activities, coalition members, data, etc makes the process take twice as long. How our Project Officers wade through the myriad of years of submitted reports is beyond me. Their task is seemingly insurmountable.

The entry of reporting data is, as someone already put, ‘busy work’. I spend a good week gathering all the materials and reports from our different coalition partners. Then it takes a day or so to compile it into an at least 20 page Word document. Then another two or so days to do the repeated cut and paste process into the COMET system. And at the final point when I hit submit, I never feel fully confident that I have painted an accurate picture of all our coalition has accomplished over the past six months. I would much rather do a comprehensive, 20-25 page report highlighting all our members, groups, accomplishments, barriers (in somewhat of a similar layout breakdown that COMET gives), but in a way that is readable, understandable and most importantly printable (not 200 pages). While I am certainly thankful that we are going into Year 7 of funding and that I have only been questioned on a report once by my Project Officer, I still feel like something is missing when sending the report along to Washington, D.C.

I can only hope that the clear frustration that is being posted by the majority of other Coordinators on this feedback form will lead to an ultimate change in the reporting system in the future. I think it would only benefit both the funded coalitions and their Project Officers in terms of both time and energy expended. Plus, wouldn’t it be nice to be able to hand our government officials a nicely packaged 20 page report every six months on what their agreement to give us this federal funding has granted us and our communities?!?!

Thank you for finally giving us a voice to air our many thoughts on the current COMET system.

User254

I feel like I am not able to really portray the work of the coalition due to the fact of my drop down box options. There are items that my coalition do that are innovative and just do not really fit into the COMET. There is also alot of repetative things that can be confusing for a new grantee.

Also, I would find it more beneficial to do quarterly reports then twice a year due to the amount of information and updating.

User255

The major issues we see seem to be the same as many others:

1. There needs to be an information clearing system after each report. There is so much in the COMET system that it is hard to navigate and items end up in the wrong places- expired goals, etc.

2. You can not always clearly see what is in the drop down boxes- only half the selection is visible.

3. An example of what is really needed would be very handy. I have heard of reports from 25 to 200 pages.

4. A lot of times, thers is not a good selection for what we are doing so we have to select something that does really make sense, which makes the report somewhat invalid and not all that useful.

We do understand that there is a lot of information that needs to be reported so an on-line tool like COMET is helpful. Maybe just work out some of the kinks in the future and provide coalitions a clear understaning of what the report should look like would help.

User109

I too agree with most of what is posted here.

1. Way too convoluted--easier navigation would be key. An information clearing system after each report was mentioned and that would be very helpful.

2. What do the DFC grantors really want to see? An example of what a "good report" looks like would be a huge help.

3. I don't understand why at some menus I can only choose from "retired" strategies...?

Thank you!

User256

The COMET system has a lot of duplication information being asked throughout the report. If there could be a way to put it in once with maybe check boxes if it hits multiple sections fo the SPF that would be great (ex: if a strategy falls under collaboration and implementation, etc.)

I would love to see an archive section. There is far too much information that needs to be scanned through that has either been deleted or closed (this is escpecially true for those coalitions further in the process). I understand the need to look back at our efforts but having the archive section would offer that ability while allowing those reporting the ease of not having to scroll through years and years of completed work to enter in the current information.

I also think there should be an other option for some of the drop down sections. Some of the initiatives our coalition undertakes doesn't really fall into any of the options given although they are environmental and working for our community. Maybe there could be an explanation box if you check the other option.

I would like to be able to delete old members off the partner list. Some partners have changed which community sector they fall under and in order to do that I have to delete the old one and then add them all over under the new sector. Meanwhile the deleted one still remains on the list which can cause confusion when returning to the system six months later.

User257

I understand that COMET is for the federal government but all the time spent entering information it should be readily available to us also.

Drop down menus are great but how about another space for our own name for each activity, etc. I have 6-8 activities that all have the same name so to update any one of them is a nightmare.

In some places an expalantion box is available for the drop down menus but you can't read the whole thing!!

Redundent - really something is amiss if I am copying information from capacity building directly to implemention. Perhaps a drop down to check when something fits inot more than one category.

All my staff hates entering information because it is confusing and terribly slow to update!

User257

Quoted: -

I wholeheartedly agree that the drop downs don't always (and sometimes never) fit the work we are doing in a local coalition

User257

Quoted: -

Thank you for finally giving us a voice to air our many thoughts on the current COMET system.

User259

This person said it all and very eloquently

User259

I like how the COMET system follows the SPF/SIG model. It really is the ONLY reporting tool that is established for coalition/prevention work. It's much better than many other systems I have to use which mainly "count" numbers, is based on "classes" or treatment. So, I like the logic model it follows.

That said, there are some glitches in the system which have already been reported on. I especially think some sort of print function that could print out reports would be helpful. It would be "cool" if the Core measures had a feature in which pie charts or other graphs could be generated from year to year "progress".

I think the calendar feature could also be set up easier. What would be nice is if we could simply use it as our daily activity log and put into the implementation objectives. It's not easily done now that way.

Overall, it's not very user friendly. But still I'd rate it as a very well organized reporting tool that just needs some TLC to make it a little easier to input data/activities, then have the ability to print out relevant reports.

User258

I was new to Comet this year and it was quite intimidating. I am still not sure if I used it correctly. I watched the training video and read the manual. It seemed really complex for the simple information it was asking.

It would be nice to enter in the date of the activity, event, community change, etc..

Name the activity.

Check which category it falls under Capacity, Implementation, Evaluation Etc. Check if it was an event, activity, CMCA, coalition building, etc.

Add how many were impacted through information disseminated, those attending, etc.

Add Coalition partners involved with the ability to add partners that arent with the coalition but joined for that moment. Ideally, yes you would want them on the coalition but its not always the case. Example DEA shows up for National RX Take Back.

Check a box on targeted population.

Check box for Completed, ongoing, and disrupted with the reason why.

Then at the end of each activity we enter. It should have the option to click Finish and enter new activity or Finish. So fo each thing we enter. We hit all that apply. Enter specific information for that event and then add another with the ability to add comments. The system should then be able to categorize each entry in the correct place, Implementation, Evaluation, etc.

User261

!. Since entering the DFC grantee arena in 1994, there have been many innovative changes that contribute to award recepient's ability to pursue community change. COMET has made the process of progress reporting much less cumbersome and time-on-task efficient. There is opportunity for each grantee to list accomplishments as some Users have commented on.

2. This feedback forum is another venue that may prove beneficial. For example, it is good to know that data can be presented in Excel and other formats. This is important to know as data input is time consuming.

3. Sustainability plans are updated annually using a Continuous Improvement Management Process with COMET allowing annual updating and tracking.

4. Having national and state data avialable for comparison might be an area the Evaluation Site could promote via COMET or the DFC site. Comparion of in-state and state-to-state could be made to allow grantee feedback on current status of project population.

User262

While it is certain that COMET is a useful tool for collecting the necessary progress and data from the coalitions, the system seems overly secure. In the eighteen months I have needed access to COMET to enter infomration, I have been denied access three times; each time because the user ID or password have expired. What is so sensitive that the system requires a change in password every six months? I cannot imagine the information being entered is confidential that it necessitates a change in password every six months. This is a frustration to the user which is unnecessary in my opinion.

User264

Although paperwork is not my strength, I appreciate the fact that the information I submit, actually gets read and used to promote the work our coalitions are doing. The one challenge I find with Comet is that many of the boxes are dictated by previous pages and limits how questions can be answered. I would prefer more fill in the boxes to allow for a better discription of the work we are doing. I would also appreciate that it would not time you out so quickly so you could have more time to write the information before submitting.

User265

Overall, I would say it's fairly decent. I am able to record all our activities and refer back to it and use it as a form of activity log. I also agree that an archive section for all objectives etc that are closed (thus eliminating uneccessary data and clutter) would be helpful.

The COMET format forces me to think the activities through using the logic model format (which is a plus) but it would be helpful if I could easily go through the steps on a particular strategy or activity from beginning (objective) to the end..(outcomes)...without having to work through each category over and over again as I work through the steps. Perhaps a dual process where you could choose the line of thinking or process of reporting out that best accomadates the data you are sharing...

I also find the scroll down choices limiting or confusing. Though I understand the framework you are trying to build...it just isn't always a clear fit nor are the community strategies as limited or as simple...kind of like fitting a square peg in a round hole.

User266

Overall it is an average system to us. If the information is printed out it makes much more sense than just putting information in. At times it seems extremely redundent.

User267

I cpet the comments simple and direct, but having to report the same information over and over again, was frustrating. I was hearing from several coaltions about how difficult it was, but when I completed the report, it did not seem that difficult to me, just lots of repeating. The directions were helpful for submitting the survey data from the state youth survey our state uses.

User267

I kept the comments simple and direct, but having to report the same information over and over again, was frustrating. I was hearing from several coaltions about how difficult it was, but when I completed the report, it did not seem that difficult to me, just lots of repeating. The directions were helpful for submitting the survey data from the state youth survey our state uses.

User268

Regarding the COMET System:

These are the things that I have noticed:

1) I'd like to be able to DELETE people who are no longer with the coalition due to

graduated from HS, have moved, deceased, etc.

2) I feel the system is redundant.

3) Sometimes I cannot find the scroll down choice that fits and there is no option

for OTHER.

4) The assessment section is not "user friendly". Takes a long time to complete.

5) Would like the option to print out customized reports with pie graphs/bar graphs/line graphs, etc.

6) Would like to submit log information (community change, media, resources, etc)

7) Like the fact that you can contact SUPPORT if needed.

8) Supplying data......easier chart?

User269

I am frustrated at times about the repetitive nature of the COMET Report. I would like to see some of the categories eliminated or reduced when they have already been addressed in previous areas. I think statistical data would be more beneficial and included in a separate section. Each year grantees are asked to conduct surveys and I think this is important information that should be reported out in detail on COMET so that we have a good assessment tool to use when planning strategies.

I have completed only two (2) COMET Reports since becoming the coordinator of our coalition. The COMET Report prior to that was done by the current Program Director who had to "piece together" the previous six-months when the previous coordinator left the organization.

When drastic changes occur, like the loss of a coordinator, or another key person in the organization, the COMET Report appears to be an overwhelming "paper chase", and it shouldn't. The core premise of the reporting should be reflective of the Coalition's achievements and a working document to show where the coalition is headed in the future.

User270

Quoted: -

Regarding the COMET System:

These are the things that I have noticed:

1) I'd like to be able to DELETE people who are no longer with the coalition due to

graduated from HS, have moved, deceased, etc.

2) I feel the system is redundant.

3) Sometimes I cannot find the scroll down choice that fits and there is no option

for OTHER.

4) The assessment section is not "user friendly". Takes a long time to complete.

5) Would like the option to print out customized reports with pie graphs/bar graphs/line graphs, etc.

6) Would like to submit log information (community change, media, resources, etc)

7) Like the fact that you can contact SUPPORT if needed.

8) Supplying data......easier chart?

I am commenting on the statement made by user 268. I agree with everything s/he says. I have been very frustrated that the drop down lists often don't show the choices we are supposed to choose from. Someone needs to go though and make sure the information we are given and choices listed are consistent. And we need an "other" category. I like the idea of giving us categories to choose from because that helps me understand the kind of information you want from us. I think it would be very helpful to be able to see sample entries for each section, so we understand the format and language we should be using, what specifics should be included and in how much detail we shouldhave in our descriptions. On the positive side, I like that the system has categories based on the SPF. I like that the reporting system is cumulative since it helps us be able to see in one report the progression of our work.

User270

Quoted: -

It would be nice to enter in the date of the activity, event, community change, etc..

Name the activity.

Check which category it falls under Capacity, Implementation, Evaluation Etc. Check if it was an event, activity, CMCA, coalition building, etc.

Add how many were impacted through information disseminated, those attending, etc.

Check a box on targeted population.

Check box for Completed, ongoing, and disrupted with the reason why.

Excellent suggestions 258. I agree that these changes would help streamline the process and provide a better fit with the reality of our work.

User270

In addition to my other comments... It would also be great if there was a way we could upload other major coalition documents in addition to our report, sustainability plan etc. I understand that you would want to discourage coalitions from uploading too many items, but adding brochures we've generated, links to newpaper articles and other major planning documents might help us give a more accurate picture of what we're doing. Thanks for listening!

User147

Quoted: -

This person said it all and very eloquently

Like user 257, I find the COMET system to be very frustrating.

User147

Like many other users of the COMET system, I find it to be somewhat intimidating and sometimes very frustrating. I have been reporting in COMET for less than two years, and I'm still trying to get the hang of it. While I know that the information needs to be collected, I believe that it could be done under much simpler terms. For instance:

Rather than go through the individual stages of Assessment, Planning, Implementation, etc., why not allow a drop down box with a selection of all that apply. Then a full description of what actually took place from the assessment phase through evaluation could be stated in one place. This way the information does not have to be repeated from section to section.

It would be absolutely wonderful if we could have the option of reporting specifically on the strategies listed in our DFC workplan. After all, isn't the workplan designed to be sort of a road map for the coalition. I realize that other projects and activites may happen during the reporting period which may not be in our workplan, but this could be covered under an "other" option! I would think that it would make things a little easier for the coalition, as well as the project officer when trying to compare our reporting to our actual workplan.

I would love to have the option to delete or at least archive information that is obsolete and no longer applies to the coalition. It is so frustrating trying to complete a report and having to go through all the old information.

In relation to risk and protective factors, it would be good to see the full list of both. This would allow us to choose the factors more closely related to the issues identified in our individual communities. Since there are various lists of risks and protective factors from several credible sources, this might be a good place to allow an option for "other", with a space to type it in.

I will post other comments as I think of them. I do appreciate the opportunity to express my opinions about the COMET system and hope that they will be seriously considered. I also commend the ONDCP and SAMHSA for all the wonderful work they are doing.

User271

I also agreed with many of the comments that have been posted. I am a year one grantee and have only entered data into COMET one time. It was very intimidating and somewhat frustrating. Repeating yourself numerous times is never fun! I also found that some of my coalition's activities did not fit in the drop-down categories. Please consider adding an "other: ________" option.

Sometimes I feel as if I spend more time reporting what I am doing than actually doing things. Time is a very valuable thing. Anything that you can do to streamline this process and cut back on the time required, would be greatly appreciated.

User147

Quoted: -

Regarding coalition membership, I think it would be much less time intensive for a coalition to simply upload a membership list. Perhaps that list is designed to identify stakeholder representation & member engagement level (supporter, implementer, leader) or whatever. We have specifically been uncomfortable entering youth names into COMET, so we've been using initials instead. It has been challenging to figure out the definition of "member". Providing a valued and appropriate spectrum of involvement is more realistic for coalition work. Our community may have a local champion who will provide impactful influence occasionally, but who doesn't get involved in the administrative maintenance of the coalition, or drug-free supporters who wish to stay informed, but don't have time to commit to the cause. At the other end of spectrum are key coalition leaders who conduct planning, evaluation and administrative duties on behalf of the coalition. In our communities, this entire spectrum of involvement is valued but we struggle to communicate these varying roles in our membership list. The term "membership" can often imply a continuous and high-engagement with an organization, whereas we might have a small number of continuous and highly engaged individuals who mobilize the other sectors of the community for specific projects in a less continuous fashion.

Regarding coalition functioning, I think a growing and improving coalition needs to be tracking its leadership development, expertise in implementing prevention interventions, and ability to sustain progress made in drug-free community goals. COMET does not currently include a self-evaluation from coalition members regarding their input on the items above. Ideally, a standard coalition member assessment (with an ability to indicate member engagement level as related to above comment) could be conducted via a short online survey (distributed to individual coalition members) which would aggregate at the local, state and federal levels so that each entity can see what technical assistance might be needed to help coalitions increase effectiveness and sustainability. Ideally, data would be tracked over time to show increased capacity and self-reported effectiveness of coalitions. While it might be scary to have local coalition member survey data available to state and federal stakeholders, data entry would only occur once, by each individual coalition member, without the need for local or state workers to re-enter the data for a DFC or STOP report. Ideally, this survey would be conducted annually, NOT in the summer, and results would be available in real-time, so that local coalition leaders could gauge participation, completion and results without waiting 6-9 months for data to trickle back down to the local level. Maybe a partnership with Zoomerang or Survey Monkey could be arranged?

I really like the idea of being able to upload the coalition membership list, rather than going through all the individual agencies and volunteers, etc.

User147

Quoted: -

Regarding the COMET System:

These are the things that I have noticed:

1) I'd like to be able to DELETE people who are no longer with the coalition due to

graduated from HS, have moved, deceased, etc.

2) I feel the system is redundant.

3) Sometimes I cannot find the scroll down choice that fits and there is no option

for OTHER.

4) The assessment section is not "user friendly". Takes a long time to complete.

5) Would like the option to print out customized reports with pie graphs/bar graphs/line graphs, etc.

6) Would like to submit log information (community change, media, resources, etc)

7) Like the fact that you can contact SUPPORT if needed.

8) Supplying data......easier chart?

I really like user 268 idea of being able to print out customized reports.

User74

Quoted: -

This person said it all and very eloquently

I have been a COMET user since its inception. From the get-go I have found COMET to be very un-friendly. I'm not a stranger to data -based reporting systems, however COMET continues to be so confusing. Some drop down boxes are set to default to 1 option, therefore the user is not able to type in the actual information which is completely different. My employer requires staff to use 'Firefox' and not Internet Explorer. When I use Firefox, it limits my view of the page. I have DFC & STOP Act grants. When entering data, it ends up being 'bundled' even though they have different goals & objectives. Past reporting info stays active on the screen and it should be filed somewhere else in the database system ~ 1 year at at a time for viewing. Thanks for the opportunity to give feedback. I look forward to seeing an 'extreme makeover' in the near future.

User272

We have been using COMET for 2 years. It seems as though we need to change the password every time we want to enter the site - which is very frustrating.

The reporting is very cumbersome given that there are so many screens. There is a tremendous amount of duplication between activities and accomplishments. Never understood the difference between entering data under activities versus progress

You have to jump around between tabs to enter data instead of all data being entered in one spot and then the program distributing it where it needs to be (based on check boxes).

It requires the actual name of all the partners who are working on the coalition and this is changing all the time. All this detail does not assist the coalition to see the effectiveness of its work.

Much too much duplication

This tool has felt more like a required activity of the funder and not at all useful to the coalition or myself in understanding how we are meeting our goals. There is no user friendly option for writing a monthly, quarterly or yearly report for the coalition to use to evaluate its work. We would like this.

User273

I didn't feel that the COMET system accurately got the information that told how we are doing on the work. It wasn't as bad as what I had to do for the SPF SIG, however.

User274

I have been using COMET for almost 5 years. The biggest issue to me is that there is a lot of duplication. For instance, an activity might be entered under both Capacity and Implementation, or Capacity and Planning, or Assessment and Evaluation, because one activity might be appropriate for two or even three of the steps in the SPF process. Many activities fit under so many of the different steps that it is very cumbersome to enter all of the info. It would be better if you could enter the activity once and then check boxes for each category it fits under instead of entering the description several times.

Under membership, there should be a way to delete members completely, instead of only being able to make them inactive.

It is also frustrating that I have to change my password so frequently. The last few times I've logged into COMET, I had to change my password.

User277

I agree with many of the comments others have reported. (1) We seem to have password problems each time we use the COMET system. (2) There is always redundancy reporting collaborative activities under capacity and then again in planning and implementation. In fact, capacity increases among collaborative members when they engage in planning and implementation activities! (3) Under Planning, "Strategies to Achieve Objectives" there is a tremendous amount of redundancy and with some activities such as policy change we are in fact using all 7 community change strategies! I think the question is vague, and overly general. I don't see how the agency is gaining detailed enough data from this question. (4) There needs to be an 'Other' column to accurately state what we are doing.

User278

Quoted: -

We have troubles logging in to the COMET program. We also have put data into the report that has not saved into the report. Data was duplicated through-out the report and was confusing while we input the information.

User53

Thank you for soliciting our input. Here's my experience with the COMET tool:

Broadly, I feel that if 'completed' information could be accessed through an archive system, as opposed to being listed with current activities, then the Comet tool would be more useful. It would be easier to use as a work management system between report periods, and the time required to enter information would be cut down by half.

Other, more specific, areas to address might be:

Providing notification when the 1000-word character limit is passed. Now, we assume that everything typed in is recorded, unless we go back.

Allowing new 5 -year grantees to start with a fresh Comet report-- a clean slate -- having learned the system better.

In the uploading of survey data in the Evaluation module, it would be helpful if specific school districts could be named (even in code) in association with the data listed. Now, the only distinguishing feature allowed (as far as I can see) is: "smaller/less than Target Area."

If we discover that we have made an error in a previous report, i.e., entered data in the wrong category, it would be helpful if we could change or delete it.

On the Implementation summary page, it would be helpful to see status (esp 'Completed') and 'last update." It would be very useful for the ongoing tracking of activities.

On 'key people involved', it would be helpful if the software would 'guess' at names that we begin to type in, referencing the names we have already entered in the Capacity Module. As it stands now, we need to select from a drop-down list which contains hundreds of people (and an accidental slip of a button loses the names temporarily selected). The process takes an excruciatingly long time.

It would be helpful if the instructions on the Comet Tool itself addressed the following:

In the Planning module, should 'Objectives' be broad or specific? Should detailed 'sub-objectives' be listed (added at the end of) existing objectives, as they arise? We have done this, and hand-written the date of the entries within the description, since the software currently doesn't trigger a new date in the Planning Module summary page. For this reason, we wonder if Project Officers even know that Objectives updates have been entered. If the P.O.s don't know to check there (because the Planning summary pages shows an old date) we should be instructed to contain our reporting for this in the Implementation module.

Again, we appreciate the opportunity to share this feedback.

User117

While I appreciate being asked for my opinion, it would have been much more effective to give feedback immediately after completing the COMET report in May. It has helped immensely to be able to review input from the other grantees here, and that has helped me focus my comments.

I am glad to see that many of my concerns and frustrations have been reported by others. I had also wondered that such an important item as this report did not come with any real training for those who are entering data. This could have been done at the national trainings we attended. The on-line training videos focus on the process of inputing information, not the process for identifying the key items to be reported.

First, in its present format, COMET serves only one purpose - reporting to the DFC powers who be. I agree wholeheartedly with the comment that it would be helpful to be able to easily share parts of this report with Coalition members, not just report that our report had been accepted by our program officer. We are a second year grantee, and I estimate that I spent 24 hours inputing the information in May, and that does not include time spent doing the revisions requested by our project officer. That amount of time twice a year should produce a product that can be easily shared to celebrate our accomplishments.

If the system was more simple, including easier to get in and out of, one would be likely to go in and work on it periodically, instead of clearing one's desk for three days to complete it to meet the deadline. Then when I completed it, and the system would not accept my work, I really questioned the process.

It is particularly frustrating to not be able to print out the information in an easily readable format. Although I am familiar with on-line reporting systems, I prefer to be able to review and edit from a printed copy.

I second the comment that the due date for the submission of the report and the cut-off date for reporting information are difficult to keep separate. Both times somehow we have had major successful accomplishments which fall into that "no man's land" of dates...leaving one frustrated that one has to wait six months now to report these items.

I also agree that the membership section is not only frustrating to edit and does not truly represent our Coalition's volunteer workforce. I agree with the suggestion that somewhere we should be able to note our partners, who often are not Coalition members.

And I also second the comments that the pulldowns in the Planning and Implementation sections are far too narrow and maybe even sometimes inappropriate for what I am trying to report. This could impact on the accuracy of what I am trying to report also.

User281

Thank you for the opportunity to provide feedback on the COMET system.

From what I've read in the other comments, I suppose "frustrating" is the most appropriate term for using COMET. We have been using the system since its inception, and it has never really seemed to come together as a unified reporting tool. There seem to have been many changes to the system, rendering certain option unusable or re-written in confusing language (I'm thinking about drop-down boxes, and their options, used throughout the system).

A couple of areas that could make COMET much less frustrating:

- Remove the need for repeated data entry. As a coalition, we use strategies and activities that support multiple objectives and/or goals. When it comes time to update these activities, we have not found an easy way to do so aside from the "brute force" method of updating data. If we could have a somewhat simpler system where, perhaps, our activities and strategies are stored in one place and changes in that table would "trickle down" to the entries throughout the system, it would save hours of data entry time.

- Allow for better views and tracking of activities, strategies, and timelines. We have started to tag our entries with a keyword at the front (e.g., STOP ACT ACTIVITY) to better keep track of our lists as they grow (this also helps in the capacity section by using the sector as the leading tag for each entry). It would be nice if COMET had a built-in system for organizing this information a bit better. In addition, it would be great if we had a visual representation (line graphs) of our activities and timelines. This could help coalition members in analyses of contribution.

Thanks again for this opportunity to provide feedback on the improvement of COMET!

User282

The COMET system would be most helpful if it could provide graphing tools for the data that is entered so that it could then be used for presentations. It would be good if you could enter other data besides the NOMs data so that this can all be tracked in one place. Updating objectives has been difficult, because we use the same activities to meet multiple objectives and the data has to be inputed multiple times. This is time consuming and not especially a productive use of our time. In some ways the system would be better if it were more like the KIT system that requires that data be inputed on at least a weekly basis. This makes for more accurate and more thorough reporting.

User280

Our coalition has been using COMET since its inception. That being said, we were excited in the beginning that not only would this be our reporting system for the DFC grant but that COMET would be a useful tool for our coalition to utilize. That has not been the case as the system is extremely repetitive and many sections are difficult to use (especially drop down sections that dictate what you say versus letting us enter your own 'section').

The idea of using the SPF model of assessment, capacity, planning, implentation and evaluation was an excellent idea. But instead of being able to report on each objective and activity- showing the 5 steps- we must report on EACH activity within ALL five steps.

All of this has meant that COMET is a tool that takes an incredible amount of manpower hours to complete. Twice a year when we complete the COMET, I look at the hours our staff members log in on the system and think about how we could have used those hours to do our prevention work.

Suggestions for a future tool:

- set it up to list the objectives and activities under the goals and have them be entered once. Then when a coalition reports in twice a year they can show their progress within the 5 SPF steps thus allowing a clear picture of where you are within the 5 steps of the SPF.

- make it be an easy system to utilize, one that allows coaltions to use throughout the reporting period. (Ideally having it set up so that excel spreadsheet can be used to list various things- it would allow a coalition to download their charts/ work and use it.)

- membership should be easy to track, allowing you to delete and add as necessary.

- if there are set drop down sections to chose from, allow coalitions to add a topic that might not be listed in the drop down list for that section.

User284

Quoted: -

All of this has meant that COMET is a tool that takes an incredible amount of manpower hours to complete. Twice a year when we complete the COMET, I look at the hours our staff members log in on the system and think about how we could have used those hours to do our prevention work.

Suggestions for a future tool:

- membership should be easy to track, allowing you to delete and add as necessary.

- if there are set drop down sections to chose from, allow coalitions to add a topic that might not be listed in the drop down list for that section.

For flexibility, llow percentages of time to be 5%. (Currently the lowest percentage is 10%.) .

Allow us to be able to determine the strategy. Currently we are restricted.

User285

I agree with User 284. I thought the COMET would allow us to have a better reporting system. Hoping ,it would be an easy way our Coalition could use it as documentation. However, I have found that it requires more time than anticipated.

My problem has been I cannot get it to remove members who are no longer there-example students who have graduated. Also,our data collected in the state does not break down by gender just grades. It seems so time consuming to go under each category to enter all activities to meet the objectives.I feel like there should be a faster and more effective way for us to be able to do report data.

User162

The areas I think that need improvement are in the coalition members, capacity building, planning & implementation.

Since we have taskforces within our coalition that are given activities to implement, in order to give them credit for their activities I am forced to list the taskforce as a member, while the members of the taskforce are already listed by the sector/agency. I also do not like that you cannot remove anyone, we have had change in our coalition and those members are still listed in our comet although they no longer are working with us, they are not inactive which to me sounds temporary these members have moved or separated from the coalition.

Under the capacity building sector we have a lot of meetings, and I have found that it gets time consuming to enter each meeting individually so I enter the one meeting and then list the dates of each of the meetings during that reporting period in the information/notes section, otherwise I could be entering 16 meetings with the same people but since they met different dates and had a different purposes they are different meetings with the same name. I would be spending the whole report reporting on meetings.

Under the planning section, I have found it too difficult to keep up with what activities are being planned and what activities are completed, so I have gone to only imputing activities that are complete unless something being planned will make an impact during the next cycle.

Under implementation, we might work a fair and hand out materials on all 4 different drugs; I do not like the fact that I have to enter this 4 times vs. just entering once and selecting each of the goals that the activity would impact.

I would like to see an area to track matching dollar amounts, in a more specific way, by actually having to list out the matches, therefore it forces coalitions to remember the match, a section where we could upload documents that are important to the coalition that help to support the story we are trying to tell. I would also like to see a way to export out the information, since in many cases it is easier to work off from excel or word and import back into the report, and it would be easier to make a report from the information entered that is not 70 pages, but more of a one page this is what we have done in 6 months document.

User286

Simply being able to upload a report would be great. Our coalition has an evaluator that provides us with a written report yearly with alot of the same information that is asked for in COMET. It is so time consuming to type this in.

User290

We appreciate the opportunity to provide feedback.

The system is repetitive, requiring the reentry of data, and it is complex. While it is organized in a sequential fashion, COMET is cumbersome and overly elaborative in the various categories. A simplified version would be helpful, as the report required an inordinate amount of time to enter. The drop down menu does not full fully allign with the llanguage of the 7 strategies, making it difficult to enter information. We printed our COMET report as there is no tracking mechanism for strategies and timelines. It would be helpful if a graphical report could be generated from the data.

A positive - the system wasn't operationally "sluggish" as many on-line reporting systems can be.

We would greatly appreciate any streamling of the system.

User287

Here are a few suggestions for improvement:

-TRAINING: improve training on what to input in this report. DITIC training was fairly helpful, but they could not speak to any content-related matters. Include this in DFC New Grantee Training in DC- make it mandatory that ALL coalition staff (not just Director) recieve this training and be familiar with how to use the COMET reporting system. Our coalition staff has gone through a lot of turnover, and it has been difficult to learn without much guidance.

-REMINDERS & NOTIFICATIONS: make it easier for DFC grantees to find links & reports in a timely fashion. I could not access this link without having to request it from someone at ICF, even though we had been reminded several times WITHOUT any link in those emails.

-REPORTING TIMELINE: Create a structure where grantees can enter information on a monthly or bi-monthly basis so that it is not as overwhelming to enter every 6 months. We have to do this with our SPF-SIG report, and we have found monthly reporting to be much less daunting than completing a huge report every 6 months. If it's an issue for GPOs, perhaps COMET could compile our monthly reports into 6-month reports for the GPOs review.

-GENERAL FORMAT: I agree with other comments- make this a more PRACTICAL & ACCESSIBLE tool so we can share our progress results with coalition members and the community, many of whom get overwhelmed by highly technical reporting terms and measures. Make it more user-friendly and less technocratic.

-CAPACITY: re-design "Coalition Membership" section to include "members-at-large", "community partners" and other forms of capacity besides coalition members with an MOU in hand.

Thank you for soliciting feedback. I hope that our comments will be taken into consideration. Perhaps there could be a way to solicit feedback immediately following COMET reporting, as others have commented? That way you could have instantaneous feedback right after it is fresh in the grantee's mind.

User293

Compared to other web based reporting systems we use, COMET is relatively easy to use but has unnecessary redundancies which make it more time consuming than it should be. If there was a way to separate the quarterly report from the years of data already entered and reported on, it would be more user friendly. E.G. when doing a quarterly report, you have to go through all entries dating back 5 or more years; also, former coalition members cannot be removed. If there was a way for the years old data to be stored and retrievable for our review but removed from the quarterly report, I think that allow for a cleaner and more succinct report.

User291

The system is very time consuming. You cannot remove strategies or objectives that no longer are employed. Frustrating navigating back and forth through sections. Data submission needs to be streamlined. The first time through the system is extremely frustrating - tutorial or not. If the current system is kept, reports should be required less often. Used customer support for access a couple of times- they were helpful. It is the navigation, redundancy and utility of the reports that are the problem. Frankly, a written report based on activity related to stated goals would be easier.

User288

For a coalition that has been entering data, since COMET began I find it very bulky and as a result less useful than it should be. So many areas include info from our first five years and our second, combined - which makes the section huge and unmanageable. For example, our objectives from the first five years are listed in addition to our second five years. In each section, we should be able to archive info. Another user commented that we should be able to delete coalition members - if we could just archive them and place them on a hidden screen that would help.

There are also sections that were changed some time ago and the identifying information was made into general choices. I am sure this is helpful for our federal officers but for the person who fills out the report it is not. I have to click on each entry to see if updates are needed - takes a great deal of time. One helpful tip - I download all the sections as PDF's and open them along with the matching section in the report as a split screen on my computer. I can zip back and forth; see what I have entered in the past, etc. This works really well and saves time.

In the early days of COMET, we did not really know what we were doing. As a result, we have information that was placed in the wrong section. Unfortunately, it is locked and even our federal officer is unable to move it. It seems that they need the ability to help us make this a useful tool and should not have their hands tied.

Over the years we have been required to do written reports that are duplicates of information included in the COMET report. It is frustrating and while this has gotten better over the years, it still happens on occasion.

User289

Ive read many of the other comments and my suggestions are going to be much the same as what I have read- I hate to be redundant but in order to drive the point home I will go ahead and list them again.

This is a very time consuming redundant process that I dread twice a year!

1. I do not like having to repeat my entry's when activities fall into more than one of the SPF process areas. It would be nice to be able to enter the informaton about our strategies and then have a drop down for the SPF process it falls into and be able to select mroe than one.

2. The drop down portion is not user friendly and has fields that really do not correlate with the work we are doing and are unable to be changed.

3. Challenges/Barriers/ and assistance should only have to be entered once- not in EVERY section.

4. the system need to allow us to tell more of a story about the work of the coalition. Allow links to blog reels and websites where information is updated on a continuous basis. Allow uploading of newsletters, news articles, photos that help paint the picture in the coalition.

5. Graphing and utilizing the data that we enter for reporting purposes would be very nice. Now we have to use two seperate systems so that our coalition can accomplish this. It would be nice to enter it all into one location and be able to print quarterly, semiannual and year end reports for our board members, the community, coalition members, fiscal agents and funders.

6. I would like to be able to remove coalition members. People move on and having them in the list when they are no longer a part of our coalition seems as though we are providing incorrect data. Also the way that the members are entered is a bit confusing as you enter the company/agency and the number of representatives. I would prefer to enter the name of the member, the agency/company/organization they belong to and then the sector(and make it so you can choose more than one sector).

7. Also when entering strategies that are ongoing it is difficult to update and add to the strategy. For example if I am working on a policy change and I have a lot of small activities that happen to work toward that policy change during one reporting period but dont actually change the policy. It is tough to go back in the next reporting period and add to that strategy with additional activites to show the updated things that have been completed. AND on the same note if a yearly activity is completed for that year and you mark it complete you can not go back in the next year and make it active again and add to it for the next year.

Thank you for your work on this project and taking the time to listen to grantees needs I hope I have been able to provide some insight into the things that would make the system more user friendly and useable for our coalition.

User294

I attended the CADCA Mid Year and participated in the COMET focus group and have read the on line comments and agree with what coalitions are saying. It is vital to gather data to support the value of coalitions so they continue to receive the funds they need to do the work.. A good tool captures useful information. Perhaps if we understood how the information is used, it would help everyone do a better job at reporting. Old information needs to be archived. We need to be able to update the membership and able to tell the real story. I am constantly amazed at how much our coalition achieves with such limited money and I don't think we are alone. Coalitions are powerful. Help us help you. THANKS

User295

Thank you for the opportunity to express my opinion of the comet system.

Being a fairly new member of our coalition I am not that familiar with this data collection system, but I know it is used by my director and our evaluator. I feel that it is very useful and important to our reporting process for all grantees. I would love to see more opportunities for training for coordinators and other assistants. This training would prove to be very productive in our ability to report all essential data in a timely manner.

Thanks and have a great day.

User299

It's a very repetitive process. You put in your meetings, which you might have to post more than once because they cover more than one issue area. It would be nice to be able to put in one event or meeting or activity, and then check off all the sections that apply to that meeting.

User304

I suggest more open questions to provide opinions and describe strategies and activities. This information is important to complement the objective questions of the instrument. The core measures collection data section is usefull.

User303

I have wanted to comment on this system for a long time. Thanks for the opportunity. When I started reading other's comments I ran across one that is so much what I wanted to say that I just copied and pasted. I hope you don't mind the redundancy, but that is how I feel each time I fill out the Comet report.

These are not my words - give credit to #289 - but I agree ---

"This is a very time consuming redundant process that I dread twice a year!

2. The drop down portion is not user friendly and has fields that really do not correlate with the work we are doing and are unable to be changed.

3. Challenges/Barriers/ and assistance should only have to be entered once- not in EVERY section.

User301

I really don't think I can add anything considering that all of the issues I have with COMET have been addressed by others.... I heartily agree with -

COMET is cumbersome, overly time consuming and redundant

Being able to remove (or archive) previous members who are no longer involved

I like the idea of more classifications for people involved with the coalition, members, members at large, community partners, etc

I totally agree that there needs to be more and better training AND it needs to happen BEFORE a grantee is expected to complete the report WITH more clarification of what information is expected in each of the categories

Archiving old data and being able to change strategies as progress moves on

Being able to upload documents to show what the coalition is doing

Thanks for asking! :)

User307

I would like to be able to upload Comprehensive Plans and other data collected not just 4 Core Measures. I would like to be able to be able to use Comet as a tool with the coalition. As it is now it is too long and non user friendly. I would not give to the entire board at this point each time to read because I do not see them reading it! They do not have the time as volunteers. I would also like to be able to print graphs and pictorials from our data inputed in Comet :)

I appreciate you asking-I also agree with ALL of the comments already addressed!!

User306

My issues with COMET have already been stated: I find it to be very redundant and I wish that we could change our strategies as we progress. I like the 4 core measures section in evaluation. I would also prefer it listed activities and such in chronological order. I do appreciate it's descriptive nature, rather than just entering codes.

User308

Don't like the repetitive process. The worst part for me is where we place our activities under each goal. There has to be an easier way to do this. I feel like I do not do that section justice because of the repetitive and difficult process to insert each activity.

User302

Although this was my first experience using Comet, I too feel that it seemed somewhat repetitive. Also, I wondered if it was really an accurate representation of all that we were doing as a coalition.

I also did not care for how we had to identify coalition members. I found myself identifying them as an organization more often then not.

User90

I have enjoyed going back and reading others comments. It has been a learning experience. Can you share what you are going to do with the information?

User118

Like- That you are able to have a site that you can compile information to guage your progress.

Dislike- COMET site is not compatible with my MAC, maybe it is my internet program, not sure (Safari)

Some of the categories don’t really seem like they cover what we are entering (events, programs, etc), you just have to choose the one that is closest. I don’t like that.

If we are to focus on Environmental Strategies, why is the reporting system more specific to them.

User309

I also agree with #289's comments. It often feels as though so much of what our coalition is about is not being captrued in the COMET report. Thanks for the opportunity to share feedback.

User305

This is my third year of completing COMET. Each year, as I'm working on it, I wonder the same thing-does it really reflect all of the hard work & effort that our coalition puts into our community outreach?

I feel like it's very repetitive-I end up typing in same info in several places in order to cover all required areas & answer all questions thoroughly. There seems like there should be a better-less tedious way to accomplish this goal!

I don't mind putting the time & effort into doing a thorough report if indeed it is reflective of the accomplishments of our coalition . It would just be nice to find a way to do that & not need to enter the same info in so many separate areas of the report.

The design of COMET also makessetting up a reporting format for the coalition difficult as well. We would like to have a standardized report format for coalition activities that gives us the best possible data for daily functioning purposes, as well as for DFC reporting (COMET). The cumbersome & repetitive nature of COMET makes this task more difficult than it probably should be to meet both needs. We have developed a format that's workable for both purposes but is also more cumbersome than I think it should need to be because we have to incorporate the COMET requirements in a feasible way.We use a Survey Monkey activity report format that we designed to track the required data-wondewr if something similar woul;d work as an overall report format for COMET-much more user friendly anyway.

There are areas of COMET that are somewhat confusing as well. You have to take extra time to sit & read the wording carefully to be sure you are answering the question correctly-if not sure, you end up inoputting even more info than needed to be sure you have it covered in every possible way.

I do believe that valuable information has been retrieved from COMEt to support the concept that grassroot, community level prevention works!!! I would just like to see the report be easier to enter needed data & be more reflective of our individual coalition accomplishments.

User298

I agree with most of the others who have commented. COMET is cumbersome and very repetitive.

I enter the same information as much as three to four times in a report under the different headins. Why can this information not be entered once and be populated via drop downs in other areas when appropriate. The process for entering would be greatly expediated and less time consuming.

The information needs to be grouped by reporting periods or at least fiscal year and then archived with the information being available to print or review as needed. A report function with drop downs would also be helpful. We also need to be able to print the information that we enterbefore submitting in report format to share with our Coalitions and others as appropriate

Coalition members who are no longer active need to be shown only during the period when they were active and only active members shown during a specific reporting period. A report feature to list all active and past members could be a useful tool.

User310

User312

Usually I enjoy doing a progress report. I love drilling down to the details of what is going on with the work and always find much to celebrate. I am a huge champion of continuous improvement processes, and I discovering the ways we can improve our work -- and then seeing how we accomplish those changes -- is very, very satisfying.

With the COMET system, all of my brainpower went into chopping up the work into the bites that the system wanted. The story was disjointed and there were pieces of work that never found a home in the system. Instead of having insights as I work, I felt confused and stymied.

Maybe I missed it, but the training I looked at all seemed focused on the movement of your mouse to various buttons. An overview was badly needed to help me comprehend how the pieces went together.

Even if I had understood the system more quickly, however, I still would have been dealing with pieces.

With a system like this, the system takes on more power than it should for a project--I will always be thinking how to fit something into COMET instead of putting 100% of my thought power into creating a drug free community.

User18

As a coalition leader I have completed two COMET reports and for the most part the process was straightforward and tedious. I have been the program director of the DFC for just over one year, and a lot of the reporting that I have completed was a follow up from the previous leadership.

I did find COMET helpful during my training and it brought me up to speed on the accomplishments and barriers that I would be facing in my community. Also, when coalition members or concerned community members ask for statistics, I have utilized COMET for easy access to the four indicators.

To be honest, there is probably no quick and fancy way to collect data, so with that in mind I believe that COMET is sufficient - is it ideal? No. But for the purposes of collecting data we are satisfied.

*To beat a dead horse - it would be nice to eliminate past coalition members to be able to keep an accurate head count.

User315

The are several problems I have with this system...

1. The risk factors do not include any that we are targeting based on the CTC model.

2. It is unclear what goes under coalition activities and implementation activities.

3. Challenges and Barriers shouldn't be required under each section.

4. Separating activities under objectives causes a lot of duplication of activities and much longer to enter data. Many activities deal with more than one objective.

I just think it is a bit clumsy as a database system.

User316

I think that COMET falls short of truly and accurately reporting the true activities of the coalition. For one thing, some fields were set by us years ago, and we are unable to change them as options to accurately portray our new directives. In other words, many activities are innacurately documented. The system is often very slow.

User297

In my experience using COMET, i feel that the system should be more useful in between reporting periods. it should be set more as a database to track and record everything that has been submitted on our workscopes. There should also be an area where we can include monthly meeting minutes and agendas. There should also be a way that we can recieve technical assistance and have more communication between other coalition coordinators to share ideas and advice on how to continue with our efforts.

User317

I am the coalition coordinator and input the data for COMET. My frustration begins when I reach the Risk/Protective Factor screens and it does not allow you to delete/remove past factors the coalition is no longer looking at. Even if we could put 'Inactive' and bump it down to the bottom of the screen, it would be helpful.

I agree with others about removing coalition members who are no longer active. Perhaps a running count can be made or persons can be marked inactive and placed at bottom of list.

I have a HUGE difficulty in the Planning & Implementation sections, as the strategies are NOT appropriate and do not allow you to change them. They are NOT the 7 strategies we continually are trained in. Being a coalition that was around prior to the installation of these strategies, COMET system has not allowed updating of strategies as we learn, grow, educate our coalitions. Frustration builds each time I do this section. I do not feel I report properly as the strategy is not appropriate for our activities.

I do appreciate the accomplishment/challenges portion in each section. It helps define our work for the period.

In the Implementation section I feel limited with the 'Activity Type' choices. I figure a way to get our coalition's work in there, but some times feel it does not fully reflect our efforts.

My last comment is about the print out. It does not reflect the screens we use to input the data. I can accept most of this, but it again takes extra time to interpret the report from what I entered to see if it truly shows what we are doing.

This is a challenge, as others have said, but I hope my comments make some sense to reduce the frustration of completing this report. Thanks for listening.

User311

I have entered data into the COMET system twice now and found it to be very confusing and frustrating. Although the COMET reporting system is new to me, I have had much experience working with different grant reporting systems in the 8 years I worked in this field. I think most of my frustration stems from not being able to delete/change previously entered data. Being the second Coordinator on board, I noticed that much of the information had been entered incorrectly and I had no way of changing it. Also, having attended the National Coalition Academy, I anticipate that our goals and objectives will be changing again in the near future. Your strategic plan should be a fluid document that changes as the needs of the community change and this system does not allow for that. The COMET system is also very repetitive and users are forced to pigeon hole their activities into very specific categories. Lastly, I feel this forum would have been more effective if offered immediately after the COMET reporting period.

User319

We have been using COMET for several years and absolutely hate it! It does not exemplify what we have accomplished because it is looking for formulais answers. We need more input that resembles what we do. There is no way to delete past projects that may no longer be valid and not needed. It is way too complicated.

User318

The COMET takes an excessive amount of time away other activities that are much more important. It took us 18 hours to complete the COMET. That's intirely too long. There must be a better way. Also, I think you should have asked for feedback the day after reporting on the COMET. I think more of us could have made specific recommendations on improving the data collection system.

User315

The Norristown Area CTC for Youth is in the 6th year of receiving DFC funding and when we applied for the Cohort 6, our objectives were complete different than the first time we applied. The system would be better if we were able to remove old objectives or start a new with Cohort 6.

User314

Pros:

Keeps us organized and places a data collection, check and balances system with our Coalition and its members. Bi-annually the entire aspect of the coalition is reviewed and discussed for the purpose of the data entry process and follow-up, in advance of our annual strategic planning meetings in May.

Reminds me how thankful I am for the laptop and technology affiliated with this grant, because this data collection tool while challenging, does stir thought into planning, reporting, and marketing the information to the community it serves. The technology eases the complexity of producing the outcome reports.

The programmers changes in the past to eleviate the "down time" of the site, or the frequency of dark time during reporting periods. Understandably the site system gets maxed with the demand to log in and remain submitting data primarily around the 2 reporting deadlines, but being a 9 year grantee, the history of lost data entry in a moment's blink has been overcome, thanks.

Cons:

When each section opens for data collection, would you please have the cursor mark at the top of the page for the entry to begin. This simple change would reduce the hardship of having to scroll down the page to locate the entry section for this year's report. For example: in the section where we state activites occuring, the page has to be scrolled down to find the entry number (1.8) and then once you add that data and press enter. You have to repeat this effort again for the next activity entered addressing the objective. Simply inverting this aspect of the system would minimize immense, repeated scrolling down by long term grantees.

Archive data should be tabbed with an archive section under each data collection section, as if scrolling thru your files on your computer. There are ample times when it is desirable to look back to the previous year's report and check where you located an activity, or what risk & protective factors were selected and where you stand with that in this report. However the data currenlty being entered should be the only info on the screens that you look at for this fiscal year's report. Long term grantees end up having data that is immense with membership names and acticites/accomplishments.

The challenge of assimilating the Coalition's achievements into the appropriate classification, and not ending up repeating the entries in two sections (to be sure it was located where the tool needed it to be reported and the coalition felt it belonged located).

The challenge of submitting a survey tool for approval for coalition data collection, then to find out the name of the survey tool was NOT included in the drop down list of approved surveys. When the coalition had implemented the tool, received approval, gathered the data and frustrated not being able to see the thorough follow-up of adding it to the list of approved measurement tools.

Minimal awareness of the progress of the data sorting from the foundational risk & protective factors selected in the early data entries. The examples of categories is an asset to newer coalitions, please keep that section (in the drop down menu). However when trying to classify a category for the target population served, or what sector of the comunity, or what category the outcomes falls under, it gets challenging. The COMET Training was very weak in explaining this aspect of the data collection flow chart format in our initial training.

Membership of the coalition, this data needs to only include the current fiscal year members affiliated with the coalition. When reporting who is responsible for the activity or outcome being achieved, there has to be a better way to include the members of the committees. Currently you must scroll down through all the names and press keys to highlight members involved and keep your finger on one key all the time, while also trying to scroll through a large membership list. Remember too this has to be done for each activity reported upon for the coalition, which becomes a hinderence to the data entry person. This section alone has to have a high percentage of error in reporting due to the complexity of the data entry tasking to keep so many names highlighted as well as be accurate when reporting. Is the goal to generate a count for the involvement of the overall coalition, or a specific count for each activity's involvement to be connected to the overall % of effort the coalition places upon this objective?

Reporting the % of coalition effort towards achieving a specific goal's objective has been to vaguely classified to generate a true perspective of the coalition's focus of overall effort. At the end of the report this is another region of higher probability for error when defining this response. I am not sure how one would correct this section, because I am not sure of the focus for this data collection. Are you determining % of effort measured toward outcomes achieved, or measuring coalition's primary focus of effort of prevention activities, or attempting to measure coalition sector's involvement in overall accomplishments of the coalitions activities (based upon the grouping of members affiliated with each activity)? Smaller scale % quanities would be the quick fix to the section with 5% increment levels or 10%, but I am unaware of the goal of this data being collected to convey a message.

Finally, highlight/create a tab that clearly states final data submission. So applicants know they have submitted their report to the site. I had this happen once, where it was thought to have been submitted when in fact it hadn't been and was sitting idle in the registry system. Plus the instructions that supplement this section help ease the concerns of new grantees, reminding them they would have 30 days to make corrections as needed by their Project Officer. This is more reassuring to the data tabulator at the end of a long, tedious process that is very important to complete accurately and timely.

User320

The reporting on Comet does not accurately reflect the work we do, the accomplishments we make and the outcomes we have. It seems that if we could report under the format of a logic model, it would simplify things greatly and be more clear and specific to our progress.

User321

The COMET reporting system as already stated is too long and repetitive. It does not allow you to elaborate on all accomplishments. Also the drop-down sections are tedious and difficult to manuver.

User322

We are completing out tenth year of DFC funding and have used the system since it began. It is my belief that newly funded communities coming in to the system would benefit greatly from understanding the utility of the system before they start and then can make it work for them as a tracking and reporting system. We came in mid strem so we did a lot of, "make it fit" stuff. Syastem asks for too much specific information.....no place to tell your story or keep a running dialog of Coalition or community activities such that end of year or grant you can have a real sense or picture of the work that ahs been done. Our story gets so complex and has so many layers that it is difficult to convey the leevl of work being done in the community because you do not have the time to do so each time you make a report. Would love to sit on a task force or committee for a couple of days just to build a new system. It is much more then we had but can do a much better job of telling the story of engaging communities in creating changes around community issues.

User323

Quoted: -

It is often difficult to tell if entries have already been made due to the long lists that are developed throughout the years. If a program is still active, it is easy to forget that it has already been entered if it was iniatially entered years ago, but is still occuring.

It is hard to track projects throughout all of their stages with the current system. The format itself is very difficult to follow.

If something that was previously entered as a challenge or barrier has turned into a success, it is difficult to know how to record it so that it does not erase the original information, but stil shows the link from the challenge to the success. I feel as though some challenges/barriers are left hanging out there when really they have been fixed, but the link between the two is not easily seen.

The implementation activity and progress screens are very hard to navigate and read. It is not easy at all to track the activities to their later progress. Entering the information is one thing, but being able to read and interpret the information is another. Once the information is in, it becomes hard to use due to the organization and layout of the program.

User324

I gnerally have trouble when documenting in the COMET system because the options available are not programmed specifically enough for our coaltion. I find myself selecting a "best fit" instead of exactly what we have done. I would also state that the drop downs are not user friendly which makes the chore of documenting our successes very time consuming. Our coalition also pays for another expensive evaluation firm which isjust as annoyingly tedious with regard to entries.

Wht I do enjoy about the COMET system is showing the fine work we have done in the community and relaying that to our funders. I suppose my frusteration is that I can not paint the whole picture easily.

User103

I agree with almost all of the comments that have been made. The system is not intuitive, it is cumbersome with much repetition. We should be able to 'archive' members who leave and objectives that are completed or eliminated. I don't want to have to change my password so often. The site is often very sluggish, taking minutes to move from page to page.

User323

I agree with previous comments that the ability to produce printed reports that can serve as a valuable resource to staff and coalition members would be invaluable! Coalition members and staff alike could really use the boost of seeing all of their hard work written out right in front of them in a concise way. It is often difficult to keep coalition members engaged and excited about our work. If staff had a document, graphs, charts, or etc. that could show all of the things that have been done in the last year, it may give us the ability to excite more people and light a bit of a fire under folks to keep up the good prevention work!

User327

The COMET system is fairly good for the most part. Some of the areas that are frustrating are: When people move away or are not longer on the coalition I can not remove them from the list. If the coalition is supporting the start up of a new coalition there is no place to add "new coalition members" for the new group.

People who are involved in program activities/objectives, this is difficult to have to scroll down through the entire list each time to select each person. If there was a way for us to view the entire list instead of four people at a time that would help.

User328

As director of the NO WAY Coalition, I believe the COMET site to be user friendly and very well put together. I do not use the site as much as my evaluator but when we work together on reports we have no problems understanding what information is needed for COMET. I have asked my evaluator to post her concerns also.

User325

Many of my opinions regarding the COMET reporting system are very similar to many of the comments already posted. I would like to say however that my greatest disappointment with the system is it's inability to accurately reflect the work of the coalition and it's impact on the community served. It becomes extremely frustrating when you have to try and describe the activities your coalition is implementing under goals and objectives that have changed as the coalition has changed.

I will say that I appreciate the convenience of having an online system to complete reports that also saves previous information reported. This is a feature that is beneficially when dealing with staff changes.

The time it takes to complete the report seems too long, mainly becasue it is unnecessarily tedious. For paper reports you understood the need to repeat information under each applicable section but with a computer system it seems there ought to be a way to link information that is pertinent to other sections without having to input the information over and over again.

There should also be a simpler way to review the completed report before submission. This would make proof reading the report simpler and more accurate.

User330

In general, the COMET tool is a good idea for multi-site reporting and evaluation. However, it is difficult to align activities with outcomes and evaluation reporting. I find that I have to fiddle with the cells in order to make some sort of logical "fit". But, this really doesn't reflect the actual work of our coalition. Perhaps its just the structure of the database behind the COMET tool that's the issue. Frankly, it would be easier to track progress on a spreadsheet with rows and columns!

User300

Thank you for allowing us the opportunity to give feedback on the COMET. I must say that in reading through other users' comments, I see many that mirror our experience.

Our feedback is that the system is repetitive, time consuming and tedious - it is not easy to navigate, and the fields are not user friendly. It was hard to know if I was entering information into the correct area. There is no easy way to proof the report prior to submission or to keep accurate track of what has/hasn't been entered in each field. It sounds as though this gets worse as time goes on.

Under Capacity: Drop down menus did not always contain the right "fit" for data.

We wish there was a way to use the data contained in the COMET report as a beneficial reporting tool to our Coalition and Steering Committee (graphs, charts, etc). Presently the system is not set up in that manner.

Like most new users of the system (May was my first submission w/ COMET), it would have been more helpful for me to complete feedback on the system in May or June while it was "fresh" in my memory.

It is our sincere hope that before you change the COMET system, you will allow coalitions the opportunity to beta test it. Thanks.

User333

Quoted: -

There should also be a simpler way to review the completed report before submission. This would make proof reading the report simpler and more accurate.

I would agree with this post in that it seems that there is duplication in the data entry requirements that become time consuming and confusing at times.

User331

Good:

The fact that we can record and report activities of Drug Free Communities in a quantitative measure.
The report will reflect the progress made toward the goals at hand.
The reporting process ensures accountability toward working DFC Goals within each coalition.

Better:

The redundancy of entries without truly being able to share the "story" - often we choose the best fit instead.

How:

Create a reporting opportunity that would allow qualitative entries to complement quantitative entries to more accurately reflect the story of each coalition instead of going through the motion in reporting and documentation.

User333

Quoted: -

Thank you for allowing us the opportunity to give feedback on the COMET. I must say that in reading through other users' comments, I see many that mirror our experience.

Under Capacity: Drop down menus did not always contain the right "fit" for data.

Like most new users of the system (May was my first submission w/ COMET), it would have been more helpful for me to complete feedback on the system in May or June while it was "fresh" in my memory.

It is our sincere hope that before you change the COMET system, you will allow coalitions the opportunity to beta test it. Thanks.

This post best describes our experience and concerns with the system. I would very much appreciate a report that could be easily read by our Coalition Members to keep them informed as well and involved in the work of the Coalition at a glance.

Thank you for giving us this opportunity. We too want the information to be useful to our Grantor's and try to do our best to fit our specific info in the drop down selections. I have not taken advantage of any of the reports thus far and will try them out.

Thank you for your support!

User333

Quoted: -

Ditto

User329

As a year 6 grantee, COMET has seen a lot of entries from our coalition and its currently in an unmanageable state. Since the inception of the grant, the nature of the work has changed from being program driven to environmental strategies focused. Even though we have changed the way we do business, we have no ability to archive anything from previous years. Essentially we are left with a tool that has become more and more unruly as time has gone on. Everytime a COMET report is due it takes hours to weed through pages and pages of irrelevant information. Archiving is essential...not just for strategies but for coalition membership as well. It would be great to be able to create a list that accurately reflects the coalition's current membership. It would also be great to have a more concrete framework. Over the years, many people (both past and present coalition staff) have made entries in COMET and since so much of it is open to interpretation and there are few parameters people reported on anything and everything for the sake of capturing something. We are now stuck with hundreds of pages information that does not reflect the work of the coalition as it relates to the DFC goals which only adds to confusion when new staff people attempt to make entries on current coalition activity.

User333

Quoted: -

I find this to be true. I would be more apt at commenting if it was soon after the last report dead line. Reading all of this makes me feel anxious to get back into the system!

User337

A positive aspect about COMET is the cumulative nature of reporting.

Negative aspects are redundancy among the SPF sub categories.

In general, I find the current framework functional and better than average.

User339

COMET is a very difficult reporting tool. It takes a lot of time to complete and does not have areas to be able to show more statistical data. Even the fore core measures inputting is difficult because not everyone is using the same format. You can not change objectives and reporting on them is difficult due to the length of lists. It takes multiple people working on all of the print outs and going back through to make sure that you are reporting on everything appropriately. Also some activities may have been added as needed by the community and may not fit into the current options. COMET should be revised and made to be much more user friendly. Also Feedback would be more beneficial if it was asked of us after inputting into COMET.

User340

Some of my thoughts on COMET -

It is entirely too time consuming.
It is hard to give a clear picture of what our coalition is doing.
It is repetitive.
Can be confusing to navigate.
Some choices for answering questions are limited. I feel like I'm often not able to choose/give the best or correct answer and have to settle for the next closest thing.

User341

I think COMET provides a tool to capture the work we are doing over a 6 month period and over the grants time frame. Also, after a few times it comes somewhat more easy.

I do often times though have a hard time fitting the work we do into a specific category that COMET provides.

User338

1. Too many questions that require a narrative, many of them repetitive. Information could be gathered and analyzed more efficiently by using a multiple choice format with a box for comments where necessary. Checking off answers as opposed to writing a narrative would save time at both ends and assure more consistent responses.

2. We found that we were going back to copy previous answers into subsequent questions because they were essentially asking for the same information.

3. Mechanically, the site is clumsy and requires the responder to click through several pages to move back and forth in the survey. e.g. when needing to check a prior answer before entering a new answer. Making negotiating the site simpler would save time.

4. Tracking reported information over time is very confusing. Interventions, strategies and activities change even for the same objective and the survey makes it difficult to keep track and convey those changes in an organized way.

5. The YRBS data that was automatically entered did not conform to the actual YRBS data we received and we were not able to change it.

I agree with the person who suggested that seeking feedback right after the survey was submitted would make it easier to recall the problems and suggestions.

User343

Quoted: -

I agree wholeheartedly with this comment. After 10 years the report is so big I always feel terrified to begin, never knowing where to start. Some of the information is so old and outdated and I've only been here 3 years, so I have no idea what to do with the old stuff. Everything is in the system twice due to the change in format a couple of years ago and that is just a crazy mess. The old information should have somehow been transferred instead of having to create a whole new set of data entry points. I often have to keep going back and forth, back and forth back and forth - it is insane the amount of time I waste looking at the old stuff and then going back to the new stuff.

User343

Quoted: -

2. We found that we were going back to copy previous answers into subsequent questions because they were essentially asking for the same information.

5. The YRBS data that was automatically entered did not conform to the actual YRBS data we received and we were not able to change it.

I agree with the person who suggested that seeking feedback right after the survey was submitted would make it easier to recall the problems and suggestions.

I agree with this - I create a Word document so that I can copy and paste the same information over and over and over again. It would be so much easier if I could enter an activity and then check off the categories.

For example I could enter Resource Fairs as an activity, there could be a comments box so that I could describe what was better or worse this year, if we changed anything blah blah blah and then I could check off on a list all of the ways the fairs meet our requirements.

- capacity building
- evaluation
- planning
- implementation
- sustainabilithy

User342

We have found COMET very difficult to use. It's very hard to track progress in a consistent and organized way. So much overlaps that it becomes confusing. Also, it's hard to document environmental strategies adequately. The drop down menus are very limited and often there is not one that corresponds with our initiative. We had to ask for a written print out of the drop down menus because they were not always complete. We were told there was not enough room to put the entire explanation. Another problem we had is in reference to "retired items". When the change was made, we really didn't find a good way of documenting the "new" definitions. The report is just so very lengthy and at times confusing that we feel we can never adequately document the work that we are doing. We find it difficult when we update our Work Plan each year to have that reflected properly in the report.

User343

I hope that the system collects the information that you need, I wish it collected the information I need. I would love to be able to print reports showing the increase in number of attendees to our events, or the change in membership / participation.

With most web pages there is the option to right click and open a link in a new tab - this allows the user to click back and forth between pages without having to save everything, and wait for another page to load. COMET lacks this tool - so each time I need to go to another section I have to save where I am and change pages, close the one I am in to open another one. For example if I am talking about progress in the implementation section or I am clicking on the people involved in a project and I realize one of the important people involved hasn't yet been added to the list of coalition members, I have to save where I am, change pages to the membership page, add the new member and go back to the progress section and start ALL OVER AGAIN holding down the control key and selecting all of the people involved again.

This report has at times taken me up to 3 days to complete - and this is just the data entry. We have to go back 6 months to make sure we are including everything we worked on and who was involved. I always say I am going to do the report monthly, so it isn't all at once, but it is just too time consuming and overwhelming that I dread it and don't go near it until I absolutely have to do it. After all of that work, all of that data entry, all of that frustration going back and forth back and forth back and forth section to section we recieve almost no feedback about the work we are doing.

Finally - and most importantly - The online training shows us how to click, how to add, how to delete - basically how to use an online reporting system. It certainly does NOT give us guidelines for entering our data, rules so that we can all be consistent, hints for where resource fairs fit in or how to qualify someone as an active coalition member. We are all making up our own rules - how can you possibly be tracking our work? There are NO RULES!! No one tells us exactly how to enter things - I might count something as capacity building and someone else might count it as something else. We have to fit our activities into pre-decided categories and sometimes I go through the series of dropdowns three or four times before I can find the right combination - that can take up to 30 minutes of clicking through, waiting for pages to refresh and getting almost all the way to the end before I can't find something on the list that fits - so back I go to the start, choose a different drop down at the beginning, wait for the page to refresh, try and find a drop down choice that fits, wait for the page to refresh - I am sure you can understand what I mean and how this can be frustrating. Most of the time I can't even read the whole choice because the window isn't as long as the writing and I haven't yet figured out how I am supposed to view the whole sentence - even when I select a choice I can't read the whole thing. To be honest there are times I've chosen a drop-down choice that really doesn't fit simply because I've run out of options and nothing fits but I want to count the work I'm doing - what kind of sense does that make?

User346

I came in on this grant in the second half of year four. As the proportion of funding for programming decreased, assessments were updated, and strategies were changed to exclusively deal with environmental strategies, the original direction and structure of the COMET report is woefully out of date. Trying to configure strategies under the old parameters is like trying to fit a square peg into a round hole.

I have thought that if we used the Strategic Prevention Planning Framework it would be intuitively a much easier way to track our progress. Since coalitions are at any one time engaged in any of those categories of assessment, capacity building, planning, etc. it would be easier to detail in a more 'free-form' manner what we have initiated within each category; add or modify existing entries (and the rationale); and retire entries that are completed. The DFC grants were given to individual communities because they are each unique and strategies will differ to a lesser or greater degree accordingly. Perhaps guidlines with plenty of examples could be used to guide the user in each of the sections of the SPF.

In addition, the report would serve us well on the local level as a 'report card' of sorts, making the tracking of our progress much cleaner, featuring those unique descriptions that will translate well in telling our story not only to our own communities; but serve as a great idea generator for other communities, and a wealth of successes to share on the national level. - Instead of putting this information together at a separate time and place.

I don't mean to leave out quantifying data as well. That tells part of the story as well and is needed to verify the validity of the strategies. I think that beyond the core measures, a guideline could be created on how to tell the community story using reliable, quantifiable data appropriate for local coalition work. These numbers and results would encourage proper evaluation of our work and accompany each entry in the different sections.

User349

The COMET systems has come a long way from when I was first introducted to it at a conference in Washington DC. However, I think there are other changes that could be made. I would prefer to be able to work in each system much better and delete things that no longer a applicable. There are several sections you can't event delete old employers that are no longer here and/or a part of this grant application. Everytime I start working on it have have to become familiar with the system again. We only use it twice a year and try to work in it in between times. I would also like to be able to print out my report so I can read it before I submit it to the project officer, have other coalition members read through it too. Often times things get left out that would be important to have in the report. At this time that is all the information I can think of about the report.

User350

I wish the reporting matched up better with the seven environmental strategies, logic model and implementation plan. All the work that we do prior to the reporting on COMET is tied to those tools but it is not reflected in our COMET report as clearly as it should be-- since that is our bottom line and our measurement for success. Also, I wish it was tied closer to the output tools given to us at the CADCA Academy. Overall, it is not an overwhelming reporting process currently so please do not make the report more important than our mission, work and outcomes but as a way to hold us accountable. I like it when we can measure apples to apples so that we do not have to reinvent the wheel for each DFC Coalition and we can prove to funders and supports alike that we are moving the bar and improving substance abuse rates in our communities.

User352

Repetitive - a lot of duplication in entry

Suggestions for improvement:

drop down menus
spell check
ability to print current data entry
average age of onset calulation

User353

The site does not allow for sufficient changes as the Coalition builds and progresses. Some of the information is repetitive. The site does not provide sufficient space to enter outcomes not related to the original Coalition goals and objectives.

User351

I think designing such a universal tool for collecting data would be a challenge. However, I find many of the questions on Comet to be repetitive with information often crossing many of the categories. The drop down boxes are vague and often confusing. I don't know that it paints a true picture of activities and if actual data as in drug surveys are the true reason for collecting this data, there has to be a better more inclusive way to it. Again, I can't even imagine trying to design a system that would best reflect grantees throughout this country without flaws so I would say overall make it shorter and cleaner.

User335

My comments are similar to what has been reported:

When it’s reporting time, it is time consuming to review the number of pages that have accumulated over the years. It would benefit the reporter to have a viewer friendly format, such as a spreadsheet with rows and columns.

It is difficult to get an accurate number of coalition members since they can not be removed.
At times it is difficult to navigate between sections.
Before submission, it would benefit the reporter if there was an overview of what will be submitted.