Evaluation and Performance Measuring Plan Template

Attachment 3a. - Evaluation and Performance Measurement Plan

Overdose Data to Action (OD2A)

Evaluation and Performance Measurement Plan

Form Approved

OMB NO: 0920-xxxx

Exp. Date: X/XX/XXXX

Public reporting burden of this collection of information is estimated to vary from 4 to 12 hours, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to CDC/Information Collection Review Office, 1600 Clifton Road, NE, MS D-74, Atlanta, GA 30333; Attn: PRA (0920-xxxx).

The purpose of Overdose Data to Action (OD2A) is to support recipients in getting high quality, complete, and timelier data on opioid prescribing and overdoses, and to use those data to inform prevention and response. As established in the OD2A Notice of Funding Opportunity (NOFO), recipients are expected to implement comprehensive and rigorous evaluations of their program activities using timely data from a variety of sources.

As outline in the OD2A NOFO, the evaluation plan submitted with your application should demonstrate:

• How key program partners or stakeholders will participate in the evaluation and performance measurement planning processes.

• Feasibility of collecting appropriate evaluation and performance data, available data sources, data management plan (DMP), and other relevant data information (e.g., performance measures proposed by the applicant).

• Describe the type of evaluations (i.e., process, outcome, or both).

• Describe key evaluation questions to be addressed by these evaluations.

• Describe other information (e.g., measures, data sources).

Applicants should also consider including action steps to address:

• How evaluation efforts will be coordinated and managed across all of their OD2A funded initiatives

• How existing data will be used in your evaluations

• How evaluation findings will be shared with and used by stakeholders to inform programmatic development and improvements throughout the cooperative agreement

The evaluation plan you submit with your application should be no longer than 10 pages.

Recipients will be required to submit a more detailed Evaluation and Performance Measurement plan, using the template that follows, six months after award. The evaluation plan will be updated annually with your Annual Progress Report. This more detailed Evaluation and Performance Measurement plan should include the DMP elements, as described in the Reporting Section of this NOFO. This template is being provided during the application process, so that applicants can understand the full scope of evaluation planning that will occur in the future and get a head start with their evaluation planning activities. Use of the template or parts of it is optional during the application process.

Overdose Data to Action

Evaluation and Performance Measurement Plan Template

[Insert Jurisdiction Name]

This template is designed to assist with the planning, execution, and monitoring of the OD2A evaluation activities. It also facilitates CDC’s technical assistance by ensuring that we offer the available and necessary support for evaluation design, data access and procurement, and facilitate connections among awardees who are engaging in similar evaluation activities. A completed version of this detailed Evaluation and Performance Measurement plan is due six months after award.

This template will guide you through reporting a detailed evaluation plan, including a logic model. Together, we will revisit this document often and use it as a resource and reference for planning. As we progress through each budget period, we will work with you to revise the template. You will resubmit revisions of this plan with the Annual Performance Report.

Definition of key terms used in this template:

Component: There are two required components in OD2A surveillance and prevention.

Strategy: There are 10 strategies in OD2A. Three surveillance strategies (1: Collect and disseminate timely emergency department (ED) data and 2: Collect and disseminate descriptions of drug overdose death circumstances are required for state recipients, D.C. and Puerto Rico, 3: Implement innovative surveillance is required for all recipients) and seven prevention strategies (Required: (4) Prescription Drug Monitoring Programs, (5) State-local Integration, (6) Linkage to Care, and (7) Providers and Health Systems Support and optional strategies: (8) Public Safety Partnerships, (9) Empowering Individuals to Make Safer Choices, and (10) Prevention Innovation Projects. (10) Prevention Innovation Projects).

Activity: There are several activities identified within each major strategy in the OD2A NOFO. These are intended to be illustrative rather than prescriptive, and therefore are not comprehensive. Proposed activities are subject to CDC approval, but any that align with the spirit and scope of the major strategy and fit within the logic model are likely to be approved and should be included in the evaluation plan.

Sub-Activity: There are sub-activities included throughout OD2A NOFO. Applicants can choose from the recommended sub-activities or proposed their own. A detailed description of the sub-activities should be provided by each applicant.

Evaluation Question: Evaluation questions can be a mixture of both process and/or outcome focused. Evaluation questions describe exactly what you are evaluating. Example evaluation questions are provided for the “Prescription Drug Monitoring Program” activity: (1) To what extent were barriers to PDMP registration and use removed? (2) To what extent were PDMP registration and use percentages changed?

Outcome: Describe general benefits related to changes in behavior, skills, knowledge, attitudes, values, condition, status or other attributes. An outcome example is: improved PDMP registration.

Evaluation Use: In this portion of the template please describe how evaluation findings will be used. The evolving nature of drug overdose prevention work requires that evaluators remain vigilant to the strategic and on-going needs of their stakeholders. Evaluators should look for ways to feedback actionable information to program implementers and decision makers in a timely and actionable manner^[2].

Indicator: Measure achievement; basically indicators are operationalized outcomes; specify how the outcome will be measured. An indicator example for the outcome “improved PDMP registration” is percentage change of prescribers registered with PDMP (from baseline).

Data Source: List the source of your data. For example, if it is your PDMP, then list the name of your PDMP. If another existing data system is being used for health outcomes, list the name of that system. If you are collecting new data, then describe the source of that data (e.g., PDMP delegates or prescribers).

Data Collection Method: In this portion of the template, we would like for you to tell us if the data already exist or if the data need to be collected. If you have baseline line to track progress. How the data are new and need to be collected, describe how the data will be collected. For example, survey, key informant interview, document review, etc. And the frequency of the data collection—for example, monthly, quarterly, twice per year, annually, etc.

Timeline: In this column, describe the timeline line (Quarter/Year) for data collection and analysis.

Overdose Data to Action (OD2A) Logic Model
Strategies and Activities	Short-term Outputs/Outcomes	Intermediate Outcomes	Long-Term Outcomes
COMPONENT 1: SURVEILLANCE
Strategy 1: Collect and disseminate timely emergency department (ED) data on suspected all drug, all opioid, heroin, and stimulant overdoses (required for SHDs, Puerto Rico, and DC)	Strategy 1-3 Timely and actionable surveillance data disseminated by recipients: To enhance the implementation of their NOFO interventions To recipients’ stakeholders working to reduce drug overdoses in their jurisdiction To CDC to rapidly inform the public and key regional and national stakeholders	Strategy 1-3 NOFO surveillance data contributed to improvements in drug overdose interventions	Decreased rate of opioid misuse and opioid use disorder Increased provision of evidence-based treatment for opioid use disorder Decreased rate of ED visits due to misuse or opioid use disorder Decreased drug overdose death rate, including prescription and illicit opioid overdose death rates.
Strategy 2: Collect and disseminate descriptions of drug overdose death circumstances using death certificates and medical examiner / coroner data (required for SHDs, Puerto Rico and DC)
Strategy 3: Implement innovative surveillance to support NOFO interventions (required)
COMPONENT 2: PREVENTION
Strategy 4: Prescription Drug Monitoring Programs (required) Universal use among providers within a state Inclusion of more timely or real-time data contained within a PDMP Actively managing the PDMP in part by sending proactive (or unsolicited) reports to providers to inform prescribing Ensuring that PDMPs are easy to use and access by providers Intrastate and interstate interoperability	Strategy 4 Increased measurable collaboration and communication Increased application of data to drive prevention and response activities between state and local efforts Increased access for state health departments to multiple data sources (data dashboards, etc.)	Strategy 4 Increased use of PDMP by providers and pharmacists Identification of high risk prescribing and patient behaviors Better tracking of opioid prescriptions Decrease in high risk prescribing behaviors
Strategy 5: Integration of State and Local Prevention and Response Efforts (required) Explicit efforts to better integrate state and local prevention efforts Capacity building for more effective and sustainable integrated surveillance, prevention, and response efforts Prevention and response strategies at the state and local level	Strategy 5: Increased local and state capacity for sustainable surveillance and prevention efforts Increased understanding of context, resources, and needs in city/county/state Increased understanding of evidence-based, scalable response approaches Increased focus on highest risk groups	Strategy 5: Greater awareness of drug and opioid overdose epidemic by state health departments, with respect to burden and resources, including at the city/county level. Increased state involvement in local-level prevention efforts Increased preparedness and response at the local level.
Strategy 6: Establishing Linkages to Care (required) Identify systems-level strategies in healthcare (e.g., emergency departments, outpatient settings), community programs (e.g., comprehensive syringe services programs) and public safety and courts (e.g., police, emergency response, diversion programs) to support care linkages with improved awareness, coordination, and technology	Strategy 6: Increased awareness and coordination of linkages to care	Strategy 6: Increased referrals to and engagement in evidence-based treatment
Strategy 7: Providers and Health Support Systems Support (required) Clinical Education and Training based on evidence-based guidelines (e.g. CDC guideline) Insurer and health systems supports	Strategy 7: Provider, health system, and payer awareness of and supports for guideline-concordant opioid prescribing, non-opioid medications, and non-pharmacological treatments	Strategy 7: Increased use of non-opioid medications and non-pharmacologic treatments for pain by patients Decrease in high risk opioid prescribing

Overdose Data to Action (OD2A) Logic Model (Continued)
Strategies and Activities	Short-term Outputs/Outcomes	Intermediate Outcomes	Long-Term Outcomes
Strategy 8: Partnerships with Public Safety and First Responders (optional) Data sharing across public health and public safety partners Programmatic collaborations to share and leverage prevention and response resources	Strategy 8: Improved coordination of Public Health and Public Safety efforts Use of shared data to inform collaborative public health/public safety prevention and response activities Greater jurisdictional awareness of opioid overdose epidemic and evidence-based approaches by public safety and first responder partners Increased opportunities/processes to link individuals to care Increased use of pre-arrest and pre-trial diversion type programs to address opioid-related behaviors	Strategy 8: Improved utilization of evidence-based approaches to prevention, intervention and referral to treatment	Decreased rate of opioid misuse and opioid use disorder Increased provision of evidence-based treatment for opioid use disorder Decreased rate of ED visits due to misuse or opioid use disorder Decreased drug overdose death rate, including prescription and illicit opioid overdose death rates
Strategy 9: Empowering individuals to make safer choices (optional) Awareness and education informed by media campaigns, translational research for public consumption, and appropriate messaging and resources	Strategy 9: Awareness of the risks of prescription and illicit opioids Awareness of non-opioid medications and non-pharmacologic treatments among prescribers and other clinical care partners	Strategy 9: Decreased initiation of opioid use and misuse Increased fidelity to opioid prescription/medication protocol Increased use of non-opioid medications and non-pharmacologic treatments among patients
Strategy 10: Prevention Innovation Projects (optional) Projects that allow states to respond to emerging threats and to promote an innovative prevention approaches and practices	Strategy 10: Prevention Innovation Projects Improved flexibility to respond to changing conditions within the jurisdiction. Promotes the development of novel prevention strategies	Strategy 10: Prevention Innovation Projects Expanded opioid prevention activities Improved jurisdictional responsiveness

Section 1: Logic Model for Overdose Data to Action (OD2A) Surveillance and Prevention Components

Please include a logic model for your three year OD2A Surveillance and Prevention Components. Please note that this logic model is specific to your selected strategies and activities and the outcomes you are tracking. The logic model should complement your work plan.

Section 2: Evaluation Plan for Overdose Data to Action (OD2A) Surveillance and Prevention Components

Surveillance Component 1

Strategies 1-3

Data Monitoring and Quality Assurance: Describe how you will constantly monitor and improve the quality of data reported to CDC.

Use and Utility of Surveillance Data

Describe how you will ensure the use of surveillance data to inform OD2A programs.

Dissemination and Impact of Surveillance Data

Describe how you will track the dissemination and impact of surveillance data.

Note: Copy and paste additional tables if more than one strategy is being evaluated/tracked.

Prevention Component 2
Strategy _#_(Please indicate which prevention strategy 4-10 is being addressed in this section of the evaluation template)
Activity (If an activity is provided in the OD2A NOFO please use that name, otherwise name and describe here.)
Description of the sub-activity/activities (Please describe the specific sub-activity being evaluated. The description should be short (300 words or less) and should relate to the description provided in your work plan)
Outcome(s) Please note the outcome(s) being address from the Overdose Data to Action Logic Model on page 5-6 of this template.
Evaluation Use Describe who will use the evaluation information and how? (if more than one stakeholder list separately) Describe how and when evaluation findings will be disseminated for each stakeholder (e.g., during monthly meeting updates, via quarterly/annual reports).
Evaluation Questions  Please include one or more evaluation questions that you aim to address in your evaluation.
Indicators	Data collection methods Mark one: _ Data exist _ Data need to be collected (new data) Baseline data: __ Known __ Unknown/TBD If data are new, how collected: Frequency of data collection:	Timeline for data collection and analysis

Note: Copy and paste additional tables if more than one sub-activity is being evaluated/tracked per strategy.

[2]^[2] Evaluation approaches like developmental evaluation or Rapid Feedback Evaluations maybe helpful models for evaluators to use while working on overdose prevention efforts.

10