Attachment 6
Summary of CDC Evaluation Plan
Evaluation of the SAMHSA PDMP Electronic Health Record (EHR) Integration and Interoperability Expansion (PEHRIIE) Program
Summary of CDC Evaluation Plan
Through its Prescription Drug Monitoring Program (PDMP) Electronic Health Records (EHR) Integration and Interoperability Expansion (PEHRIIE) program, the Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Substance Abuse Treatment has entered into cooperative agreements with officials in nine states with operational PDMPs in order to improve the integration of controlled substances prescribing data with EHRs and other health information technology (HIT) systems, as well as the interoperability of PDMPs between states. The stated goals of this two-year program, which started in October 2012, are to:
Increase provider utilization of their state’s PDMP by improving real-time access to PDMPs via the integration of PDMP data and/or access thereof within health information technologies (HIT);
Increase provider utilization of PDMP data by increasing the comprehensiveness and quality of PDMP data by increasing the interoperability of state PDMPs across state lines.
Both of these goals are expected to contribute to improve prescribing and dispensing practices, resulting in improvements in care and decreased prescription drug abuse and misuse and related health consequences such as fatal and non-fatal overdoses.
Under the cooperative agreements issued by SAMHSA, Centers for Disease Control and Prevention (CDC) is responsible for conducting a comprehensive process and outcomes evaluation of the above described PEHRIIE program. The purpose of this evaluation is to determine if successful completion of the two primary goals of this program resulted in changed provider behavior and impacted prescription drug related health outcomes in these states. In addition, this evaluation will identify cross-cutting barriers to and facilitators of successful implementation of the HIT integration and interstate interoperability projects funded through the PEHRIIE program. To address these evaluation questions, CDC will combine the results of a qualitative information collection with the results of a quantitative analysis of PDMP data from the PEHRIIE-funded states and health outcomes data from SAMHSA’s and CDC’s national datasets. These results will be supplemented with findings from a review of all documents relating to the PEHRIIE program, the individual state projects, and other publicly available information pertaining to the PDMPs in the PEHRIIE grantee states. States are required to participate in these evaluation activities as a condition of receiving funds from SAMHSA.
Qualitative Information Collection
The CDC evaluation team will be conducting a qualitative evaluation of the PEHRIIE program in order to understand the processes, challenges, and successes in implementing and sustaining integration of PDMP data with Health Information Technology (HIT) systems and interoperability of PDMP systems across states. This information collection will also capture the experiences of clinical end users with the systems being upgraded under the PEHRIIE program as well as their recommendations for how the goals of the PEHRIIE program could have been better accomplished. In order to complete this component of the evaluation, the CDC evaluation team will conduct qualitative interviews with individuals involved in the planning and implementation of the PEHRIIE projects (i.e. key project staff and stakeholders) as well as with the clinical end users (i.e. prescribers and pharmacists) of the PDMPs in the states where these projects are taking place. In-person interviews will be conducted during the course of a four-day in state visit to each of the nine PEHRIIE grantee states, using pilot-tested semi-structured interview guides. At least two evaluation team members will conduct each interview during the site visits, with additional note takers participating via conference call. Interviews with key project staff and stakeholders will take approximately 45 minutes and interviews with clinical end users will take approximately one hour. When in-person interviews cannot be scheduled during the planned site visits, interviews will be conducted via conference call.
Key Project Staff/Stakeholders Interviews. Based on preliminary conversations with the primary project coordinators in each of the nine PEHRIIE grantee states, the evaluation team anticipates conducting interviews with 91 key project staff members/stakeholders in each state as well as 14 individuals from vendors working with multiple PEHRIIE states. Individuals in the key project staff/stakeholder category include state officials and selected staff members involved in the operations and oversight of the state PDMP, including board officials of the agency where the PDMP is housed; other state officials involved in policy, technical, and legal support agencies as well as the state’s substance abuse authority; PDMP software and data sharing hub vendors; and HIT systems’ partners, including hospital staff responsible for EHR implementation at their site and their respective IT personnel.
The goal of these interviews is to solicit information that is not available through progress reports and reported quantitative data in order to gain qualitative perspectives on the PEHRIIE projects. To achieve this goal, these interviews will consist of four domains: 1, Background and Project Context; 2, Activities and Challenges; 3, Successes and Outcomes; and 4, Future Work and Lessons Learned.
Clinical End Users Interviews. The evaluation team will also conduct interviews with three clinical end users of the upgraded PDMP systems at identified medical practice sites where access to the PDMP or PDMP data itself will be integrated into providers’ EHRs and/or pharmacy dispensing software systems. These interviews will be conducted at an average of four implementation sites per state. Individuals in the clinical end users category include prescribers, pharmacists, and other pharmacy stakeholders.
The goal of these interviews is to capture the effectiveness of the PEHRIIE projects at improving PDMP data quality, accessibility, and usability and to better understand the impacts of these projects on provider behavior related to controlled substances prescribing and dispensing. To achieve these goals, these interviews will consist of four domains: 1, Personal Background; 2, Current Use of the PDMP; 3, Clinical Impact; and 4, Recommendations for Future Improvements.
Prior to the interviews, each interviewee will be provided with a summary of the evaluation plan and a brief description of the interview purpose, as well as a copy of the relevant interview guide. Potential participants will be given the option to refuse to participate in the interviews or to answer any question he or she does not feel comfortable answering. Interviewees will be audibly apprised of their privacy rights and asked to give vocal confirmation of their agreement to participate prior to each interview. This information will also be included in the distributed pre-interview materials.
During the interviews, both the attending evaluation team members and those participating via conference call will take extensive notes using standard note-taking sheets in order to capture consistent data. With explicit permission from the interviewees, interviews will be audiotaped to use for reference and clarification in cases where the written notes are unclear. After each interview, all participating team members will compare their notes and compile a summary for that interview. All notes, audio-recordings and hard copies will be stored following standard CDC security procedures. Files will be maintained by an assigned 12-character ID number generated from a combination of state and site identifiers as well as the date and time of each interview. All files will be kept for six months following the completion of the evaluation in case any information is needed and then destroyed thereafter.
Following the completion of all interviews pertaining to a given PEHRIIE grantee state, the evaluators will conduct a thematic analysis of each finalized interview summary as well as within and across case qualitative analysis.
Quantitative Data Collection
The CDC evaluation team will also be conducting a quantitative evaluation of the PEHRIIE program in order to understand the impact of the PEHRIIE projects on the use of the PDMP by medical providers (including prescribers and pharmacists) and on early health outcomes related to the misuse and abuse of controlled prescription drugs. Quantitative data will be provided from two sources: PDMP data from each of the PEHRIIE states and publicly available state-level health outcomes surveillance data from CDC and SAMHSA.
PDMP Data. As stipulated in the SAMHSA cooperative agreements, all grantee states will provide either deidentified PDMP data or quantitative measures derived there from upon request to the evaluation team during the program period. Deidentified PDMP data will be used to assess changes in patient, prescriber, and pharmacist behavior associated with PEHRIIE projects. In addition, the evaluation team will ask grantees to document use of the PDMP data by prescribers and pharmacists associated with the projects.
To protect the identities of the patients with records in each PDMP database, the majority of the identifying information will be removed from the records and a unique, project-specific identification number will be assigned in order to link all records associated with the same patient across the data collected for this evaluation. The state and zip code of patient residence will be retained in the data to enable examination of geographic area variation, particularly around the implementation sites. Identifying information relating to prescribers and dispensers will be treated similarly. The CDC evaluation team will also request data from each grantee state on prescriber and pharmacist registration with and use of the PDMP. As with the PDMP data, usage data will contain only project-specific ID numbers for prescribers and pharmacists, as well as their practice site zip codes. Finally, the evaluation team will request that each grantee state create a dichotomous variable indicating which prescribers and pharmacists are associated with implementation sites, and provide this variable along with the PDMP and usage data files. To ensure anonymity of prescribers and pharmacists associated with implementation sites, geographic and demographic strata with fewer than 25 prescribers or 25 pharmacists will not be used in the analysis of project effects. Grantees that choose to prepare the metrics themselves will follow the same data preparation procedures.
Quantitative data analysis will examine patterns of PDMP use as well as changes in the rates of risky patient behavior and inappropriate prescriber or pharmacy behavior. Within each grantee state, pre-/post-implementation changes in each set of measures will be compared in aggregate for implementation and non-implementation sites using a difference-in-differences approach.
PDMP and provider usage data will be requested for the two year period preceding the project implementation as early as feasible in project year 1. Follow-up data of both types will be requested at approximately the 8th month of project year 2, to allow for data preparation and data analysis by the evaluation team in preparation for its final report by the end of project year 2.
Due to the lag time in the availability of these metrics, an assessment of the changes associated with the PEHRIIE projects may not be possible until one to two years after the end of the grant period.
Document Review
As part of the cooperative agreements, grantees are required to submit proposals, progress reports, and similar project documents to both SAMHSA and the evaluation team. A systematic content analysis of these documents as well as publicly available information, such as laws relating to the grantee’s PDMP and other related PDMPs, will be conducted to capture both baseline status of the PDMP and changes made as part of this grant. Content analysis of these documents will also provide information for an across-case accounting of activities and outcomes that occurred as a result of the grant, including:
Number of grantees that had legislation regarding the PDMP at the end of the grant that they did not have before the grant program
Number of new data sharing agreements or memorandum of agreements between states
Number of new states that grantee systems are now interoperable with
Number of new integrations with HIT systems
Costs associated with development and implementation of integration and interoperability
Information about these indicators will allow the evaluation team to assess outcomes of the grant projects in relation to increasing interstate interoperability, integration with additional EHR systems, other enhancements to the PDMP, and costs associated with interoperability.
The primary results of each evaluation component will be included in summary reports for each of the nine PEHRIIE state grantees and a final comprehensive report of the overall evaluation findings, which will be disseminated to the primary stakeholders in the SAMHSA PEHRIIE cooperative agreement program. Because this is the first evaluation of PDMP integration with HIT and interstate interoperability, the findings from the evaluation will likely inform future collaborations between government agencies, PDMP advisory groups, and state stakeholders as they pursue additional work focused on improving PDMP efficacy and reducing the epidemic of prescription drug misuse, abuse, and overdose. Moreover, while the results of this evaluation are not explicitly generalizable beyond the specific projects being evaluated, it is anticipated that the findings of this work will serve as an informative guide for additional states looking to make similar upgrades to their PDMP and HIT systems. Finally, it is anticipated that the reports resulting from this evaluation will be used by the PEHRIIE grantee states in order to refine and expand the improvements made to their PDMPs and HIT systems beyond the funding period.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Cindy Parks Thomas |
| File Modified | 0000-00-00 |
| File Created | 2021-01-28 |