SUPPORTING STATEMENT
Part A
VALIDATION PILOT FOR THE AHRQ PATIENT SAFETY INDICATORS
PHASE II
Version June 19, 2008
Agency for Healthcare Research and Quality (AHRQ)
Table of contents
A. Justification 3
1. Circumstances that make the collection of information necessary 3
2. Purpose and use of information 6
3. Use of Improved Information Technology 10
4. Efforts to Identify Duplication 10
5. Involvement of Small Entities 11
6. Consequences if Information Collected Less Frequently 11
7. Special Circumstances 12
8. Consultation outside the Agency 12
9. Payments/Gifts to Respondents 12
10. Assurance of Confidentiality 12
11. Questions of a Sensitive Nature 13
12. Estimates of Annualized Burden Hours and Costs 13
13. Estimates of Annualized Respondent Capital and Maintenance Costs 16
14. Estimates of Annualized Cost to the Government 16
15. Changes in Hour Burden 16
16. Time Schedule, Publication and Analysis Plans 16
17. Exemption for Display of Expiration Date 16
Attachments 17
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see Attachment A), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. research that develops and presents scientific evidence regarding all aspects of health care; and
2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
As part of its patient safety initiative, AHRQ is interested in the development of measures of medical errors to document existing conditions at the start of patient safety improvement initiatives, and to monitor improvements over time. The AHRQ Patient Safety Indicators (AHRQ PSI) were made publicly available in March, 2003 as an evidence-based tool for identifying potentially preventable adverse events based on readily available hospital discharge data (“administrative data”). Continued refinement of the AHRQ PSI requires validation of the cases “flagged” in the administrative data through the collection of data from the medical record.
In Phase I of the validation pilot, the purpose was to test medical record abstraction instruments, sampling methods, and data collection applications, in addition to testing a mechanism for conducting the validation through a voluntary collaborative of participating hospitals. The intent was to determine the feasibility of incorporating a validation component into the ongoing development process of the AHRQ PSI. Phase I of the validation pilot provided information on 1) the extent to which organizations were willing to participate on a volunteer basis; 2) the most effective protocols for deploying the data abstraction, sampling and data collection tools to these different organizations; and 3) whether the data collected would be sufficient to achieve the study aims. The Phase I study aims were narrowly focused on characterizing the cases flagged from the administrative data using data from the medical record for selected Patient Safety Indictors that were relatively more common and less clinically complex1.
The purpose and intent of Phase II is the same as Phase I: provide information on 1) the extent to which organizations were willing to participate on a volunteer basis; 2) the most effective protocols for deploying the data abstraction, sampling and data collection tools to these different organizations; and 3) whether the data collected would be sufficient to achieve the study aims. However, the Phase II study aims are broader and include the public release of the tools. Specifically, the pilot will study the feasibility of:
Focusing on characterizing the cases mistakenly not flagged from the administrative data using data from the medical record. Identifying these “false negatives” is important for accurate comparative reporting, and requires more efficient sampling approaches.
Selecting indicators that are relatively less common2 and more clinical complex3 than the indicators included in Phase I. These indicators require additional years of data for adequate sample sizes, and more intensive methods of training and monitoring.
Including the public release of the data abstraction, sampling and data collection tools, which requires an assessment of reliability and regulatory compliance. Ultimately, validation of the AHRQ Patient Safety Indicators requires that these tools become publicly available in order that individual hospitals may use the tools to validate the results in their own institutions in a reliable manner. Public release recognizes that validation is not a static attribute of the indicators, but a dynamic and evolving one that requires continuous evaluation and assessment.
Phase II of the validation pilot is intended to address these three issues. Once complete, the combined lessons from Phase I and Phase II will be incorporated into a modified project design to be submitted as a full OMB Clearance Package sometime in the future.
In the Fiscal Year 2002 Senate Appropriations Report for the Departments of Labor, HHS, and Education (Rpt. 107-84), AHRQ was given the following specific requirements which relate to this pilot project:
The Committee further directs AHRQ to provide a report detailing the results of its efforts to reduce medical errors. The report should include how hospitals and other healthcare facilities are reducing medical errors; how these strategies are being shared among healthcare professionals; how many hospitals and other healthcare facilities record and track medical errors; how medical error information is used to improve patient safety; what types of incentives and/or disincentives have helped healthcare professionals reduce medical errors and; a list of the most common root causes of medical errors. The report should provide data showing the effectiveness of State requirements in reducing medical errors. The report should also describe how AHRQ is responding to some of the findings in the IOM’s report ‘To Err is Human: Building a Safer Health System.’
The Patient Safety Indicators were developed to help AHRQ meet the specific requirements as stated in the above congressional mandate. Achieving the goal of reducing medical errors requires identifying errors in practice, and undertaking initiatives to avoid and prevent them. It also requires national and regional attention to monitor and report to the public about patient safety. Widespread consensus exists that health care organizations can reduce patient injuries by learning from successful safety-improvement initiatives in other industries. Such initiatives have focused on systematically reducing opportunities for errors to occur by improving the environment for safety. These diverse steps range from technical changes, such as implementing electronic medical record systems, to cultural ones, such as improving staff awareness of patient safety risks. Clinical process interventions also have strong evidence for reducing the risk of adverse events related to a patient’s exposure to hospital care. However, local and national initiatives are better prioritized and evaluated through the use of adequate information on patient safety problems. The Patient Safety Indicators meet this information need using relatively inexpensive, readily available hospital administrative data. The data collected in Phase I and II of this validation pilot will further enable the AHRQ PSI to achieve their intended purpose.
This validation pilot is being conducted under the Support for Quality Indicators II contract from the Agency for Healthcare Research and Quality (AHRQ) (Contract No. 290-04-0004).
The AHRQ Patient Safety Indicators are one of four modules4 in the AHRQ Quality Indicators (AHRQ QI) project maintained under the Support for Quality Indicator II (SQI-II) contract to Battelle Memorial Institute, Stanford University and the University of California-Davis (the “AHRQ QI support team”). The AHRQ QI project is a component of the Data Development Portfolio at AHRQ, which includes the Health Care Utilization Project (HCUP) and the Consumer Assessment of Healthcare Providers and Systems (CAHPS). The goal of the Portfolio is to support the attainment of Agency strategic goals to promote safety, quality, efficiency and effectiveness in the healthcare system.
The AHRQ PSI development process consists of conducting literature reviews of previous published studies, convening clinical panels to review indicator definitions, and performing empirical analysis to evaluate alternative specifications for identifying potentially preventable adverse events from hospital administrative data. Continued refinement of the AHRQ PSI requires assessing whether the adverse events identified from the administrative data are documented in the medical record, and whether the adverse events documented in the medical record are identified from the administrative data. The overall goal of the development effort is to generate evidence on the scientific acceptability5 of the AHRQ PSI for use in quality improvement and comparative reporting, and to provide guidance on the appropriate interpretation and use of AHRQ PSI data.
In Phase I of the validation pilot, the data collected was used by the AHRQ QI support team to test medical record abstraction instruments, sampling methods and data collection applications for the purpose of assessing whether the adverse events flagged in the administrative data were documented in the medical record. Specifically, the information was used to address three questions:
Whether cases flagged did or did not have the clinical event. Data collection focused on characterizing that event (e.g., what, when, where, etc.)
The potential preventability of the clinical event. Data collection focused on explicit processes of care that either led to the event or might have prevented the event
Whether there were confounding factors that might be important for improving indicator specifications and for interpreting and using the AHRQ PSI rates.
In addition, the data collected by the validation pilot was used by the support team to test the feasibility of conducting a validation study to address these study aims through a voluntary collaborative of hospital participants.
The lessons drawn from Phase I of the validation pilot indicate that hospitals were willing to participate on a volunteer bases, that the protocols used to deploy the data abstraction, sampling and collection tools to these different organizations were generally effective, and that the data were sufficient to achieve many of the study aims. Of the approximately 60 organizations and 80 hospitals that initially indicated an interest in participating, about 30 organizations and 40 hospitals from 29 states completed the entire pilot. Most of the hospitals that did not participate lacked the required capacity to run the AHRQ QI software or conduct the medical record data abstraction. A few of the hospitals that did not participate had resource constraints at the time the data collection was to occur. Of the hospitals that completed the pilot, the participant survey (Attachment B) indicated that the training and documentation provided prior to data collection was useful and adequate. Most had some difficulty running the software and selecting the sample, especially initially as the staff became more familiar with the data and tools. In the end, all participating hospitals indicated a willingness to participate in future validation efforts.
The validation pilot was intended primarily to test the feasibility of gathering evidence on the criterion validity of the PSIs (based on medical record review). Positive predictive value (PPV) was defined as the crude percentage of PSI-flagged cases that were confirmed by detailed record review. False positive cases were classified as either miscoded diagnoses or correctly coded diagnoses that predated admission.
For "accidental puncture and laceration" (N=250 at 43 hospitals), PPV was 90% (9% miscoded, 1% predated admission) and about 69% of confirmed events required a reparative procedure. For "iatrogenic pneumothorax" (N=175 at 38 hospitals), PPV was 88% (4% miscoded, 8% predated admission) and about 66% of confirmed events were treated with tube thoracostomy. For "postoperative DVT/PE" (N=123 at 37 hospitals), PPV was 84% (6% miscoded, 10% predated admission), but about 22% of true positives involved arm or neck veins and 4% involved superficial leg veins. For "postoperative sepsis" (N=98 at 33 hospitals), PPV was 68% (29% miscoded, 3% predated admission). For "selected infections due to medical care" (N=189 at 37 hospitals), PPV was 63% (19% miscoded, 18% present at admission) and about 78% of confirmed events were catheter-related. Overall, the PPV of the five AHRQ PSIs in a nonrandom sample of US hospitals varied from 63% to 90%, depending on the specific PSI and on how denominator exclusions are handled. Incorporating "present at admission" data would substantially improve most of these PPVs.
In Phase II of the validation pilot, the data collected will be used by the AHRQ QI support team to test medical record abstraction, sampling and data collection tools for the purpose of addressing these additional questions:
Not only whether cases flagged in the administrative data are documented as adverse events in the medical record (that is, relatively few “false positives”), but also whether cases documented as adverse events in the medical record are flagged in the administrative data (that is, relatively few “false negatives”). The challenge is to develop a sampling method to select the cases for review efficiently.
Whether participating institutions are able to access administrative data and medical record data over a long enough period of time in order to have a sufficient study sample for indicators that are relatively less common. These indicators may occur only one per 1,000 cases or less, and will require a minimum of three years of data from which to draw the sample. (More than three years may lessen the contemporary clinical relevance of the findings).
Whether the medical record abstraction tools adequately characterize (e.g., what, when, where, etc.) adverse events that are clinically complex outcomes often involving multiple operations during a single hospital stay.
Whether the medical record abstraction, sampling and data collection tools are reliable, adequately documented, compliant with applicable standards (i.e. Section 508) and otherwise suitable for public release. Reliability is the extent to which the tools provide consistent results across institutions and abstractors with equivalent skills and methods. Reliability testing will require that the hospitals re-abstract approximately 10% of records for selected items for constructing metrics of inter-rater reliability. It will also require the team to modify the training process to ensure uniformity, for example by including mock chart training using one or two sample charts.
In addition, the data collected by the validation pilot will be used by the support team to test the feasibility of conducting a validation study to address these study aims through a voluntary and ongoing collaborative of hospital participants.
In Phase I we developed a web-based data collection application that consists of a user-interface, database and administrator tools for reporting and data extraction (the existing tool can be viewed at http://aqi.ahrq.gov). The current web-based application includes the medical record abstraction tools for all of the Phase I indicators (these tools were included in our previous supporting statement). Although the Phase II indicators will be added to the web-based application, this validation pilot will rely on paper data collection instruments. In Phase I we found that participants preferred paper tools because it facilitated the collection of data from electronic medical records. Attachments C to G1 include the medical record abstraction tools and instructions for the additional Phase II indicators.
A review of the literature including published, unpublished, and internet sources is conducted annually as part of the maintenance of the AHRQ QI. This review has identified a limited number of Patient Safety Indicator validation studies, which the AHRQ QI support team has used in the past to make refinements to the indicator specifications. However, not all of the Patient Safety Indicators have been the subject of such studies, and the studies that have been conducted have not always addressed validation systematically or in sufficient clinical detail to be informative on scientific acceptability. Instead, the studies have tended to focus on the narrow question of positive predictive value (whether the cases flagged based on the administrative record data accurately identify potentially preventable adverse events in the medical record) and on the potential benefit of adding certain clinical data elements to the administrative record (e.g., condition present on admission flags or readmission data).
The validation pilot is distinguished from these previous studies in that the data collection is specifically focused on information that will 1) enable refinement of the indicator specifications based on existing administrative data; 2) identify explicit processes of care associated with the adverse events that will provide a basis for quality improvement activity; and 3) generate guidance on interpretation and use in the context of how the PSI are actually applied in various organizational settings.
In order for the validation pilot to generate useful information on the feasibility of incorporating a validation component into the AHRQ PSI development process, the information collected should reflect the variety of settings in which the Patient Safety Indicators are actually used. This includes both smaller, rural hospitals and larger hospital systems and academic medical centers. However, we do not anticipate that any of the hospitals volunteering to participate would be considered small businesses.
Although the original development of the Patient Safety Indicators systematically reviewed the existing evidence and developed new evidence from clinical panels and empirical analysis, and although some limited studies have contributed to that evidence base since the release of the PSI in March, 2003, additional information is needed to ensure that the Patient Safety Indicators meet generally recognized criteria of scientific acceptability for use in quality improvement and comparative reporting of hospital performance. If the proposed data collection is not approved, there will be an absence of valid indicators developed for and used by hospitals in the U.S. for the purpose of reducing potentially easily avoidable medical errors.
This request is for a one time data collection to occur over approximately a three-month time period. There are no legal obstacles to reducing the burden.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
8.a. Federal Register Notice
This project is being submitted under AHRQ's generic clearance 0935-0124 and does not require publication in the Federal Register.
The AHRQ QI Support team has an ongoing collaboration with researchers at the University Healthsystem Consortium and the Department of Veteran’s Affairs on validation activities related the AHRQ PSI.
No payments or gifts will be provided to the hospitals participating in this validation pilot data collection effort. However, hospitals will receive a report summarizing aggregate results for the entire validation pilot population and will be invited to participate in a webinar with the support team to review the validation pilot findings. In addition, hospitals may use the medical record abstraction instruments for their own internal quality improvement activities.
For each hospital participating in the validation pilot data collection, we will provide a Notice of Data Use advising the participants that the data collected will be treated in a confidential manner, unless otherwise compelled by law. The notice will indicate that the data collection complies with the Privacy Rule under the provisions for use and disclosure of PHI without subject authorization to a public health authority authorized by law (45 CFR 164.512(b)). In addition, the notice will indicate that the data collection qualifies for expedited review under the HHS Protection of Human Subjects regulations as research involving minimal risk to human subjects and relying on non-research data and documents (45 CFR 46.110). No personal identifiers will be used during analysis6 that would allow an individual to be matched to his or her medical record. Only hospital identifiers will be used to monitor the data collection and to allow for examination of results across hospitals. Patient records will be aggregated with those of other patients before any information is reported to AHRQ or any other party outside of the AHRQ QI support team.
There are no questions of a sensitive nature.
We anticipate that a similar number of potential hospital participants will respond to a notice asking for volunteer participants as the number that responded to the notice in Phase I. The notice was issued through the AHRQ QI technical support listserv, and requested that organizations submit an indication of interest form (See Attachment H). The estimate of burden hours assumes participation by approximately 40 hospitals, based on our experience in Phase I. This is a relatively large sample size and includes large and small hospital systems, urban and rural settings, and academic medical centers and community facilities. States vary in the content and extent of their administrative data systems (e.g., the number of diagnosis and procedure codes reported), and the sample includes representatives from many different states. Several of the patient safety indicators occur infrequently, and obtaining adequate sample sizes to test the sufficiency of the data collection for analysis requires collecting data over a longer period of time or across a larger sample of hospitals. A larger sample of hospitals will allow the analysis to incorporate a more reliable assessment of the effects of with-in hospital clustering on the estimation of the positive predictive value and other characteristics of the indicators, leading to more accurate confidence intervals for those parameters. Each hospital will perform an average of 60 record abstractions for a total of 2,400 abstractions across the 40 hospitals.
Exhibit A shows the estimated annualized burden hours. Based on Phase I we expect 140 responses to the interest form, which will require about two minutes to complete. Of the 40 participating hospitals, two persons from each hospital will participate in a 5 hour training session (see Attachments H, C1, D1, E1, F1 and G1). The participant survey will be completed by each person that participates in this project and is expected to require 30 minutes (see Attachment B1). Each of the five abstraction tools will be used by an average of 20 hospitals7, which will use the tool an average of 24 times, for a total of 480 uses each. Each of the abstraction tools will require about one hour to complete, including time to locate the required information and perform the abstraction. The total estimated burden hours for the hospitals time to participate in this project is 2845 hours.
Exhibit B shows the estimated annualized cost burden for the responding hospitals time to participate in this project. The cost burden is estimated to be $88,934.
Exhibit A. Estimated annualized burden hours
Form Name |
Number of respondents |
Number of responses per respondent |
Hours per response |
Total burden hours |
Interest form |
140 |
1 |
2/60 |
5 |
Training |
40 |
2 |
5 |
400 |
Participant survey |
40 |
2 |
30/60 |
40 |
Foreign body left during procedure abstraction tool |
20 |
24 |
1 |
480 |
Postoperative hemorrhage or hematoma abstraction tool |
20 |
24 |
1 |
480 |
Postoperative physiologic & metabolic derangement abstraction tool |
20 |
24 |
1 |
480 |
Postoperative respiratory failure abstraction tool |
20 |
24 |
1 |
480 |
Postoperative wound dehiscence abstraction tool |
20 |
24 |
1 |
480 |
Total |
320 |
na |
na |
2845 |
Exhibit B. Estimated Annualized Cost
Form Name |
Number of respondents |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
Interest form |
140 |
5 |
$31.26 |
$156 |
Training |
40 |
400 |
$31.26 |
$12,504 |
Participant survey |
40 |
40 |
$31.26 |
$1,250 |
Foreign body left during procedure abstraction tool |
20 |
480 |
$31.26 |
$15,005 |
Postoperative hemorrhage or hematoma abstraction tool |
20 |
480 |
$31.26 |
$15,005 |
Postoperative physiologic & metabolic derangement abstraction tool |
20 |
480 |
$31.26 |
$15,005 |
Postoperative respiratory failure abstraction tool |
20 |
480 |
$31.26 |
$15,005 |
Postoperative wound dehiscence abstraction tool |
20 |
480 |
$31.26 |
$15,005 |
Total |
320 |
2845 |
$31.26 |
$88,934 |
*Mean hourly wages for Healthcare Practitioner and Technical Occupations, National Compensation Survey: Occupational wages in the United States, May 2007, “U.S. Department of Labor, Bureau of Labor Statistics.”
Participating hospitals will not be asked to maintain any records beyond the time period of the validation pilot. No additional software or equipment purchases will be necessary to support data collection or record keeping.
The total cost to the government of conducting this validation pilot is approximately $100,000, which includes the cost of development of the medical record abstraction instruments, the sampling design, the data collection application, testing, the actual data collection, analysis, and preparation of a final report. The estimated cost of the data collection component is $25,000, which includes labor costs, fringe expenses, administrative expenses, and costs of materials and supplies.
This is a new activity under this generic clearance (0935-0124).
A final report will be delivered to AHRQ in November 2008. Current publication plans include a special supplement to the journal Medical Care, a health-related, peer-reviewed journal.
Exhibit C. Estimated Project Time Schedule
Validation Pilot Phase |
Timeline |
Planning |
September, 2007 to April, 2008 |
Collaborator recruitment, training; protocol and tool development and testing |
June, 2008 to July, 2008 |
Data collection application development and testing; data reporting |
August, 2008 to September, 2008 |
Analysis and assessment |
October, 2008 to November, 2008 |
AHRQ does not seek this exemption.
Attachments:
Attachment A – Healthcare Research and Quality Act of 1999
Attachment B – Phase I Participant Survey
Attachment C – Foreign body left during procedure abstraction tool
Attachment C1 – Instructions for foreign body left during procedure abstraction tool
Attachment D – Postoperative hemorrhage or hematoma abstraction tool
Attachment D1 – Instructions for postoperative hemorrhage or hematoma abstraction tool
Attachment E – Postoperative physiologic & metabolic derangement abstraction tool
Attachment E1 – Instructions for postoperative physiologic & metabolic derangement abstraction tool
Attachment F – Postoperative respiratory failure abstraction tool
Attachment F1 – Instructions for postoperative respiratory failure abstraction tool
Attachment G – Postoperative wound dehiscence abstraction tool
Attachment G1 – Instructions for postoperative wound dehiscence abstraction tool
Attachment H – Phase II interest form
Attachment I – Training manual
1 The Patient Safety Indicators included in Phase I were Accidental Puncture and Laceration, Iatrogenic Pneumothorax, Postoperative DVT or PE, Postoperative Sepsis and Selected Infections due to Medical Care.
2 The less common indicators include Foreign Body Left in During Procedure, Postoperative Physiologic and Metabolic Derangement and Postoperative Wound Dehiscence.
3 The more clinically complex indicators include Postoperative Hemorrhage or Hematoma and Postoperative Respiratory Failure.
4 The other modules are the Inpatient Quality Indicators, the Prevention Quality Indicators, and the Pediatric Quality Indicators. See http://qualityindicators.ahrq.gov.
5 Scientifically acceptable is generally defined as 1) well defined and precisely specified; 2) reliable; 3) valid; 4) adaptable to patient preferences and a variety of contexts and settings; 5) using an adequate risk adjustment strategy; and 6) having consistent evidence linking process of care to patient outcomes. See, the National Quality Forum, a Comprehensive Framework for Hospital Performance Measure Evaluation (May, 2003).
6 Personal identifiers are used only to facilitate hospital access to the medical record and not retained for data analysis.
7 Because not every hospital will have cases for every indicator
| File Type | application/msword |
| File Title | OMB Clearance Application |
| Author | hamlin-ben |
| Last Modified By | wcarroll |
| File Modified | 2008-06-27 |
| File Created | 2008-06-27 |