SUPPORTING STATEMENT
Part A
Measure Dx: A Resource to Identify, Analyze, and Learn from Diagnostic Safety Events – Evaluation Plan for AHRQ’s ACTION III Diagnostic Safety Capacity Building Contract (TORFP: 75P00119R00265)
Version: May 13, 2022
Agency of Healthcare Research and Quality (AHRQ)
Table of contents
A. Justification 3
1. Circumstances that make the collection of information necessary 3
2. Purpose and use of information 6
3. Use of Improved Information Technology 7
4. Efforts to Identify Duplication 8
5. Involvement of Small Entities 8
6. Consequences if Information Collected Less Frequently 9
7. Special Circumstances 9
8. Federal Register Notice and Outside Consultations 9
9. Payments/Gifts to Respondents 9
10. Assurance of Confidentiality 9
11. Questions of a Sensitive Nature 10
12. Estimates of Annualized Burden Hours and Costs 10
13. Estimates of Annualized Respondent Capital and Maintenance Costs 11
14. Estimates of Annualized Cost to the Government 11
15. Changes in Hour Burden 12
16. Time Schedule, Publication and Analysis Plans 12
17. Exemption for Display of Expiration Date 12
List of Attachments 12
References 14
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. Research that develops and presents scientific evidence regarding all aspects of health care; and
2. Synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. Initiatives to advance private and public efforts to improve health care quality.
AHRQ conducts and supports research and evaluations, and supports demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
Background for this information collection
Diagnostic errors, including delayed, wrong, and missed diagnoses, account for an estimated 40,000 to 80,000 patient deaths each year1-2. Diagnostic errors are responsible for a third of all malpractice claims that result in permanent injury or death,3,4 and are considered the most harmful and costly type of medical error5. Despite the frequency of diagnostic errors and the burden they cause to patients, care teams, and healthcare organizations, the lack of validated strategies to measure these errors has hampered diagnostic improvement efforts. Measurement strategies have been developed and evaluated in research but have yet to be implemented and formally evaluated in operational settings.
In their 2015 report Improving Diagnosis in Health Care, the National Academies of Sciences, Engineering, and Medicine (NASEM) recommended that accrediting organizations require healthcare organizations to “have programs in place to monitor the diagnostic process and identify, learn from, and reduce diagnostic errors and near misses in a timely fashion.” The NASEM report does not recommend specific strategies to measure diagnostic errors but does encourage healthcare organizations to consider using various methods and leverage readily available data sources when possible.6
The Measure Dx resource (the Resource) is a modular toolkit that provides clinicians, quality and safety personnel, and healthcare organization leaders with guidance for implementing diagnostic safety measurement strategies for the purposes of learning and improvement. The Resource was developed and pilot tested (Fast Track OMB control number: 0935-0179) during the base year of an AHRQ contract awarded to the MedStar Health Research Institute and provides pragmatic recommendations for implementing measurement strategies that were identified in the AHRQ Issue Brief titled Operational Measurement of Diagnostic Safety: State of the Science. In particular, the Measure Dx resource focuses on four broad measurement strategies that were assessed to be approaching readiness for implementation in operational settings.
We are requesting full OMB approval to conduct a formal evaluation of the Measure Dx resource. AHRQ would like to further develop this resource, expanding on the initial pilot test which qualitatively examined feasibility of implementing the resource, general receptivity, and feedback for improvement. AHRQ will use the information collected through this Information Collection Request (ICR) to evaluate the efficacy and impact of the Measure Dx resource for stimulating diagnostic safety measurement activities for learning and improvement. To meet this objective, the project team will examine this sample of results via qualitative and quantitative analyses to measure fidelity, process, outcomes, and sustainability. Key outcome measures will assess the yield of diagnostic safety event detection and related diagnostic safety intelligence, and impact of the resource on diagnostic safety policies, activities, and actions taken to mitigate risks/harms. AHRQ’s ability to publicly share a diagnostic measurement resource (Measure Dx) that has been scientifically validated is expected to be of great interest to the health care community and important in helping organizations measure diagnostic safety for patient safety and quality improvement efforts.
This information collection has the following goal:
To evaluate the diagnostic safety measurement resource (Measure Dx) in order to stimulate measurement activities for learning and improvement and quantitatively and qualitatively examine:
Feasibility of the implementing the Resource with limited to no technical assistance;
User experience and satisfaction with the resource;
Impact of the resource on diagnostic safety policies or activities;
Yield of newly detected diagnostic safety events and associated learning resulting from use of the resource;
Intent to sustain use of the resource and continue with the diagnostic safety process following evaluation efforts.
To achieve the goals of this project the following information collection instruments will be completed:
Organizational Characteristics Survey (Appendix A) – designed to qualitatively describe the characteristics of the organizations engaged in evaluation (e.g., patient characteristics, practice size, and staffing).
Organizational Self-Assessment Survey (Appendix B) - designed to qualitatively assess organization’s readiness (e.g., leadership support, resources, and safety culture/infrastructure) for implementing the Measure Dx resource.
The Safer Dx Checklist (Appendix C) - A synthesis of foundational practices that health care organizations can use to advance diagnostic excellence. The checklist provides a framework for organizations to conduct a self-assessment to understand the current state of diagnostic practices, identify areas to improve, and track progress toward diagnostic excellence over time.
Pre-test Evaluation Interview Protocol (Appendix D) – designed to qualitatively assess the organization’s current policies and structures related to diagnostic safety, plans for implementing the Measure Dx resource, and initial feedback on resource materials.
Post-test Evaluation Interview Protocol (Appendix E) – designed to qualitatively assess organization’s experience with implementing the Measure Dx resource, the impact of the resource on diagnostic safety policies or activities in their organization, contextual information about whether and how Measure Dx facilitated case detection, and intent to sustain use of the resource following evaluation efforts.
Team Questionnaire (Appendix F) – adapted to help organizations self-assess diagnostic teamwork in their organization & their diagnostic team’s commitment to implementing the Measure Dx resource.
Case Review Summary Form (Appendix G) – designed to quantitatively and qualitatively summarize the diagnostic safety intelligence that participants have detected, analyzed, and/or learned from while implementing one Measure Dx strategy.
ECHO Calls Protocol (Appendix H) – The purpose of virtual ECHO calls is to foster bi-directional learning among the participating organizations, to check site progress during the implementation period and to understand “real-time” challenges, successes, and lessons learned. Standard questions for each ECHO session will be asked to foster shared learning and discussion.
This information collection is being conducted by AHRQ through its contractor, MedStar Health Research Institute, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).
AHRQ will use the information collected through this ICR to assess and enhance the feasibility of organizations in adopting the Measure Dx resource to stimulate diagnostic safety measurement activities for learning and improvement. AHRQs’ ability to publicly share a diagnostic measurement resource (Measure Dx) that has been scientifically validated is expected to be of great interest to the health care community and important in helping organizations measure diagnostic safety for patient safety and quality improvement efforts.
The specific purpose of each of the information collection instruments is described below:
Organizational Characteristics Survey (Appendix A) – This is collected once per health system, at ten health systems maximum. It is designed to describe the characteristics of the sites engaged in pilot testing (e.g., size, diagnostic team member role diversity, and familiarity with patient safety and quality improvement activities). These data will also be examined as potential moderators of impact.
Organizational Self-Assessment (Appendix B) - This is collected once per health system, at ten health systems maximum. It is a series of items embedded within Section II of the resource. The items are designed to help organizations self-assess their readiness (e.g., leadership support, resources, and safety culture/infrastructure) for implementing the Measure Dx resource and select an appropriate measurement strategy given available resources. The data will be used for evaluation of fidelity and will also be examined as potential factors that influence use and outcomes of Measure Dx.
The Safer Dx Checklist (Appendix C) - This instrument will be collected twice per health system (1 pre-intervention, 1 post-intervention), at ten health systems maximum. It is a publicly available framework designed to help organizations conduct a self-assessment of their current state of diagnostic practices, identify areas to improve, and track progress toward diagnostic excellence over time. The data will be used to evaluate the impact of Measure Dx on diagnostic safety policies and activities.
Pre-test Evaluation Interview Protocol (Appendix D) – This is collected once per health system, prior to implementation, at ten health systems maximum. This data collection instrument has been designed to qualitatively assess the presence and features of current policies and structures at the organizations that are relevant to diagnostic safety and diagnostic errors. It will also be used to obtain information about each organization’s implementation plans (e.g., their selected measurement strategy and perceived barriers and facilitators for resource implementation and/or diagnostic safety improvement efforts) and feedback to optimize resource materials.
Post-test Evaluation Interview Protocol (Appendix E) – This is collected once per health system, post-implementation, at ten health systems maximum. This data collection instrument has been designed to qualitatively assess organization’s experience with implementing the Measure Dx resource including the extent to which users adhered to the resource content (fidelity), barriers and facilitators to improving diagnostic safety with the resource, and user experience and satisfaction with the resource. It will also be used to assess the impact of the resource on diagnostic safety (e.g., yield of diagnostic safety intelligence and actions taken to change practice/safety culture) and intent to sustain use of the resource materials. .
Team Questionnaire (Appendix F) – This instrument will be collected twice per health system (1 pre-intervention, 1 post-intervention), at ten health systems maximum, and will be used to assess diagnostic teamwork over time. This instrument has been adapted from a publicly available survey to facilitate an organizational self-assessment of diagnostic teamwork and the team’s commitment to implementing the Measure Dx resource.
Case Review Summary Form (Appendix G) – This is collected twice per health system (1 mid-intervention, 1 post-intervention). This form is designed to summarize the diagnostic safety intelligence that participants have detected, analyzed, and/or learned from while implementing one Measure Dx strategy. Results obtained from this instrument, with results from the ECHO calls and post-test interviews, will be used to analyze the impact of the Resource on diagnostic safety (e.g., yield of diagnostic safety intelligence).
ECHO Calls Protocol (Appendix H) – The purpose of virtual ECHO calls is foster bi-directional learning among the participating organizations, to check site progress during the implementation period and to understand “real-time” challenges, successes, and lessons learned. Standard queries during each ECHO session will be used to foster shared discussion and site to site networking. There will be a total of 6 ECHO sessions (1 per month), each 60 minutes and attended by one site champion from each of the ten health systems.
The information collection instruments are designed to capture background site and team data (Appendices A, B, F), and to capture data related to use and impact of the Measure Dx resource (Appendices C, D, E, G, and H). The results from this evaluation via statistical analyses will help inform guidance to other organizations that may independently implement the resource when available on the AHRQ website. Every attempt will be made to recruit sites that are representative of diverse geographic locations as well as diverse patient populations served including sites that serve AHRQ priority populations.
The instruments provided in Appendices A-C, F, and G will be collected using an electronic data collection strategy. One health services manager from each of the 10 health systems will receive links to complete the Organizational Characteristics Survey (Appendix A), the Organizational Self-Assessment (Appendix B), the Safer Dx Checklist (Appendix C), and the Team Questionnaire (Appendix F) prior to resource implementation. They will then receive links to complete surveys in Appendices C and F again following implementation of the organization’s selected strategy, to assess changes in responses over time. The same ten health systems will also receive links to complete the Case Review Summary Form (Appendix G) halfway through the project period, and again following implementation of the Resource. No identifiers will be collected on the individual participants who fill out the surveys. However, surveys will include a health system-level identifier that links the surveys to a particular health system, to ensure that all health systems have completed the surveys.
The remaining information collection instruments described herein (Appendices D, E, and H) will rely on paper data collection instruments in the form of interview/focus group protocol and will to be used by the pre/post interview (Appendices D, E) and ECHO call (Appendix H) facilitators and moderators for each information collection activity. Interviews/focus groups and ECHO calls will be audio recorded for the purpose of transcription and coding. There will otherwise be no automated, electronic, or other technological collection techniques or other forms of information technology used for the information collection. In summary:
Survey Instrument |
Frequency |
Collection Method |
Organizational Characteristics Survey (Appx A) |
1x - Pre-implementation |
Electronic data |
Organizational Self-Assessment (Appx B) |
1x - Pre-implementation |
Electronic data |
The Safer Dx Checklist (Appx C) |
2x - Pre-implementation and post-implementation |
Electronic data |
Pre-test Evaluation Interview (Appx D) |
1x – Pre-implementation |
Paper |
Post-test Evaluation Interview (Appx E) |
1x – Post-implementation |
Paper |
Team Questionnaire (Appx F) |
2x- Pre-implementation and post implementation |
Electronic data |
Case Review Summary Form (Appx G) |
2x – Mid-implementation and post implementation |
Electronic data |
ECHO Calls Protocol (Appx H) |
6x – Each month of implementation |
Paper |
The first phase of the parent study involved an extensive environmental scan to review the literature, including published, unpublished, and internet sources to identify existing interventions and resources pertinent to Resource development. The environmental scan revealed key gaps in diagnostic safety measurement understanding and implementation leading to missed, delayed, and wrong diagnosis which are addressed, in part, by the Resource being evaluated under this information collection request. To our knowledge, this does not involve a duplication of any existing efforts as suggested by the gap analysis resulting from the environmental scan.
The information being collected under this request will reflect the variety of settings in which the Resource will actually be used, inpatient and outpatient settings. To our knowledge none of the practices volunteering to participate would be considered small businesses or small entities.
This request is consistent with the generic information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on June 15, 2022, Volume 87, Number 115, page 36128, and provided a sixty-day period for public comment. A copy of this notice is attached as Attachment Y. During the notice and comment period, the government received no requests for information or substantive comments.
Not applicable.
Our information collection efforts will not offer direct payments or gifts to individual respondents. The organizations engaging in evaluation activities will be sub-contractors to the MedStar Health Research Institute and will receive a flat stipend for their efforts.
Individuals and organizations will be assured of the confidentiality of their replies under Section 944(c) of the Public Health Service Act. 42 U.S.C. 299c-3(c). That law requires that information collected for research conducted or supported by AHRQ that identifies individuals or establishments be used only for the purpose for which it was supplied.
Information that can directly identify the respondent, such as name and/or social security number will not be collected. No information will allow for individual identification of participants
Participants will also receive the following confidentiality statements printed on any respondent materials: “The confidentiality of your responses are protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)]. Information that could identify you will not be disclosed unless you have consented to that disclosure.”
Information collected will be maintained in a secure HIPAA-compliant data server. All information collection will be stored using the contractor, MedStar’s REDCap™ research data capture database. REDCap™ is a mature, secure web application for building and managing online information collection instruments and data. While REDCap™ can be used to collect virtually any type of data, it is specifically geared to support data capture for research studies. The REDCap™ Consortium is composed of 1,711 active institutional partners in 96 countries who utilize and support REDCap™ in various ways. REDCap™ can be established to support data entry forms and to conduct web-enabled surveys. The project team will also use a REDCap™ project space to securely store any documents received from the practices during the project. The MedStar Health Research Institute is a REDCap™ project collaborator site with a robust history of using this method for data collection.
This ICR does not request any personally identifiable information.
This ICR does not include a form that requires a Privacy Act Statement.
Does this ICR request any personally identifiable information (see OMB Circular No. A-130 for an explanation of this term)? Please consult with your agency's privacy program when making this determination? Yes No
Does this ICR include a form that requires a Privacy Act Statement (see 5 U.S.C. §552a(e)(3))? Please consult with your agency's privacy program when making this determination. Yes No
The proposed information collection does not include any questions of a sensitive nature. Each respondent will undergo an informed consent process that will describe participant rights. We anticipate that the MedStar Health Research Institute’s Intuitional Review Board will grant a waiver of documentation for written consent as the consent form will be the only documentation linking the participant’s identity to the information collection. Each participant will receive a copy of the study information sheet for verbal consent and an information sheet on the project that outlines the participant’s rights as is standard for MedStar’s exempt studies where active recruitment is required. The consent process will highlight the participants right to answer or not answer any questions that they are asked and their right to withdraw from the interview and/or focus group at any time without penalty or repercussions.
Exhibit 1. Estimated annualized burden hours
Form Name |
Number of respondents |
Number of responses per respondent |
Hours per response |
Total burden hours |
Appendix A: Organizational Characteristics Survey |
10 |
1 |
1 |
10 |
Appendix B: Organizational Self-Assessment (from Measure Dx) |
10 |
1 |
.5 |
5 |
Appendix C –Safer Dx Checklist |
10 |
2 |
0.25 |
5 |
Appendix D: Pre-Test Interview Protocol |
20 |
1 |
1 |
20 |
Appendix E: Post-test Evaluation Interview Protocol |
20 |
1 |
1 |
20 |
Appendix F: Team Questionnaire |
10 |
2 |
0.25 |
5 |
Appendix G: Case Review Summary Form |
10 |
2 |
.75 |
15 |
Appendix H: ECHO Call Protocol |
10 |
6 |
1 |
60 |
Total |
100 |
180 |
|
Exhibit 2. Estimated annualized cost burden
Form Name |
Total burden hours |
Average hourly wage rate* |
Respondent cost |
Appendix A: Organizational Characteristics Survey |
10 |
$57.61a |
$576.1 |
Appendix B: Organizational Self-Assessment (from Measure Dx) |
5 |
$57.61a |
$288.05 |
Appendix C: Safer Dx Checklist |
5 |
$57.61a |
$288.05 |
Appendix D: Pre-Test Interview Protocol |
20 |
$136.37b |
$2,727.40 |
Appendix E: Post-test Evaluation Interview Protocol |
20 |
$136.37b |
$2,727.40 |
Appendix F: Team Questionnaire |
5 |
$57.61a |
$288.05 |
Appendix G: Case Review Summary Form |
15 |
$136.37b |
$2,045.60 |
Appendix H: ECHO Call Protocol |
60 |
$57.61a |
$3,456.60 |
Total |
140 |
NA |
$12,397.25 |
* National Compensation Survey: Occupational wages in the United States May 2021 “U.S. Department of Labor, Bureau of Labor Statistics.” (https://www.bls.gov/oes/current/oes_nat.htm#29-0000)
a Based on the mean wages for Medical and Health Services Managers (Code 11-9111)
b Based on the mean wages for Physicians (broad) (Code 29-1210)
There are no direct costs to respondents other than their time to participate in the study.
The total contractor cost to the government is estimated to be $205,738.56. As shown in Exhibit 3a, this amount includes costs for project development ($18,812.50); publication of results ($3,000); data collection activities ($93,152.31); data processing and analysis ($24,178.47); project management ($18,812.50) and overhead ($47,782.78).
Exhibit 3a. Estimated Total and Annualized Cost
Cost Component |
Total Cost |
Annualized Cost |
Project Development |
$18,812.50 |
$18,812.50 |
Data Collection Activities |
$93,152.31 |
$93,152.31 |
Data Processing and Analysis |
$24,178.47 |
$24,178.47 |
Publication of Results |
$3,000.00 |
$3,000.00 |
Project Management |
$18,812.50 |
$18,812.50 |
Overhead |
$47,782.78 |
$47,782.78 |
Total |
$205,738.56 |
$205,738.56 |
Exhibit 3b. Federal Government Personnel Cost
Activity |
Federal Personnel |
Hourly Rate |
Estimated Hours |
Cost |
Project oversight to include data collection oversight and review of results |
Project Officer GS15 |
$81.84 |
25 |
$2,046 |
Total |
$2,046 |
|||
Annual salaries based on 2020 OPM Pay Schedule for Washington/DC area: https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2020/DCB_h.pdf
This is a new information collection, thus no changes in hour burden is expected or reported here.
The information collection will begin upon OMB approval (estimated October 2022) and will include recruitment of practices and completion of all data collection activities by September 2023. We anticipate a 6-month implementation time period with quantitative analysis for evaluating the Measure Dx resource, complete by June 2023. We will seek institutional review board approval for this data collection. Upon completion of the evaluation, resource materials will be revised as needed, and an evaluation report will be developed by project period end (September 23, 2023). Publication of the updated materials by AHRQ on their website will be completed after 508 compliance review. There will also be efforts to publish details of the design and findings of this evaluation in open-access articles in one or more peer-reviewed scientific journals. All published information will only be presented in the aggregate and will be free of information that could be used to identify individual respondents or participating health systems.
AHRQ does not seek this exemption.
Appendix A: Organizational Characteristics Survey
Appendix B: Organizational Self-Assessment (from Measure Dx)
Appendix C: Safer Dx Checklist (publicly available)
Appendix D: Pre-Test Interview Protocol
Appendix E: Post-test Interview Protocol
Appendix F: Team Questionnaire (adapted)
Appendix G: Case Review Summary Form
Appendix H: ECHO Call Protocol
References
Leape LL, Berwick DM, Bates DW. Counting Deaths Due to Medical Errors—Reply. JAMA. 2002;288(19):2405. doi:10.1001/jama.288.19.2405-jlt1120-2-3
Newman-Toker DE, Pronovost PJ. Diagnostic errors the next frontier for patient safety. JAMA - J Am Med Assoc. 2009;301(10):1060-1062. doi:10.1001/jama.2009.249
CRICO Strategies, 2015. 2014 Annual Benchmarking Report: Malpractice Risks in the Diagnostic Process. Cambridge, MA, Harvard Medical Institutions, Inc.
Newman-Toker DE, Schaffer AC, Yu-Moe CW, et al. Serious misdiagnosis-related harms in malpractice claims: The “Big Three” – vascular events, infections, and cancers. Diagnosis. 2019;6(3):227-240. doi:10.1515/dx-2019-0019
Saber Tehrani AS, Lee HW, Mathews SC, et al. 25-Year summary of US malpractice claims for diagnostic errors 1986-2010: An analysis from the National Practitioner Data Bank. BMJ Qual Saf. 2013;22(8):672-680. doi:10.1136/bmjqs-2012-001550
National Academy of Medicine. Improving Diagnosis in Health Care. (Balogh EP, Miller BT, Ball JR, eds.). Washington, DC: National Academies Press; 2015. doi:10.17226/21794.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Hill, Mary A |
| File Modified | 0000-00-00 |
| File Created | 2022-09-01 |