Supporting Statement

SUPPORTING STATEMENT

Part A

Building Diagnostic Safety Capacity - Diagnostic Safety Measurement Resource Evaluation Plan

Version: December 21, 2020

Agency for Healthcare Research and Quality (AHRQ)

Table of contents

A. Justification 3

1. Circumstances that Make the Collection of Information Necessary 3

2. Purpose and use of Information 5

3. Use of Improved Information Technology 6

4. Efforts to Identify Duplication 6

5. Involvement of Small Entities 6

6. Consequences if Information Collected Less Frequently 6

7. Special Circumstances 6

8. Federal Register Notice and Outside Consultations 7

9. Payments/Gifts to Respondents 7

10. Assurance of Confidentiality 7

11. Questions of a Sensitive Nature 8

12. Estimates of Annualized Burden Hours and Costs 9

13. Estimates of Annualized Respondent Capital and Maintenance Costs 9

14. Estimates of Annualized Cost to the Government 10

15. Changes in Hour Burden 10

16. Time Schedule, Publication and Analysis Plans 10

17. Exemption for Display of Expiration Date 17

List of Attachments 11

References 12

A. Justification

1. Circumstances that make the collection of information necessary

The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:

1. research that develops and presents scientific evidence regarding all aspects of health care; and

2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and

3. initiatives to advance private and public efforts to improve health care quality.

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.

Background for this information collection

Delayed, wrong, and missed diagnoses, or diagnostic errors, account for 40,000 to 80,000 patient deaths each year^1-2. Diagnostic errors are the most harmful type of medical error³ and are responsible for 33% of malpractice claims that result in permanent injury or death⁴ of the patient. Despite the frequency and burden of diagnostic error, efforts to improve diagnostic safety are hampered by a lack of valid measurement strategies. Strategies that have been developed have been evaluated in research but have yet to be implemented in operational settings.

In the 2015 report Improving Diagnosis in Health Care, the National Academies of Sciences, Engineering, and Medicine (NASEM) recommended that accrediting organizations require healthcare organizations to “have programs in place to monitor the diagnostic process and identify, learn from, and reduce diagnostic errors and near misses in a timely fashion.” The NASEM report stops short of recommending specific measurement approaches but encourages healthcare organizations to consider using multiple methods and leverage readily available data sources when possible.⁵

The diagnostic safety measurement resource is a brief, modular toolkit that provides clinicians, quality and safety personnel, and healthcare organization leaders with guidance for implementing diagnostic safety measurement strategies for the purposes of learning and improvement. The resource provides pragmatic recommendations for implementing measurement strategies that were identified in the recent AHRQ Issue Brief titled Operational Measurement of Diagnostic Safety: State of the Science. In particular, the measurement resource focuses on three broad measurement strategies that were assessed to be approaching readiness for implementation in operational settings.

This information collection has the following goal:

1. To pilot test the diagnostic safety measurement resource (the Resource) in order to stimulate measurement activities for learning and improvement and qualitatively examine:

   1. Feasibility of the implementing the Resource with limited to no technical assistance;

   2. Receptivity of clinicians and quality and safety personnel to the Resource;

   3. Feedback on areas to improve the Resource and its implementation within various settings (e.g., hospital, outpatient practices, rural/urban, academic/non-academic).

To achieve the goals of this project the following information collection instruments will be completed using individual interviews and/or focus groups:

1. Organizational Characteristics Survey (Appendix A) - designed to qualitatively describe the characteristics of the organizations engaged in pilot testing (e.g., patient characteristics, practice size, and staffing).

2. Organizational Readiness for Implementation Change Survey (ORIC) (Appendix B) - designed to qualitatively assess each organization’s readiness for implementing new diagnostic safety measurement processes, learning and feedback activities, etc.

3. Pre-test Interview Protocol (Appendix C) – designed to observe and qualitatively assess organizational users’ understanding of Resource instructions and materials, as well as barriers to application at the beginning of pilot testing.

4. Post-test Evaluation Interview Protocol (Appendix D) - designed to qualitatively assess the barriers and facilitators of implementing the Resource in practice, users’ receptivity to and experience with using the Resource, and recommendations for improvements to improve satisfaction of use.

This information collection is being conducted by AHRQ through its contractor, MedStar Health Research Institute, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).

2. Purpose and Use of Information

AHRQ will use the information collected through this Information Collection Request to assess the feasibility of adopting a resource to facilitate measurement of diagnostic safety events with the goal of organization-level learning and improvement. A qualitative approach to information collection (e.g., individual interviews and/or focus groups) and analysis will be used to provide useful insights on stakeholders’ perceptions and opinions, identify barriers and facilitators to Resource implementation, receptivity to the Resource by stakeholders including clinicians, quality and safety personnel, and organization leadership, as well as to explore stakeholder feedback to enhance the usability of the Resource. In this context, we will examine the question “How and in what contexts do the chosen interventions work or can they be amended to work?”, rather than evaluating effectiveness or “Do they work?” This approach will not consist of statistical surveys that yield quantitative results that can be generalized to the population. Pilot testing will occur at up to 20 healthcare organizations, and feasibility of implementation will be assessed at the stakeholder and organization levels. The information collected from stakeholders will be used to revise the Resource in order to promote widespread adoption.

The specific purpose of each of the information collection instruments is described below:

1. Organizational Characteristics Survey (Appendix A). The information collection instrument will qualitatively summarize and describe the characteristics of the organizations engaged in pilot testing. The information will be used to summarize information on organizational characteristics in a qualitative manner. It will also allow for the project team to identify competing priorities that may inhibit full organizational participation with the proposed qualitative methods. This information collection will provide context for pilot testing and generate information on the type of organizations where the Resource was tested. It will thus support the ability to define the types of organizations where the Resource is found to be generally acceptable.

2. Organizational Readiness for Implementation Change (ORIC) (Appendix B). The information collection instrument will qualitatively assess individual organizations’ readiness to change. One individual from the organization will complete the instrument in collaboration with a project team member/interviewer.

3. Pre-test Interview Protocol (Appendix C). The information collected will provide qualitative information for analysis. Information collected will define the barriers and facilitators of Resource implementation. It will allow for a summary and synthesis of qualitative information and recommendations for improvement of the Resource based on initial stakeholder feedback, satisfaction, and receptivity to the Resource.

4. Post-test Evaluation Interview Protocol (Appendix D). The information collected will be qualitative and will be synthesized to assess and define the barriers and facilitators of Resource implementation. The qualitative information collected will allow for the development of recommendations for improving the Resource based on stakeholder feedback (quality and safety personnel), satisfaction, and receptivity to the Resource.

These information collection instruments (Appendices A-D) are designed to capture qualitative data. No claim is made that the results from this evaluation will be generalizable in a statistical sense, nor is the intent to conduct statistical analyses. However, every attempt will be made to recruit organizations that are representative of diverse geographic locations as well as diverse patient populations served including practices that serve AHRQ priority populations. The goals of the evaluation are aimed at soliciting stakeholder feedback through interviews on the challenges of implementation, receptivity to the Resource, and feedback on how to improve the Resource to support adoption and implementation. The information collected will be used to revise the Resource to enhance usability and receptivity.

3. Use of Improved Information Technology

The information collection described herein will rely on paper information collection instruments in the form of interview guides to be used by the interview facilitators and moderators for each information collection activity. Interviews will be audio recorded for the purpose of transcription and coding. There will otherwise be no automated, electronic, or other technological collection techniques or other forms of information technology used for the information collection.

4. Efforts to Identify Duplication

The first phase of the parent study involved an extensive environmental scan to review the literature, including published, unpublished, and internet sources to identify existing interventions and resources pertinent to Resource development. The environmental scan revealed key gaps in diagnostic safety measurement understanding and implementation leading to missed, delayed, and wrong diagnosis which are addressed, in part, by the Resource being evaluated under this information collection request. To our knowledge, this does not involve a duplication of any existing efforts as suggested by the gap analysis resulting from the environmental scan.

5. Involvement of Small Entities

The information being collected under this request will reflect the variety of settings in which the Resource will actually be used, inpatient and outpatient settings. To our knowledge none of the practices volunteering to participate would be considered small businesses or small entities.

6. Consequences if Information Collected Less Frequently

This information collection is for a one-time information collection only. All of the information needed to solicit stakeholder feedback to inform Resource revisions under this information collection request does not need to be collected more than once.

7. Special Circumstances

This request is consistent with the generic information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.

8. Federal Register Notice and Outside Consultations

8.a. Federal Register Notice

This information collection is being submitted under AHRQ’s generic clearance. A Federal Register notice is therefore not required.

8.b. Outside Consultations

Not applicable.

9. Payments/Gifts to Respondents

Our information collection efforts will not offer direct payments or gifts to individual respondents. The organizations engaging in the information collection efforts will be sub-contractors to the MedStar Health Research Institute and their partners.

10. Assurance of Confidentiality

Individuals and organizations will be assured of the confidentiality of their replies under Section 944(c) of the Public Health Service Act.  42 U.S.C. 299c-3(c). That law requires that information collected for research conducted or supported by AHRQ that identifies individuals or establishments be used only for the purpose for which it was supplied.

Information that can directly identify the respondent, such as name and/or social security number will not be collected. Participant characteristics that will be collected for individuals who participate in pilot testing will be limited to information about their functional roles within their organizations (see Appendix C). This information will be presented in aggregate and used to describe the characteristics of the group providing feedback. No information will allow for individual identification of participants.

Participants will also receive the following confidentiality statements printed on any respondent materials:

“The confidentiality of your responses are protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)]. Information that could identify you will not be disclosed unless you have consented to that disclosure.”

Information collected will be maintained in a secure HIPAA-compliant data server. All information collection will be stored using the contractor, MedStar’s REDCap™ research data capture database. REDCap™ is a mature, secure web application for building and managing online information collection instruments and data. While REDCap™ can be used to collect virtually any type of data, it is specifically geared to support data capture for research studies. The REDCap™ Consortium is composed of 1,711 active institutional partners in 96 countries who utilize and support REDCap™ in various ways. REDCap™ can be established to support data entry forms and to conduct web-enabled surveys. The Course will also use a REDCap™ project space to securely store any documents received from the practices during the project. The MedStar Health Research Institute is a REDCap™ project collaborator site with a robust history of using this method for data collection.

The information collected will be primarily qualitative in nature. Responses to interviews and/or focus groups will be recorded and notes for each session transcribed. Transcription files will be uploaded to the REDCap™ file repository for security. Paper files will be retained in a locked file cabinet within MedStar Health. A case record form will be created within REDCap™ to record output from the thematic reviews of the transcripts to facilitate reporting and feedback to participants and to the study team. This information collection request (ICR) does not contain surveys, censuses, or employ statistical methods.

This ICR does not request any personally identifiable information.

This ICR does not include a form that requires a Privacy Act Statement.

Does this ICR contain surveys, censuses, or employ statistical methods? Yes No

Does this ICR request any personally identifiable information (see OMB Circular No. A-130 for an explanation of this term)? Please consult with your agency's privacy program when making this determination Yes No

Does this ICR include a form that requires a Privacy Act Statement (see 5 U.S.C. §552a(e)(3))? Please consult with your agency's privacy program when making this determination. Yes No

11. Questions of a Sensitive Nature

The proposed information collection does not include any questions of a sensitive nature. Each respondent will undergo an informed consent process that will describe participant rights. We anticipate that the MedStar Health Research Institute’s Intuitional Review Board will grant a waiver of documentation for written consent as the consent form will be the only documentation linking the participant’s identity to the information collection. Each participant will receive a copy of the study information sheet for verbal consent and an information sheet on the project that outlines the participant’s rights as is standard for MedStar’s exempt studies where active recruitment is required. The consent process will highlight the participants right to answer or not answer any questions that they are asked and their right to withdraw from the interview and/or focus group at any time without penalty or repercussions.

12. Estimates of Annualized Burden Hours and Costs

Exhibit 1.  Estimated annualized burden hours

Form Name	Number of respondents	Number of responses per respondent	Hours per response	Total burden hours
Appendix A: Organizational Characteristics Survey	20	1	1.5	30
Appendix B: Organizational Readiness for Implementation Change (ORIC)	20	1	0.25	5
Appendix C: Pre-Test Interview Protocol	60	1	1	60
Appendix D: Post-test Evaluation Interview Protocol	60	1	1	60
Total	NA	NA	NA	155

 

Exhibit 2. Estimated annualized cost burden

Form Name	Number of respondents	Total burden hours	Average hourly wage rate*	Total cost burden
Appendix A: Organizational Characteristics Survey	20	30	$55.37a	$1,661.10
Appendix B: Organizational Readiness for Implementation Change (ORIC)	20	5	$55.37a	$276.85
Appendix C: Pre-Test Interview Protocol: Administrator	30	30	$55.37a	$1,661.10
Appendix C: Pre-Test Interview Protocol: Clinician	30	30	$102.53b	$3,075.90
Appendix D: Post-test Evaluation Interview Protocol: Administrator	30	30	$55.37a	$1,661.10
Appendix D: Post-test Evaluation Interview Protocol: Clinician	30	30	$102.53b	$3,075.90
Total	NA	155	NA	$11,411.95

* National Compensation Survey: Occupational wages in the United States May 2019 “U.S. Department of Labor, Bureau of Labor Statistics.”

^a Based on the mean wages for Medical and Health Services Managers (Code 11-9111)

^b Based on the mean wages for Family Medicine/General Practitioners (Code 29-1062)

13. Estimates of Annualized Respondent Capital and Maintenance Costs

There are no direct costs to respondents other than their time to participate in the study.

14. Estimates of Total and Annualized Cost to the Government

The total contractor cost to the government is estimated to be $202,738.56. As shown in Exhibit 3a, this amount includes costs for project development ($18,812.50); data collection activities ($93,152.31); data processing and analysis ($24,178.47), project management ($18,812.50) and overhead ($47,782.78).

Exhibit 3a.  Estimated Total and Annualized Cost

Cost Component	Total Cost	Annualized Cost
Project Development	$18,812.50	$18,812.50
Data Collection Activities	$93,152.31	$93,152.31
Data Processing and Analysis	$24,178.47	$24,178.47
Publication of Results	$0	$0
Project Management	$18,812.50	$18,812.50
Overhead	$47,782.78	$47,782.78
Total	$202,738.56	$202,738.56

Exhibit 3b. Federal Government Personnel Cost

Activity	Federal Personnel	Hourly Rate	Estimated Hours	Cost
Project oversight to include data collection oversight and review of results	Project Officer GS15	$81.84	25	$2046
Total				$ 2046

Annual salaries based on 2020 OPM Pay Schedule for Washington/DC area: https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2020/DCB_h.pdf

15. Changes in Hour Burden

This is a new information collection, thus no changes in hour burden is expected or reported here.

16. Time Schedule, Publication and Analysis Plans

The information collection will begin upon OMB approval (estimated March 2020) and will include recruitment of practices and completion of all data collection activities by November 2021. Qualitative analysis will be completed by December 2021 and materials will be revised based on stakeholder feedback by February 2022. We anticipate a 6-month pilot test timeline. Publication of the materials by AHRQ on their website will be completed after 508 compliance review. There are no other publication efforts associated with this information collection effort.

17. Exemption for Display of Expiration Date

AHRQ does not seek this exemption.

List of Attachments:

Appendix A: Organizational Characteristics Survey

Appendix B: Organizational Readiness for Implementation Change (ORIC)

Appendix C: Pre-Test Interview Protocol

Appendix D: Post-test Evaluation Interview Protocol

References

1. Leape LL, Berwick DM, Bates DW. Counting Deaths Due to Medical Errors—Reply.

JAMA. 2002;288(19):2405. doi:10.1001/jama.288.19.2405-jlt1120-2-3

2. Newman-Toker DE, Pronovost PJ. Diagnostic errors the next frontier for patient safety.

JAMA - J Am Med Assoc. 2009;301(10):1060-1062. doi:10.1001/jama.2009.249

3. Saber Tehrani AS, Lee HW, Mathews SC, et al. 25-Year summary of US malpractice claims for diagnostic errors 1986-2010: An analysis from the National Practitioner Data Bank. BMJ Qual Saf. 2013;22(8):672-680. doi:10.1136/bmjqs-2012-001550

4. Newman-Toker DE, Schaffer AC, Yu-Moe CW, et al. Serious misdiagnosis-related harms in malpractice claims: The “Big Three” – vascular events, infections, and cancers. Diagnosis. 2019;6(3):227-240. doi:10.1515/dx-2019-0019

5. National Academies of Sciences, Engineering, and Medicine. 2015. Improving Diagnosis in Health Care. Washington, DC: The National Academies Press. https://doi.org/10.17226/21794.