SUPPORTING STATEMENT
Part A
Building Diagnostic Safety Capacity - Diagnostic Calibration Resource Evaluation Plan
Version: April 28, 2021
Agency for Healthcare Research and Quality (AHRQ)
Table of contents
A. Justification 3
1. Circumstances that Make the Collection of Information Necessary 3
2. Purpose and use of Information 5
3. Use of Improved Information Technology 6
4. Efforts to Identify Duplication 6
5. Involvement of Small Entities 6
6. Consequences if Information Collected Less Frequently 7
7. Special Circumstances 7
8. Federal Register Notice and Outside Consultations 7
9. Payments/Gifts to Respondents 7
10. Assurance of Confidentiality 7
11. Questions of a Sensitive Nature 8
12. Estimates of Annualized Burden Hours and Costs 9
13. Estimates of Annualized Respondent Capital and Maintenance Costs 9
14. Estimates of Annualized Cost to the Government 10
15. Changes in Hour Burden 10
16. Time Schedule, Publication and Analysis Plans 10
17. Exemption for Display of Expiration Date 11
List of Attachments 11
References 12
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. research that develops and presents scientific evidence regarding all aspects of health care; and
2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
Background for this information collection
Delayed, wrong, and missed diagnoses, or diagnostic errors, account for 40,000 to 80,000 patient deaths each year1-2. Diagnostic errors are the most harmful type of medical error3 and are responsible for 33% of malpractice claims that result in permanent injury or death4 of the patient. Despite the frequency and burden of diagnostic error, efforts to improve diagnostic safety are hampered by a lack of valid measurement strategies. Strategies that have been developed have been evaluated in research but have yet to be implemented in operational settings.
In the 2015 report Improving Diagnosis in Health Care, the National Academies of Sciences, Engineering, and Medicine (NASEM) recommended that accrediting organizations require healthcare organizations to “have programs in place to monitor the diagnostic process and identify, learn from, and reduce diagnostic errors and near misses in a timely fashion.” The NASEM report stops short of recommending specific measurement approaches but encourages healthcare organizations to consider using multiple methods and leverage readily available data sources when possible.5
Measurement approaches will be needed at both the health care system and individual clinician level to improve diagnosis and foster diagnostic excellence. This resource, however, focuses on measurement related to individual clinicians by seeking to measure and improve clinician calibration (i.e., an assessment of one’s diagnostic performance that aligns with one’s actual diagnostic performance).6 Calibration can affect information gathering and help seeking behavior during the diagnostic process. For example, poor clinical calibration is often manifested as overconfidence in diagnostic accuracy,6-7 leading to reduced use of diagnostic resources (such as help from a colleague, diagnostic decision support, or other reference materials) when needed to get the correct diagnoses for their patients.6-8 Poor calibration could also manifest as under confidence, resulting in too much testing and delayed diagnoses. It is, therefore, important for clinicians to improve their calibration (so they are neither over- nor under-confident) to move towards diagnostic excellence.
The diagnostic safety calibration resource is a brief, pragmatic, and user-friendly tool that provides clinicians with guidance for evaluating and calibrating their own diagnostic performance for the purposes of learning and improvement. The tool is designed to be used by individual licensed clinicians, but healthcare organization leaders, quality and safety personnel, educators, and trainees may also be relevant stakeholders. The resource will provide practical recommendations for implementing calibration strategies in operational settings.
This information collection has the following goal:
To pilot test the diagnostic safety calibration resource (the Resource) in order to prepare the Resource for further evaluation and eventual implementation. The testing process will qualitatively examine:
Ease of use of the Resource with limited or no technical assistance;
Receptivity of clinicians to the Resource;
Feedback on areas to improve the Resource and feasibility of its implementation within various settings (e.g., hospital, outpatient practices, rural/urban, academic/non-academic).
To achieve the goals of this project the following information collection instruments will be completed:
Individual Respondent Characteristics Survey (Appendix A) - designed to qualitatively describe the characteristics of individual clinicians engaged in pilot testing and characteristics of their clinical practice settings.
Safety Attitude Survey (Appendix B) – A subset of items from the Safety Attitudes Questionnaire9 (specifically, the 7 items that make up the Safety Climate subscale) will be used qualitatively to further characterize the group of individual clinicians who engage in pilot testing.
Pre-test Interview Protocol (Appendix C) – designed to observe and qualitatively assess users’ understanding of Resource instructions and materials, as well as barriers to application at the beginning of pilot testing.
Post-test Evaluation Interview Protocol (Appendix D) - designed to qualitatively assess the barriers and facilitators of implementing the Resource in practice, users’ receptivity to and experience with using the Resource, and recommendations to improve usability and satisfaction.
This information collection is being conducted by AHRQ through its contractor, MedStar Health Research Institute, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).
AHRQ will use the information collected through this Information Collection Request to assess the feasibility of adopting a resource to facilitate clinicians’ self-calibration of diagnostic processes with the ultimate goal of improving the quality and safety of diagnostic processes. A qualitative approach to information collection (individual interviews) and analysis will be used to provide insights on stakeholders’ perceptions and opinions, identify barriers and facilitators to Resource implementation, and assess usability and receptivity to the Resource by intended end-users (clinicians). In this context, we will examine the question “How and in what contexts do the chosen interventions work or can they be amended to work?”, rather than evaluating effectiveness or “Do they work?” This approach will not consist of statistical surveys that yield quantitative results that can be generalized to the population.
Pilot testing will occur with up to 20 individual clinicians at different healthcare organizations, and feasibility of implementation will be assessed at the stakeholder and organization levels. The information collected from stakeholders (i.e., the individual clinicians) will be used to revise the Resource in order to promote widespread adoption.
The specific purpose of each of the information collection instruments is described below:
Individual Respondent Characteristics Survey (Appendix A). The information collection instrument will qualitatively summarize and describe the characteristics of the stakeholders engaged in pilot testing. The information will be used to summarize information on stakeholder characteristics in a qualitative manner. This information collection will provide context for pilot testing and generate information on the characteristics of individuals stakeholders and the organizations where the Resource was tested. It will thus support the ability to define the types of users for whom the Resource is found to be generally acceptable.
Safety Attitude Survey (Appendix B). The information collection instrument will qualitatively assess individual stakeholders’ perceptions of the safety climate in their respective practice settings. This information will be used to summarize information on stakeholders in a qualitative manner and provide context for interpretation of other qualitative interview data.
Pre-test Interview Protocol (Appendix C). The information collected will provide qualitative information for analysis. Information collected will define the barriers and facilitators of Resource implementation. It will allow for a summary and synthesis of qualitative information and recommendations for improvement of the Resource based on initial stakeholder feedback, satisfaction, and receptivity to the Resource.
Post-test Evaluation Interview Protocol (Appendix D). The information collected will be qualitative and will be synthesized to assess and define the barriers and facilitators of Resource implementation. The qualitative information collected will allow for the development of recommendations for improving the Resource based on stakeholder feedback, satisfaction, and receptivity to the Resource.
These information collection instruments (Appendices A-D) are designed to capture qualitative data. No claim is made that the results from this evaluation will be generalizable in a statistical sense, nor is the intent to conduct statistical analyses. However, every attempt will be made to recruit from organizations that are representative of diverse geographic locations as well as diverse patient populations served, including practices that serve AHRQ priority populations. The goals of the evaluation are aimed at soliciting stakeholder feedback through interviews on the challenges of implementation, receptivity to the Resource, and feedback on how to improve the Resource to support adoption and implementation. The information collected will be used to revise the Resource to enhance usability and receptivity.
The information collection described herein will rely on paper information collection instruments in the form of interview guides to be used by the interview facilitators and moderators for each information collection activity. Interviews will be conducted via teleconference and audio recorded for the purpose of transcription and coding. There will otherwise be no automated, electronic, or other technological collection techniques or other forms of information technology used for the information collection.
The first phase of the parent study involved an extensive environmental scan to review the literature, including published, unpublished, and internet sources to identify existing interventions and resources pertinent to Resource development. The environmental scan revealed key gaps in understanding diagnostic safety performance measurement and improvement leading to missed, delayed, and wrong diagnosis which are addressed, in part, by the Resource being evaluated under this information collection request. To our knowledge, this does not involve a duplication of any existing efforts as suggested by the gap analysis resulting from the environmental scan.
The information being collected under this request will reflect the variety of settings in which the Resource will actually be used, inpatient and outpatient settings. To our knowledge none of the stakeholders’ organizations would be considered small businesses or small entities.
This information collection is for a one-time information collection only. All of the information needed to solicit stakeholder feedback to inform Resource revisions under this information collection request does not need to be collected more than once.
This request is consistent with the generic information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
This information collection is being submitted under AHRQ’s generic clearance. A Federal Register notice is therefore not required.
Not applicable.
Clinician participants in the described information collection effort will be offered a small and reasonable token of appreciation to participate in the proposed qualitative data collection activities. The token is to compensate the participant (any licensed clinician whose scope of practice includes diagnosis) for the burden of the information collection activities and time away from standard work duties. Up to a $100 stipend will be offered for completing all proposed information collection instruments including two surveys and two 60-minute interviews.
Based on the contractor (MedStar’s) previous recruitment efforts and based on recent literature citing lack of compensation for clinician time as contributing to increasing clinician burnout and drop-out, an incentive based on standard GSA rates represents the minimum amount necessary to recruit and secure participation in information collection efforts.10 The $100 stipend11 also serves as a token of appreciation should clinicians lose clinical RVUs (relative value units) by participating in the interviews. From prior direct experience in which clinicians are frequently offered stipends as appreciation for allocating their time and expertise towards quality improvement efforts, failure to offer any incentive or an amount lower than this will hamper the ability to successfully recruit the number of clinicians required for the proposed data collection.
Individuals and organizations will be assured of the confidentiality of their replies under Section 944(c) of the Public Health Service Act. 42 U.S.C. 299c-3(c). That law requires that information collected for research conducted or supported by AHRQ that identifies individuals or establishments be used only for the purpose for which it was supplied.
Information that can directly identify the respondent, such as name and/or social security number will not be collected. Participant characteristics that will be collected for individuals who participate in pilot testing will be limited to information about their functional roles within their organizations (see Appendix C). This information will be presented in aggregate and used to describe the characteristics of the group providing feedback. No information will allow for individual identification of participants.
Participants will also receive the following confidentiality statements printed on any respondent materials:
“The confidentiality of your responses are protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)]. Information that could identify you will not be disclosed unless you have consented to that disclosure.”
Information collected will be maintained in a secure HIPAA-compliant data server. All information collection will be stored using the contractor, MedStar’s, REDCap™ research data capture database. REDCap™ is a mature, secure web application for building and managing online information collection instruments and data. While REDCap™ can be used to collect virtually any type of data, it is specifically geared to support data capture for research studies. The REDCap™ Consortium is composed of 1,711 active institutional partners in 96 countries who utilize and support REDCap™ in various ways. REDCap™ can be established to support data entry forms and to conduct web-enabled surveys. This project will also use a REDCap™ project space to securely store any documents received from the practices during the project. The MedStar Health Research Institute is a REDCap™ project collaborator site with a robust history of using this method for data collection.
The information collected will be primarily qualitative in nature. Responses to interviews and/or focus groups will be recorded and notes for each session transcribed. Transcription files will be uploaded to the REDCap™ file repository for security. Paper files will be retained in a locked file cabinet within MedStar Health. A case record form will be created within REDCap™ to record output from the thematic reviews of the transcripts to facilitate reporting and feedback to participants and to the study team. This information collection request (ICR) does not contain surveys, censuses, or employ statistical methods.
This ICR does not request any personally identifiable information.
This ICR does not include a form that requires a Privacy Act Statement.
Does this ICR contain surveys, censuses, or employ statistical methods? Yes No
Does this ICR request any personally identifiable information (see OMB Circular No. A-130 for an explanation of this term)? Please consult with your agency's privacy program when making this determination Yes No
Does this ICR include a form that requires a Privacy Act Statement (see 5 U.S.C. §552a(e)(3))? Please consult with your agency's privacy program when making this determination. Yes No
The proposed information collection does not include any questions of a sensitive nature. Each respondent will undergo an informed consent process that will describe participant rights. We anticipate that the MedStar Health Research Institute’s Intuitional Review Board will grant a waiver of documentation for written consent as the consent form will be the only documentation linking the participant’s identity to the information collection. Each participant will receive a copy of the study information sheet for verbal consent and an information sheet on the project that outlines the participant’s rights as is standard for MedStar’s exempt studies where active recruitment is required. The consent process will highlight the participants right to answer or not answer any questions that they are asked and their right to withdraw from the interview at any time without penalty or repercussions.
Exhibit 1. Estimated annualized burden hours
Form Name |
Number of respondents |
Number of responses per respondent |
Hours per response |
Total burden hours |
Appendix A: Individual Respondent Characteristics Survey |
20 |
1 |
0.25 |
5 |
Appendix B: Safety Attitude Survey |
20 |
1 |
0.25 |
5 |
Appendix C: Pre-Test Interview Protocol |
20 |
1 |
1 |
20 |
Appendix D: Post-test Evaluation Interview Protocol |
20 |
1 |
1 |
20 |
Total |
NA |
NA |
NA |
Exhibit 2. Estimated annualized cost burden
Form Name |
Number of respondents |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
Appendix A: Individual Respondent Characteristics Survey |
20 |
5 |
$103.06a |
$515.30 |
Appendix B: Safety Attitude Survey |
20 |
5 |
$103.06a |
$515.30 |
Appendix C: Pre-Test Interview Protocol |
20 |
20 |
$103.06a |
$2,061.20 |
Appendix D: Post-test Evaluation Interview Protocol |
20 |
20 |
$103.06a |
$2,061.20 |
Total |
NA |
50 |
NA |
$5,153.00 |
* National Compensation Survey: Occupational wages in the United States May 2020 “U.S. Department of Labor, Bureau of Labor Statistics.”
a Based on the mean wages for Family Medicine Physicians (Code 29-1215)
There are no direct costs to respondents other than their time to participate in the study.
The total contractor cost to the government is estimated to be $202,738.56. As shown in Exhibit 3a, this amount includes costs for project development ($18,812.50); data collection activities ($93,152.31); data processing and analysis ($24,178.47), project management ($18,812.50) and overhead ($47,782.78).
Exhibit 3a. Estimated Total and Annualized Cost
Cost Component |
Total Cost |
Annualized Cost |
Project Development |
$18,812.50 |
$18,812.50 |
Data Collection Activities |
$93,152.31 |
$93,152.31 |
Data Processing and Analysis |
$24,178.47 |
$24,178.47 |
Publication of Results |
$0 |
$0 |
Project Management |
$18,812.50 |
$18,812.50 |
Overhead |
$47,782.78 |
$47,782.78 |
Total |
$202,738.56 |
$202,738.56 |
Exhibit 3b. Federal Government Personnel Cost
Activity |
Federal Personnel |
Hourly Rate |
Estimated Hours |
Cost |
Project oversight to include data collection oversight and review of results |
|
$81.84 |
25 |
$2046 |
Total |
$ 2046 |
|||
Annual salaries based on 2020 OPM Pay Schedule for Washington/DC area: https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2020/DCB_h.pdf
This is a new information collection, thus no changes in hour burden is expected or reported here.
The information collection will begin upon OMB approval (estimated August 2021) and will include recruitment of practices and completion of all data collection activities by January 2022. Qualitative analysis will be completed by February 2022 and materials will be revised based on stakeholder feedback by March 2022. We anticipate a 6-month pilot test timeline. Publication of the materials by AHRQ on their website will be completed after 508 compliance review. There are no other publication efforts associated with this information collection effort.
AHRQ does not seek this exemption.
Appendix A: Individual Respondent Characteristics Survey
Appendix B: Safety Attitudes Survey
Appendix C: Pre-Test Interview Protocol
Appendix D: Post-test Evaluation Interview Protocol
1. Leape LL, Berwick DM, Bates DW. Counting Deaths Due to Medical Errors—Reply.
JAMA. 2002;288(19):2405. doi:10.1001/jama.288.19.2405-jlt1120-2-3
Newman-Toker DE, Pronovost PJ. Diagnostic errors the next frontier for patient safety.
JAMA - J Am Med Assoc. 2009;301(10):1060-1062. doi:10.1001/jama.2009.249
Saber Tehrani AS, Lee HW, Mathews SC, et al. 25-Year summary of US malpractice claims for diagnostic errors 1986-2010: An analysis from the National Practitioner Data Bank. BMJ Qual Saf. 2013;22(8):672-680. doi:10.1136/bmjqs-2012-001550
Newman-Toker DE, Schaffer AC, Yu-Moe CW, et al. Serious misdiagnosis-related harms in malpractice claims: The “Big Three” – vascular events, infections, and cancers. Diagnosis. 2019;6(3):227-240. doi:10.1515/dx-2019-0019
National Academies of Sciences, Engineering, and Medicine. 2015. Improving Diagnosis in Health Care. Washington, DC: The National Academies Press. https://doi.org/10.17226/21794.
Meyer AN, Payne VL, Meeks DW, et al. Physicians’ diagnostic accuracy, confidence, and resource requests: a vignette study. JAMA Intern Med. 2013;173(21):1952-1958. Doi:1731967[pii];10.1001/jamainternmed.2013.10081 [doi]
Berner ES, Graber ML. Overconfidence as a cause of diagnostic error in medicine. Am J Med. 2008;121(5, Supplement 1):S2-S23. doi:10.1016/j.amjmed.2008.01.001
Meyer AN, Singh H. Calibrating how doctors think and seek information to minimise errors in diagnosis. BMJ Qual Saf. 2016;Online First.
Sexton JB, Helmreich RL, Neilands TB, et al. . The Safety Attitudes Questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Services Research. 2006;6:44.
Stehman CR, Testo Z, Gershaw RS, et al. Burnout, Drop Out, Suicide: Physician Loss in Emergency Medicine, Part I. West J Emerg Med, 2019;20(3), 485–494. https://doi.org/10.5811/westjem.2019.4.40970
U.S. Department of Labor, Bureau of Labor Statistics. (May 2020). National Compensation Survey: Occupational wages in the United States. Retrieved from https://www.bls.gov/oes/current/oes_nat.htm
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Andrea Bradford |
| File Modified | 0000-00-00 |
| File Created | 2023-11-13 |