SUPPORTING STATEMENT
Part A
Implementation and Testing of Diagnostic Safety Resources
Version: May 22, 2024
Agency of Healthcare Research and Quality (AHRQ)
Table of contents
A. Justification 3
1. Circumstances That Make the Collection of Information Necessary.....................3
2. Purpose and Use of Information 7
3. Use of Improved Information Technology 9
4. Efforts to Identify Duplication 9
5. Involvement of Small Entities 10
6. Consequences if Information Collected Less Frequently 10
7. Special Circumstances 11
8. Consultation outside the Agency 11
9. Payments/Gifts to Respondents 11
10. Assurance of Confidentiality 12
11. Questions of a Sensitive Nature 13
12. Estimates of Annualized Burden Hours and Costs 14
13. Estimates of Annualized Respondent Capital and Maintenance Costs 17
14. Estimates of Annualized Cost to the Government 17
15. Changes in Hour Burden 17
16. Time Schedule, Publication and Analysis Plans 18
17. Exemption for Display of Expiration Date 18
List of Attachments 19
References…………………………………………………………………………..21
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see Healthcare Research and Quality Act of 1999 | Agency for Healthcare Research and Quality (ahrq.gov)), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. research that develops and presents scientific evidence regarding all aspects of health care; and
2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
Background for this information collection
Patient safety is a pillar of the Agency for Healthcare Research and Quality’s (AHRQ’s) mission to support the highest quality healthcare. While progress has been made in many areas of patient safety, the field of diagnostic safety has emerged as a particular area of concern. It is estimated that every person in the United States will experience a diagnostic error in their lifetime (Institute of Medicine, 2015) which can lead to inappropriate, delayed, or withheld treatment and ultimately poor health outcomes, distress, and increased costs. Each year, 795,000 Americans are estimated to become permanently disabled or die annually across care settings due to misdiagnosis (Newman-Toker, 2024). Analyses of malpractice claims have found that the conditions most commonly associated with harms due to diagnostic errors are vascular conditions, infections, and cancers (Newman-Toker, et al., 2019). Diagnostic errors occur in every setting. Five percent of U.S. adults who receive outpatient care in a given year experience a diagnostic error (Singh and Thomas, 2014). In the hospital setting, 6 to 17 percent of adverse events are associated with diagnostic error (Leape, et al., 1991; Zwaan, et al., 2010). Given that diagnostic error measurement is limited by a lack of precise data and standardized approaches, these rates are likely underestimates. Moreover, rates are likely even higher in vulnerable populations for whom diagnostic safety events are more prevalent and systematically undercounted due to implicit and structural bias in safety reporting systems (Giardina, Woodard, and Singh, 2023; Schulson, et al., 2020). Diagnostic errors can occur for many reasons: lack of meaningful engagement between clinicians, patients, and families; a fragmented healthcare system not designed to account for an increasingly complex diagnostic process; minimal (if any) feedback to clinicians about their diagnostic performance; and a culture that does not always support transparent disclosure of diagnostic errors (Institute of Medicine, 2015). Leaders in diagnostic excellence suggest that multi-pronged efforts are needed to address this complex problem and go beyond individual behaviors to system-level changes and empowering patients to engage in their care (Institute of Medicine, 2015; Henriksen, et al., 2017).
Improving diagnostic safety and quality is an AHRQ priority. In recognition of the multifaceted approach needed to effectively advance diagnostic safety, AHRQ recently supported the development of three tools to prevent diagnostic errors and have prioritized these tools for implementation and testing. These resources vary in the types of stakeholders they target, a critical advancement in our approach to diagnostic excellence.
Calibrate Dx. This tool, targeted to individual clinicians, invites users to select a topic or condition, review diagnostic performance on a sample of cases for insights and learning opportunities, and debrief with a peer. This resource will be tested in all settings where clinicians are involved in the diagnostic process, including both inpatient and ambulatory settings.
Measure Dx. This tool supports healthcare organizations in building sustainable teams for improving diagnostic excellence, identifying current capacity gaps, engaging in measurement strategies as part of a systematic approach to reviewing available data, and translating findings into learning opportunities. This resource will be tested in both inpatient and ambulatory settings; it is expected to be implemented more commonly in inpatient settings.
Toolkit for Engaging Patients to Improve Diagnostic Safety (Patient Toolkit). This tool prepares patients, families, and health professionals to work together as partners to improve diagnostic safety; encourages patients to prepare for visits; and encourages providers to listen for 60 seconds before interrupting the patient. This resource will be tested in ambulatory settings only.
The goal of this research is to implement and test these three diagnostic safety resources to identify specific ways in which each resource can be used to maximize its value. For each resource the following will be examined:
1) Feasibility of implementation - barriers, facilitators, success factors, and time needed for implementation
2) Level of adoption - number and type of clinicians aware of and/or using the resource, number of organizational leaders endorsing the resource
3) Effectiveness of the resource - number of diagnostic safety events (Measure Dx and Patient Toolkit), clinician self-efficacy for diagnostic decision-making (Calibrate Dx)
4) Maintenance and sustainability – the number and type of patient safety processes in place, barriers and facilitators to maintenance and sustainability
This project will implement and test these three diagnostic safety resources across a minimum of 150 sites to up to 219 sites (i.e., 50 to 73 sites per resource). An Implementation and Testing period for each resource will last 12 months, with Calibrate Dx starting implementation first and Measure Dx and the Toolkit for Engaging Patients starting implementation six months later. This timing allows for staggered recruitment to ensure adequate sample size and to pilot implementation processes with a single diagnostic safety resource first, transferring lessons learned about implementation and testing to the implementation of the two other resources. A Sustainability period will begin as soon as the 12-month Implementation and Testing Period is complete and will continue for 14 additional months for each resource.
To achieve the goals of this project the following data collections will be implemented:
Site Interest Form (Attachment A). A short form completed once by up to 1060 sites interested in participating in the project. Used to indicate interest in the project and by AHRQ to evaluate whether the site meets the minimum participation criteria.
Site Information Form (Attachment B). Completed once by site leaders at 265 sites that begin the project enrollment process, this form collects additional contact information, data on patient mix, and information on the organization’s diagnostic safety teams, resource commitments, and capacity for implementing the resources.
Safer Dx Checklist (Attachment C). Completed once by 219 sites who fully complete enrollment activities and begin implementation of one of the three resources (82.6% of the 265 sites who begin enrollment activities). The Safer Dx Checklist is a tool that allows healthcare organizations to understand the current state of their diagnostic practices, identify areas to improve, and track progress toward diagnostic excellence over time. This will be completed prior to actual implementation of the resource.
Exit Interviews Protocol (Attachment D). Completed once by an estimated 69 sites (30% of those implementing one of the three resources) that withdraw from the project, this telephone interview will collect information on why the site could not sustain their efforts or participation.
A baseline assessment of patient safety culture will be conducted once for each of the 219 sites that begin participation. Completed once by site leads depending on the setting:
SOPS® Medical Office Survey with Diagnostic Safety Supplemental Item Set (Attachment E): Completed once by the site lead for 109 ambulatory clinics.
SOPS® Hospital Survey with Diagnostic Safety Supplemental Item Set (Attachment F): Completed once by the site lead for 110 inpatient sites.
Post-training Evaluation Form (Attachment G): Completed once by 1,350 clinicians and managers (90% of the 1,500 participants) attending the project’s training sessions. The data will be used to track the perceived value of the training provided to enrolled sites.
Post-technical Assistance Evaluation Form (Attachment H): Administered up to 3 times to 1,500 clinicians and managers participating in the project’s Learning Collaborative sessions; an estimated 90% response rate to this collection with a total of 4,050 forms completed. The data will be used to track the perceived value of the technical assistance provided to enrolled sites.
Clinical Sustainability Assessment Tool (CSAT) (Attachment I): Completed by 219 site leaders once between months 9 to 12 in advance of the 14-month sustainability period. The CSAT is a self-assessment to evaluate sustainability capacity of a clinical practice.
Implementation Interviews Protocol (Attachment J). A qualitative, semi-structured interview will be conducted with 438 site leads and/or frontline staff (up to 2 individuals from each site) at two points in time during implementation (e.g., 6- and 18- months). The protocol is designed to elicit participant perspectives on implementation of the resource, capture lessons learned and best practices, and when possible, to provide support for adjustment to the implementation.
In addition to those noted above, the project will implement the following data collections specific to the individual resources.
For Measure Dx, the following data collections will be implemented:
Measure Dx Organizational Self-Assessment (Attachment K). This is one of the main components of the Measure Dx resource and is designed to gauge the organization’s readiness to engage with Measure Dx. This checklist will be completed once by up to 73 Measure Dx sites during the project onboarding process.
Measure Dx Declaration of Measurement Strategy (Attachment L). The 73 Measure Dx sites will complete this form once to indicate their selection of measurement strategy to be implemented and provide confirmation of minimum necessary capabilities.
Diagnostic Safety Event Report (Attachment M). These reports will provide aggregate information on diagnostic safety events identified during a 12-month reporting period. The report will be completed by each participating site 3 times over the course of the testing and sustainability period at 3-, 12-, and 24-months; a total of 219 reports will be completed over the course of the project. Note that the contractor is not attempting to collect these reports at Month 0. Since part of the Measure Dx resource’s goal is to support implementation of a measurement strategy, Month 3 will serve as the baseline.
Additional information on site safety culture, including use of diagnostic safety event data, activities to improve the quality of care, and the work environment will be collected through a survey at 3-, 12-, and 24-months during the implementation/sustainment. Five members of the Measure Dx team at each site will be surveyed; the expected response rate is 85% at each of the three administration periods. Depending on the setting, the following survey will be fielded:
Omnibus Safety and Culture Survey_Medical Offices (Attachment N). Completed by clinicians at 36 ambulatory clinics.
Omnibus Safety and Culture Survey_Hospitals (Attachment O). Completed by clinicians at 37 inpatient sites.
For Calibrate Dx, the following data collections will be implemented:
Calibrate Dx Survey (Attachment P). This survey collects clinicians’ reflections on their diagnostic performance for 3-5 cases, with additional metrics around time to complete the review and the number of cases reviewed. This will be completed quarterly (following the Calibrate Dx guidance for implementation) during the implementation and testing period by up to 5 clinicians per site; with an estimated a 90% response rate to this collection.
Clinician Self-Efficacy Survey (Attachment Q). The survey assesses clinician self-efficacy with diagnostic safety case review and improvement. Up to 5 clinicians per site will be asked to complete this survey two times, after training and again at the end of the testing phase; with an estimated 90% response rate to this collection.
For Patient Toolkit, the following data collections will be implemented:
Provider Characteristics Form (Attachment R). This form will be completed once by up to 15 providers at each of the 73 enrolled sites. This form collects information on practitioner type, years in practice, specialty, subspecialty, and percent of time spent in clinical practice.
Patient Toolkit Survey-Provider (Attachment S). This survey assesses provider-perceived skills and quality of communication. It will be administered to up to 15 providers at each site at five timepoints (Baseline, 3-, 6-, 9-, and 12-months), with a 90% anticipated response rate.
Provider Interview Protocol (Attachment T). A total of 50 qualitative, semi-structured interviews with site clinicians will be conducted during implementation. The interview protocol collects information related to diagnostic safety events; patient safety culture; feasibility, acceptability, utility, adoption, and spread of the Patient Toolkit; and insights into clinician experience.
Patient Toolkit Survey - Patient (Attachment U). The survey assesses patient-perceived experience and quality of communication, and collects basic patient demographics (e.g., age, gender, education, race, ethnicity). This will be administered to site patients over a 1-week period at five timepoints (Baseline, 3-, 6-, 9-, and 12-months). The survey will be provided to patients upon check-out from a healthcare visit. A total of 12,500 surveys will be completed during each 1-week period.
Patient Interview Protocol (Attachment V). A total of 50 qualitative, semi-structured interviews will be completed with site patients during implementation. The interview protocol collects information on reason for visit, provider communication, and other insights into patient experience.
This study is being conducted by AHRQ through its contractor, RAND, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).
Healthcare organizations and clinicians are looking for actionable, practical, and flexible resources to support diagnostic safety that have been rigorously evaluated. Currently, few such tools exist. The goal of this project is to implement and test three diagnostic safety resources, Calibrate Dx, Measure Dx, and the Patient Toolkit, to identify specific ways in which each resource can be used to maximize its value. AHRQ will use the collected information to evaluate the feasibility of implementation of the Calibrate Dx, Measure Dx, and Patient Toolkit resources by organizations and their adoption by clinicians, the effectiveness of the resources in supporting diagnostic safety, and the ability of organizations to sustain these resources over time.
The specific purpose of each of the information collection instruments is described below:
During recruitment and enrollment of sites, the Site Interest Form will be used to identify potential sites for this effort, and to evaluate whether the site meets the minimum participation criteria. The Site Information Form will be used to collect contact information and for describing the characteristics of participating organizations and their experience with and capacity for implementing diagnostic safety resources. The Safer Dx Checklist will be used to assess the existing diagnostic safety practices of sites. Should a site withdraw, the Exit Interviews Protocol will be used to collect information on the reasons why sites could not sustain their efforts or participation.
Several data collection instruments will be used in this evaluation across all three resources. The SOPS® Medical Office Survey with Diagnostic Safety Supplemental Item Set or SOPS® Hospital Survey with Diagnostic Safety Supplemental Item Set will be used provide a baseline assessment of patient safety culture at all the participating sites. To evaluate training and technical assistance support provided as part of implementation, the Post-training Evaluation Form and Post-technical Assistance Evaluation Form will be used to track the perceived value of the training and technical assistance provided to enrolled sites and the Implementation Interviews Protocol will provide insight from clinicians, QI team members, executives, and other site participants on emerging/identified implementation barriers and facilitators and lessons learned. The Clinical Sustainability Assessment Tool (CSAT) will be used to understand and compare the sustainability capacity across participating sites.
Specific to the Measure Dx evaluation, and as one of the main components of the resource, the Measure Dx Organizational Self-Assessment will assess the sites’ readiness to implement the resource and the Measure Dx Declaration of Measurement Strategy will collect the selection of measurement strategy to be implemented and provide confirmation of the capabilities to do so. During implementation, the Diagnostic Safety Event Reports will provide aggregate information on diagnostic safety events to aid in assessing the effectiveness of the Measure Dx resource and the Omnibus Safety and Culture Survey_Medical Offices or Omnibus Safety and Culture Survey_Hospitals will be used to assess changes in safety and culture during the Measure Dx implementation (also a measure of effectiveness of the resource).
Specific to the Calibrate Dx evaluation, the Calibrate Dx Survey will assess the adoption of the resource by clinicians and its effectiveness based on their reflections of diagnostic process, self-assessment of the case review process, and identification of improvement areas. The Clinician Self-Efficacy Survey will provide an assessment of clinician self-efficacy with diagnostic safety case review and improvement.
For the Patient Toolkit, the Provider Characteristics Form will allow for an accounting of the types of providers who have adopted the resource. Interviews with providers about their experience using the PFE Toolkit, including challenges and successes, will be used to evaluate implementation of this resource (Provider Interview Protocol). Two surveys and patient interviews will contribute to the evaluation of the effectiveness of this resource; the Patient Toolkit Survey-Provider assesses provider-perceived skills and quality of communication; and the Patient Toolkit Survey - Patient, along with the Patient Interview Protocol, will assesses patients’ experiences with provider communication.
The data collection methods for this evaluation were selected to reduce participant burden and, where possible, to allow participants a choice of response mode. In addition, technology is used for data capture and qualitative coding and analysis.
Several forms and data collection instruments will be administered using a web mode (Attachments A, E, F, G, H, I, N, and O). Site leads and participants will receive a link allowing them to complete the form online. The Site Interest Form will also be accepted as a hardcopy should organizations prefer to mail or fax these forms. All other forms (Attachments B, C, K, L, M, P, Q, R, and S) will be administered either by a fillable form that can be returned via email, mail, or fax depending on the site or participant preference.
Interviews (Attachments D, J, T and V) will be conducted by phone or video call (e.g., Microsoft Teams, Zoom) with interviewers using a hardcopy version of the protocol. Interviews will be audio-recorded and transcribed, following verbal consent from participants. Qualitative software will be used for data coding and analysis of interviews.
The patient surveys will be provided to patients upon check-out from a healthcare visit and they will be encouraged to complete the survey before leaving the office. The survey will include a QR code to allow patients to access a web version of the form. Alternatively, the patient can complete the paper survey and it will be collected at the site, minimizing the need for patients to return the paper survey by mail. The paper surveys will be formatted for data scanning, and data from all paper surveys returned to the contractor will be scanned into an electronic datafile.
AHRQ previously supported a contract to develop the CalibrateDx, MeasureDx, and Patient Toolkit resources. At that time, environmental scans confirmed that few resources for healthcare organizations with the specific goal of improving diagnostic quality and safety existed; this remains the case. After development of the CalibrateDx, MeasureDx, and Patient Toolkit, they were pilot tested for feasibility and revised (OMB No. 0935-0179, OMB No. 0935-0263). However, as these pilot tests only included a small number of organizations, AHRQ is now interested in implementing the resources more broadly and testing the impact of their use on relevant process and outcome measures, gathering more user experiences with the resources, and understanding issues associated with sustainability. To our knowledge, this does not involve a duplication of any existing efforts as suggested by gap analyses resulting from the environmental scans.
This information collection will include a range of health care settings in which the Calibrate Dx, Measure Dx, and the Patient Toolkit resources will ultimately be implemented, including health systems, hospitals, and clinics. While the information collection may include small entities, as some of the participating sites may employ a small staff, all data collection instruments and procedures are designed to minimize burden on participating sites and individual respondents. The data requested represent the absolute minimum information required for the intended uses, and the data submission process does not unduly burden small hospitals, clinics, or other organizations.
This data collection effort is in support of an evaluation of three diagnostic safety resources; Calibrate Dx, Measure Dx, and the Patient Toolkit. The frequency of the data collection activities is necessary to comprehensively assess the feasibility of implementation, level of adoption, effectiveness, and sustainability of each resource.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
8.a. Federal Register Notice
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on March 7, 2024 and page 16574 for 60 days (see Attachment W).
Consultations on the evaluation design, data collection approach and instruments – including the selection and modification (if applicable) of existing instruments such as the Safer Dx Checklist, SOPS® Surveys, and the Clinical Sustainability Assessment Tool (CSAT) as well as the interview protocols, occurred through the proposal and planning phase of this effort (i.e., June 2023 through December 2023). These consultations have provided, and will continue to provide, the opportunity to ensure the technical quality and appropriateness of the overall evaluation design, minimal burden to participant sites, and fidelity to the intent of the original instruments. Consultations have occurred with the following individuals in connection with this study (listed in alphabetical order):
Kisha Ali, Research Scientist, MedStar Health Institute for Quality and Safety
Gerard Castro, Director of Quality Improvement, Society to Improve Diagnosis in Medicine
Patricia McGaffigan, Vice President, Institute for Healthcare Improvement
Jeffrey Salvon-Harmon, VP Safety, Institute for Healthcare Improvement
Bruce Spurlock, President and CEO, Convergence Health Consulting
Kim Werkmeister, Senior Vice President, Improvement & Implementation, Convergence Health Consulting
This project is offering two types of payments to support site and respondent engagement, incentives to individual respondents and site-level honoraria as a token of appreciation for site participation. According to OMB guidance, agencies may offer incentives if they are necessary to ensure the collection of high-quality data, to reduce costs, when the study design is multifaceted, and/or to encourage response from selective groups, such as physicians. OMB further indicates that past study experience can provide justification for the use of incentives (OMB, 2016). Considerable resources will be expended in the recruitment and enrollment of sites and training and technical assistance provided to site staff participating in the evaluation. In the contractor’s experience, providing modest incentives for portions of the data collection will facilitate recruitment of study sites, reduce attrition of sites and individual participants, and increase response to surveys and interviews.
While physicians are notoriously difficult to survey, considerable research has shown that monetary incentives can significantly increase their survey response (VanGeest, Johnson, and Welch VL, 2007; Ziegenfuss, et al., 2011). Research has also shown that monetary incentives increase participation in qualitative interviews (Margolis, et al., 2017). It has been noted that offering incentives can have a positive impact on study recruitment and on data quality by ensuring a more diverse respondent pool (Stähli and Joye, 2016).
To maximize clinician and site staff response to the Omnibus Safety and Culture Survey (Medical Offices and Hospitals versions), surveys that are critical to the evaluation of the Measure Dx resource, a token of appreciation of $25 will be provided to responders. To encourage participation in interviews that are key to assessing implementation barriers for all resources (Implementation Interviews Protocol) and the effectiveness of the Patient Toolkit (Provider Interview Protocol and Patient Interview Protocol), clinicians, site staff and patients who participate in these data collections will be provided with a token of appreciation of $40.
In addition to individual incentives, a small honorarium to sites as a token of appreciation will be offered. While research has shown that incentives can have a positive impact on participant recruitment (Stähli and Joye, 2016), the contractor has also noted this positive effect for recruiting and maintaining site participation in previous research they have conducted. Thusly, to facilitate site involvement in this effort, a small honorarium of $1500 will be provided as a token of appreciation to sites who complete the project enrollment.
Strategic use of incentives and honoraria will ultimately reduce the overall costs for the activities by shortening the time for recruitment activities, limiting the need to replace individual participants, and by minimizing the size of the samples for surveys and interviews.
Data will be kept private to the extent allowed by law. Individuals and organizations will be assured of the confidentiality of their replies under Section 944(c) of the Public Health Service Act. 42 U.S.C. 299c-3(c). That law requires that information collected for research conducted or supported by AHRQ that identifies individuals or establishments be used only for the purpose for which it was supplied.
Some information that can directly identify the respondent, such as name will be collected. However, except for site name and personal name and contact information, no other personally identifiable information (PII) will be collected from clinicians and other staff members participating in the evaluation. This information is needed to enroll sites, provide training and technical support to participants, conduct interviews, and administer forms and surveys. Specific to data collection for the Patient Toolkit resource, patient surveys include a place for patient respondents to voluntarily list name and contact information, if the patient is interested in participating in a patient interview. Other than this, no identifiers or ID numbers linking name of patient to data collection instruments will be collected. Participating sites sign a data use agreement detailing how site data will be received and used.
Protocols for data collection, storage, and analyses will be approved by the contractor’s Institutional Review Board and followed accordingly. Files containing PII will be stored on internal servers in password-protected locations only accessible by approved project staff and/or stored with passwords and file encryption on fixed or removable systems/media. Hardcopy materials containing PII will be stored in locked file cabinets when not in use.
Names and contact information for participants will be replaced with anonymized IDs in analytic files. No participant names or contact information will be included in any reports, presentations, or other publications emerging from this project. Only aggregated, de-identified results will be displayed in any reports. All PII from this project will be destroyed 1 year following completion of the project.
The following confidentiality statement will be printed on data collection instruments (e.g., surveys, forms) or incorporated into verbal protocols:
This survey is authorized under 42 U.S.C. 299a. This information collection is voluntary and the confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)]. Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average x minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The data you provide will help AHRQ’s mission to produce evidence to make health care safer, higher quality, more accessible, equitable, and affordable. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (OMB control number 0935-xxxx) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857, or by email to [email protected].
Personally identifiable information (PII), including names, addresses, and email for site staff participating in the evaluation, will be collected for training, technical assistance, interviews, and administration of the surveys and other data collection forms. Names, addresses, and email for patients who volunteer for interviews will also be collected.
Surveys and other data collection forms do not ask for information of a sensitive nature (e.g., sexual behavior and attitudes, religious beliefs) other than gender and race/ethnicity that is collected in the patient survey. In addition, the patient survey collects data on the patient’s assessment of communication with their provider. Collection of these data from patients is necessary for the evaluation in order to evaluate the effectiveness of the Patient Toolkit and describe the patients who participated in the evaluation.
While patient surveys include a place to list name and contact information if the patient would like to participant in an interview (for Patient Toolkit resource only), this is voluntary. Otherwise, no identifiers or ID numbers linking name of patient to this survey will be collected.
While it is expected that the contractor’s Institutional Review Board will grant a waiver of documentation for written consent (as the consent form will be the only documentation linking the participant’s identity to the information collection), all participants in these data collection activities will undergo an informed consent process that will describe participant rights. All data that includes PII from participants will be stored in the secure facilities for no more than 1 year after the study is completed, and Federal requirements regarding collection and storage of PII will be adhered to.
This section summarizes the total burden hours for this information collection effort in addition to the cost associated with those hours.
Exhibit 1 contains estimated response burdens for each subject population participating in the evaluation’s data collection activities.
1. Site Interest Form – A physician or manager at an interested site will complete the form only once to indicate interest in participating. The form will be completed by 1,060 respondents and requires 6 minutes to complete.
2. Site Information Form – A physician or manager at an interested site will complete the form only once to provide additional contact information, data on patient mix, and information on the organization’s diagnostic safety teams, resource commitments, and capacity for implementing the resources. The form will be completed by 265 respondents and requires 20 minutes to complete.
3. Safer Dx Checklist – A physician or manager at participating sites will complete the form only once to allow the participating site to understand the current state of their diagnostic practices, identify areas to improve, and track progress toward diagnostic excellence over time. The form will be completed by 219 respondents and requires 15 minutes to complete.
4. Exit Interviews Protocol – A physician or manager at sites that withdraw from the project will complete the form once to provide information on why the site could not sustain their efforts or participation. The form will be completed by 69 respondents and requires 10 minutes to complete.
5a. SOPS® Medical Office Survey with Diagnostic Safety Supplemental Item Set – A physician or manager at participating ambulatory sites will complete the form to provide a baseline assessment of patient safety culture. The form will be completed by 109 respondents and requires 15 minutes to complete.
5b. SOPS® Hospital Survey with Diagnostic Safety Supplemental Item Set – A physician or manager at participating hospital sites will complete the form to provide a baseline assessment of patient safety culture. The form will be completed by 110 respondents and requires 15 minutes to complete.
6. Post-training Evaluation Form – A physician, nurse practitioner, physician assistant, or manager will complete the form once to indicate the perceived value of the training provided to participating sites. The form will be completed by 1350 respondents and requires 3 minutes to complete.
7. Post-technical Assistance Evaluation Form - A physician, nurse practitioner, physician assistant, or manager will complete the form up to three times to indicate the perceived value of the technical assistance provided to participating sites. The form will be completed by 1350 respondents, three times, and requires 2 minutes to complete.
8. Clinical Sustainability Assessment Tool (CSAT) – A physician or manager at participating sites will complete the form to evaluate the sustainability capacity of a clinical practice. The form will be completed by 219 respondents and requires 15 minutes to complete.
9. Implementation Interviews Protocol – A physician, nurse practitioner, physician assistant, or manager will participate in an interview two times to provide their perspectives at different stages of the implementation. The interview will be completed by up to 438 respondents, two times, and requires 1 hour to complete.
10. Measure Dx Organizational Self-Assessment - A physician, nurse practitioner, physician assistant, or manager will complete the form only once to gauge the organization’s readiness to engage with Measure Dx. The form will be completed by 73 respondents and requires 30 minutes to complete.
11. Measure Dx Declaration of Measurement Strategy - A physician, nurse practitioner, physician assistant, or manager will complete the form only once to indicate their selection of measurement strategy to be implemented and provide confirmation of minimum necessary capabilities. The form will be completed by 73 respondents and requires 5 minutes to complete.
12. Diagnostic Safety Event Report - A physician, nurse practitioner, physician assistant, or manager will complete the form three times to provide aggregate information on diagnostic safety events. The form will be completed by 73 respondents, three times, and requires 1 hour to complete.
13a. Omnibus Safety and Culture Survey_Medical Offices - A physician, nurse practitioner, physician assistant, or manager will complete the form three times to provide information on safety culture at ambulatory sites. The form will be completed by 162 respondents, three times, and requires 20 minutes to complete.
13b. Omnibus Safety and Culture Survey_Hospitals - A physician, nurse practitioner, physician assistant, or manager will complete the form three times to provide information on safety culture at inpatient sites. The form will be completed by 167 respondents, three times, and requires 20 minutes to complete.
14. Calibrate Dx Survey - A physician, nurse practitioner, or physician assistant will complete the form four times to provide reflections on their diagnostic performance for 3-5 cases, with additional metrics around time to complete the review and the number of cases reviewed. The form will be completed by 329 respondents, four times, and requires 30 minutes to complete.
15. Clinician Self-Efficacy Survey - A physician, nurse practitioner, or physician assistant will complete the form two times to provide information on their self-efficacy with diagnostic safety case review and improvement. The form will be completed by 329 respondents, two times, and requires 3 minutes to complete.
16. Provider Characteristics Form - A physician, nurse practitioner, or physician assistant will complete the form once to provide information on practitioner type, years in practice, specialty, subspecialty, and percent of time spent in clinical practice. The form will be completed by 986 respondents and requires 1 minute to complete.
17. Patient Toolkit Survey-Provider – A physician, nurse practitioner, or physician assistant will complete the form five times to provide information on provider-perceived skills and quality of communication. The form will be completed by 986 respondents, five times, and requires 2 minutes to complete.
18. Provider Interview Protocol - A physician, nurse practitioner, or physician assistant will participate in an interview once to provide information related to diagnostic safety events; patient safety culture; feasibility, acceptability, utility, adoption, and spread of the Patient Toolkit; and insights into clinician experience. The interview will be completed by up to 50 respondents and requires 45 minutes to complete.
19. Patient Toolkit Survey - Patient – Patients will complete the form only once to provide information on their experience and quality of communication, and demographics information. The form will be completed by 62,500 respondents and requires 5 minutes to complete.
20. Patient Interview Protocol – Patients will participate in an interview once to provide information on reason for visit, provider communication, and other insights into patient experience. The interview will be completed by up to 50 respondents and requires 45 minutes to complete.
For the three-year clearance period, the estimated annualized burden hours for the data collection activities are 8195.
Exhibit 1. Estimated annualized burden hours
Form Name |
Number of respondents |
Number of responses per respondent |
Hours per response |
Total burden hours |
1: Site Interest Form |
1060 |
1 |
6/60 |
106 |
2: Site Information Form |
265 |
1 |
20/60 |
88 |
3: Safer Dx Checklist |
219 |
1 |
15/60 |
55 |
4: Exit Interviews Protocol |
69 |
1 |
10/60 |
12 |
5a: SOPS® Medical Office Survey with Diagnostic Safety Supplemental Item Set |
109 |
1 |
15/60 |
27 |
5b: SOPS® Hospital Survey with Diagnostic Safety Supplemental Item Set |
110 |
1 |
15/60 |
28 |
6: Post-training Evaluation Form |
1350 |
1 |
3/60 |
68 |
7: Post-technical Assistance Evaluation Form |
1350 |
3 |
2/60 |
135 |
8: Clinical Sustainability Assessment Tool (CSAT) |
219 |
1 |
15/60 |
55 |
9: Implementation Interviews Protocol |
438 |
2 |
1 |
876 |
10: Measure Dx Organizational Self-Assessment |
73 |
1 |
30/60 |
37 |
11: Measure Dx Declaration of Measurement Strategy |
73 |
1 |
5/60 |
6 |
12: Diagnostic Safety Event Report |
73 |
3 |
1 |
219 |
13a: Omnibus Safety and Culture Survey_Medical Offices |
162 |
3 |
20/60 |
162 |
13b: Omnibus Safety and Culture Survey_Hospitals |
167 |
3 |
20/60 |
167 |
14: Calibrate Dx Survey |
329 |
4 |
30/60 |
657 |
15: Clinician Self-Efficacy Survey |
329 |
2 |
3/60 |
33 |
16: Provider Characteristics Form |
986 |
1 |
1/60 |
16 |
17: Patient Toolkit Survey-Provider |
986 |
5 |
2/60 |
164 |
18: Provider Interview Protocol |
50 |
1 |
45/60 |
38 |
19: Patient Toolkit Survey - Patient |
62500 |
1 |
5/60 |
5208 |
20: Patient Interview Protocol |
50 |
1 |
45/60 |
38 |
Total |
|
|
|
Exhibit 2 shows the estimated annualized cost burden based on the respondents’ time to complete the data collection forms. The total cost burden is estimated to be $457,431.90.
Exhibit 2. Estimated annualized cost burden
Form Name |
Number of respondents |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
1: Site Interest Form |
1060 |
106 |
$97.30a |
$10,313.80 |
2: Site Information Form |
265 |
88 |
$97.30a |
$8,562.40 |
3: Safer Dx Checklist |
219 |
55 |
$97.30a |
$5,351.50 |
4: Exit Interviews Protocol |
69 |
12 |
$97.30a |
$1,167.60 |
5a: SOPS® Medical Office Survey with Diagnostic Safety Supplemental Item Set |
109 |
27 |
$97.30a |
$2,627.10 |
5b: SOPS® Hospital Survey with Diagnostic Safety Supplemental Item Set |
110 |
28 |
$97.30a |
$2,724.40 |
6: Post-training Evaluation Form |
1350 |
68 |
$102.90b |
$6,997.20 |
7: Post-technical Assistance Evaluation Form |
1350 |
135 |
$102.90b |
$13,891.50 |
8: Clinical Sustainability Assessment Tool (CSAT) |
219 |
55 |
$97.30a |
$5,351.50 |
9: Implementation Interviews Protocol |
438 |
876 |
$102.90b |
$90,140.40 |
10: Measure Dx Organizational Self-Assessment |
73 |
37 |
$102.90b |
$3,807.30 |
11: Measure Dx Declaration of Measurement Strategy |
73 |
6 |
$102.90b |
$617.40 |
12: Diagnostic Safety Event Report |
73 |
219 |
$102.90b |
$22,535.10 |
13a: Omnibus Safety and Culture Survey_Medical Offices |
162 |
162 |
$102.90b |
$16,669.80 |
13b: Omnibus Safety and Culture Survey_Hospitals |
167 |
167 |
$102.90b |
$17,184.30 |
14: Calibrate Dx Survey |
329 |
657 |
$102.83c |
$67,559.31 |
15: Clinician Self-Efficacy Survey |
329 |
33 |
$102.83c |
$3,393.39 |
16: Provider Characteristics Form |
986 |
16 |
$102.83c |
$1,645.28 |
17: Patient Toolkit Survey-Provider |
986 |
164 |
$102.83c |
$16,864.12 |
18: Provider Interview Protocol |
50 |
38 |
$102.83c |
$3,907.54 |
19: Patient Toolkit Survey – Patient |
62500 |
5208 |
$29.76d |
$154,990.08 |
20: Patient Interview Protocol |
50 |
38 |
$29.76d |
$1,130.88 |
Total |
|
|
|
$457,431.90 |
* National Compensation Survey: Occupational wages in the United States May 2022, “U.S. Department of Labor, Bureau of Labor Statistics.”
a Based on the weighted mean hourly wage for physicians (broad) ($121.15; occupation code 29-1210; 60%) and Medical and Health Services Managers ($61.53; Code 11-9111; 40%)
b Based on the weighted mean hourly wage for physicians (broad) ($121.15; occupation code 29-1210; 70%); nurse practitioners (broad) ($59.94; occupation code 29-1170; 15%); physician assistants (broad) ($60.23; occupation code 29-1070; 10%); and medical and health services managers (broad) ($61.53; Code 11-9111; 5%)
c Based on the weighted mean hourly wage for physicians (broad) ($121.15; occupation code 29-1210; 70%); nurse practitioners (broad) ($59.94; occupation code 29-1170; 15%); and physician assistants (broad) ($60.23; occupation code 29-1070; 15%)
d Based on the mean wages for All Occupations (Code 00-0000)
There are no direct costs to respondents other than their time to participate in the study.
The total contractor cost to the Federal Government for this 4-year project is estimated to be $6,724,832 or $1,681,208 per year on average. As shown in Exhibit 3a, this amount includes costs for project development ($2,377,494); data collection activities ($2,677,845); data processing and analysis ($797,192); publication of results ($401,561); and project management ($470,740).
Exhibit 3a. Estimated Total and Annualized Cost
Cost Component |
Total Cost |
Annualized Cost |
Project Development |
$594,374 |
|
Data Collection Activities |
$2,677,845 |
$669,461 |
Data Processing and Analysis |
$199,298 |
|
Publication of Results |
$100,390 |
|
Project Management |
$117,685 |
|
Total |
$6,724,832 |
$1,681,208 |
Exhibit 3b. Federal Government Personnel Cost
Activity |
Federal Personnel |
Hourly Rate |
Estimated Hours |
Cost |
Data Collection Oversight |
COR GS 15 |
$87.93 |
30 |
$2,674 |
Review of Results |
COR GS 15 |
$87.93 |
16 |
$1,407 |
Total |
$4,081 |
|||
Annual salaries based on 2024 OPM Pay Schedule for Washington/DC area:
This is a new collection of information.
This information collection will begin upon OMB approval (estimated September 2024). Site recruitment and onboarding activities will begin immediately following OMB approval and continue for approximately 6 months, or until adequate sites have been recruited. Implementation and testing for the three resources will be staggered with Calibrate Dx beginning approximately 6 months after OMB approval (estimated March 2025), and both Measure Dx and Patient Toolkit beginning 10 months after OMB approval (estimated August 2025). Implementation for each resource will last for 12 months, after which a 14-month sustainment period will begin, during which limited data collection specific to understanding sustainability will be continued. All data collection activities are estimated to be completed by September 2027. All project findings will be synthesized into a comprehensive Project Final Report by September 2027. In addition, papers reporting the evaluation’s findings and design may be submitted to peer-reviewed and open-access publications or conferences during and after the evaluation period.
Project Time Schedule
Activity |
Time Schedule |
Recruitment and onboarding |
Months 1-6 after OMB approval |
Calibrate Dx training, implementation, testing and sustainment. |
Months 6-32 after OMB approval |
Measure Dx training, implementation, testing and sustainment. |
Months 10-36 after OMB approval |
Patient Toolkit training, implementation, testing and sustainment. |
Months 10-36 after OMB approval |
Data analysis |
Beginning 12 months after OMB approval through September 2027 |
Dissemination of findings: interim reports, briefings, and final report |
Beginning 3 months after OMB approval through September 2027 |
AHRQ does not seek this exemption.
List of Attachments:
Attachment A - Site Interest Form
Attachment B - Site Information Form
Attachment C - Safer Dx Checklist
Attachment D - Exit Interviews Protocol
Attachment E - SOPS® Medical Office Survey with Diagnostic Safety Supplemental Item Set
Attachment F - SOPS® Hospital Survey with Diagnostic Safety Supplemental Item Set
Attachment G - Post-training Evaluation Form
Attachment H - Post-technical Assistance Evaluation Form
Attachment I - Clinical Sustainability Assessment Tool (CSAT)
Attachment J - Implementation Interviews Protocol
Attachment K - Measure Dx Organizational Self-Assessment
Attachment L - Measure Dx Declaration of Measurement Strategy
Attachment M - Diagnostic Safety Event Report
Attachment N - Omnibus Safety and Culture Survey_Medical Offices
Attachment O - Omnibus Safety and Culture Survey_Hospitals
Attachment P - Calibrate Dx Survey
Attachment Q - Clinician Self-Efficacy Survey
Attachment R - Provider Characteristics Form
Attachment S - Patient Toolkit Survey-Provider
Attachment T - Provider Interview Protocol
Attachment U - Patient Toolkit Survey - Patient
Attachment V - Patient Interview Protocol
Attachment W - Federal Register Notice
References
Institute of Medicine, National Academies of Sciences Engineering and Medicine. Improving Diagnosis in Health Care. Washington, DC: The National Academies Press; 2015.
Giardina TD, Woodard LD, Singh H. Advancing Diagnostic Equity Through Clinician Engagement, Community Partnerships, and Connected Care. J Gen Intern Med. Apr 2023;38(5):1293-1295.
Henriksen K, Dymek C, Harrison MI, et al. Challenges and opportunities from the Agency for Healthcare Research and Quality research summit on improving diagnosis: a proceedings review. Diagnosis (Berl). 2017;4(2):57-66.
Kelly, B., Margolis, M., McCormack, L., LeBaron, P. A., & Chowdhury, D. (2017). What Affects People’s Willingness to Participate in Qualitative Research? An Experimental Comparison of Five Incentives. Field Methods, 29(4), 333-350.
Leape LL, Brennan TA, Laird N, et al. The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med. Feb 7 1991;324(6):377-384.
Newman-Toker DE, Schaffer AC, Yu-Moe W, et al. Serious misdiagnosis-related harms in malpractice claims: The "Big Three" - vascular events, infections, and cancers. Diagnosis (Berl). Aug 27 2019;6(3):227-240.
Newman-Toker DE, Nassery N, Schaffer AC, et al. Burden of serious harms from diagnostic error in the USA. BMJ Qual Saf. Jan 19 2024;33(2):109-120.
Office of Management and Budget. (2016). Questions and Answers When Designing Surveys for Information Collections.
Schulson LB, Novack V, Folcarelli PH, Stevens JP, Landon BE. Inpatient patient safety events in vulnerable populations: a retrospective cohort study. BMJ Qual Saf. Oct 26 2020.
Stähli E, Joye D. Incentives as a possible measure to increase response rates. The SAGE handbook of survey methodology. 2016 Mar 24:425-40.
Singh H, Meyer and Thomas EJ. The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving US adult populations. BMJ Qual Saf. Sep 2014;23(9):727-731.
VanGeest JB, Johnson TP, Welch VL. Methodologies for improving response rates in surveys of physicians: a systematic review. Eval Health Prof. 2007 Dec;30(4):303-21.
Ziegenfuss JY, Tilburt JC, Harris AM, Beebe TJ. Getting physicians to respond: the impact of incentive type and timing on physician survey response rates. Health Serv Res. 2011 Feb;46(1 Pt 1):232-42.
Zwaan L, Bruijne MdW, Cordula, Thijs A, Smits M, van der Wal GT, Daniëlle R M Patient record review of the incidence, consequences, and causes of diagnostic adverse events. Arch Intern Med. Jun 28 2010;170(12):1015-1021.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | OMB Clearance Application |
| Author | hamlin-ben |
| File Modified | 0000-00-00 |
| File Created | 2024-07-22 |