Detailed Supporting Statement

SUPPORTING STATEMENT

Part A

Building Diagnostic Safety Capacity – Patient and Family Engagement Resource Evaluation Plan

Version: April 20, 2020

Agency of Healthcare Research and Quality (AHRQ)

Table of contents

A. Justification 3

1. Circumstances that make the collection of information necessary 3

2. Purpose and use of information 5

3. Use of Improved Information Technology 7

4. Efforts to Identify Duplication 7

5. Involvement of Small Entities 7

6. Consequences if Information Collected Less Frequently 7

7. Special Circumstances 7

8. Federal Register Notice and Outside Consultations 7

9. Payments/Gifts to Respondents 8

10. Assurance of Confidentiality 8

11. Questions of a Sensitive Nature 10

12. Estimates of Annualized Burden Hours and Costs 11

13. Estimates of Annualized Respondent Capital and Maintenance Costs 12

14. Estimates of Annualized Cost to the Government 12

15. Changes in Hour Burden 13

16. Time Schedule, Publication and Analysis Plans 13

17. Exemption for Display of Expiration Date 14

List of Attachments 14

References 51

A. Justification

1. Circumstances that make the collection of information necessary

The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:

1. research that develops and presents scientific evidence regarding all aspects of health care; and

2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and

3. initiatives to advance private and public efforts to improve health care quality.

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.

Background for this information collection

The most prevalent events leading to malpractice claims in primary care include diagnostic errors, including delayed and missed diagnoses, and medication errors.^1–6 Failures in diagnosis such as incomplete evaluation of patient symptoms (39%), problems with testing (44%) and issues with patient follow-up (26%) are targets for a resource to engage patients and families in the process to mitigate diagnostic failures. ^2,7–11

The 2015 National Academies of Science, Engineering, and Medicine’s (NASEM) report, Improving Diagnosis in Health Care, identified several opportunities for patients to influence the diagnostic process.³ Clinician-patient interactions, communication, and test result follow-up are important opportunities where engagement may mitigate diagnostic error.^{2,7,8,11–19} Physicians also report seeing unprofessional behaviors that threaten patient safety, which can have a negative²⁰ impact on clinical outcomes.^21,22 Patients’, families’, and clinicians’ experiences can be a valuable source of information to improve safety and quality.¹¹

Patients and families are well positioned to communicate details about their diagnostic journey, observe clinicians’ performance, and identify factors that contribute to diagnostic errors.^11,21^,23–27 As part of this AHRQ project, an interprofessional team has been assembled to develop, obtain feedback from key stakeholders, and promote a resource to engage patients and families in the diagnostic process. The approach to resource development engages patients and families and SMEs throughout development. The goal for the engagement resource will be to provide practices and patients with simple approaches to close prevalent gaps in care leading to diagnostic failures.

This research has the following goal:

1. To pilot test a Resource to engage patients and families in the diagnostic process in order to reduce diagnostic errors (the Resource) and qualitatively examine:

   1. Feasibility of the implementing the Resource with limited to no technical assistance;

   2. Satisfaction with and receptivity to the Resource;

   3. Solicit feedback on areas to improve the Resource and delivery within ambulatory care practices.

To achieve the goals of this project the following information collection instruments will be completed using individual interviews and/or focus groups:

1. Practice-level demographic survey (Appendix A) - designed to qualitatively describe the characteristics of the practices engaged in pilot testing (e.g. patient characteristics, practice size, and staffing).

2. Pilot Test Evaluation Protocol for patients and families (Appendix B) - designed to qualitatively describe patient and family member satisfaction with the Resource and recommended changes to make the Resource more useful to them.

3. Organizational Readiness for Change Survey (Appendix C) - designed to qualitatively assess each practices’ readiness for implementing new practice processes, improvements, etc.

4. Pilot Test Interview Protocol for Ambulatory Providers (Appendix D) - designed to qualitatively assess the barriers and facilitators of implementing the Resource in practice, provider receptivity to and experience with using the Resource and recommendations for improvements to improve satisfaction of use.

5. Pilot Test Evaluation Protocol for Practice Staff (Appendix E) – designed to qualitatively assess the barriers and facilitators of implementing the Resource in practice, practice staff receptivity to and experience with using the Resource and recommendations for improvements to improve satisfaction of use.

6. Pilot Test Evaluation Protocol for Practice Administrators (Appendix F) - The information collection approach will be qualitative. Information collected will assess the barriers and facilitators of Resource implementation. It will summarize and synthesize qualitative information and recommendations for improvement based on stakeholder feedback (practice administrators), satisfaction, and receptivity to the Resource.

7. Observational Site Visit Tool (Appendix G) – Designed to collect qualitative data on the use of the Resource through direct observation during site visits.

This information collection is being conducted by AHRQ through its contractor, MedStar Health Research Institute, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).

2. Purpose and Use of Information

AHRQ will use the information collected through this Information Collection Request to assess the feasibility of adopting a resource to engage patients and families in the diagnostic process, in ambulatory care practices. A qualitative approach to data collection (e.g., individual interviews and/or focus groups) and analysis will be used to provide useful insights on stakeholders’ perceptions and opinions, identify barriers and facilitators to Resource implementation, receptivity to the Resource by stakeholders including patients, family members, clinicians, and practice staff and administrators, as well as to explore stakeholder feedback to enhance the usability of the Resource. In this context, we will examine the question “How and in what contexts do the chosen interventions work or can they be amended to work”, rather than evaluating effectiveness or “Do they work?” This approach will not consist of statistical surveys that yield quantitative results that can be generalized to the population. Pilot testing will occur at up to 20 primary care sites and feasibility of implementation will be assessed at the stakeholder and practice levels. The information collected from stakeholders will be used to revise the Resource in order to promote widespread adoption.

The specific purpose of each of the information collection instruments is described below:

1. Practice Descriptive Characteristics (Appendix A). The information collection instrument will qualitatively summarize and describe the characteristics of the practices engaged in the pilot test. The information will be used to summarize information on practice characteristics in a qualitative manner. It will also allow for the project team to identify competing priorities that may inhibit full practice participation with the proposed qualitative methods. This information collection will provide context for the pilot tests and generate information on the type of practices where the Resource interventions were pilot tested. It will thus support the ability to define the types of practices were the Resource is found to be generally acceptable.

2. Pilot Test Evaluation Protocol for Patients and Family Members (Appendix B). The information collected will provide qualitative information for analysis. Information collected will define the barriers and facilitators of Resource implementation. It will allow for a summary and synthesis of qualitative information and recommendations for improvement of the Resource based on stakeholder feedback (patients and family members), satisfaction, and receptivity to the Resource.

3. Organizational Readiness for Implementation Change (ORIC) (Appendix C). The information collection instrument will qualitatively assess individual practice’s readiness to change. One individual from the practice will complete the instrument in collaboration with a project team member/interviewer.

4. Pilot Test Interview Protocol for Ambulatory Care Providers (Appendix D). The information collected will be qualitative and will be synthesized to assess and define the barriers and facilitators of Resource implementation. The qualitative information collected will allow for the development of recommendations for improving the Resource based on stakeholder feedback (ambulatory care clinicians), satisfaction, and receptivity to the Resource.

5. Pilot Test Evaluation Protocol for Practice Staff (Appendix E). The information collected will be qualitative and will be synthesized to assess the barriers and facilitators of Resource implementation. The qualitative information will be summarized to enable the development of recommendations for improvement based on stakeholder feedback (practice staff), satisfaction, and receptivity to the Resource.

6. Pilot Test Evaluation Protocol for Practice Administrators (Appendix F). The information collected will be qualitative and will be synthesized to assess the barriers and facilitators of Resource implementation. The qualitative information will be summarized to enable the development of recommendations for improvement based on stakeholder feedback (practice administrators), satisfaction, and receptivity to the Resource.

7. Observation Tool (Appendix G). This information collection instrument is designed to collect qualitative data on the use of the Resource through direct observation of the Resource in practice during pilot test site visits.

These information collection instruments (Appendices A-G) are designed to capture qualitative data (Appendix B-G) with some quantitative data (Appendix A). No claim is made that the results from this evaluation will be generalizable in a statistical sense, nor is the intent to conduct statistical analyses. However, every attempt will be made to recruit practices that are representative of diverse geographic locations as well as diverse patient populations served including practices that serve AHRQ priority populations. The goals of the evaluation are aimed at soliciting stakeholder feedback through interviews and/or focus groups on the challenges of implementation, receptivity to the Resource, and feedback on how to improve the Resource to support adoption and implementation. The information collected will be used to revise the Resource to enhance usability and receptivity.

3. Use of Improved Information Technology

The information collection described herein will rely on paper data collection instruments in the form of interview and focus group guides to be used by the interview and/focus group facilitators and moderators for each information collection activity. Interviews and focus groups will be audio recorded for the purpose of transcription and coding. There will otherwise be no automated, electronic, or other technological collection techniques or other forms of information technology used for the information collection.

4. Efforts to Identify Duplication

The first phase of the parent study involved an extensive environmental scan to review the literature, including published, unpublished, and internet sources to identify existing interventions and resources pertinent to Resource development. The environmental scan revealed key gaps in patient and family engagement leading to missed, delayed, and wrong diagnosis which are addressed, in part, by the Resource being evaluated under this information collection request. To our knowledge, this does not involve a duplication of any existing efforts as suggested by the gap analysis resulting from the environmental scan.

5. Involvement of Small Entities

The information being collected under this request will reflect the variety of settings in which the Resource will actually be used, ambulatory care practices. This includes medium (4-9 clinicians) and large (more than 9 clinicians) ambulatory care practices. However, to our knowledge none of the practices volunteering to participate would be considered small businesses or small entities.

6. Consequences if Information Collected Less Frequently

This information collection is for a onetime data collection only. All of the information needed to solicit stakeholder feedback to inform Resource revisions under this information collection request does not need to be collected more than once.

7. Special Circumstances

This request is consistent with the generic information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.

8. Federal Register Notice and Outside Consultations

8.a. Federal Register Notice

This information collection is being submitted under AHRQ’s generic clearance. A Federal Register notice is therefore not required.

8.b. Outside Consultations

Not applicable.

9. Payments/Gifts to Respondents

Patient and family participants in the described information collection effort will be offered fair and reasonable compensation for any costs they might incur in order to participate in the proposed qualitative data collection. This includes their time and travel, parking or other transportation-related expenses. The incentive is to compensate the participant for the burden of the information collection, particularly for participants from low income populations. A $25 stipend will be offered for a 60-minute focus group for patients and family members only. Patient focus groups will be held during the day and/or evening to minimize the amount of time the respondents may miss from standard work duties.

Based on the contractor (MedStar’s) previous recruitment efforts for similar studies and the published literature, this represents the minimum amount necessary to recruit and secure participation in information collection efforts with comparable levels of effort for the participant, in the specific geographic locations where the pilot testing of the Resource will be conducted.^28,29 The proposed information collection request seeks to recruit patients and family members along the spectrum of patient and family engagement. Here, engagement in healthcare activities may range from disenfranchised or disengaged (including those with limited experience and exposure to the health system) to patients who are activated and empowered to advocate on behalf of their own healthcare and the healthcare of others.³⁰ Research demonstrates that individuals with limited experience and exposure to the healthcare system, those with lower health consumption, patients from deprived and low income neighborhoods, the elderly and youth, and those from lower socioeconomic backgrounds are less likely to participate in research.^29,31 MedStar also consulted with patient representatives from the target demographics (e.g. range of socioeconomic status, age, low and high users of healthcare) to examine the necessity of stipends to support inclusion of patients and families from the proposed ambulatory care practices representing the targeted end-users of the Resource. The $25 stipend for participation in an up to 60-minute focus group or interview is necessary to achieve an adequate response rate for our patient and family member target audiences. From prior direct experience, failure to offer any incentive or an amount lower than this will hamper the ability to successfully recruit the number of patients required for the proposed data collection.

10. Assurance of Confidentiality

Individuals and organizations will be assured of the confidentiality of their replies under Section 944(c) of the Public Health Service Act.  42 U.S.C. 299c-3(c).  That law requires that information collected for research conducted or supported by AHRQ that identifies individuals or establishments be used only for the purpose for which it was supplied.

Information that can directly identify the respondent, such as name and/or social security number will not be collected. The following participant characteristics will be collected for individuals volunteering to pilot test the Resource. These data will be presented in aggregate and used to describe the characteristics of the group providing feedback. No information will allow for individual identification of participants.

Practice Characteristics

Characteristic	Collection Method	Response Option*
Participant Category	Interview/Focus Group	Provider Staff Patient Administrator Other
Age	Interview/Focus Group	Age (years)
Sex	Interview/Focus Group	Male Female
Education Level	Interview/Focus Group	Elementary High School Diploma Some College Associate degree Bachelor’s degree Master’s degree Professional Degree Doctorate
Race	Interview/Focus Group	White Black or African American American Indian or Alaska Native Asian Two or more races
Ethnicity	Interview/Focus Group	Hispanic or Latino Not Hispanic or Latino
Location	Interview/Focus Group	City, State
Location	Interview/Focus Group	Urban Inner City Rural Suburban Other
Priority Population	Interview/Focus Group	Yes No Unknown
Self-reported Health Status	Focus Group (Patients only)	Excellent Very Good Good Fair Poor
Chronic Disease	Focus Group	Yes/No
*Each characteristic must include an option for did not respond/did not provide an answer

Participants will also receive the following confidentiality statements printed on any respondent materials:

“The confidentiality of your responses are protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)]. Information that could identify you will not be disclosed unless you have consented to that disclosure.”

Information collected will be maintained in a secure HIPAA-compliant data server. All information collection will be stored using the contractor, MedStar’s REDCap™ research data capture database. REDCap™ is a mature, secure web application for building and managing online information collection instruments and data. While REDCap™ can be used to collect virtually any type of data, it is specifically geared to support data capture for research studies. The REDCap™ Consortium is composed of 1,711 active institutional partners in 96 countries who utilize and support REDCap™ in various ways. REDCap™ can be established to support data entry forms and to conduct web-enabled surveys. The patient and family engagement Resource will also use a REDCap™ project space to securely store any documents received from the practices during the project. The MedStar Health Research Institute is a REDCap™ project collaborator site with a robust history of using this method for data collection.

The information collected will be primarily qualitative in nature (Appendices B-H). Responses to interviews and/or focus groups will be recorded and notes for each session transcribed. Transcription files will be uploaded to the REDCap™ file repository for security. Paper files will be retained in a locked file cabinet within MedStar Health. A case record form will be created within REDCap™ to record output from the thematic reviews of the transcripts to facilitate reporting and feedback to participants and to the study team. This information collection request (ICR) does not contain surveys, censuses, or employ statistical methods.

This ICR does not request any personally identifiable information.

This ICR does not include a form that requires a Privacy Act Statement.

Does this ICR contain surveys, censuses, or employ statistical methods? Yes No

Does this ICR request any personally identifiable information (see OMB Circular No. A-130 for an explanation of this term)? Please consult with your agency's privacy program when making this determination Yes No

Does this ICR include a form that requires a Privacy Act Statement (see 5 U.S.C. §552a(e)(3))? Please consult with your agency's privacy program when making this determination. Yes No

11. Questions of a Sensitive Nature

The proposed information collection does not include any questions of a sensitive nature. Each respondent will undergo an informed consent process that will describe participant rights. We anticipate that the MedStar Health Research Institute’s Intuitional Review Board will grant a waiver of documentation for this informed consent process as the consent form will be the only documentation linking the participant’s identity to the information collection. Each participant will receive a copy of the informed consent document and an information sheet on the project that outlines the participant’s rights as is standard for MedStar’s exempt studies where active recruitment is required. The consent process will highlight the participants right to answer or not answer any questions that they are asked and their right to withdraw from the interview and/or focus group at any time without penalty or repercussions.

12. Estimates of Annualized Burden Hours and Costs

Exhibit 1.  Estimated annualized burden hours

Form Name	Number of respondents	Number of responses per respondent	Hours per response	Total burden hours
Appendix A: Practice Descriptive Characteristics	20	1	1.5	30
Appendix B: Pilot Test Evaluation Protocol for Patients and Family Members	160	1	1	160
Appendix C: Organizational Readiness for Implementation Change (ORIC)	20	1	.25	5
Appendix D: Pilot Test Interview Protocol for Ambulatory Care Providers	120	1	.75	90
Appendix E: Pilot Test Evaluation Protocol for Practice Staff	120	1	1	120
Appendix F: Pilot Test Evaluation Protocol for Practice Administrators	60	1	1	60
Appendix H: Observation Tool	200	1	.5	100
Total	NA	NA	NA	565

 

Exhibit 2. Estimated annualized cost burden

Form Name	Number of respondents	Total burden hours	Average hourly wage rate*	Total cost burden
Appendix A: Practice Descriptive Characteristics	20	30	$54.68a	$1,640.40
Appendix B: Pilot Test Evaluation Protocol for Patients and Family Members	160	160	$22.11b	$3,537.60
Appendix C: Organizational Readiness for Implementation Change (ORIC)	20	5	$54.68a	$273.40
Appendix D: Pilot Test Interview Protocol for Ambulatory Care Providers	120	90	$93.61c	$8,424.90
Appendix E: Pilot Test Evaluation Protocol for Practice Staff	120	120	$20.23d	$2,427.60
Appendix F: Pilot Test Evaluation Protocol for Practice Administrators	60	60	$54.68a	$3,280.80
Appendix H: Observation Tool	200	100	$40.46e	$4,046.00
Total	NA	565	NA	$$23,630.70

* National Compensation Survey: Occupational wages in the United States May 2018 “U.S. Department of Labor, Bureau of Labor Statistics.”

^a Based on the mean wages for Medical and Health Services Managers (Code 11-9111)

^b Based on the mean wages for Misc. Healthcare Worker (Code 29-9090)

^c Based on the mean wages for Family Medicine/General Practitioners (Code 29-1062)

^d Based on the mean wages for HC Support Occupations (Code 31-000)

^e Based on the mean wages for two HC Support Occupations (Code 31-000) positions

13. Estimates of Annualized Respondent Capital and Maintenance Costs

There are no direct costs to respondents other than their time to participate in the study.

14. Estimates of Total and Annualized Cost to the Government

The total contractor cost to the government is estimated to be $202,738.56. As shown in Exhibit 3a, this amount includes costs for project development ($18,812.50); data collection activities ($93,152.31); data processing and analysis ($24,178.47), project management ($18,812.50) and overhead ($47,782.78).

Exhibit 3a.  Estimated Total and Annualized Cost

Cost Component	Total Cost	Annualized Cost
Project Development	$18,812.50	$18,812.50
Data Collection Activities	$93,152.31	$93,152.31
Data Processing and Analysis	$24,178.47	$24,178.47
Publication of Results	$0	$0
Project Management	$18,812.50	$18,812.50
Overhead	$47,782.78	$47,782.78
Total	$202,738.56	$202,738.56

Exhibit 3b. Federal Government Personnel Cost

Activity	Federal Personnel	Hourly Rate	Estimated Hours	Cost
Project oversight to include data collection oversight and review of results	Project Officer GS14	$75.57	25	$1,889.25
	Health Scientist Administrator GS 13	$63.95	25	$1,598.75
Total				$3,488.00

Annual salaries based on 2020 OPM Pay Schedule for Washington/DC area: https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2020/DCB_h.pdf

15. Changes in Hour Burden

This is a new information collection, thus no changes in hour burden is expected or reported here.

16. Time Schedule, Publication and Analysis Plans

The information collection will begin upon OMB approval (estimated May 2020) and will include recruitment of practices and completion of all data collection activities by January 2021. Qualitative analysis will be completed by February 2021 and materials will be revised based on stakeholder feedback by April 2021. We anticipate a 6-month pilot test timeline. Publication of the materials by AHRQ on their website will be completed after 508 compliance review. There are no other publication efforts associated with this information collection effort.

17. Exemption for Display of Expiration Date

AHRQ does not seek this exemption.

List of Attachments:

Appendix A – Practice Descriptive Characteristics

Appendix B – Pilot Test Evaluation Protocol for Patients and Family Members

Appendix C –Organizational Readiness for Implementation Change (ORIC)

Appendix D – Pilot Test Interview Protocol for Ambulatory Care Providers

Appendix E – Pilot Test Evaluation Protocol for Practice Staff

Appendix F – Pilot Test Evaluation Protocol for Practice Administrators

Appendix G– Observation Tool

Appendix A – Practice Descriptive Characteristics

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

Diagnostic Safety Capacity Building – Patient and Family Resource

Please complete the following information about your practice:

General Information About Your Practice

Practice Name
Location (City, State)
Select one:	Urban Inner City Rural Suburban Other (Specify)
Contact Person
Medical Director
Number of	Physicians	__________
	Nurse Practitioners	__________
	Nurses	__________
	Medical Assistants	__________
	Pharmacists	__________
	Social Workers	__________
	Case Managers Other Practice Staff	__________ __________
	Other (specify)	__________
Total Number of Patients Served by Practice
Payer Mix (Indicate % of Patients)	Self-Pay Medicare Medicaid Private Insurance Uninsured Other	_________% _________% _________% _________% _________% _________%
Race (indicate % of patients)	White Black or African American American Indian or Alaska Native Asian Native Hawaiian or Other Pacific Islander	_________% _________% _________% _________% _________%
Ethnicity (indicate % of patients)	Hispanic or Latino Not Hispanic or Latino	_________% _________%

Information about Patient Safety and Quality Improvement Activities of the Practice

	Yes	No
Does your practice routinely conduct a patient safety culture survey?	☐ Please specify which survey you use: _____________ Date of the last survey ________	☐
Is your practice part of a larger healthcare system?	☐ Please indicate which health system you are affiliated with: ___________________	☐
Is your practice currently working on any other practice improvement strategies?	☐	☐
Does your practice have or use the services of a practice facilitator?	☐	☐

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

Appendix B – Pilot Test Evaluation Protocol for Patients and Family Members

AHRQ – Building Diagnostic Safety Capacity

Patient Focus Group Script – Pilot Test Evaluation

MedStar Health Research Institute (MHRI) will conduct up to twenty (n=20) focus groups (one per practice) with patients during Pilot Testing. We will aim to recruit patients in the following manner:

• 6-8 patients, family members, and/or caregivers per primary care practice.

• Patients/family members who have been exposed to the intervention

• A diverse group of patients in terms of age, sex, race/ethnicity, income, education, and current self-reported health status.

Recruitment Criteria

Patients and family members who have been exposed to the intervention (i.e. attended the primary care practice during the period of implementation) will be eligible to participate in the focus groups. MHRI team members will work with the practice coordinators to identify and recruit patients to participate in the focus groups.

Focus Group Goals

The goals of the focus groups will be to:

• Obtain feedback on the intervention patient-focused materials

• Obtain feedback on the barriers and facilitators encountered for each intervention

• Obtain feedback on satisfaction with interventions

• Obtain feedback on receptivity and enhancements to intervention to improve adoption

Focus Group Materials

• Copies of the patient-focused materials implemented by the practice during the period of implementation

• Informed consent documents

• Paperwork for processing the Participant stipends

• Digital recorder

Focus Group Location

Focus groups will be conducted at a location within the practice’s community. Locations may include libraries and/or community centers.

Participant Stipends

Upon arriving at the focus group location and after the completion of the informed consent process, all participants will complete the required paperwork (W9) to receive the stipend for participation. The stipend for participation will be $25.

Informed Consent Procedures

Participants will complete the informed consent process at the time of arrival to the focus group.

Proposed Agenda – Patient Focus Group

Focus Groups are approximately 60 minutes each.

Agenda
Introduction	5 minutes
Background	10 minutes
Review Materials	10 minutes
General Experience with Intervention (satisfaction/barriers/enablers)	20 minutes
Enhancements	10 minutes
Closing	5 minutes
Total	60 minutes

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

AHRQ - Building Diagnostic Safety Capacity

Patient Focus Group Script – Pilot Test Evaluation

[bracketed text will depend on interviewee and topic]

WELCOME AND INTRODUCTION

• Thank you for agreeing to participate in the focus group about the strategy to improve patient and family engagement to improve diagnosis.

• My name is [INSERT NAME OF INTERVIEWER] and I am the facilitator for today’s conversation. I am here with [INSERT NAME OF PROJECT STAFF] and HE/SHE will be taking notes of our conversation.

• With your permission we will also be audio recording the session. This will help make sure that we don’t miss anything that you say and can share with other people who are working on this project. The recording will be deleted after we have the notes transcribed and are sure we have captured all your comments accurately.

• TODAY/TONIGHT we will be asking you questions about your experiences with engaging with your care team on diagnosis.

• Nothing that you say or share today will impact your treatment or care from your doctor’s/Nurse Practitioner’s office. We will not share your name or anything that you say with them.

• Do you have any questions before we begin?

GROUND RULES

• We want to hear from everyone and want to hear your honest opinions. There are no wrong answers.

• If you have something to add to the conversation, please feel free to jump in. We do have a lot to cover so we will try not to spend too much time on any one topic.

• It is important that when you do jump in that we try to make sure that we only have one person talking at any time. This will help us hear everyone’s thoughts and opinions.

• Any questions?

So, let’s get started.

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

BRIEF INTRODUCTIONS

I would like to start with some brief introductions. We will go around the table. As we go around, please tell us all your first name and something about yourself like your favorite hobby or television show.

BACKGROUND

That is great! I am really pleased to meet you all. I would like to jump right in and start asking for your impressions about improving the process of diagnosis.

DIAGNOSIS

1. How would you describe the process of discussing your healthcare symptoms with your doctor/nurse practitioner?

   1. What goes well?

   2. What could go better?

   3. How would you change the encounter if you could to improve the process of getting a diagnosis?

   4. Can you describe what might help you to better engage with your clinicians to help share your symptoms?

2. How do you think that patients or their family members can engage with their healthcare team to improve the process of diagnosis?

   1. What about asking questions? Is there something that might help you ask questions easier during your encounter?

   2. What do you do to prepare for your visit?

   3. Do you ever forget to ask the doctor or nurse something during your visit?

Your doctor’s office has been implementing a new strategy to improve diagnosis. We would like to ask you about your experiences with some of the materials you may have seen in your doctor’s office.

3. What is the first thing that comes to mind when you see this [Agenda Setting Tool/Poster]?

   1. Have you seen these materials before?

      1. Where exactly did you see them?

      2. Did you pick them up or ask anyone about them?

         1. Why? Why not?

   2. Did you think they were helpful? Why/Why not?

   3. What would make them more useful to you?

4. Can you describe for me how you used/would use the [Agenda Setting Tool/Poster]?

   1. Did you find it helpful?

   2. What would have made it better?

   3. Did you feel like you needed more information about why you should use it?

   4. Were you able to get help if you needed it?

5. If you could change one thing about the materials, what would that be?

   1. If nothing to change:

      1. Did you personally get a chance to use these tools?

      2. Did your family member?

      3. Can you describe their experience?

6. What about the format of the [Agenda Setting Tool/Poster]? Is there a better way for us to think about presenting the materials?

Is there anything else you would like to share about your experiences with the materials? If not, let’s move on to learning more about your experiences with the Agenda Setting Tool/Poster.

7. We want to get some more information about your thoughts on [Agenda Setting Tool/Poster]. Where did you first hear about the [Agenda Setting Tool/Poster]?

   1. Who first brought it to your attention, your doctor, another organization?

   2. How long ago, or when did you first get introduced to the [Agenda Setting Tool/Poster]?

8. When you were first given the [Agenda Setting Tool/Poster] what did you think?

   1. How did you feel about using it?

   2. How did you end up using the [Agenda Setting Tool/Poster], or did you end up not using it after all?

   3. If no, what prevented you from using it? Time? Challenges with the materials?

   4. Do you plan on using the [Agenda Setting Tool/Poster] in the future? If yes, how. If no, why not? Is there something we could do to help you with making it more usable?

9. How easy or challenging did you find the [Agenda Setting Tool/Poster] to be?

10. Specifically thinking about the clinical encounter and getting a diagnosis, how did you think this the [Agenda Setting Tool] addressed safety for you?

11. And what about patient engagement, how did the [Agenda Setting Tool/Poster] help you to engage or increase your engagement in your care?

CLOSING

12. Those were all the questions I had today. Are there any questions that I should have asked that I did not?

Thank you for your time and participation in this interview. Your comments will be very helpful to this project!

Appendix C – Organizational Readiness for Implementation Change (ORIC

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

Organizational Readiness for Implementation Change (ORIC) – Practice Leader/Administrator/Practice Champion

MedStar Health Research Institute (MHRI) will conduct an interview with up to twenty (n=20) practice champion/administrator/provider to assess practice readiness to change. The information is collected at the practice level, not the individual level.

Recruitment Criteria

Any practice champion/administrator/provider that has practice level line of sight on organizational and operational priorities may complete the practice-level survey.

Location and Schedule

The change readiness survey will be completed online. The survey will take approximately 12 minutes to complete.

Informed Consent Procedures

Informed consent will be completed online with a survey cover page.

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 15 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

Organizational Readiness for Implementing Change (ORIC)

	1	2	3	4		5
	Disagree	Somewhat Disagree	Neither Agree nor Disagree	Somewhat Agree		Agree
People who work here feel confident that the organization can get people invested in implementing this change.					1 2 3 4 5
People who work here are committed to implementing this change.					1 2 3 4 5
People who work here feel confident that they can keep track of progress in implementing this change.					1 2 3 4 5
People who work here will do whatever it takes to implement this change.					1 2 3 4 5
People who work here feel confident that the organization can support people as they adjust to this change.					1 2 3 4 5
People who work here want to implement this change.					1 2 3 4 5
People who work here feel confident that they can keep the momentum going in implementing this change.					1 2 3 4 5
People who work here feel confident that they can handle the challenges that might arise in implementing this change.					1 2 3 4 5
People who work here are determined to implement this change.					1 2 3 4 5
People who work here feel confident that they can coordinate tasks so that implementation goes smoothly.					1 2 3 4 5
People who work here are motivated to implement this change.					1 2 3 4 5
People who work here feel confident that they can manage the politics of implementing this change.					1 2 3 4 5

Appendix D – Pilot Test Interview Protocol for Ambulatory Care Providers

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

AHRQ – Building Diagnostic Safety Capacity

Provider Interviews – Patient and Family Engagement Resource Pilot Test Evaluation

MedStar Health Research Institute (MHRI) will conduct interviews and/or focus groups with providers from up to twenty (n=20) practices. Evaluation will be completed within 3-6 months after implementation of the Patient and Family Engagement Resource.

• 120 cognitive interviews with ambulatory care providers (up to 6 providers per practice x 20 practices); each interview will last approximately 45 minutes

Recruitment Criteria

We will aim to recruit providers and practice staff in the following manner:

• Providers and practice staff who have been exposed to the PFE Resource

• Diversity among practice staff and providers

MHRI staff will work with the practice coordinators to identify individuals to participate in the interviews.

Interview Goals

The goals of the focus groups/interviews will be to:

• Obtain feedback on the intervention Guide materials

• Obtain feedback on the barriers and facilitators encountered for the Guide

• Obtain feedback on satisfaction with the Guide

• Obtain feedback on receptivity and enhancements to the Guide to improve adoption

Materials

• Copies of the PFE Resource materials

• Informed consent documents

• Documentation for participant stipends

• Digital recorder

Location

Interviews will take place at the primary care practice at a time convenient to the provider and/or practice staff members. Interviews may also take place over the phone to enhance ability to recruit and retain clinicians.

Informed Consent Procedures

Participants will complete the informed consent process prior to starting the interview.

Participant Stipends

None.

Each interview will take no more than 45 minutes.

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 45 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

AHRQ – Building Diagnostic Safety Capacity

Provider Interviews – Patient and Family Engagement Resource Pilot Test Evaluation

WELCOME AND INTRODUCTION

• Thank you for agreeing to speak with me!

• My name is [INSERT NAME OF INTERVIEWER] and I am here to ask you a few questions about the AHRQ’s Patient and Family Engagement (PFE) resource to improve diagnosis.

• With your permission we will also be audio recording the session. This will help make sure that we don’t miss anything that you say and can share with other people who are working on this project. The recording will be deleted after we have the notes transcribed and we are sure that all of your comments are accurately.

• TODAY/TONIGHT I will be asking you questions about your experiences with the AHRQ’s Patient and Family Engagement (PFE) resource to improve diagnosis.

• Everything you say here will be kept confidential and included as part of our assessment of the feasibility of implementing the AHRQ’s PFE resource to improve diagnosis into practice. We will not share your name or attribute any of your words directly to you.

• Do you have any questions before we begin? Ok, great. Let’s get started.

DIAGNOSIS

1. How would you describe the impact of the AHRQ’s PFE resource on the diagnostic process in your personal practice?

   1. Why do you think that it was improved/not changed?

   2. Can you share with me any thoughts on what could be improved as part of the PFE resource that could make it more effective?

      1. What about changes to the patient poster? Agenda setting tool?

      2. How about the implementation planning toolkit?

      3. Training toolkit? What worked well? What would you do to improve it?

   3. How difficult was it to implement the strategy of allowing the patient to have the first minute of the clinical visit to tell their story?

      1. What impact do you feel this strategy had?

      2. Did you learn anything more from it compared to your usual practice?

      3. What was the patient’s response to this approach? How did you handle that?

2. In your opinion as a healthcare provider (doctor, nurse practitioner), what would you say are the most important things for patients to know or think about when it comes to improving diagnosis?

   1. Can you describe how this PFE Resource helped you to achieve this?

   2. What could or should we do differently to make this more effective?

Thank you for sharing. I now want to speak with you a little more about patient engagement.

3. From your experience, how did the PFE Resource materials support engagement from the patient and or their family in the diagnostic process?

   1. Can you describe any barriers to engagement that you observed?

   2. How would you describe the level of engagement you had with patients and families after you implemented the PFE Resource?

   3. From your perspective, what part of the intervention made the greatest impact?

Now I would like you to consider the materials that you have in front of you for this next series of questions.

4. Can you describe for me how you used the (Planning toolkit, patient resources, training toolkit)?

   1. Did you find it helpful?

   2. Was the process difficult to follow?

   3. Was the training and education about how to use the PFE Resource appropriate? What changes would you make to improve it?

   4. Did you feel like you needed more information about how and/or why you should use the PFE resource?

5. What about the format of the materials? Is there a better way for us to think about presenting the materials?

   1. To patients? To clinicians? To Administrators?

   2. What about an electronic version? If you had this on your phone or another electronic mobile device would that help?

Is there anything else you would like to share about your experiences with the materials? If not, let’s move on to learning more about your experiences with the PFE Resource.

6. When you were first given the PFE Resource what did you think?

   1. How did you feel about using it?

   2. Can you describe your practice’s implementation process?

   3. How did you make decisions on which elements of the PFE Resource to use?

7. How easy or challenging did you find the PFE Resource implementation to be?

   1. Were any elements of the PFE Resource that were easier or harder to implement? If yes, can you describe which ones and what made them more challenging?

   2. What can we do to make them more effective? Easier to implement? More relevant to your practice’s workflow or patient population?

8. What was the best thing about the Resource from your perspective?

9. What was your least favorite thing about the Resource?

10. What would you have changed about the Resource to make it more user friendly?

11. What do you think your patients felt about having these tools available to them?

    1. Can you give me an example of a patient experience that was positive?

    2. How about a negative one? Is there anything that could have been done to make it a more positive experience for that patient?

12. What about your practice staff? Where they on board with the change?

    1. Did they like the new approach?

    2. Was it difficult to get buy-in from them?

13. Those were all the questions I had today. Are there any questions that I should have asked that I did not?

Thank you for your time and participation in this interview. Your comments will be very helpful to this project!

Appendix E – Pilot Test Evaluation Protocol for Practice Staff

AHRQ – Building Diagnostic Safety Capacity

Staff Interviews – Patient and Family Engagement Resource Pilot Test Evaluation

MedStar Health Research Institute (MHRI) will conduct interviews and/or focus groups with staff from up to twenty (n=20) practices. Evaluation will be completed within 3-6 months after implementation of the Patient and Family Engagement Resource.

• 120-160 cognitive interviews with ambulatory care staff (6-8 staff members per practice x 20 practices); each interview will last approximately 60 minutes

Recruitment Criteria

Ambulatory care practice staff who have been exposed to the PFE Resource within their practice will be eligible to participate in the focus group. MHRI team members will work with the practice champions to identify practice staff to participate in the focus groups. We will aim to recruit practice staff in the following manner:

• Staff members who were involved in the implementation of the PFE Resource or how have had experience using the PFE Resource

Focus Group Goals

The goals of the focus groups will be to:

• Obtain feedback on the intervention PFE Resource materials

• Obtain feedback on the barriers and facilitators encountered during implementation

• Obtain feedback on satisfaction with instructions and materials

• Obtain feedback on receptivity and enhancements to the PFE Resource materials to improve adoption and implementation

Focus Group Materials

• Copies of the PFE Resource

• Informed consent documents

• Documentation for Processing of Participant stipends

• Digital recorder

Focus Group Location

Focus groups will be conducted at a location within the practice or within the practice’s community.

Participant Stipends

None.

Informed Consent Procedures

Participants will complete the informed consent process at the time of arrival to the focus group.

Proposed Agenda – Practice Staff Focus Group

Focus Groups will be planned for approximately 60 minutes each.

Agenda
Introduction	5 minutes
Background	10 minutes
Review Materials	10 minutes
General Experience with Intervention (satisfaction/barriers/enablers)	15 minutes
Enhancements	15 minutes
Closing	5 minutes
Total	60 minutes

AHRQ – Building Diagnostic Safety Capacity

Staff Interviews – Patient and Family Engagement Resource Pilot Test Evaluation

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

WELCOME AND INTRODUCTION

• Thank you for agreeing to participate in the focus group about your experiences with the Agency for Healthcare Research and Quality’s Resource for improving patient and family engagement in the diagnostic process!

• My name is [ INSERT NAME OF INTERVIEWER] and I am the facilitator for today’s conversation. I am here with [INSERT NAME OF PROJECT STAFF and HE/SHE will be taking notes of our conversation.

• With your permission we will also be audio recording the session. This will help make sure that we don’t miss anything that you say and can share with other people who are working on this project. The recording will be deleted after we have the notes transcribed and are sure we have captured all your comments accurately.

• TODAY/TONIGHT we will be asking you questions about your experiences with using the PFE Resource and your practice’s experience implementing it.

• Nothing that you say or share today will be shared in an identifiable way with your practice but represented together along with nine other practices in a report to the Agency for Healthcare Research and Quality. We will not share your name or anything that you say with them in a personally identifiable way.

• Do you have any questions before we begin?

GROUND RULES

• We want to hear from everyone and want to hear your honest opinions. There are no wrong answers.

• If you have something to add to the conversation, please feel free to jump in. We do have a lot to cover so we will try not to spend too much time on any one topic.

• It is important that when you do jump in that we try to make sure that we only have one person talking at any time. This will help us hear everyone’s thoughts and opinions.

• Any questions?

So let’s get started.

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

I would like to begin by asking you all some questions about diagnostic error and the diagnostic process in your practice.

DIAGNOSIS

1. How would you describe the impact of the PFE Resource on the safety of the diagnostic process in your practice?

1. Why do you think that it was improved?

2. What strategies within the PFE Resource (agenda setting tool, poster, one-minute of patient talking uninterrupted) do you feel had the greatest impact on the process? Can you elaborate on why you think it had that impact?

3. Given your experience with the PFE Resource, what was most effective in your practice?

   1. Did you do anything to make changes to the materials or approaches to fit your practice? Can you describe those changes? Did that work? Why/Why not?

   2. Now that you have had some experience with the PFE Resource in your practice, what would you change about it? Why? What do you think that change would achieve?

Thank you for sharing. I now want to speak with you a little more about patient engagement.

2. From your experience, how did the PFE Resource support engagement from the patient and or their family in the process of getting an accurate and timely diagnosis?

• Can you describe what you observed when patients/family members used the agenda setting tool? The poster?

• Were you able to observe the patient and/or family owning that first minute of the visit to tell their diagnosis story?

• How would you describe the level of engagement you had with patients and families after you implemented the PFE Resources?

Now I would like you to consider the materials that you have in front of you for this next series of questions.

3. I would like to focus on the practice’s implementation of the PFE Resource. Were you part of the implementation team? Can you describe for me how your practice advised the team about implementation?

• Did you use the quick start implementation guide? Do you have any feedback about what was missing in the implementation guidance? Anything that should have been there that would have been helpful? What about things in it that weren’t helpful?

• What could have made the implementation guide more useful?

4. Can you describe for me how your practice implemented the patient-facing resources? Specifically, the agenda setting tool and the poster. Did you use both?

• How did the patient’s respond to this new role?

• Can you describe the intended workflow for the patient materials? Who was responsible for orienting the patient? How was this accomplished?

  1. Was the process disruptive to the practice’s workflow?

  2. Did you feel like you had enough education and training on how to orient patients to the tools? What would have helped that we didn’t think about?

     1. How did you overcome this?

     2. What strategies did you use to help integrate the agenda setting tool into your practice?

     3. How did you track its effectiveness?

• Did you feel like you needed more information about how to or why you should use it?

5. What about the format of the materials? Is there a better way for us to think about presenting the materials?

6. When you were first given the PFE Resource what did you think?

• How did you feel about using it?

• Are there plans for your practice to continue using the PFE Resource in the future?

  1. What changes is your practice making to accommodate this sustained implementation?

  2. Why do you think that the practice is not continuing the implementation?

  3. Is there anything that would make it more sustainable/feasible to maintain the implementation?

7. What was the best thing about the PFE Resource?

8. What was your least favorite thing about the PFE Resource?

9. What would you have changed about the PFE Resource?

10. What about costs of implementing the PFE Resource? Was that a challenge at all for you?

1. Can you describe how much the time and/or effort it took from yourself or your staff to implement?

2. Was this a barrier?

3. Do you believe that this is sustainable for your practice?

   1. If yes. Can you describe what makes this approach important enough for you to continue using it?

   2. If no. Can you describe why you wouldn’t consider continuing to use it?

11. Those were all the questions I had today. Are there any questions that I should have asked that I did not?

Thank you for your time and participation in this interview. Your comments will be very helpful to this project and will help us to make important improvements to the PFE Resource!

Appendix F – Pilot Test Evaluation Protocol for Practice Administrators

AHRQ – Building Diagnostic Safety Capacity

Practice Administrators – Patient and Family Engagement Resource Pilot Test Evaluation

MedStar Health Research Institute (MHRI) will conduct interviews with administrators from up to twenty (n=20) practices. Evaluation will be completed within 3-6 months after implementation of the Patient and Family Engagement Resource.

• 20-60 cognitive interviews with practice administrators (1-3 practice administrators per practice x 20 practices); each interview will last approximately 60 minutes

Recruitment Criteria

Ambulatory care practice administrators who have been exposed to the PFE Resource within their practice will be eligible to participate in the interview. MHRI team members will work with the practice champions to identify practice administrators to participate in the interviews. We will aim to recruit practice administrators in the following manner:

• Practice administrators involved in the decision making around implementation of the PFE Resource or how have had experience using the PFE Resource

Interview Goals

The goals of the interviews will be to:

• Obtain feedback on the intervention PFE Resource materials

• Obtain feedback on the barriers and facilitators encountered during implementation

• Obtain feedback on satisfaction with instructions and materials

• Obtain feedback on receptivity and enhancements to the PFE Resource materials to improve adoption and implementation

Interview Materials

• Copies of the PFE Resource

• Informed consent documents

• Documentation for Processing of Participant stipends

• Digital recorder

Location

Interviews will be conducted at a location within the practice or within the practice’s community. Interviews may also be conducted virtually via the telephone.

Participant Stipends

None.

Informed Consent Procedures

Participants will complete the informed consent process at the time of arrival to the interview.

Proposed Agenda – Practice Administrators Interview

Interviews will be planned for approximately 60 minutes each.

Agenda
Introduction	5 minutes
Background	10 minutes
Review Materials	10 minutes
General Experience with Intervention (satisfaction/barriers/enablers)	15 minutes
Enhancements	15 minutes
Closing	5 minutes
Total	60 minutes

AHRQ – Building Diagnostic Safety Capacity

Practice Administrators – Patient and Family Engagement Resource Pilot Test Evaluation

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

WELCOME AND INTRODUCTION

• Thank you for agreeing to participate in the interview about your experiences with the Agency for Healthcare Research and Quality’s Resource for improving patient and family engagement in the diagnostic process!

• My name is [ INSERT NAME OF INTERVIEWER] and I am the facilitator for today’s conversation. I am here with [INSERT NAME OF PROJECT TEAM] and HE/SHE will be taking notes of our conversation.

• With your permission we will also be audio recording the session. This will help make sure that we don’t miss anything that you say and can share with other people who are working on this project. The recording will be deleted after we have the notes transcribed and are sure we have captured all your comments accurately.

• TODAY/TONIGHT we will be asking you questions about your experiences with using the PFE Resource and your practice’s experience implementing it.

• Nothing that you say or share today will be shared in an identifiable way with your practice but represented together along with nine other practices in a report to the Agency for Healthcare Research and Quality. We will not share your name or anything that you say with them in a personally identifiable way.

• Do you have any questions before we begin?

So let’s get started.

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

I would like to begin by asking you all some questions about diagnostic error and the diagnostic process in your practice.

DIAGNOSIS

12. Prior to the signing up for the pilot test, what was your practice doing in terms of improving diagnosis and diagnostic error mitigation?

    1. Was this a strategic priority for your organization?

    2. What were the key factors that underpinned your decision to join the pilot test?

    3. Were any of the materials provided at the time of recruitment helpful to this process?

    4. What information could we have provided to make this intervention more attractive to you as a practice administrator and leader?

13. How would you describe the impact of the PFE Resource on the safety of the diagnostic process in your practice?

4. Why do you think that it was improved?

   1. Are you actively measuring and tracking these improvements?

   2. Are there other improvements that are harder to track? If yes, what would those be?

5. What strategies within the PFE Resource (agenda setting tool, poster, one-minute of patient talking uninterrupted) do you feel had the greatest impact on the diagnostic process within your practice?

   1. Can you elaborate on why you think it had that impact?

6. What was the most effective part of the PFE Resource within your practice? Can you describe why that is?

Thank you for sharing. I now want to speak with you a little more about patient engagement.

14. As an administrator, what was the impact of the PFE Resource on your clinical team?

    1. What about your practice staff? Did the implementation result in changes in productivity or engagement from your staff? Clinicians? Others?

    2. What about burden? Was this unduly burdensome for your practice to implement?

       1. Compared to other quality improvement activities, how would you rate this activity in terms of burden for implementation? What about sustainability?

    3. Does your practice intend to continue using the PFE Resource?

       1. If yes, what parts of the resource did you find most impactful? How do you know this?

       2. What parts of the process/resource do you intend to keep?

       3. For those parts of the resource that you will continue, are you planning on making any changes to them?

       4. What would make the PFE Resource more useful? Easier to implement? Easier to sustain?

15. Do you believe that the PFE Resource impacted your patients in a positive way?

    1. Were they more or less engaged with the clinicians?

    2. Were the patient resources helpful or did they hinder patient-clinician interactions?

    3. Did patients give you any feedback about the resources or the approach?

       1. How did they describe their experiences?

       2. Where did they provide this feedback to you?

       3. What about measures of patient engagement? Did you see any changes in CAHPS scores or other satisfaction or experience surveys?

16. I would like to focus on the practice’s implementation of the PFE Resource. Can you describe for me how your practice advised the team about implementation?

• Did you use the quick start implementation guide? Do you have any feedback about what was missing in the implementation guidance? Anything that should have been there that would have been helpful? What about things in it that weren’t helpful?

• What could have made the implementation guide more useful?

17. As a practice administrator, was the format of the materials appropriate for how your practice conducts process and quality improvement implementations?

    1. Did it fit within your patient and family engagement strategies? Complement them? Were they at odds with them in any way?

    2. What changes to the materials might support greater adoption and more sustained implementation? Would you be willing to try that?

18. What about costs of implementing the PFE Resource? Was that a challenge at all for you?

    1. How will cost of the implementation influence the decision to sustain the program after the pilot test?

       1. What are the greatest influencing factors with respect to sustainability?

          1. Payment? Cost? Change fatigue? Interest?

       2. How could the PFE Resource better address these issues to make it more attractive to your clinic stakeholders?

19. How much staff time and resources did the implementation of the PFE Resource take? Is this sustainable? What would you say to other practice administrators who are contemplating implementing this resource?

20. Those were all the questions I had today. Are there any questions that I should have asked that I did not?

Thank you for your time and participation in this interview. Your comments will be very helpful to this project and will help us to make important improvements to the PFE Resource!

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

Appendix G – Observation Tool

AHRQ – Building Diagnostic Safety Capacity

Patient and Family Engagement Resource Pilot Test Evaluation

Practice Observation Tool

The practice observation tool will be used during the practice site visit to document evidence of the PFE Resource implementation within the practice. This will be conducted by a project team member and have minimal burden on the practice and respondents. The practice observation tool is a brief checklist to audit materials and evaluate fidelity to the intended implementation of each of the resources into everyday practice. Patient-clinician interactions will be observed at the discretion of the patient, clinician, and practice. We aim to observe up to 10 patient-clinician interactions per practice for evidence of within encounter adoption of the PFE Resources.

MedStar Health Research Institute (MHRI) will conduct direct observations of patient-clinician encounters for up to 10 encounters per practice (n=200 encounters). Each encounter will engage one patient, one clinician, and one or more staff members (burden estimate based on 2 staff). Total time for the encounter burden was estimated at 20 minutes per encounter.

Recruitment Criteria

Ambulatory care practice administrators who have been exposed to the PFE Resource within their practice will be eligible to participate in the interview. MHRI team members will work with the practice champions to identify practice administrators to participate in the interviews. We will aim to recruit practice administrators in the following manner:

• Practice administrators involved in the decision making around implementation of the PFE Resource or how have had experience using the PFE Resource

Interview Goals

The goals of the interviews will be to:

• Obtain feedback on the intervention PFE Resource materials

• Obtain feedback on the barriers and facilitators encountered during implementation

• Obtain feedback on satisfaction with instructions and materials

• Obtain feedback on receptivity and enhancements to the PFE Resource materials to improve adoption and implementation

Interview Materials

• Copies of the PFE Resource

• Informed consent documents

• Documentation for Processing of Participant stipends

• Digital recorder

Location

Practice observations will be conducted within the practice that is implementing the resource.

Participant Stipends

None.

Burden Hours: 100 patients x 30 minutes per patient

Informed Consent Procedures

Participants will complete the informed consent process at the time of arrival to the interview.

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

AHRQ – Building Diagnostic Safety Capacity

Patient and Family Engagement Resource Pilot Test Evaluation

Practice Observation Tool

PFE Resource Component	Observed	Description of Fidelity/Implementation in Practice
Patient facing poster	☐	Encounter Room Waiting Room Hallway Other: [Define]
Patient Agenda Tool – Preparation for Encounter	☐	Given at check in Given at exam room Given in waiting room Other
Patient One Minute Story	☐	Patient given one minute to speak at start of encounter Patient interrupted after [xxxx] seconds Patient not given opportunity Patient refused opportunity Family member spoke on behalf of patient Family member engaged in story telling with patient Other? [describe]
Patient Agenda Tool - Encounter	☐	MA acknowledged the Agenda Tool Clinician acknowledged the Agenda Tool Clinician used Agenda Tool in visit Other? [describe]

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 20 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

References

1. Singh H, Meyer AND, Thomas EJ. The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving US adult populations. BMJ Qual Saf. 2014;23(9):727-731. doi:10.1136/bmjqs-2013-002627

2. Singh H, Giardina TD, Meyer AND, Forjuoh SN, Reis MD, Thomas EJ. Types and Origins of Diagnostic Errors in Primary Care Settings. JAMA Intern Med. 2013;173(6):418. doi:10.1001/jamainternmed.2013.2777

3. National Academy of Medicine. Improving Diagnosis in Health Care. (Balogh EP, Miller BT, Ball JR, eds.). Washington, D.C.: National Academies Press; 2015. doi:10.17226/21794

4. Wallace E, Lowry J, Smith SM, Fahey T. The epidemiology of malpractice claims in primary care: a systematic review. BMJ Open. 2013;3(7):e002929. doi:10.1136/bmjopen-2013-002929

5. Singh H, Schiff GD, Graber ML, Onakpoya I, Thompson MJ. The global burden of diagnostic errors in primary care. BMJ Qual Saf. 2017;26(6):484-494. doi:10.1136/bmjqs-2016-005401

6. Singh Panesar S, DeSilva D, Carson-Stevens A, et al. How safe is primary care? A systematic review. BMJ Qual Saf. 2016;25:544-553.

7. Smith KM, Baker K, Wesley D, et al. Guide to Improving Patient Safety in Primary Care Settings by Engaging Patients and Families.; 2017. doi:10.13140/RG.2.2.35660.26242

8. M. N, O. K, B.C. D. Reducing diagnostic errors in primary care. A systematic meta-review of computerized diagnostic decision support systems by the LINNEAUS collaboration on patient safety in primary care. Eur J Gen Pract. 2015;21(Supplement 1):8-13. doi:http://dx.doi.org/10.3109/13814788.2015.1043123

9. CRICO. Risks of Ambulatory Care.; 2014.

10. Quinn GR, Ranum D, Song E, et al. Missed Diagnosis of Cardiovascular Disease in Outpatient General Medicine: Insights from Malpractice Claims Data. Jt Comm J Qual Patient Saf. 2017;43:508-516. doi:10.1016/j.jcjq.2017.05.001

11. Smith KM, Baker KM, Hemmelgarn C, Goeschel CA. Patient perceived care breakdowns in diagnosis and treatment in urgent care. In: Society for Improving Diagnosis in Medicine. ; 2017.

12. Davis Giardina T, King BJ, Ignaczak AP, et al. Root Cause Analysis Reports Help Identify Common Factors In Delayed Diagnosis And Treatment Of Outpatients. Health Aff. 2013;32(8):1368-1375. doi:10.1377/hlthaff.2013.0130

13. Giardina T, Modi V, Parrish D, et al. The Patient Portal and Abnormal Test Results: An Exploratory Study of Patient Experiences. Vol 2.; 2015. http://pxjournal.org/journalhttp://pxjournal.org/journal/vol2/iss1/20http://pxjournal.org/journal/vol2/iss1/20. Accessed June 12, 2019.

14. Zwaan L, Singh H. The challenges in defining and measuring diagnostic error. Diagnosis. 2015;2(2):97-103. doi:10.1515/dx-2014-0069

15. Giardina TD, Haskell H, Menon S, et al. Learning From Patients’ Experiences Related To Diagnostic Errors Is Essential For Progress In Patient Safety. Health Aff. 2018;37(11):1821-1827. doi:10.1377/hlthaff.2018.0698

16. Giardina TD, Baldwin J, Nystrom DT, Sittig DF, Singh H. Patient perceptions of receiving test results via online portals: a mixed-methods study. J Am Med Informatics Assoc. 2018;25(4):440-446. doi:10.1093/jamia/ocx140

17. Smith KM, Baker KM, Shofer M, Goeschel CA. Engaging patients – promising strategies to bridge the patient safety chasm in primary care. Implement Sci. 2019;14(Supplement 1):878.

18. Bell SK, Mejilla R, Anselmo M, et al. When doctors share visit notes with patients: a study of patient and doctor perceptions of documentation errors, safety opportunities and the patient-doctor relationship. BMJ Qual Saf. 2017;26:262-270. doi:10.1136/bmjqs-2015-004697

19. Nystrom DT, Singh H, Baldwin J, Sittig DF, Giardina TD. Methods for Patient-Centered Interface Design of Test Result Display in Online Portals. eGEMs (Generating Evid Methods to Improv patient outcomes). 2018;6(1):15. doi:10.5334/egems.255

20. Arora S. Project ECHO: democratising knowledge for the elimination of viral hepatitis. Lancet Gastroenterol Hepatol. 2019;4(2):91-93. doi:10.1016/S2468-1253(18)30390-X

21. Martinez W, Lehmann LS, Thomas EJ, et al. Speaking up about traditional and professionalism-related patient safety threats: a national survey of interns and residents. BMJ Qual Saf. 2017;26(11):869-880. doi:10.1136/bmjqs-2016-006284

22. Martinez W, Pichert JW, Hickson GB, et al. Qualitative Content Analysis of Coworkersʼ Safety Reports of Unprofessional Behavior by Physicians and Advanced Practice Professionals. J Patient Saf. March 2018:1. doi:10.1097/PTS.0000000000000481

23. Novak A. Improving safety through speaking up: An ethical and financial imperative. J Healthc Risk Manag. doi:10.1002/JHRM.21360

24. Fisher KA, Smith KM, Gallagher TH, Burns L, Morales CM, Mazor KM. We Want to Know: Eliciting Hospitalized Patients’ Perspectives on Breakdowns in Care. J Hosp Med. 2017;12(8). doi:10.12788/jhm.2783

25. Fisher K, Smith K, Gallagher T, Mazor K. Implementing a Program to Encourage Patients to Report Breakdowns in Care: Do We Really Want to Know? J Patient-Centered Res Rev. 2017;4(3):185. doi:10.17294/2330-0698.1543

26. Fisher KA, Smith KM, Gallagher TH, Huang JC, Borton JC, Mazor KM. We want to know: patient comfort speaking up about breakdowns in care and patient experience. BMJ Qual Saf. 2019;28(3):190-197. doi:10.1136/bmjqs-2018-008159

27. Smith KM, Mazor KM, Gallagher TH. We Want to Know. MedStar Health. http://www.wewant2know.org/. Published 2014. Accessed January 1, 2016.

28. Arthur HM, Wright DM, Smith KM. Women and Heart Disease: The Treatment May End but the Suffering Continues. Can J Nurs Res. 2001;33(3):17-29.

29. Koopmans B, Nielen MM, Schellevis FG, Korevaar JC. Non-participation in population-based disease prevention programs in general practice. BMC Public Health. 2012;12:1. doi:10.1186/1471-2458-12-856

30. Carman KL, Dardess P, Maurer M, et al. Patient and family engagement: a framework for understanding the elements and developing interventions and policies. Heal Aff. 2013;32(2):223-231. doi:10.1377/hlthaff.2012.1133

31. Bonevski B, Randell M, Paul C, et al. Reaching the hard-to-reach: a systematic review of strategies for improving health and medical research with socially disadvantaged groups. 2014. doi:10.1186/1471-2288-14-42