5 Appendix E: Post-test Evaluation Interview Protocol

Appendix E – Post-test Interview Protocol

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20xx

AHRQ –Measure Dx: A Resource To Identify, Analyze, and Learn From Diagnostic Safety Events

Post-test Interview Protocol for Quality and Safety Personnel

During the evaluation of Measure Dx, the project team will conduct post-test interviews at the end of implementation (approximately 6 months after delivery of the resource to the sites). Interviews will be conducted virtually, with participants from up to 10 implementation sites, and 1-2 representatives per site, for approximately 60 minutes each. Participants will include clinicians and/or quality and safety personnel from each site.

Interview Goals

The goals of the focus groups/interviews will be to:

• Understand the organizations’ experience with implementing the resource

  • Determine the extent to which users adhered to the resource content (FIDELITY)

  • Identify barriers and facilitators to improving diagnostic safety with the Resource (PROCESS)

  • Assess user experience and satisfaction with the resource (OUTCOMES)

• Assess the impact of the resource on diagnostic safety policies or activities (OUTCOMES)

  • Determine the yield of diagnostic safety intelligence

  • Identify actions taken to mitigate risks/harms, change practice, and patient safety culture

• Assess how organizations want to continue with the diagnostic safety process (SUSTAINABILITY)

Materials

• Copies of the Measure Dx materials

• Informed consent documents

• Digital recorder

Location

Interviews will take place at the setting at a time convenient to the participants and may be conducted remotely via videoconference.

Informed Consent Procedures

Participants will complete the informed consent process prior to starting the interview.

Participant Stipends

None.

Each interview is expected to take no more than 60 minutes.

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

WELCOME AND INTRODUCTION

• Thank you for agreeing to participate in this interview, which is part of the larger Diagnostic Safety Measurement project.

• My name is [INSERT NAME OF INTERVIEWER] and I am here to ask you a few questions about Measure Dx.

• With your permission we will also be audio recording the session. This will help make sure we capture all the information you are sharing.

• During this interview, I will be asking questions about your experience with the resource and the impact of the resource on your diagnostic safety policies and activities.

• Everything you say here will be kept confidential. Your insights will used as part of our assessment on the feasibility of implementing the resource, and its materials, into practice. We will not share your name or attribute any of your words directly to you. If at any time you wish to withdraw from taking part in this interview, please let me know and we will stop the interview.

• Do you have any questions before we begin? Ok, great. Let’s get started.

[PRIOR TO THE INTERVIEW, INTERVIEWER WILL REVIEW PAST INTERVIEW AND CALL NOTES, AS WELL AS RESPONSES TO POST-TEST SURVEYS, TO INFORM SPECIFIC QUESTIONS AND PROBES BELOW.]

FEEDBACK ON EXPERIENCE WITH RESOURCE AND FIDELITY OUTCOMES

I would like you to reflect on the process of using Measure Dx for these past 6 months.

1. Based on our past conversations and monthly calls, it sounds like your organization used Strategy [A, B, C, D] – is that correct?

   1. [PROBE IF NECESSARY (eg, unexpected response)] How did you make the decision(s) on which strategy to use [Strategy A, B, C, D]?

   2. How did [YOU/YOUR TEAM] prepare your organization for measuring diagnostic safety?

      1. PROBE: Did you discuss the strategy with other people in the organization? If so, who? What was their reaction to the strategy?

2. What implementation challenges did you face?

3. How did you address implementation challenges ?

4. Did individuals join or leave the team during the project period?

   1. PROBE: If yes, please discuss why/how that happened.

OUTCOMES OF THE TEAM’S ACTIVITIES

1. Could you share how using Measure Dx affected how your team identified and learned from diagnostic safety events?

   1. [PROBE] Why do you think that it was improved/not changed/declined?

   2. How did using Measure Dx affect the way your team’s diagnostic safety work was organized or structured?

   3. Can you describe some types of diagnostic improvement opportunities that your team identified?

   4. Can you describe any cases that resulted in changes or further investigation?

      1. PROBE: Were there any significant discoveries that warranted more immediate action? [IF YES] What types of changes were made in response to these cases?

2. What elements of the resource could be improved to make it more effective?

IMPACT ON DIAGNOSTIC SAFETY OUTCOMES

Let’s talk about any changes to policies or processes that your organization might have made because of implementing the Measure Dx. [REVIEW AND ACKNOWLEDGE ANY CHANGES NOTED ON SAFER DX CHECKLIST SINCE BASELINE. ASK FOR ELABORATION OF CHANGES USING EXAMPLE PROBES FOR QUESTION 1 BELOW.]

1. After implementing the resource, how has the organization changed in terms of addressing diagnostic errors?

1. PROBE: Have the roles/responsibilities of the [INDIVIDUAL/TEAM/PROGRAM] to address diagnostic error changed since using the resource?

2. PROBE: Has the reporting environment in your organization changed?

   1. If no, do you think it will change as a result of this resource?

3. PROBE: Do you believe staff feel more comfortable about voicing their concerns and ideas about diagnostic safety after using this resource?

4. PROBE: Have discussions of diagnostic errors in forums such as morbidity and mortality (M&M) rounds changed after using this resource?

5. PROBE: Has the way in which your executive leadership hears about diagnostic errors changed after using this resource?

2. What changes, if any, have been made to reinforce clinician training about diagnostic errors?

6. PROBE: Have there been any changes to how clinical providers are evaluated professionally as a result of this diagnostic measurement work?

3. What changes, if any, have been made to encourage patient engagement in diagnostic safety?

   1. PROBE: Have there been any changes as to how patients are educated about diagnostic error (e.g., what to do/who to call if they have questions)?

4. How are diagnostic errors being communicated or disclosed to patients?

   1. PROBE: Do you think the communication of diagnostic errors to patients will change in the future?

5. How has the resource affected [YOUR/YOUR TEAM’S] awareness of the issues surrounding diagnostic safety in your organization?

SUSTAINABILITY

1. Please describe how much the time and/or effort it took from yourself or your staff to implement the measurement strategy(ies)?

   1. Are there materials within Measure Dx that were more important to adopt than others?

      1. PROBE: If yes, could you please explain what was more important, and why?

   2. Do you believe that use of the resource will continue and/or spread in your setting?

      1. [IF YES] Can you describe why?

      2. [IF NO] Can you describe why?

2. Your organization used Strategy [A, B, C, and/or D]. Are there other measurement strategies that you think your organization will expand to include over time?

   1. Why or why not?

3. What advice would you give an organization that is deciding whether to implement this resource?

CLOSING

1. How would you summarize your experience using Measure Dx?

2. Is there anything else you would like to share that we have not discussed?

Thank you for your time and participation in this interview. Your insights will be very helpful to improve the resource and diagnostic safety in the future.

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

10