6 Appendix F

Appendix F – Pilot Test Evaluation Protocol for Practice Administrators

AHRQ – Building Diagnostic Safety Capacity

Practice Administrators – Patient and Family Engagement Resource Pilot Test Evaluation

MedStar Health Research Institute (MHRI) will conduct interviews with administrators from up to twenty (n=20) practices. Evaluation will be completed within 3-6 months after implementation of the Patient and Family Engagement Resource.

• 20-60 cognitive interviews with practice administrators (1-3 practice administrators per practice x 20 practices); each interview will last approximately 60 minutes

Recruitment Criteria

Ambulatory care practice administrators who have been exposed to the PFE Resource within their practice will be eligible to participate in the interview. MHRI team members will work with the practice champions to identify practice administrators to participate in the interviews. We will aim to recruit practice administrators in the following manner:

• Practice administrators involved in the decision making around implementation of the PFE Resource or how have had experience using the PFE Resource

Interview Goals

The goals of the interviews will be to:

• Obtain feedback on the intervention PFE Resource materials

• Obtain feedback on the barriers and facilitators encountered during implementation

• Obtain feedback on satisfaction with instructions and materials

• Obtain feedback on receptivity and enhancements to the PFE Resource materials to improve adoption and implementation

Interview Materials

• Copies of the PFE Resource

• Informed consent documents

• Documentation for Processing of Participant stipends

• Digital recorder

Location

Interviews will be conducted at a location within the practice or within the practice’s community. Interviews may also be conducted virtually via the telephone.

Participant Stipends

None.

Informed Consent Procedures

Participants will complete the informed consent process at the time of arrival to the interview.

Proposed Agenda – Practice Administrators Interview

Interviews will be planned for approximately 60 minutes each.

Agenda
Introduction	5 minutes
Background	10 minutes
Review Materials	10 minutes
General Experience with Intervention (satisfaction/barriers/enablers)	15 minutes
Enhancements	15 minutes
Closing	5 minutes
Total	60 minutes

AHRQ – Building Diagnostic Safety Capacity

Practice Administrators – Patient and Family Engagement Resource Pilot Test Evaluation

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

WELCOME AND INTRODUCTION

• Thank you for agreeing to participate in the interview about your experiences with the Agency for Healthcare Research and Quality’s Resource for improving patient and family engagement in the diagnostic process!

• My name is [ INSERT NAME OF INTERVIEWER] and I am the facilitator for today’s conversation. I am here with [INSERT NAME OF PROJECT TEAM] and HE/SHE will be taking notes of our conversation.

• With your permission we will also be audio recording the session. This will help make sure that we don’t miss anything that you say and can share with other people who are working on this project. The recording will be deleted after we have the notes transcribed and are sure we have captured all your comments accurately.

• TODAY/TONIGHT we will be asking you questions about your experiences with using the PFE Resource and your practice’s experience implementing it.

• Nothing that you say or share today will be shared in an identifiable way with your practice but represented together along with nine other practices in a report to the Agency for Healthcare Research and Quality. We will not share your name or anything that you say with them in a personally identifiable way.

• Do you have any questions before we begin?

So let’s get started.

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

I would like to begin by asking you all some questions about diagnostic error and the diagnostic process in your practice.

DIAGNOSIS

1. Prior to the signing up for the pilot test, what was your practice doing in terms of improving diagnosis and diagnostic error mitigation?

   1. Was this a strategic priority for your organization?

   2. What were the key factors that underpinned your decision to join the pilot test?

   3. Were any of the materials provided at the time of recruitment helpful to this process?

   4. What information could we have provided to make this intervention more attractive to you as a practice administrator and leader?

2. How would you describe the impact of the PFE Resource on the safety of the diagnostic process in your practice?

1. Why do you think that it was improved?

   1. Are you actively measuring and tracking these improvements?

   2. Are there other improvements that are harder to track? If yes, what would those be?

2. What strategies within the PFE Resource (agenda setting tool, poster, one-minute of patient talking uninterrupted) do you feel had the greatest impact on the diagnostic process within your practice?

   1. Can you elaborate on why you think it had that impact?

3. What was the most effective part of the PFE Resource within your practice? Can you describe why that is?

Thank you for sharing. I now want to speak with you a little more about patient engagement.

3. As an administrator, what was the impact of the PFE Resource on your clinical team?

   1. What about your practice staff? Did the implementation result in changes in productivity or engagement from your staff? Clinicians? Others?

   2. What about burden? Was this unduly burdensome for your practice to implement?

      1. Compared to other quality improvement activities, how would you rate this activity in terms of burden for implementation? What about sustainability?

   3. Does your practice intend to continue using the PFE Resource?

      1. If yes, what parts of the resource did you find most impactful? How do you know this?

      2. What parts of the process/resource do you intend to keep?

      3. For those parts of the resource that you will continue, are you planning on making any changes to them?

      4. What would make the PFE Resource more useful? Easier to implement? Easier to sustain?

4. Do you believe that the PFE Resource impacted your patients in a positive way?

   1. Were they more or less engaged with the clinicians?

   2. Were the patient resources helpful or did they hinder patient-clinician interactions?

   3. Did patients give you any feedback about the resources or the approach?

      1. How did they describe their experiences?

      2. Where did they provide this feedback to you?

      3. What about measures of patient engagement? Did you see any changes in CAHPS scores or other satisfaction or experience surveys?

5. I would like to focus on the practice’s implementation of the PFE Resource. Can you describe for me how your practice advised the team about implementation?

• Did you use the quick start implementation guide? Do you have any feedback about what was missing in the implementation guidance? Anything that should have been there that would have been helpful? What about things in it that weren’t helpful?

• What could have made the implementation guide more useful?

6. As a practice administrator, was the format of the materials appropriate for how your practice conducts process and quality improvement implementations?

   1. Did it fit within your patient and family engagement strategies? Complement them? Were they at odds with them in any way?

   2. What changes to the materials might support greater adoption and more sustained implementation? Would you be willing to try that?

7. What about costs of implementing the PFE Resource? Was that a challenge at all for you?

   1. How will cost of the implementation influence the decision to sustain the program after the pilot test?

      1. What are the greatest influencing factors with respect to sustainability?

         1. Payment? Cost? Change fatigue? Interest?

      2. How could the PFE Resource better address these issues to make it more attractive to your clinic stakeholders?

8. How much staff time and resources did the implementation of the PFE Resource take? Is this sustainable? What would you say to other practice administrators who are contemplating implementing this resource?

9. Those were all the questions I had today. Are there any questions that I should have asked that I did not?

Thank you for your time and participation in this interview. Your comments will be very helpful to this project and will help us to make important improvements to the PFE Resource!