Form 3 Attachment C -- Remote Follow-up Interview Guide for Cli

Attachment C -- Remote Follow-up Interview Guide for Clinical Staff

Form Approved
OMB No. 0935-XXXX
Exp. Date XX/XX/20XX

Learning Health System Product Evaluation

Later Clinical Staff Interview Guide (1)

For the interviewer:
The purpose of this interview is to learn about the experiences of learning health system (LHS) staff who have incorporated this product and its contents into their work. Topics covered in the interview are LHS staff experiences with the product(s), including supporting the products’ adoption, barriers and facilitators to using the product(s), resources needed to use the product(s), and the sustainability of the product(s).

The telephone interviews, each lasting 60 minutes, will take place toward the conclusion of the implementation period (month 10).

LHS and implementation product (e.g., Johns Hopkins, triage tool):

LHS leader or staff person name/role:

Interviewer Name:

Date:

Introduction

Thank you for agreeing to talk with us today. My name is ________________ and I will be conducting the interview. I am joined by _______________, who will be taking notes. We work with the American Institutes for Research (AIR), a non-profit evaluation organization that is providing the Agency for Healthcare Research and Quality (AHRQ) with support in developing, implementing, and evaluating evidence-based products to help learning health systems use the findings from Evidence-based Practice Center (EPC) systematic reviews.

The information you provide today will help AHRQ improve the uptake of evidence-based products, like the [PRODUCT NAME(S)], in learning health system settings. In our conversation today, we would like to learn about your experiences using evidence-based resources in the past, and your current experiences with the resources you were provided for [LHS’s Name for Project], and applying the evidence described in these resources to patient care. We would also like to learn about what you have observed from colleagues use of the resources and application of the evidence.

Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, # 07W41A, Rockville, MD 20857.

We are not evaluating you or your performance in any way. We are simply interested in hearing your perspectives and opinions. We have a series of questions to help guide our discussion. There are no right or wrong answers to these questions. If there are any questions that you do not feel knowledgeable about or do not feel comfortable answering, just let us know and we will move on.

With your permission, we would like to record today’s discussion. Our research staff will have access to the transcript from the interview. If, at any point, you would like to go off the record or would like a comment to be anonymous, please let us know, and we will stop the recording. We will restart the recording only after you have allowed us to do so.

Do you have any questions before we begin?

Do we have your permission to record the discussion? [If yes, turn on recorder.]

Interview¹

We would like to start with a few questions about you and your role.

1. Please tell us about your main role and responsibilities at [LHS].

1. How long have you been involved with [LHS] in this capacity?

2. How did you become involved in the [LHS name for project] implementation?

3. Were you involved in implementing [LHS name for project] or its content?

1. If yes, what was your role?

A. Experience with other products
To begin with, I would like to know about your experiences accessing information about evidence-based practices and what types of materials you have used.

Note: Staff may be using resources created from PRODUCT NAME(S) and not the actual product(s). Refer to whatever related resources the LHS implementation champion identified as being rolled out to staff.

1. Have there been situations when you were given or sought out resources to learn about patient care practices that are evidence-based?

1. If so, please tell me about those situations.

2. Please describe the resources you reviewed.

3. What was your experience with them?

1. How did you find those resources?

2. How did you use them?

3. What did you like about those resources?

4. What could have been improved?

2. How do/does [PRODUCT NAME(S) and/or related resources] compare with the materials you described? [adoption]

B. Impression and experience with the product and fit to work
Now we’d like to ask you some questions about your experience with [PRODUCT NAME(S) and/or related resources].

1. How did you learn about [the evidence-based practices]? [implementation update]

2. How did you use [PRODUCT NAME(S) and/or related resources]? [relative advantage, acceptability, and compatability]

3. What made you decide to use [PRODUCT NAME(S) and/or related resources]?
   Probe: Did someone ask you to use [PRODUCT NAME(S) and/or related resources]? Was it recommended to you? [adoption]

4. What were your overall impressions of the [PRODUCT NAME(S) and/or related resources] and its/their features? [adoption]

1. Were there any aspects of the presentation of the information that appealed to you or you found useful? [adoption]

1. What made the presentation of the information appealing or useful? [adoption]

2. Conversely, what aspects of the presentation of the information were not appealing or useful? [adoption]

1. What made the presentation of the information unappealing or not useful? [adoption]

5. What type of information did you hope to learn from reading [PRODUCT NAME(S) and/or related resources]? [relative advantage, acceptability, and compatability]

1. Did it/they meet your expectations? [appropriateness]

2. Is the content relevant to the patients you work with? [appropriateness]

3. Is the content relevant to the type of work you do? [appropriateness]

4. Was the content presented at the right level of detail? Please describe. [relative advantage, acceptability, and compatability]

1. If not, why? [relative advantage, acceptability, and compatability]

5. Was the information easy to find? Was it easy to understand? Please describe. [relative advantage, acceptability, and compatability]

6. Was there anything else that made [PRODUCT NAME(S) and/or related resources] easy or difficult to use? [relative advantage, acceptability, and compatability]

1. If yes, please explain. [relative advantage, acceptability, and compatability]

6. Were you able to easily apply what you learned from the product to your work? Please explain. [feasibility]

1. Was there anything about that [PRODUCT NAME(S) and/or related resources] that made it easy or difficult to apply the information to your work? [feasibility]

1. If so, please explain. [feasibility]

2. Have you used [PRODUCT NAME(S) and/or related resources] or learned from it/them in other ways? [fidelity]

1. If so, please explain. [fidelity]

3. Under what circumstances would you choose to use [PRODUCT NAME(S) and/or related resources]? Under what circumstances would you choose not to use it/them or to stop using it/them?

4. If you started to use the product(s) and then stopped, what led you to do so? [adoption]

5. How much time did it take you to learn about the evidence presented in use [PRODUCT NAME(S) and/or related resources]? [implementation cost]

7. Did you access other products after using [PRODUCT NAME(S) and/or related resources]? [fidelity]

1. If yes, what were those products? [fidelity]

2. How were they used? [fidelity]

3. How did they fit with [PRODUCT NAME(S) and/or related resources]? [fidelity]

8. Have you used other products similar to [PRODUCT NAME(S) and/or related resources] in the past? [fidelity]

1. If yes, please tell me about those products. [fidelity]

2. How does/do [PRODUCT NAME(S) and/or related resources] compare? [adoption]

Probe: Is the content or format more or less user friendly or implementable? Is it more or less relevant?

9. Would you seek out this/these product(s) or other tool(s) developed from the product content if it/they were designed to provide information about other clinical effectiveness topics of interest to you? [sustainability]

1. If you would not seek out the product(s) again, what changes would make the product(s) more appealing and useful? [sustainability]

10. Are there any other aspects of [PRODUCT NAME(S) and/or related resources] that we have not discussed that could be improved (e.g., the format, content)? [sustainability]

1. If so, what and how? [sustainability]

11. What parts of [PRODUCT NAME(S) and/or related resources] do you think should remain as they are? Please be specific. [relative advantage, acceptability, and compatability]

C. User reach
We’d like to ask you a few questions about the reach of the product(s)—specifically, about who is using it/them and how they are using it/them.

1. What are you hearing, if anything, from colleagues about the [PRODUCT NAME(S) and/or related resources]? [reach]

2. Are there staff who are not using the product(s) who could benefit from using it/them? [effectiveness]

1. If so, can you explain why they are not using it/them? [effectiveness]

3. Has the product helped clarify evidence so that it can be shared with patients? [effectiveness]

1. If so, how? [effectiveness]

2. If not, why? [effectiveness]

D. Application of the evidence to patient care

We’d also like to ask you about any changes to patient care that have been made as the result of what has been learned from [PRODUCT NAME(S) and/or related resources].

1. Has information learned as a result of using [PRODUCT NAME(S) and/or related resources] led to changes in clinical practices or decisionmaking consistent with the evidence? [effectiveness]

1. If yes, please explain what those changes are. [effectiveness]

2. If yes, how much time is it taking to make those changes? [cost of care]

3. Are the new practices integrated into the clinical workflow? [sustainability]

1. If so, how? [sustainability]

2. If not, please tell me how the changes impact the clinical workflow. [sustainability]

2. Has information learned as a result of using the product(s) led to any changes in the health system? [effectiveness]

1. What does the health system hope to achieve by making these changes? [effectiveness]

2. Are the changes leading to any improvements in the time and effort you or others expend? [cost of care]

1. If yes, please explain. [cost of care]

2. If no, what was the thinking behind not making changes? [cost of care]

3. What did it take to learn new practices that came from [PRODUCT NAME(S) and/or related resources]? [implementation cost]

1. How much effort did it take to learn what you needed to apply the evidence described in [PRODUCT NAME(S) and/or related resources]? [cost of care]

1. Did it require special training? [sustainability]

2. Did it require referring to reminders, such as cue cards, or other approaches to learning new practices? [sustainability]

2. How much time did it take before you became comfortable with these new practices? [implementation cost]

1. Was this about what you expected it would take? [implementation cost]

2. If more time, how much more did it take? [implementation cost]

3. If less, how much time did it take? [implementation cost]

4. Are you receiving reminders or other reinforcements to follow the new practices? [sustainability]

1. Please tell me about how the new practices are being reinforced. [sustainability]

1. By whom or by what? [sustainability]

2. How often? [sustainability]

2. Are the new practices documented in a policy or procedure manual? [sustainability]

3. Are leaders and administrators around you encouraging or expecting you to use or apply the new practices? [sustainability]

1. If so, who are they? [sustainability]

2. If not, do you know why? [sustainability]

5. Do you feel the new practices will be consistently followed once the project ends? [sustainability]

1. Why or why not? [sustainability]

E. Wrap up

1. Do you have any questions for us or anything you would like to say to help us understand your experience with [PRODUCT NAME(S) and/or related resources] and following the new practices?

Thank you so much for your time.

1 The interviewer will tailor the questions to the product being piloted. If more than one product is selected, the interview questions will be asked of each product. In cases where the products are used for different purposes, the interviewer will refer to the product that is most applicable to the question.