Form 2 Attachment B -- Remote Follow-up Interview Guide for LHS

Attachment B -- Remote Follow-up Interview Guide for LHS Leaders

Form Approved
OMB No. 0935-XXXX
Exp. Date XX/XX/20XX

Learning Health System Product Evaluation

Later Leadership Interview Guide (2)

For the interviewer:
The purpose of this interview is to learn about the experiences of Learning Health System (LHS) leadership and staff who have incorporated this product and its contents into their work. Topics covered in the interview are LHS leaders’ experiences with the product(s), including supporting the products’ adoption, barriers and facilitators to using the product(s), resources needed to use the product(s), and the sustainability of the product(s).

The telephone interviews, each lasting 60 minutes, will take place toward the conclusion of the implementation period (month 10).

LHS and implementation product (e.g., Johns Hopkins, triage tool):

LHS leader or staff person name/role:

Interviewer Name:

Date:

Introduction

Thank you for agreeing to talk with us today. My name is ________________ and I will be conducting the interview. I am joined by _______________, who will be taking notes. We work with the American Institutes for Research (AIR), a non-profit evaluation organization that is providing the Agency for Healthcare Research and Quality (AHRQ) with support in developing, implementing, and evaluating evidence-based products to help learning health systems use the findings from Evidence-based Practice Center (EPC) systematic reviews.

The information you provide today will help AHRQ improve the uptake of evidence-based products, like the [PRODUCT NAME(S)], in learning health system settings.

In our conversation today, we would like to learn about how implementation of the [PRODUCT NAME(S)] went for your health system. We would like to learn about experiences using the [PRODUCT NAME(S)] throughout the whole implementation period. Specifically, we are interested in learning about barriers and facilitators to adopting and using [PRODUCT NAME(S)], how you allocated resources to implement [PRODUCT NAME(S)], whether [PRODUCT NAME(S)] influenced practices at the health system, and, if so, how.

Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, # 07W41A, Rockville, MD 20857.

We are not evaluating you or your performance in any way. We are simply interested in hearing your perspectives and opinions. We have a series of questions to help guide our discussion. There are no right or wrong answers to these questions. If there are any questions that you do not feel knowledgeable about or do not feel comfortable answering, just let us know and we will move on.

With your permission, we would like to record today’s discussion. Our research staff will have access to the transcript from the interview. If, at any point, you would like to go off the record or would like a comment to be anonymous, please let us know, and we will stop the recording. We will restart the recording only after you have allowed us to do so.

Do you have any questions before we begin?

Do we have your permission to record the discussion? [If yes, turn on recorder.]

Interview¹

We would like to start with a few questions about you and your role. (Ask only if not interviewed in the early round of interviews.)

1. Please tell us about your main role and responsibilities at [LHS].

1. How long have you been involved with [LHS] in this capacity?

2. How did you become involved in the implementation of [PRODUCT NAME(S)]?

A. Implementation plan update (Leaders who are involved with planning only)
We’d like to get a sense of how the implementation has been going since we last spoke.

1. How has the implementation been going compared to what was planned? [implementation update]

1. Have there been any changes to the implementation from what was originally planned? [implementation update]

2. If so, what are those changes? [implementation update]

Probe: Changes to staff involved with the implementation, the timeline, how the information will be used, or how the [PRODUCT NAME(S)]’s content will be introduced to end-users.

3. What led to any of the changes you have described? [implementation update]

2. Are there any changes you would make to your approach to future implementations based on what you have learned? [implementation update]

3. Is there anything you didn’t know earlier in implementation that you wish you had known? [implementation update]

B. Experiences with the product and fit to LHS
The next topic I would like to explore is your experience implementing [PRODUCT NAME(S)].

1. What were your early impressions of the [PRODUCT NAME(S)]?² [relative advantage, acceptability, and compatibility]

1. Did your impressions of the [PRODUCT NAME(S)] change over time? [relative advantage, acceptability, and compatibility]

   1. If so, how? [relative advantage, acceptability, and compatibility]

   2. What led to those changes? [relative advantage, acceptability, and compatibility]

2. Were there any aspects of the presentation of the information that appealed to you or you found useful? [adoption]

1. What made the presentation of the information appealing or useful? [adoption]

1. Conversely, what aspects of the presentation of the information were not appealing or useful? [adoption]

1. What made the presentation of the information unappealing or not useful? [adoption]

2. How did you use the [PRODUCT NAME(S)]? [adoption]

1. Was this how you expected to use the product(s)? [adoption]

2. Did you encounter any challenges in using the product(s)? [adoption]

3. How did others use the [PRODUCT NAME[S)]? [adoption]

1. Was this how you expected them to use the product(s)? Please explain. [adoption]

2. Did they encounter any challenges in using the product(s)? [adoption]

4. What was your experience with implementing the product content? [relative advantage, acceptability, and compatibility]

5. Did the [PRODUCT NAME(S)] information meet the needs at the health system level as you expected? [appropriateness]

1. Why or why not? [appropriateness]

2. What aspects of the [PRODUCT NAME(S)] were less useful and what was more useful? [appropriateness]

1. Please explain. [appropriateness]

3. What aspects of the [PRODUCT NAME(S)] could be improved and what is essential to maintain? [appropriateness]

1. Please explain. [appropriateness]

6. Did you access other products after using [PRODUCT NAME(S)]? [feasibility]

1. If yes, what were those products? [feasibility]

2. How were they used? [feasibility]

3. How did they fit with [PRODUCT NAME(S)]? [feasibility]

4. Did the [PRODUCT NAME(S)] reduce the health system’s need for other information resources? [feasibility]

7. Have you used other products like [PRODUCT NAME(S)]? [feasibility]

1. If yes, is the [PRODUCT NAME(S)] content more or less implementable as other products designed for similar purposes? [feasibility]

1. Why or why not? [feasibility]

2. What might those other products be? [feasibility]

8. Have you used systematic reviews in the past? [adoption]

1. If so, has this product changed the way you think about systematic reviews? [adoption]

1. If so, how? [adoption]

2. If not, why? [adoption]

C. Use of product by staff
Next, I would like to ask about who within your system has used [PRODUCT NAME(S)].

1. What is your best estimate of how many administrative, leadership, clinical groups, and other types of groups used the product(s)? [reach]

1. Please tell me which groups you included in this count. [reach]

1. Specifically, who used the product(s)? [reach]

2. How many people are in [name each group]? Please provide your best estimate if you are unsure. [reach]

1. How many of these people in each group actually used the product? [reach]

2. How were staff exposed to the evidence highlighted in [PRODUCT NAME(S)]? [reach]

1. Did staff use [PRODUCT NAME(S)] to convey the key evidence descripted in the products? [adoption]

2. Did staff use or develop other resources to convey the key evidence described in the product(s)? [adoption]

3. If other resources were accessed or created, please tell me about those (e.g., who created the resources, content, format, level of detail, how used). [adoption]

3. Did the use of [PRODUCT NAME(S) and/or related resources] differ from what you expected? [relative advantage, acceptability, and compatability]

1. If yes, please tell me what happened. [relative advantage, acceptability, and compatability]

2. Why do you think this occurred? [relative advantage, acceptability, and compatability]

3. Under which circumstances did staff choose to use or not to use [PRODUCT NAME(S) and/or related resources] or its/their content? [relative advantage, acceptability, and compatability]

4. How challenging or easy was it to encourage use of the [PRODUCT NAME(S) and/or related resources]? [feasibility]

1. What factors do you think facilitated or limited use of [PRODUCT NAME(S) and/or related resources] or application of the evidence? [feasibility]

2. Under what circumstances did staff choose to use or not use the [PRODUCT NAME(S) and/or related resources] and application of the evidence? [feasibility]

5. Who used the product content? [feasibility]

1. How was the product content (not the product) used? [feasibility]

2. What challenges, if any, were encountered in applying the product content? [feasibility]

6. What feedback have you heard about the [PRODUCT NAME(S)] and its content? [relative advantage, acceptability, and compatability]

D. Change within the health system and sustainment of the change
Next, we are going to ask you some questions about any health system changes that have occurred associated with using [PRODUCT NAME(S)], and the sustainability of those changes.

1. Has the health system taken any unit specific or institution wide action steps after using the [PRODUCT NAME(S)]? [actionability]

1. If so, what were those steps? [actionability]

1. What content from the [PRODUCT NAME(S)] led to those changes? [actionability]

2. What does the health system hope to achieve by making those changes? [actionability]

3. How were changes measured? [actionability]

2. If not, was there a reason why? [actionability]

3. Are any action steps planned for the future? [actionability]

1. If so, what are they? [actionability]

2. Do you feel that the evidence from [PRODUCT NAME(S)] will be used routinely once the project ends? [sustainability]

1. Why or why not? [sustainability]

2. Are there any approaches to reinforce the new practices or processes that are in place? [sustainability]

1. Is the evidence from [PRODUCT NAME(S)] part of a decisionmaking or clinical workflow or is it included in a policy or procedure? [sustainability]

2. Has it been integrated into the EHR? [sustainability]

3. Is the evidence from [PRODUCT NAME(S)] part of a written procedure or performance expectations or is it reinforced through other means of assigning accountability? [sustainability]

3. Is use of the product(s) being reinforced at multiple levels within your health system and by multiple people? [sustainability]

1. Can you explain? [sustainability]

4. How did using the product(s) compare to accessing a systematic review report for information on a topic? [adoption]

1. Was there any difference in the time required? [adoption]

2. Was there a difference in the ease of finding information relevant to your health system? [adoption]

3. Was there a difference in what you did with the information you found? [adoption]

5. What changes to the product would make it more useful to you? [relative advantage, acceptability, and compatability]

E. Implementation resource costs and costs of care
We’d now like to hear about costs associated with using the [PRODUCT NAME(S)] and any benefits offsetting those costs.

1. What resources were required to introduce the product to decisionmakers (e.g., time to review the product(s) and prepare to share the information with others, time to do further research on the topic meeting time)? [implementation cost]

1. Was this more or less or about the same as you expected? [implementation cost]

2. What types of resources were required to plan and roll out the information and any changes across your system? [implementation cost]

Probe: This might include time for planning, coordination with others, creation of materials, and training resources, changes to the EHR or other systems, and training.

1. Was this more or less or about the same amount of resources as you expected? [implementation cost]

3. What would your system hope to gain from using the [PRODUCT NAME(S)]? [implementation cost]

1. What benefits, other than what we have discussed, make it worth the effort expended? [implementation cost]

4. Did any changes that occurred as the result of implementing the evidence lead to changes in costs to patient care? [cost of care]

Probe: These changes to costs may be related to different types of staff delivering care, the amount of staff time to deliver care, intensity of care needed, length of stay, or the use of new or different equipment, medication, or use treatment procedures.

1. If yes, what were those changes? [cost of care]

2. Who incurred the added costs or the cost savings? [cost of care]

5. Do you expect any other implementation costs or changes to patient care costs to occur in the future? [cost of care]

1. If so, what are these costs? [cost of care]

2. If not, why? [cost of care]

F. Closing

We have just a few more general questions before we wrap up.

1. What did you originally think about the EPC Program systematic reviews?

1. Has using [PRODUCT NAME(S)] changed the way you think about EPC Program systematic reviews?

2. If so, how?

2. Is there anything about the implementation and the [PRODUCT NAME(S)] that you would like to add that we have not covered?

3. Do you have any questions for us?

Thank you so much for your time.

1 The interviewer will tailor the questions to the product being piloted. If more than one product is selected, the interview questions will be asked of each product. In cases where the products are used for different purposes, the interviewer will refer to the product that is most applicable to the question.

2⁷ The interviewer will tailor the questions to the product being piloted. If more than one product is selected, the interview questions will be asked of each product. In cases where the products are used for different purposes, the interviewer will refer to the product that is most applicable to the question.