Form 4 Key Informant Interview Protocols

APPENDIX E

key informant Interview Protocol

Form Approved
OMB No. 0935-XXXX
Exp. Date XX/XX/20XX

EVALUATION OF
AHRQ HEALTHCARE HORIZON SCANNING SYSTEM

KEY INFORMANT INTERVIEW PROTOCOL

Sponsored by the Agency for Healthcare Research and Quality

Conducted by

M

Public reporting burden for this collection of information is estimated to average XX minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 540 Gaither Road, Room # 5036, Rockville, MD 20850.

athematica Policy Research

1. GENERAL INTRODUCTION AND BACKGROUND

We appreciate you taking the time to speak with us today. Before we begin, let me introduce myself and explain what we will be doing. My name is [NAME], and I work for Mathematica Policy Research, a social policy research firm. We are conducting an evaluation of the Healthcare Horizon Scanning System, which we will refer to as the “horizon scanning system.” As part of the evaluation we are talking with a variety of people who participate in the horizon scanning system in order to:

• Learn which elements of the horizon scanning system protocol are working well and the reasons why they are working well; and

• Understand which elements of the horizon scanning system protocol can be improved, how they might be improved, and the relative importance of suggested improvements.

The results of our discussion will be synthesized in a final report. Only general themes that emerge from our discussions will be reported. We will not attribute specific comments or quotes to named individuals. Participation in the interview is voluntary.

We expect this discussion to take about [NUMBER] minutes. Do you have any questions before we begin?

2. SCANNING AND LEAD SELECTION

I would like to start by talking about scanning and lead selection. (Sc, LM, A, PM, CTL)

1. Please give us one or two examples of how scanning and lead selection is working well. (Prompt: For example, what specific sources or types of sources produce particularly high quality information?) (Sc, LM)

2. What suggestions do you have about ways to change the list of sources? (Prompts: What specific sources or types of sources regularly fail to produce useful information? What additional sources or types of sources are needed?) (Sc, LM, A)

3. What suggestions do you have for how to improve the questions intended to help identify leads? (horizon scanning system protocol pp. 7–12) (Prompts: What questions should be added or removed? What suggestions do you have for making the questions easier to apply? Would providing additional guidance to scanners about assigning leads to priority areas be helpful?) (Sc, LM)

4. How is the crosscutting priority area used? (Sc, LM, A, PM, CTL)

   1. Do [you (Sc)/scanners (LM, CTL, PM)] have difficulty determining whether leads should be assigned to the crosscutting priority area? If so, what kinds of difficulties do they encounter? And in what situations? (Sc, LM, PM, CTL)

   2. Do [you (A)/analysts (CTL, PM)] have difficulty determining whether interventions should be assigned to the crosscutting priority area? If so, what kinds of difficulties do they encounter? And in what situations? (A, PM, CTL)

3. DEVELOPMENT OF INTERVENTIONS

Now I would like to talk about development of interventions and determination of which interventions are brought to the topic nomination meeting. (A)

1. How would you improve the criteria for entering interventions into the horizon scanning system? (horizon scanning system protocol p. 16) (A)

2. How often are you unsure about whether an intervention should be dropped, voted on via the SharePoint site, or voted on during a topic nomination meeting? (A)

1. What do you do when you are unsure? (Prompts: What is the process? Is there a set process? What changes, if any, should be made to that process?) (A)

4. TOPIC NOMINATION MEETINGS AND PROFILE DEVELOPMENT FOR TARGET INTERVENTIONS

Now I would like to talk about topic nomination meetings and profile development for target interventions. I understand that voting occurs during both virtual meetings and in-person topic nomination meetings. (A, PM, CTL)

[Now I would like to talk about (LM)/I would like to start by talking about (DIS, Se, RM)] communication among searchers and analysts. (LM, DIS, Se, RM)

1. What do you think about the level of preparation that analysts do before virtual meetings and in-person topic nomination meetings? (Prompts: Is it usually the right amount, more preparation than necessary, or less preparation than necessary? How often do interventions that are discussed online and expected to be labeled “track only” end up being discussed in person and labeled “advance to target?” How often do searchers need to conduct ad hoc searches to address questions that come up during the meetings?) (A, PM, CTL)

2. What suggestions do you have for improving aspects of the virtual process? (Prompt: What changes, if any, should be made to how the virtual discussion and voting takes place?) (A, PM, CTL)

3. What aspects of the in-person topic nomination meetings and voting work well? (A, PM, CTL)

4. What suggestions do you have for improving aspects of the in-person topic nomination process? (Prompts: What changes, if any, would you make to the length and frequency of the meetings? Should meetings be restricted to specific priority areas? If yes, what changes would you make? What changes, if any, would you make to who attends the meeting? Would you add additional attendees? If yes, who and what types of attendees would you add?) (A, PM, CTL)

5. What ideas do you have for improving communication among searchers and analysts? (LM, A, DIS, PM, CTL, Se)

6. What ideas do you have for improving communication among analysts and the reference management team? (A, PM, CTL, RM)

7. Please discuss any challenges that have arisen when [you are (A)/the analyst is (Se, CTL, PM)] developing an intervention profile for an intervention that advances to target. (A, Se, CTL, PM)

1. How have [you (A)/analysts (Se, CTL, PM)] responded to these challenges? (A, Se, CTL, PM) (Prompt: What challenges have you faced with conducting searches for the advance to target interventions?)

2. What changes, if any, would you make to the templates used to develop target intervention profiles to make them more useful? (horizon scanning system protocol pp. 20–21) (A)

5. EXPERT INPUT AND DETERMINATION OF POTENTIAL HIGH IMPACT INTERVENTIONS

Next I would like to talk about expert comment and determination of Potential High Impact interventions. (A, PM, CTL)

I would like to talk about the expert review process. (E, PRI)

1. Please provide a couple of examples of how the process of identifying experts and gathering comments is working well. (A, PM, CTL)

2. In what ways can selection of experts be improved? (Prompts: Should changes be made to address the balance of reviewer categories (e.g., health systems, clinical research)? If yes, how so? Should more or fewer experts be contacted on the first pass? If more or fewer, please say more about that.) (A, PM, CTL)

1. What improvements, if any, should be made to the process of identifying and balancing conflicts of interest? (A, PM, CTL)

3. In what ways can communication with experts be improved? (A, PM, CTL, E, PRI)

4. How would you describe the amount of information that you [provide (A)/receive (E)] on an intervention that [experts (A)/you (E)] are asked to review? (Prompt: Is it adequate, too much, or too little information?) (A, E)

1. Please discuss any suggestions you have for improvements to the target technology reports. (Prompt: What types of additional information in the target technology report would be helpful for filling out the horizon scanning system intervention comment form?) (E)

5. What suggestions do you have for improving the Horizon Scanning intervention comment form? (Prompt: Should parameters be added, deleted, or revised? Are changes needed to the four point rating system?) (horizon scanning system protocol pp. 50–52) (A, PM, CTL, E, PRI)

6. What suggestions do you have for improving the process of reading expert comments and comparing comments with ratings? (A, PM, CTL)

7. My final question is about the Potential High Impact reports. Have you had a chance to review at least one of those reports? (E)

1. If so, what suggestions do you have about ways to improve the Potential High Impact reports? (Prompts: Do you think the information should be framed differently? Was there sufficient clinical context? How might the Potential High Impact rating be explained more clearly?) (E)

6. INTERVENTION ARCHIVING, MONITORING, UPDATING, AND REASSESSMENT OF POTENTIAL IMPACT

Now let’s talk about intervention archiving, monitoring, updating, and reassessment of potential impact. (A, DIS, Se)

1. What lessons have been learned from automated daily searching on “track only” and “advance to target” interventions? (Prompts: Are some triggers for updates of interventions more commonly identified than others? (horizon scanning system protocol p. 28) If yes, what are those? How can communication between searchers, analysts, and reference managers be improved?) (A, DIS, Se)

2. What lessons have been learned from active searching on “advance to target” interventions? (Prompt: How could lessons from active searching be used to improve automated daily searching?) (A, DIS, Se)

3. What lessons from automated searches and active searches could be applied to improve scanning for leads? (A, DIS, Se)

7. OVERARCHING AND MISCELLANEOUS QUESTIONS

We’re almost done. Thanks so much for talking with us. (A, PM, CTL)

I have [one (A, PM, CTL)/three (PD)] overarching question[s] for you. (A, PM, CTL, PD)

1. The three main functions of the horizon scanning system are: (1) identification and prioritization of interventions for tracking and monitoring; (2) development of detailed content on target interventions and acquisition of expert opinions about the potential impact of the interventions; and (3) synthesis of perspectives of experts. Which of these functions should receive more resources? Which should receive less resources? (A, PM, CTL, PD)

2. What steps or activities of the horizon scanning system have been the most successful? (Prompts: Scanning? Intervention nomination meetings? Gathering expert reviews? Review of expert comments and scores? Creation of the Potential High Impact reports? Automated searching and intervention monitoring?) (PD)

3. What steps or activities of the horizon scanning system have been the least successful? (Prompts: Scanning? Topic nomination meetings? Gathering expert reviews? Review of expert comments and scores to determine potential high impact interventions? Automated searching and intervention monitoring?) (PD)

[If Appropriate] My final question addresses quantitative analysis and forecasting. (PM, CTL, QA)

4. [If Appropriate] What is working well with the quantitative analysis and forecasting? What suggestions do you have for improving that process? (PM, CTL, QA)

Those are all the questions I have. Do you have any final thoughts about the AHRQ Healthcare Horizon Scanning System that you’d like to share with us?

Thanks again for taking the time to talk with us today. We really appreciate your input. Have a good day. Good-bye.