Form #3 Form #3 Key Informant Interview Discussion Guide

Agency for Healthcare Research and Quality Evaluation of the National Guideline Clearinghouse™ Key Informant Interview
Discussion Guide

Agency for Healthcare Research and Quality

National Guideline Clearinghouse™ (NGC) Evaluation

Key Informant Interview Protocol

DRAFT

This guide will serve as the basis for the key informant (KI) interview. Currently, the questions included in this guide assume that key informants will include both users and non-users of the NGC. As documented in the notes throughout the guide, some questions will be specific to those who are users.

The flow of topic areas is similar to the flow of questions contained in the NGC Evaluation Survey. The first 10-15 minutes of the KI interview will pertain to the participant’s use of guidelines and, in the case of NGC Users, the participant’s experience with NGC. The second component of the KI interview has been tailored to the issues most relevant to the KI user type (e.g., guideline developer, clinician, informatics specialist, etc.)

---------------------------------------------------------------------------------------------------------------------

Screening questions would have already been asked to key informants. The interviewer will have this information on the KI before beginning the interview.

Screener questions previously asked to KI:

1) Do you use clinical guidelines in your work?

a. All informants answered Yes.

2) How many times per year, on average, do you use clinical guidelines in your work?

3) Are you aware of the National Guideline Clearinghouse?

a. Some KIs may not be aware of the NGC

4) Have you used the NGC to access information on clinical guidelines?

a. KIs may be users and non-users of the NGC.

---------------------------------------------------------------------------------------------------------------------------

Key Informant:

Welcome/Background [5 minutes]:

• Interviewer introduction

• NGC overview

• Purpose of the evaluation

• Importance of their perspective/How it will be incorporated into the evaluation

• Interview logistics

• Opportunity for questions

Discussion Questions

Background/Screening [5 minutes]

First, as you know, in preparation for this interview we have gathered background information on your use of clinical guidelines and the NGC. I am going to ask similar questions as our starting point.

Question: I understand you use clinical guidelines in your work. Can you briefly describe how and how often you use clinical guidelines in your work?

Probe: potential reasons they use clinical guidelines may include:

- Finding clinical practice guidelines - School assignment

- Comparing guidelines - Professional knowledge building

- Development of a clinical practice guideline - Supporting clinical decision-making

- Development of quality measures - Reducing errors/malpractice (risk management)

- Academic/medical research - Determining coverage of services

- Rationalizing and controlling healthcare expenditures

Question: Are you aware of the National Guideline Clearinghouse (NGC)?

b. How did you learn about the NGC?

c. How long have you been aware of the NGC?

Use of NGC & Other Guideline Sources [10 minutes]

Now, we will discuss how you use the NGC and other guideline sources when you need to locate information on clinical guidelines.

Question: Of the times you have needed to locate information on clinical guidelines, what percentage of the time do you use the NGC for this information?

a. For those times when you do not use the NGC for clinical guideline information, where do you access the information?

Probe: Potential other sources may include:

• Pubmed/Medline

• From a search using a general purpose Web search engine (e.g., Google)

• Point-of-care Web-based resources (e.g., DynaMed, Essential Evidence Plus, EMedicine)

• Medscape

• U.S. Preventive Services Task Force

• National Institute for Health and Clinical Excellence (NICE)

• Guidelines Advisory Committee (Canada)

• Guidelines International Network (G-I-N)

• Institute for Clinical Systems Improvement (ICSI)

• Colorado Clinical Guidelines Collaborative (CCGC)

• Milliman Care Guidelines®

• Expert Consensus Guidelines® (EKS®)

• Medical Specialty Society guidelines (e.g.,AACE, AAFP, AAN, AAO, AAOS, AAP, ACC, ACCP, ACEP, ACP, ACPM, ACR, AGA, ASCO, ASGE, IDSA, SAGES)

• Professional or Disease Specific Society guidelines (e.g., AASM, AARC, ACS, ADA, AHA, ASPEN, BTF, HFSA, NKF, SCCM)

• Government guidelines (e.g., CDC, NHLBI, SAMHSA, TFCPS, DoD/VHA)

• Commercial products (e.g., UpToDate; ACP Pier) (please specify)

• Other government entities

Question: How long have you been using the NGC website? (NGC users only)

Question: How often would you say you have used the NGC website in the past 12 months? (NGC users only)

a. For those who have not visited the NGC in the past 12 months, why not?

Probe: Potential reasons may include:

- No need to locate clinical guideline information

- Need the information, but go without it;

- Need the information, but use other guideline source instead of NGC

Question: Thinking of the guideline source that you frequent most often other than the NGC, how often have you used the source in the past 12 months?

Effectiveness and Influence of NGC and Other Guideline Sources [20 minutes]

Now that we have discussed how and how often you use the NGC and other guideline sources to support your work with clinical guidelines, we are going to discuss how well these tools fulfill your needs regarding clinical guidelines.

Question: We previously discussed how you use clinical guidelines. Summarize how they use guidelines from Question 1. Keeping these thoughts in mind, to what extent does the NGC site fulfill these needs? How? (NGC users only)

Probe: If there are ways in which the participants use clinical guidelines, but do not use the NGC to assist in that activity, why not?

Question: When thinking of how you use clinical guidelines, to what extent does another guideline source besides the NGC fulfill these needs? How?

Question: Overall, how would you rate your satisfaction with the NGC compared to the other guideline source you frequent most often? (NGC users only)

Options to consider in your response, may include: less satisfied with the NGC, satisfaction about equal, more satisfied with the NGC

Probe: Why more/less satisfied with the NGC?

Question: Do you find the NGC criteria for guideline inclusion appropriate?

Probe: Why/why not? Too stringent/loose?

The four criteria for NGC guideline inclusion are:

a. The guideline must contain systematically developed statements that include recommendations that assist in making decisions in specific circumstances.

b. Guideline must be produced, sponsored, or supported by an organization, association, society, or government agency. An individual without sponsorship cannot submit a guideline.

c. There must be documentation that verifies a literature search and a review of existing evidence was performed as part of developing the guideline.

d. Full text must be available in English.

Question: Do you feel the NGC saves you time regarding your work with clinical guidelines? (NGC users only)

b. If yes, how much?

c. How does it save you time?

Question: Do you trust the information provided by the NGC (both NGC content and the guideline which are included in NGC)? (NGC users only)

Probe: Why/why not?

Question: Thinking of the guideline source you most frequently use other than the NGC, do you trust the information provided?

Probe: Why/why not?

Probe. More/less than information from NGC?

Question: Clinical guidelines are used for variety of purposes. What is your perception of the influence of NGC on:

a. Achievement of organization goals?

b. Development of actionable knowledge?

c. Individual practice and learning?

d. Research?

e. The work of Quality Improvement Organizations?

Question: Generally, clinical guidelines are expected to contribute to the improvement of quality in healthcare, promote efficiency, reduce healthcare costs and improve clinical decision making. What is your perception of NGC’s impact on:

a. Healthcare quality (both in general and specifically pertaining to the type of key informant)?

b. Efficiency?

c. Healthcare costs?

d. Clinical decision making?

STAKEHOLDER SPECIFIC QUESTIONS AT THIS POINT IN THE DISCUSSION GUIDE. [15 minutes]

Guideline Developer

Question: Do you (or your organization) submit guidelines for inclusion in the NGC?

Probe: Why/why not?

a. If yes, approximately what percentage of developed guidelines do you submit to NGC?

Question: For those of you that do submit your guidelines to the NGC, how would you rate the following components of the NGC process? Options to consider in your don’t know, poor, fair, neutral, good, excellent

• Your experience with the submission process

• Providing copyright

• Preparation of the NGC Summary and abstraction of your organization's guidelines

• Verification process (per each guideline summary and/or annually)

• Dissemination of your organization’s guidelines

• Response to FDA warnings

Question: How, if at all, does NGC influence your organization’s guideline development program?

Probe: guideline topic selection, guideline development methodology, documentation and reporting of guidelines, guideline update frequency, collaboration with other guideline developers

Now that we have discussed how the NGC has influenced your organization’s guideline development program, the next three questions address how the NGC has influenced other related activities.

Question: For those of you whose organizations have created implementation tools that can be used to facilitate implementation of your organization’s guidelines, how, if at all, has NGC influenced this activity?

Question: For those of you whose organizations have created quality measures that can be used to assess the implementation of guidelines, how, if at all, has NGC influenced this activity?

Question: For those of you whose organizations have developed guidelines in a format that can be integrated into electronic medical records or other decision clinical support tools, how, if at all, has NGC influenced this activity?

Question: To the best of your knowledge, does your organization use other AHRQ-sponsored products to inform its guideline development activities? Examples include:

• Systematic evidence reviews

• Technology assessments

• Comparative effectiveness reviews

• Technical briefs

• Policymaker guides

• DEcIDE project reports

• CERT project reports

• Other products from AHRQ’s Effective Health Care Program

Probe: Why/why not? How?

Medical librarians

Question: How, if at all has NGC influenced the following activities?

• Your data collection processes

• Your approach to identifying guidelines

• Your ability to meet your client needs regarding evidence-based clinical practice guidelines

• Your ability to identify “high quality” guidelines

• Your ability to identify “current guidelines

Are there other activities that NGC has influenced? How?

Question: NGC currently uses the UMLS Metathesaurus to index the guideline summary content included on the site to support browsing and searching. Should NGC use other controlled medical vocabularies for this purpose?

Probe: Why/why not? Which ones?

Informatics Specialist

Question: Do you currently USE or DEVELOP Web 2.0 applications (e.g., blogs, wikis, RSS,) to communicate about NGC guidelines?

Probe: Why/why not? How do you use these tools?

Question: Do you utilize NGC data as an input in developing clinical information or decision support systems?

a. If not, why not?

b. If yes, what type of data and type of “Application Programming Interface (API)” would you like NGC to provide?”

Probe: Why/why not? How? If not, can you think of some ways that NGC content might be used?

Question: Do you utilize any of NGC’s RSS downloads?

Probe: Examples include: NGC content inventory (e.g., NGC Summaries, Expert Commentaries, and Guideline Syntheses or “What’s new” file. Why/why not? How? Could other NGC RSS feeds be useful for informatics specialists?

Question: Do you utilize the NGC Search Form Feature which allows developers to create search interfaces with NGC on external sites?

Probe: Why/why not? How? If not aware of this feature (describe), now that you are, would you use it?

Question: Would having an NGC output for individual guidelines available as an XML file according to the Guideline Elements Model (GEM) be useful to you in your work?

Probe: Why/why not? How?

Researchers

Question: How, if at all, has NGC influenced the following activities?

• Your data collection processes

• Your approach to identifying guidelines

• Your ability to identify “high quality” guidelines

• Your ability to identify “current guidelines

Are there other activities that NGC has influenced? How?

Question: In your research, do you use other AHRQ-sponsored products? Examples include:

• Systematic evidence reviews

• Technology assessments

• Comparative effectiveness reviews

• Technical briefs

• Policymaker guides

• DEcIDE project reports

• CERT project reports

• MEPS: Medical Expenditure Panel Survey

• HCUP: Healthcare Cost & Utilization Project

• HCUPnet: Interactive Tool for Hospital Statistics

Probe: Any others? Why/why not? How?

Healthcare Purchaser, Policymakers, Other

Question: How, if at all, has NGC influenced the following activities?

• Your clinical decision-making processes

• Your organization’s approach to public policy making

• Your efforts to convert clinical information to knowledge that can be acted on

• Utilization management

• Medical reimbursement practices

• Your organization’s implementation of clinical practice guidelines

Question: To the best of your knowledge, does your organization use other AHRQ-sponsored products to inform its guideline development activities? Examples include:

• Systematic evidence reviews

• Technology assessments

• Comparative effectiveness reviews

• Technical briefs

• Policymaker guides

• DEcIDE project reports

• CERT project reports

• Other products from AHRQ’s Effective Health Care Program

Probe: Why/why not? How?

Measure Developers

Question: How, if at all, has the NGC influenced your development and/or use of quality measures?

Probe: Any other areas where the NGC has had influence?

Question: Does your organization use NGC as an input for its measure development activities?

Probe: Why/why not? How?

Question: Does your organization submit its measures to AHRQ’s National Quality Measures Clearinghouse?

Probe: Why/why not?

Additional Considerations [5 minutes]

Question: The NGC is undergoing a redesign. We would like to discuss your thoughts and potential use of the following enhancements:

• Subject-specific email alerts regarding new and updated guideline releases

• Ratings of a guideline’s quality and/or methodologic rigor (e.g., AGREE score)

• NGC user forums (e.g., blogs, bulletin boards) that promote discussion, education, and collaboration on clinical guidelines?

• NGC summary content formatted for mobile device (e.g., cell phone, pocket PDAs) viewing

• Additional data exporting options

• The ability to download references into a citation manager utility (e.g., Endnote)

• The ability to limit searches of NGC to data contained in specific fields in the NGC summary

• The ability to export the entire NGC database

• The ability export targeted elements of the entire NGC database

• The ability to search archived guideline content

• Access to archived guidelines

Probe: Would you like to see any other additional functionality?

Question: Are there any specific changes that AHRQ could make to improve your organization’s experience with NGC?
Probe: What kinds of changes?

Question: Based on our discussion today, did you gather new knowledge of the NGC?

a. What did you learn?

b. Does this new knowledge of the NGC increase your likelihood of using it in the future?

Question: Are there any final comments or questions?

This concludes our Interview. Thank you for your time and input. AHRQ greatly appreciates your participation.

NGC Evaluation Key Informant Interview Guide 11 | Page