Form #4 In-depth Interview Guide

A ttachment C: In-depth Interview Guide

[NOTE: Interviewees will be individuals who used the HIE Evaluation Toolkit to develop an evaluation plan. They will be interviewed either after they have completed the development of an HIE evaluation plan OR 6 months after receiving the toolkit.]

A. INTRODUCTION AND OPENING

Thank you for agreeing to talk to me today. My name is [INTERVIEWER’s name] and I work for Westat.

Westat is conducting this project for the Agency for Healthcare Research and Quality, known as AHRQ. The purpose of today’s discussion is to gather your feedback on the Health Information Exchange–which we will refer to as HIE–Evaluation Toolkit based on your experience using it to develop an evaluation plan over the past few months. We will refer to this as the toolkit.

We would like to understand:

• Your use of this and other tools to develop an HIE evaluation plan.

• Your opinion of how this toolkit assisted you in developing your plan.

• Which information or sections of the toolkit were most helpful.

• If there was any information that was not available and would have been helpful as you developed your plan.

We have 90 minutes scheduled for this call. Does that still work with your schedule?

If the participant cannot dedicate at least an hour to the interview he/she should be thanked and rescheduled.

The results of this project will be provided to AHRQ staff to use in planning to enhance the HIE Evaluation Toolkit and aid its users. The information you share today will help ensure that AHRQ continues to provide relevant and timely support to the health IT and HIE community.

Before beginning, I would like to remind you about how we will use the information you provide. Your responses will be kept confidential to the extent permitted by law, including AHRQ’s confidentiality statute, 42 USC 299c-3(c). Westat may include respondent comments in reports but will not attribute the comments to specific individuals or organizations. Westat also has a system to mark specific comments as off-limits for any reports that might be made public. If any of the information we discuss today falls into this category, please let me know and I will mark the information as off-limits in our notes. Your participation in this study is voluntary. If at any time you do not feel comfortable answering a question, or do not know the answer, please let me know and I will move on. My colleague [STATE NAME] from Westat will take notes during the interview. In addition, we would like to record this session to ensure that we accurately capture your comments.

Do you mind if I record the interview?

If Yes: We will simply take notes, but not record the interview.

If No: Start the recording.

Okay. As you may recall, when you decided to participate in this project, you agreed to use the HIE Evaluation Toolkit when developing your HIE evaluation plan.

During this interview, I will be asking you a series of questions related to your experiences with the use of the HIE Evaluation Toolkit in developing your HIE evaluation plan.

Do you have any questions before I begin?

B. BACKGROUND

First, I would like to get some background information on your evaluation plan and the persons who helped develop it.

Q Are/were you materially involved in developing the evaluation plan? “Materially involved” is defined as participating in design decisions and contributing directly to a significant portion of the evaluation plan.

If No: Ask for name of people who were materially responsible for developing the evaluation plan and thank the individual for his/her time.

Q Were other staff materially involved in developing the plan?

If Yes:

Q2a. How many and what are their roles?

Q2b. How many of them also used the toolkit?

Q Were there requirements from stakeholders, funders, or others that impacted the evaluation plan such as the measures used or timeline for the evaluation?

If Yes:

Q3a. Did the toolkit help you meet those requirements?

Q Who is the audience(s) for your evaluation?

Q Please describe how far you are in the development of your HIE evaluation plan.

Q Have you started your evaluation?

If Yes:

Q6a. Have you completed it?

C. HIE TOOLKIT USAGE FOR DEVELOPING EVALUATION PLAN

Thank you. Now we will discuss the HIE Evaluation Toolkit you were provided with and used in developing your HIE evaluation plan.

Make sure the respondent has the correct version of the HIE Evaluation Toolkit that was provided to them and can refer to it during this interview.

I will be asking for your candid feedback about this toolkit. All comments – both positive and negative – are welcome. Your feedback during today’s session will help AHRQ provide useful tools to help in planning HIE evaluations.

Q At what point in your process of developing the plan did you first use the toolkit?

Q What were your first impressions of the toolkit?

Q Please describe how you used the toolkit in developing the evaluation plan.

Q Did the information in the HIE toolkit cause you to change any of your original evaluation plans (Yes/No)?

If Yes:

Q10a. How so?

Q10b. In the end, were the changes you made because of the toolkit helpful to your evaluation project?

If No:

Q10c. Could you comment on why the use of the HIE toolkit did not

lead to any changes in your evaluation plan?

D. HIE TOOLKIT USAGE FOR EVALUATION PLANNING

For the next set of questions you will be asked if you used the toolkit for specific evaluation planning purposes. Please feel free to tell me more about how you used the toolkit for these purposes and provide examples.

Q Did you use the toolkit in these specific ways:

Q11a. Follow the recommended steps in developing evaluation plan?

Q11b. Set evaluation goals?

Q11c. Identify potential evaluation measures?

Q11d. Assess the feasibility of evaluation measures?

Q11e. Assess costs of evaluation measures?

Q11f. Choose final evaluation measures?

Q11g. Plan a quantitative evaluation study involving data collection?

Q11h. Plan a qualitative evaluation study such as interview, surveys, or focus groups?

Q11i. Estimate sample size?

Q11j. Develop your evaluation budget?

Q11k. Look up resources such as links and articles suggested in the toolkit for more information?

One of the appendices provides tables with examples of HIE evaluation measures, data sources, risks, and limits.

Q Are you familiar with this appendix (Yes/No)?

If Yes to Q12:

Q12a. Did you use the information in this appendix to help select and assess potential evaluation measures?

Q12b. Are there other measures you considered for inclusion in your evaluation plan that were not included in this appendix toolkit?

Ask Q12c only if Yes to Q12b:

Q12c. Can you tell me what those are?

If No to Q12:

Q12d. Can you comment on why you did not use the appendix?

Q Are there other important ways that you used the toolkit that we have not discussed?

Next, I would like to know if you used other tools in developing your evaluation plan.

Q Did you use other tools to support your work in HIE evaluation plan development?

[probe with other available tools previously identified by Westat]

If Other Tools Identified:

Q14a. Please describe them.

Q14b. How does the toolkit compare to these other tools?

Q14c. How much would you say you used these other tools relative to the toolkit?

Q14d. What sources did you use to locate those tools?

Q14e. How useful were those sources for finding relevant tools?

Q Were there other kinds of tools you were seeking but did not find?

If Yes:

Q15a. If so, what topics and type of information would they cover?

Let’s now discuss how useful you feel the toolkit is.

Q Overall, how useful was the HIE Evaluation Toolkit in developing your evaluation plan?

1. Extremely useful

2. Very useful

3. Somewhat useful

4. Slightly useful

5. Not at all useful

Q What information and sections did you find to be most helpful?

Q Least helpful?

Q What topics or measurement areas did you want your evaluation to cover? Some examples of topics are: usage, user satisfaction, workflow impact, changes in quality, and changes in efficiencies and related costs.

If yes:

Q19a. Did the toolkit help you understand how to evaluate those topics?

Q Would you use the toolkit in the future? Why or why not?

Q Would you recommend it to others conducting HIE evaluations? Why or why not?

Q What could be done to make the toolkit more relevant for you and your role within your organization?

Q Were you satisfied with the evaluation plan you developed?

If Yes:

Q23a. What components of the toolkit contributed most to the plan?

Next, I would like to talk to you about ease of use and organization of the information in the toolkit.

Q Did you find the information provided in the toolkit to be organized in a way that made sense to you?

If Yes:

Q24a. Describe what in the organization made the most sense.

If No:

Q24b. Describe how you would have liked to see the information organized.

Q Did you find the information provided in the toolkit easy to use?

If Yes:

Q25a. Describe what made the information easy to use.

If No:

Q25b. Describe how you would have to have the information presented to make it easier to use and access.

Q Did you find the information provided easy to read and understand?

If Yes:

Q26a. Describe ways in which the toolkit made the information easy to read and understand.

Q27. Describe how you would have liked presented to make it easy to read and understand.

Q28. Did you find the information in the appendix of evaluation measures easy to use and apply?

If Yes:

Q28a. Describe ways in which the toolkit made the information easy to use and apply.

If No:

Q28b. Describe how the information in the appendix could be more useful.

Q29. Did you believe that the toolkit helped you to efficiently develop your evaluation plan?

If Yes:

Q29a. Describe how the toolkit helped you be more efficient.

If No:

Q29b. Describe ways that you feel the toolkit should have helped you be more efficient, but where it did not.

Q30. Overall how satisfied were you with the toolkit?

1. Extremely satisfied

2. Very satisfied

3. Somewhat satisfied

4. Slightly satisfied

5. Not at all satisfied

Q30a. What aspects of the toolkit were you most satisfied with?

Q30b. What aspects of the toolkit were you least satisfied with?

Only ask if the respondent has started the evaluation (question 6)

You indicated that you had started your evaluation. We have a few last questions about the evaluation and how the toolkit may have impacted the evaluation.

Q31. Did you also use toolkit while conducting your evaluation?

If Yes:

Q31a. Please explain how it was used for the evaluation.

Only ask if the respondent has completed the evaluation (question 6a)

Q32. Did you believe that the toolkit helped you accomplish the evaluation?

If Yes:

Q32a. Describe how the toolkit helped you complete your evaluation.

If No:

Q32b. Describe how the toolkit could be improved to have helped you accomplish the evaluation.

Q33. Were you satisfied with your evaluation?

If Yes:

Q33a. What components of the toolkit contributed most to the satisfactory evaluation?

Thank you. We can now wrap up the interview.

Q34. Are there any issues related to HIE Evaluation Toolkit that we have not discussed that you think would be appropriate for us to know?

Thank you. As I previously mentioned, you will receive $100 as a token of our appreciation. You should receive this within the next 60 days.

Thank you for your participation today and for helping us gather the kind of information that AHRQ can use to improve the HIE Evaluation Toolkit.

Good bye.