Awardee interviews for the evaluation of the Advance Interoperable Health Information Technology to Support Health Information Exchange program

Ahie: awardee interviews MATHEMATICA POLICY RESEARCH

Form Approved

OMB No. 0955-0003

Exp. Date 12/31/2017

Internal note: Customize each interview guide to incorporate key questions from the 1-1 project meetings. Since these are qualitative in-depth interviews, questions may not be asked in the order presented in the guide in order to preserve the conversational flow of the interview.

1. Introduction (5 minutes¹)

Thanks for taking the time to talk with us today. On the phone, we have [name research staff from Mathematica-NQF and identify which organization]. We are interested hearing from you about [program name]. We have scheduled this interview to take 90 minutes. Representatives of ONC will also be on the call with us today. [Allow ONC representatives to briefly identify themselves.] We want to hear the story about your program in your own words. Due to the number of topics we want to cover in this limited amount of time, we will need to move through the questions at a fairly rapid pace, and we ask that only one or two people take the lead in responding to each question. Different people can take the lead for different questions depending on who has more experience or knowledge relevant to a given question.

Before we begin our discussion, we want to check with you to see if we can record this interview for note taking purposes so that we can describe your program more accurately in our evaluation report. Your name will not be associated with any of your responses in the report. However, the names of organizations may be used in the report as we summarize your program, the progress you have made, and the facilitators and challenges you have faced. The recording will be destroyed after the evaluation is complete.

Before we get started, do you have any questions for us?

2. Program overview (15 minutes)

In preparation for this interview, we had an in-depth discussion with your ONC Project Officer, and we reviewed a number of your program documents, including quarterly reports, program monitoring plan, your program application, etc. So we already have reviewed a fair amount of basic information about your program that we would like to expand upon. With that in mind, please tell me about what you hoped to accomplish with your program.

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-0003. The time required to complete this information collection is estimated to average 90 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, to review and complete the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Health & Human Services, OS/OCIO/PRA, 200 Independence Ave., S.W., Suite 336-E, Washington D.C. 20201, Attention: PRA Reports Clearance Officer

Probe on the following topics as they come up in the response to the question above: Overall goal of the project; Implementation approach; Motivation for participation; Target population (how was this selected)

3. Identifying, marketing, recruiting, and enrolling eligible professionals (EPs) and non-eligible professionals (non-EPs) (5 minutes)

We understand that your target population included [list awardee-specific target population]. How did you identify, recruit, and enroll participants from the target population into your program?

Probe with the following questions if they are not addressed in the awardee representatives’ initial response and if time allows.

1. How did you market the program to your target populations, and how did you recruit potential participants?

   1. What marketing strategies/activities did you find most effective?

2. In your opinion, what factors went into a person’s decision to participate in the program? (e.g., built upon work previously done in the organization)

3. What were the challenges involved in identifying, recruiting, and enrolling participants?

4. Quantitative measures (5 - 10 minutes)

Describe your approach to measuring progress toward your programmatic goals. What milestones did you set? Did your milestones change over the course of the program? If so, why did they change?

Probe with the following questions if they are not addressed in the awardee representatives’ initial response and if time allows.

1. Can you describe the process you went through to select your initial metrics for tracking the progress of your milestone achievements?

2. How did you make the projections for your metrics? (e.g., based on data from a similar project)

3. In what ways did your health IT landscape or target populations change over the period of performance?

   1. How if at all did you change your metrics to reflect changes in the health IT landscape/target populations

4. Do you feel that your final metrics accurately represent your program progress?

   1. If not, what would you change?

5. Implementation challenges and strategies used to address them (15 minutes)

Describe some of the successes you achieved and challenges you encountered over the course of your program. What do you think went well? What barriers did you face and how did you overcome them?

Probe with the following questions if they are not addressed in the awardee representatives’ initial response and if time allows.

1. What factors affected your ability to achieve the goals you identified for your program? For example, these factors might include characteristics of the local environment, the culture of the implementing sites, and your program’s design. The factors might also include individuals in leadership roles and program processes.

[For each factor that the interviewee identifies:]

1. How did [factor] help you in your program implementation?

2. How did [factor] create challenges for you in your program implementation?

3. Did you overcome the challenges?

   1. If so, how?

1. Overall, were some challenges more difficult to overcome than anticipated?

2. What strategies were effective in training and educating EPs and non-EPs to increase adoption and utilization of HIE services? What strategies were not effective?

3. What strategies were effective for workflow redesign support? What strategies were not effective?

4. What strategies were effective for engaging with vendors? What strategies were not effective?

5. What lessons did you learn from the implementation process?

6. Were any unexpected costs incurred (e.g. for hiring staff, etc.) beyond the required 3:1 matching ratio for the award?

6. ONC-provided tools (5 -10 minutes)

How did ONC support your program? What ONC activities did you find most helpful? What could ONC have done differently?

Probe with the following questions if they are not addressed in the awardee representatives’ initial response and if time allows.

1. We understand that ONC offered you guidance in the form of meetings and tools to help you implement your program. These meetings and tools included Community of Practices (CoPs), technical assistance resources, site visits, in-person workshops, kick-off meetings, and annual meetings. Did you use any of these resources?

2. Did you find the resources helpful to you in accomplishing your programmatic objectives? If so, how?

3. Where some resources more useful to you than others? (probe: ask for elaboration as necessary)

   1. Did you find the CoP in particular to be helpful? If so how?

4. Do you have any ideas on how some resources could have been improved? (probe: ask for elaboration as necessary)

5. Is there any other type of resource that would have been useful as you implemented your program? If so, what is this resource? Why do you think it would have been helpful?

6. Could ONC have done anything differently to better support your program?

7. Sustainability (5 minutes)

What impact did this program have on the overall sustainability of health information exchange efforts in your state? To what extent will you be able to continue program activities in the future?

Probe with the following questions if they are not addressed in the awardee representatives’ initial response and if time allows.

1. In what ways do you plan to sustain the activities conducted as part of this cooperative agreement after the program has ended?

   1. If so, do you anticipate any issues related to continuing these activities?

      1. If so, what are these issues?

      2. Do you have a plan for overcoming these issues? If so, what is it?

   2. If not, why do you not intend to continue these activities?

2. What is the viability of the HIE in your state? Do you consider your organization and the work you do to be sustainable?

3. Do you think this program contributed to the sustainability or viability of your state HIE?

8. Replicability (5 minutes)

What lessons did you learn from this program? What advice would you give to another state/HIE trying to increase adoption and use of HIE services?

Probe with the following questions if they are not addressed in the awardee representatives’ initial response and if time allows.

1. Do you plan to share your work under the [program name] with other organizations? If so, how many?

   1. How are you disseminating this information? (e.g., via learning collaboratives, webinars, conferences)

2. Do you foresee any challenges involved in sharing information on the program with others?

   1. If so, do you have a plan to overcome these challenges? Can you briefly describe this plan?

9. Program alignment with federal, state, and local efforts (10 minutes)

How did you work with other federal, state, and local stakeholders to advance health information exchange in your state?

Probe with the following questions if they are not addressed in the awardee representatives’ initial response and if time allows.

1. What strategic partnerships did you form through your program to drive statewide or regional alignment of existing health IT initiatives? Were there previous existing partnerships that had an impact on your program?

2. How did these efforts impact your program or contribute toward achievement of your programmatic objectives?

3. Do you see anything that will prevent more extensive utilization of health information exchange services in your state?

4. Did you partner with any other AHIE awardees? If so, were there specific initiatives or efforts that you and other awardees focused on?

5. Are there any partnerships that developed during the program period that will continue post-award?

10. ADT (only for Delaware, Oregon, Rhode Island, and Utah) (15 minutes)

Now we would like to focus specifically on your work as one of the ADT CoP awardees. In your own words, tell us about your program. How did you leverage or build upon your existing work in the Advance Interoperable HIE Program? What were you trying to accomplish? Who did you work with, and what did you do?

Probe with the following questions if they are not addressed in the awardee representatives’ initial response and if time allows.

1. What are your inter-state program goals and objectives?

2. What are your intra-state program goals and objectives?

3. Describe your target audience and why they were selected. What was your overall approach (including partnerships) to using this funding to facilitate widespread adoption and routing of ADT messages?

   1. Have there been any changes regarding your planned connections?

4. How did you engage stakeholders and partners?

5. Which marketing, outreach, enrollment, processes did you implement? What were some of the successes and challenges of these approaches?

6. Were there any unanticipated challenges and outcomes?

7. What were the environmental, political, local, state, economic, or market influences?

8. What strategies were most effective in training and educating new users to increase adoption and widespread routing of ADT messages?

9. What strategies or approaches did you use to establish or expand your provider directory (if applicable)? What challenges did you encounter during this process?

10. What strategies or partnerships were most effective for facilitating cross-state exchange?

11. What strategies or approach did you use to determine required content for ADT messages?

    1. What challenges did you encounter during this process?

12. What strategies or approaches did you use to develop trust agreements between participating entities?

    1. What challenges did you encounter during this process?”

13. What barriers prevent more extensive adoption and widespread routing of ADT messages?

14. Regarding the community of practice, did you participate in any of the meetings and utilize any of the information developed or discussed there?

    1. If so, how often? How could they have been improved?

15. In what ways did the ADT CoP, the ADT in-person workshop, and the topic briefs help you accomplish your programmatic objectives?

16. What could ONC have done better to support your program?

17. Did any specific deliverables/webinars/meetings/CoPs or other events stand out to you as particularly helpful/informative?

18. What lessons have you learned through the ADT CoP work?

19. How does this program contribute to the sustainability/viability of the HIE?

11. Wrap up (5 minutes)

1. Looking back over the two years of the program, is there anything you would have done differently?

2. Is there anything that we haven’t asked about that you think is important for us to know with respect to the evaluation of either the [program name] or ADT?

1 Guidance on time given in parentheses for each section are suggestions. Interviewers will modify the amount of time spent on particular questions depending on the flow of the conversation and the need to probe for particular details.

6