2 Virtual site visit interviews

Appendix C

RUDI Provider and Subrecipient Discussion Guide

RUDI Provider and Subrecipient Discussion Guide

Before interviewing, gather following information from the provider’s RUDI survey data and initial introductory/scheduling call with the point of contact:

Information	Source
Respondent information
Interviewee title and description of role in organization:	Introductory call
Role in intervention implementation:	Introductory call
Name of subrecipient organization	Sample selection document – Use this to fill “NAME OF ORGANIZATION” fills
Name of recipient for this subrecipient	RSR data – fill NAME OF RECIPIENT below
Provider information
Sampled Group 1 (used HAB resources), Group 2 (used other resources not HAB), or Group 3 (accessed but did not use HAB resources)	Sample selection document – Use to select a streamlined version of this guide tailored to the group
Provider type:	Linked survey and RSR data
Size:	Linked survey and RSR data
Geographic location:	Linked survey and RSR data
Intervention implemented and link to page on TargetHIV website:	Survey Q8
Resources used:	Survey Q2, Q19 a, b, c, d
Intervention description (intervention type, care continuum outcome, population):	Survey Q5 a, b, c
[Group 1 only:] How accessed intervention information:	Survey Q9-10
[Group 3 only:] Reasons for not using HAB resources that they accessed	Survey, Q6a – e1
[Groups 2 and 3:] Sources of non-HAB resources used	Survey, Group 2: 6e1 Group 3: 4e1
Other
Name of AETC nearest the selected organization	Welcome | AIDS Education and Training Centers National Coordinating Resource Center (AETC NCRC) (aidsetc.org) – fill NAME OF AETC below

Note: We will not speak the italicized material below; rather, we include it as an aid to the interviewer during the interview. Bracketed words that are not italicized indicate where we intend to tailor the guide based on the information collected before the interview and summarized in the box above.

Introduction¹

Thank you for taking the time to speak with us today. My name is [NAME] and I am joined on the call today by my colleague from Mathematica [NAME]. As you know from the invitation we sent, we are working with the HIV/AIDS Bureau (HAB) at HRSA on a study to learn about how Ryan White recipients and subrecipients access and use the intervention resources and materials that HAB disseminates to improve services for people with HIV. We are calling the efforts you take to improve care and services for people with HIV “interventions.” You could use the term “program” or “strategies” as well to describe these efforts.

We would like to follow up on your organization’s survey responses to talk in more depth about how [PROVIDER ORGANIZATION] accesses and/or uses intervention resources and materials that HAB and other organizations make available, feedback you have, and potential opportunities to improve the usefulness of those resources. This will help HAB maximize the impact of the resources they make available to help end the HIV epidemic in the U.S. We will summarize what we learn from you and others we interview for this study in a report to HAB.

Please note, we are not evaluating your organization, this is not a HAB site visit, and there are no wrong answers. Nothing today will affect your participation in the Ryan White HIV/AIDS Program. Your participation is totally voluntary. If you’d like to skip a question or stop the interview at any point, that is okay. We will pull together information from interviews with multiple organizations to share overall themes and illustrative examples in a final report. We won’t share any identifying information about you or your organization, but our final report may include anonymized examples from our discussion. Also, the information you provide will only be used for the RUDI Evaluation. Do you consent to participate in the study?

[Interviewer: Record respondent’s verbal response: Yes/No. If NO, we CANNOT conduct the interview. Thank the respondent for their time and end the call.]

If OK, we would like to record the audio of our discussion to ensure accurate notes. We will not share the recording with anyone outside of the Mathematica research team and the transcriptionist. Nor will we share the transcript outside the Mathematica research team. We will destroy the transcript and recording from this interview at the end of our study. Is it okay with you if we record the interview?

[Interviewer: Record respondent’s verbal response: Yes/No. If NO, we can conduct the interview and take notes but not record it.]

1. Background

Let’s start with some background.

1. Could you please briefly describe your role and responsibilities at [NAME OF ORGANIZATION] and how long you worked there?

   1. Which Ryan White funding stream do you work under, if you know that? [e.g., Part A, B, C, D, F?]

2. We know some background characteristics of your organization from the information you provide to HAB, but one additional characteristic we’d like to know is to what extent is your organization the type that likes to innovate or try new strategies to improve HIV services early, versus wait until most other organizations have something in place before adopting it?

2. How providers access resources

Now let’s get into how you personally access resources to help support improvement of care and service for people living with HIV. By resources, I mean supporting information and tools – anything from information on interventions you read on the TargetHIV or other HAB websites, to webinars you might view online, to things you may download, including implementation manuals and tools or toolkits.

1. Where do you tend to access resources to help you improve HIV care and services?

1. Why do you go there?

2. How often do you access resources there?

3. Do you typically go looking for a resource or are you pointed there through links?

   1. [If links:] What are the most common sources of the links you use, such as webinar slides, emails, listservs, documents?

4. [If subrecipient] Does your RWHAP recipient, that is, [NAME OF RECIPIENT] suggest you link to and use resources? How?

5. [If recipient] Does your HAB project officer suggest you link to and use resources? How?

2. [If not previously mentioned] Are you familiar with the intervention resources available on the TargetHIV website? By intervention resources, I mean resources that would help you implement a deliberate change strategy to improve the delivery of care and treatment and health outcomes for people with HIV.

1. How often you do you access intervention resources on TargetHIV?

2. [Groups 1 and 2:] Why do you use TargetHIV as a source of information for HIV service improvements, and which types of resources do you access? [Note: Examples are innovative ideas for improving HIV care, for advice and tools to support a goal for improving HIV services.]

3. Are you familiar with the intervention resources disseminated through the AIDS Education and Training Centers—either through the national AETC website, or through your regional AETC staff or website (for you that would be [NAME OF AETC])? [Note: Make sure to understand which source(s) they use. Probe as needed if they use them:]

1. Are you familiar with the National Coordinating Resource Center and the National Clinician Consultation Center, which are also national AETCs?

2. How often you do you access the resources available on the AETC website(s)?

3. [If they use AETC resources:] Can you explain why you use the AETC as a source of information for service improvement, and what types of resources you access?

[Note: Both the TargetHIV website and the AETC staff and websites help to disseminate HAB resources. Please listen carefully if they reference a specific website in moving through the sections below.]

3. Provider feedback on the usefulness of intervention resources

Now I’d like to ask about your organization’s use of resources to support your efforts to improve services for people with HIV.

1. [Groups 1 and 2:] In the online RUDI survey that you or one of your colleagues completed, you mentioned that your organization has used resources in the last 2 years to help implement an intervention to improve HIV service:

[Group 1]: [NAME OF INTERVENTION]: Does that sound right to you?

[Group 2:] Does that sound right to you, and if so, how would you briefly describe the intervention? [Note: Resources could be a guidance manual, toolkits, staff training videos, webpages on TargetHIV that provide intervention information; see provider responses to Q8 in survey]

1. [Groups 1 and 2:] Other than the one just mentioned, what if any other interventions have you implemented recently to improve service for people with HIV? [Note: If organization is a SPNS grantee, ask about an intervention that was not funded by SPNS.]

1. [If more than one intervention] Which intervention should we discuss first?[Or steer this to a different intervention if it is the second interview for the virtual site visit and makes sense to do so.]

2. I’d like to get some more information about the interventions:

   1. Why did you choose this intervention and what did it aim to improve? [Note: Probe as necessary (e.g., about focus on specific population) to classify the intervention; see Q9a, Q9b, Q9c survey response for classifications.]

   2. What core elements or activities did this entail?

      1. Probe if not clear: Did you need to change staffing patterns to implement it?

   3. Who were the primary staff involved and their roles?

[Group 2 skip to Q10; Group 3 skip to Q13, Group 1 continue]

4. What HAB resources, materials, or products did [name of organization] use to implement [name of intervention]? [Note: If they listed a resource page on their survey, but don’t mention it, probe to help ensure complete response]

1. How did you hear about [name of resources]? [Note: For example, through a prompt from an external source such as an email or webinar, a peer recommendation.]

2. How did you access [name of resources]?

• How often do you access HAB resources through this source?

5. Did you need to tailor or modify [name of HAB resources] to fit your local needs?

   1. [If they made changes:] What changes did you make and why? [Probe: content, process]

   2. [If they made changes:] What was your experience with modifying the intervention materials? Was it easy or difficult? Why? Did you test to see if the modified version worked well, and if so, how and what did you learn?

6. What was helpful about the [name of HAB resources] that [NAME OF ORGANIZATION] used to implement [intervention name]? [Note: Listen to open answer and probe as needed on resources’ impact on areas below:]

   1. Selecting intervention or its components

   2. Staffing or operationalizing intervention (e.g., implementation plan)

   3. Conducting outreach to population

   4. Choosing content to interact with population

   5. Improving care delivery

   6. Identifying measures of success

   7. Other

7. What was not helpful about [name of HAB resources] that you used to implement [NAME OF INTERVENTION]?

8. [If they characterized their use of resources as overall at least somewhat successful:] You mentioned these resources were at least somewhat helpful to your effort to improve HIV care, was there anything about this provider organization that helped you do that, that other provider organizations might not have? For instance, some organizations might not have resources or time to think about how to improve care or might not have the capacity to look for resources or use any new ideas, or might not have the relationships with other organizations or professionals where people share ideas: what made your organization able to do it?

9. Was there anything that you wanted to do to improve care, a need that you’ve seen, that you weren’t able to for some reason – anything that held you back from searching for or using new resources to improve care?

[Circle back and repeat Qs 3-8 for another intervention, if time allows]

10. [Groups 1 and 2]: What resources, if any, have you used from sources other than HAB to support your work to improve care for people living with HIV? [Note: Non-HAB resources might include NASTAD, Stop the Virus, and state or local health departments.]

    1. How did you hear about and access the resources? [Note: For example, through a prompt from an external source such as an email or webinar, a peer recommendation.]

    2. What led you to seek out those resources? For example, what were you trying to achieve?

11. What about the non-HAB resources you used to implement [NAME OF INTERVENTION] was helpful?

    1. How did using [name the non-HAB resources] help you to implement and sustain the intervention? [Note: Probe as needed on resources’ impact on:]

       1. Selecting intervention or its components

       2. Staffing or operationalizing intervention (e.g., implementation plan)

       3. Conducting outreach to population

       4. Choosing content to interact with population

       5. Improving care delivery

       6. Identifying measures of success

       7. Other

12. [If both HAB and non-HAB resources were used:] How did HAB resources compare to non-HAB resources you used to implement [NAME OF INTERVENTION]?

13. [Group 2 only:] [If they characterized their use of resources as overall at least somewhat successful:] You mentioned these resources were at least somewhat helpful to your efforts to improve HIV care, was there anything about this provider organization that helped you do that, that other provider organizations might not have? For instance, some organizations might not have resources or time to think about how to improve care or might not have the capacity to look for resources or use any new ideas, or might not have the relationships with other organizations or professionals where people share ideas: what made your organization able to do it?

14. [Group 2 only:] Was there anything that you wanted to do to improve care, a need that you’ve seen, that you weren’t able to pursue new resources to help with for some reason – anything that held you back from searching for or using new resources to improve care?

15. [Group 3:] In the online RUDI survey that you or one of your colleagues completed, you mentioned that your organization accessed resources from TargetHIV but did not use them to help implement improvement to HIV services. Why did you choose not to use them? [Have the survey responses for Q6a-e handy, use them to probe as needed but do not try to validate or hold them accountable for the survey response.]

    1. Is there anyone else here at [NAME OF ORGANIZATION] who you think was more likely than you to use TargetHIV resources to help improve care for people living with HIV?

       1. [If yes:] might we have their contact information to potentially follow up with them?

4. Implementation and intervention outcomes [Groups 1 and 2 only]

1. How successful, overall, was [name of organization’s] implementation of the intervention?

1. How do you think about success - for example, measures of implementation success and outcomes did you use?

1. Did you happen to set any benchmarks for these measures in advance? If yes, what did you choose?

2. Did you do anything specific to evaluate the quality of the activities you implemented or look at how closely you were able to implement the original design of [name intervention]? If yes, what did you do?

3. [If the intervention was implemented across multiple sites:] How did the metrics or their performance vary across sites?

2. Can you share any data on measures that indicate how implementation has gone? For example, trend lines showing improvements.

2. Can you share any measures that indicate the intervention’s effect on the outcomes of the HIV care continuum, such as measures for linkage to care, receipt of medical care, retention in care, or viral suppression? For example, trend lines showing measure improvements.

3. What effect did implementing the intervention have on the organization or your staff’s capacity? [Note: For example, increase or decrease in staff capacity, clearer understanding of staff roles, etc.]

5. Opportunities to strengthen impact of HAB resources

1. Were there any resources or products you didn’t have that would have helped you implement the intervention? Please describe.

2. [Groups 1 and 3:] What would have made the HAB resources you accessed more useful or accessible?

3. How could HAB improve the way it disseminates its intervention resources? [Probe on:]

1. Ways you’d like to see them publicized for greater awareness?

2. Product types or characteristics?

3. Content?

4. Population focus?

5. Types of assistance the resources offer (for example, in person or on the phone)?

4. [Groups 1 and 2:] Are there any lessons we haven’t yet discussed that you learned from your experience implementing [intervention name] at your practice?

   1. If you were to do it again, what would you change about what resources you used or what you did?

5. Are there any resources that HAB could disseminate that would help others build on what you learned from your implementation and not have to learn hard lessons from their own experience?

6. Is there anything else you would like to share about your experience accessing and using disseminated resources by HAB?

Thank you for your time! [Stop recording]

1 This discussion guide is a “master” discussion guide. As such, this introduction as well as the questions that follow may be asked of multiple respondents within the organization selected for a virtual site visit. We will create an organization-specific interview strategy of who will be interviewed (1-4 individuals) collaboratively with the point of contact for the organization, on a 10-minute introductory scheduling call that considers who would be important to interview. Prior to each interview the interviewee will have received information about the study and an outline of topics to be covered on the interview.