Supporting_Statement_B_HIV_Youth PASSBACK.v2_clean

Supporting Statement B

Building Futures: Supporting Youth Living with HIV

OMB Control No. 0906-XXXX

2. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

Under the Building Futures: Supporting Youth Living with HIV project, the Health Resources and Services Administration (HRSA), HIV/AIDS Bureau (HAB) contracted DSFederal and partners (hereafter mentioned as the project team) to conduct site visits with 10 Ryan White HIV/AIDS Program (RWHAP) providers with patients with high rates of viral load suppression (specialized) and 16 RWHAP providers with patients with low rates of viral load suppression (performance improvement).

To identify these providers, the project team relied on 2014 Ryan White Services Report (RSR) data. Each RWHAP-funded provider submits an RSR client-level data file with client demographic, service, and health outcomes information on an annual basis. HRSA/HAB compiled the 2014 data and provided the project team with provider-level statistics for analysis.

HRSA/HAB aims to target providers that serve at least 20 youth. Of the 1,788 providers funded by the RWHAP in 2014, 472 served at least 20 HIV-positive youth. The project team then categorizes these providers as specialized or performance improvement based on their youth retention in care and viral load suppression rates. To ensure these rates are meaningful, the project team restricts the universe even further to exclude providers with fewer than 10 HIV-positive youth with an outpatient/ambulatory medical care (OAMC) visit before 9/1/2014 (the denominator in the retention in care measure). This restriction yields 307 providers.

Within this sample universe, the project team establishes two pools: specialized providers and performance improvement providers. Providers are categorized by their 2014 youth retention in care and viral load suppression rates. The viral load suppression rate is the number of virally suppressed (<200 copies) HIV-positive youth (ages 13-24) with least one OAMC visit divided by the total number of HIV-positive youth with at least one OAMC visit. The retention in care rate is the total number of HIV-positive youth who received an OAMC service before 9/1/2014 and another OAMC service at least 90 days later divided by the total number of HIV-positive youth who received an OAMC service before 9/1/2014.

Note that the viral load suppression measure when defining specialized providers includes clients with missing viral load data. That way, the project team ensures providers not only have high suppression rates, but also report the required data for their clients. The methodology is slightly changed for calculating viral load suppression rates for the performance improvement providers by excluding clients with no viral load data from the denominator. This means that poor performance is a true reflection of viral load suppression and not a function of poor data quality.

To select the specialized providers, the project team requires providers to have viral load suppression rates of 70% or above for their youth client population, which results in 34 providers. To select the performance improvement providers, the project team requires providers to have viral load suppression rates of less than 50% and retention in care rates of less than 60% for their youth client population. These criteria yield 19 providers. Within each pool, HRSA/HAB selects the final 10 specialized providers and 16 performance improvement providers based on geographic and client population diversity.

The below table provides a summary of site visit selection criteria, respondents, and data collection methods.

	Specialized	Performance Improvement
Number of Providers/Site Visits	10	16
Selection Criteria	Youth viral load suppression rate of 70% or more	Youth viral load suppression rate of less than 50% and youth retention in care rate of less than 60%
Type (and Number) of Respondents per Site Visit	Program Manager (1) Clinical Director (1) Program Administrative Staff (5) Clinicians Case workers Intake staff Youth (7) Total = 14	Program Manager (1) Clinical Director (1) Program Administrative Staff (5) Clinicians Case workers Intake staff Youth (12) Total = 19
Data Collection Method	Organizational Online Questionnaire Program Manager and Clinical Director Interview Program and Administrative Staff Interview Youth Focus Group Youth Interview	Organizational Online Questionnaire Program Manager and Clinical Director Interview Program and Administrative Staff Interview Youth Focus Group Youth Interview Panel/advisory board of young people living with HIV

2. Procedures for the Collection of Information

During site visits, the project team will collect qualitative data through in-person interviews with provider staff members and interviews and focus groups with HIV-positive youth. Conducting in-person interviews allows the project team’s researchers to connect with participants, respond to individual answers in real time, and ask relevant follow up questions. Importantly, focus groups allow participants to hear, address, and respond to the stories and responses of other participants. This leads to a richer, more meaningful discussion that follows the flow and direction set by multiple participants. The specific data collection methods are described below.

Both types of providers (specialized and performance improvement) will complete an online or telephone questionnaire prior to the site visit. During each site visit, project team researchers will complete an onsite observational tool while accompanied by a provider staff member. In addition, there will be interviews at each site with program managers, clinical directors, and program and administrative staff (e.g., mental health workers, intake staff). Researchers will also conduct one interview with an HIV-positive youth and one focus group with approximately six HIV-positive youth. Only at the performance improvement sites, researchers will facilitate a Youth Panel.

The project team has developed a strategy for staffing site visits to ensure that site visits meticulously capture 1) replicable evidence-based interventions and promising practices (specialized) and 2) promising strategies for adoption and recommended actions to improve outcomes (performance improvement).

Both types of site visits will include one clinical expert and one organizational assessment expert, experienced in outcome oriented, youth-focused HIV care and treatment. To the extent possible, visits to the performance improvement sites will also include one young person living with HIV (Youth Consultant).

Team Member	Site Visit Type	Proposed expertise/skill set
Clinical Expert	Specialized Performance Improvement	Clinical specialist in adolescent medicine Current or past medical director of an HIV practice serving youth Leadership position in pediatric/adolescent HIV Knowledge of complex clinic settings Knowledge of multidisciplinary teams Experience with setting values/tone in clinical site visit assessments Knowledge of RWHAP Knowledge of performance improvement/ evaluation strategies Skills in interviewing and conducting focus groups Prior site visit experience
Organizational Assessment Expert	Specialized Performance Improvement	Knowledge of organizational assessment strategies and practices Knowledge of evidence-based interventions Knowledge of RWHAP Knowledge of performance improvement/ evaluation strategies Knowledge of organizational system assessment Knowledge of implementation science Expertise in research, analysis, and writing Skills in interviewing and conducting focus groups Skills in documenting evidence-based/best/promising practices Skills in facilitating planning sessions Prior site visit experience
Youth Consultant	Performance Improvement only	Youth leader aged 20-29 Experience with receiving HIV care in RWHAP clinic Experience as peer advocate/ serving on consumer advisory board Experience as mentor/transition from youth to adult clinic

During site visits, the project team will record interviews and focus groups and take detailed notes. The recordings will serve as a mechanism to clarify and enhance notes; the project team does not plan on transcribing the recordings. To facilitate standardized data collection and analysis across the various information sources, the project team has categorized all data collection instruments according to five areas of assessment: clinic systems and infrastructure, clinical standard and models of care, provider knowledge, skills and attitudes, collaboration with youth and families, and community presences and linkages. During the qualitative interviews, the project team will use structured notetaking worksheets to document observations in the five areas, and these worksheets include columns for notes on core intervention and implementation components, as well as on the adaptable components of interventions. Information across data sources will be analyzed to determine which provider attributes contribute to positive health outcomes for youth and which attributes could be strengthened to improve performance.

The project team will summarize findings of each specialized site visit in a Specialized Site Visit Report. The report format is informed by the Active Implementation Framework (AIF)¹ and literature on replicating evidence-based practices,² and reports will document evidence-based best practices and promising strategies, models of care, environment suited for implementation, and resources needed to maximize effectiveness. A Performance Improvement Site Visit Report will summarize each performance improvement site visit, documenting evidence-based best practices and promising strategies for adoption and recommendations for actions to improve engagement, retention, and suppression for youth living with HIV.

The provider sites selected are not meant to be statistically representatives of the RWHAP provider population overall. In addition, the project team does not plan to generate quantitative statistics based on the information collected through interviews and focus groups. Therefore, the project team will not use advanced statistical analysis to assess findings.

3. Methods to Maximize Response Rates and Deal with Nonresponse

The project team intends to maximize response rate by having the HRSA/HAB Project Officers associated with providers strongly encourage participation. In addition, specialized site visit providers will be motivated to have their programs and successful approaches highlighted in technical assistance materials. Performance improvement providers will be motivated to receive in-depth and site-specific technical assistance to improve performance during the site visits. HRSA/HAB expects full participation from these providers; however, the project team has identified a back-up sample of providers in the event that a one or two declines to participate or is nonresponsive.

4. Tests of Procedures or Methods to be Undertaken

The project team piloted three instruments, the pre-site visit questionnaire, program manager interview guide, and HIV-positive youth interview guide, with the following representative providers.

• Adolescent AIDS Program, Montefiore Medical Center, Bronx, NY, Dr. Donna Futterman, Director, and Dr. Alisha Liggett, Attending Physician

• SUNY Downstate Medical Center, Brooklyn, NY, Dr. Jeffrey Birnbaum, Director

• Sidney Borum at Fenway, Boston, MA, Dr. Ralph Vetters, Medical Director

• Grady Infectious Disease Program, Pediatric Department, Atlanta, GA, Stephanie Hackett, Physician’s Assistant

To select pilot RWHAP providers and participants, we relied on professional connections and suggestions from HRSA/HAB. In addition, the project team piloted the instruments providers not already selected for the site visits.

Either in person or with a screen-sharing platform, the project team reviewed the questions with the providers and gained feedback on relevance and appropriateness of tone.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Two members of Mission Analytics Group, a DSFederal partner, conducted the analysis to select the sample:

• Coombs, Elizabeth (Ellie), MPP, 415-796-0159, [email protected]

• Davidson, Jeffrey 415-852-7137, [email protected]

A team of consultants and staff from DSFederal and its partners will collect and analyze the data. The project team staff members and consultants include:

Name	Contact Information
DSFederal Team and Consultant Site Visitors
Stephanie Bogdewic, MPH, CAI	[email protected]
Elizabeth Coombs, M.P.P. Mission Analytics	[email protected] 415-263-9885
Alice Douglas, MPH, CAI	[email protected]
Leslie Herman, MSPH, CAI	[email protected]
Julia Hidalgo, ScD., M.S.W., M.P.H., Positive Outcomes, Inc.	410-212-7554
Debbie Isenberg	617-932-1145 (H) 617-733-6140 (C) [email protected]
Warren Yiu Kee Ng , M.D.	[email protected]; 347-306-8138 (H) 212-305-7256 (W)
Casey Selwyn, MPhil CAI	[email protected] 212-594-7741 x 271
Echo Wang, MPH DSFederal, Inc.	[email protected]
Youth Consultants
Darion Banister	[email protected]
De'Andre' Gulley	[email protected]
Devin Hursey	[email protected]
Justin LiGreci	[email protected]
Shaquille "Justice" Long	[email protected]
Maxwell Monaghan	301 Ridgedale Circle Rochester NY, 14616 585-319-0891
Matthew Rodriguez	750 Third Ave., 6th Floor NYC 10017 [email protected]
Cedric Sturdevant	[email protected] 601-672-1882
Adam Thompson	[email protected] 864.354.8468 University of Buffalo, Millard Fillmore College – Certificate in Practice Facilitation (2015) Georgetown University - Bachelor of Arts – Theology/Christian Ethics (2010)
Leanna Thornton	130 Lyn Court Park Rochester NY, 14612 585-456-7032
Subject Matter Expert Consultants
Alicia Beatty	[email protected] www.circleofcarephilly.org 215-985-2657 1700 Market Street, 18th Floor, PA 19103
Jeff Birnbaum, M.D., M.P.H.	SUNY Downstate Medical Center Department of Pediatrics 450 Clarkson Avenue, MSC 1241 Brooklyn, New York 11203 718-282-1199  [email protected]
Lori DeLorenzo, MSN, RN Director Organizational Ideas	Work: 540-951-0576 [email protected]
Donna Futterman, M.D.	Montefiore Medical Center  111 East 210th Street  Bronx, NY 10467 Tel: 718.882.0232  [email protected] [email protected]
Marie Hayes	Comprehensive Diagnostic and Treatment Center (CDTC), Broward Health Fort Lauderdale FL 954- 728-1082, [email protected]
Lisa B. Hightow-Weidman, M.D., M.P.H.	101 Manning Dr Chapel Hill, NC 27514 919- 966-2536
Nancy Hutton, M.D. Johns Hopkins Univ School of Medicine	David M. Rubenstein Child Health Building Room 2019 200 N. Wolfe Street Balitmore, MD 21287 Harriet Lane Compassionate Care [email protected] 410.614.4750
Rebecca M. Johnson, ChangeLab Solutions, California State University, East Bay	ChangeLab Solutions Oakland CA 94612 510-302-3380
Sarah Levine Henry J. Kaiser Family Foundation, Washington, DC	202-347-5270
Alisha Liggett, MD Montefiore Medical Center	[email protected]
Meribeth Meixner Reed, R.N., Ph.D., Thomas Edison State College, NJ	[email protected] 61 North Bay Ln. #4, Port Ludlow WA 98365 360-437-7846
Jeffrey A. Menzer, R.N., ACRN	120 - E Street, S.E. Washington, DC 20003-2613 202-321-7563 [email protected]
Nancy P. Miertchen	Thomas Street Health Center Harris Health System Office:  713/873-4185 Cell:  713/817-3524 FAX:  713/440-1269 [email protected]
Linda Neuhauser, Clinical Professor of Community Health and Human Development at UC Berkeley	PHONE: (510) 643-9177 FAX: (510) 642-9792 50 University Hall MC 7360, University of California [email protected]
Angulique Outlaw, Ph.D.	Wayne State University 2100 Undergraduate | Library 5155 Gullen Mall | Detroit, Michigan 48202 Tel (313) 577-3030
Ana Maria Puga, M.D., F.A.A.P.	[email protected] 954-728-8080 ext. 1017
Susan Rubinstein, Independent Consultant	217 Ivy Rock Ln., Haverton PA 19083 Phone 610-609-6367 [email protected]
Karen L. Schlanger, PH.D.	560 Hillcrest Avenue, Athens, GA 30606 (706) 850‐0610 [email protected]
Vicki Tepper, Ph.D.	410-706-8931 410-706-7553 737 W. Lombard Street, 244 Baltimore, MD [email protected]
Jacob E. Tenenbaum, D.P.A., M.P.H.	301 649-7665 [email protected]
Ivy Turnbill, PhD, Deputy Executive Director, AIDS Alliance	The AIDS Institute, DC 1705 Desales St NW #700, Washington, DC 20036 Phone:(202) 835-8373 [email protected]
Mildred Williamson, Ph.D., MSW Lecturer, University of Illinois, Chicago School of Public Health	University of Illinois Chicago School of Public Health [email protected]

1 Damschroder et al., “Fostering Implementation of Health Services Research Findings into Practice.”

2 Metz, Bowie, and Blasé, “Seven Activities for Enhancing the Replicability of Evidence-Based Practices.”

6