Supporting Statement Part B AHRQ Clean

The data collection planned under this project is part of a comprehensive evaluation strategy to assess the adoption of the Comprehensive Unit-Based Safety Program (CUSP) for antibiotic stewardship (AS) in acute care, long-term care and ambulatory care settings; measure the effectiveness of the interventions in the participating facilities or units; and evaluate the characteristics of teams that are associated with successful implementation and improvements in outcomes. A key component of the AHRQ Safety Program for Improving Antibiotic Use is the recruitment of facilities within sites and delivery systems that will be supportive of and likely to spread the CUSP AS model.

1. Respondent Universe and Sampling Methods

This data collection request covers the activities with the Cohort 1 pilot facilities, Cohort 2 acute care hospitals, Cohort 3 long-term care facilities, and Cohort 4 ambulatory care facilities.

Recruitment Methods:

Cohort 1 (Pilot Facilities)

Facility recruitment for the Cohort 1 pilot period has focused on identifying Integrated Delivery Systems (IDSs) that have indicated a willingness to participate in the project and which meet the following criteria:

Each system should consist of regional clusters of coordinated facilities, each containing at least ten (10) facilities that will participate in the project, with at least two (2) facilities from each of the settings participating from each system: acute care hospitals, long-term care facilities, and ambulatory settings.
The recruited integrated systems should represent some geographic diversity, covering at least two (2) HHS regions.

The AHRQ Safety Program for Improving Antibiotic Use has recruited three Integrated Delivery Systems (IDSs) for the pilot period:

Geisinger Health System,
Johns Hopkins Health System (JHHS), and
Carolinas HealthCare System.

All participating IDSs meet the criteria and will commit at least 10 facilities to participate in the project, with at least two facilities from each of the three settings of interest (acute care, long-term care, and ambulatory care) At least one unit from each of the facilities will participate in the project. Exhibit B.1 below outlines some basic information about each IDS.

The 30 sites (i.e. participating units) within the three IDSs participating in the Cohort 1 pilot period of the AHRQ Safety Program for Improving Antibiotic Use include diverse locations with integrated delivery systems and vary based on type of facility, geographic location (both Northern and Southern populations), urbanity, number of beds, number of providers, type of Electronic Health Record (EHR) system, and teaching status. While the three IDSs have been selected to participate in the pilot to meet the needs stated above, AHRQ is open to considering alternative pilot facilities as recommended by OMB.

Exhibit B.1: Integrated Delivery Systems Participating in the Pilot Period

IDSs

HHS Region

Urbanity

Sites

Existing Antibiotic Stewardship Program

Number of Sites Participating From Each Setting

LTC

Amb

Acute

Carolinas HealthCare System

Primarily Urban

Over 900 care locations, including 40 hospitals, 24 urgent care centers, 5 LTC, and 600 ambulatory care sites

TBD

Geisinger Health System

Primarily Rural

14 hospitals, 7 LTC, and 40 ambulatory care sites

TBD

Johns Hopkins Health System

Urban

6 academic and community hospitals, 4 surgery centers, 10 LTC (Lorien Health Systems), 39 primary and specialty care sites

Cohort 2 (Acute Care Hospitals), Cohort 3 (Long-Term Care Facilities) and Cohort 4 (Ambulatory Care Facilities)

Targets. For Cohorts 2, 3, and 4, the project will recruit 250-500 acute care hospitals (at least 25 from each of the 10 HHS Geographic Regions); 250-500 long-term care facilities (at least 25 from each HHS region); and 250-500 ambulatory care facilities (at least 25 from each HHS region).

Facility Characteristics. To meet the recruitment targets the project will cast a wide net and ensure that eligibility criteria garners broader participation rather than unnecessarily excluding potential sites. In addition to ensuring coverage of all 10 HHS regions, the project will seek to ensure diversity of recruited facilities. The characteristics by setting are highlighted below:

Hospitals. In Cohort 2, all hospitals will be eligible for participation including federal, short term, and long term acute care hospitals. Other characteristics include urban and rural, academic and non-academic hospitals, small, medium and large hospitals, independent and part of larger health systems.
Long Term Care. For Cohort 3, the project will recruit LTC facilities that are both urban and rural, and vary by facility size, ownership (for-profit, not-for-profit, and government-owned facilities, and chain and independent facilities).
Ambulatory Care. For Cohort 4, the project will recruit ambulatory care facilities that are both urban and rural, small, medium and large practices, and small, independent and hospital owned.

Recruitment Strategy. The project will use a multi-pronged strategy for recruitment. Through a national roster of partners (e.g., patient safety foundations, Hospital Information Networks, QIN-QIOs, other quality-focused organizations, national health associations), the project will ensure coverage of all 10 HHS regions. Exhibit B.2 below details the organizations supporting the recruitment efforts, their respective audiences, target delivery sites, and HHS regions covered for this project.

Exhibit B.2: Cohorts 2, 3, and 4 Recruitment Partners

Organizations	Dissemination Vehicles/Venues	Delivery Site	HHS Regions Covered
American Academy of Family Physicians (AAFP)	National Conference, CME Events, Chapter Events, AAFP News	Ambulatory	1-10
Infectious Diseases Society of America	ISDA News, ID Week (Oct 26-30, 2016)	Hospital, LTC, Ambulatory	1-10
Joint Commission on Accreditation of Healthcare Organizations (JCT)	CJCP Education (education modules), Conferences & Seminars, Webinars, eCourses, JCR Quality and Safety Network	Hospital,	1-10
Medical Group Management Association (MGMA)	Annual Conference, Webinars, Online Courses, Educational Conferences	Ambulatory	1-10
National Patient Safety Foundation (NPSF)	NPSF Annual Patient Safety Congress, NPSF Lucian Leape , Patient Safety Week, Patient Safety Blog, NPSF News, Webcasts, PS Blog and President's Corner,	Hospital, LTC, Ambulatory	1-10
SHEA	SHEA Spring Conference, ID Week, SHEA News, SHEA-Medscape	Hospital, LTC, Ambulatory	1- 10
Society for Infectious Diseases Pharmacists	Annual Meeting	Hospital, LTC, Ambulatory	1-10
American Medical Association (AMA)	JAMA, National Advocacy Conference, AMA Education Center, AMA Annual Meeting	Hospital, LTC, Ambulatory	1-10
Vizient, Inc. (VHA-UHC)	Vizient Clinical Connections Summit, Vizient Webinars	Hospital, Ambulatory	1-10
Pediatric Infectious Diseases Society	International Pediatric Antimicrobial Stewardship Conference, ID Week, Hot Topics in Infection & Immunity in Children, St. Jude/PIDS Transplant ID Symposium & Infectious Disease Research Conference	Hospital, Ambulatory	1-10
Genesis HealthCare	Rehab.com, GRS News	LTC	1-10
Wisconsin Hospital Association	Physician Leadership Conference, Rural Health Conference, Seminars, Webinars (at least monthly),	Hospital	5
Michigan Hospital Association Keystone Center	MHA Patient Safety & Quality Symposium, Corporate Sponsorship Program, MHA Major Meetings, Healthcare Leadership Academy	Hospital	5
Premier, Inc.	Health Care Summit, Premier Annual Breakthroughs Conference and Exhibition, HFMA ANI Conference	Hospital	1-10

Additional recruitment activities will include the following:

Work with the Technical Expert Panel (TEP) members from delivery organizations to gauge their interest in participating in this program.
Leverage the AHRQ patient safety listserv of over 55,000 contacts, the Armstrong Institute (AI) listserv of 8,000 contacts that have participated in prior patient safety interventions with AI/JHU, the National Patient Safety Foundation listserv, and the Infectious Disease Society of America listserv.
Use press releases and early publications from the Cohort 1 pilot to generate national interest in the AHRQ Safety Program for Improving Antibiotic Use. The project will develop press releases on the program to disseminate in relevant email mailing lists and websites.
Use free social media to recruit acute care hospitals, including periodically posting on the Johns Hopkins School of Medicine’s (JHM) Facebook (522k likes), Twitter (430k followers), and LinkedIn (18k followers).
Host informational webinars to garner interest in the project.
Develop a website for the AHRQ Safety Program for Improving Antibiotic Use that will assist with recruitment.
Collaborate with federal partners including CDC and CMS to ensure synergistic efforts across related projects.

Respondent Selection and Sample Sizes

Cohort 1 will perform as a pilot project to test the antibiotic stewardship course design and intervention strategies. For the unit-level measures, including Structural Assessment, Team Antibiotic Review Form, and EHR data, the project will sample all the participating units. For the individual-level measure for the AHRQ Surveys on Patient Safety Culture, the project will sample all the eligible staff within each participating unit.

For the semi-structured qualitative interviews with the Cohort 1 IDS staff, the project will aim to recruit respondents from two acute care facilities, two long-term care facilities and two ambulatory care sites in each of the three IDSs for a total of 18 sites. At each institution, the project aims to sample 10 frontline health care workers and five Stewardship Leaders/Master Trainers, for a total of 270 respondents.

For Cohort 2 (acute care), Cohort 3 (long-term care), and Cohort 4 (ambulatory care), the AHRQ Safety Program for Improving Antibiotic Use will collect data from all participating units when possible.

Each facility lead will conduct the structural assessment at pre- and post- intervention in order to track changes in facility infrastructure and policy related to antibiotic use (i.e. per the survey design, the project will collect site-level data twice).
For the unit-level measures, including the Team Antibiotic Review Form and EHR data, the project will sample all the participating units.
For the individual-level measure for the AHRQ Surveys on Patient Safety Culture, the project will sample all the eligible staff within each participating unit as instructed by the AHRQ survey user guide; because unit size varies, the project estimates the average number of respondents to be 25 for each unit.

Cohorts 2, 3 and 4 will include respondents at each participating unit as detailed in Exhibit B.3 below.

Exhibit B.3: Cohort 2, 3, and 4: Respondent Universe

Setting	Respondent Universe	Additional Information
Acute care setting	There will be 3 levels of participants: 1. Antibiotic Stewardship Team 2. Unit champions (e.g., physician, pharmacist, and nurse lead) 3. Frontline staff
LTC setting	1. Antibiotic Stewardship Team 2. Team leaders 3. Frontline staff	For LTC participating facilities without an existing stewardship team, team leaders will be trained to understand the principles of antibiotic stewardship.
Ambulatory care setting	The project will ask each ambulatory care participating clinic to identify a “master trainer*.”	*A physician, nurse practitioner, and/or physician assistant. Intended for primary care providers or urgent care clinics All healthcare workers at each participating clinic will be encouraged to participate.

Response Rates

We anticipate a 90% cooperation rate from the participating facilities to the Structural Assessments, submission of EHR data (e.g., antibiotic use, C. difficile rates, urine cultures, etc.), response to qualitative interviews, and completion of Team Antibiotic Review forms.

Completion rates for the AHRQ Surveys on Patient Safety Culture, based on previous CUSP interventions focusing on VAP and CLABSI), are anticipated to range between 30-50%. For those interventions, the hospital or unit response rates for the AHRQ Patient Safety surveys varied from baseline to follow-up and from project to project. The response rates averaged 55% at baseline and between 16-28% at follow-up.

2. Information Collection Procedures

Primary Data Collection

The project will collect a range of data to contribute substantively to the evaluation and facilitate analysis of progress over time. Primary data collection includes surveys, assessment tools, and qualitative discussions (see Exhibit B.4 below). The project will analyze EHR data submitted from individual sites as secondary data sources.

Exhibit B.4: Summary of Primary Data Collection Activities

Primary Data Collection Sources	Data Collection Tools	Target population	Data collection frequency
Surveys	AHRQ Patient Safety Culture Change surveys: HSOPS, NHSOPS, MOSOPS	All staff at the participating sites	At baseline (pre-intervention) End of intervention
Assessment Tools	Structural Assessment Tool (different tools for acute care, long term care, and ambulatory care settings)	Unit leader at the participating sites	At baseline (pre-intervention) At the end of intervention (during/post)
Antibiotic Appropriateness Assessment Data	Team Antibiotic Review Form (assessment by Stewardship Team¹)	Patients who are prescribed antibiotics at the participating sites (data will be collected by Stewardship Team and will not contain protected health information)	Every month from the beginning of intervention (acute care and long-term care only)
Qualitative, In-Person Interviews (Cohort 1 only)	A semi-structured discussion guide modified for the different stakeholder groups (frontline staff, unit champions, and the Stewardship Team) and three different health care settings	Frontline staff, unit leaders, antibiotic Stewardship Team and organizational leaders from the three IDSs Recruit respondents from two acute care facilities, two long-term care facilities, and two ambulatory care facilities in each IDS for a total of 18 sites. At each institution, aim to sample 10 frontline staff and five stewardship champions/leaders	At the beginning of the intervention End of the intervention period

Surveys. The project will administer the AHRQ Patient Safety Culture Change survey to all AHRQ Safety Program for Improving Antibiotic Use participating staff at the beginning and end of the intervention. This survey has three versions, adapted for the three types of health care settings where the AHRQ Safety Program for Improving Antibiotic Use will be implemented:

The Hospital Survey on Patient Safety (HSOPS) will be utilized to evaluate safety culture for acute care hospitals (see Attachment F).
The Nursing Home Survey on Patient Safety (NHSOPS) will be administered in long term care (see Attachment G).
The Medical Office Survey on Patient Safety (MOSOPS) will be administered in ambulatory care centers (see Attachment H).

Each survey asks questions about patient safety issues, medical error, and event reporting in the respective setting. For patient safety culture surveys, the project will reach out to all eligible providers and staff in order to collect data according to AHRQ’s User Guide. All staff on the unit that is implementing the AHRQ Safety Program for Improving Antibiotic Use will be asked to complete the survey. As unit size varies, the project estimates the average number of respondents to be 25 for each unit. Participating staff should have enough knowledge about the day-to-day activities in the unit and interact regularly with other staff working in the unit in order to provide informed answers. These staff members, who spend all or most of their time at work within the unit, may include the following:

Staff who have direct contact or interaction with patients;
Staff who may not have direct contact or interaction with patients but whose work directly affects patient care;

Assessment Tools

The project will mainly collect primary data by administering assessment tools to staff members involved with the AHRQ Safety Program for Improving Antibiotic Use at each IDS:

Structural Assessment. The project will administer the Structural Assessment tool to the Stewardship Team in all three settings at baseline (pre-intervention) and at the end of the intervention period (see Attachment A through C for assessment tools for each of the three settings). This online assessment tool will ask the Stewardship Team lead five to seven questions to understand facility infrastructure and capacity to carry out the AHRQ Safety Program for Improving Antibiotic Use.

Team Antibiotic Review Form. Additionally, the Stewardship Team will conduct a monthly assessment to achieve a better understanding of the use of antibiotics being prescribed in acute care settings (see Attachment D for the antibiotic use review form) and in long-term care settings. Every four weeks, the Stewardship Team will review at least 10 patients’ medical records and fill out the appropriateness assessment tool to see if the CUSP “four moments of antibiotic prescribing” were considered.

Secondary Data Collection

EHR Data Extracts. The AHRQ Safety Program for Improving Antibiotic Use project, upon review and approval from our Technical Expert Panel (TEP), identified the following unit-level antibiotic usage and clinical measures related to antibiotic stewardship for each type of setting, which will be used to evaluate the effectiveness of the program. To be specific, the project will use these measures to answer 1) if the AHRQ Safety Program for Improving Antibiotic Use is effective to improve antibiotic use, and 2) the extent of the improvement. The TEP identified these measures as appropriate AS measures or these measures have been used in public reporting to monitor antibiotic use (e.g., CDC NHSN AUR Module²).

Every month between baseline (the three month period prior to the beginning of intervention) and the end of the intervention (after the nine month intervention period), each onsite data coordinator (with assistance from local IT) will extract the measures via their EHR system, and submit the data to the project evaluation team. The data will include the rates of antibiotic use by class and unit; CDI lab events for acute care hospitals; rates of antibiotic use by class, rates of starts by class, and CDI events for long-term care facilities; and ICD-10 codes for ambulatory care facilities. Attachment K contains the clinical measure template and Attachment L contains the instructions for conducting the data extract.

Exhibit B.5 describes the related clinical measures for the AHRQ Safety Program for Improving Antibiotic Use (including the appropriateness of antibiotic use which will be collected using the Team Antibiotic Review Forms described above), by setting.

Exhibit B.5: Clinical Measures for the AHRQ Safety Program for Improving Antibiotic Use

Setting	Clinical measures	Description
Acute care setting	Days of antibiotic therapy (DOTs) per 1,000 days present	A day of antibiotic therapy is defined by any amount of an antibiotic administered to a patient on a single calendar day. Each drug is counted independently. Days present are defined as the aggregate number of days patients were admitted to that hospital unit.
	Clostridium difficile laboratory event episodes per 10,000 patient days by unit (quarterly measure)	As prior antibiotic use is the risk factor with the greatest association for C. difficile infections, C. difficile rates will be evaluated on a quarterly basis during the base and intervention period. These data are currently reportable to CDC's National Healthcare Safety Network on a quarterly basis by acute care facilities in the United States. C. difficile laboratory event is defined as the number of all non-duplicate c. difficile toxin-positive laboratory results. Patient-days are defined as the aggregate number of days patients were admitted to that hospital unit.
	Appropriate use of antibiotics (through primary data collection; only during intervention period)	The Stewardship Team will review antibiotic use with the frontline team on at least 10 patients per month. The Stewardship Team will fill out the Team Antibiotic Review Form (Appendix B) which will not contain any PHI. The team will utilize the tool to assess if the four moments in antibiotic prescribing were considered.
Long term care setting	Days of antibiotic therapy (DOTs) per 1,000 resident-days	A day of antibiotic therapy is defined by any amount of an antibiotic administered to a patient on a single calendar day. Each drug is counted independently. Patient-days are defined as the aggregate number of days that patients resided in the long-term care setting.
	Rate of new antibiotic starts initiated in nursing home per 1,000 patient-days	Antibiotic starts is defined as prescriptions written after the resident has been admitted to the long-term care setting. Patient-days are defined as the aggregate number of days patients resided in the long-term care setting.
	Clostridium difficile laboratory event episodes per 10,000 resident days by unit (quarterly measure)	As prior antibiotic use is the risk factor with the greatest association for C. difficile infections, C. difficile rates will be evaluated on a quarterly basis during the base and intervention period. C. difficile laboratory event is defined as the number of all non-duplicate c. difficile toxin-positive laboratory results. Patient-days are defined as the aggregate number of days patients resided in the long-term care setting.
	Number of urine cultures obtained per 1,000 resident-days	Urine culture collection is defined as the number of urine cultures obtained. Patient-days are defined as the aggregate number of days patients resided in the long-term care setting.
Ambulatory care setting	Antibiotic therapy prescription per 100 patient visits	An antibiotic therapy prescription is defined by any amount of an antibiotic administered to a patient on a single calendar day. Each drug is counted independently. Patient-visits are defined as the aggregate number of visits made by any patient to an ambulatory care setting.
Ambulatory care setting	Proportion of antibiotic use by non-antibiotic appropriate respiratory condition	Proportion of patients with ICD-10 codes for non-antibiotic-appropriate respiratory conditions (non-specific URIs, acute bronchitis, influenza) prescribed antibiotic.

Evaluation Design

The pre-intervention baseline will be three months, and the during/post-intervention period will be nine months. The AHRQ Safety Program for Improving Antibiotic Use proposes a pre-post evaluation design, comparing the Structural Assessments and AHRQ Surveys on Patient Safety Culture data from each of the three facility settings to themselves from pre-intervention/baseline to post-intervention/end of intervention. The project will examine the change from baseline to post-intervention and investigate the impacts of other factors on those changes. For data elements that are collected monthly (i.e. Team Antibiotic Review Forms and EHR data), the project will apply an interrupted time series (ITS) analysis to investigate the performance of antibiotic usage and clinical outcomes over time and the impact from the AHRQ Safety Program for Improving Antibiotic Use, because the scope of the project neither includes selection of external comparison sites nor access to EHR data from non-participating sites.

Cohort 1 (Pilot Facilities)

The Cohort 1 pilot implementation data collection is designed to be powered at 80% to detect a decrease³ from baseline to post-intervention period⁴ for: 1) 61 days of antibiotic therapy (DOT) per 1,000 patient-days in acute care settings,⁵ 2) 24 DOT per 1,000 patient-days in long-term settings,⁶ and 3) 2.9 antibiotic prescriptions per 100 patient-visits in ambulatory care settings.⁷ The following assumptions were used in the power calculations: 1) 1-sided significant level of 0.05⁸, 2) ten units from each type of settings, and 3) within-unit correlation of 0.4⁹.

Exhibit B.6 below provides the power and the corresponding mean difference that the project will be able to detect from the baseline to post-intervention period, given different levels of significance and within-unit correlations based on the primary outcomes for each type of settings. Please note that we have included these illustrative power calculations in the spirit of transparency. The Pilot Period is intended to help test instrumentation and feasibility of our data collection schema, which will inform our Option Period data collection/analysis design, for which we have developed pro forma power calculations.

Exhibit B.6: Power Analysis for Cohort 1 (n=10, for each setting)

Correlation within unit	1-sided significance of 0.05				2-sided significance of 0.05
	Power	Detectable Difference			Power	Detectable Difference
	Power	DOT per 1,000 patient-days for acute care setting	DOT per 1,000 patient-days for LTC setting	Number of antibiotic prescription per 100 patient visits for ambulatory care setting	Power	DOT per 1,000 patient-days for acute care setting	DOT per 1,000 patient-days for LTC setting	Number of antibiotic prescription per 100 patient visits for ambulatory care setting
0.2	60%	53.7	21.5	2.6	60%	62.6	25.0	3.0
	70%	61.4	24.5	2.9	70%	70.3	28.1	3.4
	80%	70.3	28.1	3.4	80%	79.2	31.7	3.8
	90%	82.8	33.1	4.0	90%	91.7	36.7	4.4
0.4	60%	46.5	18.6	2.2	60%	54.2	21.7	2.6
	70%	53.1	21.3	2.6	70%	60.9	24.3	2.9
	80%	60.9	24.4	2.9	80%	68.6	27.4	3.3
	90%	71.7	28.7	3.4	90%	79.4	31.8	3.8
0.6	60%	38.0	15.2	1.8	60%	44.3	17.7	2.1
	70%	43.4	17.4	2.1	70%	49.7	19.9	2.4
	80%	49.7	19.9	2.4	80%	56.0	22.4	2.7
	90%	58.5	23.4	2.8	90%	64.8	25.9	3.1

Cohort 2 (Acute Care Hospitals), Cohort 3 (Long-Term Care Facilities) and Cohort 4 (Ambulatory Care Facilities)

The Cohort 2, 3, and 4 implementation data collection is designed to be powered at 80% to detect a decrease from baseline to post-intervention period for: 1) 12.8 DOT per 1,000 patient-days in acute care settings, 2) 5.1 DOT per 1,000 patient-days in LTC settings, and 3) 0.6 antibiotic prescriptions per 100 patient-visits in ambulatory care settings. The following assumptions were used in the power calculations: 1) 1-sided significant level of 0.05, 2) 250 units from each type of settings, 3) within-unit correlation of 0.4, and 4) response rate of 90%.

Exhibit B.7 below provides the power and the corresponding effect size the project will be able to detect from baseline to post-intervention period, given different levels of significance and within-unit correlations based on the primary outcomes for each type of settings.

Exhibit B.7: Power Analysis for Cohorts 2, 3, and 4 (n=250*0.9, for one setting^*)

Correlation within unit	1-sided significance of 0.05				2-sided significance of 0.05
	Power	Detectable Difference			Power	Detectable Difference
	Power	DOT per 1,000 patient-days for acute care setting	DOT per 1,000 patient-days for LTC setting	Number of antibiotic prescription per 100 patient visits for ambulatory care setting	Power	DOT per 1,000 patient-days for acute care setting	DOT per 1,000 patient-days for LTC setting	Number of antibiotic prescription per 100 patient visits for ambulatory care setting
0.2	60%	11.3	4.5	0.5	60%	13.2	5.3	0.6
	70%	12.9	5.2	0.6	70%	14.8	5.9	0.7
	80%	14.8	5.9	0.7	80%	16.7	6.7	0.8
	90%	17.4	7.0	0.8	90%	19.3	7.7	0.9
0.4	60%	9.8	3.9	0.5	60%	11.4	4.6	0.5
	70%	11.2	4.5	0.5	70%	12.8	5.1	0.6
	80%	12.8	5.1	0.6	80%	14.5	5.8	0.7
	90%	15.1	6.0	0.7	90%	16.7	6.7	0.8
0.6	60%	8.0	3.2	0.4	60%	9.3	3.7	0.4
	70%	9.1	3.7	0.4	70%	10.5	4.2	0.5
	80%	10.5	4.2	0.5	80%	11.8	4.7	0.6
	90%	12.3	4.9	0.6	90%	13.7	5.5	0.7

^* For Cohorts 2, 3, and 4, the project plans to recruit 250 to 500 facilities for each type of setting (acute care, ambulatory care, LTC). The power analysis is based on the lower end of the projections (250 facilities per type). The analysis will be able to detect smaller change as the sample size increases.

Qualitative Analysis (Cohort 1 only). The project will also conduct semi-structured qualitative interviews during Cohort 1 to assess the factors, including site-specific social norms and prescribing behavior, which facilitate/hinder implementation of the intervention. These one-on-one in-depth discussions will occur during three one-week site visits, one at each of the IDS locations, with stewardship champions/organizational leaders and frontline staff in the units where the program is being implemented. Stewardship champions/organizational leaders will include the individuals who are charged with implementing the AHRQ Safety Program for Improving Antibiotic Use at each site as well as other facility managers or leaders as deemed relevant. Frontline healthcare workers will include physicians, pharmacists, nurse practitioners, physician assistants, nurses and others deemed relevant to prescribing by setting (e.g., we will sample more nurses in the LTC setting than in other settings). These site visits will occur during the three-month baseline period and at the end of the Cohort 1 intervention period.

3. Methods to Maximize Response Rates

The data collection planned under this project is part of an evaluation to assess the adoption of the CUSP for AS in acute care, long-term care and ambulatory care settings. The results of the evaluation will be used to inform the CUSP for AS program; they will not yield generalizable results or be used for statistical estimation purposes. The project will recruit facilities that have indicated a willingness to participate in the project and meet the inclusion criteria, and who will supportive of and likely to spread the CUSP for AS model.

To encourage the participating facilities to complete and submit the Structural Assessments, Team Antibiotic Review Forms, AHRQ Surveys on Patient Safety, and EHR data, the project will implement the following strategies:

Ensure team understands the value of data collection for their ongoing efforts to sustain their antibiotic stewardship efforts (i.e., share data reports and performance results with unit staff).
Offer technical assistance webinars and onboarding calls (including information related to submitting data) to participating sites
Prompt the on-site data coordinator and unit/facility leader for feedback from each participating unit regarding the data collection activities and AS-related performance
Work closely with each on-site data coordinator to address any data collection issues and develop site-specific data collection strategies when necessary
Discuss data management and submission methods with the participating sites early in the process
Provide sites flexibility to submit data (e.g., via a central data collection platform, via IDS’s own data collection system, or using pre-coded password-protected spreadsheet)
Provide step-by-step instructions on data sharing via the central data collection platform
Ensure that data collection tools are simple, easy to use, and succinct
Ensure missing or inaccurate data is identified early and that issues are resolved in a systematic, highly-reliable way
Share data collection strategies/best practices to simplify processes through coaching calls

4. Tests of Procedures

The Cohort 1 pilot period will perform as a pilot project to test the program design and implementation. The AHRQ Safety Program for Improving Antibiotic Use will also conduct semi-structured qualitative interviews during Cohort 1 to assess the factors which facilitated/hindered implementation of the intervention. The information collected will allow the project to:

Gauge the feasibility of collecting the same information when the pilot scales.
Identify areas where staff wanted/needed more training.
Identify the challenges and barriers of implementing the AHRQ Safety Program for Improving Antibiotic Use. Findings will inform the implementation approach for the subsequent cohorts.

The AHRQ Surveys on Patient Safety Culture are publicly available and have been widely used to assess patient safety culture in a variety of health care settings.

5. Statistical Consultants

Johns Hopkins University and NORC at the University of Chicago will serve as the primary consultants for statistical aspects of the design and analysis of the evaluation data.

Exhibit B.8: List of Statistical Consultants

Name	Title and Institution	Telephone Number
Roy Ahn, ScD, MPH	Associate Director, NORC	312/759-4068
Pranita Tamma, MD, MHS	Assistant Professor of Pediatrics in Division of Infectious Diseases and Director of the Pediatric Antimicrobial Stewardship Program, John Hopkins University	410/614-1492
Sara Cosgrove, MD, MS, FSHEA, FIDSA	Professor of Medicine in the Division of Infectious Diseases and Director of the Department of Antimicrobial Stewardship, John Hopkins University	443/287-4590
Sai Loganathan	Senior Research Scientist, NORC	301/634-9346
Yue Gao, MPH	Research Scientist, NORC	301/634-9440
Michael Kohut, PhD	Qualitative Researcher, John Hopkins University	410/637-7228

The data will be collected by NORC at the University of Chicago.

1 Please note this measure, as a component in antibiotic prescribing appropriateness, will be collected via primary data collection. Other components, such as days of antibiotic therapy days per 1,000 patient days, will be collected via secondary data collection through EHR extraction and are included in the secondary data collection section.

2 https://www.cdc.gov/nhsn/pdfs/training/aur-training.pdf

3 Assuming the standard deviation is the same for baseline and post-intervention period. Based on prior studies, we set up the value as 50 for DOT per 1,000 patient-days in acute care setting, 20 for DOT per 1,000 patient-days in LTC setting and 2.4 for number of antibiotic prescriptions per 100 patient-visits in ambulatory care setting.

4 Of note, data will be kept in a longitudinal way to use all 12 repeated measures for each outcome in the analysis. We aim to detect the pre- and post- difference using interrupted time series method, which is a commonly used method in interventions for improving antibiotic use.

5 Morrill HJ, Caffrey AR, Gaitanis MM, LaPlante KL. Impact of a prospective audit and feedback antimicrobial stewardship program at a Veterans Affairs medical center: a six-point assessment. PLoS One. 2016;11(3):e0150795.

6 Daneman N, Bornskill SE, Gruneir A, et al. Variability in antibiotic use across nursing homes and the risks of antibiotic-related adverse outcomes for individual residents. JAMA Intern Med. 2015;175(8):1331-9.

7 Fleming-Dutra KE, Hersh AL, Shapiro DJ, et al. Prevalence of inappropriate antibiotic prescriptions among US ambulatory care visits, 2010-2011. JAMA. 2016;315(17):1864-73.

8 We assumed a 1-sided significance level given that we expect the intervention would improve the outcome (very likely to be a positive change only). However, we also presented the result with an assumption of 2-sided significance level as a standard assumption.

9 Campbell MK, Mollison J, Grimshaw JM. Cluster trials in implementation research: estimation of intracluster correlation coefficients and sample size. Stat Med. 2001;20(3):391-9. In this reference, within-patient correlation is estimated as ICC = 0.47 (95% CI 0.29 to 0.65). Based on this reference, we assumed the within-unit correlation as 0.4 and range from 0.2 to 0.6.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	SChowdhury
File Modified	0000-00-00
File Created	2021-01-22

Supporting Statement Part B AHRQ Clean