SUPPORTING STATEMENT
Part B
The AHRQ Safety Program for Methicillin-Resistant Staphylococcus aureus (MRSA) Prevention
June 17, 2021
Agency for Healthcare Research and Quality (AHRQ)
B. Collections of Information Employing Statistical Methods 4
1. Respondent Universe and Sampling Methods 4
2. Information Collection Procedures 9
Attachment
A: Federal Register Notice
Attachment B: ICU/Non-ICU Gap
Analysis
Attachment C: Surgical Services Gap Analysis
Attachment D: LTC Gap Analysis
Attachment E: ICU/non-ICU Team Checkup Tool
Attachment F: Surgical Services Team Checkup Tool
Attachment G: LTC Team Checkup Tool
Attachment H: Survey of Patient Safety Culture –– HSOPS
Attachment I: Survey of Patient Safety Culture –– NHSOPS
Attachment J: Clinical Outcomes Data for ICU/non-ICU
Attachment K: Clinical Outcomes Data for Surgical Services
Attachment L: Clinical Outcomes Data for LTC
Attachment M: Completion Guide for ICU/non-ICU Clinical Outcomes Data
Attachment N: Completion Guide for Surgical Services Clinical Outcomes Data
Attachment O: Completion Guide for LTC Clinical Outcomes Data
Attachment P: Schedule of Evaluation Activities
The data collection planned under this program is part of a comprehensive evaluation strategy to assess the adoption of the AHRQ Safety Program for Methicillin-Resistant Staphylococcus aureus (MRSA) Prevention in ICU, non-ICU, surgical, and long-term care (LTC) settings; measure the effectiveness of the interventions in the participating facilities or units; and evaluate the characteristics of teams that are associated with successful implementation and improvements in outcomes. A key component of the AHRQ Safety Program for MRSA Prevention is the recruitment of facilities within sites and delivery systems that will be supportive of and likely to spread the AHRQ Safety Program for MRSA Prevention model.
This data collection request covers planned activities with four cohorts: ICU units, non-ICU units, surgical services units, and LTC facilities.
Recruitment Methods:
The program will be rolled out using a phased approach with four cohorts: ICU (Cohort 1) and non-ICU (Cohort 2) settings, surgical settings (Cohort 3), and LTC facilities (Cohort 4). The recruitment methods and targets for each are described below.
Targets
ICU and Non-ICU Settings (Cohorts 1 and 2): Facility recruitment for the ICU and non-ICU settings will target hospitals with high cumulative attributable differences (CADs) of MRSA invasive infections (defined as MRSA bacteremia) based on National Healthcare Safety Network (NHSN) data. The program will then over-recruit from these regions with the highest quintile of CAD scores and move to lower quintiles if necessary to reach the recruitment targets. The program will collaborate with the Centers for Disease Control and Prevention’s (CDC) Division of Healthcare Quality Promotion (DHQP) Surveillance Branch with AHRQ’s facilitation to obtain these data and/or use other data sources such as the CMS Hospital Compare data.
From within these hospitals with high CADs of MRSA invasive infections, the program will recruit at least 400 ICU and 400 non-ICU units. Examples of these units include the following:
ICU settings
Cardiac Intensive Care
Coronary Care and Cardiothoracic Intensive Care
Medical Intensive Care
Medical-Surgical Intensive Care
Pediatric Intensive Care
Thoracic Surgery Intensive Care
Other Intensive Care
Non-ICU settings
Medical/surgical unit
Medical unit
Surgical unit
Surgical Services (Cohort 3): The program will target at least 300 hospital surgical service units including high risk surgical service areas of orthopedic, cardiothoracic, and neurosurgery units. Similar to the strategy for the rest of the hospital cohorts, we will identify hospitals with high CAD scores using the Hospital Compare and, if feasible, NHSN data.
Long-Term Care (Cohort 4): For LTC facilities, the program will recruit at least 300 LTC facilities. LTC facilities may include nursing homes, dementia care facilities, residential and continuing care facilities, skilled nursing facilities (SNF), and hospice facilities. The LTC program is not designed for stand-alone long-term acute care hospitals (LTACHs), assisted living facilities, rehabilitation facilities, adult day care, home health programs, or facilities specializing in the care of developmentally disabled or pediatric populations.
Facility Characteristics
To meet the recruitment targets, the program will cast a wide net and ensure that eligibility criteria garners broader participation rather than unnecessarily excluding potential sites. In addition to ensuring a broad geographic coverage, the program will seek to ensure diversity of recruited sites. The program anticipates that the existing distribution of targeted hospitals and LTC facilities will ensure a broad national geographic reach and include diverse locations of integrated delivery systems and nursing home chains with various types of facilities, urbanicity, number of beds, number of providers, types of Electronic Health Record (EHR) systems, and teaching status. To illustrate the broad distribution of hospitals, in Exhibit B.1 we have calculated the CAD scores using publicly reported hospital-level infection rates from Hospital Compare and generated a list of states and HHS regions with CAD scores that are in the top quintile of the distribution.
The characteristics by setting are further highlighted below:
Hospital ICU and non-ICU Settings. In ICU and non-ICU cohorts (Cohorts 1 and 2), the following hospitals will be eligible for participation: general acute care hospitals, women’s hospitals, children’s hospitals, and oncology hospitals. Other characteristics include urban and rural, academic and non-academic hospitals, small, medium and large hospitals, independent and part of larger health systems. The following hospitals are ineligible for the program: military hospitals, orthopedic hospitals, freestanding inpatient psychiatric facilities, surgical hospitals, Veterans’ Affairs hospitals, and small rural critical access hospitals.
Hospital Surgical Setting. For the surgical setting cohort (Cohort 3), the program will recruit facilities that are both urban and rural, small, medium and large hospitals.
LTC Setting. For the LTC cohort (Cohort 4), the program will recruit LTC facilities that are both urban and rural, and vary by facility size, number of beds, nursing home rating, ownership (for-profit, not-for-profit, and chain and independent facilities).
Exhibit B.1. Location of Hospitals in Highest Quintile of CAD Scores*
HHS Region |
MRSA Bacteremia: Observed Cases |
CAD (i.e., number of MRSA Bacteremia Cases that must be prevented to meet HHS Goal ) |
Number of Hospitals in the top quintile of CAD rates |
State with the highest number of hospitals in the top quintile of CAD rates |
Number of hospitals in the state that are in the top quintile of CAD rates |
Region 1 – Boston: Connecticut, Maine, Massachusetts, New Hampshire, Rhode Island, and Vermont |
99 |
59 |
14 |
Connecticut |
6 |
Region 2 - New York: New Jersey, New York, Puerto Rico, and the Virgin Islands |
734 |
430 |
84 |
New York |
56 |
Region 3 – Philadelphia: Delaware, District of Columbia, Maryland, Pennsylvania, Virginia, and West Virginia |
627 |
313 |
76 |
Pennsylvania |
27 |
Region 4 – Atlanta: Alabama, Florida, Georgia, Kentucky, Mississippi, North Carolina, South Carolina, and Tennessee |
1996 |
1180 |
210 |
Florida |
76 |
Region 5 – Chicago: Illinois, Indiana, Michigan, Minnesota, Ohio, and Wisconsin |
807 |
449 |
105 |
Ohio |
37 |
Region 6 – Dallas: Arkansas, Louisiana, New Mexico, Oklahoma, and Texas |
842 |
506 |
115 |
Texas |
60 |
Region 7 - Kansas City: Iowa, Kansas, Missouri, and Nebraska |
196 |
112 |
28 |
Missouri |
16 |
Region 8 – Denver: Colorado, Montana, North Dakota, South Dakota, Utah, and Wyoming |
64 |
28 |
13 |
Colorado |
6 |
Region 9 - San Francisco: Arizona, California, Hawaii, Nevada, American Samoa, Commonwealth of the Northern Mariana Islands, Federated States of Micronesia, Guam, Marshall Islands, and Republic of Palau |
750 |
428 |
123 |
California |
73 |
Region 10 – Seattle: Alaska, Idaho, Oregon, and Washington |
86 |
44 |
13 |
Washington |
6 |
*CAD rate based on Hospital Compare data (https://data.cms.gov/provider-data/topics/)
Recruitment Strategy. The program will target hospitals with high CADs of MRSA invasive infections based on NHSN data and over-recruit from these regions with the highest quintile(s) of CAD scores. The program will collaborate with the CDC‘s DHQP Surveillance Branch and/or use other data sources such as Hospital Compare data to identify sites with high CAD scores and/or high levels of MRSA invasive infection rates.
The NHSN data for recruitment will only include the CAD ranking, not the actual CAD scores themselves due to data privacy regulations. CDC will provide the list of all hospitals with CAD scores so the program has a complete picture of the number of hospitals and can expand the targeted list as needed to reach recruitment goals. The program team will be flexible and have the data if needed to expand targeted recruitment efforts beyond hospitals in the top quintile of CAD scores. The program will ensure sites include all HHS regions.
The program will develop recruitment tools and enrollment materials for each cohort, working across program stakeholders including internal clinical and CUSP experts, Technical Expert Panel members, AHRQ, and Quality Innovation Network – Quality Improvement Organization (QIN-QIO) Implementation Advisers (IAs) for feedback and to ensure effective messaging for each cohort’s target audience. Recruitment materials will list program benefits of participation and requirements.
CDC will notify hospitals with high CADs of MRSA invasive infections based on NHSN data that their contact information and CAD ranking has been shared with the AHRQ Safety Program for MRSA Prevention. The program team will then reach out directly to the targeted hospitals to introduce them to the Program and encourage them to apply. The direct outreach will provide a brief overview of the program, benefits, expectations of hospitals, and program data collection requirements.
Respondent Selection and Sample Sizes
The AHRQ Safety Program for MRSA Prevention will collect unit- and facility-level data from all participating units and facilities when possible. There are two types of teams in each unit or facility who will participate: one Infection Prevention (IP) coordinating team and a number of individual unit-based CUSP teams (i.e., frontline providers and staff in each unit; CUSP teams) who will implement specific IP practices. An IP coordinating team member will participate with and inform each of the CUSP teams, becoming an active member of the unit-based CUSP team. In all cohorts, frontline staff may include rounding clinicians, bedside nurses, nurse practitioners, respiratory therapists, unit assistants, nursing educators and any other unit-based staff, such as housekeeping. The IP Coordinating team includes physicians, nursing unit leaders, and infection preventionists.
Participating
facilities will be asked to collect the following data:
Gap Analysis. One unit leader and one Infection Preventionist at each facility will complete the gap analysis at month1, (start of the implementation period) and month 18, (end of the implementation period to assess the needs of participating units, prioritize areas for improvement, and advocate for institution-level and unit-level resources . The endline analysis will assess the unit’s progress in building infrastructure and capacity to sustainably reduce MRSA infections.
Team Checkup Tool. One frontline staff within CUSP teams per unit or facility will implement the Team Checkup Tool (TCT), an assessment of implementation fidelity. The TCT will be collected monthly during the implementation period.
Patient safety culture assessment. At month 1 and month 18 of the implementation period, we will administer the AHRQ Surveys on Patient Safety Culture (i.e., HSOPS and NHSOPS), with frontline staff on individual, unit-based CUSP teams or facility as instructed by the AHRQ survey user guide. Because unit size varies, the program estimates the average number of respondents to be 25 for each unit or facility.
Outcome measures. Clinical outcome measures will be collected for the 12 month period prior to the start of the implementation and during the 18 month implementation.
Primary data collection. For clinical outcome measures that are not available in the National Healthcare Safety Network (NHSN) data and for those hospitals with participating ICU, non-ICU, and surgical services units that do not confer rights to their NHSN data to the program, clinical outcome measures will be collected quarterly from unit-level EHR extracts. Clinical outcome data in LTC settings will be obtained via unit-level EHR data extracts, pulled on a monthly basis. The EHR data will be collected by unit/facility leaders with support from IT staff at each facility.
Secondary data collection. The secondary data collection strategy includes use of NHSN data from hospitals that confer rights to the AHRQ Safety Program for MRSA Prevention to use their NHSN data for the evaluation. NHSN data will serve as data sources for some clinical outcomes in ICU, non-ICU, and surgical services units. Clinical outcome measures in LTC settings are not available in NHSN.
Response Rates
Based on the actual data submission rates for the AHRQ Safety Program for Improving Antibiotic Use, a similar implementation project with data collection and which included acute cohort hospital units and LTC facilities, we anticipate a 70-75% cooperation rate from the participating facilities for the Gap Analysis, Team Checkup Tool, and quarterly/monthly submissions of the required EHR data.
We anticipate approximately 90% of hospitals in ICU and non-ICU settings to confer rights for the AHRQ Safety Program for MRSA Prevention to pull their NHSN data for the evaluation. In surgical settings, we expect approximately 80% of hospitals to confer rights to the program to pull their NHSN data for the evaluation. Similar quality improvement projects have achieved 100% NHSN conferral response rates from acute care hospitals for similar outcome measures that are mandated to be reported to NHSN. A 90% conferral rate is assumed for ICU and non-ICU to be conservative since the AHRQ Safety Program for MRSA Prevention is a new project that sites may not be familiar with at the time of recruitment. An 80% conferral rate is assumed for surgical services since these data are not all mandated to be reported to NHSN, although most hospitals report them.
We anticipate a 10% cooperation rate for the optional MRSA point prevalence survey for the ICU and non-ICU settings.
Completion rates for the AHRQ Surveys on Patient Safety Culture, based on previous CUSP interventions focusing on VAP, CLABSI, and antibiotic stewardship, are anticipated to range between 30-50%. For those interventions, the hospital or unit response rates for the AHRQ Patient Safety surveys varied from baseline to follow-up and from project to project. The response rates averaged 55% at baseline and between 16-47% at follow-up.
This section describes procedures for collecting primary data (i.e., unit-level EHR data extracts, surveys, implementation assessment, and gap analyses) and secondary data (i.e., NHSN data).
Primary Data Collection
The program will collect a range of data to contribute substantively to the evaluation and facilitate analysis of progress over time. The program will collect primary data from EHR extracts of unit-level clinical outcome data and administration of assessment tools to staff involved with the AHRQ Safety Program for MRSA Prevention at each site. Primary data collection includes surveys and assessment tools (see Exhibit B.2 below).
During program registration (pre-implementation) each participating site will be asked to identify an AHRQ Safety Program for MRSA Prevention infection prevention (IP) lead and/or data coordinator who will facilitate data collection at their site (i.e., a member of the IP team and/or IT support may lead data coordination at each site). Each site will receive access to the secure website portal to submit data, accompanied by a data collection and submission guide. The guide will contain information on the purpose of the data collection, types of data to be collected and submitted, data collection and submission timeline for each data element, and step-by-step instructions for completing and submitting the forms on the website portal. For all cohorts, the implementation period will be 18 months.
Secondary Data Collection
The secondary data collection strategy includes use of NHSN data from hospitals that confer rights to the AHRQ Safety Program for MRSA Prevention to use their NHSN data for the evaluation. NHSN data will serve as secondary data sources for clinical outcomes in ICU, non-ICU, and surgical services units. Clinical outcome measures in LTC settings are not available in NHSN.
For hospitals that confer NHSN rights to the program for the ICU and non-ICU cohorts, the secondary data will include:
Hospital onset MRSA invasive infection (MRSA bacteremia LabID Day 3 or after of admission
Patient days
Central Line-Associated Blood Stream Infections with causative organism(s)
Central Line Days
Community onset MRSA invasive infection (MRSA bacteremia LabID prior to Day 3 after admission)
For hospitals that confer NHSN rights to the program for the surgical services cohort, the secondary data will include:
Surgical site infection (SSI) events and causative organisms
Number of surgical procedures performed, by type of surgical procedure
In order to utilize NHSN data, participating hospitals would need to confer rights to their NHSN data to the AHRQ Safety Program for MRSA Prevention. Any entity, such as hospital systems, state health departments, and quality improvement organizations, can maintain a Group in NHSN. These entities can share data with partners and agencies, such as the AHRQ Safety Program for MRSA Prevention, using NHSN’s Group function. These hospitals can join Groups and provide access to data requested by the Group within the NHSN application. Groups are responsible for creating a Confer Rights Template of data they are requesting from facilities; the template is automatically sent to the Group’s member facilities upon completion.
After facilities confer rights and join the MRSA Prevention Program group in NHSN, they will not need to take any further action. The data they submit to NHSN will automatically become available to the MRSA program team for analysis. We will have access to the 12 months of retrospective data prior to start of implementation and be able to do quarterly pulls of monthly data during the 18 month implementation.
Exhibit B.2: Summary of Primary Data Collection Activities
Primary Data Collection Sources |
Data Collection Tools |
Target population |
Data collection frequency |
Surveys |
AHRQ Surveys of Patient Safety Culture (HSOPS, NHSOPS) |
Frontline staff on CUSP teams at all participating sites.
Estimated 25 staff per facility or unit
|
All cohorts:
|
Infrastructure Assessments
|
Gap Analysis (GA)
|
Unit leader and Infection Preventionist at the participating sites
|
All cohorts: GA – Once at the start of implementation (month 1) and once post-implementation period (month 18)
|
Implementation Assessment |
|
1 staff per facility, unit or service
|
All cohorts:
|
EHR extracts of unit-level data (unless conferring rights to NHSN) |
|
Data coordinator at participating sites
Estimated 1 staff per facility, unit or service |
All cohorts:
|
Surveys. The program will administer the AHRQ Surveys of Patient Safety Culture (see Attachments H and I) to all AHRQ Safety Program for MRSA Prevention staff at the participating units or facilities at the beginning and end of the implementation period. The survey has separate versions adapted for hospital settings and LTC settings:
The Hospital Survey on Patient Safety (HSOPS) will be utilized to evaluate safety culture for ICU, Non-ICU and surgical units.
The Nursing Home Survey on Patient Safety (NHSOPS) will be administered in LTC facilities.
Each survey asks questions about patient safety issues, medical error, and event reporting in the respective setting. For patient safety culture surveys, the program will reach out to all eligible providers and staff in order to collect data according to AHRQ’s User Guide. All eligible providers and staff on the unit or facility that is implementing the AHRQ Safety Program for MRSA Prevention will be asked to complete the survey. As unit and facility size vary, the program estimates the average number of respondents to be 25 for each unit. Participating staff should have enough knowledge about the day-to-day activities in the unit or facility and interact regularly with other staff working in the unit or facility in order to provide informed answers. These staff, who spend all or most of their time at work within the unit or facility, may include the following:
Staff who have direct contact or interaction with patients or residents;
Staff who may not have direct contact or interaction with patients or residents but whose work directly affects patient or resident care.
Infrastructure Assessment. There is one assessment of setting infrastructure: the Gap Analysis. The program will administer the Gap Analysis to unit leaders and Infection Preventionists in all four cohorts at the start of the implementation period (month 1) and endline (month 18) (see Attachments B, C, and D for assessment tools for each of the four cohorts). This online assessment tool has two parts. The first part addresses infection prevention program structure, activities, and resources, and should be completed by the Infection Prevention Team. The second part addresses infection control activities, specifically those related to MRSA prevention, on the participating unit and should be completed by the Project Lead for the participating unit, in collaboration with the infection preventionist working with the unit. Each part of the assessment will ask unit leaders and Infection Preventionists up to 65 questions to understand facility infrastructure and capacity to carry out the AHRQ Safety Program for MRSA Prevention.
Implementation Assessment Tools. There is one fidelity assessment tool: the Team Checkup Tool. This tool will be filled out by frontline staff who participate in their unit-based CUSP teams. They will submit assessments through a web portal designed for the AHRQ Safety Program for MRSA Prevention.
Team Checkup Tool. The program will use the Team Checkup Tool to monitor key actions of staff. This Tool asks about use of safety guidelines, tools, and resources throughout three different phases: Assessment (1), Planning, Training, and Implementation (2), and Sustainment (3). See Attachments E, F, and G.
Collection of clinical outcomes
The primary outcome for this study for the ICU, non-ICU, and Surgical Services Cohorts is the change in MRSA infection rate. The primary outcome for the LTC Cohort is transfer of facility resident to an acute care hospital with reason of infection or suspected infection. We will collect primary outcome data from units/facilities participating in the Program through EHR data extracts.
EHR Data Extracts. The AHRQ Safety Program for MRSA Prevention identified the following unit-level clinical measures related to MRSA prevention for each type of setting, which will be used to evaluate the effectiveness of the program. Outcomes are specific to the cohort.
The data for the ICU and non-ICU cohorts will include:
Hospital onset MRSA invasive infection (MRSA bacteremia LabID Day 3 or after of admission) *
Community onset MRSA invasive infection (MRSA bacteremia LabID prior to Day 3 after admission) *
Patient days*
Central Line-Associated Blood Stream Infections with causative organism(s) *
Central line days*
Hospital-onset bacteremia (i.e., including MSSA)
Point prevalence survey - Patients with positive MRSA nasal surveillance tests (one day, semi-annually) (optional measure)
Point prevalence survey - Total surveillance tests in the unit during the chosen day (one day, semi-annually) (optional measure)
The data for the surgical services cohort will include:
Surgical site infection (SSI) events and causative organisms*
Number of surgical procedures performed, by type of surgical procedure*
*Signifies clinical outcomes available in NHSN data. Sites that confer NHSN rights to the program will not need to submit these data via the EHR data extract tool. If a participating hospital decides not to confer rights to their NHSN data to the program team, then all clinical outcome data will be collected via EHR data extracts.
The data for the LTC cohort will include:
Transfer of facility resident(s) to an acute care hospital, with reason of suspected or confirmed infection
Transfer of facility resident(s) to an acute care hospital, with reason other than infection
All-cause bacteremia with causative organisms
Resident days
The program will use these measures to answer 1) if the AHRQ Safety Program for MRSA Prevention program is effective to improve MRSA prevention, and 2) the extent of the improvement.
During each month or quarter between the start of the implementation period (month 1) and the end of the implementation period (month 18), each onsite data coordinator (i.e., ICU team leadership or facility leader), with assistance from local IT -will extract the measures via their EHR systems and submit data to the program evaluation team. ICU, non-ICU, and surgical settings will provide this monthly data on a quarterly basis, while LTC settings will do so on a monthly basis. For ICU, non-ICU, and surgical settings, this will result in a total of six quarterly submissions during the 18 month implementation, and 18 implementation data points. Quarterly data submissions will be due one month after the close of each quarter. LTC settings will provide 18 monthly submissions during the implementation, also resulting in 18 implementation data points. For the initial data pull, all settings will also provide monthly retrospective data on a quarterly basis from the previous 12 months. This will result in a total of four quarterly submissions and 12 retrospective data points
Attachments J, K, and L contain the clinical measure template.
Evaluation Design
The AHRQ Safety Program for MRSA Prevention proposes an interrupted time series (ITS) design to assess MRSA invasive infections (defined as MRSA bacteremia) in ICU, non-ICU, and Surgical Services settings, transfer of facility residents to acute care hospitals with reason of suspected or confirmed infection in LTC settings, and secondary clinical outcomes, using 18 months of implementation data and 12 months of retrospective data. The other proposed primary data collection activities will allow the team to assess program participation, effectiveness of educational materials, implementation effectiveness, and the impact on patient safety culture at participating sites.
To assess program impact on unit-level clinical outcomes, the primary outcome to be measured in ICU and non-ICU units is hospital-onset MRSA invasive infection (MRSA bacteremia LabID Day 3 or after of admission).1 The secondary outcomes to be measured include all-cause bacteremia (i.e., including Methicillin Susceptible Staphylococcus aureus (MSSA)), optional semi-annual MRSA colonization point prevalence surveys of MRSA colonization, and data on other MDROs. The primary outcome to be measured in surgical units is SSI events and causative organisms. The primary outcome to be measured in LTC settings is transfer of facility resident(s) to an acute care hospital, with reason of suspected or confirmed infection.
The multiple baseline data points will allow us to establish the pre-implementation trends in clinical outcomes crucial to the ITS design. Our ITS model will test whether trends in clinical outcomes during the implementation period are statistically different from baseline trends. This will include tests for both level and slope changes, as well as slope changes with a possible lag.
We plan to recruit at least 300 hospital surgical and LTC facilities, and 400 ICU and non-ICU units, each. The power analysis is based on the lower end of the projections (270 or 360 facilities per type) to allow for possible withdrawals or underrecruitment. The analysis will be able to detect smaller change as the sample size increases.
ICU (Cohort 1) and Non-ICU (Cohort 2)
For the main outcome of interest, the Cohort 1 and 2 implementation data collection is designed to be powered at 80 percent to detect a decrease2 from baseline to post-implementation for the number of quarterly MRSA invasive infections (i.e., MRSA bacteremia) per 100 admissions in ICU and non-ICU settings.3 The following assumptions were used in the power calculations: (1) significance level of 0.054; (2) a sample size of 400 units for each of type of two settings (ICU/non-ICU; and (3) a within-unit correlation of 0.2, 0.4, or 0.65. We also assume an intra-cluster correlation of 0.3.
Exhibit B.3 below provides the power and the corresponding mean difference that the project will be able to detect from the baseline to the post-implementation period, given different levels of significance and within-unit correlations based on the primary outcomes for each type of setting.6
Exhibit B.3: Power Analysis for Cohorts 1 and 2 (n=400 for each setting)
Correlation within unit |
Clusters (# Hospitals) (ICC=0.3) |
Cluster Size (# units per hospital) |
Detectable Difference for the Number of Monthly MRSA Invasive Cases per 100,000 Patient Days for ICU or non-ICU Setting |
|
1-sided significance of 0.05 |
2-sided significant of 0.05 |
|||
0.2 |
100 |
4 |
1.30 |
1.47 |
200 |
2 |
1.08 |
1.22 |
|
400 |
1 |
0.95 |
1.07 |
|
0.4 |
100 |
4 |
1.13 |
1.27 |
200 |
2 |
0.93 |
1.05 |
|
400 |
1 |
0.82 |
0.92 |
|
0.6 |
100 |
4 |
0.92 |
1.04 |
200 |
2 |
0.76 |
0.86 |
|
400 |
1 |
0.67 |
0.75 |
|
Surgical services (Cohort 3) and LTC Facilities (Cohort 4)
For our main outcome of interest, the implementation period data collection is designed to be powered at 80% to detect a decrease from the start of implementation to endline for monthly hospital-onset MRSA invasive infections (i.e., MRSA bacteremia) per 10,000 patient/resident days in Surgical settings, and monthly transfers of LTC residents to acute care hospitals for infections. Monthly MRSA infection data will be collected on a quarterly basis from EHR or NHSN data over a 12-month baseline (i.e., pre-implementation) period for a total of 12 retrospective data points, and over an 18-month period during implementation for a total of 18 implementation data points.
The following assumptions were used in the power calculations: 1) 1-sided significance level of 0.05, 2) 300 units from each type of settings, 3) within-unit correlation of 0.4, and 4) response rate of 90%.
Exhibit B.4 below provides the power and the corresponding effect size the program will be able to detect from baseline to endline, given different levels of significance and within-unit correlations based on the primary outcomes for each type of setting.
Exhibit B.4: Power Analysis for Cohorts 3 and 4 (n=270 per cohort)
Correlation within unit |
1-sided significance of 0.05 |
2-sided significance of 0.05 |
||||||
Power |
Detectable Difference |
Power |
Detectable Difference |
|||||
Number of Monthly MRSA invasive infections per 10,000 patient days for surgical setting |
Number of Monthly transfers of LTC residents to an acute care hospital7 |
|
Number of Monthly MRSA invasive infections per 10,000 patient days for surgical setting |
Number of Monthly transfers of LTC residents to an acute care hospital |
|
|||
0.2 |
60% |
0.88 |
1.23 |
|
60% |
1.03 |
1.44 |
|
70% |
1.00 |
1.41 |
|
70% |
1.15 |
1.61 |
|
|
80% |
1.15 |
1.61 |
|
80% |
1.30 |
1.82 |
|
|
90% |
1.36 |
1.90 |
|
90% |
1.50 |
2.10 |
|
|
0.4 |
60% |
0.76 |
1.07 |
|
60% |
0.89 |
1.02 |
|
70% |
0.87 |
1.22 |
|
70% |
1.00 |
1.40 |
|
|
80% |
1.00 |
1.40 |
|
80% |
1.13 |
1.58 |
|
|
90% |
1.17 |
1.64 |
|
90% |
1.30 |
1.82 |
|
|
0.6 |
60% |
0.62 |
0.87 |
|
60% |
0.73 |
1.02 |
|
70% |
0.71 |
0.99 |
|
70% |
0.81 |
1.14 |
|
|
80% |
0.81 |
1.14 |
|
80% |
0.92 |
1.29 |
|
|
90% |
0.96 |
1.34 |
|
90% |
1.06 |
1.49 |
|
|
*n=270 is based on 300 units participating in the study per cohort and a 90% response rate.
The data collection planned under this program is part of an evaluation to assess the adoption of the AHRQ Safety Program for MRSA Prevention in ICU, Non-ICU, surgical services, and LTC settings. The results of the evaluation will be used to inform the AHRQ Safety Program for MRSA Prevention; they will not yield generalizable results or be used for statistical estimation purposes. In addition to selecting facilities that have high CAD rates (see Exhibit B.1), the program will recruit facilities that have indicated a willingness to participate in the program, meet the inclusion criteria, and who will support and likely spread the AHRQ Safety Program for MRSA Prevention model.
To encourage the participating facilities to complete and submit the data collection elements, the program will implement the following strategies:
Encourage hospitals to confer rights to their NHSN data to the program for the purposes of the evaluation, thereby decreasing burden of data extraction from EHRs.
Ensure that data collection tools are simple and easy to use, including providing sites with a template for the monthly EHR data extracts.
Ensure sites understand the value of data collection for their ongoing efforts to sustain their MRSA prevention efforts (i.e., share quarterly benchmarking reports with participating facilities so they can track their progress).
Offer webinars on the data collection elements required for the program, and provide sites with a data collection and submission guide with step-by-step instructions for reporting and submitting data.
The implementation advisers will review the data collection templates, data collection submission dates, and submission methods with the participating sites early in the process.
Provide timely responses to data collection inquiries and/or issues received from sites to the dedicated Help Desk email inbox.
Prompt the site leaders for feedback from each participating unit regarding the data collection activities and MRSA-related performance, for early identification of potential issues.
Perform ongoing quality control review of data submitted and provide feedback to sites on data quality issues. Participating sites will be provided with instructions on how to correct data issues and resubmit data, if needed.
Provide sites with a secure portal to submit data. Users will receive a login and password to access the site.
Share data collection strategies/best practices to simplify processes through coaching calls
Similar to prior assessments of CUSP implementation, the AHRQ Safety Program for MRSA Prevention will use validated instruments when possible. All instruments are reviewed by the program TEP and other subject matter experts. The AHRQ Safety Program for MRSA Prevention TEP will provide ongoing recommendations for the best measures of implementation.
AHRQ Surveys of Patient Safety Culture. The AHRQ Surveys on Patient Safety Culture are publicly available and have been widely used to assess patient safety culture in a variety of health care settings. Psychometric analyses of the Hospital Survey on Patient Safety Culture (HSOPS) data from 331 U.S. hospitals found acceptable properties on 12 dimensions and 42 items at the individual, unit, and hospital levels.i A smaller study of 454 hospital staff found moderate-to-strong validity and reliability for all by one subscale.ii The initial pilot of the Nursing Home Survey of Patient Safety Culture (NHSOPS) concluded that this survey also has high reliability and factor structure.iii
Gap Analysis. The Gap Analysis has been developed specifically for this program to assess the status of MRSA prevention programs at the sites. It will address the need to understand the current status of each site, both at the start of the implementation period and endline.
Team Checkup Tool. The Team Checkup Tool (TCT) supports assessment of implementation progress, adherence to protocol, cultural change, and implementation effectiveness. The TCT has been used in several similar implementation studies, and has undergone psychometric assessment. Specifically, in a multi-centered clustered randomized controlled trial of a team-based QI intervention conducted at 46 ICUs, the TCT demonstrated temporal stability, construct validity, and measure responsiveness.iv In a more recent study of the TCT for measuring implementation of QI activities, Marsteller et al. calculated high item-level and scale-level content validity using the content validity index.v
Johns Hopkins University and NORC at the University of Chicago will serve as the primary consultants for statistical aspects of the design and analysis of the evaluation data.
Exhibit B.5: List of Statistical Consultants
Name |
Title and Institution |
Telephone Number |
Roy Ahn, ScD, MPH |
Vice President, NORC |
312/759-4068 |
Erik Scherpf, Ph.D |
Senior Research Methodologist, NORC |
301/634-9437 |
Yea-Jen Hsu, Ph.D |
Assistant Scientist, Johns Hopkins University |
443/540-0957 |
The data will be collected by NORC at the University of Chicago.
1 Morrill HJ, Caffrey AR, Gaitanis MM, LaPlante KL. Impact of a prospective audit and feedback antimicrobial stewardship program at a Veterans Affairs medical center: a six-point assessment. PLoS One. 2016;11(3):e0150795.
2 For the power analysis, we assume that the standard deviation is the same for the baseline and post-implementation periods. Based on facility-level data on hospital-associated MRSA infections from the Centers for Medicare and Medicaid Services (CMS) Hospital Compare database, we set the value of the standard deviation at 6 for MRSA infections per 100,000 patient-days (MRSA cases per 100,000 patient days had a mean of 3.14 and standard deviation of 5.8 using this data). Since the data are at the facility level, we could not distinguish between differences in this measure between the ICU and non-ICU settings.
3 With higher frequency data collection (e.g., monthly data), the minimum detectable difference in the power analysis results constitute an upper bound and more observations improve power slightly.
4 We assume a 1-sided significance level given that we expect the intervention would improve the outcome (e.g., a decrease in infections). However, we also presented the result with an assumption of 2-sided significance level as a standard assumption.
5 Campbell MK, Mollison J, Grimshaw JM. Cluster trials in implementation research: estimation of intracluster correlation coefficients and sample size. Stat Med. 2001;20(3):391-9. In this reference, within-patient correlation is estimated as ICC = 0.47 (95% CI 0.29 to 0.65). Based on this reference, we assumed the within-unit correlation as 0.4 and range from 0.2 to 0.6.
6 Simulations showed that an ITS design with 3 pre-implementation periods and powered at 80 percent would, with a cluster size of 4 (implying 100 clusters), be able to detect a 1.36 unit change in the outcome level mean due to intervention; with a cluster size of 2, it would be able to detect a 1.09 unit change in the outcome level mean. With no clustering of units within hospitals, the ITS design would be able to detect a .925 unit change in the level mean. These results are for two-sided tests with a 0.05 significant level. The simulation assumed the following pattern of correlation of outcomes across time: 0.5 for measures one-period apart, 0.4 for measures two periods apart, and 0.3 for measures 3 periods apart.
7 The power analysis for transfers of LTC residents to an acute care hospital due to suspected or confirmed infection is informed by the Centers for Medicare and Medicaid Services Skilled Nursing Facility 30-day All-Cause Readmission Measure (SNFRM), which estimates the rate of all-cause, unplanned hospital readmissions.
i Sorra, J. S., & Dyer, N. (2010). Multilevel psychometric properties of the AHRQ hospital survey on patient safety culture. BMC health services research, 10(1), 199. https://doi.org/10.1186/1472-6963-10-199
ii Blegen, M. A., Gearhart, S., O'Brien, R., Sehgal, N. L., & Alldredge, B. K. (2009). AHRQ's hospital survey on patient safety culture: psychometric analyses. Journal of patient safety, 5(3), 139-144. DOI: 10.1097/PTS.0b013e3181b53f6e
iii Sorra, J., Carpenter, J., Streagle, S., et al. Pilot testing and psychometric analysis of the Nursing Home Survey on Resident Safety. Rockville, MD: Agency for Healthcare Research and Quality, 2008.
iv Chan, K. S., Hsu, Y. J., Lubomski, L. H., & Marsteller, J. A. (2011). Validity and usefulness of members reports of implementation progress in a quality improvement initiative: findings from the Team Check-up Tool (TCT). Implementation science, 6(1), 115. doi: 10.1186/1748-5908-6-115
v Marsteller, J. A., Hsu, Y. J., Chan, K. S., & Lubomski, L. H. (2017). Assessing content validity and user perspectives on the Team Check-up Tool: expert survey and user focus groups. BMJ quality & safety, 26(4), 288-295. DOI: 10.1136/bmjqs-2015-004635
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | SChowdhury |
| File Modified | 0000-00-00 |
| File Created | 2021-07-27 |