Cms-10102 Supporting Statement B - 03-07-24

The Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey is administered to a random sample of adult inpatients between 48 hours and six weeks after discharge. Patients admitted in the medical, surgical, and maternity care service lines are eligible for the survey. HCAHPS is not restricted to Medicare patients. Hospitals may use an approved survey vendor or collect their own HCAHPS data, if approved by CMS to do so.

Starting with January 2025 hospital discharges, the FY 2025 IPPS/LTCH PPS proposed rule is proposing the use of an updated version of the HCAHPS Survey. The proposed updates to the HCAHPS Survey would result in a survey of 32 items that would form 11 measures, with seven composite multi-item measures and four single-item measures for the Hospital IQR and PCHQR Programs. Seven measures, four multi-item composite measures and three single-item measures from the current HCAHPS Survey, would remain unchanged in the HCAHPS Survey. In addition, the HCAHPS Survey will include three new measures (two multi-item measures and one single-item measure) and one revised multi-item measure.

The need for these proposed updates to the HCAHPS Survey was identified by the measure developer through literature review, focus groups and cognitive interviews with patients and caregivers, and discussions with technical experts. The proposed updated content to the HCAHPS Survey was empirically tested in a large-scale mode experiment in 2021, as described in the final FY 2024 IPPS/LTCH PPS final rule (88 FR 59196 through 59197). The measure developer has determined that these proposed updates to the HCAHPS Survey will amount to a minimal change in burden because the combination of removal and addition of survey items will result in only an additional 1.0 minute to complete the survey. The time required to complete the 32-item survey is estimated to average eight minutes. Additionally, we note that these proposed updates resulting in a 32-item survey is a return to an earlier item count

and should not represent an undue increased burden for patients or survey administrators given that before the “Communication About Pain” questions were removed in the CY 2019 OPPS/ASC Final Rule (83 FR 59140 through 59149), the HCAHPS Survey also consisted of 32 items.

HCAHPS can be currently implemented in four survey modes: mail, telephone, mail with telephone follow-up, or active interactive voice recognition (IVR), each of which requires multiple attempts to contact patients. Per the 2024 IPPS Rule, beginning in January 2025 hospitals can also administer their survey by web with mail follow-up, web with telephone follow-up, or web with mail and telephone follow-up. The IVR survey mode will be dropped in 2025.

Hospitals must survey patients throughout each month of the year. IPPS hospitals must achieve at least 300 completed surveys over four calendar quarters. HCAHPS is available in official English, Spanish, Chinese, Russian, Vietnamese, Portuguese, German, Tagalog, and Arabic versions. CMS will add additional translations over time as needed. The HCAHPS Survey sampling protocol promotes the following: 1) standardized administration of the HCAHPS Survey by hospital/survey vendors, and 2) comparability of resulting data across all participating hospitals. The FY 2024 IPPS/LTCH PPS final rule describes the most recent updates to HCAHPS survey administration requirements and additional background information for the Hospital VBP Program, the Hospital IQR Program, and the PCHQR Program (88 FR 59083 through 59089, 88 FR 59196 through 59201, and 88 FR 59229 through 59232, respectively). For more details including information about patient eligibility for the HCAHPS survey, please refer to the current HCAHPS Quality Assurance Guidelines, which can be found on the official HCAHPS On-Line Web site at: https://hcahpsonline.org/en/quality-assurance/.

Respondent Universe and Sampling

The HCAHPS Survey is broadly intended for patients of all payer types who meet the following criteria:

Eighteen (18) years or older at the time of admission
Admission includes at least one overnight stay in the hospital
Non-psychiatric MS-DRG/principal diagnosis at discharge
Alive at the time of discharge

There are a few categories of otherwise eligible patients who are excluded from the sample frame. These are:

“No-Publicity” patients – Patients who request that they not be contacted
Court/Law enforcement patients (i.e., prisoners); patients residing in halfway houses are included
Patients with a foreign home address (U.S. territories – Virgin Islands, Puerto Rico, Guam, American Samoa, and Northern Mariana Islands -- are not considered foreign addresses and are not excluded)
Patients discharged to hospice care (Hospice-home or Hospice-medical facility)
Patients who are excluded because of state regulations
Patients discharged to nursing homes and skilled nursing facilities

Hospitals/Survey vendors must retain documentation that verifies all exclusions and ineligible patients for a minimum of three years. This documentation is subject to review.

Hospitals/Survey vendors participating in HCAHPS are responsible for generating complete, accurate, and valid sample frame data files each month that contain all administrative information on all patients who meet the eligible population criteria. The following steps must be followed when creating the sample frame:

The sample frame for a particular month must include all eligible hospital discharges between the first and last days of the month (e.g., for January, any qualifying discharges between the 1st and 31st)
If a hospital is conducting sampling at the end of each month, they must create the sample frame in a timely manner in order to initiate contact for all sampled patients within 42 days of discharge
Patients with missing or incomplete addresses and/or telephone numbers must not be removed from the sample frame. Instead, every attempt must be made to find the correct address and/or telephone number
Patients whose eligibility status is uncertain must be included in the sample frame

The hospital/survey vendor must retain the sample frame (i.e., the entire list of eligible HCAHPS patients from which each hospital’s sample is pulled) for 3 years. Confidentiality note: Patient-identifying information within the sample frame will not be part of the final data submitted to CMS, nor will any other protected health information.

Hospitals must submit at least 300 completed HCAHPS Surveys in a rolling four-quarter period (unless the hospital is too small to obtain 300 completed surveys). The absence of a sufficient number of HCAHPS eligible discharges is the only acceptable reason for submitting fewer than 300 completed HCAHPS Surveys in a rolling four-quarter period. In that not all sampled patients who are contacted to complete the survey will actually do so, guidance is provided hospitals/survey vendors as to how many discharges are needed to reach the required 300 completed surveys per four rolling quarters of data (a 12- month reporting period).

Data Collection Procedures

Statistical Methodology for Stratification and Sample Selection

The basic sampling procedure for HCAHPS entails drawing a random sample of all eligible discharges from a hospital on a monthly basis. Sampling may be conducted either continuously throughout the month or at the end of the month, as long as a random sample is generated from the entire month. The HCAHPS sample must be drawn according to this uninterrupted random sampling protocol and not according to any “quota” system. Hospitals/Survey vendors must sample from every month throughout the entire 12-month reporting period and not stop sampling or curtail ongoing survey administration activities even if 300 completed surveys have been attained.

Sampling for HCAHPS is based on the eligible discharges (HCAHPS sample frame) for a calendar month. If every eligible discharge for a given month has the same probability of being sampled, then an equiprobable approach is being used. Stratified sampling is where eligible discharges are divided into non-overlapping subgroups, referred to as strata, before sampling.

There are three options for sampling patients for the HCAHPS Survey: Simple Random Sampling (SRS), Proportionate Stratified Random Sampling (PSRS), and Disproportionate Stratified Random Sampling (DSRS). Once a sample type is used within a quarter it must be maintained throughout that quarter; “Sample Type” can only be changed at the beginning of a quarter. For more information about HCAHPS sampling, please see HCAHPS Quality Assurance Guidelines, V18.0, pp. 61-84, at www.hcahpsonline.org/en/quality-assurance/.

SRS: Simple Random Sampling is the most basic sampling type; patients are randomly selected from all eligible discharges for a month. Strata are not used when employing SRS and each patient has equal opportunity of being selected into the sample, making SRS equiprobable. Census sampling is considered a form of simple random sampling.
PSRS: Proportionate Stratified Random Sampling uses strata definitions and random sample selection from all strata at equal rates. Since the sampling rates of the strata are “proportionate,” PSRS is also considered equiprobable.
DSRS: Disproportionate Stratified Random Sampling involves sampling within strata at different rates, and thus, DSRS requires information about the strata. By definition, DSRS is not an equiprobable sampling approach as DSRS allows for dissimilar sampling rates across strata. DSRS means that all eligible discharges do not have an equal chance of being selected for inclusion in the monthly sample. To account for this, CMS requires additional information from hospitals and survey vendors who choose to use DSRS as a sampling type. Hospitals/survey vendors must submit an Exceptions Request Form and then be approved to use DSRS.

Estimation Procedures.

Not applicable to the HCAHPS Survey.

Degree of Accuracy Needed for the Purpose Described in the Justification.

IPPS hospitals are expected to achieve at least 300 completed surveys over a 12 month period, if possible, to attain the desired degree of accuracy; please see below. HCAHPS scores based on fewer than 100 or 50 completed surveys are publicly reported but the lower reliability of these scores is noted with an appropriate footnote in public reporting. HCAHPS scores based on fewer than 25 completed surveys are not reported on the Care Compare Web site. However, these hospitals do receive their HCAHPS scores in their confidential Preview Reports. IPPS hospitals must achieve at least 100 completed HCAHPS surveys during the 12 month Performance Period in order for a Person and Community Engagement Domain score to be calculated in the Hospital Value-Based Purchasing program. The HCAHPS Quality Assurance Guidelines, V18.0, pp. 67-70, describe how to calculate the sample size necessary to attain at least 300 completed surveys, www.hcahpsonline.org/en/quality-assurance/. Additional information about the validity and reliability of the HCAHPS Survey is provided in Elliott, Lehrman, et al. (2010). “Do Hospitals Rank Differently on HCAHPS for Different Patient Subgroups?” Medical Care Research and Review, 67(1):56- 73.

Unusual Problems Requiring Specialized Sampling Procedures.

Not applicable to the HCAHPS Survey.

Maximizing Response Rates/Non-response and Issues of Accuracy, Reliability, and Validity

Implementation of the HCAHPS Survey according to the protocols contained in the current HCAHPS Quality Assurance Guidelines, V18.0, helps hospitals to attain the greatest response rate. CMS examines hospital response rates every quarter and works with hospitals and survey vendors whose response rate is significantly below the national average. Among other tactics, CMS strongly encourages hospitals to offer the HCAHPS Survey in the language spoken at home by their patients. Analysis of HCAHPS data indicates that the patient-mix adjustment applied to survey results adequately addresses the non-response bias that would exist without patient-mix adjustment; see “The Effects of Survey Mode, Patient Mix, and Nonresponse on CAHPS Hospital Survey Scores.” Elliott, Zaslavsky et al. (2009) Health Services Research, 44 (2): 501-518.

Information on statistical tests of the original 27-item HCAHPS Survey can be found in the original 2006 OMB package. Here we focus on the most recent tests of the validity and reliability of the currently implemented 29-item HCAHPS Survey (at the recommended 300 completed surveys in a 12-month public reporting period) for seven HCAHPS measures that will remain unchanged on the updated HCAHPS Survey . For the three new and one revised measures that will first appear on the updated HCAHPS Survey in 2025, the tests for validity and reliability are from the 2021 HCAHPS mode experiment.

In terms of hospital-level reliability, the signal-to-noise ratio indicates how much of what you measure is “signal” (true variation in performance), rather than “noise” (measurement error). Unit or hospital-level reliability (Spearman-Brown reliability) is the proportion of variance in hospital-level scores that reflect true variation among hospitals, not noise due to limited numbers of patient surveys; see Table 1.

Table 1: Hospital-Level Spearman-Brown Reliabilities of HCAHPS Linear-Mean Scores of HCAHPS Measures at 300 Completed Surveys, 2.3 million discharges, January – December 2022

Communication with Nurses	0.90
Communication with Doctors	0.89
Communication about Medicines	0.89
Cleanliness	0.91
Discharge Information	0.86
Overall Rating	0.94
Recommend Hospital	0.94
Responsiveness of Hospital Staff ¹	0.78
Care Coordination ¹	0.79
Restfulness of Hospital Environment ¹	0.87
Information about Symptoms ¹	0.71

¹Estimaes based on the 9,642 respondents to the 2021 HCAHPS Mode Experiment. Evidence from prior mode experiments suggest that reliability estimates from mode experiment tend to be lower than what is subsequently found in implementation data.

Generally accepted standards for reliability are: 0.7: Adequate (all 11 HCAHPS measures exceed this); 0.8: Good (eight measures exceed this); 0.85: Very good (eight measures exceed this); and 0.9: Excellent (four measures reach this level). It should be noted that at more than the recommended 300 completed surveys in 12 month reporting period, HCAHPS reliability is even higher.

Evidence of the internal consistency of the updated HCAHPS Survey can be found in Table 2, which displays the Cronbach’s Alpha statistics for the seven HCAHPS measures that are comprised of two or three survey items. Four of these multi-item composite measures are from the current HCAHPS Survey and three are new or revised measures.

Table 2: Cronbach’s Alphas of HCAHPS Composite Measures, 2.3 million discharges, January – December 2022

Communication with Nurses	0.84
Communication with Doctors	0.87
Communication about Medicines	0.67
Discharge Information	0.45
Responsiveness of Hospital Staff ¹	0.75
Care Coordination ¹	0.77
Restfulness of Hospital Environment ¹	0.74

¹ Estimates based on the 9,642 respondents to the 2021 HCAHPS Mode Experiment.

Criterion validity indicates the extent to which the HCAHPS Survey measures what it purports to measure: patient experience of care. The patient-level correlation of specific HCAHPS measures with the “Hospital Rating” and “Recommend the Hospital” single-item global measures can be used to assess validity (criterion validity); see Table 3. We expect moderate, positive correlations of the other nine measures in the updated HCAHPS Survey with these two global patient experience of care constructs. If the correlations were too high, this may indicate redundancy or halo effect. If too low, would indicate lack of validity. We find moderate, positive correlations between the nine specific HCAHPS measures in the updated HCAHPS Survey and the two global measures (Hospital Rating and Recommend the Hospital). All 18 of these correlations are between 0.31 and 0.68; 16 of 18 correlations exceed 0.35.

Table 3: HCAHPS Patient-level Correlations*, 2.3 million discharges, July 2021 – June 2022

	Hospital Rating	Recommend the Hospital
Communication with Nurses	0.68	0.61
Communication with Doctors	0.56	0.52
Communication about Medicines	0.51	0.46
Cleanliness	0.45	0.40
Discharge Information	0.34	0.31
Responsiveness of Hospital Staff¹	0.61	0.53
Care Coordination¹	0.67	0.62
Restfulness of Hospital Environment¹	0.55	0.48
Information about Symptoms¹	0.49	0.45

¹ Estimates based on the 9,642 respondents to the 2021 HCAHPS Mode Experiment.

*Patient-level Pearson correlations of rescaled linear means of HCAHPS measures for patients discharged between July 2021 and June 2022 (2.3 million completed surveys), unless otherwise noted. Note: All correlations are significant at p<0.001.

The correlations of the nine specific measures with the two global measures are larger at the hospital level than at the patient level, in part because of the better measurement at the hospital level (as intended) than at the patient level; see Table 4. These hospital level correlations ranged from 0.48 to 0.88.

Table 4: Hospital Correlations for HCAHPS Measures, January – December 2022 discharges (4,424 hospitals, 2.3 million surveys)

	Hospital Rating	Recommend the Hospital
Communication with Nurses	0.88	0.82
Communication with Doctors	0.80	0.75
Communication about Medicines	0.78	0.72
Cleanliness	0.68	0.61
Discharge Information	0.66	0.66
Responsiveness of Hospital Staff¹	0.78	0.71
Care Coordination¹	0.84	0.78
Restfulness of Hospital Environment¹	0.64	0.50
Information about Symptoms¹	0.56	0.48

¹ Estimates based on the 9,642 respondents to the 2021 HCAHPS Mode Experiment.

The goal of HCAHPS is to collect information from patients using a standardized, national survey and to present the information based on those surveys to consumers, providers, and hospitals. One of the methodological issues associated with making comparisons between hospitals is the need to adjust appropriately for patient-mix differences. Patient-mix refers to patient characteristics that are not under the control of the hospital that may affect measures of patient experiences, such as demographic characteristics and health status. The basic goal of adjusting for patient-mix is to estimate how different hospitals would be rated if they all provided care to comparable groups of patients.

CMS applies patient-mix adjustment to control for patient characteristics that affect ratings and that are differentially distributed across hospitals. Most of the patient-mix items are included in the “About You” section of the instrument, while others are taken from administrative records. Based on the 2021 mode experiment, and consistent with previous studies of patient-mix adjustment in CAHPS and in previous hospital patient surveys, we will employ the following variables in the patient-mix adjustment model:

Self-reported general health status (specified as a linear variable)
Self-reported mental or emotional health status (specified as a linear variable)
Education (specified as a linear variable)
Service line (male medical, female medical, male surgical, female surgical, or maternity care)
Age (specified as a categorical variable)
Lag time between patient discharge and survey completion or disposition
Age by service line interaction
Language spoken at home (English, Spanish, Chinese, and Russian/Vietnamese/Portuguese/German/Tagalog/Arabic/Other)
Planned hospital stay (indicator of “Yes, definitely” vs. other responses)

Once the data are adjusted for patient-mix, there is a fixed adjustment for each of the reported measures for mode of administration (discussed in detail below). The patient-mix adjustment employs a regression methodology, also referred to as covariance adjustment.

On the survey there are two items that capture the race and ethnicity of the respondent. These items are not included in the patient-mix adjustment model but are used as analytic variables to support the congressionally-mandated report, “The National Healthcare Quality and Disparities Report.” This report provides annual, national level breakdowns of HCAHPS scores by race and ethnicity. Many hospitals collect information on race and ethnicity through their administrative systems, but coding is not standard. Thus, it was determined that administrative data are not adequate to support the analyses needed for the reports and the items should be included in the survey.

Tests of Procedures or Methods

Hospitals began using the HCAHPS Survey in 2006 under the auspices of the Hospital Quality Alliance, a private/public partnership that includes CMS and the American Hospital Association, the

Federation of American Hospitals, and the Association of American Medical Colleges; the Joint Commission on Accreditation of Healthcare Organizations; the National Quality Forum; AARP; and the Agency for Healthcare Research and Quality (AHRQ).

Throughout the HCAHPS development process, CMS solicited and received a great deal of public input. As a result, the HCAHPS questionnaire and methodology went through several iterations prior to national implementation. The accumulated lessons learned from the pilot testing, public comments, input from stakeholders, numerous team discussions, and the National Quality Forum’s review and endorsement (CBE #0166) through their consensus development process led to the national implementation in 2006 of the 27-item HCAHPS Survey and the HCAHPS data collection protocol that allows hospitals to integrate their own supplemental questions. The resulting core questionnaire was comprised of questions in several dimensions of primary importance to the target audience: communication with doctors and nurses, responsiveness of hospital staff, cleanliness of the hospital environment, quietness of the hospital environment, nurse communication, pain management, communication about medicines, and discharge information. The HCAHPS Survey was re-endorsed by the consensus-based entity in 2012, 2015, and 2019. The updated HCAHPS survey has been approved by the 2023-2024 Measures Under Consideration process for review by the Measures Application Partnership, a multi-stakeholder partnership that provides recommendations to HHS on the selection of quality and efficiency measures for CMS programs.

Since national implementation began in 2006, CMS has continually refined and clarified HCAHPS survey protocols, and created new translations of the survey in Chinese, Russian, Vietnamese, Portuguese, German, Tagalog, and Arabic in addition to the original English and Spanish versions. CMS has annually updated the HCAHPS Quality Assurance Guidelines (currently version 18.0), improved the appearance and accessibility of HCAHPS results on the CMS public reporting Web site Care Compare (https://www.medicare.gov/care-compare/), and made information about HCAHPS quickly and easily available through its official HCAHPS On-Line website, www.hcahpsonline.org ., including aggregate state and national scores, scores for individual survey items, coefficients for patient-mix and survey mode adjustment, and benchmarks for the service lines, as well as the survey instrument, training materials, and a series of user-friendly podcasts to further understanding of HCAHPS content, implementation, adjustment and scoring. The HCAHPS Project Team has published a number of analyses of HCAHPS results in peer-reviewed scientific journals and made numerous presentations at professional conferences; a bibliography of the publications can be found at: https://hcahpsonline.org/#HCAHPSPublications. Other researchers have also analyzed HCAHPS data; the PubMed Web site (https://pubmed.ncbi.nlm.nih.gov/) lists over 500 published articles that employ HCAHPS data.

Since public reporting of hospitals HCAHPS results was inaugurated in March 2008, a growing number of healthcare, consumer and professional organizations, state governments, media outlets and others have adopted or incorporated HCAHPS scores, in part or in whole, for their own purposes. These activities, external to CMS, have had the effect of extending knowledge about HCAHPS and increasing the impact of survey results. The content of the HCAHPS Survey, its methodology and administration protocols, and its ambition to measure and publicly report consumers’ experiences in a uniform and standardized manner have influenced other surveys developed within CMS as well as those undertaken by hospitals and healthcare systems in the United States and abroad.

There are distinct roles for hospitals or their survey vendors and the federal government in the national implementation of HCAHPS. The federal government is responsible for support and public reporting, including:

Conducting training on data collection and submission procedures
Providing on-going technical assistance
Ensuring the integrity of data collection
Accumulating HCAHPS data from individual hospitals
Producing patient-mix adjusted hospital-level estimates
Conducting research on the presentation of data for public reporting
Publicly reporting the comparative hospital data

Hospitals or their survey vendors are responsible for data collection, including developing a sampling frame of relevant discharges, drawing the sample of discharges to be surveyed, collecting survey data from sampled discharges, and submitting HCAHPS data to CMS in a standard format. We have formatted the data files so hospitals/vendors will submit to CMS de-identified data files following 45 CFR Section §164.514. Hospitals maintain business associate agreements with their contracted approved survey vendors to collect and submit HCAHPS survey data through the secure QualityNet Exchange portal and data warehouse.

CMS began its collaboration with the Health Services Advisory Group (HSAG) in 2003 to coordinate the national implementation of the Hospital CAHPS Survey. HSAG’s role is to provide technical assistance and training for vendors and hospitals, data validation, data processing, analysis, and adjustment, and oversight of self-administering hospitals and survey vendors. HSAG also produces electronic data files and a hospital level extract file for public reporting of the HCAHPS scores.

Names and telephone numbers of individuals consulted on statistical aspects of the survey and implementation design and names of agency units, contractor(s), grantee(s), or other persons(s) who collect and/or analyze the information for the agency

Marc N. Elliott, PhD

Distinguished Chair in Statistics; Senior Principal Researcher

RAND Corporation

(310) 393-0411, x7931 [email protected]

Christopher Cohea, MS

Director, Research and Analysis

Health Services Advisory Group, Inc.

(602) 471-3673 [email protected]

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Justification of the Hospital CAHPS Survey
Author	CMS
File Modified	0000-00-00
File Created	2024-09-06

Cms-10102 Supporting Statement B - 03-07-24

CMS-10102 SUPPORTING STATEMENT B - 03-07-24.docx

National Implementation of Hospital Consumer Assessment of Health Providers and Systems (HCAHPS) (CMS-10102)

OMB: 0938-0981

TABLE OF CONTENTS

Introduction

Respondent Universe and Sampling

Data Collection Procedures

Statistical Methodology for Stratification and Sample Selection

Estimation Procedures.

Degree of Accuracy Needed for the Purpose Described in the Justification.

Unusual Problems Requiring Specialized Sampling Procedures.

Maximizing Response Rates/Non-response and Issues of Accuracy, Reliability, and Validity

Tests of Procedures or Methods

Names and telephone numbers of individuals consulted on statistical aspects of the survey and implementation design and names of agency units, contractor(s), grantee(s), or other persons(s) who collect and/or analyze the information for the agency