Cms-10102 Supporting Statement B -

The Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey is administered to a random sample of adult inpatients between 48 hours and six weeks after discharge.

Patients admitted in the medical, surgical, and maternity care service lines are eligible for the survey. HCAHPS is not restricted to Medicare patients. Hospitals may use an approved survey vendor or collect their own HCAHPS data, if approved by CMS to do so. HCAHPS can be implemented in four survey modes: mail, telephone, mail with telephone follow-up, or active interactive voice recognition (IVR), each of which requires multiple attempts to contact patients. Hospitals must survey patients throughout each month of the year. IPPS hospitals must achieve at least 300 completed surveys over four calendar quarters. HCAHPS is available in official English, Spanish, Chinese, Russian, Vietnamese, Portuguese, German, Tagalog, and Arabic versions. The HCAHPS Survey sampling protocol promotes the following: 1) standardized administration of the HCAHPS Survey by hospital/survey vendors and 2) comparability of resulting data across all participating hospitals. The survey and its protocols for sampling, data collection, coding and submission can be found in the HCAHPS Quality Assurance Guidelines manual (Version 18.0, March 2023) on the official HCAHPS On-Line Web site, www.hcahpsonline.or g.

1. Respondent Universe and Sampling

The HCAHPS Survey is broadly intended for patients of all payer types who meet the following criteria:

Eighteen (18) years or older at the time of admission
Admission includes at least one overnight stay in the hospital
Non-psychiatric MS-DRG/principal diagnosis at discharge
Alive at the time of discharge

There are a few categories of otherwise eligible patients who are excluded from the sample frame. These are:

“No-Publicity” patients – Patients who request that they not be contacted
Court/Law enforcement patients (i.e., prisoners); patients residing in halfway houses are included ➢ Patients with a foreign home address (U.S. territories – Virgin Islands, Puerto Rico, Guam, American Samoa, and Northern Mariana Islands -- are not considered foreign addresses and are not excluded)
Patients discharged to hospice care (Hospice-home or Hospice-medical facility)
Patients who are excluded because of state regulations
Patients discharged to nursing homes and skilled nursing facilities

Hospitals/Survey vendors must retain documentation that verifies all exclusions and ineligible patients for a minimum of three years. This documentation is subject to review.

Hospitals/Survey vendors participating in HCAHPS are responsible for generating complete, accurate, and valid sample frame data files each month that contain all administrative information on all patients who meet the eligible population criteria. The following steps must be followed when creating the sample frame:

The sample frame for a particular month must include all eligible hospital discharges between the first and last days of the month (e.g., for January, any qualifying discharges between the 1st and 31st)
If a hospital is conducting sampling at the end of each month, they must create the sample frame in a timely manner in order to initiate contact for all sampled patients within 42 days of discharge
Patients with missing or incomplete addresses and/or telephone numbers must not be removed from the sample frame. Instead, every attempt must be made to find the correct address and/or telephone number
Patients whose eligibility status is uncertain must be included in the sample frame

The hospital/survey vendor must retain the sample frame (i.e., the entire list of eligible HCAHPS patients from which each hospital’s sample is pulled) for 3 years. Confidentiality note: Patient-identifying information within the sample frame will not be part of the final data submitted to CMS, nor will any other PHI.

Hospitals must submit at least 300 completed HCAHPS Surveys in a rolling four-quarter period (unless the hospital is too small to obtain 300 completed surveys). The absence of a sufficient number of HCAHPS eligible discharges is the only acceptable reason for submitting fewer than 300 completed HCAHPS Surveys in a rolling four-quarter period. In that not all sampled patients who are contacted to complete the survey will actually do so, guidance is provided hospitals/survey vendors as to how many discharges are needed to reach the required 300 completed surveys per four rolling quarters of data (a 12- month reporting period).

2. Data Collection Procedures

a. Statistical Methodology for Stratification and Sample Selection

The basic sampling procedure for HCAHPS entails drawing a random sample of all eligible discharges from a hospital on a monthly basis. Sampling may be conducted either continuously throughout the month or at the end of the month, as long as a random sample is generated from the entire month. The HCAHPS sample must be drawn according to this uninterrupted random sampling protocol and not according to any “quota” system. Hospitals/Survey vendors must sample from every month throughout the entire 12-month reporting period and not stop sampling or curtail ongoing survey administration activities even if 300 completed surveys have been attained.

Sampling for HCAHPS is based on the eligible discharges (HCAHPS sample frame) for a calendar month. If every eligible discharge for a given month has the same probability of being sampled, then an equiprobable approach is being used. Stratified sampling is where eligible discharges are divided into non-overlapping subgroups, referred to as strata, before sampling.

There are three options for sampling patients for the HCAHPS Survey: Simple Random Sampling (SRS),

Proportionate Stratified Random Sampling (PSRS), and Disproportionate Stratified Random Sampling

(DSRS). Once a sample type is used within a quarter it must be maintained throughout that quarter;

“Sample Type” can only be changed at the beginning of a quarter. For more information about HCAHPS sampling, please see HCAHPS Quality Assurance Guidelines, V18.0, pp. 61-84, at www.hcahpsonline.org/en/quality -assurance /.

SRS: Simple Random Sampling is the most basic sampling type; patients are randomly selected from all eligible discharges for a month. Strata are not used when employing SRS and each patient has equal opportunity of being selected into the sample, making SRS equiprobable. Census sampling is considered a form of simple random sampling.
PSRS: Proportionate Stratified Random Sampling uses strata definitions and random sample selection from all strata at equal rates. Since the sampling rates of the strata are “proportionate,” PSRS is also considered equiprobable.
DSRS: Disproportionate Stratified Random Sampling involves sampling within strata at different rates, and thus, DSRS requires information about the strata. By definition, DSRS is not an equiprobable sampling approach as DSRS allows for dissimilar sampling rates across strata. DSRS means that all eligible discharges do not have an equal chance of being selected for inclusion in the monthly sample. To account for this, CMS requires additional information from hospitals and survey vendors who choose to use DSRS as a sampling type. Hospitals/survey vendors must submit an Exceptions Request Form and then be approved to use DSRS.

b. Estimation Procedures.

Not applicable to the HCAHPS Survey.

c. Degree of Accuracy Needed for the Purpose Described in the Justification.

IPPS hospitals are expected to achieve at least 300 completed surveys over a 12 month period, if possible, to attain the desired degree of accuracy; please see below. HCAHPS scores based on fewer than 100 or 50 completed surveys are publicly reported but the lower reliability of these scores is noted with an appropriate footnote in public reporting. HCAHPS scores based on fewer than 25 completed surveys are not reported on the Care Compare Web site. However, these hospitals do receive their HCAHPS scores in their confidential Preview Reports. IPPS hospitals must achieve at least 100 completed HCAHPS surveys during the 12 month Performance Period in order for a Person and Community Engagement Domain score to be calculated in the Hospital Value-Based Purchasing program. The HCAHPS Quality Assurance Guidelines, V18.0, pp. 67-70, describe how to calculate the sample size necessary to attain at least 300 completed surveys, www.hcahpsonline.org/en/quality -assurance /. Elliot, et al. provide additional information about the validity and reliability of the HCAHPS Survey; see Elliott, Lehrman, et al. (2010). “Do Hospitals Rank Differently on HCAHPS for Different Patient Subgroups?” Medical Care Research and Review, 67(1):56- 73.

d. Unusual Problems Requiring Specialized Sampling Procedures.

Not applicable to the HCAHPS Survey.

3. Maximizing Response Rates/Non-response and Issues of Accuracy, Reliability, and Validity

Implementation of the HCAHPS Survey according to the protocols contained in the current HCAHPS Quality Assurance Guidelines, V18.0, helps hospital to attain the greatest response rate. CMS examines hospital response rates every quarter and works with hospitals and survey vendors whose response rate is significantly below the national average. Among other tactics, CMS strongly encourages hospitals to offer the HCAHPS Survey in the language spoken at home by their patients. Analysis of HCAHPS data indicates that the patient-mix adjustment applied to survey results adequately addresses the non-response bias that would exist without patient-mix adjustment; see “The Effects of Survey Mode, Patient Mix, and

Nonresponse on CAHPS Hospital Survey Scores.” Elliott, Zaslavsky et al. (2009) Health Services Research, 44 (2): 501-518.

Information on statistical tests of the original 27-item HCAHPS Survey can be found in the original 2006 OMB package. Here we focus on the most recent tests of the validity and reliability of the current, 29item HCAHPS Survey at the recommended 300 completed surveys in a 12-month public reporting period.

In terms of hospital-level reliability, the signal-to-noise ratio indicates how much of what you measure is

“signal” (true variation in performance), rather than “noise” (measurement error). Unit or hospital-level reliability (Spearman-Brown reliability) is the proportion of variance in hospital-level scores that reflect true variation among hospitals, not noise due to limited numbers of patient surveys; see Table 1.

Table 1: Hospital-Level Spearman-Brown Reliabilities of HCAHPS Top-Box Scores of HCAHPS Measures at 300 Completed Surveys, 2.3 million discharges, January – December 2022

Communication with Nurses	0.90
Communication with Doctors	0.89
Responsiveness of Hospital Staff	0.94
Communication about Medicines	0.86
Cleanliness	0.91
Quietness	0.93
Discharge Information	0.86
Overall Rating	0.93
Recommend Hospital	0.94
Care Transition	0.88

Generally accepted standards for reliability are: 0.7: Adequate (all HCAHPS measures exceed this); 0.8: Good (all exceed this); 0.85 Very good (all exceed this); and 0.9: Excellent (6 of 10 reach this level). It should be noted that at more than the recommended 300 completed surveys in 12 month reporting period, HCAHPS reliability is even higher.

Evidence of the internal consistency of the HCAHPS Survey can be found in Table 2, which displays the

Cronbach’s Alpha statistics for the six publicly reported HCAHPS composite measures, which are made up of two or three survey items.

Table 2: Cronbach’s Alphas of HCAHPS Composite Measures, 2.3 million discharges, January – December 2022

Communication with Nurses	0.78
Communication with Doctors	0.80
Responsiveness of Hospital Staff	0.68
Communication about Medicines	0.65
Discharge Information	0.45
Care Transition	0.82

Criterion validity indicates the extent to which HCAHPS measures what they purport to measure: patient experience of care. The patient-level correlation of specific HCAHPS measures with “Hospital Rating” and “Recommend the Hospital” can be used to assess validity (criterion validity); see Table 3. We expect moderate, positive correlations of the other eight HCAHPS measures with patient experience of care constructs. If the correlations were too high, this may indicate redundancy or halo effect. If too low, would indicate lack of validity. We find moderate, positive correlations between the eight specific HCAHPS measures and the two global measures (Rating and Recommendation). All 16 of these correlations are between 0.32 and 0.68; 13 of 16 correlations equal or exceed 0.35.

Table 3: HCAHPS Patient-level Correlations, 2.3 million discharges, July 2021 – June 2022

HCAHPS PATIENT-LEVEL CORRELATIONS *


Communication with Nurses	1	0.58	0.59	0.53	0.41	0.35	0.31	0.49	0.68	0.61
Communication with Doctors		1	0.42	0.47	0.31	0.29	0.32	0.46	0.56	0.52
Responsiveness of Hosp. Staff			1	0.45	0.37	0.35	0.24	0.40	0.55	0.49
Comm. About Medicines				1	0.36	0.31	0.38	0.50	0.51	0.46
Cleanliness of Hospital Env.					1	0.29	0.21	0.31	0.44	0.40
Quietness of Hospital Env.						1	0.15	0.28	0.38	0.32
Discharge Information							1	0.36	0.34	0.31
Care Transition								1	0.53	0.50
Hospital Rating									1	0.80
Recommend the Hospital										1

*Patient-level Pearson correlations of rescaled linear means of HCAHPS measures, for patients discharged between July 2021 and June 2022 (2.3 million completed surveys). Note: All correlations are significant at p<0.001.

*These results encompass all hospitals that received HCAHPS scores. Because not all hospitals report their results on Hospital Compare, the values on that Web site may differ from those shown here.

The correlations of the eight specific measures with the two global measures are larger at the hospital level than at the patient level, in part because of the better measurement at the hospital level (as intended) than at the patient level; see Table 4. These hospital level correlations ranged from 0.55 to 0.84, with none less than 0.50.

Table 4: Hospital Top-Box Correlations for HCAHPS Measures, January – December 2022 discharges (4,149 hospitals, 2.3 million surveys)

HCAHPS HOSPITAL LEVEL CORRELATIONS

	Nurse	Doctor	Staff	Rx Comm	Clean	Quiet	Discharge	CTM	Recommend	Rating
Nurse	1
Doctor	0.82	1
Staff	0.84	0.73	1
Rx Comm	0.78	0.73	0.75	1
Clean	0.67	0.58	0.67	0.59	1
Quiet	0.67	0.66	0.68	0.62	0.58	1
Discharge	0.62	0.53	0.52	0.54	0.42	0.33	1
CTM	0.79	0.74	0.69	0.73	0.59	0.62	0.63	1
Recommend	0.74	0.67	0.63	0.63	0.58	0.55	0.60	0.81	1
Rating	0.84	0.75	0.74	0.71	0.68	0.66	0.63	0.83	0.91	1

The goal of HCAHPS is to collect information from patients using a standardized, national survey and to present the information based on those surveys to consumers, providers and hospitals. One of the methodological issues associated with making comparisons between hospitals is the need to adjust appropriately for patient-mix differences. Patient-mix refers to patient characteristics that are not under the control of the hospital that may affect measures of patient experiences, such as demographic characteristics and health status. The basic goal of adjusting for patient-mix is to estimate how different hospitals would be rated if they all provided care to comparable groups of patients.

CMS applies patient-mix adjustment to control for patient characteristics that affect ratings and that are differentially distributed across hospitals. Most of the patient-mix items are included in the “About You” section of the instrument, while others are taken from administrative records. Based on the mode experiment, and consistent with previous studies of patient-mix adjustment in CAHPS and in previous hospital patient surveys, we employ the following variables in the patient-mix adjustment model:

Self-reported general health status (specified as a linear variable)
Self-reported mental or emotional health status (specified as a linear variable)
Education (specified as a linear variable)
Type of service (medical, surgical, or maternity care) by gender
Age (specified as a categorical variable)
Lag time between patient discharge and survey completion or disposition
Age by service line interaction
Language (English, Spanish, Chinese, and

Russian/Vietnamese/Portuguese/German/Tagalog/Arabic/Other) spoken at home

Once the data are adjusted for patient-mix, there is a fixed adjustment for each of the reported measures for mode of administration (discussed in detail below). The patient-mix adjustment employs a regression methodology, also referred to as covariance adjustment.

On the survey there are two items that capture the race and ethnicity of the respondent. These items are not included in the patient-mix adjustment model but are used as analytic variables to support the congressionally-mandated report: “The National Healthcare Quality and Disparities Report.” This report provides annual, national level breakdowns of HCAHPS scores by race and ethnicity. Many hospitals collect information on race and ethnicity through their administrative systems, but coding is not standard. Thus, it was determined that administrative data are not adequate to support the analyses needed for the reports and the items should be included in the questionnaire.

4. Tests of Procedures or Methods

Hospitals began using the HCAHPS Survey in 2006 under the auspices of the Hospital Quality

Alliance, a private/public partnership that includes CMS and the American Hospital Association, the

Federation of American Hospitals, and the Association of American Medical Colleges; the Joint Commission on Accreditation of Healthcare Organizations; the National Quality Forum; AARP; and the Agency for Healthcare Research and Quality (AHRQ).

Throughout the HCAHPS development process, CMS solicited and received a great deal of public input. As a result, the HCAHPS questionnaire and methodology went through several iterations prior to national implementation. The accumulated lessons learned from the pilot testing, public comments, input from stakeholders, numerous team discussions, and the National Quality Forum’s review and endorsement

(CBE #0166) through their consensus development process led to the national implementation in 2006 of the 27-item HCAHPS Survey and the HCAHPS data collection protocol that allows hospitals to integrate their own supplemental questions. The resulting core questionnaire is comprised of questions in several dimensions of primary importance to the target audience: doctor communication, responsiveness of hospital staff, cleanliness of the hospital environment, quietness of the hospital environment, nurse communication, pain management, communication about medicines, and discharge information. The HCAHPS Survey was re-endorsed by the consensus-based entity in 2012, 2015, and 2019.

Since national implementation began in 2006, CMS has continually refined and clarified HCAHPS survey protocols, created new translations of the mail version of the survey in Chinese, Russian, Vietnamese, Portuguese, German, Tagalog, and Arabic in addition to the original English and Spanish versions. In the telephone mode, the HCAHPS Survey is available in English, Spanish, Chinese, and Russian; in the active IVR mode, the HCAHPS Survey is available in both English and Spanish. CMS has added and removed survey items over time (currently the survey contains 29 items), annually updated the HCAHPS Quality Assurance Guidelines (currently version 18.0), improved the appearance and accessibility of HCAHPS results on the CMS public reporting Web site Care Compare: https://www.medicare.gov/care compare/ and made information about HCAHPS quickly and easily available through its official HCAHPS On-Line website, www.hcahpsonline.or g.The HCAHPS Project Team has published a number of analyses of HCAHPS results in peer-reviewed scientific journals and made numerous presentations at professional conferences; a bibliography of the publications can be found at:

https://hcahpsonline.org/globalassets/hcahps/home/bibliography_feb_2020.pd f .

Since public reporting of hospitals HCAHPS results was inaugurated in March 2008, a growing number of healthcare, consumer and professional organizations, state governments, media outlets and others have adopted or incorporated HCAHPS scores, in part or in whole, for their own purposes. These activities, external to CMS, have had the effect of extending knowledge about HCAHPS and increasing the impact of survey results. The content of the HCAHPS Survey, its methodology and administration protocols, and its ambition to measure and publicly report consumers’ experiences in a uniform and standardized manner have influenced other surveys developed within CMS as well as those undertaken by hospitals and healthcare systems in the United States and abroad.

There are distinct roles for hospitals or their survey vendors and the federal government in the national implementation of HCAHPS. The federal government is responsible for support and public reporting, including:

conducting training on data collection and submission procedures
providing on-going technical assistance
ensuring the integrity of data collection
accumulating HCAHPS data from individual hospitals
producing patient-mix adjusted hospital-level estimates
conducting research on the presentation of data for public reporting
publicly reporting the comparative hospital data

Hospitals or their survey vendors are responsible for data collection, including: developing a sampling frame of relevant discharges, drawing the sample of discharges to be surveyed, collecting survey data from sampled discharges, and submitting HCAHPS data to CMS in a standard format. We have formatted the data files so hospitals/vendors will submit to CMS de-identified data files following 45 CFR Section §164.514. Hospitals maintain business associate agreements with their contracted survey vendors to collect and submit HCAHPS survey data through the secure QualityNet Exchange portal and data warehouse.

CMS began its collaboration with the Health Services Advisory Group (HSAG) in 2003 to coordinate the national implementation of the Hospital CAHPS Survey. HSAG’s role is to provide technical assistance and training for vendors and hospitals, data validation, data processing, analysis, and adjustment, and oversight of self-administering hospitals and survey vendors. HSAG also produces electronic data files and a hospital level extract file for public reporting of the HCAHPS scores.

5. Names and telephone numbers of individuals consulted on statistical aspects of the survey and implementation design and names of agency units, contractor(s), grantee(s), or other persons(s) who collect and/or analyze the information for the agency

Marc N. Elliott, PhD

Distinguished Chair in Statistics; Senior Principal Researcher

RAND Corporation

(310) 393-0411, x7931 [email protected]

Christopher Cohea, MS

Director, Research and Analysis Health Services Advisory Group, Inc.

(602) 471-3673 [email protected]

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Justification of the Hospital CAHPS Survey
Author	CMS
File Modified	0000-00-00
File Created	2023-12-12