Dcapp, 3-31-15 -- Hcahps Supporting Statement B - 33015

April 2015

National Implementation of the Hospital Consumer
Assessment of Healthcare Providers and Systems
(HCAHPS) Survey
CMS 10102

OMB Supporting Statement - Part B

Prepared by
Division of Consumer Assessment & Plan Performance
Centers for Medicare & Medicaid Services
7500 Security Boulevard
Baltimore, MD 21244

1

TABLE OF CONTENTS
Section

Page

Introduction………………………………………………………………………………………….

3

1. Respondent Universe and Sampling…………………………………………………………

3

2. Data Collection Procedures………………………………………………………………….

5

a. Statistical Methodology for Stratification and Sample Selection………………………

5

b. Estimation Procedures………………………………………………………………….

6

c. Degree of Accuracy Needed for the Purpose Described in the Justification……………

6

d. Unusual Problems Requiring Specialized Sampling Procedures………………………..

7

e. Any Use of Periodic Data Collection Cycles to Reduce Burden………………………..

7

3. Maximizing Response Rates/Non-response and Issues of Accuracy,
Reliability and Validity...........................................................................................................

7

4. Past and Ongoing Tests of Procedures, Training, and Quality Improvement Activities;
Collaborator and Contractor Participation……………………………………………………

16

5. Names and Telephone Numbers of Individuals Consulted on Statistical Aspects
of the Survey and Implementation Design and Names of Agency Units, Contractor(s),
Grantee(s), or Other Persons(s) who Collect and/or Analyze the Information for the
Agency…………………………………………………………….…………………………

23

TABLES
Table 1. Number of Participating HCAHPS Hospital per Year……………………………………

5

Table 2. Number of Completed HCAHPS Surveys per Year……………………………………….

5

Table 3. HCAHPS Survey Response Rates per Year………………………………………………..

5

Table 4. Hospital-Level Spearman-Brown Reliabilities of HCAHPS Top-Box Scores at
300 Completed Surveys, 3.1 Million Discharges, 2013……………………………………

8

Table 5. Cronbach’s Alphas of HCAHPS Composite Measures at 300 Completed Surveys,
3.1 Million Discharges, 2013………………………………………………………………

9
1

Table 6. Top-box Correlations for HCAHPS Measures at 300 Completed Surveys,
3.1 Million Discharges, Patient-Level, 2013……………………………………………..

10

Table 7. Top-Box Correlations for HCAHPS Measures, ~4,000 Hospitals, Hospital Level,
2013………………………………………………………………………………………

10

Table 8. Average Top-Box Inter-Item Correlations for Individual HCAHPS Items,
3.1 Million Discharges, Patient Level, 2013………………………………………………

11

Table 9. HCAHPS Top-Box Scores, ~4,000 Hospitals, 3.1 Million Discharges, 2013…………….

12

Table 10. HCAHPS Survey Development and Implementation Timeline, 2002-2006……………..

17

LIST OF ATTACHMENTS
Attachment A -- HCAHPS Survey Instrument (Mail) and Supporting Material (included with Supporting
Statement - Part A)

2

OMB SUPPORTING STATEMENT – Part B:
National Implementation of the Hospital CAHPS Survey
CMS-10102
Introduction
The Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey is
administered to a random sample of adult inpatients between 48 hours and six weeks after discharge.
Patients admitted in the medical, surgical and maternity care service lines are eligible for the survey.
HCAHPS is not restricted to Medicare patients. Hospitals may use an approved survey vendor or collect
their own HCAHPS data, if approved by CMS to do so. HCAHPS can be implemented in four survey
modes: mail, telephone, mail with telephone follow-up, or active interactive voice recognition (IVR), each
of which requires multiple attempts to contact patients. Hospitals must survey patients throughout each
month of the year. IPPS hospitals must achieve at least 300 completed surveys over four calendar
quarters. HCAHPS is available in official English, Spanish, Chinese, Russian, Vietnamese, and
Portuguese versions. The HCAHPS Survey sampling protocol promotes the following: 1) standardized
administration of the HCAHPS Survey by hospital/survey vendors and 2) comparability of resulting data
across all participating hospitals. The survey and its protocols for sampling, data collection, coding and
submission can be found in the HCAHPS Quality Assurance Guidelines manual (Version 10.0, March
2015) on the official HCAHPS On-Line Web site, www.hcahpsonline.org.

1. Respondent Universe and Sampling
The HCAHPS Survey is broadly intended for patients of all payer types who meet the following criteria:





Eighteen (18) years or older at the time of admission
Admission includes at least one overnight stay in the hospital
Non-psychiatric MS-DRG/principal diagnosis at discharge
Alive at the time of discharge

There are a few categories of otherwise eligible patients who are excluded from the sample frame. These
are:
 “No-Publicity” patients – Patients who request that they not be contacted
 Court/Law enforcement patients (i.e., prisoners); patients residing in halfway houses are included,
 Patients with a foreign home address (U.S. territories – Virgin Islands, Puerto Rico, Guam,
American Samoa, and Northern Mariana Islands are not considered foreign addresses and are not
excluded)
 Patients discharged to hospice care (Hospice-home or Hospice-medical facility)
3

 Patients who are excluded because of state regulations
 Patients discharged to nursing homes and skilled nursing facilities
Hospitals/Survey vendors must retain documentation that verifies all exclusions and ineligible patients for
a minimum of three years. This documentation is subject to review.

Hospitals/Survey vendors participating in HCAHPS are responsible for generating complete, accurate,
and valid sample frame data files each month that contain all administrative information on all patients
who meet the eligible population criteria. The following steps must be followed when creating the sample
frame:
 The sample frame for a particular month must include all eligible hospital discharges between the
first and last days of the month (e.g., for January, any qualifying discharges between the 1st and
31st)
 If a hospital is conducting sampling at the end of each month, they must create the sample frame
in a timely manner in order to initiate contact for all sampled patients within 42 days of discharge
 Patients with missing or incomplete addresses and/or telephone numbers must not be removed
from the sample frame. Instead, every attempt must be made to find the correct address and/or
telephone number.
 Patients whose eligibility status is uncertain must be included in the sample frame
The hospital/survey vendor must retain the sample frame (i.e., the entire list of eligible HCAHPS patients
from which each hospital’s sample is pulled) for 3 years. Confidentiality note: Patient-identifying
information within the sample frame will not be part of the final data submitted to CMS, nor will any
other PHI.

Hospitals must submit at least 300 completed HCAHPS Surveys in a rolling four-quarter period (unless
the hospital is too small to obtain 300 completed surveys). The absence of a sufficient number of
HCAHPS eligible discharges is the only acceptable reason for submitting fewer than 300 completed
HCAHPS Surveys in a rolling four-quarter period. In that not all sampled patients who are contacted to
compete the survey will actually do so, guidance is provided hospitals/survey vendors as to how many
discharges are needed to reach the required 300 completed surveys per four rolling quarters of data (a 12month reporting period).

Information on the number of hospitals participating in the HCAHPS Survey, the number of completed
surveys, and the annual response rate, from 2008 through 2013, can be found in Table 1, Table 2, and
Table 3, respectively. Information for 2014 is not yet complete.
4

Table 1. Number of Hospitals Participating in the HCAHPS Survey per Year
2008

2009

2010

2011

2012

2013

3,791

3,811

3,850

3,887

3,938

4,093

Table 2. Number of Completed HCAHPS Surveys per Year
2008

2009

2010

2011

2012

2013

2,326,696

2,561,603

2,739,257

2,900,161

3,126,350

3,099,832

Table 3. HCAHPS Survey Response Rates per Year
2008

2009

2010

2011

2012

2013

33.5

33.0

32.8

32.2

33.0

32.3

2. Data Collection Procedures
a. Statistical Methodology for Stratification and Sample Selection.
The basic sampling procedure for HCAHPS entails drawing a random sample of all eligible discharges
from a hospital on a monthly basis. Sampling may be conducted either continuously throughout the
month or at the end of the month, as long as a random sample is generated from the entire month. The
HCAHPS sample must be drawn according to this uninterrupted random sampling protocol and not
according to any “quota” system. Hospitals/Survey vendors must sample from every month throughout
the entire 12-month reporting period and not stop sampling or curtail ongoing survey administration
activities even if 300 completed surveys have been attained.

Sampling for HCAHPS is based on the eligible discharges (HCAHPS sample frame) for a calendar
month. If every eligible discharge for a given month has the same probability of being sampled, then an
equiprobable approach is being used. Stratified sampling is where eligible discharges are divided into
non-overlapping subgroups, referred to as strata, before sampling.

There are three options for sampling patients for the HCAHPS Survey: Simple Random Sampling (SRS),
Proportionate Stratified Random Sampling (PSRS), and Disproportionate Stratified Random Sampling
5

(DSRS). Once a sample type is used within a quarter, it must be maintained throughout that quarter;
“Sample Type” can only be changed at the beginning of a quarter. For more information about HCAHPS
sampling, please see HCAHPS Quality Assurance Guidelines, V10.0, pp. 49-70, at
http://www.hcahpsonline.org/qaguidelines.aspx.
 SRS: Simple Random Sampling is the most basic sampling type; patients are randomly selected
from all eligible discharges for a month. Strata are not used when employing SRS and each
patient has equal opportunity of being selected into the sample, making SRS equiprobable. Census
sampling is considered a form of simple random sampling.
 PSRS: Proportionate Stratified Random Sampling uses strata definitions and random sample
selection from all strata at equal rates. Since the sampling rates of the strata are “proportionate,”
PSRS is also considered equiprobable.
 DSRS: Disproportionate Stratified Random Sampling involves sampling within strata at different
rates, and thus, DSRS requires information about the strata. By definition, DSRS is not an
equiprobable sampling approach as DSRS allows for dissimilar sampling rates across strata.
DSRS means that all eligible discharges do not have an equal chance of being selected for
inclusion in the monthly sample. To account for this, CMS requires additional information from
hospitals and survey vendors who choose to use DSRS as a sampling type. Hospitals/survey
vendors must submit an Exceptions Request Form and then be approved to use DSRS.

b. Estimation Procedures.
Not applicable to the HCAHPS Survey.

c. Degree of Accuracy Needed for the Purpose Described in the Justification.
IPPS hospitals are expected to achieve at least 300 completed surveys over a 12 month period, if possible,
to attain the desired degree of accuracy; please see below. HCAHPS scores based on fewer than 100 or
50 completed surveys are publicly reported but the lower reliability of these scores is noted with an
appropriate footnote in public reporting. IPPS hospitals must achieve at least 100 completed HCAHPS
surveys during the 12 month Performance Period in order for a Patient and Caregiver Centered
Experience of Care/Care Coordination domain score to be calculated in the Hospital Value-Based
Purchasing program. The HCAHPS Quality Assurance Guidelines, V10.0, pp. 54-57, describe how to
calculate the sample size necessary to attain at least 300 completed surveys,
http://www.hcahpsonline.org/qaguidelines.aspx. Elliot, et al. provide additional information about the
6

validity and reliability of the HCAHPS Survey; see Elliott, Lehrman, et al. (2010). “Do Hospitals Rank
Differently on HCAHPS for Different Patient Subgroups?” Medical Care Research and Review, 67(1):5673.
d. Unusual Problems Requiring Specialized Sampling Procedures.
CMS recognizes that some small hospitals may not have 300 HCAHPS-eligible discharges in the course
of a year and so are unable to obtain 300 completed surveys in a 12-month report period. In such cases,
IPPS hospitals must sample all eligible discharges (that is, conduct a census) and attempt to obtain as
many completes as possible

e. Any Use of Periodic Data Collection Cycles to Reduce Burden.
There is no use of periodic (less frequent than annual) data collection cycles for the HCAHPS Survey.
Great effort was expended considering how often HCAHPS data should be collected. We solicited and
received much comment on this issue when the HCAHPS Survey was being developed. Two options for
the frequency of data collection were suggested: once during the year or continuous sampling. The
majority of hospitals/vendors suggested continuous sampling would be easier to integrate into their
current data collection processes. Thus we decided to require sampling of discharges on a continuous
basis (i.e., a monthly basis) and cumulate these samples to create rolling estimates based on 12-months of
data. We chose to pursue the continuous sampling approach for the following reasons:






It is more easily integrated with many existing survey processes used for internal improvement,
Improvements in hospital care can be more quickly reflected in hospital scores (e.g., 12-month
estimates could be updated on a quarterly or semi-annual basis),
Hospital scores are less susceptible to unique events that could affect hospital performance at a
specific point in time,
It is less susceptible to gaming (e.g., hospitals being on their best behavior at the time of an annual
survey), and
There is less variation in time between discharge and data collection.

3. Maximizing Response Rates/Non-response and Issues of Accuracy, Reliability, and Validity
Implementation of the HCAHPS Survey according to the protocols contained in the current HCAHPS
Quality Assurance Guidelines, V10.0, helps hospital to attain the greatest response rate. CMS examines
hospital response rates every quarter and works with hospitals and survey vendors whose response rate is
significantly below the national average. Among other tactics, CMS strongly encourages hospitals to
offer the HCAHPS Survey in the language spoken at home by their patients (see 50966 Federal Register /
Vol. 78, No. 160 / Monday, August 19, 2013 / Rules and Regulations, pp. 325-326).
7

Analysis of HCAHPS data indicates that the patient-mix adjustment applied to survey results adequately
addresses the non-response bias that would exist without patient-mix adjustment; see “The Effects of
Survey Mode, Patient Mix, and Nonresponse on CAHPS Hospital Survey Scores.” Elliott, Zaslavsky et
al. (2009) Health Services Research, 44 (2): 501-518.

Information on statistical tests of the 27-item HCAHPS Survey can be found in the original 2006 OMB
package. Here we focus on the most recent tests of the validity and reliability of the current, 32-item
HCAHPS Survey at the recommended 300 completed surveys in a 12-month public reporting period.

In terms of hospital-level reliability, the signal-to-noise ratio indicates how much of what you measure is
“signal” (true variation in performance), rather than “noise” (measurement error). Unit or hospital-level
reliability (Spearman-Brown reliability) is the proportion of variance in hospital-level scores that reflect
true variation among hospitals, not noise due to limited numbers of patient surveys; see Table 4.

Table 4: Hospital-Level Spearman-Brown Reliabilities of HCAHPS Top-Box Scores at 300
Completed Surveys, 3.1 million discharges, 2013.
Communication with Nurses
Communication with Doctors
Responsiveness of Hospital Staff
Pain Management
Communication about Medicines
Cleanliness
Quietness
Discharge Information
Overall Rating
Recommend Hospital
Care Transition

0.87
0.85
0.90
0.80
0.82
0.86
0.93
0.83
0.90
0.92
0.84

Generally accepted standards for reliabilty are: 0.7: Adequate (all HCAHPS measures exceed this); 0.8:
Good (all exceed this); 0.85 Very good (7 of 11 reach this); and 0.9: Excellent (4 of 11 reach this level).
It should be noted that at more than the recommended 300 completed surveys in 12 month reporting
period, HCAHPS reliability is even higher.

8

Evidence of the internal consistency of the HCAHPS Survey can be found in Table 5, which displays the
Cronbach’s Alpha statistics for the seven publicly reported HCAHPS composite measures, which are
made up of two or three survey items.
Table 5: Cronbach’s Alphas of HCAHPS Composite Measures (composed from 2 or 3 survey
questions) at 300 Completed Surveys, 3.1 million discharges, 2013.
Communication with Nurses
Communication with Doctors
Responsiveness of Hospital Staff
Pain Management
Communication about Medicines
Discharge Information
Care Transition

0.86
0.88
0.72
0.83
0.67
0.51
0.80

Criterion validity indicates the extent to which HCAHPS measures what it purports to measure: patient
experience of care. The correlation of specific HCAHPS measures with “Rating” and “Recommendation”
can be used to assess validity (criterion validity); see Table 6. We expect moderate, positive correlations
of the other nine HCAHPS measures with patient experience of care constructs. If the correlations were
too high, this may indicate redundancy or halo efffect. If too low, would indicate lack of validity. We
find moderate, positive correlations between specific HCAHPS measures and global measures (Rating
and Recommendation). All correlations are between 0.22 and 0.50; 13 of 18 correlations exceed 0.33.

9

Table 6: Top-Box Correlations for HCAHPS Measures at 300 Completed Surveys, 3.1 million
discharges, patient-level, 2013.

Nurse
Doctor
Staff
Pain
RX Com.
Clean
Quiet
Dischg.
Rating
Recom.
Care
Trans.

Nurse

Doctor

Staff

Pain

1
0.52
0.48
0.51
0.48
0.35
0.29
0.24
0.50
0.47

1
0.35
0.41
0.42
0.26
0.25
0.24
0.39
0.38

1
0.43
0.38
0.30
0.29
0.16
0.38
0.34

1
0.42
0.29
0.28
0.21
0.41
0.38

0.40

0.37

0.31

0.35

RX
Comm.

Discha
rge

Rating

Recom
md.

Clean

Quiet

1
0.31
0.28
0.29
0.38
0.36

1
0.24
0.15
0.32
0.29

1
0.10
0.25
0.22

1
0.23
0.23

1
0.61

1

0.43

0.25

0.23

0.29

0.40

0.41

Care
Trans.

1

The correlations of the nine specific measures with the two global measures are larger at the hospital level
than at the person level, in part because of the better measurement at the hospital level (as intended) than
at the person level; see Table 7. These hospital-level correlations range from 0.39 to 0.76, with all but
one exceeding 0.49.

Table 7: Top-Box Correlations for HCAHPS Measures, ~4,000 Hospitals, hospital level, 2013.

Nurse
Doctor
Staff
Pain
RX
Comm.
Clean
Quiet
Disch.
Rating
Recom.
Care
Trans.

Nurse

Doctor

Staff

Pain

1
0.75
0.83
0.75

1
0.68
0.62

1
0.68

1

0.75
0.66
0.60
0.50
0.78
0.65

0.64
0.53
0.62
0.39
0.61
0.48

0.69
0.70
0.60
0.42
0.66
0.51

0.64
0.53
0.52
0.47
0.68
0.58

0.73

0.59

0.63

0.61

RX
Comm.

Discha
rge

Rating

Recom
md.

Clean

Quiet

1
0.56
0.54
0.48
0.64
0.52

1
0.49
0.33
0.59
0.46

1
0.24
0.54
0.39

1
0.56
0.49

1
0.88

1

0.63

0.53

0.47

0.52

0.76

0.72

Care
Trans.

1

10

Discriminant validity (specific measures assess specific aspects of patient experience construct, not just
the general patient experience contstruct) is demonstrated by the higher inter-item correlations within the
7 composite measures than across (between) measures, as seen in Table 7. Table 8 displays mean patientlevel inter-item correlations within composites (bottom row), which range from 0.47 to 0.58 except for
Discharge at 0.28 (the Discharge composite is uniquely composed of two dichotomous items, which
artificially suppresses correlations) and mean inter-item correlations between composites for items in
different composites (which range from 0.13 to 0.38). The second-last row of the table contains the mean
between-composite inter-item correlation involving the composite in the column (e.g. mean inter-item
correlation is 0.50 within the Nurse Communication composite and 0.31 for correlations of Nurse
Communication items with items in other composites). In all cases, the mean correlation within
composites is much higher than those between composites, providing clear evidence of discriminant
validity. This can also be demonstrated via factor analysis of HCAHPS results; see “Exploratory Factor
Analyses of the CAHPS Hospital Pilot Survey Responses across and within Medical, Surgical, and
Obstetric Services.” O’Malley, Zaslavsky, et al. (2005) Health Services Research, 40 (6): 2078-2095.

Table 8: Average Top-Box Inter-Item Correlations for Individual HCAHPS Items, 3.1 million
discharges, patient level, 2013.

Nurse
Doctor
Staff
Pain
RX Comm.
Discharge
Care
Transition
Avg. Cross
Avg. Within

Nurse
0.50
0.35
0.37
0.38
0.34
0.16

Doctor

Staff

Pain

RX Comm.

Discharge

0.55
0.27
0.31
0.31
0.16

0.47
0.35
0.31
0.13

0.55
0.32
0.15

0.46
0.20

0.28

0.28
0.31
0.50

0.27
0.28
0.55

0.25
0.28
0.47

0.27
0.30
0.55

0.32
0.30
0.46

0.20
0.17
0.28

Care
Transition

0.58
0.26
0.58

Three Care Transition questions were added to the HCAHPS Survey in January 2013. The Care
Transition Measure was originally endorsed by the National Quality Forum as NQF #0288. In 2012,
CMS tested the validity of the CTM as a part of the HCAHPS Survey in a mode experiment. Briefly,
20,670 inpatients discharged January 1, 2012 through March 31, 2012 were randomly assigned to each of
the four HCAHPS survey modes: Mail only; Mixed mode (Mail with Telephone follow-up); Telephone
11

only (CATI); and Touch-tone IVR only (TT-IVR). With respect to validity of the CTM, we found
Pearson correlations (linear scoring) with 10 existing HCAHPS measures ranging from 0.30 to 0.51. The
highest correlations were with Recommend Hospital (0.51), Nurse Communication (0.50),
Communication About Medicines (0.49), Overall Hospital Rating (0.48). The patient-mix adjustment the
survey mode effect patterns were similar to other HCAHPS measures.
The 2012 mode experiment analysis provides support for the construct validity of the HCAHPS Survey,
finding strong correlation of each of the composites and individual measures with the two overall
measures and confirming the intended factor structure. Like all HCAHPS measures CTM has good
psychometric properties. We found no evidence of a ceiling effect; there is substantial room for quality
improvement. While there are strong links with Nurse Communication and Communication about
Medicines, there was generally a lack of redundancy with HCAHPS measures. Results from the national
implementation of the Care Transition Measure in 2013 indicate there is ample room for improvement on
this aspect of patient experience, as compared with the ten original HCAHPS measures; see Table 9.

Table 9: HCAHPS Top-Box Scores, ~4,000 hospitals, 3.1 million discharges, 2013.
HCAHPS Composite
Communication with Nurses
Communication with Doctors
Responsiveness of Hospital Staff
Pain Management
Communication about Medicines
Cleanliness
Quietness
Discharge Information
Overall Rating
Recommend Hospital
Care Transition

Mean
79.12
81.82
67.85
70.88
64.30
73.61
61.47
85.73
70.80
71.12
51.27

Std
Dev
5.68
5.36
9.12
5.98
7.01
7.86
10.23
4.45
9.11
10.08
7.23

Recent HCAHPS results that include the new Care Transition Measure continue to affirm the high level
of reliability of the HCAHPS Survey; see above. These results demonstrate that all 11 HCAHPS
measures reliably distinguish hospital performance and are sufficiently reliable for hospital-level public
reporting and performance incentives at the recommended 300 completed surveys using top-box scoring.

The goal of HCAHPS is to collect information from patients using a standardized, national survey and to
present the information based on those surveys to consumers, providers and hospitals. One of the
12

methodological issues associated with making comparisons between hospitals is the need to adjust
appropriately for patient-mix differences. Patient-mix refers to patient characteristics that are not under
the control of the hospital that may affect measures of patient experiences, such as demographic
characteristics and health status. The basic goal of adjusting for patient-mix is to estimate how different
hospitals would be rated if they all provided care to comparable groups of patients.

CMS applies patient-mix adjustment to control for patient characteristics that affect ratings and that are
differentially distributed across hospitals. Most of the patient-mix items are included in the “About You”
section of the instrument, while others are taken from administrative records. Based on the mode
experiment, and consistent with previous studies of patient-mix adjustment in CAHPS and in previous
hospital patient surveys, we employ the following variables in the patient-mix adjustment model:









Self-reported general health status (specified as a linear variable)
Education (specified as a linear variable)
Type of service (medical, surgical, or maternity care)
Age (specified as a categorical variable)
Admission through emergency room (discontinued in 2010)
Lag time between discharge and survey
Age by service line interaction
Language other than English spoken at home

Once the data are adjusted for patient-mix, there is a fixed adjustment for each of the reported measures
for mode of administration (discussed in detail below). The patient-mix adjustment employs a regression
methodology, also referred to as covariance adjustment.

On the survey there are two items that capture the race and ethnicity of the respondent. These items are
not included in the patient-mix adjustment model but used as analytic variables to support two
congressionally-mandated reports: “The National Healthcare Disparities Report” and “The National
Healthcare Quality Report.” These reports provide annual, national-level breakdowns of HCAHPS scores
by race and ethnicity. Many hospitals collect information on race and ethnicity through their
administrative systems, but coding is not standard. Thus it was determined that administrative data are
not adequate to support the analyses needed for the reports and the items should be included in the
questionnaire.

CMS added five new items to the HCAHPS Survey beginning with July 2012 discharges on a voluntary
basis. These items officially became a required part of the HCAHPS Survey beginning with January 2013
13

discharges; results were publicly reported for the first time in December 2014. Three new items (survey
questions 23, 24 and 25) were added to the ‘core’ of the survey in order to create the new Care Transition
Composite measure, which became the seventh HCAHPS composite measure. Two new items (survey
questions 26 and 28) were added to the “About You” section of the survey: one concerning emergency
room admission, and one concerning self-reported overall mental health.
The three-item Care Transition Measure (CTM) assesses patients’ perspectives on the coordination of
hospital discharge care. The CTM was developed by Dr. Eric A. Coleman of the University of Colorado,
a leading researcher in the area of care transitions, and has been endorsed by the NQF as a voluntary
consensus standard, NQF #0228. The new CTM items complement the two existing HCAHPS discharge
questions, provide detailed information on the transition to post-hospital care, and incentivize quality
improvement in this critical aspect of care.
In 2013, CMS also added two new survey items to the “About You” section of the HCAHPS Survey. The
first asks patients whether they entered the hospital through the emergency room. The second asks
patients about their overall mental health. Each of these items conforms to CAHPS standards and had
been tested in the HCAHPS Three-State Pilot Study in 2003. Neither item was adopted in the national
implementation of HCAHPS, primarily to keep the survey as short as possible. At that time, emergency
room admission information could be obtained from hospital administrative data, and patients’ overall
mental health status was found to have relatively little impact on patient-mix adjustment in the ThreeState Pilot Study. In response to requests from users and stakeholders, CMS added these items to the
HCAHPS Survey. CMS does not publicly report the results of these two items.
From 2006 to 2010, “emergency room admission” as a point of origin for hospital patients was an
administrative code that was provided by hospitals. CMS employed this administrative code as a patientmix adjustment for HCAHPS scores. However, effective July 1, 2010, the “Point of Origin for
Admission or Visit” code for Emergency Room (code 7) was discontinued for use by Medicare Systems
and thus became unavailable for HCAHPS. CMS has been unable to find any other hospital
administrative variable or code that could be used to reliably capture admission through the emergency
room. The best alternative CMS has found is a patient-reported item that had been developed for
HCAHPS and was tested in 2003. This patient-reported emergency room admission item was added to
the “About You” section of the HCAHPS Survey in July 2012 on a voluntary basis; it became a required
survey item in January 2013.
14

Although we chose not to add a survey item about patient’s overall mental health status in the national
implementation of HCAHPS in 2006, we continued to receive inquiries and requests from hospitals and
researchers on this topic. Some researchers claim that mental health status is an important factor in how
patients respond to HCAHPS survey items. The continuing interest in this topic, coupled with the direct
impact of HCAHPS performance on hospital payments beginning October 2012, led us to decide to add
an overall mental health item to the HCAHPS Survey. The overall mental health survey item we have
chosen very closely resembles the Overall General Health item in the HCAHPS Survey, has been
extensively tested, and is currently included in several other CAHPS surveys. CMS added the overall
mental health item to the “About You” section of the HCAHPS Survey in July 2012 on a voluntary basis;
it became a required survey item in January 2013. We provided a copy of the revised HCAHPS Survey in
our 2012 OMB package. There have been no changes to the HCAHPS Survey since then.

The addition of five new items in 2012 necessitated a new mode experiment to investigate how each survey
mode (mail, telephone, mail with telephone follow-up, and active Interactive Voice Response) affects
responses to the new Care Transition Measure items. This mode experiment, called HCAHPS Mode
Experiment 3, closely resembled the original HCAHPS mode experiment. HCAHPS Mode Experiment 3
was conducted from January to March 2012 and encompassed 47 hospitals from around the USA that
represented key hospital characteristics. Briefly, a sample of 20,670 patients discharged during that period
was randomly assigned in equal numbers to the four HCAHPS modes. Survey results were compared and
mode effects were estimated to adjust for their impact on survey response. The CMS HCAHPS
subcontractors RAND and Health Services Advisory Group conducted the mode experiment and analyzed the
data. We found strong evidence for the reliability and validity of the Care Transition Measure:
-

-

-

-

CTM composite score
o Top-box: 46%
o Middle-box: 46%
o Bottom-box : 8%
o
Cronbach’s alpha
o 0.82 (Linear scoring)
o 0.80 (top-box scoring)
o
Hospital-level reliability (Spearman-Brown), at 300 or more completed surveys
o 0.88 (linear scoring)
o 0.84 (top-box scoring)
o
Pearson correlations (linear scoring) with 10 existing HCAHPS measures
15

o Range 0.30 to 0.51
o Patient-mix adjustment and survey mode effects are similar to other HCAHPS measures
In summary, the Care Transition Measure displays strong psychometric properties; there is ample room for
quality improvement (top-box score is lower than the other 10 HCAHPS measures; see Table 8); no evidence
of a ceiling effect; CTM is not redundant with existing HCAHPS measures; and the relatively strong
correlation with Nurse Communication and Communication about Medicines suggests the importance of care
coordination in those areas.

To implement the expanded survey, CMS began training participating hospitals and survey vendors on the
new items in 2012 and updated the survey materials, telephone scripts, and file specifications for these items.
The Care Transition Measure became the seventh HCAHPS composite publicly reported on the Hospital
Compare Web site in December 2014.

In the FY 2013 Inpatient Prospective Payment System rule, CMS added the new HCAHPS Survey items
to the Annual Payment Update (APU) requirements of IPPS hospitals. CMS will propose adding the Care
Transition Measure as the ninth dimension in the Patient Experience of Care Domain in the Hospital
Value-Based Purchasing program for IPPS hospitals in the 2015 Inpatient Prospective Payment System
Proposed Rule.

4. Past and Ongoing Tests of Procedures, Trainings, and Quality Improvement Activities;
Collaborator and Contractor Participation (collaborators and contractors are bolded).
Hospitals began using the HCAHPS Survey in 2006 under the auspices of the Hospital Quality Alliance,
a private/public partnership that includes CMS and the American Hospital Association, the Federation
of American Hospitals, and the Association of American Medical Colleges; the Joint Commission on
Accreditation of Healthcare Organizations; the National Quality Forum; AARP; and CMS’ Agency
for Healthcare Research and Quality (AHRQ).

Beginning in 2002, HCAHPS has undertaken extensive research, planning, development and testing
activities. CMS originally partnered with AHRQ to develop a standard instrument and data collection
and reporting procedures that capture patients’ perspectives of their hospital care. AHRQ is a leader in
developing instruments for measuring patient perspectives on health care. Table 10 summarizes the

16

process of creating and implementing the HCAHPS survey throughout the period starting July 2002
through the first public reporting of HCAHPS results in March of 2008.

Table 10: HCAHPS Survey Development and Implementation Timeline, 2002-2008.
Activity
Published a “call for measures” in the Federal Register and received 7
submissions from Avatar, Edge Health Care Research Healthcare Financial
Management Association, Jackson Organization, Press Ganey Associates,
National Research Corporation, Peace Health, Professional Research
Consultants, and SSM Health Care.
Completed literature review.
Held a Web chat to answer questions about HCAHPS.
Provided draft domains to CMS.
Reviewed measures submitted in response to Federal Register Notice
FRN).
Held Stakeholders Meeting to solicit suggestions and comments.
Held vendors meeting to solicit suggestions and comments.
AHRQ delivered 66 item draft survey to CMS for use in pilot test.
Developed data collection and sampling methods, and developed analysis
plan.
Published FRN soliciting comments on draft HCAHPS.
Completed hospital recruitment for pilot.
Began data collection for CMS 3-state pilot test.
Published a FRN soliciting comments on draft HCAHPS and asked for
input about implementation issues.
Analyzed data from CMS pilot test.
Review of instrument by CAHPS Cultural Comparability team.
Began CT pilot test of the HCAHPS instrument.
Held HCAHPS Stakeholders’ Meeting at AHRQ.
Revised HCAHPS instrument to 32 items.
AHRQ submitted revised 32-items HCAHPS Instrument to CMS.
Published a FRN soliciting input for 32-item HCAHPS instrument and
implementation strategy.
Started coordination of national implementation with HSAG, the AZ QIO.
Completed CT pilot test of HCAHPS.
AHRQ submitted Analysis Report of the CMS 3-state pilot to CMS.
Continued discussions with hospitals, vendors, consumers to follow-up on
FRN comments from February.
Revised 25-item HCAHPS Instrument submitted by AHRQ to CMS.
Submitted HCAHPS to NQF for its consensus development process.

Timeframe

July 2002

Sept-Nov 2002
Oct 2002
Oct 2002
Nov 2002
Nov 2002
Nov 2002
Jan 2003
Jan-Feb 2003
Feb 2003
Mar 2003
June 2003
June 2003
Sept–Nov 2003
Fall 2003
Fall 2003
Nov 2003
Nov 2003
Dec 2003
Dec 2003-Feb 2004
January 2004
Jan 2004
Jan 2004
March – Sept 2004
Oct 2004
November 2004
17

Started developing training documents for national implementation.
Started discussions regarding data transmission via QNET & other issues
with the IFMC, the IA QIO.
Formed the Data Integrity Group.
Received endorsement for 27-item HCAHPS from the National Quality
Forum.
Modified survey instrument and protocol as 27-items.
Abt Associates, Inc. receives OMB approval for cost-benefit analysis.
Established the Data Submission and Reporting Group.
Abt Associates, Inc. submits final report of the cost-benefit analysis.
Published FRN soliciting comments on draft CAHPS Hospital Survey.
Received final approval.
Mode Experiment.
National Implementation begins.
HCAHPS participation linked to RHQDAPU program (“pay for
reporting”).
First public reporting of HCAHPS results.

December 2004
April 2005
June 2005
May 2005
May 2005
June 2005
July 2005
October 2005
November 2005
December 2005
February – May 2006
October 2006
July 2007
March 2008

Throughout the HCAHPS development process, CMS solicited and received a great deal of public input.
As a result, the HCAHPS questionnaire and methodology went through several iterations prior to national
implementation. The accumulated lessons learned from the pilot testing, public comments, input from
stakeholders, numerous team discussions, and the National Quality Forum’s review and endorsement
(NQF #0166) through their consensus development process led to the national implementation in 2006 of
the 27-item HCAHPS Survey and the HCAHPS data collection protocol that allows hospitals to integrate
their own supplemental questions. The resulting core questionnaire is comprised of questions in several
dimensions of primary importance to the target audience: doctor communication, responsiveness of
hospital staff, cleanliness of the hospital environment, quietness of the hospital environment, nurse
communication, pain management, communication about medicines, and discharge information. The
HCAHPS Survey was re-endorsed by the NQF in 2012 and 2015.

In response to numerous requests from within CMS, the Department of Health and Human Services, and
external stakeholders, the HCAHPS Survey was expanded for the first time by the addition of five new
items, which began with July 2012 discharges and on a voluntary basis. These 5 new items officially
became a required part of the HCAHPS Survey beginning with January 2013 discharges; results were
publicly reported for the first time in December 2014. Three new items were added to the ‘core’ of the
survey in order to create the new Care Transition Composite measure, which became the seventh
18

HCAHPS composite measure, and two new items were added to the “About You” section of the survey:
one concerning emergency room admission, and one concerning self-reported overall mental health. The
five new items raised the total number of items to 32. The 32-item HCAHPS Survey currently in use can
be found in Attachment A – HCAHPS Survey Instrument (Mail) and Supporting Materials.

The HCAHPS implementation plan changed significantly as a result of the public input we received prior
to national implementation. CMS made the following major changes in the implementation approach,
which have served to lessen the burden on hospitals/survey vendors:






reduced the number of mailings for the “mail only” survey protocol from three to two;
reduced the number of follow-up phone calls for the “telephone only” survey protocol from ten to
five;
added active interactive voice response (IVR) as a mode of survey administration;
eliminated the 50% response rate requirement;
reduced the number of patient discharges to be surveyed.

Since national implementation began in 2006, CMS has continually refined and clarified HCAHPS survey
protocols, created new translations of the mail version of the survey in Chinese, Russian, Vietnamese, and
Portuguese in addition to the original English and Spanish versions (in the telephone and IVR modes, the
HCAHPS Survey is available in both English and Spanish), added five new survey items, including three
items that form the new composite measure, Care Transition, annually updated the HCAHPS Quality
Assurance Guidelines (currently version 10.0), improved the appearance and accessibility of HCAHPS
results on the Hospital Compare website (http://www.medicare.gov/hospitalcompare/search.html), and
made information about HCAHPS quickly and easily available through its official HCAHPS On-Line
website, www.hcahpsonline.org. There has been steady and significant improvement in HCAHPS scores
since national implementation; see Elliott, Lehrman, Goldstein et al. (2010), “Hospital Survey Shows
Improvements in Patient Experience.” Health Affairs, 29 (11): 2061-2067. The HCAHPS Project Team
has published a number of analyses of HCAHPS results in peer-reviewed scientific journals and made
numerous presentations at professional conferences; a bibliography of the publications can be found at
http://www.hcahpsonline.org/home.aspx.

Since public reporting of hospitals HCAHPS results was inaugurated in March 2008, a growing number
of healthcare, consumer and professional organizations, state governments, media outlets and others have
adopted or incorporated HCAHPS scores, in part or in whole, for their own purposes. These activities,
external to CMS, have had the effect of extending knowledge about HCAHPS and increasing the impact
19

of survey results. The content of the HCAHPS Survey, its methodology and administration protocols, and
its ambition to measure and publicly report consumers’ experiences in a uniform and standardized manner
have influenced other surveys developed within CMS as well as those undertaken by hospitals and
healthcare systems in the United States and abroad.

There are distinct roles for hospitals or their survey vendors and the federal government in the national
implementation of HCAHPS. The federal government is responsible for support and public reporting,
including:








conducting training on data collection and submission procedures,
providing on-going technical assistance,
ensuring the integrity of data collection,
accumulating HCAHPS data from individual hospitals,
producing patient-mix adjusted hospital-level estimates,
conducting research on the presentation of data for public reporting, and
publicly reporting the comparative hospital data.

Hospitals or their survey vendors are responsible for data collection, including: developing a sampling
frame of relevant discharges, drawing the sample of discharges to be surveyed, collecting survey data
from sampled discharges, and submitting HCAHPS data to CMS in a standard format. We have
formatted the data files so hospitals/vendors will submit to CMS de-identified data files following 45 CFR
Section §164.514. Hospitals maintain business associate agreements with their contracted survey vendors
to collect and submit HCAHPS survey data through the secure QualityNet Exchange portal and data
warehouse.

CMS began its collaboration with the Health Services Advisory Group (HSAG) in 2003 to coordinate
the national implementation of the Hospital CAHPS Survey. HSAG’s role is to provide technical
assistance and training for vendors and hospitals, data validation, data processing, analysis, and
adjustment, and oversight of self-administering hospitals and survey vendors. HSAG also produces
electronic data files and a hospital level extract file for public reporting of the HCAHPS scores.

In the spring of 2006, CMS conducted a large-scale experiment to assess the impact of mode of survey
administration, patient characteristics and patient non-response on HCAHPS results. This first mode
experiment was based on a nationwide random sample of short-term acute care hospitals. Hospitals from
each of CMS' ten geographic regions participated in the Mode Experiment. A hospital's probability of
being selected for the sample was proportional to its volume of discharges, which guaranteed that each
20

patient would have an equal probability of being sampled for the experiment. The participating hospitals
contributed patient discharges from a four-month period: February, March, April, and May 2006. Within
each hospital, an equal number of patients was randomly assigned to each of the four modes of survey
administration. Sample selection and surveying were conducted by the National Opinion Research
Center of the University of Chicago, and the data was analyzed by the RAND Corporation.

A randomized mode experiment of 27,229 discharges from 45 hospitals was used to develop adjustments
for the effects of survey mode (Mail Only, Telephone Only, Mixed mode, or Active Interactive Voice
Response) on responses to the HCAHPS Survey. In general, patients randomized to the Telephone Only
and Active Interactive Voice Response provided more positive evaluations than patients randomized to
Mail Only and Mixed (Mail with Telephone follow-up) modes. These mode effects varied little by
hospital, and were strongest for global items (rating and recommendation), and the Cleanliness & Quiet,
Responsiveness, Pain Management, and Discharge Information composites. Adjustments for these mode
effects are necessary to make the reported scores independent of the survey mode that was used. These
adjustments are applied to HCAHPS results before they are publicly reported on the Hospital Compare
website. The mode adjustments can be found in the “Mode Adjustment” section of the HCAHPS website,
www.hcahpsonline.org.

The Mode Experiment also provided valuable information on the impact of salient patient characteristics
and non-response bias on HCAHPS results. This analysis was needed because hospitals do not provide
care for comparable groups of patients but, as demonstrated in the HCAHPS Three-State Pilot Study,
some patient characteristics may affect measures of patient experiences of care. The goal of patient-mix
adjustment, which is also known as case-mix adjustment, is to estimate how different hospitals would be
rated if they provided care to comparable groups of patients. As suggested by the Three-State Pilot Study,
a set of patient characteristics not under control of the hospital was selected for analysis. In summary, the
most important patient-mix adjustment items were patients’ self-reported health status, education, service
line (maternity, medical or surgical care) and age. In addition, after mode and patient-mix adjustments
have been made, non-response effects were found to be negligible. A report on patient-mix and nonresponse adjustments, as well as the mode adjustments and current and past patient-mix adjustments, is
available on the HCAHPS On-Line website, http://www.hcahpsonline.org/modeadjustment.aspx. We
also looked at the extent to which each domain contributes to measurement in priority areas established by
an independent, expert body on quality measurement, the National Quality Forum (NQF). The
HCAHPS domains “communication with doctors,” “communication with nurses,” “communication about
21

medications” contribute to the NQF’s priority on improving care coordination and communication. The
HCAHPS “pain control” domain contributes to the NQF’s pain management priority, while the HCAHPS
“discharge information” domain contributes to the priority on improving self-management and health
literacy.

CMS, with assistance from HSAG and its sub-contractor, the National Committee on Quality
Assurance (NCQA), has developed and conducted annual training sessions for self-administering
hospitals and survey vendors participating in the HCAHPS Survey, as well as others interested in this
program. HCAHPS Introductory Training was first offered at the CMS headquarters in January 2006, and
then by webinar in January and April 2006. Since then, HCAHPS Introductory Training has been held
annually, by webinar. In addition, CMS developed the HCAHPS Update Training program. HCAHPS
Update Training was first offered in May 2007 and has been offered annually by webinar since then.
HCAHPS Introductory Training is required for self-administering hospitals and survey vendors that wish
to join HCAHPS. HCAHPS Update Training provides information on important changes to the HCAHPS
program and is required for all self-administering hospitals and survey vendors participating in HCAHPS.

As part of an initiative to add five-star quality ratings to its Compare Web sites, CMS will add HCAHPS
Star Ratings to the Hospital Compare Web site in April 2015. Star ratings will make it easier for
consumers to use the information on the Compare Web sites and spotlight excellence in healthcare
quality. Twelve HCAHPS Star Ratings will appear on Hospital Compare: one for each of the 11 publicly
reported HCAHPS measures, plus the new HCAHPS Summary Star Rating. HCAHPS Star Ratings will
be the first star ratings to appear on Hospital Compare. CMS plans to update the HCAHPS Star Ratings
each quarter. In preparation for the addition of HCAHPS Star Ratings to Hospital Compare, we have
posted documents on the HCAHPS On-Line Web site that explain the calculation of the star ratings and
address frequently asked questions; see http://www.hcahpsonline.org/StarRatings.aspx.

Mode effects can be addressed with precise analytic adjustments developed in large-scale mode
experiments. These adjustments are specific to the administered survey items. A mode experiment was
conducted in 2012 to develop adjustments for newly added HCAHPS care transition items. In 2016,
CMS will conduct another mode experiment of sufficient scale to examine and identify changes in mode
effects since the last mode experiment and if necessary, provide precise adjustment estimates for use with
the current 32-item HCAHPS survey. The purpose of the planned HCAHPS mode experiment
(Document Identifier: CMS Form Number CMS-10542: HCAHPS Mode Experiment) is to conduct a
22

mode experiment of sufficient sample and scale to determine if the mode adjustments currently employed
for the 32-item HCAHPS core survey need revision. An additional goal is to collect empirical evidence
on the effect of the number of additional supplemental items, added by hospitals, on survey response rate
and patterns of response to the HCAHPS core demographic items (known as “About You” items).

5. Names and telephone numbers of individuals consulted on statistical aspects of the survey and
implementation design and names of agency units, contractor(s), grantee(s), or other persons(s) who
collect and/or analyze the information for the agency.

Marc N. Elliott, PhD
Distinguished Chair in Statistics; Senior Principal Researcher
RAND Corporation
(310) 393-0411, x7931
[email protected]
Christopher Cohea, MS.
Senior Research Analyst
Health Services Advisory Group, Inc.
(602) 471-3673
[email protected]

23