Form CMS-10102 Appendix B

Supporting Statement for the National Implementation of the Hospital CAHPS Survey

Appendix B
Replication of Patient-Level Psychometric Analysis of the
HCAHPS Instrument Across Two Samples

1

Replication of Patient-Level Psychometric Analysis of the
HCAHPS Instrument Across Two Samples

Analyses were conducted to examine the psychometric properties of the revised form of the
HCAHPS questionnaire used in two samples: (1) 3-State Pilot (3SP)1, N=19,568 and (2) Connecticut
(CT), N=1,675.

The basic unit of reporting for the HCAHPS survey measure is the hospital. Thus, it is most
appropriate to focus on the psychometric features of the measures at the hospital level. Hospital-level
reliability captures the extent to which variation in scores on a composite reflects variation between
hospitals, as opposed to random variation in patient response within hospitals. Hospital level
correlations for construct validity capture the extent to which hospitals with high scores on the
composites also have high scores on patient willingness to recommend the hospital and the overall
rating. Hospital-level reliabilities of the composites and hospital-level correlations of the composites
with the global ratings for the three-state pilot were presented previously. Because of more limited
data available in the Connecticut pilot individual-level analyses were conducted (which also can be
informative) and are presented below.

The HCAHPS measure was compiled into the following seven composites: Communication with
Nurses (n=3), Communication with Doctors (n=3), Communication about Medicine (n=2), Nursing
Services (n=2), Discharge Information (n=2), Pain Control (n=2), and Physical Environment (n=2).
(One item from the doctor communication and nurse communication composites was subsequently
dropped.) Within both samples, the reliability of the seven composites was estimated using the

1

The 3SP dataset is comprised of data obtained in Arizona, Maryland, and New York.

2

internal consistency method (Cronbach’s alpha coefficient). The construct validity of the composites
was evaluated with regard to their relationships to an overall rating of the hospital (Hospital Rating)
and whether the patient would recommend the hospital to others (Hospital Recommendation). The
results of these analyses indicate that the HOSPITAL CAHPS SURVEY measure performed
similarly in the CT and 3SP data sets.

The alpha coefficients across the two data sets were comparable (see Table 1): in the 3SP data file the
alpha coefficients ranged from .51 to .88 and in the CT sample the alphas ranged from .50 to .87. The
same four of seven composites within both samples had alpha coefficients greater than .70. These
were Communication with Nurses, Communication with Doctors, Nursing Services, and Pain
Control. Analyses also revealed the same relationships across data sets of items to competing
composites (see Table 1). Within both samples, the items comprising the Nursing Services composite
(Q22, Q9) were correlated as or more strongly with the Communication with Nurses composite than
with their own composite. And, items within the Physical Environment composite were correlated as
or more strongly with three other composites (Communication with Nurses, Nursing Services, and
Pain Control) than they were with their own composite.

These relationships were found in the

analyses presented in HOSPITAL CAHPS SURVEY Three-State Pilot Study Analysis Report at
www.cms.hhs.gov/quality/hospital

. For purposes of this analysis it is important to note that the

relationships found in the 3SP data were replicated in the CT data.

With one exception, the results of the construct validity analyses were also very similar in the two
data sets (see Table 1, aR2 values). Compared to the 3SP sample, Communication with Doctors had a
somewhat stronger relationship to global ratings of hospital care in the CT sample. Table 2 presents a
comparison of the descending rank order of correlations of the composite scores with the following
3

global ratings: hospital rating, nurses rating, doctors rating, and hospital recommendation. As
illustrated in the table, three composites emerge consistently, within both samples, as being among
the most highly correlated with the global ratings: Nurse Communication, Nursing Services, and Pain
Control.

4

Table 1: A Comparison of the Patient-Level Psychometric Analyses of the Seven-Factor HospitalLevel Structure between the 3-State Pilot and Connecticut Samples

3 State Pilot Sample
Quest
#

Question Label

(1) Communication with Nurse

Integrity of
Composites
Substant Alpha &
ial Corr.
Itemnd
w2
Total
Composite
Corr.
α = .86

Relationship of Item and Composite-Level Scores to
Hospital Rating and Recommendation**
Hospital Rating
2

aR =0.47*

.34

t-value=
101.55
38.70
48.02
21.62
t-value=
41.15
13.07
21.06
1.20#
t-value=
12.41
9.73
5.46
t-value=
36.05
18.35
25.22
t-value=
20.33
11.49
14.54
t-value=
39.96
11.98
23.95
t-value=
52.08
45.11

.34

22.64

Q5
RN Listen
Q4
RN Respect
Q6
RN Explain
(2) Communication with Doctors

.77
.73
.69
α = .88

aR =0.24

Q12
MD Listen
Q11
MD Respect
Q13
MD Explain
(3) Communication about Medication

.81
.76
.73
α = .67

aR =0.18

Q40
Allergies to Medicines
Q41
Side-Effects of Medicine
(4) Nursing Services

.51
.51
α = .72

aR =0.36

.56
.56
α = .51

aR =0.08

Q49
Symptoms may have
Q48
Help for you at home?
(6) Pain Control

.35
.35
α = .83

aR =0.30

Q32
Pain Controlled
Q33
Pain Help All Can
(7) Physical Environment

.71
.71
α = .51

aR =0.26

Q22
How often Bathroom
Q9
Help when Call Button
(5) Discharge Information

Q17

Room Clean

Q18

Room Quite

1(.56)
1(.63)

1(.43)
4(.44)
6(.34)
1(.35)
4(.39)

2

2

2

2

2

2

Recommend Hospital
2

aR =0.37

2

aR =0.19

2

aR =0.14

2

aR =0.28

2

aR =0.07

2

aR =0.24

2

aR =0.19

t-value=
79.02
28.59
39.35
16.22
t-value=
34.67
9.20
19.13
1.49#
t-value=
7.99
5.97
3.92
t-value=
28.67
16.17
18.39
t-value=
19.30
12.28
12.01
t-value=
34.16
9.94
20.84
t-value=
33.19
31.94

11.21

* aR2 = Adjusted R-squared, how much variance in the dependent variable is accounted for by the set of items
in the composite controlling for the effect of number of variables (i.e. all things being equal, a larger
set of items will account for a larger percentage of the variance.
** t-values listed in grey cells are for the unique relationship of this composite to the criterion variable
controlling for the other composites. t-values in cells adjacent to the item are for the unique
relationship of that item controlling for the other report items in the questionnaire, therefore these are
the same values as those depicted in Table 5. Probability of t-value is less than 0.01 unless otherwise
denoted.
α = Cronbach’s alpha coefficient, an estimate of internal consistency reliability.
# = p > 0.01

5

Connecticut Sample
Quest
#

Question Label

(1) Communication with Nurse
Q5
Q4
Q6

Integrity of
Composites
Substan Alpha &
tial Corr.
Itemnd
w2
Total
Compos
Corr.
ite
α = .85

RN Listen
RN Respect
RN Explain

Relationship of Item and Composite-Level Scores to
Hospital Rating and Recommendation**
Hospital Rating

2

aR =0.45*

.34

.34

9.79

Q12
MD Listen
Q11
MD Respect
Q13
MD Explain
(3) Communication about Medication

.80
.74
.71
α = .69

aR =0.19

Q40
Allergies to Medicines
Q41
Side-Effects of Medicine
(4) Nursing Services

.52
.52
α = .71

aR =0.35

.55
.55
α = .50

aR =0.08

Q49
Symptoms may have
Q48
Help for you at home?
(6) Pain Control

.33
.33
α = .81

aR =0.32

Q32
Pain Controlled
Q33
Pain Help All Can
(7) Physical Environment

.68
.68
α = .51

aR =0.27

Q18

Room Quite

1(.38)
4(.39)
6(.35)
1(.35)
4(.40)

t-value=
11.57
2.57#
7.22
1.14#
t-value=
1.33#
.88#
1.54#
t-value=
13.10
9.94
5.42
t-value=
8.16
2.53#
7.49
t-value=
13.12
6.07
6.45
t-value=
11.98
8.71

aR =0.22

aR =0.30

Room Clean

2

t-value=
16.80
1.27#
7.92
6.42
t-value=
2.39#
-0.18#
3.51
t-value=
12.08
7.28
7.41
t-value=
5.65
1.99#
5.23
t-value=
16.58
7.07
8.44
t-value=
19.07
14.17

2

α = .87

Q17

t-value=
17.47
3.67
14.30
1.62#

aR =0.34

.75
.71
.68

1(.52)
1(.61)

2

t-value=
26.01
7.57
14.81
5.65

(2) Communication with Doctors

Q22
How often Bathroom
Q9
Help when Call Button
(5) Discharge Information

Recommend Hospital

2

2

2

2

2

2

aR =0.15

2

aR =0.29

2

aR =0.08

2

aR =0.25

2

aR =0.19

6.01

* aR2 = Adjusted R-squared, how much variance in the dependent variable is accounted for by the set of items
in the composite controlling for the effect of number of variables (i.e. all things being equal, a larger
set of items will account for a larger percentage of the variance.
** t-values listed in grey cells are for the unique relationship of this composite to the criterion variable
controlling for the other composites. t-values in cells adjacent to the item are for the unique
relationship of that item controlling for the other report items in the questionnaire, therefore these are
the same values as those depicted in Table 5. Probability of t-value is less than 0.01 unless otherwise
denoted.
α = Cronbach’s alpha coefficient, an estimate of internal consistency reliability.
# = p > 0.01

6

Table 2: Comparison of Rank Order (Descending) of Correlations of Composite Scores with Global
Ratings between the 3 State Pilot and Connecticut Samples

3 State Pilot
Hospital Rating

Nurses Rating

Doctors Rating

Recommend Hospital

Nurse Com

.68

Nurse Com

.79

Doctor Com

.79

Nurse Com

.60

Nursing Services

.60

Nursing Services

.66

Nurse Com

.48

Nursing Services

.52

Pain

.54

Pain

.54

Pain

.44

Pain

.48

Physical Environ.

.50

Physical Environ.

.47

Nursing Services

.42

Doctor Com

.43

Doctor Com

.48

Doctor Com

.42

Medicine

.37

Physical Environ.

.42

Medicine

.42

Medicine

.41

Physical Environ.

.33

Medicine

.37

Discharge

.28

Discharge

.25

Discharge

.25

Discharge

.26

Connecticut
Hospital Rating

Nurses Rating

Doctors Rating

Recommend Hospital

Nurse Com

.67

Nurse Com

.76

Doctor Com

.80

Nurse Com

.58

Nursing Services

.59

Nursing Services

.67

Nurse Com

.51

Nursing Services

.54

Pain

.55

Pain

.53

Pain

.44

Pain

.48

Doctor Com

.54

Doctor Com

.48

Nursing Services

.44

Doctor Com

.47

Physical Environ.

.51

Physical Environ.

.45

Medicine

.37

Physical Environ.

.43

Medicine

.43

Medicine

.44

Physical Environ.

.36

Medicine

.38

Discharge

.27

Discharge

.25

Discharge

.25

Discharge

.27

7