National Implementation of Hospital Consumer Assessment of Health Providers and Systems (HCAHPS)

Appendix B

Replication of Patient-Level Psychometric Analysis of the

HCAHPS Instrument Across Two Samples

Replication of Patient-Level Psychometric Analysis of the

HCAHPS Instrument Across Two Samples

Analyses were conducted to examine the psychometric properties of the revised form of the HCAHPS questionnaire used in two samples: (1) 3-State Pilot (3SP)¹, N=19,568 and (2) Connecticut (CT), N=1,675.

The basic unit of reporting for the HCAHPS survey measure is the hospital. Thus, it is most appropriate to focus on the psychometric features of the measures at the hospital level. Hospital-level reliability captures the extent to which variation in scores on a composite reflects variation between hospitals, as opposed to random variation in patient response within hospitals. Hospital level correlations for construct validity capture the extent to which hospitals with high scores on the composites also have high scores on patient willingness to recommend the hospital and the overall rating. Hospital-level reliabilities of the composites and hospital-level correlations of the composites with the global ratings for the three-state pilot were presented previously. Because of more limited data available in the Connecticut pilot individual-level analyses were conducted (which also can be informative) and are presented below.

The HCAHPS measure was compiled into the following seven composites: Communication with Nurses (n=3), Communication with Doctors (n=3), Communication about Medicine (n=2), Nursing Services (n=2), Discharge Information (n=2), Pain Control (n=2), and Physical Environment (n=2). (One item from the doctor communication and nurse communication composites was subsequently dropped.) Within both samples, the reliability of the seven composites was estimated using the internal consistency method (Cronbach’s alpha coefficient). The construct validity of the composites was evaluated with regard to their relationships to an overall rating of the hospital (Hospital Rating) and whether the patient would recommend the hospital to others (Hospital Recommendation). The results of these analyses indicate that the HOSPITAL CAHPS SURVEY measure performed similarly in the CT and 3SP data sets.

The alpha coefficients across the two data sets were comparable (see Table 1): in the 3SP data file the alpha coefficients ranged from .51 to .88 and in the CT sample the alphas ranged from .50 to .87. The same four of seven composites within both samples had alpha coefficients greater than .70. These were Communication with Nurses, Communication with Doctors, Nursing Services, and Pain Control. Analyses also revealed the same relationships across data sets of items to competing composites (see Table 1). Within both samples, the items comprising the Nursing Services composite (Q22, Q9) were correlated as or more strongly with the Communication with Nurses composite than with their own composite. And, items within the Physical Environment composite were correlated as or more strongly with three other composites (Communication with Nurses, Nursing Services, and Pain Control) than they were with their own composite. These relationships were found in the analyses presented in HOSPITAL CAHPS SURVEY Three-State Pilot Study Analysis Report at www.cms.hhs.gov/quality/hospital . For purposes of this analysis it is important to note that the relationships found in the 3SP data were replicated in the CT data.

With one exception, the results of the construct validity analyses were also very similar in the two data sets (see Table 1, aR² values). Compared to the 3SP sample, Communication with Doctors had a somewhat stronger relationship to global ratings of hospital care in the CT sample. Table 2 presents a comparison of the descending rank order of correlations of the composite scores with the following global ratings: hospital rating, nurses rating, doctors rating, and hospital recommendation. As illustrated in the table, three composites emerge consistently, within both samples, as being among the most highly correlated with the global ratings: Nurse Communication, Nursing Services, and Pain Control.

Table 1: A Comparison of the Patient-Level Psychometric Analyses of the Seven-Factor Hospital-Level Structure between the 3-State Pilot and Connecticut Samples

3 State Pilot Sample
		Integrity of Composites		Relationship of Item and Composite-Level Scores to Hospital Rating and Recommendation**
Quest #	Question Label	Substantial Corr. w 2nd Composite	Alpha & Item-Total Corr.	Hospital Rating		Recommend Hospital
(1) Communication with Nurse			α = .86	aR2=0.47*	t-value= 101.55	aR2=0.37	t-value= 79.02
Q5	RN Listen		.77		38.70		28.59
Q4	RN Respect		.73		48.02		39.35
Q6	RN Explain		.69		21.62		16.22
(2) Communication with Doctors			α = .88	aR2=0.24	t-value= 41.15	aR2=0.19	t-value= 34.67
Q12	MD Listen		.81		13.07		9.20
Q11	MD Respect		.76		21.06		19.13
Q13	MD Explain		.73		1.20#		1.49#
(3) Communication about Medication			α = .67	aR2=0.18	t-value= 12.41	aR2=0.14	t-value= 7.99
Q40	Allergies to Medicines		.51		9.73		5.97
Q41	Side-Effects of Medicine		.51		5.46		3.92
(4) Nursing Services			α = .72	aR2=0.36	t-value= 36.05	aR2=0.28	t-value= 28.67
Q22	How often Bathroom	1(.56)	.56		18.35		16.17
Q9	Help when Call Button	1(.63)	.56		25.22		18.39
(5) Discharge Information			α = .51	aR2=0.08	t-value= 20.33	aR2=0.07	t-value= 19.30
Q49	Symptoms may have		.35		11.49		12.28
Q48	Help for you at home?		.35		14.54		12.01
(6) Pain Control			α = .83	aR2=0.30	t-value= 39.96	aR2=0.24	t-value= 34.16
Q32	Pain Controlled		.71		11.98		9.94
Q33	Pain Help All Can		.71		23.95		20.84
(7) Physical Environment			α = .51	aR2=0.26	t-value= 52.08	aR2=0.19	t-value= 33.19
Q17	Room Clean	1(.43) 4(.44) 6(.34)	.34		45.11		31.94
Q18	Room Quite	1(.35) 4(.39)	.34		22.64		11.21

* aR² = Adjusted R-squared, how much variance in the dependent variable is accounted for by the set of items in the composite controlling for the effect of number of variables (i.e. all things being equal, a larger set of items will account for a larger percentage of the variance.

** t-values listed in grey cells are for the unique relationship of this composite to the criterion variable controlling for the other composites. t-values in cells adjacent to the item are for the unique relationship of that item controlling for the other report items in the questionnaire, therefore these are the same values as those depicted in Table 5. Probability of t-value is less than 0.01 unless otherwise denoted.

α = Cronbach’s alpha coefficient, an estimate of internal consistency reliability.

# = p > 0.01

Connecticut Sample
		Integrity of Composites		Relationship of Item and Composite-Level Scores to Hospital Rating and Recommendation**
Quest #	Question Label	Substantial Corr. w 2nd Composite	Alpha & Item-Total Corr.	Hospital Rating		Recommend Hospital
(1) Communication with Nurse			α = .85	aR2=0.45*	t-value= 26.01	aR2=0.34	t-value= 17.47
Q5	RN Listen		.75		7.57		3.67
Q4	RN Respect		.71		14.81		14.30
Q6	RN Explain		.68		5.65		1.62#
(2) Communication with Doctors			α = .87	aR2=0.30	t-value= 16.80	aR2=0.22	t-value= 11.57
Q12	MD Listen		.80		1.27#		2.57#
Q11	MD Respect		.74		7.92		7.22
Q13	MD Explain		.71		6.42		1.14#
(3) Communication about Medication			α = .69	aR2=0.19	t-value= 2.39#	aR2=0.15	t-value= 1.33#
Q40	Allergies to Medicines		.52		-0.18#		.88#
Q41	Side-Effects of Medicine		.52		3.51		1.54#
(4) Nursing Services			α = .71	aR2=0.35	t-value= 12.08	aR2=0.29	t-value= 13.10
Q22	How often Bathroom	1(.52)	.55		7.28		9.94
Q9	Help when Call Button	1(.61)	.55		7.41		5.42
(5) Discharge Information			α = .50	aR2=0.08	t-value= 5.65	aR2=0.08	t-value= 8.16
Q49	Symptoms may have		.33		1.99#		2.53#
Q48	Help for you at home?		.33		5.23		7.49
(6) Pain Control			α = .81	aR2=0.32	t-value= 16.58	aR2=0.25	t-value= 13.12
Q32	Pain Controlled		.68		7.07		6.07
Q33	Pain Help All Can		.68		8.44		6.45
(7) Physical Environment			α = .51	aR2=0.27	t-value= 19.07	aR2=0.19	t-value= 11.98
Q17	Room Clean	1(.38) 4(.39) 6(.35)	.34		14.17		8.71
Q18	Room Quite	1(.35) 4(.40)	.34		9.79		6.01

* aR² = Adjusted R-squared, how much variance in the dependent variable is accounted for by the set of items in the composite controlling for the effect of number of variables (i.e. all things being equal, a larger set of items will account for a larger percentage of the variance.

** t-values listed in grey cells are for the unique relationship of this composite to the criterion variable controlling for the other composites. t-values in cells adjacent to the item are for the unique relationship of that item controlling for the other report items in the questionnaire, therefore these are the same values as those depicted in Table 5. Probability of t-value is less than 0.01 unless otherwise denoted.

α = Cronbach’s alpha coefficient, an estimate of internal consistency reliability.

# = p > 0.01

Table 2: Comparison of Rank Order (Descending) of Correlations of Composite Scores with Global Ratings between the 3 State Pilot and Connecticut Samples

3 State Pilot
Hospital Rating		Nurses Rating		Doctors Rating		Recommend Hospital
Nurse Com	.68	Nurse Com	.79	Doctor Com	.79	Nurse Com	.60
Nursing Services	.60	Nursing Services	.66	Nurse Com	.48	Nursing Services	.52
Pain	.54	Pain	.54	Pain	.44	Pain	.48
Physical Environ.	.50	Physical Environ.	.47	Nursing Services	.42	Doctor Com	.43
Doctor Com	.48	Doctor Com	.42	Medicine	.37	Physical Environ.	.42
Medicine	.42	Medicine	.41	Physical Environ.	.33	Medicine	.37
Discharge	.28	Discharge	.25	Discharge	.25	Discharge	.26

Connecticut
Hospital Rating		Nurses Rating		Doctors Rating		Recommend Hospital
Nurse Com	.67	Nurse Com	.76	Doctor Com	.80	Nurse Com	.58
Nursing Services	.59	Nursing Services	.67	Nurse Com	.51	Nursing Services	.54
Pain	.55	Pain	.53	Pain	.44	Pain	.48
Doctor Com	.54	Doctor Com	.48	Nursing Services	.44	Doctor Com	.47
Physical Environ.	.51	Physical Environ.	.45	Medicine	.37	Physical Environ.	.43
Medicine	.43	Medicine	.44	Physical Environ.	.36	Medicine	.38
Discharge	.27	Discharge	.25	Discharge	.25	Discharge	.27

1 The 3SP dataset is comprised of data obtained in Arizona, Maryland, and New York.