Attachment A - ICH CAHPS Pilot Test Summary Report

Attachment A:
ICH CAHPS Pilot Test Summary Report

ICH CAHPS Pilot Test

Summary Report

Beginning in October 2004, the RAND team drew a sample of 30 facilities in accordance with the criteria identified in the field test protocol. The grantees then worked closely with CMS and the Networks to determine the best way in which to draw an accurate sample of dialysis patients from each of the participating facilities. Because it was close to the Christmas holiday season by the time OMB clearance was received and a strategy for drawing the patient sample reached, the team agreed to start the field test of the instrument in mid-January 2005. Field test operations were completed in early April 2005.

The purpose of the pilot test was to examine the reliability and validity of the draft ICH CAHPS questions, identify those questions most robust for public reporting and those most appropriate for internal quality improvement, and evaluate different data collection strategies (including a comparison of telephone with mixed mode—mail plus telephone follow-up). The pilot test included patients sampled from total 32 facilities. These facilities were drawn from a list with at least 10 annual cases, and which are not more than 50% pediatric. The criteria for facility selection included:

• Region of country (NE, South, Midwest, West)

• Rural versus urban location (RUCA 1, 2–4-, 5–10)

• Size of facility—very small (10–24 patients), small to medium (25–59), medium to large (60–119), and very large (120 or more).

• Facilities that are part of a large dialysis organization (LDO)

• Hospital-based versus non-hospital based facilities (the latter divided into profit non non-profit)

• Racial/ethnic mix of facilities’ patients (i.e., oversampling of high Hispanic facilities to achieve sufficient Spanish completes)

The facilities were allocated to regions proportionately to their number: 11 Midwest, 9 South, 5 Northeast, and 5 West. Two facilities were later added to the original total of 30.

All currently dialyzing in-center hemodialysis patients with at least 3 months of experience on hemodialysis at their current facility were eligible for the survey. The number of patients sampled for the pilot was 3143.

For purposes of the pilot study, for facilities with up to 200 patients, a census of patients were drawn, for facilities with more than 200 patients, a systematic random sample of 200 patients were drawn. Patient lists were drawn from each facility. The sampling frame included: name, address, telephone number, gender, race-ethnicity, total time on in-center hemodialysis at the facility and total time at the current center.

The survey methodology included two conditions to which the patients were randomly assigned by facility:

1. Telephone only

2. Mixed mode (i.e., two mailings with telephone follow-up)

For both modes an introductory letter explaining the purpose of the survey and the conditions of participation was sent out to all sampled patients. For patients surveyed by telephone, CATI was used; up to 10 attempts were made to reach respondents at different times of days and days of the week. In the mixed mode, a survey was mailed out about 1 week after the pre-notification letter was sent. About two weeks after that, a reminder/thank you sealed letter was mailed. A second survey was mailed about four weeks after the initial one was sent. After four weeks, telephone interviews were conducted.

Because of the visual problems, fatigue, literacy, and/or other cognitive difficulties prevalent in this population, and based on information from other patient surveys conducted with dialysis patients, the team estimated a likely response rate of about 40%.

The final number of respondents was 1454, with an overall response rate of 46%; 56% responded by telephone, while 44% responded by mail. The field test showed that the vast majority of items on the survey performed well in the field; based on psychometric criteria, only 4 items were recommended for deletion.

Description	Number	Percent
Total Sample	3,143	—
Phone	1,781	56.7
Mixed Mode	1,362	43.3
Total Completes	1,454	46.3
Completes by phone	818	56.3
Completes by mail	636	43.7
Completes by Mode Assignment	—	—
Phone	695	39.0
Assigned to phone/Completed by phone	655	36.8
Assigned to phone/Completed by mail	40	2.2
Mixed Mode	759	55.7
Assigned to mixed mode/Completed by mail	596	43.8
Assigned to mixed mode/Completed by phone	163	12.0

(continued on next page)

Description	Number	Percent
Total Non-Completes	1,689	54.0
Final Refusal	2	0.1
Refuse to confirm number	35	2.1
Respondent refusal	148	8.8
Household refusal	3	0.2
Break-off	6	0.4
Deceased	98	5.8
Ill or Incapable	160	9.5
Respondent language barrier	29	1.7
Respondent away for duration of study	9	0.5
Ineligible	22	1.3
Strange noise	1	0.1
Field period ended	425	25.2
Wrong number	122	7.2
Computer line	12	0.7
Not in service	284	16.8
Temporarily not in service	41	2.4
Unpublished	47	2.8
Mail survey/Ineligible	17	1.0
Mail survey/Deceased	9	0.5
Mail survey/Ill or Incapable	1	0.1
Mail survey/Undeliverable	218	12.9

The instrument team will revise the survey in accordance with the findings of the analysis team, and feedback from Networks and facilities involved in a quality improvement project based on the results obtained from the pilot test of the survey.

Findings

The nine composite structure suggested by the CAHPS survey topic headings to measure the experience of In-Center Hemodialysis (CH) patients was not confirmed by statistical analysis. Both a three- and four-composite structure were confirmed. Structures with an intermediate number of composites were also not confirmed. The measurement properties of the three-composite structure were better than those of the four-composite structure.

Background

The Agency for Healthcare Research and Quality (AHRQ) and the Centers for Medicare & Medicaid Services (CMS) co-sponsored the development of a CAHPS survey designed to measure quality of care rendered by in-center hemodialysis ICH) facilities via patient surveys (ICH-CAHPS). A total of 3,143 hemodialysis patients were randomly selected from 32 facilities that participated in a field test of this survey. RAND collected the survey data and created the SAS data set used in the analysis of the field test findings. Analyses are based on the total final updated SAS data file observations (N = 1454) sent to AIR by RAND on the evening of May 16, 2005. The purpose of the analyses reported was to assess the general psychometric properties of the 41 report items without regard to subgroups. Thus, analyses according to subgroups defined by mode of administration (telephone versus mixed mode) or language of survey (English versus Spanish) are not reported here.

Analytic Methods

We used both confirmatory and exploratory methods to describe the structure underlying responses to the 41 quality report items. After the structure was described we evaluated the measurement properties of the composites therein according to classical psychometric methods (restriction of range, item-total correlations, internal consistency reliability, and power of item and composite scores to discriminate among facilities).

Confirmation of hypothesized composite structure. Observations were computed as the means of five imputations calculated by SAS PROC MI, rounded to the nearest legitimate whole number. We compared the correlation of each item to the composite it was hypothesized to belong to, with its correlations to all the scales that it was not hypothesized to belong to. In this analysis, our concern was with the Type II error (that we might prematurely reject a possible structure) and so we used liberal criteria in determining whether the structure was confirmed. We deemed as acceptable, items that were more highly correlated with their scale whether or not this correlation was significantly higher.

Identification of composite structure. In the absence of confirmation for the hypothesized structure, we used standard CAHPS methods for conducting the exploratory factor analysis” principal factor analysis, with oblique (Promax) rotation to simple structure. The covariance matrix was computed using a maximum likelihood procedure to estimate variance for missing data (the multiple imputation procedure from SAS with the EM function). The default method in SAS for identifying the number of composites in the structure underlying item responses is the proportion criterion. Because this criterion is controversial, we also examined the scree plot of the eigenvalues.

Evaluation of Composite Measure Properties. For each composite, we computed the item-total correlations, the percent of respondents at the highest- and lowest- possible composite score, and Cronbach’s α (an estimate of internal consistency and reliability). For each item and composite, we estimated the ability of the score to discriminate among facilities using standard CAHPS methods for computing unit-level (in this case, center-level) reliability. Center-level reliability is based on the F-statistic for testing differences among centers on an item or composite. We used the CAHPS macro (version 34b), with unimputed data as input, to calculate the global F-test, which is intended to determine whether there is evidence for differences among center means. The F-statistic provides a summary of the ratio of between-center variance (numerator) and with-center variance (denominator). The more real differences there are among centers, relative to random variation, the larger the F-statistic is expected to be. We transformed the F value using the formula (f-1)/F in order to obtain the inter-unit reliability (IUR). The IUR score indicates the fraction of the variation among center scores that is due to real differences rather than chance.

Results

Here we describe our results in the chronological order of the structures that we evaluated so that the reader can follow our logic.

Nine composites (number of items, “i” = 41). We conducted confirmatory analysis of the composite structure defined by the ICH CAHPS questionnaire headings using the procedure described in the second paragraph under Analytic Methods.

• Kidney Doctor communication (i = 7)

• Staff Communication (i = 8)

• Privacy (i = 2)

• Complaints (i = 3)

• Staff Professionalism (i = 6)

• Patient Involvement (i = 6)

• Patient Education (i = 3)

• Patient Safety (i = 6)

• Patient Rights (i = 2)

The nine composite structure was not supported. Seven of the nine composites contained items that correlated more highly with a competing scale than their own. Since the structure was not confirmed, we do not discuss measurement properties of these composites but they are summarized in Table 1. We also found that almost everyone responded that they had not made a complaint to item Q53 “in the last 12 months, did you make a complaint to any …agencies?” Due to the lack of variability, this item was not included in subsequent analyses.

Three composites. (i = 40) We conducted exploratory factor analysis (EFA) to identify an alternative to the failed 9 composite structure. The scree plot of the eigenvalues supported this three composite solution.

• Kidney Doctor Communication (i = 7)

• Dialysis Facility Care and Operations (i = 22)

• Patient Empowerment (i = 11)

We then evaluated the measurement properties of the composite structure, which were found to be quite good. The item total correlations for Kidney Doctor Communication were all above 0.40. Nineteen of the 22 item-total correlations for Dialysis Facility Care and Operations were above 0.40. Six of the 11 item-total correlations for Patient Empowerment were above 0.40. Internal consistency reliabilities for the three scales ranged from 0.75 to 0.93. Thirty percent of respondents received the highest score on the Kidney Doctor Communication composite, but less than 10% were at the ceiling on the other two composites. Although the Kidney Doctor Communication composite will not be especially good at differentiating among the higher quality facilities, the other two composites will provide information on those facilities.

Summary. In Table 1 we summarize the results above and in Table 2, we present detailed results at the item and composite level of the psychometric analyses related to the three-composite structure identified by EFA.

Table 1. Psychometric Evaluation of Alternative Composite Structures

Psychometric Criteria	Number of Composites
	9	6	5	4	3
Composite Structure Confirmed?	N	N	N	Y	Y
Range of Possible Composite Scores(α)1	0%	33%	40%	25%	66%
Internal Consistency of Composite2	44%	83%	80%	75%	100%
Composite Score Reliability3	100%	100%	100%	100%	100%

Table Key: “Y” is better than “N”. For percents, higher is better than lower.

¹Percentage of scales with <10% of respondents at the ceiling.

²Percentage of scales with internal consistency reliabilities >0.70

³Percentage of scales with ICH-reliability >0.50

Table 1 demonstrates that the 3-composite structure had the best measurement properties overall. Table 2 provides more detail on that structure.

Table 2a. ICH CAHPS—Field Test Data: 3-Composite Structure—Kidney Doctor

Q#	Kidney Doctor	ICHr −0.64	%@Ceiling = 30.1	Alpha = 0.89
Q3	Dr. Listen	0.68		0.78
Q4	Dr. Explain	0.58		0.74
Q5	Dr. Respect	0.65		0.76
Q6	Dr. Spend Time	0.70		0.75
Q7	Dr. Cared	0.69		0.78
Q8	Dr. Keep You Informed	0.60		0.61
Q10	Dr. Informed	0.61		0.56

Table 2b. ICH CAHPS—Field Test Data: 3-Composite Structure—Dialysis Facility Care & Ops

Q#	Dialysis Facility Care & Ops	ICHr = 0.51	%@Ceiling = 2.7	Alpha = 0.93
Q11	Staff Listen	0.62		0.77
Q12	Staff Explain	0.64		0.74
Q13	Staff Respect	0.67		0.77
Q14	Staff Spend Time	0.68		0.76
Q15	Staff Cared	0.75		0.79
Q16	Staff Make you Comfortable	0.70		0.75
Q17	Staff Keep Info Private	014		0.37
Q19	Staff Cover You	0.70		0.48
Q20	Comfortable Asking Staff	0.31		0.49
Q24	Staff Insert Needle w/o Pain	0.37		0.44
Q25	Staff Check You	0.62		0.70
Q27	Staff Respond to Problems	0.43		0.76
Q28	Staff Manage Problems	0.03		0.70
Q29	Staff Professional	0.72		0.72
Q30	Staff Change Gloves	**		0.33
Q31	Staff Explain Tests	0.65		0.56
Q32	Staff Discuss Diet	0.39		0.37
Q38	Staff Include Family	0.50		0.46
Q40	On Machine within 15 minutes	0.86		0.46
Q41	Center calm and quiet	0.67		0.53
Q42	Center Clean	0.76		0.50
Q51	Satisfied with Problems Handled	**		0.71

Table 2c. ICH CAHPS—Field Test Data: 3-Composite Structure—Patient Empowerment

Q#	Patient Empowerment	ICHr = 0.53	%@Ceiling = 8.7	Alpha = 0.75
Q22	Know How to Care for Self	0.29		0.33
Q33	Staff Give Patient Rights	0.60		0.39
Q34	Staff Review Patient Rights	0.58		0.46
Q35	Staff Talk About Your Health	0.38		0.44
Q36	Know How to Get off machine	0.80		0.23
Q52	Info about Complaints Process	0.65		0.43
Q21	Staff Ask about How ESRD Affect Life	0.58		0.35
Q44	Dr./Staff Talk about Treatment	0.53		0.56
Q46	Dr./Staff Explain Transplant	0.30		0.46
Q47	Dr./Staff Talk about Peritoneal	0.68		0.33
Q48	You’re Involved in Choosing Treatment	0.44		0.45