Mode Experiment_Patient_Mix_Adjustment

Attachment E:
ICH CAHPS Mode Experiment and Patient Mix Adjustment

E-1

ICH CAHPS Mode Experiment and Patient Mix Adjustment

In center hemodialysis (ICH) facilities and their contracted ICH CAHPS Survey vendors
can administer the ICH CAHPS Survey using one of three approved data collection modes: mailonly mode, telephone-only mode, and mixed mode (mail with telephone follow-up of mail
survey nonrespondents). For patients to be able to use the ICH CAHPS Survey results to make
objective and meaningful comparisons between dialysis facilities, adjustments may be needed to
account for significant sources of bias in the survey results that are outside the control of the
dialysis facilities. Potential sources of bias include the data collection mode; variability in
various patient-mix factors such as demographic characteristics, disease status, and health status;
and variation in response propensity across patients within dialysis facilities.
To address these issues, CMS conducted a randomized mode experiment from January
through April 2014 with a sample of patients receiving ICH to determine whether they respond
differently to the ICH CAHPS Survey based on data collection mode, patient mix, and
nonresponse propensity factors. The results from the mode experiment analysis will be used to
adjust comparative ICH CAHPS Survey results before they are publicly reported for ICH
facilities on Dialysis Facility Compare on the Medicare website.
The sample frame for the ICH CAHPS Survey mode experiment consisted of all patients
who met survey eligibility criteria—patients must have been 18 years of age or older at the end
of the sampling window and must have received in-center dialysis care from their current ICH
facility for 3 months or longer. Those known to be deceased were excluded from the sampling
frame. The sample was selected using a two-stage sampling process. The first stage included
developing regional strata constructed from the 50 states and U.S. territories in which Medicarecertified ICH facilities are located. These strata were only needed for the mode experiment and
are not used for national implementation. The second stage of sampling for the mode experiment
was to select patients within each stratum using a simple random sample. The sample size in
each stratum was proportional to the number of ICH patients in that stratum.
The goal of the sampling process for the mode experiment was to obtain approximately
1,570 respondents for each of the three modes: mail-only mode, telephone-only mode, and mixed
mode (mail with telephone follow-up). Given expected response rates for each mode, sample
sizes of 4,800 patients for mail only, 3,733 for phone, and 3,360 for mixed mode were identified
to achieve the planned 1,570 responses for each mode. Therefore, a total of 11,893 eligible

E-2

ICH CAHPS Mode Experiment and Patient Mix Adjustment

patients were sampled from the ICH patient population. After the sample was selected, the
sampled patients were randomly assigned to one of the three data collection modes using the
inverse of the estimated response rates. The number of respondents for each mode allowed a 5
percentage point difference to be detected with 80% power with an alpha-level of 0.05.
A total of 3,557 surveys were completed, including 1,355 for the mail-only mode, 994 for
telephone-only mode, and 1,208 for the mixed mode. After adjusting for sampled patients found
to be ineligible for the ICH CAHPS Survey, an overall response rate of 33.6% was achieved. The
response rates for the mail-only and telephone-only modes were similar at 30.4% and 30.7%,
respectively. The response rate for the mixed mode was higher at 41.8%.
Mode Experiment Analyses
The mode experiment analyses were conducted in two stages. The first stage involved the
mode and patient-mix analyses. The second stage included the nonresponse analyses and an
assessment of whether the nonresponse analysis contributed explanatory power to the ICH
CAHPS Survey results beyond the explanatory power already accounted for by the mode and
patient-mix analyses.
The mode and patient-mix analyses included two types of variables. These were
dependent variables that represented the outcome measures calculated from the ICH CAHPS
Survey data and independent variables that represented factors that may affect the dependent
variables. The six dependent variables included three variables calculated from individual ICH
CAHPS Survey items (global ratings) and three variables calculated from multiple survey items
(composite measures):
• Global rating of nephrologist
• Global rating of dialysis center staff
• Global rating of dialysis center
• Nephrologists’ communication and caring composite
• Quality of dialysis center and operations composite
• Providing information to patients composite
The method used for coding the global rating measures was the same for all three global
ratings, and applied a binary, “top box” approach. This means that each of these global ratings
was coded as 1 if the respondent’s rating was 9 or 10, and 0 if the respondent’s rating was any
E-3

ICH CAHPS Mode Experiment and Patient Mix Adjustment

score from 0 to 8. The scoring for all three of the composite measures was calculated as the
proportion of times the patient’s response included the most positive response category for the
ICH CAHPS Survey questions included in that composite.
The 21 independent variables were those hypothesized to affect the outcome measures
represented by the dependent variables, and that could be considered for use as patient-mix
adjusters for ICH CAHPS scores for ICH facilities. They included the following:
• Mode of survey administration
• Duration at dialysis center
• Overall health
• Overall mental health
• High blood pressure
• Diabetes
• Heart disease
• Deaf or serious difficulty hearing
• Blind or serious difficulty seeing
• Difficulty concentrating, remembering, or making decisions
• Difficulty walking or climbing stairs
• Difficulty dressing or bathing
• Difficulty doing errands alone
• Age
• Gender
• Education
• How well the patient speaks English
• Does the patient speak a language other than English at home
• Did someone help the patient complete this survey
• Total years on dialysis
• Was diabetes the primary cause of end stage renal disease (ESRD)
Multivariate regression analysis models were used to assess the degree to which the
outcome measures represented by the dependent variables were affected by the mode and patient
mix characteristics represented by the independent variables. Six multivariate regression models
were estimated for the mode and patient mix analysis, including one regression model for each of
E-4

ICH CAHPS Mode Experiment and Patient Mix Adjustment

the six dependent variables. Each of the regression models was initially estimated using all 21 of
the independent variables. Independent variables that were not statistically significant for any of
the six regression models were then dropped from the regression models in four steps, in small
increments of two or three variables per step, in sequential estimations of the regression models
until the remaining independent variables were all statistically significant in at least one of the
six regression models. The regression models for all six dependent variables were then finalized
to include all of the independent variables that were statistically significant in at least one of the
six regression models. These were the independent variables identified as the case mix adjusters
for public reporting of the ICH CAHPS outcomes measures for individual ICH facilities.
The nonresponse analyses included four independent variables that were available for
both survey respondents and nonrespondents: age, sex, diabetes as the cause of ESRD, and total
years on dialysis. The nonresponse analyses included two stages. The first stage included logistic
regression analysis to model survey response propensity using the independent variables. This
logistic regression analysis identified the independent variables that significantly affected the
binary dependent variable that represented survey response or nonresponse status. Nonresponse
weights were then calculated from the results of the logistic regression analysis for all of the ICH
CAHPS Survey respondents.
The second stage analyzed the correlations between the nonresponse weights calculated
for the survey respondents and the residuals from each of the six mode and patient mix
regression analysis models for each respondent. If one or more of correlation coefficients was
statistically significant, then the nonresponse weights would be found to add some explanatory
power to the analysis of the ICH CAHPS Survey outcome measures. However, if none of the
correlations were significant, then the mode and patient-mix analysis would be found sufficient
for case mix adjustment of the ICH CAHPS Survey results, because the nonresponse weights
would not add any explanatory power.
Mode and Patient-Mix Analyses Results
A total of 14 independent variables were found statistically significant in at least one of
the six mode and patient-mix regression models:
• Mode of survey administration
• Overall health
E-5

ICH CAHPS Mode Experiment and Patient Mix Adjustment

• Overall mental health
• Heart disease
• Deaf or serious difficulty hearing
• Blind or serious difficulty seeing
• Difficulty concentrating, remembering, or making decisions
• Difficulty dressing or bathing
• Age
• Sex
• Education
• Does the patient speak a language other than English at home
• Did someone help the patient complete this survey
• Total years on dialysis

These 14 independent variables were identified for use as mode and patient-mix adjusters
for the ICH CAHPS Survey results to be publicly reported for ICH facilities. The number of
independent variables significant in any one model varied from a high of 10 variables for the
model for the quality of dialysis center care and operations composite dependent variable to a
low of 4 variables for the model for the nephrologists global rating dependent variable. These
differences in the number of significant variables were expected because the dependent variables
represent a range of different outcomes, some of which are more affected by these 14
independent variables than others. The same set of 14 independent variables was used in all six
of the final mode and patient mix regression models for ease of use in application of these results
for adjusting ICH CAHPS scores for individual dialysis facilities.
The results of the logistic regression analysis conducted for the nonresponse analyses
found three independent variables to be significant predictors of nonresponse in the ICH CAHPS
Survey mode experiment, including age, diabetes as the cause of ESRD, and total years on
dialysis. These logistic regression results were then used to calculate nonresponse weights for
each of the ICH CAHPS Survey respondents.
In the six correlation analyses of the nonresponse weights for the survey respondents with
the residuals for those same survey respondents calculated in the six mode and patient mix

E-6

ICH CAHPS Mode Experiment and Patient Mix Adjustment

regression models, none of the correlation coefficients were found to be statistically significant.
As a result, the nonresponse analysis was not found to add any explanatory power for the ICH
CAHPS Survey results beyond the explanatory power provided by the mode and patient mix
analysis variables.
Next Steps
The ICH CAHPS outcome measure scores will be adjusted for public reporting for the
individual ICH facilities using the results of the mode and patient mix analyses. Nonresponse
adjustments will not be conducted because the nonresponse analyses did not add any explanatory
power beyond that provided by the mode and patient mix analyses. The variables applied to the
ICH CAHPS score adjustments will be those included as independent variables in the final mode
and patient mix regression models. They include the 14 variables listed above.
The ICH CAHPS outcome measure score adjustments will be made using the coefficients
for the 14 independent variables included in each of the six final regression models for the ICH
CAHPS outcome measures. The complete set of regression models is available in a companion
ICH CAHPS report.1 The score adjustments are made sequentially to the raw ICH CAHPS
scores, first for the patient mix variables and then for the mode variable. The patient mix variable
coefficients will be re-estimated for each reporting period based on the relationship observed for
those variables and the ICH CAHPS outcomes in that period. The mode variable coefficients will
be taken from the mode experiment results.

1

Trisolini M., Brown G., Hendershott A., Augustine C., Ingber M., Lynch J., & J. Butler. (2014). Mode experiment
report, final report, In-Center Hemodialysis CAHPS Survey. Prepared for the U.S. Centers for Medicare and
Medicaid Services, Contract No. HHSM 500-2012-0000187. Research Triangle Park, North Carolina: RTI
International.

E-7