memo on year-to-year variability

Attachment X: Analyses Addressing
2017 OMB Questions About Year-toYear Variability of Disenrollment
Reason Scores

1

Analyses Regarding Year-to-Year Variability of Disenrollment Reason Scores
As part of 2017 Medicare Advantage and Prescription Drug Plan Disenrollment Reasons Survey
clearance process, OMB made the following request:
“When four years of data are available, CMS will evaluate the within plan temporal
variability in quality scores available to consumers and adjust the frequency of the data
collection accordingly. CMS will look at the temporal and geographic variability in the
distribution of disenrollment reasons across all plans (analyses will include comparisons
at the 10th, 25th, 50th, 75th, 90th percentiles). If there is very little change in the quality
scores across time, CMS will consider collecting the information less frequently.”
CMS asked its contractor, the RAND Corporation, to conduct several analyses to determine
whether the annual collection of the disenrollment survey provides valuable information that
could not be provided through less-frequent fielding. The three analyses RAND conducted were:
(1) Contract-level variability in reasons for disenrollment composite scores over time;
(2) Geographic-level variability in the distribution of disenrollment reasons over time; and
(3) Examining the degree to which disenrollment reasons composites predict future
disenrollment.
RAND focused its analyses on MA-PD contracts, as all five composite reasons for disenrollment
measures that are reported to contracts and beneficiaries on Medicare Plan Finder are captured
by the MA-PD version of the survey.1 The five composite measures of reasons for disenrollment
(and their abbreviated labels) are:
•
•
•
•
•

Financial Reasons (“Financial”)
Problems with Prescription Drug Benefits and Coverage (“Benefits and Coverage”)
Problems with Coverage of Doctors and Hospitals (“Doctors and Hospitals”)
Problems Getting Needed Care, Coverage, and Cost Information (“Patient Experience
with the Plan”)
Problems Getting Information and Help from the Plan (“Patient Experience with Rx
Drugs”)

This memo summarizes the findings of RAND’s analyses.
Analysis #1: Contract-Level Variability in Reasons for Disenrollment Composite Scores
Over Time
To assess how much disenrollment reasons scores for individual contracts change over time,
the RAND team calculated the percentage of variance in the 2017 composite disenrollment
reasons measures across contracts that could not be predicted from the same measure using
data that was three years old (2014), two years old (2015), or one year old (2016). RAND
focused on MA-PD contracts to assess the impact on all five disenrollment reasons composite
measures.

1

The PDP version of the survey does not capture two composite disenrollment reason measures that
relate to medical coverage.

2

Figure 1 plots the percent of each disenrollment measure’s 2017 variance that is not predicted
from the prior year’s data. Larger percentages, represented by longer bars, indicate less ability
for a prior year’s contract-level scores to accurately represent 2017 contract-level disenrollment
reason scores.
For two of the five composite measures shown, less than half of the measures’ variability is
predicted from the previous year’s scores (2016, green bars). For the three remaining
measures, less than one-third of the variance for each is predicted from the prior year’s scores
(2016, green bars). Using older data (from 2015 and 2014), less than half of the measures’
variability in 2017 is predicted by using 2015 or 2015 contract-level scores. These findings
indicate that using prior year’s results as a proxy for current year contract-level disenrollment
reasons omits a substantial amount of information on a contract’s true performance in 2017.
Figure 1: A Large Percentage of 2017 Composite Variances is not Predictable by the
Corresponding Composite Measure in Previous Years (2016, 2015, and 2014)

Looking at a specific contract as an example, imagine that a beneficiary was trying to judge how
often specific reasons for disenrollment applied to Contract X2 in 2017 performance. The true
performance of Contract X in 2017, if that data were available to the beneficiary, was poor for all
non-financial composites and middling for the Financial composite but would not have
been measured accurately using older data. This contract scored better-than-average for all but
one of the non-financial composite measures in 2014; however, its performance began to
decline in 2016 for non-financial reasons measures and was worse-than-average for all
measures except for coverage of Doctors and Hospitals by 2017. Even its 2016 performance
would have given misleadingly-optimistic information to Medicare beneficiaries for three of the
four non-financial composites. Similarly, if the contract’s sponsor only had access to the 2016

2

This example is a real MA plan that has been anonymized.

3

data, the older, less-relevant data could have substantial delayed the sponsor identifying a
problem and designing appropriate quality improvement initiatives.
Analysis #2: Geographic-level Variability in the Distribution of Disenrollment Reasons
Over Time
To examine how composite reasons-for-leaving scores vary geographically and over time, the
RAND team calculated disenrollment reasons composite scores at the state level for calendar
years 2016 and 2017.
State-specific scores were calculated for the five MA-PD composite disenrollment reasons
measures for 2016 and 2017 and then each state was assigned to a category based on its
percentile score relative to all states; using categories defined by OMB in its request for
analyses: <10th%, 10th – 24th%, 25th-49th%, 50th – 74th%, 75th-89th%, and 90th+%. Note that
higher disenrollment reason composite scores, and higher percentile categories indicate worse
performance because they mean that a higher proportion of beneficiaries cite the reasons for
leaving the plan. The 2016 and 2017 scores used 2016 percentile cutoffs for comparability.
To ensure stable measurement, the analysis was limited to states with at least 100
disenrollment survey responses for a given composite measure in both 2016 and 2017, which
left 43 states with the exception of the Benefits and Coverage composite measure, where there
were only 42 states that met the 100-response threshold.3
RAND generated pairs of state maps for each disenrollment reason composite measure, colorcoding states to indicate their composite-score percentile category. When comparing the 2016
to 2017 maps, the color differences illustrate changes. In Appendix A of this memo, we have
included three pairs of maps showing 3 different composite measures as examples: (1)
Financial reasons for leaving, (2) reasons related to Experiences with the Health Plan, and (3)
coverage of Doctors and Hospitals.
To identify “transitions” between percentile categories, RAND cross tabulated the composite
score categories in 2016 and 2017 for each of the five MA composite measures (Table 1, b-f). In
each “transition” table, the shaded diagonal contains the number of states which remained
unchanged (i.e., in the same composite percentile category in each of the two years), while
those in the cells above or below the diagonal contain the number of states in which
beneficiaries endorsed the composites less often (which is better) or more often (which is
worse) in 2017.
More states were below the diagonal for the reasons for disenrollment related to Benefits and
Coverage (1c) and coverage of Doctors and Hospitals (1f) composites, indicating increased
endorsement of those disenrollment reasons composite measures in 2017 compared to 2016.
For the other three composites in Tables 1b, 1d, and 1 e (Financial reasons for Leaving, Patient
Experience with Rx Drugs, and Patient Experience with the Health Plan), more states fell above
the diagonal, indicating a decrease in endorsement of those disenrollment reasons composite
measures in 2017.
Table 1a summarizes the percentile category shifts between 2016 and 2017. Across all
composite measures, between 30% and 35% of the states remained in the same percentile
3

Note, not all states offer Medicare Advantage plans. Alaska has no MA plans.

4

category– the rest of the states moved to a higher or lower percentile category in one year’s
time. These results indicate a substantial amount of change in performance over time, such that
relying on older information would give a poor signal of current performance.
Table 1a: Most states changed percentile categories from 2016 to 2017
Change in
performance
category from
2016 to 2017:
Moved to a
higher
percentile
category
(% of states)

Change in
Change in
performance
performance
category from
Change in
category from 2016 to 2017:
performance
2016 to 2017:
Moved to a
category from
No change in lower percentile 2016 to 2017:
category
category
Number of
(% of states)
(% of states)
states

Financial

30%

30%

40%

43

Benefits and Coverage

36%

33%

31%

42

Patient Experience with Health Plan

33%

30%

37%

43

Patient Experience with Rx Drugs

28%

33%

40%

43

Doctors and Hospitals

35%

35%

30%

43

Notes: The results shown include only those states providing at least 100 disenrollment survey responses in both
2017 and 2016. A higher percentile category indicates that a higher percentage of that state’s disenrollees
endorsed the reasons for disenrollment compared to those living in lower percentile category states

Tables 1b through 1f provide the details of the results summarized in Table 1a.
Table 1b: Financial
2017 performance based on 2016 quantile thresholds

2016
performance
based on
2016 quantile
thresholds

90 - 100th

75 - 89th

50 - 74th

25 - 49th

10 - 24th

0 - 9th

90 - 100th

0

2

0

0

0

0

75 - 89th

0

1

1

2

2

1

50 - 74th

0

2

3

4

1

1

25 - 49th

0

1

3

6

2

0

10 - 24th

0

0

1

3

2

1

0 - 9th

0

0

0

0

3

1

5

Table 1c: Benefits and Coverage
2017 performance based on 2016 quantile thresholds
75 - 89th

50 - 74th

25 - 49th

2
4
5
6
0

1
1
1
5
4

0

0

75 - 89th

0

50 - 74th

0

25 - 49th

0

10 - 24th

0

0
2
1
0
1

0 - 9th

0

1

90 - 100th
2016
performance
based on
2016 quantile
thresholds

90 - 100th
0

10 - 24th
0

0 - 9th
0

0

0

3
1
2

0

2

0

0
0

Table 1d: Patient Experience with Health Plan
2017 performance based on 2016 quantile thresholds

2016
performance
based on
2016 quantile
thresholds

90 - 100th

90 - 100th
0

75 - 89th
1

50 - 74th
1

25 - 49th
0

10 - 24th
0

0 - 9th
0

75 - 89th

0

3

2

3

0

0

50 - 74th

1

2

2

5

1

0

25 - 49th

0

1

4

3

3

0

10 - 24th

0

0

2

1

4

0

0 - 9th

0

1

0

0

2

1

Table 1e: Patient Experience with Rx Drugs
2017 performance based on 2016 quantile thresholds

2016
performance
based on
2016 quantile
thresholds

90 - 100th

90 - 100th
0

75 - 89th
2

50 - 74th
1

25 - 49th
1

10 - 24th
0

0 - 9th
0

75 - 89th

0

0

4

0

3

0

50 - 74th

1

1

6

1

2

0

25 - 49th

0

3

2

3

3

0

10 - 24th

0

0

1

2

3

0

0 - 9th

0

1

0

0

1

2

Table 1f: Doctors and Hospitals
2017 performance based on 2016 quantile thresholds

2016
performance
based on
2016 quantile
thresholds

90 - 100th

90 - 100th
1

75 - 89th
2

50 - 74th
3

25 - 49th
4

10 - 24th
5

0 - 9th
6

75 - 89th

1

1

0

0

0

0

50 - 74th

0

2

5

1

0

0

25 - 49th

0

1

6

5

0

0

10 - 24th

1

0

6

3

0

1

0 - 9th

0

1

2

2

1

0

6

Focusing on specific states further illustrates changes in scores over time. For example, the
coverage of Doctors and Hospitals composite score for Mississippi in 2016 was in the betterthan-average 25th-49th percentile group (low scores / few problems / few reasons for leaving are
good). However, in 2017 Mississippi moved to the worst percentile group (90 th-100th percentile,
indicating it had among the highest rate of problems / reasons for leaving among states for the
coverage of Doctors and Hospitals composite.
The performance of Mississippi on the coverage of Doctors and Hospitals composite measure
could not have been measured accurately by using previous years’ data. Had CMS program
staff used disenrollment reasons data that was even one year older than 2017, they would have
falsely concluded that contracts in the state were doing better-than-average in terms of issues
with coverage of Doctors and Hospitals compared to other states; when in fact Mississippi’s
2017 performance was among the worst in this area.
Analysis # 3: Disenrollment Reasons Composites Predict Changes in Disenrollment
Rates
Helpful to the question of assessing how frequently surveying should occur, is understanding
the relationship between disenrollment reasons cited by beneficiaries and disenrollment rates.
To explore this, RAND used disenrollment survey data for 2016 and 2017 for 345 MA contracts
and modeled contract-specific changes in the disenrollment rates between 2016 and 2017 as a
function of the 2016 contract-level financial reasons composite score and a hybrid contract-level
composite measure which we formed by averaging the 4 non-financial composite scores for
each contract.
A significant (p=0.013) positive association was observed between the average score on the
non-financial composite (how often disenrollees cites nonfinancial reasons for disenrolling) and
changes in contract-level disenrollment rates from 2016 to 2017.
RAND modeled the 2016-2017 change in contract-specific disenrollment rates by predicting the
2017 disenrollment rate from the hybrid non-financial reasons composite, controlling for the
2016 disenrollment rate and the financial reasons composite. They then calculated the expected
change in disenrollment rate for a plan with a median 2016 disenrollment rate (9.7%) according
to whether it had a low (5th percentile), medium (50 th percentile), or high (95th percentile) 2016
nonfinancial composite reasons score. Table 2 presents the expected changes in disenrollment
rate.
A median disenrollment rate contract in 2016 with a disenrollment rate of 9.7% with a typical
(50th percentile, ”medium’) rate of non-financial reasons for 2016 disenrollment would be
expected to see a 0.7 percentage point increase in its disenrollment rate by 2017, resulting in a
10.4% disenrollment rate in 2017. In contrast, a contract with a “low” ( 5th percentile) rate of nonfinancial reasons would expect its 2017 disenrollment rate to drop by 0.8 percentage points to
8.9.%. Finally, a contract with a high (95th percentile) rate of non-financial reasons in 2016
would expect its voluntary disenrollment rate to rise by 2.4 percentage points to 12.1%.

7

Table 2: 2017 Contract Disenrollment Rates Increased When a Contract’s 2016
Disenrollees Cited More Non-Financial Reasons
2016 Non-Financial Reasons Composite Score
Low
5th percentile

Medium
50th percentile

High
95th percentile

2016 disenrollment rate

9.7%

9.7%

9.7%

Estimated change in
disenrollment rate for a
contract at the median
disenrollment rate of 9.7% in
2016

-0.8%

+0.7%

+2.4%

8.9%
(9.7%-0.8%=8.9%)

10.4%
(9.7%+0.7%=10.4%)

12.1%
(9.7%+2.4%=12.1%)

Predicted 2017
Disenrollment Rate

Low, medium, and high levels correspond to the 5th, 50th, and 95th percentiles in the distributions of
2016 non-financial composite scores.

As examples consider two specific contracts “A” and “B,”4 which had similar and slightly high
voluntary disenrollment rates of 11.6% and 12.1%, respectively, in 2016. Contract A had a
worse-than-average non-financial composite score in 2016: its disenrollees endorsed 26.4% of
non-financial reasons. Contract B, on the other hand, had a much-better/lower-than-average
non-financial composite score of 3.8% in 2016, with very little endorsement of non-financial
reasons for leaving. While Contract A’s voluntary disenrollment rate then rose by 4.1 percentage
points to 15.7% in 2017, Contract B’s. voluntary disenrollment rate fell by 6.0 percentage points
to 6.1% in 2017. Contracts that use annual data to address the specific reasons for which their
beneficiaries voluntarily disenroll may be able to reduce future disenrollment.
This analysis shows that knowing the levels of disenrollment reasons from the most recent year
provides useful information to contracts, as it is predictive of how contract disenrollment can be
expected to change in the next year. This information is useful to contracts who can focus
quality improvement efforts to avoid losing members; improvements that contracts make
support Medicare beneficiaries who want good quality contracts. Providing contracts with recent
information on reasons for disenrollment is particularly useful as an early warning to contracts,
who could make changes to improve care and services and reduce beneficiary disenrollment
and churning which is costly to contracts and beneficiaries alike.

4

These are two real contracts that have been anonymized.

8

Appendix 1: State-Level Composite-Score Category Relative to the 2016 Thresholds, Financial
Reasons for Leaving Composite, Patient Experience with the Health Plan Composite, and
Coverage of Doctors and Hospitals Composite

9

10

11