memo on year-to-year variability

Analyses Regarding Year-to-Year Variability of Disenrollment Reason Scores
As part of 2017 Medicare Advantage and Prescription Drug Plan Disenrollment Reasons Survey
clearance process, OMB made the following request:
“When four years of data are available, CMS will evaluate the within plan temporal
variability in quality scores available to consumers and adjust the frequency of the data
collection accordingly. CMS will look at the temporal and geographic variability in the
distribution of disenrollment reasons across all plans (analyses will include comparisons
at the 10th, 25th, 50th, 75th, 90th percentiles). If there is very little change in the quality
scores across time, CMS will consider collecting the information less frequently.”
CMS’s contractor for the survey, the RAND Corporation, completed the requested analyses and
submitted the results of those analyses with the 2019 OMB submission. RAND has repeated
those analyses and provides updated results. The purpose of the analyses was to determine
whether the annual collection of the disenrollment survey provides valuable information that
could not be provided through less-frequent fielding. The three analyses RAND conducted
were:
(1) Contract-level variability in reasons for disenrollment composite scores over time;
(2) Geographic-level variability in the distribution of disenrollment reasons over time; and
(3) Examining the degree to which disenrollment reasons composites predict future
disenrollment.
RAND focused its analyses on MA-PD contracts, as all five composite reasons for disenrollment
measures that are reported to contracts and beneficiaries on Medicare Plan Finder are captured
by the MA-PD version of the survey. 1 The five composite measures of reasons for
disenrollment are:
•
•
•
•
•

Financial Reasons
Problems with Prescription Drug Benefits and Coverage
Problems with Coverage of Doctors and Hospitals
Problems Getting the Plan to Provide and Pay for Needed Care
Problems Getting Information and Help from the Plan

This memo summarizes the findings of RAND’s analyses.
Analysis #1: Contract-Level Variability in Reasons for Disenrollment Composite Scores
Over Time
To assess how much disenrollment reasons scores for individual contracts change over time,
the RAND team calculated the percentage of variance in the 2021 composite disenrollment
reasons measures (the most recent year available) across contracts that could not be predicted
from the same measure using data that was three years old (2018), two years old (2019), or one
year old (2020). RAND focused on MA-PD contracts to assess the impact on all five
disenrollment reasons composite measures.

The PDP version of the survey does not capture two composite disenrollment reason measures that
relate to medical coverage.

1

1

Figure 1 plots the percent of each disenrollment measure’s 2021variance that is not predicted
from the prior year’s data. Larger percentages, represented by longer bars, indicate less ability
for a prior year’s contract-level scores to accurately represent 2021 contract-level disenrollment
reason scores.
Less than half of the variability in the Problems Getting the Plan to Provide and Pay for Needed
Care composite in 2021 was predictable from the prior year’s scores (2020, green bars. For
three of the remaining composites, between 50 and 60% of the 2021 scores’ variability was
predictable using 2020 data, and for the Problems with Getting Information and Help from the
Plan composite, approximately 70% of the 2021 variance was predictable from the 2020 data.
Using older data (2018, purple bars and 2019, blue bars), less than half of the 2021 variability
was predictable using 2018 or 2019 scores, and for the Problems with Prescription Drug
Benefits and Coverage composite, less than 20% of the variance was predictable using 2018
data, and less than 10% using 2019.
Figure 1: A Large Percentage of 2021 Composite Variances is not Predictable by the
Corresponding Composite Measure in Previous Years (2020, 2019, and 2018)

Looking at a specific contract as an example, imagine that a beneficiary was trying to judge how
often specific reasons for disenrollment applied to Contract X2 in 2021 performance. The true
2

This example is a real MA plan that has been anonymized.

2

performance of Contract X in 2021, if that data were available to the beneficiary, was poor for
the Problems with Prescription Drug Benefits and Coverage and Financial Reasons composite
but above average for the Problems with Coverage of Doctors and Hospitals composite.
However, this pattern was not reflected in the contract’s performance in prior years. This
contract scored in the lowest performance category for the Problems with Coverage of Doctors
and Hospitals in 2018, but steadily improved from 2019 to 2021. In contrast, the contract had
middling performance in 2018 for both the Problems with Prescription Drug Benefits and
Coverage and the Financial Reasons composites but had declined to the lower performance
categories of these composites by 2021. Even this contract’s 2020 performance would have
given especially misleadingly optimistic information to Medicare beneficiaries for the Problems
with Drug Benefits and Coverage composite, which declined sharply from 2020 to 2021. If the
contract’s sponsor only had access to the 2020 data, the older, less-relevant data could have
substantially delayed the sponsor from identifying a problem and designing appropriate quality
improvement initiatives.
Analysis #2: Geographic-level Variability in the Distribution of Disenrollment Reasons
Over Time
To examine how composite reasons-for-leaving scores vary geographically and over time, the
RAND team calculated disenrollment reasons composite scores at the state level for calendar
years 2020 and 2021.
State-specific scores were calculated for the five MA-PD composite disenrollment reasons
measures for 2020 and 2021 and then each state was assigned to a category based on its
percentile score relative to all states; using categories defined by OMB in its request for
analyses: <10th%, 10th – 24th%, 25th-49th%, 50th – 74th%, 75th-89th%, and 90th+%. Note that
higher disenrollment reason composite scores, and higher percentile categories indicate worse
performance because they mean that a higher proportion of beneficiaries cite the reasons for
leaving the plan. The 2020 and 2021 scores used 2020 percentile cutoffs for comparability.
To ensure stable measurement, the analysis was limited to states with at least 100
disenrollment survey responses for a given composite measure in both 2020and 2021, which
left 39 states that met the 100-response threshold. 3
RAND generated pairs of state maps for each disenrollment reason composite measure, colorcoding states to indicate their composite-score percentile category. When comparing the 2020
to 2021 maps, the color differences illustrate changes. In Appendix A of this memo, we have
included three pairs of maps showing 3 different composite measures as examples: (1)
Financial Reasons for leaving, (2) reasons related to Problems Getting the Plan to Provide and
Pay for Needed Care, and (3) Problems with Coverage of Doctors and Hospitals.
To identify “transitions” between percentile categories, RAND cross tabulated the composite
score categories in 2020 and 2021 for each of the five MA composite measures (Table 1, b-f).
In each “transition” table, the shaded diagonal contains the number of states which remained
unchanged (i.e., in the same composite percentile category in each of the two years), while
those in the cells above diagonal contain the number of states in which beneficiaries endorsed

3

Note, not all states offer Medicare Advantage plans. Alaska has no MA plans.

3

the composites less often (which is better) in 2021 than in 2020, while those below the diagonal
endorsed the composites more often (which is worse) in 2021 than in 2020.
More states were below the diagonal for the reasons for disenrollment related to the Problems
with Getting Information and Help from the Plan (1e) composite, indicating increased
endorsement of that disenrollment reasons composite measure in 2021 compared to 2020. For
the other four composites in Tables 1b, 1c, 1d, and 1f (Financial Reasons, Problems with
Prescription Drug Benefits and Coverage Problems Getting the Plan to Provide and Pay for
Needed Care, and Problems with Coverage of Doctors and Hospitals more states fell above the
diagonal, indicating a decrease in endorsement of those disenrollment reasons composite
measures in 2021 compared to 2020.
Table 1a summarizes the percentile category shifts between 2020 and 2021. Across all
composite measures, between 23% and 49% of the states remained in the same percentile
category– the rest of the states moved to a higher or lower percentile category in one year’s
time. These results indicate a substantial amount of change in performance over time, such
that relying on older information would give a poor signal of current performance.
Table 1a: Most states changed percentile categories from 2020 to 2021

Financial Reasons
Problems with Prescription Drug
Benefits and Coverage
Problems Getting the Plan to Provide
and Pay for Needed Care
Problems with Getting Information and
Help from the Plan
Problems with Coverage of Doctors and
Hospitals

Moved to a
higher percentile
(% of states)

No change
in category
(% of states)

Moved to a
lower percentile
category
(% of states)

3%

38%

59%

36%

23%

41%

23%

49%

28%

38%

36%

26%

18%

46%

36%

Notes: The results shown include the 39 states which provided at least 100 disenrollment survey
responses on each of the 5 composite measures in both 2021 and 2020. A higher percentile
category indicates that a higher percentage of that state’s disenrollees endorsed the reasons for
disenrollment compared to those living in lower percentile category states.

4

Tables 1b through 1f provide the details of the results summarized in Table 1a.
Table 1b: Financial Reasons
2021 performance based on 2020 quantile thresholds

2020
performance
based on
2020 quantile
thresholds

90 - 100th
75 - 89th
50 - 74th
25 - 49th
10 - 24th
0 - 9th

90 - 100th
0
0
0
0
0
0

75 - 89th
2
2
0
0
0
0

50 - 74th
1
4
2
0
0
0

25 - 49th
0
2
6
5
0
1

10 - 24th
0
0
1
3
4
0

0 - 9th
0
0
0
3
1
2

Table 1c: Problems with Prescription Drug Benefits and Coverage
2021 performance based on 2020 quantile thresholds

2020
performance
based on
2020 quantile
thresholds

90 - 100th
75 - 89th
50 - 74th
25 - 49th
10 - 24th
0 - 9th

90 - 100th
1
1
3
2
0
0

75 - 89th
0
1
1
0
0
0

50 - 74th
1
2
1
3
0
1

25 - 49th
0
1
3
1
2
0

10 - 24th
1
1
3
3
5
1

0 - 9th
0
0
0
1
0
0

Table 1d: Problems Getting the Plan to Provide and Pay for Needed Care
2021 performance based on 2020 quantile thresholds

2020
performance
based on
2020 quantile
thresholds

90 - 100th
75 - 89th
50 - 74th
25 - 49th
10 - 24th
0 - 9th

90 - 100th
2
1
0
0
0
0

75 - 89th
1
6
2
1
0
0

50 - 74th
0
0
4
1
0
0

25 - 49th
1
1
4
7
2
2

10 - 24th
0
0
0
0
0
0

0 - 9th
0
0
1
0
3
0

Table 1e: Problems with Getting Information and Help from the Plan
2021 performance based on 2020 quantile thresholds

2020
performance
based on
2020 quantile
thresholds

90 - 100th
75 - 89th
50 - 74th
25 - 49th
10 - 24th
0 - 9th

90 - 100th
1
2
0
0
1
0

75 - 89th
1
2
5
1
0
0

50 - 74th
2
1
3
3
0
0

25 - 49th
0
1
2
8
0
2

10 - 24th
0
1
0
0
0
1

0 - 9th
0
0
1
0
1
0

5

Table 1f: Problems with Coverage of Doctors and Hospitals
2021 performance based on 2020 quantile thresholds

2020
performance
based on
2020 quantile
thresholds

90 - 100th
75 - 89th
50 - 74th
25 - 49th
10 - 24th
0 - 9th

90 - 100th
2
2
0
0
0
0

75 - 89th
2
2
2
0
0
0

50 - 74th
0
3
6
2
0
1

25 - 49th
0
1
3
3
0
0

10 - 24th
0
0
0
4
4
0

0 - 9th
0
0
0
0
1
1

Focusing on specific states further illustrates changes in scores over time. For example, the
Problems with Coverage of Doctors and Hospitals composite score for New Hampshire in 2020
was in the best 0-9th percentile group (lowest scores / fewest problems / fewest reasons for
leaving are good). However, in 2021 New Hampshire moved to a worse-than-median group
(50th-74th percentile), indicating it had a higher rate of problems / reasons for leaving than most
of the other states for the Problems with Coverage of Doctors and Hospitals composite.
Another example is the state of Connecticut, which in 2020 had one of the worst (highest scores
/ most problems / most reasons for leaving are bad) scores for the Problems Getting the Plan to
Provide and Pay for Needed Care composite. In the following year, however, Connecticut’s
score had improved, moving into a better-than-median group (25th to 49th percentile), indicating
that it showed a lower rate of problems / reasons for leaving in 2021 than at least half of the
other states for this composite.
The performance of these states’ composite measure scores could not have been measured
accurately by using previous years’ data. Had CMS program staff used disenrollment reasons
data that was even one year older than 2021, they would have falsely concluded that contracts
in the New Hampshire were doing much better than most states in terms of Problems with
Coverage of Doctors and Hospitals when in fact they were not. Program staff might also have
concluded that contracts in the state of Connecticut were scoring very poorly on Problems
Getting the Plan to Provide and Pay for Needed Care when in fact they were scoring better than
contracts in most other states in 2021.
Analysis # 3: Disenrollment Reasons Composites Predict Changes in Disenrollment
Rates
Helpful to the question of assessing how frequently surveying should occur, is understanding
the relationship between disenrollment reasons cited by beneficiaries and disenrollment rates.
To explore this, RAND used disenrollment survey data for 2020 and 2021 for 395 MA contracts
and modeled contract-specific changes in the disenrollment rates between 2020 and 2021 as a
function of the 2021 contract-level financial reasons composite score and a hybrid contract-level
composite measure which we formed by averaging the 4 non-financial composite scores for
each contract.
A significant (p<0.0001) positive association was observed between the average score on the
non-financial composite (how often disenrollees cites nonfinancial reasons for disenrolling) and
changes in contract-level disenrollment rates from 2020 to 2021.

6

RAND modeled the 2020-2021 change in contract-specific disenrollment rates by predicting the
2021 disenrollment rate from the hybrid non-financial reasons composite, controlling for the
2020 disenrollment rate and the financial reasons composite. They then calculated the
expected change in disenrollment rate for a plan with a median 2020 disenrollment rate (9.7%)
according to whether it had a low (5th percentile), medium (50th percentile), or high (95th
percentile) 2020 nonfinancial composite reasons score. Table 2 presents the expected changes
in disenrollment rate.
A median disenrollment rate contract in 2020 with a disenrollment rate of 13.0% with a typical
(50th percentile, ”medium’) rate of non-financial reasons for 2020 disenrollment would be
expected to see a 1.9 percentage point increase in its disenrollment rate by 2021, resulting in a
14.9% disenrollment rate in 2021. In contrast, a contract with a “low” ( 5th percentile) rate of
non-financial reasons would expect its 2021 disenrollment rate to drop by 0.2 percentage points
to 11.8.%. Finally, a contract with a high (95th percentile) rate of non-financial reasons in 2020
would expect its voluntary disenrollment rate to rise by 4.8 percentage points to 17.8%.
Table 2: 2021 Contract Disenrollment Rates Increased When a Contract’s 2020 Disenrollees
Cited More Non-Financial Reasons
2020 Non-Financial Reasons Composite Score
5th

Low
percentile

Medium
percentile

50th

95th

High
percentile

2020 median
disenrollment rate

13.0%

13.0%

13.0%

Estimated change in
disenrollment rate for a
contract at the median
disenrollment rate 13.0%
in 2020

-0.2%

+1.9%

+4.8%

12.8%

14.9%

17.8%

(13.0%-0.2%=12.8%)

(13.0%+1.9%=14.9%)

(13.0%+4.8%=17.8%)

Predicted 2021
Disenrollment Rate

Low, medium, and high levels correspond to the 5th, 50th, and 95th percentiles in the distributions of 2020
non-financial composite scores.

As examples consider two specific contracts “A” and “B,” 4 which had similar and slightly high
voluntary disenrollment rates of 13.6% and 13.0%, respectively, in 2020. Contract A had a
worse-than-average non-financial composite score in 2020: its disenrollees endorsed 24.1% of
non-financial reasons. Contract B, on the other hand, had a much-better/lower-than-average
non-financial composite score of 7.4% in 2020, with very little endorsement of non-financial
reasons for leaving. While Contract A’s voluntary disenrollment rate then rose by 4.6 percentage
points to 18.2% in 2021, Contract B’s. voluntary disenrollment rate fell by 8.2 percentage points

4

These are two real contracts that have been anonymized.

7

to 4.8% in 2021. Contracts that use annual data to address the specific reasons for which their
beneficiaries voluntarily disenroll may be able to reduce future disenrollment.
This analysis shows that knowing the levels of disenrollment reasons from the most recent year
provides useful information to contracts, as it is predictive of how contract disenrollment can be
expected to change in the next year. This information is useful to contracts who can focus
quality improvement efforts to avoid losing members; improvements that contracts make
support Medicare beneficiaries who want good quality contracts. Providing contracts with
recent information on reasons for disenrollment is particularly useful as an early warning to
contracts, who could make changes to improve care and services and reduce beneficiary
disenrollment and churning which is costly to contracts and beneficiaries alike.

8

Appendix 1: State-Level Composite-Score Categories Relative to the 2020 Thresholds for the
Financial Reasons Composite; the Problems Getting the Plan to Provide and Pay for Needed
Care Composite; and the Problems with Coverage of Doctors and Hospitals Composite.

9

10

11