Memo on year-to-year variability

Analyses Regarding Year-to-Year Variability of Disenrollment Reason Scores

As part of 2017 Medicare Advantage and Prescription Drug Plan Disenrollment Reasons Survey clearance process, OMB made the following request:

“When four years of data are available, CMS will evaluate the within plan temporal variability in quality scores available to consumers and adjust the frequency of the data collection accordingly. CMS will look at the temporal and geographic variability in the distribution of disenrollment reasons across all plans (analyses will include comparisons at the 10th, 25th, 50th, 75th, 90th percentiles). If there is very little change in the quality scores across time, CMS will consider collecting the information less frequently.”

CMS’s contractor for the survey, the RAND Corporation, completed the requested analyses and submitted the results of those analyses with the 2019 OMB submission. RAND has repeated those analyses and provides updated results. The purpose of the analyses was to determine whether the annual collection of the disenrollment survey provides valuable information that could not be provided through less-frequent fielding. The three analyses RAND conducted were:

1. Contract-level variability in reasons for disenrollment composite scores over time;

2. Geographic-level variability in the distribution of disenrollment reasons over time; and

3. Examining the degree to which disenrollment reasons composites predict future disenrollment.

RAND focused its analyses on MA-PD contracts, as all five composite reasons for disenrollment measures that are reported to contracts and beneficiaries on Medicare Plan Finder are captured

by the MA-PD version of the survey.¹ disenrollment are:

• Financial Reasons

The five composite measures of reasons for

• Problems with Prescription Drug Benefits and Coverage

• Problems with Coverage of Doctors and Hospitals

• Problems Getting the Plan to Provide and Pay for Needed Care

• Problems Getting Information and Help from the Plan This memo summarizes the findings of RAND’s analyses.

Analysis #1: Contract-Level Variability in Reasons for Disenrollment Composite Scores Over Time

To assess how much disenrollment reasons scores for individual contracts change over time, the RAND team calculated the percentage of variance in the 2021 composite disenrollment reasons measures (the most recent year available) across contracts that could not be predicted from the same measure using data that was three years old (2018), two years old (2019), or one year old (2020). RAND focused on MA-PD contracts to assess the impact on all five disenrollment reasons composite measures.

¹ The PDP version of the survey does not capture two composite disenrollment reason measures that relate to medical coverage.

Figure 1 plots the percent of each disenrollment measure’s 2021variance that is not predicted from the prior year’s data. Larger percentages, represented by longer bars, indicate less ability for a prior year’s contract-level scores to accurately represent 2021 contract-level disenrollment reason scores.

Less than half of the variability in the Problems Getting the Plan to Provide and Pay for Needed Care composite in 2021 was predictable from the prior year’s scores (2020, green bars. For three of the remaining composites, between 50 and 60% of the 2021 scores’ variability was predictable using 2020 data, and for the Problems with Getting Information and Help from the Plan composite, approximately 70% of the 2021 variance was predictable from the 2020 data. Using older data (2018, purple bars and 2019, blue bars), less than half of the 2021 variability was predictable using 2018 or 2019 scores, and for the Problems with Prescription Drug Benefits and Coverage composite, less than 20% of the variance was predictable using 2018 data, and less than 10% using 2019.

Figure 1: A Large Percentage of 2021 Composite Variances is not Predictable by the Corresponding Composite Measure in Previous Years (2020, 2019, and 2018)

Looking at a specific contract as an example, imagine that a beneficiary was trying to judge how often specific reasons for disenrollment applied to Contract X² in 2021 performance. The true

² This example is a real MA plan that has been anonymized.

performance of Contract X in 2021, if that data were available to the beneficiary, was poor for the Problems with Prescription Drug Benefits and Coverage and Financial Reasons composite but above average for the Problems with Coverage of Doctors and Hospitals composite.

However, this pattern was not reflected in the contract’s performance in prior years. This contract scored in the lowest performance category for the Problems with Coverage of Doctors and Hospitals in 2018, but steadily improved from 2019 to 2021. In contrast, the contract had middling performance in 2018 for both the Problems with Prescription Drug Benefits and Coverage and the Financial Reasons composites but had declined to the lower performance categories of these composites by 2021. Even this contract’s 2020 performance would have given especially misleadingly optimistic information to Medicare beneficiaries for the Problems with Drug Benefits and Coverage composite, which declined sharply from 2020 to 2021. If the contract’s sponsor only had access to the 2020 data, the older, less-relevant data could have substantially delayed the sponsor from identifying a problem and designing appropriate quality improvement initiatives.

Analysis #2: Geographic-level Variability in the Distribution of Disenrollment Reasons Over Time

To examine how composite reasons-for-leaving scores vary geographically and over time, the RAND team calculated disenrollment reasons composite scores at the state level for calendar years 2020 and 2021.

State-specific scores were calculated for the five MA-PD composite disenrollment reasons measures for 2020 and 2021 and then each state was assigned to a category based on its percentile score relative to all states; using categories defined by OMB in its request for analyses: <10^th%, 10^th – 24^th%, 25^th-49^th%, 50^th – 74^th%, 75^th-89^th%, and 90^th+%. Note that higher disenrollment reason composite scores, and higher percentile categories indicate worse performance because they mean that a higher proportion of beneficiaries cite the reasons for leaving the plan. The 2020 and 2021 scores used 2020 percentile cutoffs for comparability.

To ensure stable measurement, the analysis was limited to states with at least 100 disenrollment survey responses for a given composite measure in both 2020and 2021, which left 39 states that met the 100-response threshold.³

RAND generated pairs of state maps for each disenrollment reason composite measure, color- coding states to indicate their composite-score percentile category. When comparing the 2020 to 2021 maps, the color differences illustrate changes. In Appendix A of this memo, we have included three pairs of maps showing 3 different composite measures as examples: (1) Financial Reasons for leaving, (2) reasons related to Problems Getting the Plan to Provide and Pay for Needed Care, and (3) Problems with Coverage of Doctors and Hospitals.

To identify “transitions” between percentile categories, RAND cross tabulated the composite score categories in 2020 and 2021 for each of the five MA composite measures (Table 1, b-f). In each “transition” table, the shaded diagonal contains the number of states which remained unchanged (i.e., in the same composite percentile category in each of the two years), while those in the cells above diagonal contain the number of states in which beneficiaries endorsed

³ Note, not all states offer Medicare Advantage plans. Alaska has no MA plans.

the composites less often (which is better) in 2021 than in 2020, while those below the diagonal endorsed the composites more often (which is worse) in 2021 than in 2020.

More states were below the diagonal for the reasons for disenrollment related to the Problems with Getting Information and Help from the Plan (1e) composite, indicating increased endorsement of that disenrollment reasons composite measure in 2021 compared to 2020. For the other four composites in Tables 1b, 1c, 1d, and 1f (Financial Reasons, Problems with Prescription Drug Benefits and Coverage Problems Getting the Plan to Provide and Pay for Needed Care, and Problems with Coverage of Doctors and Hospitals more states fell above the diagonal, indicating a decrease in endorsement of those disenrollment reasons composite measures in 2021 compared to 2020.

Table 1a summarizes the percentile category shifts between 2020 and 2021. Across all composite measures, between 23% and 49% of the states remained in the same percentile category– the rest of the states moved to a higher or lower percentile category in one year’s time. These results indicate a substantial amount of change in performance over time, such that relying on older information would give a poor signal of current performance.

Table 1a: Most states changed percentile categories from 2020 to 2021

Moved to a higher percentile (% of states)		No change in category (% of states)	Moved to a lower percentile category (% of states)
Financial Reasons	3%	38%	59%
Problems with Prescription Drug Benefits and Coverage	36%	23%	41%
Problems Getting the Plan to Provide and Pay for Needed Care	23%	49%	28%
Problems with Getting Information and Help from the Plan	38%	36%	26%
Problems with Coverage of Doctors and Hospitals	18%	46%	36%

Notes: The results shown include the 39 states which provided at least 100 disenrollment survey responses on each of the 5 composite measures in both 2021 and 2020. A higher percentile category indicates that a higher percentage of that state’s disenrollees endorsed the reasons for disenrollment compared to those living in lower percentile category states.

Tables 1b through 1f provide the details of the results summarized in Table 1a.

Table 1b: Financial Reasons

2021 performance based on 2020 quantile thresholds

	90 - 100th	75 - 89th	50 - 74th	25 - 49th	10 - 24th	0 - 9th
90 - 100th	0	2	1	0	0	0
75 - 89th	0	2	4	2	0	0
50 - 74th	0	0	2	6	1	0
25 - 49th	0	0	0	5	3	3
10 - 24th	0	0	0	0	4	1
0 - 9th	0	0	0	1	0	2

2020

performance based on 2020 quantile thresholds

Table 1c: Problems with Prescription Drug Benefits and Coverage

2021 performance based on 2020 quantile thresholds

	90 - 100th	75 - 89th	50 - 74th	25 - 49th	10 - 24th	0 - 9th
90 - 100th	1	0	1	0	1	0
75 - 89th	1	1	2	1	1	0
50 - 74th	3	1	1	3	3	0
25 - 49th	2	0	3	1	3	1
10 - 24th	0	0	0	2	5	0
0 - 9th	0	0	1	0	1	0

2020

performance based on 2020 quantile thresholds

Table 1d: Problems Getting the Plan to Provide and Pay for Needed Care

2021 performance based on 2020 quantile thresholds

	90 - 100th	75 - 89th	50 - 74th	25 - 49th	10 - 24th	0 - 9th
90 - 100th	2	1	0	1	0	0
75 - 89th	1	6	0	1	0	0
50 - 74th	0	2	4	4	0	1
25 - 49th	0	1	1	7	0	0
10 - 24th	0	0	0	2	0	3
0 - 9th	0	0	0	2	0	0

2020

performance based on 2020 quantile thresholds

	90 - 100th	75 - 89th	50 - 74th	25 - 49th	10 - 24th	0 - 9th
90 - 100th	1	1	2	0	0	0
75 - 89th	2	2	1	1	1	0
50 - 74th	0	5	3	2	0	1
25 - 49th	0	1	3	8	0	0
10 - 24th	1	0	0	0	0	1
0 - 9th	0	0	0	2	1	0

Table 1e: Problems with Getting Information and Help from the Plan

2021 performance based on 2020 quantile thresholds

2020

performance based on 2020 quantile thresholds

Table 1f: Problems with Coverage of Doctors and Hospitals

2021 performance based on 2020 quantile thresholds

	90 - 100th	75 - 89th	50 - 74th	25 - 49th	10 - 24th	0 - 9th
90 - 100th	2	2	0	0	0	0
75 - 89th	2	2	3	1	0	0
50 - 74th	0	2	6	3	0	0
25 - 49th	0	0	2	3	4	0
10 - 24th	0	0	0	0	4	1
0 - 9th	0	0	1	0	0	1

2020

performance based on 2020 quantile thresholds

Focusing on specific states further illustrates changes in scores over time. For example, the Problems with Coverage of Doctors and Hospitals composite score for New Hampshire in 2020 was in the best 0-9th percentile group (lowest scores / fewest problems / fewest reasons for leaving are good). However, in 2021 New Hampshire moved to a worse-than-median group (50th-74th percentile), indicating it had a higher rate of problems / reasons for leaving than most of the other states for the Problems with Coverage of Doctors and Hospitals composite.

Another example is the state of Connecticut, which in 2020 had one of the worst (highest scores

/ most problems / most reasons for leaving are bad) scores for the Problems Getting the Plan to Provide and Pay for Needed Care composite. In the following year, however, Connecticut’s score had improved, moving into a better-than-median group (25th to 49th percentile), indicating that it showed a lower rate of problems / reasons for leaving in 2021 than at least half of the other states for this composite.

The performance of these states’ composite measure scores could not have been measured accurately by using previous years’ data. Had CMS program staff used disenrollment reasons data that was even one year older than 2021, they would have falsely concluded that contracts in the New Hampshire were doing much better than most states in terms of Problems with Coverage of Doctors and Hospitals when in fact they were not. Program staff might also have concluded that contracts in the state of Connecticut were scoring very poorly on Problems Getting the Plan to Provide and Pay for Needed Care when in fact they were scoring better than contracts in most other states in 2021.

Analysis # 3: Disenrollment Reasons Composites Predict Changes in Disenrollment Rates

Helpful to the question of assessing how frequently surveying should occur, is understanding the relationship between disenrollment reasons cited by beneficiaries and disenrollment rates. To explore this, RAND used disenrollment survey data for 2020 and 2021 for 395 MA contracts and modeled contract-specific changes in the disenrollment rates between 2020 and 2021 as a function of the 2021 contract-level financial reasons composite score and a hybrid contract-level composite measure which we formed by averaging the 4 non-financial composite scores for each contract.

A significant (p<0.0001) positive association was observed between the average score on the non-financial composite (how often disenrollees cites nonfinancial reasons for disenrolling) and changes in contract-level disenrollment rates from 2020 to 2021.

RAND modeled the 2020-2021 change in contract-specific disenrollment rates by predicting the 2021 disenrollment rate from the hybrid non-financial reasons composite, controlling for the 2020 disenrollment rate and the financial reasons composite. They then calculated the expected change in disenrollment rate for a plan with a median 2020 disenrollment rate (9.7%) according to whether it had a low (5^th percentile), medium (50^th percentile), or high (95^th percentile) 2020 nonfinancial composite reasons score. Table 2 presents the expected changes in disenrollment rate.

A median disenrollment rate contract in 2020 with a disenrollment rate of 13.0% with a typical (50^th percentile, ”medium’) rate of non-financial reasons for 2020 disenrollment would be expected to see a 1.9 percentage point increase in its disenrollment rate by 2021, resulting in a 14.9% disenrollment rate in 2021. In contrast, a contract with a “low” ( 5^th percentile) rate of non-financial reasons would expect its 2021 disenrollment rate to drop by 0.2 percentage points to 11.8.%. Finally, a contract with a high (95^th percentile) rate of non-financial reasons in 2020 would expect its voluntary disenrollment rate to rise by 4.8 percentage points to 17.8%.

Table 2: 2021 Contract Disenrollment Rates Increased When a Contract’s 2020 Disenrollees Cited More Non-Financial Reasons

2020 Non-Financial Reasons Composite Score

	Low 5th percentile	Medium 50th percentile	High 95th percentile
2020 median disenrollment rate	13.0%	13.0%	13.0%
Estimated change in disenrollment rate for a contract at the median disenrollment rate 13.0% in 2020	-0.2%	+1.9%	+4.8%
Predicted 2021 Disenrollment Rate	12.8% (13.0%-0.2%=12.8%)	14.9% (13.0%+1.9%=14.9%)	17.8% (13.0%+4.8%=17.8%)

Low, medium, and high levels correspond to the 5^th, 50^th, and 95^th percentiles in the distributions of 2020 non-financial composite scores.

As examples consider two specific contracts “A” and “B,”⁴ which had similar and slightly high voluntary disenrollment rates of 13.6% and 13.0%, respectively, in 2020. Contract A had a worse-than-average non-financial composite score in 2020: its disenrollees endorsed 24.1% of non-financial reasons. Contract B, on the other hand, had a much-better/lower-than-average non-financial composite score of 7.4% in 2020, with very little endorsement of non-financial reasons for leaving. While Contract A’s voluntary disenrollment rate then rose by 4.6 percentage points to 18.2% in 2021, Contract B’s. voluntary disenrollment rate fell by 8.2 percentage points

⁴ These are two real contracts that have been anonymized.

to 4.8% in 2021. Contracts that use annual data to address the specific reasons for which their beneficiaries voluntarily disenroll may be able to reduce future disenrollment.

This analysis shows that knowing the levels of disenrollment reasons from the most recent year provides useful information to contracts, as it is predictive of how contract disenrollment can be expected to change in the next year. This information is useful to contracts who can focus quality improvement efforts to avoid losing members; improvements that contracts make support Medicare beneficiaries who want good quality contracts. Providing contracts with recent information on reasons for disenrollment is particularly useful as an early warning to contracts, who could make changes to improve care and services and reduce beneficiary disenrollment and churning which is costly to contracts and beneficiaries alike.

Appendix 1: State-Level Composite-Score Categories Relative to the 2020 Thresholds for the Financial Reasons Composite; the Problems Getting the Plan to Provide and Pay for Needed Care Composite; and the Problems with Coverage of Doctors and Hospitals Composite.

1