CMS-10836 Poor Plan Performance - Beneficiaries

Medicare Plan Performance Warning Information: Guide 3
Poor Plan Performance
IDI Discussion Guide (90-minute online interviews)

Interviewer: _________________________
Participant:

_________________________

Date/Time:

_________________________

Objectives
●
●
●

Gauge reactions to and interest in information on poor plan performance.
Understand how beneficiaries may use plan performance information when making
plan decisions.
Gain insights on best ways to share poor plan performance information with
beneficiaries.

Introduction (3 min)
Thank you for agreeing to speak with me today. My name is ______ and I’ll be leading our
conversation. This is opinion research, which means there are no wrong answers. I am
interested in everything you have to say. Since we are talking about your opinions, you are
the expert.
Before we get started, there are a few things I want to mention. First, I am an independent
researcher. You will not offend me or get me in any trouble. My only goal is to hear your
honest opinions. So please be as open and candid as possible.
We have a number of questions to get through and a few things to look at and read. So, if
at some point I ask that we move on, it’s because I want to make sure that we finish on
time. It’s not because I am not interested in what you have to say.
I am recording our conversation so I don’t miss anything. The recording is confidential and
is not shared outside the research team. There are also other people watching this
interview to help me take notes and to understand what was said. Ultimately, the findings
from this interview will be combined with other interviews we’re conducting, and we’ll write a
report with some recommendations. Your name won’t be used in the report.
The sponsor of this study is the Centers for Medicare and Medicaid Services, specifically,
the Medicare program, or CMS. I do not work for the Medicare program. Your participation
will not affect any medical services you may be receiving or are eligible to receive.

OMB Control Number 0938-XXXX (Expires XX/XX/XXXX)

1

Do you have any questions before we get started?
Warm-up (7 min)
I like to start by getting to know more about you.
1. Where do you live and how long have you lived there?
2. How long have you had Medicare coverage? [KEEP BRIEF]
3. What are the main things you expect your Medicare plan to do? [PROBE TO
UNDERSTAND ISSUES RELATED TO QUALITY OR PERFORMANCE]
4. In general, how do you know if your Medicare Advantage or prescription drug plan is
“good” or not? By “good,” I don’t mean affordable or if your doctor participates, I
mean if the quality of the plan itself is good, or how well it operates or serves
beneficiaries? We call this “plan quality.”
Plan Performance Indicators: Introduction (10 min)
Today we are going to talk about how Medicare monitors plans, and collects information on
the performance of plans.
5. To begin, how would you expect Medicare to monitor plan performance? What kind
of activities would you expect it to conduct?
a. What would you want to know about this monitoring?
b. What would you expect or hope Medicare would share with beneficiaries?
6. I want to share some information with you about how Medicare monitors plan
performance. [SHARE OVERVIEW TO INCLUDE AUDITS, ENFORCEMENT
ACTIONS, AND OTHER PLAN PERFORMANCE INDICATORS]
a. What is your reaction to this information?
b. What more, if anything, would you like to know about this monitoring?
7. Now I want to share some information with you specifically about various possible
indicators of poor plan performance. [SHARE HIGH LEVEL OVERVIEW OF PLAN
PERFORMANCE INDICATORS. SHOW MODEL THAT VISUALLY DEPICTS
VARIOUS WAYS/TIMES THAT CMS GATHERS DATA ON PLAN
PERFORMANCE, INCLUDING POSSIBLE INDICATORS OF POOR PLAN
PERFORMANCE.]
a. What is your reaction to this information?
8. In general, how interested are you in plan performance information about your
Medicare plan? Please rate your interest on a scale from 1, not at all interested, to
10, very interested. Please explain your answer.
a. What most interests you about plan performance information? Why?

OMB Control Number 0938-XXXX (Expires XX/XX/XXXX)

2

b. What least interests you about plan performance information? Why?
c. How, if at all, would you use this information (e.g., general interest, making
plan decisions, etc.)?
d. When might you use it?
Plan Performance Indicators: Examples and Details (45 min)
Now we are going to talk in more detail about some of the information Medicare currently
collects related to poor plan performance.
9. I’m going to describe each of these for you, and ask you some questions about it.
[GO THRU EACH INDIVIDUALLY, ASKING ALL QUESTIONS BELOW.]
a. In your own words, what is it telling you?
b. Would you see this as an indicator of poor plan performance?
c. What questions do you have about this information?
d. Overall, how helpful is this kind of information as you think about your plan?
Please rate how helpful it is on a scale from 1, not at all helpful, to 10, very
helpful. Please explain your answer. [MAKE NOTE OF ANYTHING RATED 7
OR HIGHER]
ITEMS TO BE REVIEWED; ORDER SHOWN WILL VARY ACROSS INTERVIEWS.
ALONG WITH DESCRIPTIONS FOR EACH, CREATE A 1-PG VERSION THAT CAN BE
USED AS A “CHEAT SHEET” TO REMIND PARTICIPANTS OF ALL THESE
INDICATORS:
o Complaints
o Grievances
o Financial solvency
o Denials:
▪ Initial denials
▪ Initial denial overturn rate
▪ Coverage denial overturn rate
▪ Pharmacy rejections/drug coverage denials
o Initial appeals (and appeals measures in general)
o IRE overturn rate
o Failure to effectuate
o Revamped Beneficiary Access and Performance Problems (BAPP) measure
o Call center hold times
o Enrollment-related measures
o Survey data (e.g., disenrollment reasons)
o Marketing misrepresentation
10. Let’s look again at the things you said you were most interested in [RE-SHOW
EVERYTHING RANKED 7 OR HIGHER, SHOW 1-PG REMINDER AS-NEEDED].
a. What would be most helpful? Top 3-5?
b. Would having this kind of information change how you thought about your
plan? Why or why not?
c. How, if at all, would you use this information (e.g., general interest, making
plan decisions, etc.)?

OMB Control Number 0938-XXXX (Expires XX/XX/XXXX)

3

d. When would you use it?
11. Now I want to describe some other kinds of information that Medicare collects about
plans. [GO THRU EACH INDIVIDUALLY, ASKING ALL QUESTIONS BELOW.]
a. What is your reaction to this information?
b. Overall, how helpful is this kind of information as you think about your plan?
Please rate how helpful it is on a scale from 1, not at all helpful, to 10, very
helpful. Please explain your answer.
ITEMS TO BE REVIEWED; ORDER SHOWN WILL VARY ACROSS INTERVIEWS:
o Plan enrollment (# of beneficiaries)
o Length of time plan has been in existence
o Plan that is too new or too small to be rated in several areas
Ways to Share Information (20 min)
Thank you. Now I want to get your feedback on how to share this kind of information.
12. Where would you expect to learn about this kind of plan performance information?
Why?
a. IF NOT DISCUSSED: How might you expect to find it?
13. Are you familiar with Medicare’s Plan Finder website? Have you used it before?
a. [IF NOT: SHOW SITE]
b. What’s your reaction to this site?
14. Are you familiar with Medicare Star Ratings? Have you used them before?
a. [FOR ALL: SHOW STAR RATINGS IN DETAIL]
b. What’s your reaction to this information?
c. How, if at all, would you use this information (e.g., general interest, making
plan decisions, etc.)?
d. Would it be helpful to have comparative information on Star Ratings, e.g.,
where a plan ranks relative to other plans? Why or why not?
i. IF YES: How might that be helpful? How would you use that
information?
15. Now I want to share some examples of how the type of information we’ve been
talking about might be shared. [SHOW EXAMPLES OF WAYS TO SHARE
INFORMATION AND EXPLORE THE BEST PLACE TO SHARE THIS TYPE OF
INFORMATION] What is your reaction to:
a. Adding some of this information to Star Ratings
b. Language on the Plan Finder website
c. Indicators or icons on the Plan Finder website
d. Links from Plan Finder to a separate website with more information (including
existing letters or reformatted information)
e. Information hosted on a separate website not directly linked from Plan Finder
f. Others?

OMB Control Number 0938-XXXX (Expires XX/XX/XXXX)

4

16. Which of these options to share information is most helpful? Why?
a. Does it depend on the type of information being shared? How so?
17. Let’s assume a plan you were considering enrolling in had some indicators of poor
plan performance. Would you want to know about that if the information was from
[ASK FOR EACH AND HAVE THEM EXPLAIN WHY]:
a. The last year?
b. The last two years?
i. Does your answer change if the plan has had multiple/repeated
issues with some of these indicators? IF YES: How so?
ii. Should Medicare highlight in some way if a plan has had
multiple/repeated issues with some of these indicators?
18. Thinking about all the information you consider about a plan: what other things do
you consider?
a. If it was available to you, would you consider the kind of information we’ve
been talking about today?
b. In general, how important is this kind of information to you as you make
decisions about a plan vs. everything else you consider? Please rate how
important it is on a scale from 1, not at all important, to 10, very important.
Please explain your answer.
Wrap-up (5 min)
TIME ALLOWING: Before we wrap up, I’m going to check to see if there are any additional
questions for you from anyone else on the team. [CHECK FOR QUESTIONS]
19. This has been really helpful. Thanks so much for participating in this project. Do you
have any other thoughts or comments you’d like to share?
If you have any questions about your Medicare options, please go to Medicare.gov. Thanks
again for participating. We really appreciate your assistance. Have a great day.
PRA Disclosure Statement
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a
collection of information unless it displays a valid OMB control number. The valid OMB
control number for this information collection is 0938-XXXX (Expires XX/XX/XXXX). This is
a voluntary information collection. The time required to complete this information collection
is estimated to average 1.9 hours per response, including the time to review instructions,
search existing data resources, gather the data needed, and complete and review the
information collection. If you have comments concerning the accuracy of the time
estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security
Boulevard, Attn: PRA Reports Clearance Officer, Mail Stop C4-26-05, Baltimore, Maryland
21244-1850.

OMB Control Number 0938-XXXX (Expires XX/XX/XXXX)

5

****CMS Disclosure**** Please do not send applications, claims, payments, medical
records or any documents containing sensitive information to the PRA Reports
Clearance Office. Please note that any correspondence not pertaining to the
information collection burden approved under the associated OMB control number
listed on this form will not be reviewed, forwarded, or retained. If you have questions
or concerns regarding where to submit your documents, please contact Memuna
Ifedirah at [email protected].

OMB Control Number 0938-XXXX (Expires XX/XX/XXXX)

6