CMS-10836 Enforcement and Compliance Information - Intermediaries

Medicare Plan Performance Warning Information: Guide 2 (i)
Enforcement and Compliance Information
IDI Discussion Guide (90-minute online interviews)

Interviewer: _________________________
Participant:

_________________________

Date/Time:

_________________________

Objectives
●
●
●

Gauge reactions to and interest in information on enforcement and compliance.
Understand how intermediaries may use enforcement and compliance information
when discussing plan decisions with beneficiaries.
Gain insights on best ways to share plan enforcement and compliance information
with intermediaries.

Introduction (3 min)
Thank you for agreeing to speak with me today. My name is ______ and I’ll be leading our
conversation. This is opinion research, which means there are no wrong answers. I am
interested in everything you have to say. Since we are talking about your opinions, you are
the expert.
Before we get started, there are a few things I want to mention. First, I am an independent
researcher. You will not offend me or get me in any trouble. My only goal is to hear your
honest opinions. So please be as open and candid as possible.
We have a number of questions to get through and a few things to look at and read. So, if
at some point I ask that we move on, it’s because I want to make sure that we finish on
time. It’s not because I am not interested in what you have to say.
I am recording our conversation so I don’t miss anything. The recording is confidential and
is not shared outside the research team. There are also other people watching this
interview to help me take notes and to understand what was said. Ultimately, the findings
from this interview will be combined with other interviews we’re conducting, and we’ll write a
report with some recommendations. Your name won’t be used in the report.
The sponsor of this study is the Centers for Medicare and Medicaid Services, specifically,
the Medicare program, or CMS. I do not work for the Medicare program.

OMB Control Number 0938-XXXX (Expires XX/XX/XXXX)

1

Do you have any questions before we get started?
Warm-up (7 min)
I like to start by getting to know more about you.
1. How long have you been working as/with a [SHIP counselor / broker / advocacy
group]?
a. [IF A YEAR OR LESS]: Previous to this role, have ever you done similar work
in some other capacity?
b. What does that work look like on a day-to-day basis [PROBE TO
UNDERSTAND THEIR ENGAGEMENT WITH BENEFICIARIES]?
2. What do you find most challenging about this work? [KEEP BRIEF]
3. What are some of the most common things you get asked about by
beneficiaries/beneficiaries care about? [LISTEN FOR QUALITY OR
PERFORMANCE ISSUES]
a. What are some of the most common concerns that beneficiaries have about
Medicare plans? [PROBE TO UNDERSTAND ISSUES RELATED TO
QUALITY OR PERFORMANCE]
4. In general, how could you determine if a Medicare Advantage or prescription drug
plan is “good” or not? By “good,” I don’t mean affordable or if a beneficiary’s doctor
participates, I mean if the quality of the plan itself is good, or how well it operates or
serves beneficiaries? I’ll refer to this as “plan quality.” [IF RESPONDENT REFERS
TO HEALTHCARE QUALITY, CLARIFY THIS IS REFERRING TO PLAN
OPERATIONS AND HOW WELL THE PLAN CONDUCTS THOSE.]
Enforcement and Compliance Information: Introduction (15 min)
Today we are going to talk about how Medicare monitors plans and collects information on
the performance of plans.
5. To begin, what do you know about how Medicare monitors plan performance? What
kind of activities would you expect it to conduct?
a. What would you expect or hope Medicare would share with intermediaries
who help beneficiaries, like yourself?
b. What would you expect or hope Medicare would share with beneficiaries?
c. Do you think sharing any information on this topic would help beneficiaries to
have more trust in Medicare? IF YES: What would be most helpful to share?
6. I want to share some information with you about how Medicare monitors plan
performance. [SHARE OVERVIEW TO INCLUDE AUDITS, ENFORCEMENT
ACTIONS, AND OTHER PLAN PERFORMANCE INDICATORS]
a. What is your reaction to this information? How much of it was familiar to you
already?

OMB Control Number 0938-XXXX (Expires XX/XX/XXXX)

2

b. How have you learned about this in the past?
7. Now I want to share some information with you specifically about enforcement and
compliance. [SHARE MORE DETAILED INFORMATION ON ENFORCEMENT AND
COMPLIANCE, INCLUDING DEFINITIONS, CONTEXT, THE DIFFERENCE
BETWEEN COMPLIANCE AND ENFORCEMENT, TYPES OF EACH, THAT
ENFORCEMENT ACTIONS ARE OFTEN TAKEN AFTER ISSUES ARE
RESOLVED, ETC.]
a. What is your reaction to this information? How much of it was familiar to you
already?
b. How have you learned about this in the past?
8. In general, how interested are you in compliance information about the parent
organizations of Medicare plans? Please rate your interest on a scale from 1, not at
all interested, to 10, very interested. Please explain your answer.
a. What most interests you about compliance information? Why?
b. How, if at all, would you use this information (e.g., general interest, informing
discussions with beneficiaries, etc.)?
c. When might you use it?
9. In general, how interested are you in enforcement information about the parent
organizations of Medicare plans? Please rate your interest on a scale from 1, not at
all interested, to 10, very interested. Please explain your answer.
a. What most interests you about enforcement information? Why?
b. How, if at all, would you use this information (e.g., general interest, informing
discussions with beneficiaries, etc.)?
c. When might you use it?
Enforcement and Compliance Information: Examples and Details (45 min)
Now we are going to look at some of the information Medicare currently shares about
enforcement and compliance actions.
10. I want to get your reaction to some information Medicare shares now about
compliance. These are based on real examples, but we have changed plan names.
[SHOW EXAMPLES BASED ON Part C and Part D Compliance Actions | CMS,
AND READ ACCOMPANYING INFORMATION TO DESCRIBE EACH. INCLUDE
INFORMATION ON THE RANGE OF SCENARIOS THAT MAY BE INCLUDED.]
a. Notice of non-compliance: What is your reaction to this information? In your
own words, what is it telling you? How important is this information for
beneficiaries to know and understand?
b. Warning letter: What is your reaction to this information? In your own words,
what is it telling you? How important is this information for beneficiaries to
know and understand?
c. Corrective action plan request: What is your reaction to this information? In
your own words, what is it telling you? How important is this information for
beneficiaries to know and understand?

OMB Control Number 0938-XXXX (Expires XX/XX/XXXX)

3

11. What questions do you have about this information?
12. Overall, how helpful is this kind of compliance information as you think about your
work and/or educating and assisting beneficiaries? Please rate how helpful it is on a
scale from 1, not at all helpful, to 10, very helpful. Please explain your answer.
a. What’s most helpful? What’s least helpful?
b. Would having this kind of information be useful in any way to your work?
c. Would having this kind of information change how you (or someone else)
assists beneficiaries when they are making decisions on a plan? Why or why
not?
i. Would having this kind of information impact how you viewed a plan’s
performance?
d. How, if at all, would you use this information (e.g., general interest, informing
discussions with beneficiaries, advocacy work, etc.)?
e. When would you use this information?
f. Is this kind of information official/impartial enough such that you could/would
share this information with a beneficiary to advise them about a plan if you
had concerns?
13. Now I want to get your reaction to some information Medicare shares now about
enforcement. These are based on real examples, but we have changed plan names.
[SHOW EXAMPLES BASED ON Part C and Part D Enforcement Actions | CMS,
AND READ ACCOMPANYING INFORMATION TO DESCRIBE EACH. INCLUDE
INFORMATION ON THE RANGE OF SCENARIOS THAT MAY BE INCLUDED.]
a. Civil Money Penalties: What is your reaction to this information? In your own
words, what is it telling you? How important is this information for
beneficiaries to know and understand?
b. Intermediate Sanctions: What is your reaction to this information? In your own
words, what is it telling you? How important is this information for
beneficiaries to know and understand?
c. Terminations: What is your reaction to this information? In your own words,
what is it telling you? How important is this information for beneficiaries to
know and understand?
14. What questions do you have about this information?
15. Overall, how helpful is this kind of enforcement information as you think about your
work and/or educating and assisting beneficiaries? Please rate how helpful it is on a
scale from 1, not at all helpful, to 10, very helpful. Please explain your answer.
a. What’s most helpful? What’s least helpful?
b. Would having this kind of information be useful in any way to your work?
c. Would having this kind of information change how you (or someone else)
assists beneficiaries when they are making decisions on a plan? Why or why
not?
i. Would having this kind of information impact how you viewed a plan’s
performance?
d. How, if at all, would you use this information (e.g., general interest, informing
discussions with beneficiaries, advocacy work, etc.)?
e. When would you use this information?

OMB Control Number 0938-XXXX (Expires XX/XX/XXXX)

4

f.

Is this kind of information official/impartial enough such that you could/would
share this information with a beneficiary to advise them about a plan if you
had concerns?

16. I shared two types of information with you. First, we looked at compliance
information [REFERENCE EXAMPLES], then, we looked at enforcement
information [REFERENCE EXAMPLES]. As a reminder, the difference between
compliance and enforcement is [REFER TO PREVIOUS EXPLANATION]. Now that
you’ve seen examples of each, what are you most interested in [ASK AND THEN
HAVE THEM EXPLAIN THEIR ANSWER]:
a. Compliance information
b. Enforcement information
c. Both equally
d. Neither
17. Now I want to share some enforcement and compliance information in a different
format. [SHOW REFORMATTED INFORMATION INCLUDING NAME OF PLAN,
YEAR FORMED, ENROLLMENT, NAME OF PARENT ORG, INFORMATION FOR
EACH PLAN VS. NATIONAL AVERAGES ON NONCs, WARNING LETTERS,
CAPS, CMPs, INTERMEDIATE SANCTIONS, AND TERMINATIONS]
a. Example with no compliance or enforcement information: What is your
reaction to this information? In your own words, what is it telling you? How
important is this information for beneficiaries to know and understand?
b. Example with compliance information only: What is your reaction to this
information? In your own words, what is it telling you? How important is this
information for beneficiaries to know and understand?
c. Example with enforcement information only: What is your reaction to this
information? In your own words, what is it telling you? How important is this
information for beneficiaries to know and understand?
18. What questions do you have about this information?
19. Overall, how helpful is this kind of information as you think about your work and/or
educating and assisting beneficiaries? Please rate how helpful it is on a scale from
1, not at all helpful, to 10, very helpful. Please explain your answer.
a. What’s most helpful? What’s least helpful?
b. Would having this kind of information be useful in any way to your work?
c. Would having this kind of information change how you (or someone else)
assists beneficiaries when they are making decisions on a plan? Why or why
not?
i. Would having this kind of information impact how you viewed a plan’s
performance?
d. How, if at all, would you use this information (e.g., general interest, informing
discussions with beneficiaries, advocacy work, etc.)?
e. When would you use this information?
f. Is this kind of information official/impartial enough such that you could/would
share this information with a beneficiary to advise them about a plan if you
had concerns?

OMB Control Number 0938-XXXX (Expires XX/XX/XXXX)

5

20. Now that you have a better sense of the kind of work that Medicare does in this
area, what are your thoughts? [LISTEN TO UNDERSTAND IF CMS WORK IS
PERCEIVED AS ADEQUATE, OR IF THEY THINK MORE SHOULD BE
DONE/THEY NEED TO BE MORE VIGILANT.]
21. [IF TIME] How helpful is this kind of information vs. what I first showed you? [SHOW
AS A REMINDER IF NEEDED]
a. Which do you prefer?
b. Why?
Ways to Share Information (15 min)
Thank you. Now I want to get your feedback on how to share this kind of information.
22. Where would you expect to learn about this kind of enforcement and compliance
information? Why?
a. IF NOT DISCUSSED: How might you expect to find it?
b. Where do you think beneficiaries would expect to learn about this kind of
enforcement and compliance information? Why?
23. Now I want to share some examples of how information might be shared. [SHOW
EXAMPLES OF WAYS TO SHARE INFORMATION, INCLUDING AN OVERVIEW
OF PLAN FINDER WEBSITE FOR THOSE NOT FAMILIAR; KEEP IN MIND
PURPOSE OF PLAN FINDER VS. OTHER SOURCES LIKE MEDICARE & YOU
TO EXPLORE WHAT THE BEST PLACE IS FOR THIS TYPE OF INFORMATION]
What is your reaction to:
a. Language on the Plan Finder website
b. Indicators or icons on the Plan Finder website
c. Links from Plan Finder to a separate website with more information (including
existing letters or reformatted information)
d. Information hosted on a separate website not directly linked from Plan Finder
e. Others?
24. Which of these options to share information is most helpful? Why? Who do you think
the target audience of this information should be?
a. Does it depend on the type of information being shared? How so?
b. Does it depend on the engagement level of the beneficiary? [PROBE TO
UNDERSTAND IF WHERE OR HOW INFORMATION IS DISPLAYED
WOULD DIFFER BASED ON WHETHER A BENEFICIARY IS MORE OR
LESS ENGAGED]
c. Is there some other way Medicare could communicate information related to
compliance and enforcement that would be helpful to your work or for
assisting beneficiaries?
d. Thinking of your own work and all the information you (or your organization)
share, when, if ever, would you share enforcement and compliance and
information with beneficiaries? Why?

OMB Control Number 0938-XXXX (Expires XX/XX/XXXX)

6

25. Let me ask specifically about plans which Medicare has sanctioned [RE-SHARE
DEFINITION IF NEEDED].
a. CMS policy lets beneficiaries enroll in sanctioned plans through the plan
directly and through a broker, but not through the Medicare Plan Finder.
Should CMS show a sanctioned plan on Medicare Plan Finder along with
information that the plan is sanctioned, and that you can’t enroll here? Or
should Medicare Plan Finder not show a sanctioned plan, and why? [SHOW
EXAMPLES MODELED OFF HOW CMS HAS HANDLED THIS
PREVIOUSLY]
26. Thinking about the timeliness of enforcement and compliance information. Are you
interested in information from: [ASK FOR EACH AND HAVE THEM EXPLAIN
WHY]:
a. The last year?
b. The last two years?
c. The last five years?
d. Six or more years ago?
i. Does your answer change if the plan has had multiple/repeated
enforcement or compliance issues? IF YES: How so?
i. Should CMS highlight in some way the fact that the plan has had
multiple/repeated enforcement or compliance issues?
Wrap-up (5 min)
TIME ALLOWING: Before we wrap up, I’m going to check to see if there are any additional
questions for you from anyone else on the team. [CHECK FOR QUESTIONS]
27. This has been really helpful. Thanks so much for participating in this project. Do you
have any other thoughts or comments you’d like to share?
We really appreciate your assistance. Have a great day.
PRA Disclosure Statement
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a
collection of information unless it displays a valid OMB control number. The valid OMB
control number for this information collection is 0938-XXXX (Expires XX/XX/XXXX). This is
a voluntary information collection. The time required to complete this information collection
is estimated to average 1.95 hours per response, including the time to review instructions,
search existing data resources, gather the data needed, and complete and review the
information collection. If you have comments concerning the accuracy of the time
estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security
Boulevard, Attn: PRA Reports Clearance Officer, Mail Stop C4-26-05, Baltimore, Maryland
21244-1850.

OMB Control Number 0938-XXXX (Expires XX/XX/XXXX)

7

****CMS Disclosure**** Please do not send applications, claims, payments, medical
records or any documents containing sensitive information to the PRA Reports
Clearance Office. Please note that any correspondence not pertaining to the
information collection burden approved under the associated OMB control number
listed on this form will not be reviewed, forwarded, or retained. If you have questions
or concerns regarding where to submit your documents, please contact Memuna
Ifedirah at [email protected].

OMB Control Number 0938-XXXX (Expires XX/XX/XXXX)

8