CR_OMB SummaryofRevisions 100520

Part C and D Complaints Resolution Performance Measure
Summary of OMB Revisions
Ten organizations submitted 136 comments in response to the OMB PRA Package [Document
Identifier: CMS-10308] that was posted in the Federal Register: February 25, 2010 (Volume 75,
Number 37). The tables below summarize the revisions made to the Advance Letter, Explanation
for Inclusion of Survey Questions, Supporting Statements A and B, and the Survey Instrument in
response to these comments.
Advance Letter
Two main issues that arose were the reading level and clarity of the letter. As such, CMS has
revised the letter to read at the 8th grade level. A few clarity issues that the organizations
mentioned pertained to the logistics of scheduling surveys with beneficiaries, explanation of the
purpose of the survey, specific reference to the beneficiary’s MAO or Part D sponsor (not
Medicare), and an explanation of IMPAQ International’s role. The table below summarizes
these revisions to the letter.
Effect to
Reporting
Burden

Category

Section

Change/Reason

Response to
Public
Comments

Advance
Letter

The reading level of the letter was lowered to 8th grade.

None

Response to
Public
Comments

Advance
Letter

Language was added to the Advance Letter to describe the
calling process and how beneficiaries should expect to be
contacted.

None

Response to
Public
Comments

Advance
Letter

Language was added to the Advance Letter to describe the
intended use of collected data as opposed to: "Your answers
will be kept strictly confidential and be used only for research
purposes." The purpose of the data collection (to improve how
complaints are handled and to inform the development of a plan
rating system) was explained.

None

Response to
Public
Comments

Advance
Letter

The advance letter has been revised to make explicit reference
to the MAO or Part D sponsor as well as the role of IMPAQ
International (contractor). This will reduce confusion about the
topic and source of the complaints.

None

Explanations for Inclusion of Survey Questions
There was a concern that the proposed indicators did not correspond with the survey questions.
CMS has revised the explanation of the proposed indicators, together with the descriptions of
each survey question, to better explain their importance in developing performance measures.

IMPAQ International, LLC

1

Summary of Revisions

Part C and D Complaints Resolution Performance Measure
Summary of OMB Revisions
Category

Section

Response to
Public
Comments

Explanations
for Inclusion
of Survey
Questions

Effect to
Reporting
Burden

Change/Reason
This document describing the proposed indicators has been
revised to better explain their importance, meanings, and how
they will help develop the performance measures.
Specifically:
– Rationale for use of "resolution" vs. "final outcome"
– Use of 4-point Likert scale vs. 5-point Likert scale (and
addition of "I don't know/NA" answer choice)
– Strengthening of rationale for inclusion of Veracity of Plan's
Description indicator

None

Supporting Statements A and B
There was some concern about the purpose and calculation of indicators such as “Veracity of
Complaints Resolution,” and “Beneficiary Awareness of Resolution.” Further information
regarding these indicators has been added to the supporting statements. Some organizations
were concerned with how the results of the survey will be utilized and if/when that information
would be shared publicly. At this time, it is undetermined which data will be shared with the
plans.
Some comments suggested that the survey instrument should undergo an evaluation by a quality
measurement organization. Because this is a preliminary effort to assess the possibility of
developing performance measures, this type of review is unnecessary at this time.
Some plans mentioned that overall satisfaction may be affected by dissatisfaction with the final
outcome and resolution of the complaint. CMS has revised the language to ask beneficiaries
about several aspects of their satisfaction besides the final outcome. Similarly, concerns about
confusion between “resolution” and “final outcome” have been addressed by emphasizing in the
survey that CMS is interested in the series of actions taken by the plan to resolve complaints
(e.g., the final outcome), rather than emphasizing the resolution.
It was suggested that the proposed sampling plan would not allow plans sufficient time to inform
the beneficiary of the resolution. CMS has chosen to allow a delay of 7 days after the complaint
is closed before contacting the beneficiary. In addition, CMS has provided further clarity on the
types of contracts and plans that would be subject to data collection; this entailed an explicit
exclusion of 800 series contracts and the inclusion of small contracts and contracts with a small
number of complaints.
Some plans expressed more general concerns regarding the survey. These included how
beneficiary-appointed representatives would be contacted, the uniformity of the survey approach
for both telephone and mail-in, the reliability of collected data, and analyses to control for factors
IMPAQ International, LLC

2

Summary of Revisions

Part C and D Complaints Resolution Performance Measure
Summary of OMB Revisions
outside of the plan’s control. The table below summarizes the actions CMS has taken to address
these and other concerns in Supporting Statements A and B.
Several comments objected to the possible inclusion of complaints that are outside the scope of
plans. CMS has chosen to exclude complaints that are likely to be outside of the scope of plans,
primarily some issues related to enrollment complaints.
Effect to
Reporting
Burden

Category

Section

Change/Reason

Response to
Public
Comments

Supporting
Statement A

Clarification was added regarding how CMS intends to
disseminate results of the survey. CMS will follow a process
for this performance measure similar to the one it uses for other
measures. At this time, it is undetermined what level of data
will be shared with plans.

None

Response to
Public
Comments

Supporting
Statement A

Information regarding the use and calculation of the "Veracity
of Complaint Resolution" indicator has been added.

None

Response to
Public
Comments

Supporting
Statement A

Information regarding the use and calculation of indicators for
new survey questions has been added.

None

Response to
Public
Comments

Supporting
Statement A

The calculation of the "Beneficiary Awareness of Resolution"
indicator has been clarified. The description has been revised
so that it more clearly refers to the answer choices for this
survey question and explains how the answers will influence the
calculation of the indicator.

None

Response to
Public
Comments

Supporting
Statement A

The Information Users section has been updated to reflect that
CMS may opt not to use the results of the survey for
performance measurement. This is a preliminary gathering of
information to determine the possibility of developing a
performance measure..

None

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

Information has been added regarding the participation of
representatives in the survey. This explanation includes how
representatives will be contacted (through beneficiaries and/or
CTM logs) and how data from representatives can be used in
the survey data analysis. A question has been added to identify
individuals other than the beneficiary who participate in the
survey.

None

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

Language has been added to emphasize that this is a preliminary
survey effort to assess the beneficiary's satisfaction with the
complaint resolution.

None

IMPAQ International, LLC

3

Summary of Revisions

Part C and D Complaints Resolution Performance Measure
Summary of OMB Revisions
Effect to
Reporting
Burden

Category

Section

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

The pretest and pilot test methodologies have been clarified to
emphasize efforts to achieve reliable data and remove
complaints that are not within the plan’s domain.

None

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

Clarification has been added to indicate that the telephone and
written survey instrument will have a uniform format.

None

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

It has been clarified that members of 800 series contracts will
be excluded from the data collection effort.

None

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

Controlling the validity of complaints: Approaches to control
for factors affecting satisfaction with the final outcome and the
complaint resolution have been described. Also, language has
been added to underscore that several factors will be considered
in the development of the performance measure so as not to rely
on a single aspect of the beneficiary's experience.

None

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

The sampling strategy will exclude complaints that are outside
the scope of the plan (restricted by CMS guidelines),
particularly regarding issues related to enrollment complaints.

None

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

The difference between "resolution" and "final outcome" has
been clarified. An emphasis on “final outcome or decision”
rather than “resolution” puts the focus on the series of actions
the plan took, regardless of whether the beneficiary believes
his/her complaint was resolved.

None

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

Additional explanation has been added to the sampling plan
(supporting statement B) and background (supporting statement
A) regarding the selected data collection period. CMS is
interested in the months with the largest number of complaints
in order to achieve the most statistically valid sample..

None

Response to
Public
Comments

Supporting
Statement B

Call center procedures have been further defined and clarified
regarding how staff encourage participation without being
forceful. The explanation of the CATI system has been refined
to prevent misunderstandings of the purpose of the system.

None

IMPAQ International, LLC

Change/Reason

4

Summary of Revisions

Part C and D Complaints Resolution Performance Measure
Summary of OMB Revisions
Effect to
Reporting
Burden

Category

Section

Change/Reason

Response to
Public
Comments

Supporting
Statement B

The sampling framework has been modified to accommodate 7
days between complaint closure and initial contact with the
beneficiary to allow time for beneficiaries to receive
notification of their complaint resolution.

None

Response to
Public
Comments

Supporting
Statement B

Explanation has been added regarding the decision to proceed
with collecting data on small contracts and contracts with a
small number of complaints.

None

Survey Instrument
Many organizations had general comments regarding the survey instrument. Some believed that
the wording of the questions might yield negative responses and not allow for neutral responses.
CMS has reworded the survey to be more specific on the key complaint resolution process and
include answer choices of “I Don’t Know/NA.” The latter will provide an answer that allows
beneficiaries a choice outside the 4-point likert scale to accommodate beneficiaries who feel that
they do not yet have a resolution or are unsure/do not remember. Other concerns included
language in the introduction that did not clarify why beneficiaries are being contacted, that did
not fully explain the role of IMPAQ, and that lacked clarity regarding the complaint process
(whether the beneficiary filed a complaint against their MAO/PDP sponsor/Medicare.).
Concerns also were expressed regarding specific questions on the survey instrument. Some
plans commented that the use of the word “resolution” would not elicit the intended responses
from beneficiaries; therefore, the word “resolved” has been replaced with “settled” in question 1
to prevent bias to respond negatively unless the final outcome was in favor of the beneficiary.
Also, in question 2 and question 3, “resolution” has been replaced with “final outcome.”
Revising the wording will help beneficiaries to focus on the actions taken by the plan and not
their opinion of the decision.
CMS has decided to remove questions 4, 5, 6, and 8 from the survey instrument. A new question
2 has been created to include beneficiary satisfaction with the level of communication given by
the plan and other aspects of beneficiary satisfaction such as the courtesy of the plan
representative and explanation of the final outcome.
With regard to questions 5 and 6, after the results of the pretest were reviewed, it became clear
that beneficiary responses were not aligned with the intended purpose of the survey questions.
Question 5 yielded responses about beneficiary burden rather than about the filing of duplicate
complaints. Furthermore, respondents did not differentiate between their satisfaction with the
amount of time it took to resolve their complaint and their satisfaction with how the complaint

IMPAQ International, LLC

5

Summary of Revisions

Part C and D Complaints Resolution Performance Measure
Summary of OMB Revisions
was handled by the plan; it appeared that these two satisfaction ideas were confounded in the
beneficiary’s mind.
Clarifying text has been added to question 7, to address concerns that beneficiary satisfaction
would be dependent on whether or not the complaint was resolved. Language has been added to
this question, asking the beneficiary to focus on the way the complaint was handled, regardless
of whether or not he/she is satisfied with the final outcome. This overall satisfaction question is
now Q6 in the revised survey.
It was suggested that questions 9 and 10 be rewritten because of their negative connotations.
CMS has revised both of these questions to be more neutral. Question 9 is now question 8, and
question 10 is now question 3 in the revised survey instrument.
CMS has added or reformulated some questions to the survey instrument without affecting the
reporting burden. To set a baseline for the beneficiary’s satisfaction with the plan, a question
was added to obtain information on how satisfied beneficiaries are with their plan. This will be
question 7 on the revised survey. Respondents are now asked to identify whether they are the
beneficiary or a representative. This is question 9 on the revised survey. Last, an open-ended
question was placed at the end of the survey. This allows respondents an opportunity to provide
feedback about the complaints process and make suggestions for improvement. The table below
summarizes the actions CMS has taken to address comments on the survey instrument. Again,
there was no change in the reporting burden.
Effect to
Reporting
Burden

Category

Section

Response to
Public
Comments

Survey
Instrument General

The survey questions have been revised to reflect more neutral
wording.

None

Response to
Public
Comments

Survey
Instrument Introduction

Details have been added to the survey introduction explaining
why beneficiaries are being contacted and specifying terms such
as "Medicare," MAO, or Part D sponsor, and the role of the
contractor in conducting the survey.

None

Response to
Public
Comments

Survey
Instrument Q1

"Resolved" has been replaced with "settled" in this question to
prevent beneficiary bias. An "I don't know" answer choice has
been added for beneficiaries who feel they do not yet have a
resolution or are unsure/do not remember.

None

Response to
Public
Comments

Survey
Instrument Q2

"Resolution" has been replaced with "final outcome or
decision" in this question to prevent beneficiary bias and to
guide the beneficiary towards the actions taken by the plan as
opposed to the beneficiary's opinion of the decision. This

None

IMPAQ International, LLC

Change/Reason

6

Summary of Revisions

Part C and D Complaints Resolution Performance Measure
Summary of OMB Revisions
Category

Section

Effect to
Reporting
Burden

Change/Reason
question is now Q4 in the new survey instrument.

Response to
Public
Comments

Survey
Instrument Q3

"Resolution" has been replaced with "final outcome" in this
question. An "NA" answer choice is available for beneficiaries
who do not believe they have received a final outcome or who
do not remember the resolution of their complaint. This
question is now Q5 in the new survey instrument.

None

Response to
Public
Comments

Survey
Instrument Q4

Question 4 has been removed, and issues of plan
communication with the beneficiary have been incorporated in
Q2 of the new survey instrument.

Decrease

Response to
Public
Comments

Survey
Instrument Q5

Question 5 has been removed from the survey. (CMS has
decided to drop questions about repeat complaints or multiple
attempts to contact the plan)

Decrease

Response to
Public
Comments

Survey
Instrument Q6

Question 6 has been removed from the survey. Some issues
related to the amount of time it took to resolve a complaint have
been incorporated in Q2 of the new survey instrument.

Decrease

Response to
Public
Comments

Survey
Instrument Q7

Clarifying text has been added to this question asking the
beneficiary to disregard whether or not he/she is satisfied with
the final outcome. The purpose of this question is to provide an
overall satisfaction rating. This question is now Q6 in the new
survey instrument.

None

Response to
Public
Comments

Survey
Instrument Q8

This question has been removed. The aspects of the complaint
process that were included in this question have now been
incorporated in Q2 of the new survey instrument. This revision
will allow all respondents to rate their satisfaction with
components of the process (as opposed to the skip pattern in the
original instrument).

Decrease

Response to
Public
Comments

Survey
Instrument Q9

This question has been reworded to be more neutral and to ask
about the likeliness of the beneficiary to stay with the plan.
This question is now Q8 in the new survey instrument.

None

Response to
Public
Comments

Survey
Instrument Q10

The word "problems" has been removed from this question and
the wording is more neutral (both in the question stem and in
the answer choices). This question is now Q3 in the new survey
instrument.

None

Response to
Public
Comments

Survey
Instrument –
New question

Add question to assess beneficiary satisfaction with aspects of
the complaint handling process. Beneficiaries will rate their
satisfaction with components of the handling process such as
length of the complaint process and courtesy of the plan
representative. In a simplified form, this satisfaction question
addresses issues from the original Q6 and Q8.

Increase

This question is now Q2 in the new survey instrument.

IMPAQ International, LLC

7

Summary of Revisions

Part C and D Complaints Resolution Performance Measure
Summary of OMB Revisions
Category

Section

Response to
Public
Comments

Survey
Instrument –
New question

Effect to
Reporting
Burden

Change/Reason
Add question to assess beneficiary satisfaction with the plan.
This question sets a baseline for the beneficiary's overall
satisfaction with the plan.

Increase

This question is now Q7 in the new survey instrument.
Response to
Public
Comments

Survey
Instrument –
New question

Add question to identify survey respondent. This is a
demographic question to differentiate between respondent and
proxy.
This question is now Q9 in the new survey instrument.

Increase

Response to
Public
Comments

Survey
Instrument New question

Add question to elicit beneficiary feedback. Beneficiaries will
be asked for feedback at the end of the survey, e.g., whether
they have any comments or suggestions for improvement of the
complaint resolution process.

Increase

This question is now Q10 in the new survey instrument.

IMPAQ International, LLC

8

Summary of Revisions