CR_OMB Supporting Statement A 100903 clean

Part C and D Complaints Resolution Performance Measure
CMS-10308

OMB Supporting Statement – Part A
September 3, 2010

Contents
A. Background ............................................................................................................................... 1
B. Justification ............................................................................................................................... 5
1. Need and Legal Basis...............................................................................................5
2. Information Users ....................................................................................................5
3. Use of Information Technology ...............................................................................5
4. Duplication of Efforts ..............................................................................................6
5. Small Businesses ......................................................................................................6
6. Less Frequent Collection .........................................................................................7
7. Special Circumstances .............................................................................................7
8. Federal Register/Outside Consultation ....................................................................7
9. Payments/Gifts to Respondents .............................................................................14
10. Confidentiality .......................................................................................................14
11. Sensitive Questions ................................................................................................15
12. Burden Estimates (Hours & Wages) ......................................................................15
13. Capital Costs ..........................................................................................................16
14. Cost to Federal Government ..................................................................................16
15. Changes to Burden .................................................................................................17
16. Publication Tabulation Dates .................................................................................17
17. Expiration Date ......................................................................................................26
18. Certification Statement ..........................................................................................26

Page i

A. Background
The Health Maintenance Organization Act of 1976, the Balanced Budget Act of 1997 (BBA),
and the Medicare Prescription Drug, Improvement, and Modernization Act of 2003 (MMA)
introduced private insurers into the Medicare program. With the HMO Act of 1976, health
maintenance organizations (HMOs) began to be offered as a Medicare option. The BBA
established the Medicare + Choice program, which gave beneficiaries the option of enrolling in a
variety of private plans including HMOs, preferred provider organizations (PPOs), providersponsored organizations (PSOs), private fee-for-service (PFFS) plans, and medical savings
accounts (MSAs) coupled with high-deductible insurance plans. Title I of the MMA established
the new prescription drug benefit under Part D of Title XVIII of the Social Security Act (Act).
Title II of the MMA modified Part C of the Act to rename the Medicare + Choice program as the
Medicare Advantage (MA) program. These programs are administered by the Centers for
Medicare & Medicaid Services (CMS).
Part C Sponsors provide medical coverage through at-risk arrangements with CMS. Part C
Sponsors include: Local Coordinated Care Plans, which include HMOs, PPOs, and PSO plans;
Private fee-for-service plans (PFFS); Special needs plans (SNPs); MSAs; and Regional PPOs.
Under Sections 1876 and 1833(a)(1)(A) of the Social Security Act, an HMO or CMP can
participate in the Medicare program by receiving ―reasonable cost‖ reimbursement for furnishing
covered services to enrolled beneficiaries. 1833 Cost Plans (or Heath Care Prepayment Plans)
must either be union- or employer-sponsored and must not provide inpatient hospital services for
its enrollees.
Part D Sponsors provide prescription drug benefit coverage through private at-risk prescription
drug plans that offer drug-only coverage (Prescription Drug Plans), or through Medicare
Advantage (MA) plans that offer integrated prescription drug and health care coverage (MA-PD
plans). MA plans that offer this coverage can be risk-based or cost-based plans. A Prescription
Drug Plan (PDP) serves one or more PDP regions.
The right of consumers to make informed health care treatment decisions is a tenet that has
gained ascendancy in recent years. The August 8, 2006 Executive Order mandating that Federal
agencies promote transparency of health care quality and pricing data was the most recent
official acknowledgement of this right.1 Due to this Executive Order, as of CY 2008,
performance measurement ratings for Medicare Part C & Part D can be found online on
Medicare Options Compare and the Medicare Prescription Drug Plan Finder (MPDPF),
respectively. Both of these web sites provide rating information for beneficiary use. Plans are
1

―Executive Order 13410: Promoting Quality and Efficient Health Care in Federal Government Administered or
Sponsored Health Care Programs,‖ 73 Fed. Reg., 51089 (August 8, 2006).
http://edocket.access.gpo.gov/2006/pdf/06-7220.pdf

IMPAQ International, LLC

1

OMB Supporting Statement

assigned a performance-based star rating, which helps beneficiaries make informed choices
among the many plan alternatives available to them under Medicare Parts C and D. Plans are
assigned scores for each category on a scale of one to five stars, with one star indicating poor
performance and five stars indicating excellent performance. Currently, Medicare Advantage
Organizations are rated on how well they perform in five different domains,2 which have a total
of 33 different measures. Prescription Drug Plan sponsors are rated on how well they perform in
four different domains,3 which, combined, have 19 different measures. CMS intends to follow a
procedure with the proposed performance measure similar to the one it currently uses for other
measures.
To maximize the ability of beneficiaries to compare different plans, CMS seeks to expand these
measures to provide more comprehensive and sensitive performance measures. The focus of the
current project is to assess the satisfaction of beneficiaries with the process by which their
complaints were resolved by the plans in which they were enrolled, and to evaluate the final
outcome through an objective exploration of beneficiaries’ complaint resolution experiences.
The agency does not have access to this information through regular administrative or reporting
requirement mechanisms. The proposed data collection effort would assist CMS in obtaining this
critical information. CMS has the option to use the results from this data collection effort for
program monitoring (internal use) or for public reporting purposes via the Medicare Choice
website or other alternative means. The survey will target a sample of complaints—filed by
beneficiaries or their representatives—that have been closed in the Complaints Tracking Module
(CTM) by a plan during the first quarter of CY 2011. The selected timeframe has been chosen in
order to collect data for the months with the greatest number of complaints, which will therefore
likely provide the most statistically valid sample (further detail regarding the proposed sampling
plan is provided in Supporting Statement B). The proposed surveys will occur within 7 to 21
calendar days of the complaint closure and will collect beneficiaries’ opinions on the complaint
resolution process and their satisfaction with the final outcome, among other issues.
Several substantive issues are involved in this data collection request. First, CMS will follow
several approaches to control for factors affecting satisfaction with the final outcome and the
complaint resolution process. The key aspect is that several variables will be considered in the
development of the preliminary measure so as not to rely on a single aspect of the beneficiary's
experience. This includes the use of beneficiary, plan, and complaint characteristics recorded in
other CMS datasets. Second, CMS will emphasize that the primary issue of interest is the "final
2

3

The domains are: ratings of health plan responsiveness and care; managing chronic (long-lasting) conditions;
health plan telephone customer service; staying healthy: screenings, tests, and vaccines; and health plan member
complaint, appeals, and choosing to leave the health plan.
The domains are: drug plan customer service; drug plan member complaints, members who choose to leave, and
Medicare audit findings; member experience with drug plan; drug pricing and patient safety.

IMPAQ International, LLC

2

OMB Supporting Statement

outcome or decision," to prevent confusion with a beneficiary's opinion of the decision. A focus
on the ―final outcome or decision‖ rather than ―resolution‖ puts the focus on the series of actions
the plan took, regardless of whether the beneficiary believes his/her complaint was resolved.
Third, CMS recognizes the potential for complaints that are outside the scope of the plan
(restricted by CMS guidelines), particularly regarding some issues related to enrollment,
complaints and will exclude such complaints from the sampling framework.
The premise of the proposed data collection is to conduct a full study of the entire population
(contracts). CMS will review the results of the survey responses and the analysis and then decide
whether to convert the information gathered through the survey to a set of performance measures
to be used by Medicare beneficiaries for the next plan year. As mentioned before, it is also
possible that CMS may choose to use the results for monitoring purposes only for any number of
reasons.
There may be some potential and yet-undefined issues that would have to be addressed prior to
using the information from the survey for public reporting purposes. Some potential issues are
listed and described below:
a) Are beneficiaries responding distinctly to issues of complaint settlement and resolution?
Can we produce performance measures that inform the difference between these two? To
some extent, the pilot test will provide some insight here. However, we may need the
larger scale data collection to provide more information on this issue.
b) Have the analytical methods used for composing the performance measures produce
results that provide distinct outcomes across beneficiary satisfaction levels and across
contracts?
c) A low response rate for certain contracts and/or low number of complaints during the
data collection period (real-time data collection).
d) Positive response from industry to preliminary results regarding measurement of
beneficiary satisfaction of complaint resolution.
It is important to note that CMS has addressed most technical issues with assessing the
representativeness of the complaints in the sampling universe and the survey instrument has been
vetted with several survey and Medicare experts including CMS staff involved on other CMS
surveys (such as the Consumer Assessment of Healthcare Providers and Systems (CAHPS)
survey). Thus, any issues that are reviewed during analysis are likely to be outside of these
technical aspects. Points (a) and (c) above demonstrate the necessity of a larger scale data
collection (6,500 beneficiaries) in order to ensure the high quality of the data and its viability for
the development of a performance measure. CMS will review and analyze the responses to the
survey and discuss the results with subject-matter experts at CMS and other institutions as well
as selected Part C and D contracts to assess its usability and/or representativeness of beneficiary
satisfaction with the complaint resolution process.

IMPAQ International, LLC

3

OMB Supporting Statement

In addition to the vetting and technical reviews of the survey completed in advance, CMS will
conduct a pilot test of 100 beneficiaries (in addition to the pre-test consisting of cognitive
interviews conducted with 9 beneficiaries) to work out any technical and operational issues with
the instrument or study logistics. Under very limited circumstances, CMS will consider minor
adjustments to the instrument. The pilot test will simulate all tasks and activities of the full-scale
study from mailing the advance letters and survey sample management to the statistical analysis
of the survey responses and contract and beneficiary information.
It is important to note that all survey responses will be submitted to a rigorous analytical review
using multinomial and regression analyses to produce measures that are controlled by contract
characteristics (enrollment, type of contract, etc) and beneficiary characteristics. We will also
compute correlation factors among different survey responses and other contract and beneficiary
factors. The multinomial and regression analyses will support statements of the likelihood of a
beneficiary being satisfied or not with a statistical level of certainty. Further information on the
proposed analysis is included in Section B.16.a. Tabulations, Analysis.
As in the past, CMS strives to share information and communicate with plans on issues of
performance measures. However, at this time, it is undetermined what level of data will be
shared with plans regarding the results of the preliminary data collection effort and whether the
data will be used to develop performance measures in this area. It should be noted that even after
all technical issues have been addressed, CMS has vetted the results with the industry and there
is consensus that the results of the study meet CMS objectives, CMS may still choose to not
using the information for public reporting of performance measures and use the information for
program monitoring.

IMPAQ International, LLC

4

OMB Supporting Statement

B. Justification
1. Need and Legal Basis
This clearance package seeks approval to conduct a survey as part of the Part C and D
Complaints Resolution Performance Measure project. This preliminary survey effort is
sponsored by the Centers for Medicare & Medicaid Services (CMS) and is being implemented,
under contract to CMS, by IMPAQ International, LLC. The purpose of the project is to develop
and support implementation of a performance measure for the Medicare Advantage (Part C) and
Prescription Drug (Part D) program that represents, from the beneficiary’s perspective, the way
in which plans handle complaints.
The proposed data collection is necessary because a survey is the only way to collect information
about the resolution process from the beneficiary’s perspective. Currently, there is no other data
source that collects such information for Part C and Part D Medicare plans.
The proposed survey instrument is attached to this statement. An accompanying document,
―Explanations for the Inclusion of Survey Questions,‖ presents the rationale for the inclusion of
each question in the survey.
2. Information Users
Data collected from the proposed surveys will be used by CMS to construct performance
measures of the veracity and effectiveness of plan complaint resolution from the beneficiary’s
perspective. If CMS chooses to publish a performance metric using the results of the survey, it
will do so in November 2011.
3. Use of Information Technology
Computer Assisted Telephone Interviewing (CATI) will be used by interviewers to conduct
telephone surveys. Paper surveys will be sent through the mail to beneficiaries who cannot be
reached by phone. The mail-in surveys will be identical to the telephone surveys and will not be
analyzed separately. Telephone interviews are more cost-effective and impose less burden on
respondents than do in-person interviews. The CATI system is installed on each interviewer’s
computer and assists the interviewer in conducting the survey by presenting each survey
question and answer choices and automatically following skip patterns. Only the interviewer has
contact with the system. CATI is more cost effective than paper and pencil interviewing for
many reasons, including the fact that CATI programs accept only valid responses and can be
programmed to check for logical consistency across answers. Interviewers are thus able to
correct errors during the interview, eliminating the need to call back respondents to obtain
missing data. Also, calls will be made through an auto-dialer, linked to the CATI system,
virtually eliminating dialing error. The automated call scheduler will simplify the scheduling

IMPAQ International, LLC

5

OMB Supporting Statement

and rescheduling of calls to respondents at their convenience and can assign cases to specific
interviewers, for example, those who are fluent in Spanish.
The two modes of data collection (telephone and mail-in) include all the same questions and
answers as the survey instrument is identical in both cases. The instrument was designed to be
administered as easily by phone as by paper-and-pencil self-administration. The sole difference
between the modes is that FAQs will be available to beneficiaries completing the telephone
survey. However, this is not anticipated to affect the results of the data collection and the data
will not be separated or treated differently in analysis. The FAQs will include general
information that will also be found in the Advance Letter and the cover page of the survey
instrument. For example, FAQs will cover questions beneficiaries may have about why they are
being contacted, how their contact information was compiled, whether participation in the survey
will affect their current benefits, and IMPAQ International’s role in the data collection. As these
types of questions will also be covered by the mailed materials, there is little reason to expect the
mode of data collection to effect the uniformity of responses.
As detailed in Supporting Statement B (Section 3.a. Response Rates) telephone surveys are the
primary mode of data collection. If any differences arise between the two modes, the effects will
be minimized by the relatively small number of surveys completed by paper-and-pencil.
Methods to distinguish any differences are described in the analysis overview in section B.16.a.
Tabulations, Analysis Plan. The statistical analysis has taken into account the two modes of data
collection and a control variable will be included in the multinomial analysis to assess the
statistical effect of the results regarding beneficiary satisfaction.
4. Duplication of Efforts
This survey will be conducted to collect key information from CMS beneficiaries. No other
survey data collection effort has been conducted or has been planned to collect similar
information. The study also will use administrative data from the Complaints Tracking Module
(CTM) and other CMS datasets, such as the Medicare Beneficiary Database (MBD, Common
Tables). CTM data are not sufficient to conduct the study because they do not include the
perspective of beneficiaries; hence, survey data are needed to supplement the CTM and other
CMS data.
5. Small Businesses
The survey will only involve individual beneficiaries; therefore, it will not pose a burden to
small businesses. Members of 800 series contracts will be excluded from the data collection
effort for reasons described in Supporting Statement B (Section 1: Respondent Universe and
Sampling).

IMPAQ International, LLC

6

OMB Supporting Statement

6. Less Frequent Collection
The survey will be the primary source of data for the construction of the beneficiary-focused
performance measures. The survey will collect information about the resolution process and the
resolution from the beneficiary’s perspective. Currently, there is no other data source that
collects such information for Part C and Part D Medicare plans.
The beneficiaries affiliated with the sampled complaints for this project will only be surveyed
once. Therefore, less frequent data collection is equivalent to not being able to collect any data
and would result in an inability to construct the performance measures.
7. Special Circumstances
In all respects, the data will be collected in a manner consistent with Federal guidelines. The
statistical survey will produce valid and reliable results that can be generalized to the universe of
the study, and it will include only statistical data classifications that have been reviewed and
approved by OMB. The survey will include a pledge of confidentiality that is supported by
authority established in statute or regulation and by disclosure and data security policies that are
consistent with the pledge. It will not unnecessarily impede sharing of data with other agencies
for compatible confidential use.
8. Federal Register/Outside Consultation
a. Federal Register Notice and Comments
The first Federal Register Notice and OMB PRA Package [Document Identifier: CMS-10308]
was published in the Federal Register on February 25, 2010 (Vol. 75, No. 37, page 8723). A
copy of the publication is attached to this package. In response, ten organizations submitted 136
comments.
The tables below summarize the revisions made to the Advance Letter, Explanation for Inclusion
of Survey Questions, Supporting Statements A and B, and the Survey Instrument.
Advance Letter
Two main issues that arose were the reading level and clarity of the letter. As such, CMS has
revised the letter to read at the 8th grade level. A few clarity issues that the organizations
mentioned pertained to the logistics of scheduling surveys with beneficiaries, explanation of the
purpose of the survey, specific reference to the beneficiary’s MAO or Part D sponsor (not
Medicare), and an explanation of IMPAQ International’s role. The table below summarizes
these revisions to the letter.

IMPAQ International, LLC

7

OMB Supporting Statement

Effect to
Reporting
Burden

Category

Section

Change/Reason

Response to
Public
Comments

Advance
Letter

The reading level of the letter was lowered to 8th grade.

None

Response to
Public
Comments

Advance
Letter

Language was added to the Advance Letter to describe the
calling process and how beneficiaries should expect to be
contacted.

None

Response to
Public
Comments

Advance
Letter

Language was added to the Advance Letter to describe the
intended use of collected data as opposed to: "Your answers
will be kept strictly confidential and be used only for research
purposes." The purpose of the data collection (to improve how
complaints are handled and to inform the development of a plan
rating system) was explained.

None

Response to
Public
Comments

Advance
Letter

The advance letter has been revised to make explicit reference
to the MAO or Part D sponsor as well as the role of IMPAQ
International (contractor). This will reduce confusion about the
topic and source of the complaints.

None

Explanations for Inclusion of Survey Questions
There was a concern that the proposed indicators did not correspond with the survey questions.
CMS has revised the explanation of the proposed indicators, together with the descriptions of
each survey question, to better explain their importance in developing performance measures.

Category

Section

Response to
Public
Comments

Explanations
for Inclusion
of Survey
Questions

Effect to
Reporting
Burden

Change/Reason
This document describing the proposed indicators has been
revised to better explain their importance, meanings, and how
they will help develop the performance measures.
Specifically:
– Rationale for use of "resolution" vs. "final outcome"
– Use of 4-point Likert scale vs. 5-point Likert scale (and
addition of "I don't know/NA" answer choice)
– Strengthening of rationale for inclusion of Veracity of Plan's
Description indicator

None

Supporting Statements A and B
There was some concern about the purpose and calculation of indicators such as ―Veracity of
Complaints Resolution,‖ and ―Beneficiary Awareness of Resolution.‖ Further information
regarding these indicators has been added to the supporting statements. Some organizations
were concerned with how the results of the survey will be utilized and if/when that information
would be shared publicly. At this time, it is undetermined which data will be shared with the
plans.

IMPAQ International, LLC

8

OMB Supporting Statement

Some comments suggested that the survey instrument should undergo an evaluation by a quality
measurement organization. Because this is a preliminary effort to assess the possibility of
developing performance measures, this type of review is unnecessary at this time.
Some plans mentioned that overall satisfaction may be affected by dissatisfaction with the final
outcome and resolution of the complaint. CMS has revised the language to ask beneficiaries
about several aspects of their satisfaction besides the final outcome. Similarly, concerns about
confusion between ―resolution‖ and ―final outcome‖ have been addressed by emphasizing in the
survey that CMS is interested in the series of actions taken by the plan to resolve complaints
(e.g., the final outcome), rather than emphasizing the resolution.
It was suggested that the proposed sampling plan would not allow plans sufficient time to inform
the beneficiary of the resolution. CMS has chosen to allow a delay of 7 days after the complaint
is closed before contacting the beneficiary. In addition, CMS has provided further clarity on the
types of contracts and plans that would be subject to data collection; this entailed an explicit
exclusion of 800 series contracts and the inclusion of small contracts and contracts with a small
number of complaints.
Some plans expressed more general concerns regarding the survey. These included how
beneficiary-appointed representatives would be contacted, the uniformity of the survey approach
for both telephone and mail-in, the reliability of collected data, and analyses to control for factors
outside of the plan’s control. The table below summarizes the actions CMS has taken to address
these and other concerns in Supporting Statements A and B.
Several comments objected to the possible inclusion of complaints that are outside the scope of
plans. CMS has chosen to exclude complaints that are likely to be outside of the scope of plans,
primarily some issues related to enrollment complaints.
Effect to
Reporting
Burden

Category

Section

Change/Reason

Response to
Public
Comments

Supporting
Statement A

Clarification was added regarding how CMS intends to
disseminate results of the survey. CMS will follow a process
for this performance measure similar to the one it uses for other
measures. At this time, it is undetermined what level of data
will be shared with plans.

None

Response to
Public
Comments

Supporting
Statement A

Information regarding the use and calculation of the "Veracity
of Complaint Resolution" indicator has been added.

None

Response to
Public
Comments

Supporting
Statement A

Information regarding the use and calculation of indicators for
new survey questions has been added.

None

IMPAQ International, LLC

9

OMB Supporting Statement

Effect to
Reporting
Burden

Category

Section

Change/Reason

Response to
Public
Comments

Supporting
Statement A

The calculation of the "Beneficiary Awareness of Resolution"
indicator has been clarified. The description has been revised
so that it more clearly refers to the answer choices for this
survey question and explains how the answers will influence the
calculation of the indicator.

None

Response to
Public
Comments

Supporting
Statement A

None

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

The Information Users section has been updated to reflect that
CMS may opt not to use the results of the survey for
performance measurement. This is a preliminary gathering of
information to determine the possibility of developing a
performance measure..
Information has been added regarding the participation of
representatives in the survey. This explanation includes how
representatives will be contacted (through beneficiaries and/or
CTM logs) and how data from representatives can be used in
the survey data analysis. A question has been added to identify
individuals other than the beneficiary who participate in the
survey.
Language has been added to emphasize that this is a preliminary
survey effort to assess the beneficiary's satisfaction with the
complaint resolution.

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

The pretest and pilot test methodologies have been clarified to
emphasize efforts to achieve reliable data and remove
complaints that are not within the plan’s domain.

None

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

Clarification has been added to indicate that the telephone and
written survey instrument will have a uniform format.

None

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

It has been clarified that members of 800 series contracts will
be excluded from the data collection effort.

None

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

Controlling the validity of complaints: Approaches to control
for factors affecting satisfaction with the final outcome and the
complaint resolution have been described. Also, language has
been added to underscore that several factors will be considered
in the development of the performance measure so as not to rely
on a single aspect of the beneficiary's experience.

None

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

The sampling strategy will exclude complaints that are outside
the scope of the plan (restricted by CMS guidelines),
particularly regarding issues related to enrollment complaints.

None

IMPAQ International, LLC

10

None

None

OMB Supporting Statement

Effect to
Reporting
Burden

Category

Section

Change/Reason

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

The difference between "resolution" and "final outcome" has
been clarified. An emphasis on ―final outcome or decision‖
rather than ―resolution‖ puts the focus on the series of actions
the plan took, regardless of whether the beneficiary believes
his/her complaint was resolved.

None

Response to
Public
Comments

Supporting
Statement A
&
Supporting
Statement B

Additional explanation has been added to the sampling plan
(supporting statement B) and background (supporting statement
A) regarding the selected data collection period. CMS is
interested in the months with the largest number of complaints
in order to achieve the most statistically valid sample..

None

Response to
Public
Comments

Supporting
Statement B

Call center procedures have been further defined and clarified
regarding how staff encourage participation without being
forceful. The explanation of the CATI system has been refined
to prevent misunderstandings of the purpose of the system.

None

Response to
Public
Comments

Supporting
Statement B

The sampling framework has been modified to accommodate 7
days between complaint closure and initial contact with the
beneficiary to allow time for beneficiaries to receive
notification of their complaint resolution.

None

Response to
Public
Comments

Supporting
Statement B

Explanation has been added regarding the decision to proceed
with collecting data on small contracts and contracts with a
small number of complaints.

None

Survey Instrument
Many organizations had general comments regarding the survey instrument. Some believed that
the wording of the questions might yield negative responses and not allow for neutral responses.
CMS has reworded the survey to be more specific on the key complaint resolution process and
include answer choices of ―I Don’t Know/NA.‖ The latter will provide an answer that allows
beneficiaries a choice outside the 4-point likert scale to accommodate beneficiaries who feel that
they do not yet have a resolution or are unsure/do not remember. Other concerns included
language in the introduction that did not clarify why beneficiaries are being contacted, that did
not fully explain the role of IMPAQ, and that lacked clarity regarding the complaint process
(whether the beneficiary filed a complaint against their MAO/PDP sponsor/Medicare.).
Concerns also were expressed regarding specific questions on the survey instrument. Some
plans commented that the use of the word ―resolution‖ would not elicit the intended responses
from beneficiaries; therefore, the word ―resolved‖ has been replaced with ―settled‖ in question 1
to prevent bias to respond negatively unless the final outcome was in favor of the beneficiary.
Also, in question 2 and question 3, ―resolution‖ has been replaced with ―final outcome.‖
Revising the wording will help beneficiaries to focus on the actions taken by the plan and not
their opinion of the decision.

IMPAQ International, LLC

11

OMB Supporting Statement

CMS has decided to remove questions 4, 5, 6, and 8 from the survey instrument. A new question
2 has been created to include beneficiary satisfaction with the level of communication given by
the plan and other aspects of beneficiary satisfaction such as the courtesy of the plan
representative and explanation of the final outcome.
With regard to questions 5 and 6, after the results of the pretest were reviewed, it became clear
that beneficiary responses were not aligned with the intended purpose of the survey questions.
Question 5 yielded responses about beneficiary burden rather than about the filing of duplicate
complaints. Furthermore, respondents did not differentiate between their satisfaction with the
amount of time it took to resolve their complaint and their satisfaction with how the complaint
was handled by the plan; it appeared that these two satisfaction ideas were confounded in the
beneficiary’s mind.
Clarifying text has been added to question 7, to address concerns that beneficiary satisfaction
would be dependent on whether or not the complaint was resolved. Language has been added to
this question, asking the beneficiary to focus on the way the complaint was handled, regardless
of whether or not he/she is satisfied with the final outcome. This overall satisfaction question is
now Q6 in the revised survey.
It was suggested that questions 9 and 10 be rewritten because of their negative connotations.
CMS has revised both of these questions to be more neutral. Question 9 is now question 8, and
question 10 is now question 3 in the revised survey instrument.
CMS has added or reformulated some questions to the survey instrument without affecting the
reporting burden. To set a baseline for the beneficiary’s satisfaction with the plan, a question
was added to obtain information on how satisfied beneficiaries are with their plan. This will be
question 7 on the revised survey. Respondents are now asked to identify whether they are the
beneficiary or a representative. This is question 9 on the revised survey. Last, an open-ended
question was placed at the end of the survey. This allows respondents an opportunity to provide
feedback about the complaints process and make suggestions for improvement. The table below
summarizes the actions CMS has taken to address comments on the survey instrument. Again,
there was no change in the reporting burden.
Effect to
Reporting
Burden

Category

Section

Change/Reason

Response to
Public
Comments
Response to
Public
Comments

Survey
Instrument General
Survey
Instrument Introduction

The survey questions have been revised to reflect more neutral
wording.

None

None

Response to
Public

Survey
Instrument -

Details have been added to the survey introduction explaining
why beneficiaries are being contacted and specifying terms such
as "Medicare," MAO, or Part D sponsor, and the role of the
contractor in conducting the survey.
"Resolved" has been replaced with "settled" in this question to
prevent beneficiary bias. An "I don't know" answer choice has

IMPAQ International, LLC

12

None

OMB Supporting Statement

Effect to
Reporting
Burden

Category

Section

Change/Reason

Comments

Q1

Response to
Public
Comments

Survey
Instrument Q2

Response to
Public
Comments

Survey
Instrument Q3

Response to
Public
Comments
Response to
Public
Comments
Response to
Public
Comments
Response to
Public
Comments

Survey
Instrument Q4
Survey
Instrument Q5
Survey
Instrument Q6
Survey
Instrument Q7

Response to
Public
Comments

Survey
Instrument Q8

been added for beneficiaries who feel they do not yet have a
resolution or are unsure/do not remember.
"Resolution" has been replaced with "final outcome or
decision" in this question to prevent beneficiary bias and to
guide the beneficiary towards the actions taken by the plan as
opposed to the beneficiary's opinion of the decision. This
question is now Q4 in the new survey instrument.
"Resolution" has been replaced with "final outcome" in this
question. An "NA" answer choice is available for beneficiaries
who do not believe they have received a final outcome or who
do not remember the resolution of their complaint. This
question is now Q5 in the new survey instrument.
Question 4 has been removed, and issues of plan
communication with the beneficiary have been incorporated in
Q2 of the new survey instrument.
Question 5 has been removed from the survey. (CMS has
decided to drop questions about repeat complaints or multiple
attempts to contact the plan)
Question 6 has been removed from the survey. Some issues
related to the amount of time it took to resolve a complaint have
been incorporated in Q2 of the new survey instrument.
Clarifying text has been added to this question asking the
beneficiary to disregard whether or not he/she is satisfied with
the final outcome. The purpose of this question is to provide an
overall satisfaction rating. This question is now Q6 in the new
survey instrument.
This question has been removed. The aspects of the complaint
process that were included in this question have now been
incorporated in Q2 of the new survey instrument. This revision
will allow all respondents to rate their satisfaction with
components of the process (as opposed to the skip pattern in the
original instrument).

Response to
Public
Comments
Response to
Public
Comments

Survey
Instrument Q9
Survey
Instrument Q10

Response to
Public
Comments

Survey
Instrument –
New question

Response to
Public
Comments

Survey
Instrument –
New question

Response to
Public
Comments

Survey
Instrument –
New question

IMPAQ International, LLC

This question has been reworded to be more neutral and to ask
about the likeliness of the beneficiary to stay with the plan.
This question is now Q8 in the new survey instrument.
The word "problems" has been removed from this question and
the wording is more neutral (both in the question stem and in
the answer choices). This question is now Q3 in the new survey
instrument.
Add question to assess beneficiary satisfaction with aspects of
the complaint handling process. Beneficiaries will rate their
satisfaction with components of the handling process such as
length of the complaint process and courtesy of the plan
representative. In a simplified form, this satisfaction question
addresses issues from the original Q6 and Q8.
This question is now Q2 in the new survey instrument.
Add question to assess beneficiary satisfaction with the plan.
This question sets a baseline for the beneficiary's overall
satisfaction with the plan.
This question is now Q7 in the new survey instrument.
Add question to identify survey respondent. This is a
demographic question to differentiate between respondent and
proxy.

13

None

None

Decrease

Decrease

Decrease

None

Decrease

None

None

Increase

Increase

Increase

OMB Supporting Statement

Category

Section

Response to
Public
Comments

Survey
Instrument New question

Effect to
Reporting
Burden

Change/Reason
This question is now Q9 in the new survey instrument.
Add question to elicit beneficiary feedback. Beneficiaries will
be asked for feedback at the end of the survey, e.g., whether
they have any comments or suggestions for improvement of the
complaint resolution process.
This question is now Q10 in the new survey instrument.

Increase

b. Consultation Outside of the Agency
The following individuals were consulted in designing the data collection plan and developing
the questionnaire:
Name
Oswaldo Urdapilleta
Alicia Schoua-Glusberg
Jasmine Ainetchian
Camellia Bollino
Julie Young
Donald Nichols
Peg Stessman
Kathy Goeser

Affiliation
IMPAQ International
IMPAQ International
IMPAQ International
IMPAQ International
IMPAQ International
IMPAQ International
Strategic Health Solutions
Strategic Health Solutions

Telephone Number
(202) 289 0004 x503
(847) 864 5677
(202) 289 0004 x502
(443) 718 4356
(443) 539 9766
(443) 539 0218
(402) 452 3333
(402) 452 3333

No unresolved problems were identified by any of these individuals.
9. Payments/Gifts to Respondents
There will be no respondent payments for this survey.
10. Confidentiality
IMPAQ International will follow procedures for ensuring and maintaining confidentiality
consistent with provisions of the Privacy Act of 1974. Respondents will receive information
about confidentiality protection in an advance letter describing the survey (provided as an
attachment to this package) and again at the outset of the interview as part of the interviewer's
introductory comments. Respondents will be informed that all information they provide will be
treated confidentially. Interviewers will be trained in confidentiality procedures and will be
prepared to describe these procedures in full detail, if needed, or to answer any related questions
from the respondents. For example, if asked about confidentiality, the interviewer will explain
that the answers will be combined with those of others and presented in summary form only, that
no identifiable information about participants will be made public, and that the answers will not
affect past or future eligibility for any programs.

IMPAQ International, LLC

14

OMB Supporting Statement

All data items that identify respondents will be kept only by the contractor, IMPAQ
International, for use in assembling records data and conducting the interviews. Any data
received by CMS will not contain personal identifiers, thus precluding individual identification.
In addition, the following safeguards will be employed to carry out confidentiality assurances:
All employees at IMPAQ sign a confidentiality pledge that emphasizes the importance of
confidentiality and sets forth the obligations of staff.
Identifying information is maintained in a separate file from interview data. The files are
linked only with a sample identification number.
Access to link-files containing sample identification numbers connecting the research
data and the respondents' identification is limited to a few individuals who have a need to
know this information.
Access to any hard-copy documents is strictly limited. Physical precautions include use
of locked files and cabinets, shredders for discarded materials, and interview control
procedures.
11. Sensitive Questions
The survey of CMS beneficiaries contains a minimal set of items that may be considered
sensitive in nature. These questions are related to adverse medical episodes experienced by
beneficiaries with complaints. These questions are needed to evaluate the frequency and degree
to which beneficiaries suffer as a result of the amount of time spent by plans to determine a final
outcome to complaints. As described in item A10, all respondents will be assured of
confidentiality at the outset of the interview. All survey responses will be held in strict
confidence and reported in aggregate, summary format, eliminating the possibility of individual
identification. IMPAQ International will comply with the requirements of the Privacy Act of
1974, in collecting all information.
12. Burden Estimates (Hours & Wages)
The total annual hour burden for respondents for the proposed information collection is shown in
Table 1 below. Total burden hours are based on 100 pilot test responses and 5,200 main survey
responses (5,300 total responses).

IMPAQ International, LLC

15

OMB Supporting Statement

Table 1: Annual Hour Burden for Respondents
Frequency of
Data
Collection
100
Once

Total Survey
Respondents

Cite/reference
Complaints
Resolution:
Pilot Test
Complaints
Resolution:
Main Survey
Total

10 minutes

Annual
Hour
Burden
17

Average Time per
Respondent

5,200

Once

10 minutes

867

5,300

Once

10 minutes

884

The total annualized cost to respondents of collecting this information is shown in Table 2
below.
Table 2: Annualized Cost to Respondents

Respondent
Category

Total
Number of
Hours
4,431*
739

Number of
Respondents

Not working
Working
full-time
Working
part-time
Total

$0.00

Estimated Data
Collection Cost
to Respondents
$0.00

Estimated
Cost per
Respondent
$0.00

Hourly
Rate

493*

82

$22.96**

$1,883

$3.82

376*

63

$13.06**

$823

$2.19

5,300

884

$2,706

* Based on U.S. Bureau of Labor Statistics figures for labor force participation of workers 65+
** Mean hourly earnings based on the National Compensation Survey, Dec. 2007-Jan. 2009
13. Capital Costs
This is a new, one-time survey. There will be no capital or start-up costs incurred by
respondents. There are no record keepers. There will be no costs to respondents for operations,
maintenance, or purchase of services.
14. Cost to Federal Government
The cost to the Federal government of conducting the survey is $522,623, which is the total
contractor cost of conducting the survey.

IMPAQ International, LLC

16

OMB Supporting Statement

15. Changes to Burden
This is a new data collection effort.
16. Publication Tabulation Dates
a. Tabulations
All survey data will be combined with CMS administrative data, including the Medicare
Beneficiary database, HPMS contract information, and the Complaints Tracking Module (CTM)
records. Development of the preliminary performance measures will be derived from survey and
administrative data. All measures are preliminary and will be run through exploratory, driver,
and risk-adjusted analyses. The preliminary measures are listed below under each respective
research domain:
Beneficiary Satisfaction
o Beneficiary satisfaction with the complaint handling process
o Beneficiary satisfaction with the plan
Resolution Effectiveness
o Beneficiary awareness of resolution
o Beneficiary satisfaction with final outcome
o Veracity of plan’s description of final outcome or decision
Plan Effectiveness
o Beneficiary experiences during complaint resolution process
o Areas for improvement in the complaint handling process
The research domains are described below and followed by details of how the indicators are
linked to the survey instrument questions and how they will be used in analysis.
Research Domains:
Beneficiary Satisfaction
In any industry, it is necessary to study the satisfaction of the consumer. There are many ways to
define this term. To truly understand satisfaction, the researcher must evaluate consumers’
emotions, which fluctuate within individuals and vary in range across individuals. 4 Measuring
the consumer’s emotions cannot be accomplished externally; therefore, optimal surveys ask
consumers to rate their own emotions. Satisfaction is then the favorability of their subjective
assessment of the organization, company, or group.5 In the present context, the rating is a
4

H. K. Hunt (1977), ―CS/D-Overview and Future Research Direction,‖ in Conceptualization and
Measurement of Customer Satisfaction and Dissatisfaction, ed. H. K. Hunt, L. Hu, and P. M. Bentler.
5 R. A. Westbrook (1980), ―A Rating Scale for Measuring Product\Service Satisfaction,‖ Journal of Marketing,
44: 68-72.

IMPAQ International, LLC

17

OMB Supporting Statement

representation of the beneficiary’s sense of fulfillment with the customer service experience.6
Have all his/her needs been met? Have all expectations been met?
It is important to note that a plan may comply with all CMS requirements, yet still receive a poor
rating in this measure. Without regard to the plan’s level of compliance with CMS requirements,
beneficiaries reflect on their subjective evaluation of the plan’s ability to resolve the complaint
and adequately address their issues.
The following indicators from the beneficiary survey correspond to the beneficiary satisfaction
domain:
Beneficiary satisfaction with the complaint handling process
Beneficiary satisfaction with the plan
For these satisfaction questions and others, CMS is using a 4-point Likert scale rather than a 5point Likert scale (with a neutral option) to encourage beneficiaries to provide an opinion on
each question. The use of a 4-point scale will improve the survey results. However, an N/A
answer choice is available to beneficiaries who find the questions not be applicable such as
beneficiaries who believe their complaint has not been resolved.
Resolution Effectiveness
Resolution effectiveness relates to both effectiveness and veracity. Questions of resolution
effectiveness assess whether a complaint was resolved satisfactorily from the beneficiary’s
perspective. This is a subjective, but clear-cut, outcome measure of the complaint resolution
process. However, resolution effectiveness also addresses resolution veracity, by comparing the
beneficiary’s knowledge of the complaint resolution with the plan’s stated resolution, which can
be found in the CTM. While the plan may have appropriately resolved the complaint and
addressed all of the beneficiary’s concerns, the beneficiary may not have been informed of the
plan’s decision. Veracity refers to the matching of beneficiary and plan information about the
resolution. Moreover, the resolution effectiveness research question separates the plan’s ability
to properly address and resolve complaints from its ability to handle complaints well and to the
beneficiary’s satisfaction.
The following indicators from the beneficiary survey correspond to the resolution effectiveness
domain:
Beneficiary awareness of resolution
Beneficiary satisfaction with final outcome
6

R. L. Oliver (1997), Satisfaction: A Behavioral Perspective on the Consumer (New York: McGraw-Hill).

IMPAQ International, LLC

18

OMB Supporting Statement

Veracity of plan’s description of final outcome or decision
Plan Effectiveness
Plan effectiveness refers to the complaint resolution process, as evaluated by the beneficiary.
Questions about plan effectiveness will demonstrate how the beneficiary believes the complaint
was handled by the plan. Regardless of whether the plan resolved the complaint to the
beneficiary’s satisfaction (what the outcome was), the perceived difficulty of the process may
affect the beneficiary’s subjective evaluation of the plan’s performance. Subjects included in
this domain are communication, timeliness, and consequences for the beneficiary.
The following indicators from the beneficiary survey correspond to the plan effectiveness
domain:
Beneficiary experiences during complaint resolution process
Areas for improvement in the complaint handling process
―Beneficiary Experiences during the Complaint Resolution Process‖ is calculated from the
percentage of beneficiaries who experienced any potential problems while waiting for a
complaint to be resolved. ―Contact by Plan‖ is the percentage of complaints where the
beneficiary was contacted by the plan. ―Repeat Complaints‖ highlights the percentage of
beneficiaries who contacted the plan more than once before their complaint was resolved.
Indicators:
In principle, the exploratory analysis – described in greater detail in the following section – will
include tabulations of survey item responses and beneficiary and plan characteristics. A driver
analysis could determine statistically which areas most impact overall customer satisfaction, and
it may be possible to estimate the direction and magnitude in which the drivers impact overall
satisfaction. We could determine which specific attributes have the most impact on overall
customer satisfaction and, therefore, would warrant primary attention and resources for CMS.
For measure development, we will conduct analyses to construct risk-adjusted measures, using
beneficiary characteristics, plan characteristics, and complaint characteristics.
These analyses will minimize measurement bias associated with confounding factors affecting
the satisfaction measures. For example, beneficiaries with certain characteristics may have
higher or lower levels of satisfaction than the average beneficiary. Overall, preliminary
performance measures may be risk-adjusted to account for beneficiary and plan characteristics.
In all analyses, we will include information regarding whether the respondent is the beneficiary
or a representative, since we assume that there may be differences between the satisfaction levels
of these two groups.

IMPAQ International, LLC

19

OMB Supporting Statement

Beneficiary satisfaction with the complaint handling process. Several measures will be
developed to assess how satisfied complainants are with different aspects of the complaint
resolution process. The main aspects include the following: length of the complaint process;
courtesy of the plan representative; time your plan took to contact you; amount of time spent
handling your complaint; awareness of the complaints process; and explanation of the final
outcome. Beneficiary satisfaction on these issues will be measured through the following survey
questions:
Q2. Thinking about the aspects of the complaints process, regardless of whether
you agree or disagree with the final outcome, please indicate how satisfied
you are with each of the following: [specific items follow]
Q6. Whether you agree or disagree with the final outcome, how would you rate
your overall satisfaction with the way your complaint was handled by the
plan?
The responses to each item in Question 2 and to Question 6 will together produce a preliminary
measure of the complaint resolution process. The items in Question 2 will highlight the extent to
which plans treated the complainant courteously, provided the complainant with understandable
explanations of the final outcome, and provided the complainant with enough information about
how the complaint was resolved. Question 6 will measure how satisfied beneficiaries are with
the overall process undertaken by plans to resolve their complaint.
In responding to these questions (and each item within Question 2), the complainant must answer
―very satisfied,‖ ―satisfied,‖ ―dissatisfied,‖ ―very dissatisfied,‖ or ―I don’t know/NA.‖ These
responses will be translated into numerical values, where complainants who are very satisfied
will have the greatest value (2), and those who are very dissatisfied will have the smallest value
(-2). Each preliminary measure will be calculated as the mean value of the numerically
translated responses about the complainants’ satisfaction. Therefore, larger values will indicate
better plan performance in handling complaints to complainants’ satisfaction. It is important to
note that the survey emphasizes the final outcome rather than the settlement (or resolution).
Final satisfaction with the process by which the complaint was handled (Q2) will be tabulated
against satisfaction with each of the main aspects of the process (Q6). The responses to Question
6 will be used to isolate complaints where the beneficiary is satisfied with the final outcome, but
has concerns about various aspects of the process.
As with the other measures, this measure will be tabulated against beneficiary and plan
characteristics, and we will run risk-adjustment models. No single item will define satisfaction
for each aspect of the complaint resolution process. In particular, different complaint categories
may affect each aspect of the process in different ways.

IMPAQ International, LLC

20

OMB Supporting Statement

Beneficiary satisfaction with the plan. An indicator of overall satisfaction with the plan will be
included in the analysis. This indicator will be used to analyze the satisfaction measures listed
above as well as allow for further analysis of critical satisfaction issues that may have significant
repercussions in a beneficiary’s selection of a plan. The information for overall beneficiary
satisfaction will be derived from the following survey questions:
Q7. Based on your recent experience, how satisfied are you with [Plan name]?
Q8. How likely are you to stay with this plan?
Both questions address issues of the beneficiary’s satisfaction with the plan including health care
and the complaint resolution process. Question 8 will indicate to what extent a beneficiary’s
experience with the complaint resolution process affects his/her opinion of the plan. Being able
to tabulate both questions will be insightful since we will learn about the overall satisfaction with
the plan. On the one hand, if the complaint resolution process was so unsatisfactory that the
beneficiary is willing to switch plans, it will provide context for plans with low ―Resolution
Handling‖ indicators. On the other hand, analysis of these responses may show that even
beneficiaries with unsatisfactory complaint resolution experiences are not unhappy enough to
actually switch plans.
Beneficiary Awareness of Resolution. This measure will capture the percentage of a plan’s
complainants who either are aware or agree that a settlement to their complaint has been
implemented or reached. Since the complaints in our sample have been closed in the CTM, the
expectation is that a complaint has been settled and that the beneficiary is aware of this
resolution.
From the survey, the response to the following question will be used to calculate this measure:
Q1. According to our records, the complaint you filed about [Complaint
Category] was recently closed by the plan. Was the complaint settled?
Complainants may answer say ―yes,‖ ―no,‖ or ―I don’t know.‖ To calculate this measure, the
numerator will be a count of the number of sampled complaints in which the complainant
answered ―yes.‖ The denominator will be the total count of sampled complaints in which the
complainant indicates either ―yes‖ or ―no‖ regarding the resolution of his/her complaint.
Therefore, the exclusion criteria for the denominator will be a response of ―I Don’t Know‖ or
those who did not answer that question.
Depending on the outcomes of the exploratory analysis and tests of the correlation between
satisfaction with the final outcome and the awareness of a resolution, CMS likely will treat this
measure as a monitoring measure. Using the data as a monitoring measure makes sense also

IMPAQ International, LLC

21

OMB Supporting Statement

because of the potential that some beneficiaries would respond affirmatively (―yes, the complaint
was settled‖) only if the complaint had been resolved in their favor.
Beneficiary satisfaction with final outcome. This measure will relay the extent to which
complainants are satisfied with the final outcomes or decisions that plans have provided
regarding their complaints. The assessment of satisfaction with the final outcome will be
collected through the following question:
Q5. How satisfied are you with the final outcome of your complaint?
Responses to this question will be limited to four options (―very satisfied,‖ ―satisfied,‖
―dissatisfied,‖ and ―very dissatisfied‖). These will be translated into numerical values, where
complainants who are very satisfied will have the greatest value (2), and those who are very
dissatisfied will have the smallest value (-2).
The raw measure will be calculated as the mean value of the numerically translated responses
about the complainants’ satisfaction. Therefore, larger values will indicate better plan
performance in resolving complaints to complainants’ satisfaction. As with the other measures,
this measure will be tabulated against beneficiary and plan characteristics, and we will run riskadjustment models. In this way, no single item will define satisfaction with the final outcome.
Veracity of plan’s description of final outcome. This measure will assist in the review of the
accuracy of plans’ descriptions of their complaint resolution (outcome, decisions, etc) in the
CTM. Accuracy will be determined through comparison with the complainants’ descriptions of
the final outcome regardless of whether the beneficiary sees it as a resolution. This information
will be gathered in the survey through the following question:
Q4. What was the final outcome or decision regarding your complaint?
In the comparison of the plan (CTM records) and beneficiary responses to Q4, we will check to
ensure that any major action steps described by the plan—and which should be known by the
complainant—also occur in the complainant’s description of the final outcome. The accuracy of
this measure depends upon the criteria for ―major action‖ and ―should be known by the
complainant.‖ Considerable time will be spent developing the criteria for what qualifies as a
major action in a resolution and what percentage of the steps in a plan’s description must also be
found in the complainant’s description. The criterion ―should be known‖ will ensure that a plan
does not receive a lower score for internal activities.
A dichotomous variable will be created for each included complaint. This variable will equal 1
for those complaints in which there is a match between the plan’s resolution description in the
CTM and the complainant’s description. The value for this measure will be a function of the
mean of the dichotomous variable and other variables used to control for categories of

IMPAQ International, LLC

22

OMB Supporting Statement

complaints. Overall, the measure will represent the percentage of resolved sample complaints
for which the veracity of the plan’s resolution is supported by the complainant’s description. As
mentioned above, this measure also will be risk-adjusted with a full set of variables relevant to
the question at hand. Larger values will indicate that a plan is more reliable in describing its
resolution.
Beneficiary experiences during complaint resolution process. The performance measure will
assess whether beneficiaries encounter incidents while waiting for their complaints to be
resolved. One measure will be created to account for several of the critical beneficiary
experiences during the complaint process. Subject Matter Experts emphasized the issues
included in the survey as those most likely to have an impact on the beneficiary while their
complaint is being settled. These issues include: delay in receiving care or medications; health
complications; loss of health insurance coverage; and financial hardship. The information about
beneficiary experiences will be derived from the following survey question:
Q3. During the complaint process, did you experience any of the following?
[specific items follow]
The responses to this question will be used to provide more knowledge about the scale of
beneficiaries’ experiences. As with the other measures, this measure will be tabulated against
beneficiary and plan characteristics, and we will run risk-adjustment models. The implication is
that no single item associated with the risks, difficulties, and problems of a particular complaint
will define the measure.
With this indicator and others, the pilot test (100 surveys) may indicate the need to implement a
rating scale for the answer choices to Question 3. The purpose of doing so would be to gather
information on the severity of the beneficiary experiences. If this change was made, it would be
in order to increase the utility of this question and its indicator, the nature of the instrument and
the original intent of the question, survey and data collection would not be altered.
Areas for improvement in the complaint handling process. This information will not be used in
the development of a performance measure. However, information from this open-ended
question will allow us to make further recommendations concerning CTM guidelines and
identify which complaint categories are most likely to be flagged by beneficiaries or cause strong
dissatisfaction by beneficiaries. The open-ended responses will come from the following
question:
Q10. Do you have any suggestions or comments about how your plan could
handle complaints better?

IMPAQ International, LLC

23

OMB Supporting Statement

Analysis:
Several statistical analyses will be implemented to untangle the reasons behind a beneficiary’s
assessment of a plan sponsor’s resolution, or a beneficiary’s satisfaction with a plan sponsor’s
complaint resolution process. This information is important for two reasons: (1) to define a set of
monitoring and/or performance measures that can be used to validate the plan’s resolution of
beneficiary complaints closed by plans, from the perspective of the beneficiaries themselves; and
(2) to provide critical elements that should be monitored or included in CMS Standard Operating
Procedures, CMS guidance on handling of complaints, documentation standards, and other CMS
documents on the subject.
For the analysis of survey responses, several administrative datasets will be used. For example,
the Common Medicare Enrollment tables from the Medicare Enrollment Database (EDB) will be
used to identify (1) beneficiary enrollment at the contract level, (2) beneficiary state and county
codes, (3) election periods used for enrollment, (4) Part A and Part B entitlement, and (5) Part D
eligibility. The Medicare Beneficiary Database (MBD) will be used to identify contract
characteristics and contract service areas, while HPMS will be used to identify contract status
and other contract characteristics.
In the analysis of the survey and complementary administrative data, we will look at descriptive
statistics, testing, and the development of measures based on beneficiary survey responses. The
information provided by beneficiaries will be very valuable in establishing a correlation with a
plan’s own assessment of its complaints resolution process.
Our analysis will be both quantitative and qualitative. First, we will use the quantifiable items
from the survey to define and calculate measures of beneficiary satisfaction and exploratory
analysis of patterns and correlations between beneficiary satisfaction and beneficiary and/or Part
C/D sponsor characteristics. We will use information from HPMS CTM and other CMS data sets
to explore who the beneficiary respondents are and to develop models for assessing patterns for
certain populations (Low-Income-Status beneficiaries) or type of contracts (MA, MA-PD, PDP).
Our reason for the exploratory analysis is that certain responses to beneficiary satisfaction have
confounding factors that should be controlled in developing MAO and PDP sponsor ratings. For
example, it could be the case that certain populations have a higher likelihood of being
dissatisfied with Parts C and D sponsors; if a contract has a large proportion of this population,
the plan sponsor will have lower ratings than it should have if we were to control for the share of
the ―prone to be dissatisfied‖ population. Similar arguments can be made regarding the type of
plans.
In the initial analysis, we will prepare descriptions of the survey respondents (gender, age,
marital status, health status, Low-Income Subsidy Status) and the plan sponsors (Contract and
organization type, enrollment size, complaint type, complaint categories and proportion of
subpopulation enrollees) in the study sample. In addition to these exploratory tables, we will test

IMPAQ International, LLC

24

OMB Supporting Statement

nonresponse rates for those beneficiaries who did not respond to our survey to explore whether
they are different from the survey respondents.
In a similar fashion, we will present univariate descriptive statistics for the MAO and Part D
sponsors. It is important to emphasize that the survey responses are valid at the contract level.
There is no need to adjust the survey results. However, further exploration of the results by
contract characteristics would contribute to CMS understanding of the complaint resolution
process. For example, are contracts with a large proportion of LIS beneficiaries more likely to
have higher rates of disatisfaction, or are PFFS more likely to have more satisfied beneficiaries
given the nature of the organization type? In addition, we will compare the data collected at the
contract level to the overall universe of complaints.
Following the description of the beneficiaries and the contracts in the study sample, we will
proceed to present bivariate descriptive statistics. In an early phase of the analysis, we will assess
patterns across populations or Parts C/D sponsor characteristics to the beneficiary satisfaction
measures. This information will provide insight on population/contract correlations to
beneficiary satisfaction and some of the underlying factors influencing beneficiary satisfaction.
We will also conduct cross tabulations of beneficiary satisfaction to consequences or problems
associated with the complaint. Such tabulations will provide insights on how certain
problems/consequences affect beneficiaries’ perspective on satisfaction. Another table could be
developed for satisfaction with the time it took to get a resolution.
After the draft measures have been calculated, we will conduct additional analyses to further
refine the measure specifications based on the following criteria:
What are the underlying distributions of performance data, such as the mean, median,
standard deviation, and percentile scores?
How much dispersion is there across plans? Is the dispersion random or does it appears
to systematically affect certain types of plans?
Are there too many contracts that are subject to the data suppression rules or that have a
missing data issue?
We will also conduct a qualitative review of beneficiaries’ responses, which will focus on the
open-ended questions. Examples of such measures might be the following:
Whether the complaint resolution recorded in the CTM data conforms to the beneficiary’s
response regarding the final outcome
Whether the resolution provided by the plan conforms to the beneficiary’s original
request; What aspects of the plan’s handling of the complaint were unsatisfactory to the
beneficiary

IMPAQ International, LLC

25

OMB Supporting Statement

We will review a sample of beneficiaries’ responses in order to identify the key themes of their
perceived/articulated responses. Once these themes have been identified, the results of these
initial reviews will then be captured in keywords (with similar typologies identified as well),
which will be used in the next search of the qualitative data. If frequent occurrences of those
keywords or similar typologies are found, these issues/recommendations will be summarized as
key findings in the Beneficiary Experience Reports. For keywords with less frequency, our
summary results will highlight those for which some practical policy adjustments can be offered.
Through this qualitative review of beneficiary responses, we can report on aspects of the
beneficiary experience that may not have been captured elsewhere in the survey.
Last, we will conduct multivariate analysis that could be developed using data from the survey
and other CMS data sources. Logistic and multinomial analysis for several questions of
beneficiary satisfaction would provide information on the factors that influence beneficiary
responses on a particular aspect of beneficiary satisfaction. These results would further CMS’
knowledge about what is driving beneficiary satisfaction. Overall, these models allow predicting
rates of beneficiary satisfaction given certain beneficiary and plan characteristics. On the basis of
these results, we will risk-adjust beneficiary responses across contracts.
b. Publication Plans
The final report on the Part C and Part D Complaint Resolution Measures will be submitted to
CMS in draft form in July 2011 and in final form in August 2011. The report will describe the
data collection and analysis process and make recommendations for future improvements. The
report also will contain summary statistics of the sampled surveys and the performance
measures. The measure statistics will be stratified by various plan and complaint characteristics.
c. Time Schedule
The project began in September 2009 and will end in September 2011. The instruments were
prepared between October 2009 and December 2009. The data collection will start in January
2011. The sample intake period will end either in March 2011 or when we have reached our
sample goal, whichever is earlier. The analysis of the survey data and the construction of the
monitoring and/or performance measures will be complete in August 2011. If CMS chooses to,
CMS could publish the performance measures in November 2011.
17. Expiration Date
The expiration date will be displayed on the advance letter and on the hard copy version of the
questionnaire.
18. Certification Statement
There are no exceptions taken to item 19 of OMB Form 83-1.

IMPAQ International, LLC

26

OMB Supporting Statement