CMS-R-246_REVISED_Supporting_Statement_Parts_B

SUPPORTING STATEMENT FOR MEDICARE CAHPS SURVEYS
SUPPORTING STATEMENT – Part B
B.1

Respondent universe and sample

CMS is requiring all MA, MA-PD and Stand Alone PDP plans that have at least 600 eligible enrollees July
of the previous year to participate in an independent third party vendor administration of this survey
(hereinafter referred to as Medicare CAHPS). The Medicare CAHPS survey is also conducted among a
sample of persons enrolled in the Medicare FFS plan for purposes of allowing comparisons of measures
obtained from all surveys. For the national Medicare CAHPS survey, the names and addresses of
sampled beneficiaries shall be obtained from the Integrated Data Repository (IDR) on or shortly after
January each year. Persons with Medicare 18 years old or older who have been continuously enrolled
for 6 months or longer in the same Medicare contract and who are not institutionalized are included in
the sampling frame. A random sample of between 600 and 900 eligible beneficiaries per reporting unit
is selected depending on the size of the contract. Sample sizes are designed to produce estimates with
a reliability of 0.8. Medicare health and prescription drug plans are surveyed at the contract
organization level and this level will also define the sampling and reporting unit. For Medicare FFS
enrollees the sampling and reporting unit is defined at the state or sub-state level for large states.
Most sampling units will have about 800 members. A small number of contracts with between 600 and
800 enrollees will have samples comprised of virtually all of their enrollees. If there are less than 600
eligible beneficiaries in an organization at the contract, the survey will not be conducted for that
contract.
The survey will be conducted through use of a randomized sample of Medicare enrollees as described
above from sampling and reporting units in all 50 states, the District of Columbia, the US Virgin Islands,
and Puerto Rico. Some states will be divided into smaller units if they have large numbers of enrollees.
Because of changing enrollment patterns and the need to employ the most recent information
available, sampling experts from RAND and Harvard will prepare the final sample design based on the
current CMS enrollment databases available each year just prior to sample draw.
Demographic and geographic information on non-respondents is obtained from the sample frame at
the time the sample is drawn and used in developing weights for preparing survey results that reflect
the full Medicare population. Weighting is done on a stratified basis at the contract and geographic
area level to further assure that the measures prepared from the survey results reflect the Medicare
population. Case-mix adjustment methods are also employed for comparing performance between
contracts.
B.2

Information collection procedures

The administration of the survey consists of vendors (or CMS in the case of FFS Medicare enrollees)
mailing a pre-notification letter signed by the CMS Privacy Officer prior to the mailing of the first
questionnaire a week to ten days later; a second questionnaire is mailed to non-respondents
approximately three weeks after the initial survey mailing. Telephone follow-up of non-respondents to

8

SUPPORTING STATEMENT FOR MEDICARE CAHPS SURVEYS
the mail portion of the survey is conducted beginning about two weeks after the mailing of the second
questionnaire. A minimum of 6 call-back attempts are required to reach the sample member.
B.3

Methods to maximize response rates

The CAHPS survey has developed a mixed-mode data collection protocol, as described above, that uses
a pre-notification letter alerting sample members that a survey will be mailed to them shortly, a first
mailing of the full questionnaire booklet, followed by a second mailing to those who do not respond to
the earlier mailing of the questionnaire. For those who also do not respond to the second mailing of
the questionnaire, CAHPS employs a telephone follow-up through which it offers sample members the
opportunity to complete the survey by phone. The mailing materials to all sample members also
include a toll-free telephone number that allows recipients to call in to ask questions about the survey.
Overall this system has resulted in response rates of between 64-83 percent on average over the first
twelve years of national data collection in Medicare Advantage and Medicare FFS CAHPS, varying
somewhat by plan type, contract, and region of the country.
Efforts are employed to maximize response rates including testing of the survey questions prior to
their inclusion in the questionnaires to ensure that beneficiaries comprehend the questions and can
answer with minimal effort. Second, the survey is conducted in both English and Spanish language to
meet the needs of most of our sampled beneficiaries. Also the method of administration – a prenotification letter, two mailings of the questionnaire, and telephone follow-up of non-respondents – is
a multi-pronged, comprehensive strategy that avoids the weaknesses of reliance upon mail or
telephone administration alone.
B.4

Tests of procedures or methods

The Medicare CAHPS survey has been tested within the Medicare population using a variety of
methods similar to those used in development of commercial CAHPS and other large health care
surveys. The core CAHPS questions were developed by the CAHPS consortium led by the Agency for
Healthcare Research and Quality and modified for use by the CMS. Testing of both the core questions
and supplemental questions added by CMS included a multi-state field testing of the full set of CAHPS
questionnaires among Medicare health and prescription drug plan enrollees, and tests of the timing for
the two mailings of the survey, as well as training of survey interviewers for the telephone follow-up
data collection. As noted above modifications have been made following several implementations of
the annual survey based on lessons learned from prior year collections. Each modification in turn is
tested among persons enrolled in Medicare prior to its use on the survey form or its effect on data
collection. See below also for additional detail regarding statistical design modifications.

B.5

Statistical and questionnaire design consultants

We receive ongoing input from statisticians in developing, designing, conducting, and analyzing the
information collected from this survey. This statistical expertise will continue to be available from
RAND and Harvard Medical School.
9

SUPPORTING STATEMENT FOR MEDICARE CAHPS SURVEYS

Analysis of the Medicare CAHPS survey will be conducted using methodologies and programs
developed by the Agency for Healthcare Research and Quality and the CAHPS Consortium and used by
other CAHPS surveyors including the National Committee for Quality Assurance over the last dozen
years. These analytic programs are documented in the CAHPS Health Plan Survey and Reporting Kit
and include a set of SAS files which comprise the CAHPS Analysis Program known as the CAHPS macro.
The macro allows users to analyze and statistically adjust the survey data in order to make valid
comparisons of performance across plan types.
The programs prepare several measures of plan experiences in two broad categories – global ratings of
the care and services received and reports of specific experiences using the plan. The CAHPS macro is
updated occasionally to address new survey questions and issues and has been updated to include
data collected in the MA-PD and PDP CAHPS, such as data on enrollee experiences with and ratings of
their Medicare prescription drug plans, both MA-PDs and Stand Alone PDPs.
The CAHPS data analysis programs use multivariate analysis to control for differences in plan
enrollments according to specific enrollee characteristics that have been empirically found to affect
enrollees’ perceptions of their care and plan experiences, but for which the plan has no control, such
as age, education, health status, and whether or not a spouse or family member assisted the enrollee
in completing the survey questionnaire. This set of analysis has been documented in a series of CaseMix Adjustment Reports that present reasons why specific enrollee characteristics are used in the
adjustment process and why other factors are not. For example, prior analyses of many CAHPS survey
data files show that age and health status affect enrollees’ perceptions of their plan and care
experiences in systematic ways. By adjusting for these effects, the CAHPS measures produced from
the CAHPS macro present measures that control for differences in the proportions of enrollees in each
plan having these characteristics.
Ongoing statistical consultation is provided by:
Marc N. Elliot, Ph.D.
RAND
1776 Main Street
Santa Monica, CA 90401-3208
Tel: 310-393-0411

Alan Zaslavsky, Ph.D.
Associate Professor of Statistics
Harvard University, Department of
Health Care Policy, Harvard
Medical School

10