CMS-R-246 - Supporting Statement B

SUPPORTING STATEMENT – Part B

Medicare Advantage, Medicare Part D, and Medicare Fee-For-Service Consumer Assessment of

Healthcare Providers and Systems (CAHPS) Survey

CMS-R-246, OMB 0938-0732

B.1 Respondent universe and sample

CMS is requiring all MA-Only, MA-PD, and Stand Alone PDP contracts that have at least 600 eligible enrollees July of the previous year to participate in an independent third party vendor administration of this survey (hereinafter referred to as Medicare CAHPS). The Medicare CAHPS survey is also conducted among a sample of persons enrolled in Medicare FFS for purposes of allowing comparisons of measures obtained from the surveys. For the national Medicare CAHPS survey, the names and addresses of sampled beneficiaries shall be obtained from the Integrated Data Repository (IDR) on or shortly after January each year. Persons with Medicare 18 years old or older who have been continuously enrolled for 6 months or longer in the same Medicare contract and who are not institutionalized are included in the sampling frame. A random sample of between 600 and 800 eligible beneficiaries per reporting unit is selected depending on the size of the contract. Sample sizes are designed to produce estimates with a reliability of 0.8. Medicare health and prescription drug plans are surveyed at the contract organization level, and this level will also define the sampling and reporting unit. For Medicare FFS enrollees the sampling and reporting unit is defined at the state or sub-state level for large states. Most sampling units will have about 800 members. A small number of contracts with between 600 and 800 enrollees will have samples comprised of virtually all of their enrollees. If there are less than 600 eligible beneficiaries in an organization at the contract, the survey will not be required for that contract.

The survey is conducted through use of a randomized sample of Medicare enrollees as described above from sampling and reporting units in all 50 states, the District of Columbia, and Puerto Rico. Some states will be divided into smaller units if they have large numbers of enrollees. Because of changing enrollment patterns and the need to employ the most recent information available, sampling experts from RAND and Harvard will prepare the final sample design based on the current CMS enrollment databases available each year just prior to sample draw.

Demographic and geographic information on non-respondents is obtained from the sample frame at the time the sample is drawn and used in developing weights for preparing survey results that reflect the full Medicare population. Weighting is done on a stratified basis at the contract and geographic area level to further assure that the measures prepared from the survey results reflect the Medicare population. Case-mix adjustment methods are also employed for comparing performance between contracts.

B.2 Information collection procedures

The administration of the survey consists of vendors (or CMS in the case of FFS Medicare enrollees) mailing a pre-notification letter signed by the CMS Medicare Drug Benefit and C & D Data Group Director prior to the mailing of the first questionnaire a week to ten days later; a

second questionnaire is mailed to non-respondents approximately three weeks after the initial survey mailing. Telephone follow-up of non-respondents to the mail portion of the survey is conducted beginning about three weeks after the mailing of the second questionnaire. Five call-back attempts are required to reach the sample member.

B.3 Methods to maximize response rates

The CAHPS survey has developed a mixed-mode data collection protocol, as described above, that uses a pre-notification letter alerting sample members that a survey will be mailed to them shortly, a first mailing of the full questionnaire booklet, followed by a second mailing to those who do not respond to the earlier mailing of the questionnaire. For those who also do not respond to the second mailing of the questionnaire, CAHPS employs a telephone follow-up through which it offers sample members the opportunity to complete the survey by phone. The mailing materials to all sample members also include a toll-free telephone number that allows recipients to call in to ask questions about the survey. Overall this system has resulted in response rates of between 32-65 percent on average over the last nine years of national data collection in MA, PDP, and FFS CAHPS, varying somewhat by plan type, contract, and region of the country.

Table 4. Historical Response Rates

Year	MA Response Rate	PDP Response Rate	FFS Response Rate	Total Response Rate
2019	38.4%	38.9%	32.0%	36.4%
2018	41.0%	41.3%	34.8%	39.0%
2017	42.3%	44.4%	35.1%	40.3%
2016	42.2%	40.2%	35.9%	39.7%
2015	41.4%	39.5%	34.8%	39.0%
2014	44.9%	40.1%	35.5%	41.2%
2013	46.1%	42.8%	42.2%	44.6%
2012	47.9%	44.2%	43.4%	45.9%
2011*	46.5%	40.0%	49.8%	46.9%
2010	61.7%	57.1%	57.3%	59.8%
2009	64.8%	57.7%	58.3%	61.8%
2008	64.9%	54.9%	57.5%	60.7%
2007	50.70%	47.80%	47.80%	48.90%

*In 2011, MA and PDP contracts were surveyed by multiple vendors for the first time.

**The requirement to collect and submit 2020 data was eliminated due to COVID-19 in CMS1744-IFC, so burden was reduced although hard to tell extent since data collection had already begun.

Efforts are employed to maximize response rates including testing of the survey questions prior to their inclusion in the questionnaires to ensure that beneficiaries comprehend the questions and can answer with minimal effort. Second, the survey is conducted in English, Spanish, Chinese, Korean, Vietnamese, and Tagalog to meet the needs of most of our sampled beneficiaries. Also the method of administration – a pre-notification letter, two mailings of the questionnaire, and telephone follow-up of non-respondents – is a multi-pronged, comprehensive strategy that avoids the weaknesses of reliance upon mail or telephone administration alone. We are also exploring the feasibility of a mixed mode implementation of the survey that incorporates the web mode of data collection. This would be submitted under a separate package for testing.

B.4 Tests of procedures or methods

The Medicare CAHPS survey has been tested within the Medicare population using a variety of methods similar to those used in development of commercial CAHPS and other large health care surveys. The core CAHPS questions were developed by the CAHPS consortium led by the Agency for Healthcare Research and Quality (AHRQ) and modified for use by CMS. Testing of both the core questions and supplemental questions added by CMS included a multi-state field testing of the full set of CAHPS questionnaires among Medicare health and prescription drug plan enrollees, tests of the timing for the two mailings of the survey, as well as training of survey interviewers for the telephone follow-up data collection. Modifications have been made following several implementations of the annual survey based on lessons learned from prior year collections, and to reflect the wording used in AHRQ’s 5.0 Health Plan Survey. Each modification in turn is tested among persons enrolled in Medicare prior to its use on the survey form or its effect on data collection. See below also for additional detail regarding statistical design modifications.

B.5 Statistical and questionnaire design consultants

We receive ongoing input from statisticians in developing, designing, conducting, and analyzing the information collected from this survey. This statistical expertise will continue to be available from RAND and Harvard Medical School.

Analysis of the Medicare CAHPS survey will be conducted using methodologies and programs developed by AHRQ and the CAHPS Consortium and used by other CAHPS surveyors including NCQA over the last 15 years. These analytic programs are documented in the CAHPS Health

Plan Survey and Reporting Kit and the MA and PDP CAHPS website; the documentation includes a set of SAS files that comprise the CAHPS Analysis Program known as the CAHPS macro. The macro allows users to analyze and statistically adjust the survey data in order to make valid comparisons of performance across contract types.

The programs prepare several measures of patient experiences in two broad categories – global ratings of the care and services received and reports of specific experiences using the plan. The CAHPS macro is updated occasionally to address new survey questions and issues and has been updated to include data collected in the MA-PD and PDP CAHPS, such as data on enrollee experiences with and ratings of their Medicare prescription drug plans, both MA-PDs and PDPs.

The CAHPS data analysis programs use multivariate analysis to control for differences in plan enrollments according to specific enrollee characteristics that have been empirically found to affect enrollees’ perceptions of their care and plan experiences, but for which the plan has no control, such as age, education, health status, and whether or not a spouse or family member assisted the enrollee in completing the survey questionnaire. This analysis has been documented in a series of Case-Mix Adjustment Reports that present reasons why specific enrollee characteristics are used in the adjustment process and why other factors are not. For example, prior analyses of many CAHPS survey data files show that age and health status affect enrollees’ perceptions of their plan and care experiences in systematic ways. By adjusting for these effects, the CAHPS measures produced from the CAHPS macro present measures that control for differences in the proportions of enrollees in each plan having these characteristics.

Ongoing statistical consultation is provided by:

Marc N. Elliot, Ph.D.

RAND

1776 Main Street

Santa Monica, CA 90401-3208

Tel: 310-393-0411

Alan Zaslavsky, Ph.D.

Professor of Statistics

Department of Health Care Policy

Harvard Medical School