CMS-R-246_MCAHPS Supporting Statement B

SUPPORTING STATEMENT – Part B

Medicare Advantage, Medicare Part D, and Medicare Fee-For-Service Consumer Assessment of

Healthcare Providers and Systems (CAHPS) Survey

CMS-R-246, OMB 0938-0732

B.1 Respondent universe and sample

CMS is requiring all MA-Only, MA-PD, and Stand-Alone PDP contracts that have at least 600 eligible enrollees in July of the previous year to participate in an independent third-party vendor administration of this survey (hereinafter referred to as Medicare CAHPS). The Medicare CAHPS survey is also conducted among a sample of persons enrolled in Medicare FFS for purposes of allowing comparisons of measures obtained from the surveys. For the national Medicare CAHPS survey, the names and addresses of sampled beneficiaries shall be obtained from the Integrated Data Repository (IDR) on or shortly after January each year. Persons with Medicare 18 years old or older who have been continuously enrolled for 6 months or longer in the same Medicare contract and who are not institutionalized are included in the sampling frame. A random sample of between 600 and 800 eligible beneficiaries per reporting unit is selected depending on the size of the contract. Sample sizes are designed to produce estimates with a reliability of 0.8. Medicare health and prescription drug plans are surveyed at the contract organization level, and this level will also define the sampling and reporting unit. For Medicare FFS enrollees the sampling and reporting unit is defined at the state or sub-state level for large states. Most sampling units will have about 800 members. A small number of contracts with between 600 and 800 enrollees will have samples comprised of virtually all of their enrollees. If there are less than 600 eligible beneficiaries in an organization at the contract, the survey will not be required for that contract.

The survey is conducted through use of a randomized sample of Medicare enrollees as described above from sampling and reporting units in all 50 states, the District of Columbia, and Puerto Rico. For the FFS survey, some states will be divided into smaller units if they have large numbers of enrollees. Because of changing enrollment patterns and the need to employ the most recent information available, sampling experts from RAND and Harvard will prepare the final sample design based on the current CMS enrollment databases available each year just prior to sample draw.

Demographic and geographic information on non-respondents is obtained from the sample frame at the time the sample is drawn and used in developing weights for preparing survey results that reflect the full Medicare population. Weighting is done on a stratified basis at the contract and geographic area level to further assure that the measures prepared from the survey results reflect the Medicare population. Case-mix adjustment methods are also employed for comparing performance between contracts and between MA and FFS.

B.2 Information collection procedures

Beginning with survey administration in 2024, CMS added a web mode to the data collection protocol. The web mode was tested under CMS-10793, OMB 0938-1432 and was found to increase MA response rates by approximately four percentage points. The administration of the survey consists of vendors (or CMS in the case of FFS Medicare enrollees) mailing a prenotification letter signed by the CMS Medicare Drug Benefit and C & D Data Group Director prior to sending an invitation to complete the survey four days later. Sampled enrollees with an available email address will receive an email invite to complete the survey online and those without an available email address will receive a letter with a personalized URL. The questionnaire will be mailed to those who do not complete the survey via web ten days later; a second questionnaire is mailed to non-respondents approximately three weeks after the initial survey mailing. Telephone follow-up of non-respondents to the mail portion of the survey is conducted beginning about three weeks after the mailing of the second questionnaire. Five call-back attempts are required to reach the sample member.

B.3 Methods to maximize response rates

The CAHPS survey has developed a mixed-mode data collection protocol, as described above, that uses a pre-notification letter alerting sample members that a survey will be sent to them shortly. As noted above, CMS tested data collection by web and found that it increased MA response rates. An email invitation to complete the survey via web will be sent to sampled enrollees with an available email address. Those sampled enrollees without an available email address will receive a letter with a personalized URL to complete the survey online. A first mailing of the full questionnaire booklet will be sent to non-respondents, followed by a second mailing to those who do not respond to the earlier mailing of the questionnaire. For those who also do not respond to the second mailing of the questionnaire, CAHPS employs a telephone follow-up through which it offers sample members the opportunity to complete the survey by phone. The mailing materials to all sample members also include a toll-free telephone number and an email address that allows recipients to reach out to ask questions about the survey. Overall this system has resulted in response rates of between 28-65 percent on average over the last fifteen years of national data collection in MA, PDP, and FFS CAHPS, varying somewhat by plan type, contract, and region of the country. CMS anticipates that as the availability of email addresses increases, the web mode could contribute to higher response rates.

Table 4. Historical Response Rates

Year	MA Response Rate	PDP Response Rate	FFS Response Rate	Total Response Rate
2023	35.8%	38.3%	30.2%	34.8%
2022	35.2%	38.8%	27.9%	33.9%
2021	36.4%	37.9%	28.6%	34.3%
2020	N/A	N/A	N/A	N/A
2019	38.4%	38.9%	32.0%	36.4%
2018	41.0%	41.3%	34.8%	39.0%
2017	42.3%	44.4%	35.1%	40.3%
2016	42.2%	40.2%	35.9%	39.7%
2015	41.4%	39.5%	34.8%	39.0%
2014	44.9%	40.1%	35.5%	41.2%
2013	46.1%	42.8%	42.2%	44.6%
2012	47.9%	44.2%	43.4%	45.9%
2011*	46.5%	40.0%	49.8%	46.9%
2010	61.7%	57.1%	57.3%	59.8%
2009	64.8%	57.7%	58.3%	61.8%
2008	64.9%	54.9%	57.5%	60.7%
2007	50.70%	47.80%	47.80%	48.90%

*In 2011, MA and PDP contracts were surveyed by multiple vendors for the first time.

**The MA & PDP CAHPS Survey was not administered in 2020 due to COVID-19.

Efforts are employed to maximize response rates including testing of the survey questions prior to their inclusion in the questionnaires to ensure that beneficiaries comprehend the questions and can answer with minimal effort. Second, the survey is available in English, Spanish, Chinese, Korean, Vietnamese, and Tagalog to meet the needs of most of our sampled beneficiaries. Also, the method of administration – a pre-notification letter, an email with reminder (beginning in 2024) or letter survey invitation, up to two mailings of the questionnaire for non-respondents, and telephone follow-up of non-respondents – is a multi-pronged, comprehensive strategy that avoids the weaknesses of reliance upon mail or telephone administration alone. We have tested a mixed mode implementation of the survey that incorporates the web mode of data collection and found that response rates increased by approximately four percentage points. We will implement the web mixed mode beginning in 2024.

B.4 Tests of procedures or methods

The Medicare CAHPS survey has been tested within the Medicare population using a variety of methods similar to those used in development of commercial CAHPS and other large health care surveys. The core CAHPS questions were developed by the CAHPS consortium led by the Agency for Healthcare Research and Quality (AHRQ) and modified for use by CMS. Testing of both the core questions and supplemental questions added by CMS included a multi-state field testing of the full set of CAHPS questionnaires among Medicare health and prescription drug plan enrollees, as well as tests of the survey administration protocols. Modifications have been made following several implementations of the annual survey based on lessons learned from prior year collections, and to reflect the wording used in AHRQ’s 5.1 Health Plan Survey. Under CMS-10793, OMB 0938-1432 we tested a web mode data collection protocol, AHRQ’s 5.1 Health Plan Survey wording clarifications for explicit references to care received via telehealth (phone or video), and new questions about language spoken at home and unfair treatment. See below also for additional detail regarding statistical design modifications.

B.5 Statistical and questionnaire design consultants

We receive ongoing input from statisticians in developing, designing, conducting, and analyzing the information collected from this survey. This statistical expertise will continue to be available from RAND and Harvard Medical School.

Analysis of the Medicare CAHPS survey will be conducted using methodologies and programs developed by AHRQ and the CAHPS Consortium and used by other CAHPS surveyors including NCQA over the last several decades. These analytic programs are documented in the CAHPS Health Plan Survey and Reporting Kit and the MA and PDP CAHPS website; the documentation includes a set of SAS files that comprise the CAHPS Analysis Program known as the CAHPS macro available at https://www.ahrq.gov/cahps/surveys-guidance/helpfulresources/analysis/index.html. The macro allows users to analyze and statistically adjust the survey data in order to make valid comparisons of performance across contract types.

The programs prepare several measures of patient experiences in two broad categories – global ratings of the care and services received and reports of specific experiences using the plan. The CAHPS macro is updated occasionally to address new survey questions and issues and has been updated to include data collected in the MA-PD and PDP CAHPS, such as data on enrollee experiences with and ratings of their Medicare prescription drug plans, both MA-PDs and PDPs.

The CAHPS data analysis programs use multivariate analysis to control for differences in plan enrollments according to specific enrollee characteristics that have been empirically found to affect enrollees’ perceptions of their care and plan experiences, but for which the plan has no control, such as age, education, health status, and whether or not a spouse or family member assisted the enrollee in completing the survey questionnaire. For example, prior analyses across multiple CAHPS surveys show that age and health status affect enrollees’ perceptions of their plan and care experiences in systematic ways. By adjusting for these effects, the CAHPS measures produced from the CAHPS macro present measures that control for differences in the proportions of enrollees in each plan having these characteristics.

Ongoing statistical consultation is provided by:

Marc N. Elliott, Ph.D.

RAND

1776 Main Street

Santa Monica, CA 90401-3208

Alan Zaslavsky, Ph.D.

Professor of Statistics

Department of Health Care Policy

Harvard Medical School