CMS-R-246 - Supporting Statement B [rev 5-8-2015 by OSORA PRA]

Supporting Statement Part B

Medicare Advantage, Medicare Part D, and Medicare Fee-For-Service Consumer Assessment of Healthcare Providers and Systems (CAHPS) Survey

CMS-R-246, OMB 0938-0732

B.1 Respondent universe and sample

CMS is requiring all MA-only, MA-PD, and Stand Alone PDP contracts that have at least 600 eligible enrollees July of the previous year to participate in an independent third party vendor administration of this survey (hereinafter referred to as Medicare CAHPS). The Medicare CAHPS survey is also conducted among a sample of persons enrolled in the Medicare FFS plan for purposes of allowing comparisons of measures obtained from all surveys. For the national Medicare CAHPS survey, the names and addresses of sampled beneficiaries shall be obtained from the Integrated Data Repository (IDR) on or shortly after January each year. Persons with Medicare 18 years old or older who have been continuously enrolled for 6 months or longer in the same Medicare contract and who are not institutionalized are included in the sampling frame. A random sample of between 600 and 800 eligible beneficiaries per reporting unit is selected depending on the size of the contract. Sample sizes are designed to produce estimates with a reliability of 0.8.  Medicare health and prescription drug plans are surveyed at the contract organization level and this level will also define the sampling and reporting unit. For Medicare FFS enrollees the sampling and reporting unit is defined at the state or sub-state level for large states. Most sampling units will have about 800 members. A small number of contracts with between 600 and 800 enrollees will have samples comprised of virtually all of their enrollees. If there are less than 600 eligible beneficiaries in an organization at the contract, the survey will not be conducted for that contract.

The survey will be conducted through use of a randomized sample of Medicare enrollees as described above from sampling and reporting units in all 50 states, the District of Columbia, the US Virgin Islands, and Puerto Rico. Some states will be divided into smaller units if they have large numbers of enrollees. Because of changing enrollment patterns and the need to employ the most recent information available, sampling experts from RAND and Harvard will prepare the final sample design based on the current CMS enrollment databases available each year just prior to sample draw.

Demographic and geographic information on non-respondents is obtained from the sample frame at the time the sample is drawn and used in developing weights for preparing survey results that reflect the full Medicare population. Weighting is done on a stratified basis at the contract and geographic area level to further assure that the measures prepared from the survey results reflect the Medicare population. Case-mix adjustment methods are also employed for comparing performance between contracts.

B.2 Information collection procedures

The administration of the survey consists of vendors (or CMS in the case of FFS Medicare enrollees) mailing a pre-notification letter signed by the CMS Privacy Officer prior to the mailing of the first questionnaire a week to ten days later; a second questionnaire is mailed to non-respondents approximately three weeks after the initial survey mailing. Telephone follow-up of non-respondents to the mail portion of the survey is conducted beginning about two weeks after the mailing of the second questionnaire. Five call-back attempts are required to reach the sample member.

B.3 Methods to maximize response rates

The CAHPS survey has developed a mixed-mode data collection protocol, as described above, that uses a pre-notification letter alerting sample members that a survey will be mailed to them shortly, a first mailing of the full questionnaire booklet, followed by a second mailing to those who do not respond to the earlier mailing of the questionnaire. For those who also do not respond to the second mailing of the questionnaire, CAHPS employs a telephone follow-up through which it offers sample members the opportunity to complete the survey by phone. The mailing materials to all sample members also include a toll-free telephone number that allows recipients to call in to ask questions about the survey. Overall this system has resulted in response rates of between 36-65 percent on average over the last eight years of national data collection in MA, PDP, and FFS CAHPS, varying somewhat by plan type, contract, and region of the country.

	Year	MA	PDP	FFS	Total
Response Rate	2014	44.9%	40.1%	35.5%	41.2%
	2013	46.1%	42.8%	42.2%	44.6%
	2012	47.9%	44.2%	43.4%	45.9%
	2011*	46.5%	40.0%	49.8%	46.9%
	2010	61.7%	57.1%	57.3%	59.8%
	2009	64.8%	57.7%	58.3%	61.8%
	2008	64.9%	54.9%	57.5%	60.7%
	2007	50.7%	47.8%	47.8%	48.9%

*In 2011, MA and PDP contracts were surveyed by multiple vendors for the first time

Efforts are employed to maximize response rates including testing of the survey questions prior to their inclusion in the questionnaires to ensure that beneficiaries comprehend the questions and can answer with minimal effort. Second, the survey is conducted in English, Spanish, and Chinese to meet the needs of most of our sampled beneficiaries. Also the method of administration – a pre-notification letter, two mailings of the questionnaire, and telephone follow-up of non-respondents – is a multi-pronged, comprehensive strategy that avoids the weaknesses of reliance upon mail or telephone administration alone.

B.4 Tests of procedures or methods

The Medicare CAHPS survey has been tested within the Medicare population using a variety of methods similar to those used in development of commercial CAHPS and other large health care surveys. The core CAHPS questions were developed by the CAHPS consortium led by the Agency for Healthcare Research and Quality and modified for use by CMS. Testing of both the core questions and supplemental questions added by CMS included a multi-state field testing of the full set of CAHPS questionnaires among Medicare health and prescription drug plan enrollees, tests of the timing for the two mailings of the survey, as well as training of survey interviewers for the telephone follow-up data collection. Modifications have been made following several implementations of the annual survey based on lessons learned from prior year collections. Each modification in turn is tested among persons enrolled in Medicare prior to its use on the survey form or its effect on data collection. See below also for additional detail regarding statistical design modifications.

B.5 Statistical and questionnaire design consultants

We receive ongoing input from statisticians in developing, designing, conducting, and analyzing the information collected from this survey. This statistical expertise will continue to be available from RAND and Harvard Medical School.

Analysis of the Medicare CAHPS survey will be conducted using methodologies and programs developed by the Agency for Healthcare Research and Quality and the CAHPS Consortium and used by other CAHPS surveyors including the National Committee for Quality Assurance over the last 15 years. These analytic programs are documented in the CAHPS Health Plan Survey and Reporting Kit and include a set of SAS files which comprise the CAHPS Analysis Program known as the CAHPS macro. The macro allows users to analyze and statistically adjust the survey data in order to make valid comparisons of performance across plan types.

The programs prepare several measures of plan experiences in two broad categories – global ratings of the care and services received and reports of specific experiences using the plan. The CAHPS macro is updated occasionally to address new survey questions and issues and has been updated to include data collected in the MA-PD and PDP CAHPS, such as data on enrollee experiences with and ratings of their Medicare prescription drug plans, both MA-PDs and PDPs.

The CAHPS data analysis programs use multivariate analysis to control for differences in plan enrollments according to specific enrollee characteristics that have been empirically found to affect enrollees’ perceptions of their care and plan experiences, but for which the plan has no control, such as age, education, health status, and whether or not a spouse or family member assisted the enrollee in completing the survey questionnaire. This analysis has been documented in a series of Case-Mix Adjustment Reports that present reasons why specific enrollee characteristics are used in the adjustment process and why other factors are not. For example, prior analyses of many CAHPS survey data files show that age and health status affect enrollees’ perceptions of their plan and care experiences in systematic ways. By adjusting for these effects, the CAHPS measures produced from the CAHPS macro present measures that control for differences in the proportions of enrollees in each plan having these characteristics.

Ongoing statistical consultation is provided by:

Marc N. Elliot, Ph.D. RAND 1776 Main Street Santa Monica, CA 90401-3208 Tel: 310-393-0411

Alan Zaslavsky, Ph.D. Professor of Statistics Harvard University, Department of Health Care Policy, Harvard Medical School