Supporting Statement B

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Respondent universe and sample

CMS is requiring all Medicare MA, MA-PD and free-standing PDP plans that have had a contract effective for at least one year (defined in this start-up year as effective on or before Jan 1, 2006 to participate in an independent third party administration of this survey (hereinafter referred to as Medicare CAHPS). The Medicare CAHPS survey will also be conducted among a sample of persons enrolled in the Original Medicare plan for purposes of allowing comparisons of measures obtained from all surveys. For the national Medicare CAHPS survey, the names and addresses of sampled beneficiaries shall be obtained from the Medicare Beneficiary Database (MBD) files on or shortly after January each year. Beneficiaries who have been continuously enrolled for 6 months or longer and who are not institutionalized are included in the sampling frame. A random sample of 600 eligible beneficiaries per reporting unit is selected. For health and prescription drug plans, reporting units are defined as the contract organization. For Original Medicare enrollees the reporting unit is defined at the state or territorial level. If there are less than 600 eligible beneficiaries in an organization, all of the beneficiaries are selected

The survey will be conducted through use of a randomized sample of Medicare enrollees in all 50 states, the District of Columbia, the US Virgin Islands, and Puerto Rico. Because of changing enrollment patterns and the need to employ the most recent information available, sampling experts from RAND, Harvard, and Westat will prepare the final sample design based on the current CMS enrollment databases available each year just prior to sample draw. Current plans are for 600 enrollees to be drawn from each of the MA, MA-PD, and Stand-alone PDPs, as well as sufficient numbers of additional enrollees in Original Medicare to produce state-level estimates. The sampling plan will be finalized annually by January. A data collection plan has also been developed and tested to assure sufficient survey response to provide for statistically significant CAHPS measurements in all Medicare health and prescription drug plans and in all states.

The response universe for this survey has grown considerably. The MMA legislation has increased the size and scope of the Medicare CAHPS surveys. For the current survey, the number of plans to be included in the survey has grown from 208 in the 2005 MA survey, to now include 509 MA-PD plans and 81 freestanding PDPs. For plans that cover large geographic areas or have national coverage (ie Private Fee For Service), we are splitting the organizations into multiple sampling units. With the current growth in the number of plans we estimate there will be up to 1,100 sampling units annually.

Demographic and geographic information on non-respondents is obtained from the sample frame at the time the sample is drawn and used in developing weights for preparing survey results that reflect the full Medicare population. Weighting is done on a stratified basis at the plan and geographic area level to further assure that the measures prepared from the survey result reflect the Medicare population. As noted above, case-mix adjusted methods are employed for comparing performance between plans, but non-adjusted measures are also available for use in quality improvement efforts at the plan and Medicare program levels.

B.2 Information collection procedures

The administration of the survey consists of a pre-notification letter signed by the CMS Privacy Officer sent out prior to the first questionnaire mailing, the first questionnaire mailing, a postcard reminder, and a second mailing. We conduct telephone follow-up of non-respondents.

B.3 Methods to maximize response rates

For the first round of Medicare CAHPS, we achieved a 74 percent response rate. From the first round of the survey, we learned that it would be helpful to lengthen the data collection period to get the most out of the first two mailings and to increase the period of time for telephone follow-up. For the fifth and sixth rounds of the survey, we achieved an 82 percent and 83 percent response rate, respectively.

The CAHPS survey has developed a data collection protocol that uses a pre-notification letter alerting sample members that a survey will be mailed to them shortly, a first mailing of the full questionnaire booklet, followed by a reminder postcard and a second mailing to those who do not respond to the earlier questionnaire. For those who also do not respond to the second mailing of the questionnaire, CAHPS employs a telephone follow-up through which it offers sample members the opportunity to complete the survey by phone. This system has resulted in response rates of between 70-80% on average over the past nine years of national data collection in Medicare CAHPS.

Additionally, a variety of efforts have been made to maximize our response rate. First, extensive testing of the individualized questions and their order within the survey, ensures that beneficiaries easily comprehend the questions and can answer with minimum effort. Second, the method of administration chosen, pre-notification letter, two mailouts and a reminder postcard, and telephone followup of non-respondents – is a multi-pronged, comprehensive strategy that avoids the weaknesses of reliance upon mail or telephone contact alone.

B.4 Tests of procedures or methods

Not applicable. No tests of new procedures or methods are performed.

B.5 Statistical and questionnaire design consultants

We are receiving ongoing input from statisticians to develop, design, conduct, and analyze the information collected from this survey. This statistical expertise will be available from RAND and Harvard Medical School.

Analysis of the Medicare CAHPS survey will be conducted using methodologies and programs developed by the Agency for Healthcare Research and Quality and the CAHPS Consortium and used by other CAHPS surveyors including the National Committee for Quality Assurance. These analytic programs are documented in the CAHPS Health Plan Survey and Reporting Kit and include a set of SAS files which comprise the CAHPS Analysis Program known as the CAHPS macro. The macro allows users to analyze and statistically adjust the survey data in order to make valid comparisons of performance across health plans.

The programs prepare several measures of health plan experiences in two broad categories, 1. enrollee ratings of their plan and health care services received and 2. reports of specific experiences regarding getting needed care, getting care quickly, patient-doctor communication, helpfulness of physician office staff, and getting information from a plan’s customer service. The CAHPS macro is updated occasionally to address new survey questions and issues and is being updated to address data collected in the 2006 Medicare CAHPS, including data on enrollee experiences in Medicare prescription drug plans (PDPs), both Medicare Advantage PDs and Stand Alone PDPs.

The CAHPS data analysis programs use multivariate analysis to control for differences in plan enrollments according to specific enrollee characteristics that have been empirically found to affect enrollees’ perceptions of their care and plan experiences, but for which the plan has no control, such as age, education, health status, and whether or not a spouse or family member assisted the enrollee in completing the survey questionnaire. This set of analysis has been documented in a series of Case-Mix Adjustment reports that present reasons why specific enrollee characteristics are used in the adjustment process and why other factors are not. For example, prior analyses of many CAHPS survey data files show that age and health status affect enrollees’ perceptions of their plan and care experiences in systematic ways. By adjusting for these effects, the CAHPS measures produced from the CAHPS macro present measures that control for differences in the proportions of enrollees in each plan having these characteristics.

The results of these analyses are final measures that might have been obtained from plans that had the same proportions of enrollees according to each of the case-mix adjustment factors.