CMS-10793 Supporting Statement B

SUPPORTING STATEMENT – Part B

Medicare Advantage and Prescription Drug Plan Consumer Assessment of Healthcare Providers and Systems (CAHPS) Survey Field Test

(CMS-10793, OMB 0938-XXXX)

B.1 Respondent universe and sample

The Medicare Advantage (MA) & Prescription Drug Plan (PDP) CAHPS Survey is administered annually to MA and PDP enrollees with six months of continuous enrollment in their contract. The survey is administered to adult enrollees in the U.S. who do not reside in an institution, and the standard sample size is 800 for MA contracts and 1,500 for PDP contracts. The web field test sample will be drawn from the remaining, eligible sample frame used to conduct the 2022 MA & PDP CAHPS Surveys. We propose to draw sample from 50 MA and PDP contracts for each of two study arms (100 enrollees per contract, 50 contracts, 5,000 enrollees total). The 50 contracts will be selected at random.

B.2 Information collection procedures

Sampled enrollees will be randomly assigned into one of two study arms, an experimental arm that will include the web mode, and a comparison arm that will consist of the typical MA & PDP CAHPS survey administration procedures.

In the experimental arm, all sampled enrollees will receive a mailed pre-notification letter in advance of survey administration, the letter will be personalized to the enrollee and will include the URL for the web version of the survey and a PIN code that is unique to the enrollee. The enrollee may enter the URL and PIN code to access the web version of the survey, which can be completed on either a computer or a mobile device such as a smartphone or tablet. Four days after the pre-notification letter is mailed, enrollees will be sent an invitation to the web survey. The invitation will be sent by email to enrollees with email addresses, and via a letter to those for whom an email address is not available¹. A reminder invitation email will be sent five days after the initial invitation to enrollees with email addresses. Five days later, the secondary mode (mail) will be initiated with all enrollees who have not completed the web survey. Enrollees who don’t respond to the first mail survey, will receive a second mail survey approximately 3 weeks later. Approximately three weeks after a second mail survey is sent, phone administration of the survey will be attempted with all non-respondent enrollees.

In the comparison arm, all sampled enrollees will first receive a mailed pre-notification letter prior to the mailing of the first questionnaire four days later. A second questionnaire will be mailed to non-respondents approximately four weeks after the initial survey mailing. Telephone follow-up of non-respondents to the mail portion of the survey will be conducted beginning about three weeks after the mailing of the second questionnaire.

B.3 Methods to maximize response rates

This data collection effort will use a mixed-mode data collection protocol (as described above) that uses a pre-notification letter alerting sample members that a survey will be mailed to them shortly, a first mailing of the full questionnaire booklet, followed by a second mailing to those who do not respond to the earlier mailing of the questionnaire. For those who also do not respond to the second mailing of the questionnaire, CAHPS employs a telephone follow-up through which it offers sample members the opportunity to complete the survey by phone. The mailing materials to all sample members also include a toll-free telephone number that allows recipients to call in to ask questions about the survey. Overall this system has resulted in response rates of between 36-65 percent on average over the last nine years of national data collection in MA and PDP CAHPS, varying somewhat by plan type, contract, and region of the country.

Additionally, this field test will test the addition of a web mode to the survey administration procedures, as described above. Recent research demonstrates the potential to improve response rates with the use of a web-based mode. In particular, in the emergency department setting, a CMS-funded test demonstrated that a higher survey response rate could be achieved by using a web survey as the initial mode of administration, followed by mail, phone, or both.² Based on this research, CMS believes adding web to the existing mixed mode protocol for MA & PDP CAHPS has the potential to improve response rates, in particular among younger enrollees. In 2021, the response rate for MA and PDP enrollees under age 65 was 24% compared to 39% for enrollees age 65 and older.

Table 1. Historical Response Rates

Year	MA Response Rate	PDP Response Rate
2021	36.4%	37.9%
2020*	N/A	N/A
2019	38.4%	38.9%
2018	41.0%	41.3%
2017	42.3%	44.4%
2016	42.2%	40.2%
2015	41.4%	39.5%
2014	44.9%	40.1%
2013	46.1%	42.8%
2012	47.9%	44.2%
2011**	46.5%	40.0%
2010	61.7%	57.1%
2009	64.8%	57.7%
2008	64.9%	54.9%
2007	50.7%	47.8%

*The MA & PDP CAHPS Survey was not administered in 2020

**In 2011, MA and PDP contracts were surveyed by multiple vendors for the first time

B.4 Tests of procedures or methods

The proposed field test will test a new web-based mode of survey administration, along with proposed new survey content recommended by stakeholders. We propose to administer this field test with unused sample from the same Medicare population that is eligible to receive the annual survey.

See below also for additional detail regarding statistical design modifications.

B.5 Statistical and questionnaire design consultants

The statistical expertise for this field test effort was provided by the RAND Corporation.

The proposed design is of sufficient scale to support comparison of the effect of mode of administration on responses and response rate. We will have 80% power with a 2-sided test at alpha=0.05 to detect a beneficiary-level effect size of 0.12 standard deviations (0.1 is “very small” and 0.2 is “small” by standard heuristics). We will have 80% power to detect a 3.8% difference in response rates between the experimental and comparison arms using a 2-sided test and alpha=.05. The analysis will employ fixed effects for contracts to control for contract differences in performance, an approach used previously to assess changes in MA & PDP CAHPS survey content³. Analyses will be restricted to data from the 50 contracts sampled for the field test and 2022 administration of MA & PDP CAHPS. If overall differences are detected using the initial model, additional models using random-effect contract intercepts and slopes will test whether changes are uniform or vary by contract.

Differences in mean responses between the experimental and comparison arms will inform CMS decision-making about the inclusion of new survey items.

We propose to strongly recommend the use of the web-first mode if the response rate for the experimental arm is significantly higher than the comparison arm, to recommend further study if it is not significantly different from the comparison arm, and to recommend against it if the experimental arm has a response rate significantly lower than the comparison arm.

Statistical consultation is provided by:

Marc N. Elliott, Ph.D. RAND Corporation

1776 Main Street

Santa Monica, CA 90401-3208

Tel: 310-393-0411

1 Sending an invitation by letter to those without an email address is consistent with other CMS CAHPS efforts (such as OAS CAHPS).

2 Parast L., M. Mathews, et al. (2019). Effects of Push-to-Web Mixed Mode Approaches on Survey Response Rates: Evidence from a Randomized Experiment in Emergency Departments. Survey Practice, 12(2). https://doi.org/10.29115/SP-2019-0008.

3 Beckett, M. K., Elliott, M. N., Burkhart, Q., Cleary, P. D., Orr, N., Brown, J. A., Gaillot, S., Liu, K., & Hays, R. D. (2019). The effects of survey version on patient experience scores and plan rankings. Health services research, 54(5), 1016–1022. https://doi.org/10.1111/1475-6773.13172.