CMS-10549.GenIC#4 - Mini Supporting Statement

DEPARTMENT OF HEALTH & HUMAN SERVICES

Centers for Medicare & Medicaid Services

7500 Security Boulevard

Baltimore, Maryland 21244-1850

Application to Use Burden/Hours from Generic PRA Clearance:

Generic Clearance for Questionnaire Testing and Methodological Research for the Medicare Current Beneficiary Survey (MCBS) (CMS-10549)

Generic Information Collection (GenIC #4):

MCBS Testing of Revised Advance Letter

Office of Enterprise Data and Analytics (OEDA)

Centers for Medicare & Medicaid Services (CMS)

Under the CMS Generic Clearance (OMB No. 0938-1275, exp.05/31/2021) for the Medicare Current Beneficiary Survey (MCBS), CMS requests approval to test revisions to the text and format of the Community advance letter mailed to sampled beneficiaries prior to attempting the first interview.

Background Information

Testing of the revised version of the MCBS Community advance letter will be conducted under the split ballot field testing activity approved under the existing MCBS Generic Clearance. As noted in the Generic Clearance, this involves testing of alternate versions of material that have been revised to address problems identified from previous survey rounds. CMS seeks to test the impact of implementing recommended revisions to the advance letter through a split ballot experiment for cases in the 2019 Incoming Panel as part of routine data collection in Fall 2019. The purpose of the split ballot experiment is to analyze differences in response rates and other paradata between the 2019 Incoming Panel sample who receive the revised advance letter and those who receive the current version of the advance letter.

Justification

As a panel study, CMS selects a new sample of Medicare beneficiaries each year in the fall. Adding new sample to the survey each year offsets the loss of sample which consists of those beneficiaries who have completed their participation in the MCBS (after 11 interviews over a 4 year period) as well as due to natural attrition. The data collection period for the Incoming Panel is about five months – from the last week of July through December 31 of the calendar year. Sampled beneficiaries receive a letter and the MCBS Community brochure in the mail prior to the first visit from a field interviewer to ask for their participation in the survey.

NORC at the University of Chicago is the contractor for this survey. Dr. Don Dillman, an internationally-recognized subject matter expert on survey design and respondent-facing materials, recently provided NORC general guidance regarding best practices on advanced mail materials for in-person surveys. NORC and CMS then collaborated on ways to improve the advance letter. The resulting recommendations included simplifying letter text, highlighting the request for respondent participation, reducing extraneous information and agency references unfamiliar to the target audience, and formatting updates to modernize the letter. Our expectations are that the revisions will bring the letter, which has not been revised since 2014, in line with best practices in the field of survey research and respondent communication. CMS believes the revised letter text is clearer, more direct, and easier to understand than the current version, and may yield higher levels of respondent cooperation. As the text and appearance of the advance letter have not been updated in many years, this experiment affords the opportunity to do so while also testing the effects of the revisions.

Testing Plan

Experiment Design

The split ballot experiment will be implemented using the 2019 Incoming Panel. Consistent with the split ballot recruitment processes approved in the Generic Clearance, cases to be included in the experiment will be part of the routine sampling process used in production for the 2019 Incoming Panel. No separate screening or sampling will be needed.

To identify the cases in the experimental group, beneficiaries that are part of the 2019 Incoming Panel will be randomly assigned to the experimental group or to the control group. Beneficiaries in the experimental group will receive the revised letter while those in the control group will receive the current letter. Letters will be mailed to all sampled beneficiaries approximately two business days before the field period begins in late July 2019. All steps will be taken to ensure the letters are mailed at the same time following the standard study protocol. No other changes to the questionnaire or interviewing protocol are associated with this experiment. All interviewing procedures would remain unchanged.

Table 1 below shows the expected number of respondents by group, with the sample size estimated to be similar to the number of beneficiaries sampled for the 2018 Incoming Panel (approximately 11,000). Note that the projected response rates assume no difference between the two groups. With these sample sizes, detection of just under 3% difference between these two groups will be possible, providing sufficient power to measure the effectiveness of the revised letter.

Table 1: Projected Sample Sizes upon Case Release

Experimental Group	Letter Version	2019 Panel Sample (estimated)
Control	Current	5,500
Experimental	Revised	5,500

This split ballot experiment will be conducted with existing respondents as part of the regular MCBS production. As noted in the Generic Clearance, the burden for respondent time for this kind of split ballot experiments is captured in MCBS clearance 0938-0568. No additional respondent burden is expected.

Analysis of key outcomes

To assess the effects of the revised advance letter, we will compare key indicators between the experimental and control groups, including, but not limited to, interview completion rate, success at first in-person contact, elapsed time between letter mailing and first successful contact, elapsed time between letter mailing and interview completion, and number of total contacts. Factors such as respondent age and interview location will be considered and covariates will be controlled for in all analyses conducted. Results from this experiment will be used by CMS to inform and/or decide on revisions to the advance letter in future rounds of data collection. If CMS determines that the experimental version is preferred, a request to use that version will be submitted to OMB for use in the 2020 MCBS.

Please note that the experiment version of the advance letter has been reviewed and approved by NORC’s Institutional Review Board.

If you have any questions or would like to discuss this request, please do not hesitate to contact the CMS MCBS Project Director, Debra Reed-Gillette.

Attachments:

A – Revised Advance Letter

B – Current (Control) Advance Letter

Enclosure

OMB No. 0938-0568 | Expires 9/30/2021