Mini Supporting Statement

Application to Use Burden/Hours from Generic PRA Clearance:

Generic Clearance for Questionnaire Testing and Methodological Research for the Medicare Current Beneficiary Survey (MCBS) (CMS-10549)

Generic Information Collection:

Testing Revisions to Advance Mailing Materials for the Medicare Current Beneficiary Survey

Office of Enterprise Data and Analytics (OEDA)

Centers for Medicare & Medicaid Services (CMS)

CMS requests approval to test revisions to the materials that are mailed to sampled beneficiaries prior to attempting the first interview. This request was initially approved by OMB on 05/02/2020 under CMS-10549 GenIC#5 (OMB No. 0938-1275) and implementation was planned for the MCBS Fall 2020 Round 88 data collection period. However, due to the coronavirus pandemic, MCBS data collection for that round was conducted by phone only. These advanced mail materials are designed for in-person data collection and therefore, the schedule for the testing of new mail materials was delayed. CMS is planning to return to in-person data collection for Fall 2021 Round 91 and therefore, is resubmitting this request in full as part of this revised Generic Clearance package.

The text and format of the currently-used brochure will be revised; we will also test the introduction of a new Frequently Asked Questions document. The materials are mailed in July of each year; the effectiveness of the new materials will be measured during the July – December 2021 data collection period.

Background Information

As a panel study, CMS selects a new sample of Medicare beneficiaries each year in the fall. Adding new sample to the survey each year offsets the loss of sample, which consists of those beneficiaries who have completed their participation in the MCBS (after 11 interviews over a 4-year period) as well as due to natural attrition. The data collection period for the Incoming Panel is about five months – from the last week of July through December 31 of the calendar year. Sampled beneficiaries receive a letter and the MCBS Community brochure in the mail prior to the first visit from a field interviewer to ask for their participation in the survey. This advance letter mailing represents the beneficiary’s introduction to the survey and is the first contact with incoming sample members. Over the past several years, CMS has worked to improve and modernize the advance materials with the goal of increasing participation in the first interview.

On December 3, 2018, CMS-10549.GenIC #4: MCBS Testing of Revised Advance Letter was approved by OMB. A new letter was tested from July 2019 through December 2019. Results indicate that the test letter performed slightly better than the original letter. In Attachment 2 of this Generic Clearance package, CMS has provided to OMB a summary of this experiment. This request focuses on the other materials in the mailing – namely, whether revising the content and design of the brochure makes a difference in cooperation of selected sample and also whether a new FAQ inserted in the mailing materials performs better than the brochure.

Justification

In 2021, CMS will test a revised version of the MCBS Community Brochure and a new Frequently Asked Questions document under the field testing in production activity approved as part of the MCBS Generic Clearance. CMS seeks to test the impact of implementing revisions to the advanced mail materials through a split ballot experiment for cases in the 2021 Incoming Panel as part of routine data collection in Fall 2021. The purpose of the experiment is to analyze differences in completion rates and other paradata between the 2021 Incoming Panel sample who receive several variations of the new material.

As a panel study, CMS selects a new sample of Medicare beneficiaries each year for the Fall Round¹. Adding new sample to the survey each year offsets the loss of sample, which consists of those beneficiaries who have completed their participation in the MCBS (after 11 interviews over a 4-year period) as well as due to natural attrition. The data collection period for the Incoming Panel is about five months – from the last week of July through December 31 of the calendar year. Sampled beneficiaries receive an advance mailing containing a letter and brochure in the mail prior to the first visit from a field interviewer to ask for their participation in the survey.

In 2018, NORC at the University of Chicago, the contractor for this survey worked with Dr. Don Dillman, an internationally-recognized subject matter expert on survey design and respondent-facing materials, to develop a set of best practices for advance mail materials. Dillman’s recommended changes addressed enclosures such as the use of brochures or Frequently Asked Questions documents. Although brochures are considered a standard practice in advance mailings for survey research, Dillman makes the case that there is not a compelling body of literature to support this practice, and in fact, in some instances, a glossy brochure may distract from the purpose of the letter. Thus, he recommends testing the use of a Frequently Asked Questions document as a potential replacement for the brochure in order to better directly address beneficiaries’ questions and concerns. He also noted that if the brochure would continue to be used, it should be redesigned to have a simpler and more direct message, look, and feel.

This points to a potential improvement to the MCBS advance mailing, which represents an important opportunity to establish legitimacy and lay the foundation for respondent cooperation. Thus, CMS proposes to test the enclosure of a new Frequently Asked Questions document as well as a redesigned brochure, to help determine the value of the brochure in contrast to a new type of document. The revisions to the current brochure are intended to highlight common respondent concerns and establish legitimacy while also modernizing the look and feel of the design. The current brochure was last updated in 2014 and can be improved by using clear, direct text and simple messaging. In order to test both the revised brochure and the new Frequently Asked Questions document, we propose a three-pronged experiment design to test two different conditions in comparison to the current brochure.

Testing Plan

Experiment Design

The experiment will be implemented using 12,000 cases selected for the 2021 MCBS Incoming Panel. Cases to be included in the experiment will be part of the routine sampling process used in production for the 2021 Incoming Panel.

Beneficiaries that are part of the 2021 Incoming Panel will be randomly assigned to one of three groups: two experimental groups and a control group. All beneficiaries will receive the identical versions of the envelope and advance letter. Beneficiaries in Experimental Group 1 will receive the advance letter with the new Frequently Asked Questions document. Those in Experimental Group 2 will receive the advance letter, new Frequently Asked Questions document, and a revised brochure. Experimental Group 3, the control group, will receive the advance letter and the current brochure currently in use. Sample identified as likely Spanish-speaking will be included in the experiment and receive all materials in both English and Spanish.

The advanced mail materials will be mailed to sampled beneficiaries the week of July 12, 2021; data collection for the Fall 2021 Round 91 is scheduled to begin July 19, 2021. All steps will be taken to ensure standard protocols are followed for each experimental group, so as not to subject any group to differential treatment. No other changes to the questionnaire or interviewing protocol are associated with this experiment. All interviewing procedures will remain unchanged.

Table 1 below shows the expected number of respondents by group (n=12,000).² With these sample sizes, detection of just under 3% difference in response rates between these three groups will be possible, providing sufficient power to measure the effectiveness of the updated materials.

Table 1: Projected Sample Sizes upon Case Release

Experimental Group	Materials Mailed	Sample Size (estimated)
Experimental Group 1	New Frequently Asked Questions Sheet	4,000
Experimental Group 2	New Frequently Asked Questions Sheet and Revised Brochure	4,000
Control	Current Brochure	4,000

This split ballot experiment will be conducted with the existing 2021 incoming baseline sample as part of the regular MCBS Fall data collection effort. As noted in the Generic Clearance, the burden for respondent time for this kind of experiment is captured in MCBS clearance 0938-0568, which covers MCBS data collection. No additional respondent burden is expected.

Analysis of Key Outcomes

To assess the effects of the revised brochure and introduction of the Frequently Asked Questions document, we will compare key indicators between the experimental and control groups, including, but not limited to, interview completion rate, success at first in-person contact, elapsed time between advanced material mailing and first successful contact, elapsed time between advanced material mailing and interview completion, and number of total contacts. Factors such as respondent age and interview location will be considered and covariates will be controlled for in all analyses conducted. Results from this experiment will be used by CMS to inform and/or decide on revisions to the advance mailing in future rounds of data collection. If CMS determines that one of the experimental versions is preferred, a request to use that version will be submitted to OMB for use in 2022.

Please note that the revised brochure and the new Frequently Asked Questions document have been reviewed and approved by NORC’s Institutional Review Board.

If you have any questions or would like to discuss this request, please do not hesitate to contact the CMS MCBS Project Director, Debra Reed-Gillette.

1 MCBS data collection occurs in three rounds annually: Fall (July – December), Winter (January – April), and Summer (May – August).

2 The final number of MCBS beneficiaries selected may be slightly lower; the exact number of the sample is not determined until the Spring of 2021.