MBECS Supporting Statement Part B_

SUPPORTING STATEMENT

Part B

Medicare Beneficiary Experiences with Care Survey (MBECS) System

March 10, 2021

Centers for Medicare and Medicaid Services (CMS),

Office of Minority Health

Contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent universe and sampling methods 3

2. Information Collection Procedures 5

4. Tests of Procedures 8

5. Statistical Consultants 8

B. Collections of Information Employing Statistical Methods

The Medicare Beneficiary Experiences with Care Survey (MBECS) data collection system will allow the Centers for Medicare and Medicaid Services’ Office of Minority Health (CMS OMH) to better understand and measure differences in the health care experiences of minority Medicare beneficiaries compared with those of the general Medicare population.

1. Respondent universe and sampling methods

For each of the MBECS data collection efforts, the universe of respondents will be identified leveraging one or more datasets as described below, creating a sampling frame. The underlying data will come from the following and may be used by themselves or potentially combined, as needed. This respondent data will come from:

1. Existing flags in the CMS Virtual Research Data Center (also known as the Chronic Conditions Warehouse, or CCW);

2. International Classification of Diseases - Tenth Edition (ICD-10) codes from CMS’ Medicare Fee-For-Service (FFS) “final action” claims from institutional and non-institutional providers;

3. Algorithms that leverage these data sources to identify beneficiaries who are likely in the population of interest.

Table 1 shows each minority population of interest and how each universe will be identified. The primary groups of interest are listed in the far left column; for each group, any sub-groups of interest are also indicated, along with the definition and identification method associated with each. The far right column provides the estimated size of each population as indicated by the relevant CCW flags, ICD-10 code, or algorithm in current datasets. Estimated population sizes are based on the most recent data available and may fluctuate as new data becomes available.

Table 1. MBECS Data Collection System Minority Populations of Interest*

Population of Interest	Sub-groups (if any)	Definition	Estimated Population Size
Severe Mental Illness (SMI)		Beneficiaries 18 years and over who have severe mental illness and/or intellectual disability. Excludes beneficiaries with end stage renal disease (ESRD).	381,553
	Hispanic	SMI beneficiaries who are Hispanic	28,338
	Non-Hispanic Black or African American	SMI beneficiaries whose RTI Race Code = Black or African American	57,425
	Non-Hispanic Other, including White	SMI beneficiaries whose RTI Race Code are not Hispanic or NH Black	295,790
Substance Use Disorder (SUD)		Beneficiaries 18 years and over who have SUD. Excludes beneficiaries with ESRD.	1,908,617
	Hispanic	SUD beneficiaries who are Hispanic	125,634
	Non-Hispanic Black or African American	SUD beneficiaries whose RTI Race Code = Black or African American	280,693
	Non-Hispanic Other, including White	SUD beneficiaries whose RTI Race Code are not Hispanic or NH Black	1,502,290

*Estimates presented in Table 1 were generated using the 2018 Master Beneficiary Summary File – Base with Medicare Part A/B/C/D, 2018 Master Beneficiary Summary File Other Chronic or Potentially Disabling Conditions Segment.

CMS OMH’s priority minority populations vary in size. Differential sampling will impact the design effective and overall effective sample size. The targeted sample size try to balance the need for a sufficient number of completes to create subgroup estimates, and compare to other subgroups of interest, while also minimizing the design effect that can be influenced by oversampling these subgroups. As shown below in Table 2, a target size of 400-700 completed surveys for Hispanics and Non-Hispanic Blacks allows for these estimates and comparisons, while minimizing the design effect as best possible.

Table 2 shows the anticipated design effect and margin of error associated with the target completes for each subgroup of interest. The expected margin of errors ranges from 1.8-2.9 for larger groups of interest (e.g., SMI as a whole, Non-Hispanic Other including white), and from 5.2-7.2 for the smaller minority populations.

Table 2. Estimated Measures of Error for MBECS Population Domains

MBECS Population Domains	MBECS Target Completes	Estimated Design Effect (DEFF)	Margin of Error (percentage points)
Serious Mental Illness (SMI)	3,000	2.0	2.4
Hispanic	400	2.2	7.2
Non-Hispanic Black or African American	600	2.0	5.6
Non-Hispanic Other, including White	2,000	1.8	2.9
Substance Use Disorders (SUD)	5,000	2.0	1.8
Hispanic	400	2.2	7.2
Non-Hispanic Black or African American	700	2.0	5.2
Non-Hispanic Other, including White	3,900	1.8	2.0

*A design effect of 2.0 is assumed for the Total across Domains a-f and a design effect of 1.2 is assumed for each sub-group sample. A correlation of .10 between the population domain estimate and each benchmark estimate is assumed.

The sample will be selected using a simple random sample stratified by each of the six strata (primary group crossed by race/ethnicity). The sample will be drawn to reflect current estimated response rates determined at the time of sample selection,

2. Information Collection Procedures

MBECS Core Questionnaire

The MBECS questionnaire uses items from three surveys relevant to the Medicare population: the Fee for Service (FFS) Consumer Assessment of Healthcare Providers and Systems (CAHPS®), Medicare Advantage (MA) CAHPS, and Nationwide Adult Medicaid (NAM) CAHPS surveys. Questions cover the following areas using a 6-month look-back period:

• Health care utilization (e.g. health care needed, getting appointments, prescription medicines)

• Experience with care (e.g. global rating of health care, wait times)

• Care from personal health care provider and specialists (e.g. number of visits, global ratings, care coordination)

• Experience with health plan (e.g. customer service, health plan rating)

MBECS Population-Specific Questionnaire Modules

For each surveyed population, the MBECS questionnaire will include a brief module that contains population-specific questions. These questions will be included in individual information collection requests as they have not yet been identified. Population-specific questions will be taken from already approved Federal surveys (e.g., the National Health Interview Survey, the Medicare Current Beneficiary Survey, the Current Population Survey, etc.).

Respondent Recruitment and Data Collection

The MBECS data collection system will be based on enhanced CAHPS methodology using sequential multi-mode (web, mail, and telephone) approach. Respondent materials (web survey invitation letters, Self-Administered Questionnaire cover letters, etc.) will be population-specific and included in individual information collection requests. CMS OMH anticipates that the data collection schedule for all minority populations of interest will be as follows:

• Week 0: Respondents will first be sent a web invitation letter that explains the purpose of the survey and provides the web survey URL and personal access code (PAC) to securely access the survey. Any respondents whose web invitation letter is returned as undeliverable will undergo enhanced locating techniques (described below in “Methods to Maximize Response Rates”).

• Week 3: Three weeks after the web invitation mailing, non-responders to the web survey will be mailed the first self-administered questionnaire (SAQ) with a cover letter. (See Attachment A for the SAQ version of the core questionnaire; population-specific module questions and SAQ cover letters will be submitted as part of individual information collection requests.)

• Week 7: After four weeks, non-responders to both the web survey and first questionnaire mailing will be sent a second SAQ package with cover letter.

• Weeks 11-17: Four weeks after the second SAQ mailing, non-responders to the web survey and mailed questionnaires will be contacted via telephone to complete the questionnaire with a trained telephone interviewer.

Throughout the data collection period, respondents will be able to call a toll-free line to ask questions or complete the survey via telephone. The web survey will remain open for the duration of the data collection period.

All MBECS surveys will be available in Spanish language.

Prior to any individual survey under MBECS, CMS OMH will evaluate whether incentives may be necessary to improve response rates and reduce nonresponse for the target population.

Cleaning and Imputation

After each MBECS data collection period ends, the data collection contractor will conduct general data cleaning, including ensuring 1) appropriate skip patterns were applied; 2) logical consistency throughout the survey; and 3) responses are within the expected ranges. During data cleaning, imputation methods will be implemented where necessary to account for item nonresponse. The imputation methods will be based on hot-deck methods, where cases with similar characteristics (donors) are used to fill in cases with missing data.

Weighting

For appropriate estimation and analysis, the contractor will create weights to accurately account for differential response, aggregation across minority populations, and survey nonresponse.

The weighting process will consist of three broad steps: calculation of the baseweight, a weight to account for nonresponse, and a final raked survey weight. The baseweight will simply be the inverse of the sampling fraction for the group of interest. The nonresponse adjustment will be done within homogeneous groups or adjustment cells, to account for differential nonresponse (e.g., age groups, race/ethnicity, etc.). The determination of adjustment cells will be based on logistic regression analysis to see what demographic variables may be associated with the propensity to respond. After the nonresponse weighting adjustment, the data will be adjusted (“raked”) to known population totals. These population totals will be based on known accessible tabulations that will likely be made from the most recent Master Beneficiary Summary File and any other data files that are used to help define the sampling frame.

Data Analysis

Data analysis will include comparison of MBECS survey estimates to benchmark estimates from the FFS, MA, and NAM CAHPS surveys, and to other source surveys in the case of population-specific module questions. The variables of interest will be dependent on the subgroups of interest, the variables that are available for both the targeted MBECS population and benchmark data, and the currency of information.

3. Methods to Maximize Response Rates

The MBECS data collection system will use a sequential multi-mode method of survey administration—a web invitation letter, two mailings of the paper questionnaire (SAQ), and telephone follow-up with non-responders. This sequential multi-mode approach avoids the weaknesses associated with mail or telephone administration alone and increases opportunities for respondents to complete the survey. Research has shown that including telephone follow-up with non-responders to a mail survey increases the overall response rate¹ and increases the socio-demographic representativeness of the sample.² Including an option for respondents to complete the survey via the web minimizes respondent burden by allowing respondents to complete the survey at a convenient time, allowing respondents to stop and re-enter the survey if needed, and handling skip patterns based on responses to each question. In addition, the MBECS system will build on recommendations³ to include a web option for CAHPS survey administration by using sequential—rather than concurrent—multi-mode administration (as is currently the case with CAHPS survey administration that includes an internet enhancement option).

For each individual survey under MBECS, CMS OMH will evaluate whether incentives may be necessary to improve response rates and reduce nonresponse for the target population. Finally, some respondents may be difficult to reach via mail alone or reluctant to participate; thus non-responders to the mail portion of the survey will be contacted via telephone to complete the survey.

During data collection efforts, weekly review of demographic distributions will be conducted to determine if any targeted outreach needs to be conducted. For example, if the response rate for Non-Hispanic Blacks significantly lags behind other groups during the same period, additional calls to that subgroup would be conducted to increase representativeness and decrease the potential for nonresponse bias.

In order to maximize the likelihood of reaching respondents by mail or telephone, MBECS surveys will use the LexisNexis® Accurint® service for all address/telephone records to ensure the most up-to-date information is used for contacting respondents. This method has been shown to be more effective in getting up-to-date address and telephone records than similar methods, such as Telematch and Directory Assistance (M. Elliot, personal communication, May 22, 2017).

Following the completion of each MBECS survey, CMS OMH’s contractor will conduct a nonresponse bias analysis to assess how well the final respondents to the survey represent the target population of Medicare beneficiaries. The contractor will use available comparison characteristics from the sample frame to assess to what extent the respondents differ from non-responders, and whether this needs to be taken into account when weighting the dataset. Additional demographic comparisons (e.g., rural vs. urban, age group comparison, race/ethnicity comparisons) can be made to benchmark estimates for the population of interest, where possible.

4. Tests of Procedures

CAHPS surveys are used extensively by CMS to support efforts to improve health care in the U.S. As part of their development and on-going maintenance, all CAHPS surveys undergo cognitive testing, field testing, and psychometric analyses.⁴

Regarding population-specific questionnaire modules, information about testing and development of source surveys will be provided with individual information collection requests.

5. Statistical Consultants

Data will be collected by NORC at the University of Chicago. Statistical consulting on weighting will be provided by Elizabeth Allen, Senior Statistician at NORC at the University of Chicago.

List of Attachments:

Attachment A – MBECS Core Questionnaire

Attachment B – Federal Register Notice #1

1 Beebe, T. J., Davern, M. E., McAlpine, D. D., Call, K. T., & Rockwood, T. H. (2005). Increasing Response Rates in a Survey of Medicaid Enrollees: The Effect of a Prepaid Monetary incentive and Mixed Modes (Mail and Telephone). Medical Care, 43(4), 411-414.

2 Zaslavsky, A., Zaborski, L., & Cleary, P. (2002). Factors Affecting Response Rates to the Consumer Assessment of Health Plans Study Survey. Medical Care, 40(6), 485-499. Retrieved from http://www.jstor.org/stable/3768129

3 IMPAQ International, LLC. National Implementation of the Original Medicare Fee-For-Service CAHPS Survey Data Collection and Data File Preparation: Annual Report for the 2016 Survey. Columbia, MD. Prepared for the Centers for Medicare and Medicaid Services, 2016.

4 Hargraves JL, Hays RD, Cleary PD. Psychometric properties of the Consumer Assessment of Health Plans Study (CAHPS) 2.0 Adult Core Survey. Health Serv Res. 2003 Dec;38(6 Pt 1):1509-27.