Report: Impact of Nonresponse on Medicare Current Beneficiary Survey Estimates

Impact of Nonresponse on Medicare Current Beneficiary
Survey Estimates
John Kautter, Ph.D., Galina Khatutsky, M.S., Gregory C. Pope, M.S., James R. Chromy, Ph.D., and
Gerald S. Adler, M.Phil.

The Medicare Current Beneficiary Survey
(MCBS) has been used by policymakers and
research analysts to provide information on
a wide array of topics about the Medicare
Program. Nonresponse bias is potentially one
of the most important threats to the validity
of the estimates from the MCBS. In this article we present results of our methodological
study that analyzes the impact of nonresponse
on MCBS estimates, including initial round
unit nonresponse, panel attrition, and item
nonresponse. Our findings indicate that for
most of the measures studied, the bias caused
by differences between nonrespondents and
respondents in the MCBS was substantially
reduced or eliminated by the nonresponse
procedures currently employed.
INTRODUCTION
The MCBS is a continuous, multipurpose survey of a representative national
sample of the Medicare population, conducted by CMS. The central goals of the
MCBS are to determine expenditures
and sources of payment for all services
used by Medicare beneficiaries, including
copayments, deductibles, and non-covered
services; to ascertain all types of health
insurance coverage and relate coverage to
sources of payment; and to trace processes
over time, such as changes in health status, spending down to Medicaid eligibility,
John Kautter, Galina Khatutsky, Gregory C. Pope, and James
R. Chromy are with RTI International. Gerald S. Adler is with
the Centers for Medicare and Medicaid Services (CMS). The
research in this article was supported by CMS under Contract
Number 500-01-0027. The statements expressed in this article
are those of the authors and do not necessarily reflect the views
or policies of RTI International, or CMS.

and the impacts of program changes. The
MCBS is the most important survey of
Medicare beneficiaries, and has been used
by policymakers and research analysts to
provide information on a wide array of topics about the Medicare Program (Kautter
and Pope, 2004).
The MCBS operates as a rotating panel
survey. New panels are selected from the
population of beneficiaries eligible for
Medicare as of January 1 of the year
of induction.1 Initial interviews for each
new supplement are conducted in the fall
interview round. MCBS beneficiaries are
interviewed three times per year, and each
interview round is administered over a 4month period. Beneficiaries remain in the
sample for 4 years; each fall approximately
one-fourth of the sample is replaced.2
Because of its key role in informing
Medicare policymakers, obtaining accurate estimates from the MCBS is of critical importance. Like virtually all surveys,
the MCBS is subject to several forms of
nonresponse. These include unit nonresponse, in which the sampled beneficiary
is not interviewed, and item nonresponse,
in which the interviewed beneficiary does
not answer a certain question. In addition,
in longitudinal surveys like the MCBS,
there is the potential for beneficiaries to
answer one or more rounds of the survey
and stop participating (panel attrition).
Consequences of nonresponse include:
The sampling frame actually consists of a special 5-percent
sample of such persons maintained by CMS. In recent years
the sampling frame had to be expanded beyond 5 percent of the
Medicare Enrollment File due to the small number of eligibles
in some ZIP Codes.
2 For more on the MCBS, visit: http://www.cms.hhs.gov/mcbs.
1

Health Care Financing Review/Summer 2006/Volume 27, Number 4

71

(1) biases in point estimators, (2) inflation
of the variances of point estimators, and
(3) biases in customary estimators of precision (Dillman et al., 2002). Nonresponse
bias is potentially one of the most important threats to the validity of the estimates
from surveys like the MCBS.3
In this article we present results of a
methodological study initiated by CMS
that analyzed the impact of nonresponse
on MCBS estimates (Kautter et al., 2003).
We present results of our analyses of initial
round unit nonresponse, panel attrition,
and item nonreponse. After providing a
definition of nonresponse bias and its components, we provide a brief overview of the
MCBS survey design. Then we explain our
methodology for analyzing initial round
unit nonresponse in the MCBS and our
analytic findings for initial round unit nonresponse, and similarly, for panel attrition
and item nonresponse. Finally, we offer
conclusions.
DEFINITION OF NONRESPONSE
BIAS AND ITS COMPONENTS
The concern about survey nonresponse
is that nonrespondents will differ from
respondents with regard to the survey variables, in which case the survey estimates
based on the respondents alone will be
biased estimates of the overall population
parameters (Kalton, 1983). If there is no
difference between respondents and nonrespondents regarding survey variables,
then there is no potential for bias due to
nonresponse.
To illustrate this point, suppose the aim
of a survey is to estimate a population
mean. In the case of a survey that fails
to collect data for the nonrespondents,
the sample statistic used to estimate the
population mean is the respondent sample
Westat, Inc., administers the MCBS for CMS and employs
multiple procedures to minimize nonresponse.
3

72

mean. The bias arising from using the
respondent sample mean as an estimator
for the population mean is:
b = mR – m
where b is the bias, mR is the population
mean for respondents, and m is the overall
population mean. This expression shows
that the nonresponse bias is a function of
the difference in the population mean for
respondents and the overall population
mean.
The nonresponse bias can also be
expressed in terms of the response rate
and the difference in the respondent and
nonrespondent means:
b = (1 – RR)(mR – mNR)
where b is the bias, RR is the response
rate in the population, and mR and mNR are
the population means for respondents and
nonrespondents (Kalton, 1983). This last
expression shows that the nonresponse
bias is a function of two quantities: (1) the
response rate in the population; and (2)
the difference in the population means for
respondents and nonrespondents. Note
that if there is a 100-percent response
rate (RR = 1), or if there is no difference
between the respondent and nonrespondent means (mR – mNR = 0), then the nonresponse bias is zero. For a given difference in the respondent and nonrespondent
means, the nonresponse bias falls as the
response rate increases, and for a given
response rate, the nonresponse bias falls
as the difference in the respondent and
nonrespondent means falls.
MCBS SURVEY DESIGN
A basic understanding of the MCBS
survey design is important for understanding our methodology for analyzing nonresponse. We provide an overview of the
MCBS sample design and survey weighting procedures.

Health Care Financing Review/Summer 2006/Volume 27, Number 4

Sample Design
The sample design for the MCBS is a
stratified area probability design with three
stages of selection. The induction sample
is based on a multistage sample using clusters of counties (primary sampling units
[PSUs]) at the first stage, and ZIP Code
clusters at the second stage; this clustering
helps control data collection costs because
face-to-face personal interviews are conducted to collect the data. The final stage
of sampling is at the level of beneficiaries
in the 5 percent sample with addresses in
the selected ZIP-Code clusters. At the final
stage of selection, the beneficiary sample
is stratified within seven age categories.
The target sample size for the continuing
annual sample is 12,000 responding beneficiaries. Beneficiaries eligible for Medicare
by disability (under age 65), as well as
the oldest old (85 or over), are oversampled (Apodaca, Judkins, and Lo, 1992).
Thus, when analyzing the MCBS, weights
must be employed for each respondent to
account for the differential sampling probabilities.
Sur vey Weights
Like many complex surveys, the MCBS
uses survey weights to account for differential probabilities of selection and to
adjust for nonresponse (Judkins and Lo,
1993). In this section we describe the
steps used to create the survey weights
for a panel of beneficiaries in the MCBS
(Westat, Inc., 2001).
Base Weights—For a panel in their initial
round of the MCBS, to account for differential probabilities of selection, base
weights are computed from their inverse
of probability of selection.

Poststratification Weights—After base
weights are created for a panel in their
initial round of the MCBS, poststratification adjustments are applied to ensure
consistency between the characteristics of
sampled beneficiaries, properly weighted,
and the national Medicare population.
Nonresponse Adjusted Weights—The
post-stratified weights for a panel in their
initial round of the MCBS are then adjusted for nonresponse at the initial round.
Potential predictors of initial round unit
response include Medicare and Medicaid
entitlement status, Medicare managed
care enrollment, medical reimbursements,
physicians’ fee ratios and practice cost indices, and demographic, socioeconomic, and
geographic variables. Cells for adjusting
weights for nonresponse are based either
on chi-square tests of association or models of response propensity. The resulting
weights are the panel’s initial round nonresponse weights.4 For the panel’s second
survey year, nonresponse weights are created by adjusting the initial round nonresponse weights to account for conditional
nonresponse in the panel’s second year
of the survey. The conditional response
rate in year two is defined as respondents divided by eligibles, where eligibles
are restricted to initial round respondents
who are alive on January 1 of year two.
Similar procedures are used to derive nonresponse weights for the panel’s third and
fourth years of the survey. In addition to
administrative data, survey data provided
in prior survey years can be used to adjust
for panel attrition. Candidate variables for
panel attrition adjustment include health,
functioning, demographic, geographic, utilization, and interview variables.
A further set of weights, the cross-sectional survey weights,
are also created for the MCBS. They are used in cross-sectional
analyses of the MCBS.
4

Health Care Financing Review/Summer 2006/Volume 27, Number 4

73

INITIAL ROUND UNIT
NONRESPONSE
Methodology
Our sample for analyzing initial round
unit nonresponse is confined to MCBS eligibles (respondents and nonrespondents)
in their initial round of the survey. For
this study, incoming eligibles are pooled
across 3 years of MCBS data (1997-1999)
to maximize sample size for our analyses.
Medicare administrative records, including claims, provide a unique opportunity
to analyze the impact of nonresponse on
MCBS estimates, since they provide data
on respondents and nonrespondents alike.
Claims for services received by persons
enrolled in managed care are not available
from these records. The data for beneficiaries in long-term care facilities are typically
provided by facility staff rather than the
beneficiary, and response rates are close to
100 percent. Beneficiaries who are eligible
for Medicare by end stage renal disease
(ESRD) are a small unique subpopulation.
For these reasons, the analysis sample
for studying potential bias was limited to
community-based, non-ESRD beneficiaries
enrolled in traditional Medicare fee-forservice (FFS). Our 3-year merged analytic
file for analyzing MCBS initial round unit
nonresponse has a sample size of 14,315.
Proxy measures, defined as variables
known for both respondents and nonrespondents that serve as proxies for studying the effects of nonresponse, were identified and used to compare respondents
and nonrespondents. Postratification survey weights for the selected sample were
obtained for this exercise; the weights
incorporated a poststratification adjustment to align the selected sample with the
complete frame, but no adjustments for

74

nonresponse. For the initial round, proxy
measures were based primarily on administrative record data.
The proxy measures used in comparing respondents and nonrespondents were
of two types based on the time that they
become available. The first type included
demographic measures that are available
from the sampling frame and could be (or
were) utilized in the weight adjustment process. The second type were CMS administrative record data that only became available for both respondents and nonrespondents some time after the survey had been
completed and may not have been available for application in the weight adjustment process for the current round. This
second type included selected diagnoses,
counts of services received by type, expenditures for health care, and hierarchical
condition categories (HCC) diagnostic cost
groups (DCG) risk scores, or HCC-DCG
risk scores, which were developed for risk
adjustment of Medicare managed care capitation payments (Pope et al., 2004).
The HCC-DCG risk score is an expenditure-weighted index of a beneficiary’s
diagnoses that predicts the relative risk of
future Medicare expenditures. A beneficiary’s HCC-DCG risk score is calculated by
dividing the beneficiary’s predicted expenditures by per capita expenditures for the
entire Medicare FFS population. An HCCDCG risk score above 1.0 indicates that
a beneficiary is predicted to have greater
future medical expenditures than the average Medicare FFS beneficiary (i.e., is sicker
than average), whereas an HCC-DCG risk
score below 1.0 indicates the beneficiary is
predicted to have lower than average future
health care costs, i.e., is healthier than average. In short, the HCC-DCG risk score is a
summary index of a beneficiary’s diagnostic
disease profile or burden, incorporating

Health Care Financing Review/Summer 2006/Volume 27, Number 4

both numbers and severity of serious disorders. Multiple diseases are aggregated into
a single index score using the metric of their
impact on future medical expenditures.
To measure nonresponse bias, estimates
were compared based on the respondent
sample and the eligible sample (respondents and nonrespondents). Comparisons
were first based on the poststratification
weights (before nonresponse adjustment)
and then recomputed using the nonresponse adjusted weights (after nonresponse
adjustment) for the respondent data and
the poststratification weight for the eligible
sample. The evaluation of the statistical significance of bias estimates was performed
individually on a large number of estimates.
No corrections for multiple comparisons
were applied, because the real interest was
in the individual comparisons. But if one
wished to assess the overall impact on bias,
a few statistically significant results among
a large number of measures would be likely
by chance even if the overall impact on bias
was low or negligible.
We did not attempt to directly evaluate
nonresponse bias for variables available for
survey respondents only. For these variables,
we cannot compare survey respondents to
nonrespondents or to eligibles. However, to
the extent that survey-only and administrative variables are correlated, it is reasonable
to infer that the degree of bias in administrative and survey variables is related. That
is, a large bias for administrative variables
implies the potential for a large bias among
survey variables. Conversely, if little bias is
observed among administrative variables,
our confidence of lack of significant bias
among survey variables is increased.
FINDINGS
Table 1 presents response rates overall and by selective demographic and eligibility characteristics. The overall 1997-

1999 initial round MCBS response rate for
our analysis sample is 82.6 percent. This
response rate is roughly comparable to
what is expected for large national health
surveys administered in person (Aday,
1996). Response rates by subcategory are
relatively consistent without many large
variations among the groups, the largest
difference in response rates being the
lower response in metropolitan versus nonmetropolitan areas. While the variations in
response rates are relatively small, vulnerable groups associated with poorer health
status respond at an equal or slightly
higher rate to the MCBS. For example,
Medicaid enrollees have a response rate of
85.7 percent compared with 82.0 percent
for those without Medicaid. Similarly, the
sickest beneficiaries with the highest HCCDCG risk scores have a higher response
rate than the healthiest beneficiaries with
low scores (85.6 and 79.1 percent, respectively).
Consistently, multiple logistic regression
analysis of response (Table 2) showed that
males, Medicaid enrollees, Black persons,
southerners, non-metropolitan residents,
younger beneficiaries, and those in poorer
health (upper quintiles of the HCC-DCG
risk score) were more likely to respond to
the initial round of the MCBS. The higher
response rate of many sicker, more vulnerable groups is surprising, and is contrary to
findings from recent nonresponse analyses
of other major Medicare surveys, such as
the Health Outcomes Survey (Khatutsky
et al., 2002). We speculate that these differences may arise from the different modes
of administration of the surveys, inperson
for the MCBS versus mail with telephone
follow up for the Health Outcomes Survey.
Table 3 compares MCBS initial round eligibles, respondents, and nonrespondents
by demographic, enrollment, and health
status characteristics. Nonrespondents
are further decomposed into refusals and

Health Care Financing Review/Summer 2006/Volume 27, Number 4

75

Table 1
MCBS Initial Round Unit Response Rates by Demographic, Eligibility, and Health Status
Characteristics1

Characteristic

Eligibles

All Sample

N
14,315

Age
Under 65 Years
65-74 Years
75-84 Years
85 Years or Over

Sex
Male
Female

Race
White
Black
Other

Original Reason for Medicare Entitlement
Aged
Disabled

Medicaid Status
No Medicaid
Medicaid

Current Reason for Medicare Entitlement
Aged
Disabled

Metropolitan Area Status
Non-Metropolitan
Metropolitan

Census Regions
North East
North Central
South
West
Other3

HCC-DCG Risk Score Quintiles4
0-20% (Lowest Score)
20-40%
40-60%
60-80%
80-100% (Highest Score)

Mortality
Died in the Year Following Initial Round
Survived the Year Following Initial Round

Respondents

Response Rate

Statistical
Significance2

N
11,817

Percent
82.6

—

2,609
5,391
4,664
1,651

2,174
4,442
3,837
1,364

83.3
82.4
82.3
82.6

—
—
—
—

6,280
8,035

12,079
1,463
773

13,477
831

12,127
2,188

11,706
2,609

4,114
10,201

2,917
3,565
5,385
2,216
232

2,892
2,840
2,864
2,858
2,861

5,268
6,549

9,938
1,244
635

11,096
715

9,941
1,876

9,643
2,174

3,675
8,142

2,336
2,906
4,540
1,833
202

2,288
2,286
2,356
2,439
2,448

83.9
81.5

82.3
85.0
82.2

82.3
86.0

82.0
85.7

82.4
83.3

89.3
79.8

80.1
81.5
84.3
82.7
87.1

79.1
80.5
82.3
85.3
85.6

***
—
—
—
**
—
—
—

**
—
—

***
—
—

*
—
—

***
—
—

**
—
—
—
—
—

**
—
—
—
—
—

890
13,425

739
11,078

83.0
82.5

—
—

*p<0.1.
**p<0.05.
***p<0.01.
1

MCBS community, fee-for-service sample. Beneficiaries with end stage renal disease are excluded.

2

Statistical significance testing for distribution.

3

Other includes Puerto Rico and other territories.

Diagnosis-based health status index computed from provider bills (claims). A higher hierarchical condition categories-diagnostic cost group (HCC-DCG)
score indicates poorer health.

4

NOTES: MCBS is Medicare Current Beneficiary Survey. Data unweighted.
SOURCE: RTI analysis of the 1997-1999 MCBS.

76

Health Care Financing Review/Summer 2006/Volume 27, Number 4

Table 2
Logistic Regression Model Estimating Likelihood of MCBS Initial Round Unit Response1

Characteristic
Age
Under 65 Years
65-74 Years (Omitted)
75-84 Years
85 Years or Over

Sex
Male (Omitted)
Female

Medicaid
Non-Enrolled (Omitted)
Enrolled

Race
White (Omitted)
Other
Black

Current Reason for Medicare Entitlement
Originally Entitled to Medicare by Age (Omitted)
Originally Entitled to Medicare by Disability

Census Regions
North East (Omitted)
North Central
South
West
Other

Metropolitan Area Status
Non-Metropolitan (Omitted)
Metropolitan

HCC-DCG Risk Score Quintiles2
Up to 20% (Omitted)
20-40%
40-60%
60-80%
80-100%

Panel
1997(Omitted)
1998
1999

Estimate

Standard
Error

Odds
Ratio

0.00
—
-0.15
-0.24

—
-0.15

—
0.14

—
-0.06
0.21

—
-0.02

—
-0.02
0.11
0.07
0.69

—
-0.75

—
0.03
0.26
0.48
0.53

—
0.01
0.08

0.07
—
0.06
0.09

—
0.05

—
0.08

—
0.10
0.09

—
0.10

—
0.06
0.06
0.07
0.21

—
0.06

—
0.08
0.07
0.07
0.08

—
0.05
0.06

1.00
—
0.86
0.79

—
0.86

—
1.15

—
0.94
1.23

—
0.98

—
0.98
1.12
1.07
1.99

—
0.47

—
1.04
1.30
1.61
1.70

—
1.01
1.08

Statistical
Significance
—
—
***
***
—
***
—
*
—
—
**
—
—

—
*
—
***
—
***
—
—
***
***
***
—
—
—

*p<0.1.
**p<0.05.
***p<0.01.
1

MCBS community, fee-for-service sample. Beneficiaries with end stage renal disease are excluded.

Diagnosis-based health status index computed from provider bills (claims). A higher hierarchical condition categories-diagnostic cost group (HCC-DCG)
score indicates poorer health.

2

NOTES: MCBS is Medicare Current Beneficiary Survey. Data weighted by poststratification weight adjusted for sampling design, but not for nonresponse. N=14,308.

other nonrespondents, where other nonrespondents represent Medicare beneficiaries who are unlocatable, physically or
mentally incompetent without available
proxy, out of area, etc. Initial round MCBS
nonrespondents are significantly healthier

than respondents, by 18 percent in terms
of lower current Medicare expenditures
($3,526 on average for nonrespondents
versus $4,309 for respondents) and by 11
percent in terms of the HCC-DCG risk
score (0.86 on average for nonrespondents

Health Care Financing Review/Summer 2006/Volume 27, Number 4

77

Table 3
Comparison of MCBS Initial Round Eligibles, Respondents, and Nonrespondents, by Selected
Characteristics1

Characteristic

Eligibles Respondents

All

Nonrespondents
Refusals
Other2

Statistical
Significance3

All Sample
N=14,315 N=11,817 N=2,498
N=1,852
N=646

Percent

Age

Under 65 Years
13.6
13.9
12.4
8.2
26.2
—
**
65-74 Years
44.0
43.9
44.2
46.5
36.7
—
**
75-84 Years
32.6
32.4
33.5
36.0
25.2
—
**
85 Years or Over
9.9
9.8
9.9
9.3
11.9
—
—

Sex
Male
43.4
44.2
39.8
37.6
47.2
*
**
Female
56.6
55.8
60.2
62.4
52.8
*
**

Race

White
86.0
85.6
87.5
91.7
73.5
*
**
Black
8.8
9.1
7.2
5.3
13.6
*
**
Other
5.3
5.3
5.3
3.0
12.9
—
**

Original Reason for Entitlement
Aged
94.0
93.7
95.1
96.1
91.8
*
**
Disabled
6.0
6.3
5.0
3.9
8.3
*
**

Medicaid Status
No Medicaid
87.2
86.6
89.8
94.3
75.0
*
**
Medicaid
12.8
13.4
10.2
5.7
25.0
*
**
Current Reason for Entitlement
Aged
86.4
86.2
87.6
91.8
73.8
—
Disabled
13.6
13.9
12.4
8.2
26.2
—
Metropolitan Area Status
Non-Metropolitan
27.5
29.8
16.7
15.8
19.9
*
Metropolitan
72.5
70.2
83.3
84.2
80.1
*

Census Regions
North East
20.3
19.7
23.1
23.4
22.2
*
North Central
25.5
25.2
26.8
29.1
19.3
—
South
37.5
38.4
33.6
32.8
36.3
*
West
15.0
14.9
15.3
14.5
18.1
—
4
Other
1.6
1.8
1.2
0.2
4.2
*
Mortality
Died in the Year Following Initial Round
Survived the Year Following Initial Round

Mean HCC-DCG Score5

Total Medicare Expenditures (Dollars)
% Users

6.0
94.0

0.95

4,172
88.5

**
**

**
**

—
**
—
**
**

6.0
94.0

0.97

4,309
89.8

6.0
94.1

0.86

3,526
85.0

4.9
95.2

0.80

2,935
87.5

9.5
90.5

1.05

5,458
77.0

—
—

*
—
*
*

**
**
**
—
**
**

Expenditures for Inpatient Services (Dollars) 2,025
% Users
17.6

2,092
18.3

1,708
14.5

1,371
13.2

2,811
18.8

*
*

**
**

Expenditures for Part B Services (Dollars) 1,252
% Users
87.9

1,285
88.7

1,093
84.3

1,019
87.1

1,336
75.5

*
*

**
**

*Statistically significant difference between respondents and all nonrespondents (p<0.05).
**Statistically significant difference between refusals and other nonrespondents (p<0.05).
1 MCBS community, fee-for-service sample. Beneficiaries with end stage renal disease are excluded.
2 Includes out of area, unlocatable, physically and mentally impaired without a proxy, and other types of nonrespondents.
3 Statistical significance testing on eligible and respondent differences is equivalent to statistical significance testing on all nonrespondent and respondent differences.
4 Other includes Puerto Rico and other territories.
5 Diagnosis-based health status index computed from provider bills (claims). A higher hierarchical condition categories-diagnostic cost group (HCCDCG) score indicates poorer health.
NOTES: MCBS is Medicare Current Beneficiary Survey. Data weighted by post-stratification weights not adjusted for nonresponse.
SOURCE: RTI analysis of the 1997-1999 MCBS.

78

Health Care Financing Review/Summer 2006/Volume 27, Number 4

versus 0.97 on average for respondents).
However, as is the case with sociodemographic characteristics, as a result of the
high initial round response rate, the differences between eligibles and respondents5
with respect to health status is much less
pronounced than the differences between
nonrespondents and respondents (e.g., 3
percent difference in current Medicare
expenditures—$4,172 for eligibles versus
$4,309 for respondents—versus the 18 percent difference between nonrespondents
and respondents).
Our analysis (Table 3) also demonstrates
that nonrespondents are not a homogeneous group. Refusals, which represent
approximately three-quarters of all nonrespondents, are substantially healthier than
other nonrespondents, and account for the
overall better health of nonrespondents
relative to respondents.6
Overall, the results in Table 3 suggest
that although statistically significant differences occur between nonrespondents and
respondents on such demographic characteristics as sex, race, and geographic
distribution, and although nonrespondents
overall appear to be healthier than respondents, because of the high initial MCBS
response rate, the magnitude of the differences between eligibles and respondents is
relatively small, and thus unlikely to create
a major potential for bias.
Table 4 analyzes the effect of existing
MCBS unit nonresponse weighting adjustments on selected variables by comparing
pre- and post- nonresponse adjustment
estimates. The differences in eligible and
respondent means (proportions), each
adjusted by only poststratification weights
5 When comparing eligibles and respondents, we cannot perform

standard hypothesis tests that assume independent samples.
However, statistical tests on the difference between eligibles and
respondents are equivalent to statistical tests on the difference
between nonrespondents and respondents (Kalton, 1983).
6 Other nonrespondents are also a diverse group representing
a mix of very sick and expensive nonrespondents and relatively
healthy respondents (Kautter et al., 2003).

are shown in this table. The differences
are estimates of nonresponse bias before
MCBS nonresponse adjustment weighting. Correspondingly, the table presents
the differences in eligible and respondent
means (proportions), each adjusted by
not only poststratification weights, but by
nonresponse weights as well. The differences are the estimates of nonresponse
bias after MCBS nonresponse adjustment
weighting.
As shown in Table 4, current MCBS nonresponse adjustments align the distribution
of respondents across sociodemographic
characteristics to be far more consistent
with the eligible sample. All comparisons
of respondents with eligibles are statistically significant before nonresponse adjustment. Only four comparisons (enrollment
in Medicaid, health status, total Medicare
expenditures, and percent utilizing inpatient
services) remain statistically significant
after nonresponse adjustment and, even
in those cases, the magnitude of the bias
measure is reduced. Although initial round
nonresponse bias is small and is further
reduced by MCBS nonresponse weights,
it is not entirely eliminated. For example,
after nonresponse weights are applied,
the estimated bias in mean Medicare total
expenditures falls from $137 (3.3 percent)
to $85 (2.0 percent),7 for a 38 percent
reduction in the estimated bias.
PANEL ATTRITION
Methodology
Longitudinal surveys include any type
of survey for which at least some of the
units are measured more than once. These
include traditional panel surveys, with
fixed or rotating panels, retrospective longitudinal surveys, and cohort followups
7 The

former estimate is statistically significantly different from
zero at the 1 percent level, the latter at the 10 percent level.

Health Care Financing Review/Summer 2006/Volume 27, Number 4

79

80

Health Care Financing Review/Summer 2006/Volume 27, Number 4

Respondents Not Adjusted for Nonresponse
Respondents Adjusted for Nonresponse
Weight
Weight

MCBS Initial
Poststratification Poststratification

Statistical Poststratification Nonresponse

Statistical
Eligibles (E)
Respondents (R ) Difference (R-E) Significance Eligibles (E) Respondents (R ) Difference (R-E) Significance

SOURCE: RTI analysis of the 1997-1999 MCBS.

NOTES: MCBS is Medicare Current Beneficiary Survey.

health status index computed from provider bills (claims). A higher hierarchical condition categories-diagnostic cost group (HCC-DCG) score indicates poorer health.

community, fee-for-service sample. Beneficiaries with end stage renal disease are excluded.

2 Diagnosis-based

1 MCBS

***p<0.01.

**p<0.05.

*p<0.1.

Age (%)
Under 65 Years
13.6
13.9
0.25
*
13.6
13.8
0.17
—
65-74 Years
44.0
43.9
-0.05
—
44.0
44.2
0.18
—
75-84 Years
32.6
32.4
-0.19
—
32.6
32.3
-0.29
—
85 Years or Over
9.9
9.8
-0.01
—
9.9
9.8
-0.06
—

Female (%)
56.6
55.8
-0.76
***
56.6
56.4
-0.19
—

White (%)
86.0
85.6
-0.31
**
86.0
85.7
-0.28
—

Enrolled in Medicaid (%)
12.8
13.4
0.55
***
12.8
13.2
0.38
***

Metropolitan Area Status
72.5
70.2
-2.27
***
72.5
72.0
-0.51
—

Original Reason for Medicare
Entitlement: Disability (%)
6.0
6.3
0.23
**
6.0
6.2
0.12
—

Mean HCC-DCG Score2
0.95
0.97
0.02
***
0.95
0.96
0.01
**

Total Medicare Expenditures (Dollars)
4,172
4,309
137
***
4,172
4,257
85
*
% Users
88.5
89.8
1.33
***
89.0
89.4
0.46
—

Expenditure for Inpatient Services (Dollars)
2,025
2,092
67
**
2,025
2,072
47
—
% Users
17.6
18.3
0.66
***
17.6
17.9
0.34
**

Expenditures for Part B Services (Dollars)
1,252
1,285
34
***
1,252
1,273
21
—
% Users
87.9
88.7
0.76
***
87.9
88.3
0.38
—

Characteristic

Table 4
Effect of MCBS Initial Round Nonresponse Adjustment on Selected Characteristics1

(Duncan and Kalton, 1987). Although the
increased availability of longitudinal survey data has been one of the most important developments in applied social science
research over the last few decades, the
most potentially damaging threat to the
value of longitudinal survey data is the
presence of biasing attrition, i.e., attrition
that is selectively related to outcome variables of interest (Fitzgerald, Gottschalk,
and Moffitt, 1998).
In addition to analyzing initial round unit
nonresponse, an analysis of panel attrition in the MCBS is performed. Two pairs
of MCBS file years, i.e., 1997–1998 and
1998–1999, are pooled to construct three
separate panel attrition analysis samples:
• Second Year Panel Attrition Analysis
Sample—First year respondents eligible
in the second year, used to analyze panel
attrition between the first and second
MCBS survey years.
•Third Year Panel Attrition Analysis
Sample—Second year respondents eligible in the third year, used to analyze
panel attrition between the second and
third MCBS survey years.
• Fourth Year Panel Attrition Analysis
Sample—Third year respondents eligible in the fourth year, used to analyze
panel attrition between the third and
fourth MCBS survey years.
Similar to our initial round unit nonresponse sample, our sample for analyzing
panel attrition is restricted to beneficiaries
residing in the community, enrolled in traditional FFS Medicare, and not eligible for
Medicare by ESRD. The sample sizes for
our second, third, and fourth year panel
attritions are, respectively, 7,544, 6,345,
and 5,437.
The conditional response rate for the
second (third, fourth) year panel attrition
analysis sample is defined as second (third,
fourth) year attrition sample respondents
divided by eligibles, where eligibles are

restricted to first (second, third) year
respondents who are alive on January 1 of
the second (third, fourth) year. Conditional
nonresponse bias can be decomposed into
the conditional response rate and the difference in conditional population means for
respondents and nonrespondents. For variables available for both respondents and
nonrespondents, we first calculate conditional response rates for each panel attrition
analysis sample, stratifying by sociodemographics. Then we estimate differences in
respondent and nonrespondent conditional
means and proportions for demographic,
enrollment, health status, and service utilization measures. Note that unlike for the
initial round unit nonresponse analysis, for
the panel attrition analysis, in addition to
administrative data, survey data provided
in prior survey years could also be used as
proxy measures, e.g., self-reported general
health status, prescription drug expenditures, etc.
In addition to analyzing the components
of conditional nonresponse bias, we analyze bias directly by comparing means
(proportions) for respondents and eligibles. We estimate bias before and after
applying MCBS adjustments for panel attrition. Each set of response analyses (i.e.,
second, third, and fourth year) was conditional on the response in the prior MCBS
year. Consequently, the largest potential
for bias occurred in the initial round,
because the conditional response rates
in subsequent years remained relatively
high and increased by year as the more
reluctant respondents were removed from
the pool of persons sampled for each subsequent year.
The cumulative response rate for a panel
in their second (third, fourth) MCBS year
was approximated by calculating the product of their first year response rate and
their conditional response rates through
their second (third, fourth) survey year.

Health Care Financing Review/Summer 2006/Volume 27, Number 4

81

Finally, in order to assess the cumulative
impact of nonresponse over the 4 survey
years, the cumulative nonresponse bias at
each year was approximated by summing
the estimated biases up to and including
that year.
FINDINGS
As shown in Table 5, the conditional
response rates for our second, third, and
fourth year panel attrition analysis samples
are 88.9, 94.7, and 96.8 percent, respectively, yielding progressively declining
attrition rates of 11.1, 5.3, and 3.2 percent.
Declining attrition rates are common in
longitudinal surveys like the MCBS (U.S.
Bureau of the Census, 1998).
Response rates by subcategory are
relatively consistent without many large
variations among the groups. As shown in
Table 5, one factor affecting response rate
remains prominent in all three attrition
samples: the statistically significant lower
response rate in metropolitan areas versus
non-metropolitan areas. In addition, the
higher response rate for Medicaid enrollees compared with non-Medicaid enrollees
is statistically significant in two of the three
attrition samples.
Although Medicaid eligibility is often
correlated with poor health status, we
do not find any consistent evidence that
conditional response rates are different
for those in poor health versus those in
better health. However, our first year nonresponse analysis indicated that sicker
beneficiaries have a higher propensity to
respond to the MCBS (Tables 1 and 2).
These two findings suggest that different
beneficiary characteristics affect first year
nonresponse and panel attrition.
Our findings also suggest that beneficiary
characteristics affecting the response propensity vary for each attrition sample. For
example, as shown in Table 6, nonrespon82

dents in the second year attrition sample
were not statistically different from respondents in terms of self-reported general
health status. Nonrespondents in the third
year attrition sample, however, assessed
their own health significantly differently
than respondents. More than 12 percent of
nonrespondents reported their health as
poor (versus 7.9 percent for respondents),
and 23.9 percent reported their health as
fair (versus 17.8 percent for respondents).
Finally, in the fourth year attrition sample,
again no difference in self-reported general health status is found between respondents and nonrespondents.
Although we find statistically significant
differences between respondents and nonrespondents on such sociodemographic
characteristics as age, income, and geographic distribution, because of the high
MCBS conditional response rates for each
attrition sample, the magnitude of the differences between eligibles and respondents is relatively small, and thus unlikely
to create a major potential for bias.
Our descriptive findings are supported
by multiple logistic regression analyses
(Table 7). Metropolitan area residence is
found to be a consistent and significant
factor affecting the probability of response
in all MCBS attrition samples when other
factors are held constant. In addition,
Medicaid enrollment is a significant factor
in two of the three attrition samples.
For all three attrition samples, MCBS
adjustments for panel attrition aligned the
distribution of respondents across sociodemographic and enrollment characteristics
to be far more consistent with the eligible
sample (Kautter et al., 2003). In particular,
across all attrition samples, the adjustments were effective in correcting the
Medicaid and metropolitan area distributions. In every attrition sample there are
some analysis variables for which MCBS
adjustments for panel attrition were not as

Health Care Financing Review/Summer 2006/Volume 27, Number 4

Table 5
MCBS Conditional Response Rates for Panel Attrition Analysis Samples, by Demographics,
Eligibility, and Health Status Characteristics1

Characteristic

Second Year
Statistical
Third Year
Statistical
Fouth Year
Statistical
Attrition Sample Significance2 Attrition Sample Significance2 Attrition Sample Significance2

N=7,544

N=6,345

N=5,437

Percent

All Sample
88.9
—
94.7
—
96.8

Age

**

Under 65 Years
89.0
—
93.1
—
95.8
65-74 Years
89.9
—
94.9
—
96.6
75-84 Years
88.1
—
94.9
—
97.5
85 Years or Over
87.5
—
96.1
—
97.1

Sex

Male
89.4
—
94.9
—
96.9
Female
88.5
—
94.6
—
96.7

Living Alone
*
Yes
88.3
—
94.4
—
96.7
No
89.2
—
95.5
—
97.1

Income Categories
*
Under $15,000
89.1
—
95.5
—
97.4
$15,001-$30,000
90.6
—
94.4
—
96.7
$30,001-$50,000
90.0
—
93.6
—
95.4
Over $50,000
88.3
—
94.5
—
96.5

Race
White
88.8
—
94.7
—
96.7
Black
89.7
—
95.0
—
96.4
Other
89.9
—
93.6
—
99.0

Medicaid Status
***
**
No Medicaid
88.3
—
94.5
—
96.7
Medicaid
92.5
—
96.1
—
97.5

Current Reason for Entitlement

***

Aged
88.9
—
95.1
—
97.0
Disabled
89.0
—
93.1
—
95.8
Metropolitan Area Status

***

**

Non-Metropolitan
92.3
—
95.7
—
97.7
Metropolitan
87.4
—
94.2
—
96.4

HCC-DCG Risk Score Quintiles3

*

0-20 % (Lowest Score)
88.3
—
94.3
—
96.1
20-40 %
89.5
—
94.6
—
97.7
40-60 %
90.3
—
95.4
—
97.2
60-80 %
89.3
—
95.0
—
96.7
80-100 % (Highest Score)
87.1
—
94.2
—
96.5

General Health

***

Excellent
89.9
—
95.3
—
96.5
Very Good
88.9
—
95.5
—
97.2
Good
89.4
—
95.5
—
97.0
Fair
88.4
—
93.3
—
97.0
Poor
87.9
—
91.9
—
95.8

With ADL Difficulties
None
89.1
—
94.7
—
96.8
1-2
89.2
—
95.0
—
97.0
3-4
87.8
—
94.4
—
96.1
5-6
87.2
—
93.0
—
98.1

See footnotes at the end of the table.

Health Care Financing Review/Summer 2006/Volume 27, Number 4

—
—
—
—
—
—
—
—
—
—
—
—
—
—
—
—
—
—
—
—
**
—
—
—
—
—
—
—
—
—
—
—
—
—
—
—
—

83

Table 5—Continued
MCBS Conditional Response Rates for Panel Attrition Analysis Samples, by Demographics,
Eligibility, and Health Status Characteristics1

Characteristic

Second Year
Statistical
Third Year
Statistical
Fouth Year
Statistical
Attrition Sample Significance2 Attrition Sample Significance2 Attrition Sample Significance2

N=7,544

N=6,345

N=5,437

Percent

With IADL Difficulties
None
88.8
—
95.2
—
97.0
1-2
89.4
—
94.0
—
96.4
3-4
87.5
—
94.5
—
96.6
5-6
90.7
—
94.9
—
98.2

—
—
—
—

*p<0.1.
**p<0.05.
***p<0.01.
MCBS community, fee-for-service sample. Beneficiaries with end stage renal disease are excluded. The conditional response rate for the second
(third, fourth) year panel attrition analysis sample is defined as second (third, fourth) year attrition sample respondents divided by eligibles, where eligibles are restricted to first (second, third) year respondents who are alive on January 1 of the second (third, fourth) year.

1

2

Statistical significance testing for distribution.

Diagnosis-based health status index computed from provider bills (claims). A higher hierarchical condition categories-diagnostic cost group (HCCDCG) score indicates poorer health.

3

NOTES: MCBS is Medicare Current Beneficiary Survey. ADLs are activities of daily living. IADLS are instrumental activities of daily living. Data for
second (third, fourth) year attrition sample weighted by first (second, third) year nonresponse adjusted weights.
SOURCE: RTI analysis of the 1997-1999 MCBS.

effective. However, there is no particular
pattern for what types of variables require
additional adjustment.
In addition to analyzing conditional nonresponse in the MCBS, an analysis of cumulative nonresponse is conducted. Cumulative
response rates in the MCBS are found to
be comparable to other large national surveys (U.S. Bureau of the Census, 1998).
While conditional response rates increase
over the MCBS interview cycle, cumulative response rates decrease. As shown in
Table 8, after 4 years of longitudinal data
collection, the overall response rate for
analysis of the complete longitudinal data
set was 67.3 percent. Subpopulations exhibited a range of overall response rates. As an
example, the range in cumulative response
rates after four survey years is 77.0 percent
for non-metropolitan residents and 63.3
percent for metropolitan residents.
As shown in Table 9, there is a slight
upward trend in the cumulative nonresponse bias for certain variables, such as
Medicare total expenditures (from $85 to
$108) and inpatient expenditures (from $47
to $74), indicating that further nonresponse
84

adjustments might be warranted. However,
overall, MCBS nonresponse weighting procedures were found to be effective in
adjusting for cumulative nonresponse.
ITEM NONRESPONSE
Methodology
Item nonresponse in the MCBS is also
analyzed. The analysis sample for item
nonresponse is confined to 1999 MCBS
Access to Care file8 respondents and, like
the analyses of initial round unit nonresponse and panel attrition, are restricted
to beneficiaries residing in the community,
enrolled in traditional FFS Medicare, and
not eligible for Medicare by ESRD. There
are 12,524 beneficiaries meeting these criteria.
Item nonresponse rates are derived by
calculating the ratio of item nonrespondents
to item eligibles. The following response
categories are assumed to be item nonresponse: (1) not ascertained, (2) don’t know,
There are two data files from the MCBS that are released in
annual Access to Care and Cost and Use files.
8

Health Care Financing Review/Summer 2006/Volume 27, Number 4

Health Care Financing Review/Summer 2006/Volume 27, Number 4

85

Second Year Attrition Sample
Third Year Attrition Sample
Fourth Year Attrition Sample

Statistical

Statistical

Statistical
Eligible Respondent Nonrespondent Significance Eligible Respondent Nonrespondent Significance Eligible Respondent Nonrespondent Significance

See footnotes at the end of the table.

General Health
Excellent
16.0 16.1
14.9
—
14.5 14.6
13.0

14.3
Very good
25.5 25.5
25.6
—
27.0 27.3
23.0
*
26.3
Good
29.2 29.3
28.5
—
32.0 32.3
27.5
*
32.9
Fair
18.8 18.6
20.2
—
18.2 17.8
23.9
***
18.4
Poor
10.3 10.3
10.9
—
8.1 7.9
12.6
***
7.9

14.2
26.5
33.0
18.4
7.9

16.4
22.0
32.4
19.5
9.7

—
—
—
—
—

Sample
7,544
6,708
836
—
6,345
6,009
336
—
5,437
5,264
173
—

Percent

Percent

Percent
Age

Under 65 Years
13.6 13.6
13.3
—
12.9
12.6
16.8
**
11.6
11.5
14.7
—
65-74 Years
45.5 45.9
41.9
**
43.1
43.1
42.6
—
39.7
39.6
44.3
—
75-84 Years
31.9 31.6
34.5
*
34.6
34.7
33.5
—
38.5
38.7
32.1
*
85 Years or Over
9.0 8.9
10.3
—
9.4
9.5
7.1
—
10.1
10.2
8.9
—

Female
56.1 55.9
57.4
—
56.0
55.8
58.9
—
56.5
56.4
58.9
—

Income Categories
Under $15,000
42.9 43.0
42.3
—
40.9
41.3
33.6
***
41.1
41.5
30.4
***
$15,001-$30,000
25.0 25.5
21.2
***
32.8
32.6
35.1
—
33.9
33.8
36.0
—
$30,001-$50,000
12.9 13.1
11.2
—
16.9
16.8
18.9
—
15.0
14.7
23.1
***
Over $50,000
9.5 9.4
10.6
—
9.5
9.3
12.5
*
10.0
10.0
10.6
—

Race

White
86.0 85.8
87.4
—
86.1
86.2
85.7
—
86.8 86.7
90.3
—
Black
8.7 8.7
8.1
—
8.6
8.6
7.3
—
8.4 8.4
8.0
—
Other
5.4 5.5
4.5
—
5.3
5.2
7.0
—
4.8 4.9
1.7
**

Medicaid Status
12.9 13.5
8.5
—
12.8
13.0
8.2
—
12.8
13.0
7.2
**

Metropolitan Area Status 71.6 70.5
80.4
***
71.2
70.9
77.6
***
69.7
69.4
77.0
**

Mean HCC-DCG Risk
Score2
0.93
0.92
0.96
*
0.94
0.93
1.02
**
0.97
0.97
0.97
—

Total Medicare
Expenditures
4,019
4,060
3,695
—
3,584
3,520
4,709
**
3,789
3,803
3,369
—

Inpatient Ependitures 1,966
2,009
1,628
—
1,634
1,592
2,380
**
1,708
1,715
1,482
—

Prescription Drug
Expenditures
N/A
N/A
N/A
—
872 872
865
—
866 867
859
—

Characteristic

Table 6
Comparison of MCBS Eligibles, Respondents, and Nonrespondents for Panel Attrition Analysis Samples, by Selected Characteristics1

86

Health Care Financing Review/Summer 2006/Volume 27, Number 4

Second Year Attrition Sample
Third Year Attrition Sample
Fourth Year Attrition Sample

Statistical

Statistical

Statistical
Eligible Respondent Nonrespondent Significance Eligible Respondent Nonrespondent Significance Eligible Respondent Nonrespondent Significance

SOURCE: RTI Analysis of the 1997-1999 MCBS.

NOTES: MCBS is Medicare Current Beneficiary Survey. Data for second (third, fourth) year panel attrition analysis sample weighted by first (second, third) year nonresponse adjusted weights. ADLs are activities of daily
living. IADLS are instrumental activities of daily living. Statistical testing between respondents and nonrespondents is equivalent to statistical testing between respondents and eligibles. Beneficiary characteristics are measured prior to the year in which response status is determined. NA is not available. A higher HCC-DCG score indicates poorer health.

health status index computed from provider bills (claims). A higher hierarchical condition categories-diagnostic cost group (HCC-DCG ) score indicates poorer health.

community, fee-for-service sample. Beneficiaries with end stage renal disease are excluded.

2 Diagnosis-based

1 MCBS

***p<0.01.

**p<0.05.

*p<0.1.

With IADL Difficulties
None
51.7 51.7
52.3
—
54.8 55.1
50.7
*
54.5 54.6
53.0
—
1-2
30.5 30.7
29.3
—
29.7 29.5
33.5
—
29.0 28.9
33.3
—
3-4
11.3 11.1
13.0
—
10.2 10.2
10.6
—
10.3 10.3
11.1
—
5-6
6.4
6.5
5.5
—
5.2 5.2
5.3
—
6.1 6.3
2.6
**

Sample
7,544
6,708
836
—
6,345
6,009
336
—
5,437
5,264
173
—

Percent

Percent

Percent
With ADL difficulties
None
67.8 67.8
67.2
—
72.4 72.4
72.6
—
72.9 72.8
74.9
—
1-2
21.3 21.4
20.6
—
18.8 18.8
17.1
—
18.1 18.2
16.8
—
3-4
7.0
6.9
7.5
—
5.7 5.6
6.2
—
5.6 5.6
6.3
—
5-6
4.0
3.9
4.7
—
3.1 3.0
4.1
—
3.3 3.3
2.0
—

Characteristic

Table 6—Continued
Comparison of MCBS Eligibles, Respondents, and Nonrespondents for Panel Attrition Analysis Samples, by Selected Characteristics1

Table 7
Logistic Regression Models Estimating Likelihood of MCBS Conditional Response for Panel
Attrition Analysis Samples1

Characteristic

Odds Ratio
Second Year
Statistical
Third Year
Statistical
Fouth Year
Statistical
Attrition Sample Significance Attrition Sample Significance Attrition Sample Significance

Age (65-74 Years Category Omitted)
Under 65 Years
0.83
—
0.71
—
0.69
75-84 Years
0.82
**
1.05
—
1.35
85 Years or Over
0.77
*
1.29
—
1.25

—
—
—

Sex (Male Category Omitted)
Female
0.94
—
0.78
**
0.76

*

Medicaid Status
(Non-Medicaid Category Omitted)
Enrolled in Medicaid
1.84
***
1.92
***
1.62

—

Race (White Category Omitted)
Black
1.06
—
1.12
—
0.99
Other
0.98
—
0.57
**
3.06

—
—

Original Reason for Entitlement
(Aged Category Omitted)
Disabled
0.71
**
1.15
—
1.12

—

Census Regions
(North East Category Omitted)
North Central
1.09
—
1.13
—
1.29
South
1.18
—
1.05
—
1.17
West
1.24
*
1.46
*
1.68
2
Other
4.21
***
4.01
**
1.81

—
—
*
—

Metro Area (Non-Metro Category
Omitted)
Metropolitan Area Status
0.60
***
0.71
**
0.71

*

HCC-DCG Quintiles
(0-20% Quintile Category Omitted)3
20-40%
1.13
—
0.89
—
1.18
40-60%
1.26
*
1.04
—
0.92
60-80%
1.20
—
0.96
—
0.86
80-100%
0.98
—
0.86
—
0.79

—
—
—
—

Marital Status (Non-Married
Category Omitted)
Married
0.60
—
0.94
—
0.89

—

Education (Absence of College
Degree Category Omitted)
College Degree
1.04
—
1.07
—
1.15

—

Income
(Over $50,000 Category Omitted)
Under $15,000
1.32
***
1.65
—
1.32
$15,001-$30,000
1.64
***
1.34
—
1.01
$30,001-$50,000
1.58
—
1.21
**
0.67

—
—
—

Self-Reported General Health Status
(Excellent Category Omitted)
Very Good General Health
0.97
—
1.05
—
Good
0.94
—
0.97
—
Fair
0.82
—
0.59
**
Poor
0.84
—
0.50
**

1.54
1.34
1.23
0.97

*
—
—
—

Difficulty with ADLs
(0 Category Omitted)
1-2
1.03
—
1.29
—
1.14
3-4
0.87
—
1.15
—
0.89
5-6
0.76
—
0.83
—
1.08

—
—
—

See footnotes at the end of the table.

Health Care Financing Review/Summer 2006/Volume 27, Number 4

87

Table 7—Continued
Logistic Regression Models Estimating Likelihood of MCBS Conditional Response for Panel
Attrition Analysis Samples1

Characteristic

Odds Ratio
Second Year
Statistical
Third Year
Statistical
Fouth Year
Statistical
Attrition Sample Significance Attrition Sample Significance Attrition Sample Significance

Difficulty with IADLs
(0 Category Omitted)
1-2
1.14
—
0.86
—
0.86
3-4
0.98
—
1.98
—
0.92
5-6
1.50
—
1.05
—
2.42

—
—
—

*p<0.1.
**p<0.05.
***p<0.01.
1 MCBS community, fee-for-service sample. Beneficiaries with end stage renal disease are excluded.
2 Other includes Puerto Rico and other territories.
3 Diagnosis-based health status index computed from provider bills (claims). A higher hierarchical condition categories-diagnostic cost group
(HCC-DCG) score indicates poorer health.
NOTES: MCBS is Medicare Current Beneficiary Survey. Beneficiary charcteristics are measured prior to the year in which response status is determined. Data for second (third, fourth) year panel attrition analysis sample weighted by first (second, third) year nonresponse adjusted weights. For
second, third, and fourth year panel attrition samples, N = 7,540, N = 6,340, and N = 5,433, respectively.
SOURCE: RTI Analysis of the 1997-1999 MCBS.

and (3) refused. In addition, we rank the
survey variables by item nonresponse, and
examine the distribution of the item nonresponse rates. Finally, survey variables
with relatively high item nonresponse rates
are selected, and the distribution of nonresponse categories is examined to determine
how prevalent the don’t know response
category is and whether it is a legitimate,
meaningful response. For the selected survey variables, item respondents and nonrespondents are compared using available
proxy measures.
FINDINGS
As shown in Table 10, item nonresponse
is generally low in the 1999 MCBS Access
to Care file. The mean item nonresponse
rate across survey variables is 1.6 percent,
and the majority of the variables have
negligible item nonresponse of at most 0.3
percent. However, the distribution of item
nonresponse rates across survey variables
in the MCBS is skewed, with 10 percent
of survey variables having an item nonresponse rate of at least 5.4 percent. For
example, the survey question on income
has a 6.7-percent item nonresponse rate.
88

It is important to note though that some
variables with high item nonresponse rates
are only asked of a small subset of survey
participants. For example, for the survey
question “Need help three months from
now with toileting,” the item nonresponse
rate is 22.2 percent, but the number of
eligibles for this survey question is only
18. With these small sample sizes, the data
have limited utility even if all eligible persons responded.
Our analysis of response category distributions among item nonrespondents revealed
certain patterns. “Refusals” and “not ascertained” item nonresponse choices are rare.
The great majority of item nonrespondents
select “don’t know” and they often select
this choice because there is no other appropriate valid response category available. For
example, for the survey question “Current
Veteran’s Administration disability rating,”
there are 54 item nonrespondents to the
question, with 53 answering “don’t know”
(Table 10). In particular, survey participants often select “don’t know” if they have
trouble recalling a certain health or preventive event, or do not remember details about
their military service history. Survey participants also select this answer choice to

Health Care Financing Review/Summer 2006/Volume 27, Number 4

Table 8
Cumulative Response Rates Across MCBS Interview Cycle, by Demographic, Eligibility, and
Health Status Characteristics1
Characteristic

First Year (Initial Round)

All Sample

Age
Under 65 Years
65-74 Years
75-84 Years
85 Years or Over

Sex
Male
Female

Race
White
Black
Other

Original Reason for Entitlement
Aged
Disabled

Medicaid status
No Medicaid
Medicaid

Current Reason for Entitlement
Aged
Disabled

Metropolitan Area Status
Non-Metropolitan
Metropolitan

Census Regions
North East
North Central
South
West
Other2

Mortality
Died in the Following Year
Survived the Following Year

HCC-DCG Risk Score Quintiles3
0-20% (Lowest Score)
20-40%
40-60%
60-80%
80-100% (Highest Score)

N=14,315

82.6

83.3
82.4
82.3
82.6

83.9
81.5

82.3
85.0
82.2

82.3
86.0

82.0
85.7

82.4
83.3

89.3
79.8

80.1
81.5
84.3
82.7
87.1

83.0
82.5

79.1
80.5
82.3
85.3
85.6

Second Year

Third Year

Fourth Year

N=7,544

73.4

74.2
74.1
72.5
72.3

75.0
72.2

73.0
76.2
73.8

73.3
74.8

72.3
79.3

73.2
74.2

82.4
69.7

68.9
72.2
75.8
74.4
83.1

71.9
73.5

69.9
72.0
74.3
76.2
74.6

Percent

N=6,345

69.5

69.0
70.3
68.8
69.4

71.2
68.2

69.2
72.4
69.1

69.4
71.0

68.3
76.2

69.6
69.0

78.9
65.7

64.9
68.3
71.6
71.4
81.1

68.0
69.6

65.9
68.2
70.9
72.4
70.3

N=5,437
67.3
66.2
67.9
67.1
67.4
69.0
66.0
66.9
69.8
68.4
67.2
69.1
66.1
74.3
67.5
66.1
77.0
63.3
62.5
66.0
69.3
69.6
80.5
65.4
67.4
63.3
66.6
68.9
70.0
67.8

MCBS community, fee-for-service sample. Beneficiaries with end stage renal disease are excluded. The cumulative response rate for a panel in
their second (third, fourth) MCBS year is approximated by calculating the product of their first year response rate and their conditional response rates
through their second (third, fourth) survey year.

1

2

Other includes Puerto Rico and other territories.

Diagnosis-based health status index computed from provider bills (claims). A higher hierarchical condition categories-diagnostic cost group (HCCDCG) score indicates poorer health.

3

NOTES: MCBS is Medicare Current Beneficiary Survey. Data are unweighted.
SOURCE: RTI Analysis of the 1997-1999 MCBS.

Health Care Financing Review/Summer 2006/Volume 27, Number 4

89

Table 9
Cumulative Nonresponse Bias Across MCBS Interview Cycle, Before and After MCBS
Nonresponse Adjustment1

Characteristic

First Year (Initial Round)
Before
After

Second Year
Before
After

Third Year
Before
After

Fourth Year
Before
After

Age

Under 65 Years
0.25
0.17
0.29
0.20
0.06
0.15
-0.04
0.18
65-74 Years
-0.05
0.18
0.37
0.32
0.40
0.29
0.23
0.09
75-84 Years
-0.19
-0.29
-0.52
-0.41
-0.45
-0.43
-0.23
-0.29
85 Years or Over
-0.01
-0.06
-0.18
-0.15
-0.05
-0.04
-0.01
-0.03

Female
-0.76
-0.19
-0.93
-0.30
-1.10
-0.51
-1.18
-0.71

Race

White
-0.31
-0.28
-0.49
-0.35
-0.47
-0.30
-0.59
-0.37
Black
0.33
0.18
0.40
0.23
0.47
0.27
0.48
0.28
Other
-0.01
0.09
0.09
0.12
0.00
0.03
0.11
0.08

Originally Disabled
0.23
0.12
0.06
-0.09
0.09
-0.09
0.11
-0.07

Medicaid Status
0.55
0.38
1.12
0.37
1.38
0.48
1.57
0.53

Currently Disabled2
0.25
0.17
0.29
0.20
0.06
0.15
-0.05
0.18

Metropolitan Area Status
-2.27
-0.51
-3.38
-0.64
-3.74
-0.76
-4.00
-0.78

Census Regions
North East
-0.59
-0.10
-1.24
-0.16
-1.42
-0.29
-1.62
-0.45
North Central
-0.28
0.12
-0.39
-0.09
-0.36
-0.05
-0.33
-0.02
South
0.83
0.14
1.27
0.16
1.23
0.01
1.23
-0.01
West
-0.07
-0.15
0.11
-0.03
0.25
0.13
0.39
0.26
Other
0.10
-0.01
0.23
0.12
0.29
0.20
0.32
0.22

Mean HCC-DCG Risk Score
0.02
0.01
0.01
0.01
0.00
0.00
0.00
0.00

HCC-DCG Risk Score Quintiles3

0-20% (Lowest Score)
-0.80
-0.36
-0.88
-0.48
-0.92
-0.44
-1.06
-0.54
20-40%
-0.55
-0.42
-0.42
-0.29
-0.43
-0.33
-0.28
-0.21
40-60%
-0.05
-0.05
0.23
0.19
0.36
0.29
0.40
0.33
60-80%
0.66
0.45
0.74
0.54
0.82
0.56
0.81
0.54
80-100% (Highest Score)
0.74
0.39
0.32
0.05
0.19
-0.05
0.15
-0.08

Total Medicare Expenditures
$137.32
$85.11 $178.89
$138.83
$115.03
$90.74 $129.77 $108.02

Inpatient
$67.35
$47.39 $110.74
$96.66
$68.42
$64.12 $76.33 $74.10

Mortality

Died in the Following Year
0.02
-0.07
0.15
-0.11
0.17
-0.12
0.18
-0.13
MCBS community, fee-for-service sample. Beneficiaries with end stage renal disease are excluded. The cumulative nonresponse bias at each year
is approximated by summing the estimated biases up to and including that year.

1

2

Current reason for Medicare entitlement is disability—equivalent to under 65 age group.

3

Diagnosis-based health status index computed from provider bills (claims).

NOTES: MCBS is Medicare Current Beneficiary Survey. HCC-DCG is hierarchical condition categories-diagnostic cost group.
SOURCE: RTI Analysis of the 1997-1999 MCBS.

indicate their lack of knowledge about their
health insurance coverage details or when
they have difficulty predicting needing help
with activities of daily living (ADLs). We
suggest that only some of these “don’t
know” responses, e.g., for variables such as

90

income and education, should be classified
as true missing data. For other items, e.g.,
knowledge or amount of information that
survey participants possess, these responses, if retained or reclassified, can provide
additional valuable information.

Health Care Financing Review/Summer 2006/Volume 27, Number 4

Table 10
Distribution of MCBS Item Nonresponse Rates1

Item Nonresponse Rate2

Mean Item Nonresponse Rate

Quantiles
100% (Maximum)
90%
75%
50% (Median)
25%

Selected Variables
Plan 1 Cover Stay in Nursing Home3
Need Help 3 Months from Now with Toileting4
Current Veteran's Administration Disability Rating3
Does Doctor Make House Calls3
Income5
High School Grade Completed5
1

1.6

28.9
5.4
1.3
0.3
0.0

26.6
22.2
13.9
9.8
6.7
0.7

Item Nonrespondents
Item Eligibles All Item Nonrespondents Don't Know
—

—
—
—
—
—

8,550
18
388
11,675
12,524
12,524

—

—
—
—
—
—

2,276
4
54
1,141
839
86

—

—
—
—
—
—

2,266
4
53
1,141
261
68

MCBS community, fee-for-service sample. Beneficiaries with end stage renal disease are excluded.

Item nonresponse rates are derived by calculating the ratio of item nonrespondents to item eligibles. The following response categories are assumed
to be item nonresponse: not ascertained; don't know; and refused.

2

3

Knowledge question.

4

Predictive question.

5

Sensitive question.

NOTES: MCBS is Medicare Current Beneficiary Survey.
SOURCE: RTI Analysis of the 1999 MCBS Access to Care File.

The survey variables we examined
indepth may be broadly classified into
three groups (Kautter et al., 2003):
• Sensitive questions, such as income
and education, that are known in survey research for yielding lower item
response rates.
• Questions related to recall of certain
past events such as eye exams, prostate
cancer tests, and whether the Medicare
& You handbook was received.
• Questions assessing beneficiary knowledge of various issues related to health
and health insurance coverage.
Comparison of health and demographic
characteristics of item respondents and
nonrespondents to these variables revealed
several patterns. While both income and
education variables are considered sensitive items, item nonresponse to education
is much lower than for income.9 Item
nonresponse to the income question is
associated with the characteristics gener9 The

item nonresponse rate for income is 6.7 versus 0.7 percent
for education (Table 10).

ally associated with higher income (e.g.,
male, white, no Medicaid), whereas item
nonresponse to the education question
is associated with characteristics generally associated with lower education (older,
sicker, Medicaid).
Nonrespondents to questions requiring
recall tend to be older, have a higher
proportion of minorities and higher rates
of Medicaid enrollment, and are likely
to be in significantly poorer health than
respondents. However, item respondents
and nonrespondents to knowledge questions appear to have fewer differences in
demographic and health status characteristics. Item nonrespondents to knowledge
questions tend to be less educated than
respondents. While there are some other
variations in demographic characteristics
such as race or Medicaid enrollment, we
could not detect any other particular patterns that set apart one group from another. There are no consistent differences in
health status between the two groups.

Health Care Financing Review/Summer 2006/Volume 27, Number 4

91

SUMMARY AND CONCLUSIONS
This study had several objectives in evaluating the impact of nonresponse on MCBS
estimates: (1) to examine unit nonresponse
for beneficiaries in their initial interview
round, (2) to evaluate panel attrition, (3)
and to measure item nonresponse. For
initial round nonresponse, although statistically significant differences occurred
between respondents and nonrespondents
on such demographic characteristics as
sex, race, and geographic distribution,
the magnitude of the differences between
eligibles and respondents was relatively
small and unlikely to cause a major potential for bias. However, current nonresponse
adjustments were not as effective for health
status, expenditure, and service utilization characteristics. Although initial nonresponse bias was small and further reduced
by MCBS nonresponse weights, it was not
entirely eliminated.
Beneficiar y characteristics affecting
the response propensity varied for each
panel attrition sample, but because of the
high MCBS conditional response rates for
each sample, the magnitude of the differences between eligibles and respondents
was relatively small, and thus unlikely to
create bias. Cumulative response rates
were found to be comparable to other
large national surveys. While conditional
response rates increased over the MCBS
interview cycle, cumulative response rates
decreased. Finally, item nonresponse was
generally low in the MCBS, with the exception of several items pertaining to recall
of past events and knowledge of certain
health insurance information.10
Nonresponse in panel surveys can be a
serious problem because it is cumulative
over all rounds of the survey. Our findings indicate that for most of the measures
10

92

Income also yields a relatively high item nonresponse rate.

studied, the bias caused by differences
between nonrespondents and respondents
in the MCBS was substantially reduced or
eliminated by the nonresponse procedures
currently employed.
References
Aday, L.A.: Designing and Conducting Health
Surveys. Second Edition. Jossey-Bass Publishers.
San Francisco, CA. 1996.
Apodaca, R., Judkins, D., Lo, A., et al.: Sampling
from HCFA Lists, Proceedings of the Section on
Survey Research Methods. American Statistical
Association. 1992.
Dillman, D.A., Eltinge, J.L., Groves, R.M., et al.:
Survey Nonresponse in Design, Data Collection,
and Analysis. Groves, R.M., Dilman, D.L., Eltinge,
J.L., and Little, R.J.A. (eds.): Survey Nonresponse.
John Wiley & Sons, Inc. New York, NY. 2002.
Duncan, C.G. and Kalton G.: Issues of Design and
Analysis of Surveys Across Time. International
Statistical Review 55:97-117, 1983.
Fitzgerald, J., Gottschalk, P., and Moffitt, R.: An
Analysis of Sample Attrition in Panel Data: The
Michigan Panel Study of Income Dynamics. The
Journal of Human Resources. 33(2):251-299, Spring
1998.
Judkins, D. and Lo, A.: Components of Variance and
Nonresponse Adjustment for the Medicare Current
Beneficiary Survey. Proceedings of the Section
on Survey Research Methods of the American
Statistical Association. 1993.
Kalton, G.: Introduction to Survey Sampling. Sage
Publications. Thousand Oaks, CA. 1983.
Kautter, J. and Pope, G.C.: CMS Frailty Adjustment
Model. Health Care Financing Review 26(2):1-20,
Winter 2004-2005.
Kautter, J., Khatutsky, G., Pope, G.C., et al.: Impact
of Nonresponse on MCBS Estimates. Final Report
to the Centers for Medicare & Medicaid Services.
September 2003.
Khatutsky, G., McCall, N.T., Pope, G.C., et al.:
Health Status Nonresponse Bias in the Medicare Feefor-Service Health Outcomes Survey. Final Report to
the Centers for Medicare & Medicaid Services. July
31, 2002.
Pope, G.C., Kautter, J., Ellis, R.P., et al.: Risk
Adjustment of Medicare Capitation Payments
Using the CMS-HCC Model. Health Care Financing
Review 25(4):119-141, Summer 2004.

Health Care Financing Review/Summer 2006/Volume 27, Number 4

U.S. Bureau of the Census: Survey of Income and
Program Participation (SIPP) Quality Profile 1998.
SIPP Working Paper Number 230. Third Edition.
Washington DC. 1998.
Westat, Inc.: MCBS: Report on Weighting for
Round 25 (1999 Access to Care). Memorandum
to the Centers for Medicare & Medicaid Services.
February 21, 2001.
Reprint Requests: John Kautter, Ph.D., RTI International, 1440
Main Street, Suite 310, Waltham, MA 02451. E-mail: jkautter@
rti.org

Health Care Financing Review/Summer 2006/Volume 27, Number 4

93