2 CMS-10249 - Supporting Statement part B MFP 2014 - 11.26.14

Supporting Statement Part B
Supporting Statement for Paperwork Reduction Act
Administrative Requirements for Section 6071 of the Deficit Reduction Act
CMS-10249, OMB 0938-1053
The MFP Quality of Life Survey will employ statistical methods.
1.

Universe, Sample and Response Rates

The universe for the Quality of Life survey is the total number of Medicaid
beneficiaries enrolled in the MFP program in the 45 participating grantees.
However, because the analysis will be conducted separately by state and by five
eligibility groups within each state, there are potentially 225 different
“subpopulations” for this survey. States differ in terms of how many of the five
eligibility groups they include in their MFP program, and how many beneficiaries
they expect to enroll in each group. Table 1 displays the target number of
program participants by state and eligibility group.
The survey is to be collected at three points in time for each sample member: a
baseline conducted while the beneficiary is still residing in the institution but after
all arrangements have been made for their transition to the community; a first
follow-up at 11 months after returning to the community; and a second follow-up
at about 24 months after return to the community.
Baseline interviews will be administered to all MFP participants in each eligibility
group in each state who enter the program during the first three years of
operation, until the number of completed baseline interviews in that
subpopulation in that state exceeds 750. No sampling will take place until that
target is achieved, because this is the minimum sample size needed to have the
desired precision for the analysis of which characteristics of beneficiaries are
associated with favorable changes in quality of life under the program. 1 Once a
state transitions 750 people in a target population in the first three years, MPR
will discuss with the state and CMS the possibility of sampling, the appropriate
sampling rate to support the required analysis, and the method for identifying
1

The rationale for this sample size is that we need a sufficient number of observations at the followup interviews to be confident that the differences in quality of life between two subgroups of participants
within a given state/eligibility group cell reflect true differences between the subgroups in the effects of
MFP on quality of life, and not simply chance differences. For example, we will test for whether elderly
participants who have a cognitive impairment at baseline are more likely than elderly participants without
such impairment to report a change in whether they are treated with respect by their caregivers. From the
tests, we will conclude that the effects of MFP on respect are larger for those with a cognitive impairment
only if the observed difference between the two subgroups is larger than what might have occurred simply
by chance. However, unless we have data on roughly 600 participants or more at the first follow-up, we
cannot be confident that the difference is greater than what might have occurred by chance unless the
observed difference is very large. Obtaining a follow-up sample of 600 requires that we have about 750
surveyed at baseline, assuming that only 80 percent of those who complete the baseline are expected to
complete the second follow-up interview.

1

those to be interviewed. In general, if the state is successful in reaching its target
number of participants and does not wish to survey all future participants, we will
develop a plan to select a sample equal to one-third of future participants for
whom a baseline interview will be attempted.
The planned approach, rather than sampling from the beginning in state/eligibility
subgroup cells with targets of more than 750 individuals, provides protection
against inadequate samples in states that target a large number of participants but
fall far short of expectations. Our past experience with many social programs
suggests that states and other program operators typically overestimate the
number of program participants they will attract. Even if a state does reach 750
participants before the end of the 36-month intake period, it may still not be
worthwhile to institute sampling, if relatively few additional participants are
likely to enroll in MFP during the remaining months. Furthermore, some states
may decide that they would rather have more observations on later enrollees in
order to improve the precision of analyses of changes over time in the
effectiveness of the MFP program that may occur as the program matures. States
with large enrollment targets for a given eligibility group will need to weigh this
disadvantage to sampling against the benefit of lower survey costs.
Follow up interviews will be attempted for everyone who receives a baseline
interview. This approach ensures that adequate samples will be obtained,
regardless of whether states are able to achieve their target number of enrollees in
future years. 2 If sampling is implemented for any target groups, the data will be
weighted in our analyses so that estimates accurately reflect the full population of
enrollees.
We expect to have very high response rates at baseline, because beneficiaries will
be in an institution preparing for return to the community. Thus, they will be easy
to locate. Furthermore, they are expected to be very amenable to answering
questions about their quality of life in the institution, since the purpose of the
MFP program is to enable them to satisfy their desire to return to the community.
An additional factor expected to lead to high baseline response rates is that
transition planners, who will be collecting various other types of information from
the participant to facilitate their transition, will administer the baseline in most
states. The trust that participants will have established with these planners,
2

The concern here is that a state with a sizable target (say, 1500 enrollees), evenly distributed over
three years, may meet its enrollment target for the first year (500 in this example), but then taper off
substantially. If we select a 50 percent sample of participants to receive the baseline, based on the expected
enrollment and the target of 750, we would have 250 cases in year one, but would fall far short of the target
of 750 over the 3-year life of the study, if the program was only able to recruit half its target in the second
and third years. In this case, even though 1000 beneficiaries would have participated, we would have only
500 completed baselines. Furthermore, analysis for the final report will have two years of follow up data
only for beneficiaries who enroll during the grantee’s first 12 months of operations, so having 750
completed baselines during the grantee’s first year will provide the desired level of precision for that
analysis.

2

combined with participants’ strong desire to return to the community and the ease
of locating them, is expected to lead to baseline response rates of 95 percent.
We expect response rates of about 90 percent at the first follow up, which will be
attempted with all baseline respondents. While respondents will no longer be in
institutions at follow up, they typically will be easily located, because nearly all
will be receiving Medicaid-covered personal care and other services in their home
under the Medicaid program, along with other transitional MFP services available
only during the first 12 months after the beneficiary transitions from the
institution to the community. Thus, case managers and program staff will know
where to find them, and high response rates are expected for the same reasons as
at the baseline.
We expect the response rate to the second (24-month) follow-up to be somewhat
lower, as participants’ will no longer be participating in MFP. However, we
expect it to still be quite high (80 percent of baseline respondents) because earlier
respondents will be familiar with the survey by this time, interested in discussing
their own well-being, and still receiving home and community based services
from Medicaid home care providers. An 80 percent response rate will yield an
analysis sample of 600 cases for data from the second follow-up, for cells with a
baseline sample of 750 completed interviews.
2.

Procedures for the Collection of Information

Procedures and Methodology for Sample Selection. As the bolded numbers in
Table 1 indicate, sampling will be used in at most 9 of the 45 grantees , and
within only one of the four eligibility groups in six of these states. In all other
states and cells, baseline interviews will be attempted with all MFP participants.
Participants will enter the program throughout the three-year intake period, but
the exact number who will enter and the timing of their entry is uncertain.
Participants must be interviewed very soon after they decide to participate in MFP
and find suitable housing in the community, while they are still in the institution.
Thus, no list frame will be available from which to draw a sample.
Once 750 participants in a given state/eligibility group have completed baseline
interviews, if sampling is to occur it must be done as additional participants are
identified. States will be required to submit to MPR the names, eligibility group,
and contact information of each individual to be transitioned to the community, as
they are identified. MPR will randomly assign each such participant in cells
designated for sampling to either the survey sample or to the non-survey sample.
Program participants who fall into cells designated for sampling and who enter
after the first 750 baseline respondents will have a one in three chance of being
selected for the survey.
Allocation of the Survey Sample. If each state enrolls exactly the targeted
number of beneficiaries, from each eligibility group, the baseline survey sample

3

will be allocated as indicated in Table 2, assuming a response rate of .95 and a
sampling rate of one-third for all participants once the target of 750 baselines
have been completed.
Statistical Precision and Minimum Detectable Difference. The rationale for
sampling only if the number of completed baselines exceeds 750 is derived from
the desire to have 80 percent power to detect differences of approximately 10
percentage points between two equal-sized subgroups within a given
state/eligibility group cell, for a binary outcome measure with a mean of .50,
using a two-tailed test conducted at the .10 level. For example, we will test
whether, within a particular cell, those with a given characteristic (such as having
a cognitive impairment) are less likely than those without this characteristic to
rate the quality of their life at follow-up higher than they rated it at baseline.
Assuming equal sizes for the two subgroups being compared, samples of 750
baselines would result in 600 completed follow-up surveys at 24-months. This
sample size yields a minimum detectable difference of 10.2 percentage points,
using the following standard formula:
MDD = 2.487 *s * sqrt (2/300) = .102,
where the standard error s is equal to 0.5 under our assumption of a binary
outcome with a mean of .50. In practice, we will use logistic regression models to
draw such these comparisons across subgroups defined by a number of different
factors. Thus, the precision of our estimates may be slightly greater than this
estimate based on a simple comparison of means.

4

3.
Methods to Maximize Response Rates and Deal with Issues of
Nonresponse
As discussed above, response rates should be quite high at each round, given that
sample members will be easy to locate. Other factors that should lead to high
response rates are (1) the short time required to complete the survey (20 minutes,
on average), (2) the focus of the survey on an issue of considerable importance
and relevance to the respondent (their own quality of life), and (3) the low
literacy level (fourth grade) required to complete the survey. Potential
respondents who have difficulty speaking or hearing (or who do not speak
English) will be offered the opportunity to receive assistance from a family
member in understanding the questions or providing their own answer. The
survey will be offered in Spanish as well as English, and arrangements will be
made to use telephonic translation services to complete interviews with potential
respondents who speak other languages and have no one available to translate the
questions for them.
Given that we expect response rates for even the 24-month follow-up to be 80
percent or higher, no elaborate method for addressing issues of nonresponse are
expected to be necessary. However, sample weights to account for nonresponse
will be constructed as the inverse of the predicted probability of response obtained
from a model we will estimate. The re-weighted sample should be more
representative of the population on observable factors. We will also examine the
difference in results obtained for the full sample and for respondents-only, using
outcomes available from administrative data for all participants. Similarity of
such results for the administrative data will increase confidence that the data on
survey respondents adequately represents the population for outcomes that are
measurable only with the survey.
4.

Tests of Procedures or Methods to be Undertaken

The survey questions were adapted from existing surveys conducted on a similar
target population, then examined for literacy level. The primary survey from
which the questions were drawn was the Participant Experience Survey, which is
used to collect information on the quality of life for individuals receiving home
and community based services, and with which states participating in the study
were familiar. In addition, we drew some questions from the National Core
Indicators survey, the Ask Me! survey, and the Cash and Counseling survey, all of
which were developed to collect information on individuals receiving personal
care and other services in their homes. The survey was then reviewed by
representatives from the states participating in the demonstration and modified in
response to their concerns, focusing on simplifying the survey and response
categories as much as possible without eliminating the essential content.
The survey was pre-tested on 9 individuals who were receiving similar types of
services in the community. Three of the pretest respondents selected were

5

individuals who had intellectual disabilities or developmental disabilities, to
ensure that this targeted subpopulation would be able to answer the survey
questions as well. No problems were uncovered. The average time to complete
the survey was 20 minutes.
5.

Individuals Consulted on Statistical Aspects of Design

The person responsible for the statistical aspects of the sample design and analysis
is:
•
Randall S. Brown, Ph.D., Mathematica Policy Research, Inc. (609) 2752393
Mathematica Policy Research, Inc., is conducting the evaluation under contract to
CMS (contract number HHSM-500-2005-00025I [0002]) Dr. Brown is a senior
advisor for the study. He has primary responsibility for the project design and
data collection strategy.

6

TABLE 1
MONEY FOLLOWS THE PERSON (MFP) DEMONSTRATION GRANT
INFORMATION BY STATE

State
Alabama
Arkansas
California
Colorado
Connecticut
Delaware
District of
Columbia
Florida
Georgia
Hawaii
Idaho
Illinois
Indiana
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Rhode Island
South Carolina
Tennessee
Texas
Vermont
Virginia
Washington
West Virginia

Number of
Transitions
Proposed
1210
700
1,566
450
5,235
233

Elderly
269
187
359
288
1,801
80

Physically
Disabled
259
248
536
50
2,212
181

ID/DD
335
217
484
72
130
15

914

300

180

434

0

0

1,703
1,642
502
265
1,625
1,846
568
1,225
884
1,008
104
3,873
2,192
3,065
2,461
595
1,256
420
520
347
506
670
1,725
715
277
3,178
899
299
2,568
600
300
2,225
11,751
375
1,229
4,291
520

1,208
284
249
145
793
1,023
0
306
163
280
63
2,146
1,358
1,680
741
72
333
112
256
80
203
600
513
307
174
1,008
255
101
1,397
594
240
1,195
3,749
323
373
2,016
168

328
506
232
90
242
763
0
582
154
338
21
1,364
510
1,385
179
142
538
162
256
110
34
0
664
186
28
1,398
497
142
580
66
60
980
3,053
52
385
1,882
294

0
852
21
30
176
60
568
295
317
390
0
256
142
0
357
138
357
98
8
82
269
0
0
222
75
527
147
49
238
0
0
50
4,947
0
471
345
0

167
0
0
0
414
0
0
30
222
0
0
87
182
0
0
243
0
0
0
53
0
70
0
0
0
245
0
0
353
0
0
0
2
0
0
48
58

0
0
0
0
0
0
0
12
28
0
20
20
0
0
28
0
28
48
0
22
0
0
548
0
0
0
0
7
0
0
0
0
0
0
0
0
0

7

Mental
Illness
312
48
42
36
1,092
7

Other
35
0
145
4
0
0

Wisconsin
Totals

1,127
68,514

489
28,012

492
22,052

135
12,974

11
4,120

0
1,356

Note: The bolded numbers indicate state targeted populations where sampling would be allowed.

8