CMS-10249 Supporting Statement part B [rev 12-30-11]

Supporting Statement Part B

Supporting Statement for Paperwork Reduction Act

Administrative Requirements for Section 6071 of the Deficit Reduction Act

CMS-10249, OMB 0938-1053

The MFP Quality of Life Survey will employ statistical methods.

1. Universe, Sample and Response Rates

The universe for the Quality of Life survey is the total number of Medicaid beneficiaries enrolled in the MFP program in the 44 participating states. However, because the analysis will be conducted separately by state and by five eligibility groups within each state, there are potentially 255 different “subpopulations” for this survey. States differ in terms of how many of the five eligibility groups they include in their MFP program, and how many beneficiaries they expect to enroll in each group. Table 1 displays the target number of program participants by state and eligibility group.

The survey is to be collected at three points in time for each sample member: a baseline conducted while the beneficiary is still residing in the institution but after all arrangements have been made for their transition to the community; a first followup at 11 months after returning to the community; and a second followup at about 24 months after return to the community.

Baseline interviews will be administered to all MFP participants in each eligibility group in each state who enter the program during the first three years of operation, until the number of completed baseline interviews in that subpopulation in that state exceeds 750. No sampling will take place until that target is achieved, because this is the minimum sample size needed to have the desired precision for the analysis of which characteristics of beneficiaries are associated with favorable changes in quality of life under the program.¹ Once a state transitions 750 people in a target population in the first three years, MPR will discuss with the state and CMS the possibility of sampling, the appropriate sampling rate to support the required analysis, and the method for identifying those to be interviewed. In general, if the state is successful in reaching its target number of participants and does not wish to survey all future participants, we will develop a plan to select a sample equal to one-third of future participants for whom a baseline interview will be attempted.

The planned approach, rather than sampling from the beginning in state/eligibility subgroup cells with targets of more than 750 individuals, provides protection against inadequate samples in states that target a large number of participants but fall far short of expectations. Our past experience with many social programs suggests that states and other program operators typically overestimate the number of program participants they will attract. Even if a state does reach 750 participants before the end of the 36-month intake period, it may still not be worthwhile to institute sampling, if relatively few additional participants are likely to enroll in MFP during the remaining months. Furthermore, some states may decide that they would rather have more observations on later enrollees in order to improve the precision of analyses of changes over time in the effectiveness of the MFP program that may occur as the program matures. States with large enrollment targets for a given eligibility group will need to weigh this disadvantage to sampling against the benefit of lower survey costs.

Follow up interviews will be attempted for everyone who receives a baseline interview. This approach ensures that adequate samples will be obtained, regardless of whether states are able to achieve their target number of enrollees in future years.² If sampling is implemented for any target groups, the data will be weighted in our analyses so that estimates accurately reflect the full population of enrollees.

We expect to have very high response rates at baseline, because beneficiaries will be in an institution preparing for return to the community. Thus, they will be easy to locate. Furthermore, they are expected to be very amenable to answering questions about their quality of life in the institution, since the purpose of the MFP program is to enable them to satisfy their desire to return to the community. An additional factor expected to lead to high baseline response rates is that transition planners, who will be collecting various other types of information from the participant to facilitate their transition, will administer the baseline in most states. The trust that participants will have established with these planners, combined with participants’ strong desire to return to the community and the ease of locating them, is expected to lead to baseline response rates of 95 percent.

We expect response rates of about 90 percent at the first follow up, which will be attempted with all baseline respondents. While respondents will no longer be in institutions at follow up, they typically will be easily located, because nearly all will be receiving Medicaid-covered personal care and other services in their home under the Medicaid program, along with other transitional MFP services available only during the first 12 months after the beneficiary transitions from the institution to the community. Thus, case managers and program staff will know where to find them, and high response rates are expected for the same reasons as at the baseline.

We expect the response rate to the second (24-month) followup to be somewhat lower, as participants’ will no longer be participating in MFP. However, we expect it to still be quite high (80 percent of baseline respondents) because earlier respondents will be familiar with the survey by this time, interested in discussing their own well-being, and still receiving home and community based services from Medicaid home care providers. An 80 percent response rate will yield an analysis sample of 600 cases for data from the second followup, for cells with a baseline sample of 750 completed interviews.

2. Procedures for the Collection of Information

Procedures and Methodology for Sample Selection. As the bolded numbers in Table 1 indicate, sampling will be used in at most 9 of the 44 states, and within only one of the four eligibility groups in six of these states. In all other states and cells, baseline interviews will be attempted with all MFP participants. Participants will enter the program throughout the three-year intake period, but the exact number who will enter and the timing of their entry is uncertain. Participants must be interviewed very soon after they decide to participate in MFP and find suitable housing in the community, while they are still in the institution. Thus, no list frame will be available from which to draw a sample.

Once 750 participants in a given state/eligibility group have completed baseline interviews, if sampling is to occur it must be done as additional participants are identified. States will be required to submit to MPR the names, eligibility group, and contact information of each individual to be transitioned to the community, as they are identified. MPR will randomly assign each such participant in cells designated for sampling to either the survey sample or to the non-survey sample. Program participants who fall into cells designated for sampling and who enter after the first 750 baseline respondents will have a one in three chance of being selected for the survey.

Allocation of the Survey Sample. If each state enrolls exactly the targeted number of beneficiaries, from each eligibility group, the baseline survey sample will be allocated as indicated in Table 2, assuming a response rate of .95 and a sampling rate of one-third for all participants once the target of 750 baselines have been completed.

Statistical Precision and Minimum Detectable Difference. The rationale for sampling only if the number of completed baselines exceeds 750 is derived from the desire to have 80 percent power to detect differences of approximately 10 percentage points between two equal-sized subgroups within a given state/eligibility group cell, for a binary outcome measure with a mean of .50, using a two-tailed test conducted at the .10 level. For example, we will test whether, within a particular cell, those with a given characteristic (such as having a cognitive impairment) are less likely than those without this characteristic to rate the quality of their life at followup higher than they rated it at baseline. Assuming equal sizes for the two subgroups being compared, samples of 750 baselines would result in 600 completed followup surveys at 24-months. This sample size yields a minimum detectable difference of 10.2 percentage points, using the following standard formula:

MDD = 2.487 *s * sqrt (2/300) = .102,

where the standard error s is equal to 0.5 under our assumption of a binary outcome with a mean of .50. In practice, we will use logistic regression models to draw such these comparisons across subgroups defined by a number of different factors. Thus, the precision of our estimates may be slightly greater than this estimate based on a simple comparison of means.

3. Methods to Maximize Response Rates and Deal with Issues of Nonresponse

As discussed above, response rates should be quite high at each round, given that sample members will be easy to locate. Other factors that should lead to high response rates are (1) the short time required to complete the survey (20 minutes, on average), (2) the focus of the survey on an issue of considerable importance and relevance to the respondent (their own quality of life), and (3) the low literacy level (fourth grade) required to complete the survey. Potential respondents who have difficulty speaking or hearing (or who do not speak English) will be offered the opportunity to receive assistance from a family member in understanding the questions or providing their own answer. The survey will be offered in Spanish as well as English, and arrangements will be made to use telephonic translation services to complete interviews with potential respondents who speak other languages and have no one available to translate the questions for them.

Given that we expect response rates for even the 24-month followup to be 80 percent or higher, no elaborate method for addressing issues of nonresponse are expected to be necessary. However, sample weights to account for nonresponse will be constructed as the inverse of the predicted probability of response obtained from a model we will estimate. The re-weighted sample should be more representative of the population on observable factors. We will also examine the difference in results obtained for the full sample and for respondents-only, using outcomes available from administrative data for all participants. Similarity of such results for the administrative data will increase confidence that the data on survey respondents adequately represents the population for outcomes that are measurable only with the survey.

4. Tests of Procedures or Methods to be Undertaken

The survey questions were adapted from existing surveys conducted on a similar target population, then examined for literacy level. The primary survey from which the questions were drawn was the Participant Experience Survey, which is used to collect information on the quality of life for individuals receiving home and community based services, and with which states participating in the study were familiar. In addition, we drew some questions from the National Core Indicators survey, the Ask Me! survey, and the Cash and Counseling survey, all of which were developed to collect information on individuals receiving personal care and other services in their homes. The survey was then reviewed by representatives from the states participating in the demonstration and modified in response to their concerns, focusing on simplifying the survey and response categories as much as possible without eliminating the essential content.

The survey was pre-tested on 9 individuals who were receiving similar types of services in the community. Three of the pretest respondents selected were individuals who had mental retardation or developmental disabilities, to ensure that this targeted subpopulation would be able to answer the survey questions as well. No problems were uncovered. The average time to complete the survey was 20 minutes.

5. Individuals Consulted on Statistical Aspects of Design

The person responsible for the statistical aspects of the sample design and analysis is:

• Randall S. Brown, Ph.D., Mathematica Policy Research, Inc. (609) 275-2393

Mathematica Policy Research, Inc., is conducting the evaluation under contract to CMS (contract number HHSM-500-2005-00025I [0002]) Dr. Brown is a senior advisor for the study. He has primary responsibility for the project design and data collection strategy.

TABLE 1
money follows the person (MFP) demonstration grant information by state

State	Number of Transitions Proposed	Elderly	Physically Disabled	MR/DD	Mental Illness	Other
Arkansas	700	187	248	217	48	0
California	1,566	359	536	484	42	145
Colorado	450	288	50	72	36	4
Connecticut	5,235	1,801	2,212	130	1,092	0
Delaware	233	80	181	15	7	0
District of Columbia	914	300	180	434	0	0
Florida	1,703	1,208	328	0	167	0
Georgia	1,642	284	506	852	0	0
Hawaii	502	249	232	21	0	0
Idaho	265	145	90	30	0	0
Illinois	1,625	793	242	176	414	0
Indiana	1,846	1,023	763	60	0	0
Iowa	568	0	0	568	0	0
Kansas	1,225	306	582	295	30	12
Kentucky	884	163	154	317	222	28
Louisiana	1,008	280	338	390	0	0
Maine	104	63	21	0	0	20
Maryland	3,873	2,146	1,364	256	87	20
Massachusetts	2,192	1,358	510	142	182	0
Michigan	3,065	1,680	1,385	0	0	0
Minnesota	2,461	741	179	357	0	28
Mississippi	595	72	142	138	243	0
Missouri	1,256	333	538	357	0	28
Nebraska	420	112	162	98	0	48
Nevada	520	256	256	8	0	0
New Hampshire	347	80	110	82	53	22
New Jersey	506	203	34	269	0	0
New Mexico	670	600	0	0	70	0
New York	1,725	513	664	0	0	548
North Carolina	715	307	186	222	0	0
North Dakota	277	174	28	75	0	0
Ohio	3,178	1,008	1,398	527	245	0
Oklahoma	899	255	497	147	0	0
Oregon	299	101	142	49	0	7
Pennsylvania	2,568	1,397	580	238	353	0
Rhode Island	600	594	66	0	0	0
South Carolina	300	240	60	0	0	0
Tennessee	2,225	1,195	980	50	0	0
Texas	11,751	3,749	3,053	4,947	2	0
Vermont	375	323	52	0	0	0
Virginia	1,229	373	385	471	0	0
Washington	4,291	2,016	1,882	345	48	0
West Virginia	520	168	294	0	58	0
Wisconsin	1,127	489	492	135	11	0
Totals	68,514	28,012	22,052	12,974	4,120	1,356

Note: The bolded numbers indicate state targeted populations where sampling would be allowed.

1 The rationale for this sample size is that we need a sufficient number of observations at the followup interviews to be confident that the differences in quality of life between two subgroups of participants within a given state/eligibility group cell reflect true differences between the subgroups in the effects of MFP on quality of life, and not simply chance differences. For example, we will test for whether elderly participants who have a cognitive impairment at baseline are more likely than elderly participants without such impairment to report a change in whether they are treated with respect by their caregivers. From the tests, we will conclude that the effects of MFP on respect are larger for those with a cognitive impairment only if the observed difference between the two subgroups is larger than what might have occurred simply by chance. However, unless we have data on roughly 600 participants or more at the first followup, we cannot be confident that the difference is greater than what might have occurred by chance unless the observed difference is very large. Obtaining a followup sample of 600 requires that we have about 750 surveyed at baseline, assuming that only 80 percent of those who complete the baseline are expected to complete the second followup interview.

2The concern here is that a state with a sizable target (say, 1500 enrollees), evenly distributed over three years, may meet its enrollment target for the first year (500 in this example), but then taper off substantially. If we select a 50 percent sample of participants to receive the baseline, based on the expected enrollment and the target of 750, we would have 250 cases in year one, but would fall far short of the target of 750 over the 3-year life of the study, if the program was only able to recruit half its target in the second and third years. In this case, even though 1000 beneficiaries would have participated, we would have only 500 completed baselines. Furthermore, analysis for the final report will have two years of follow up data only for beneficiaries who enroll during the grantee’s first 12 months of operations, so having 750 completed baselines during the grantee’s first year will provide the desired level of precision for that analysis.

8