CMS-10427 Supporting_DocB_10 24 12

PART B

For-Profit PACE Study

Supporting Statement for Collection of Information Employing Statistical Methods

June 20, 2012

Nancy Duda

David Jones

Frank Yoon

Jennifer Ott

Contract Number: HHSM-500-2005-00025I/HHSM-500-T0005 Mathematica Reference Number: 06965.400 Submitted to: Centers for Medicare & Medicaid Services 7500 Security Boulevard, WB-06-05 Baltimore, MD 21244 Project Officer: Julia A. Zucco Submitted by: Mathematica Policy Research P.O. Box 2393 Princeton, NJ 08543-2393 Telephone: (609) 799-3535 Facsimile: (609) 799-0005 Project Director: David Jones

PART B For-Profit Pace Study Supporting Statement for Collection of Information Employing Statistical Methods June 20, 2012 Nancy Duda David Jones Frank Yoon Jennifer Ott

CONTENTS

Chapter Page

PART B. Supporting Statement for Paperwork Reduction Act Submission 1

B. Collection of Information Employing Statistical Methods 1

1. Respondent Universe and Sampling Methods 1

2. Procedures for the Collection of Information 4

3. Methods to Maximize Response Rates and Deal with Issues of Nonresponse 6

4. Test of Procedures or Methods to be Undertaken 8

5. Individuals Consulted on Statistical Aspects of Design and Individuals Collecting and/or Analyzing Data 9

APPENDIX A1: Advance MAILING to pace enrollees

APPENdix A2: Information Brochure for pace enrollee

Appendix B: For Profit Pace survey

TABLES

B.1 Pace Enrollees by For-Profit Versus Not-For-Profit Status 2

B.2 Target PACE Samples 3

B.3 Design effects, Statistical Power, and Detectable Differences 6

PART B. Supporting Statement for Paperwork Reduction Act Submission

B. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The study of for-profit PACE is designed to determine whether there are differences in access to and quality of care delivered to enrollees in for-profit PACE plans taking part in the for-profit demonstration compared to those in not-for-profit PACE plans. We will estimate differences in access and quality of care at these sites by com­paring the experience of for-profit PACE enrollees with that of not-for-profit enrollees. Because the only PACE plans participating in the for-profit demonstration are in Pennsylvania, we will compare the experiences of enrollees in these plans (four total sites operated by a common organization) to not- for-profit enrollees also in Pennsylvania.¹ Because of the small number of plans in the demonstration and their common ownership, this is essentially a case study of these plans, which limits the generalizability of the findings, but may provide useful insights into potential differences between for-profit and non-profit PACE plans.

The general strategy is to (1) identify all PACE plans in Pennsylvania, (2) identify the universe of enrollees in the for-profit and not-for-profit PACE plans, (3) select a sample of for-profit PACE enrollees from the 5 for-profit sites with data that will allow them to be matched to enrollees at the not-for-profit sites, (4) select not-for-profit plans operating in areas with similar population demographics to the five for-profit sites, and (5) use propensity score matching to form strata to select a comparison group of not-for-profit enrollees that distributionally match the characteristics of for-profit PACE enrollees in Pennsylvania.

a. Selection of For-Profit PACE Enrollees

The sampling frame for the for-profit PACE enrollees includes all individuals who are enrolled in for-profit PACE plans in Pennsylvania when the sample is drawn in 2012 and who are alive when the initial survey is administered. Because the target sample of for-profit enrollees is close to the total number of enrollees (reported later in Tables B.1 and B.2) in for-profit plans and there are no major systematic differences between for-profit and not-for-profit enrollees that would lead us to take a targeted sample of for-profit enrollees, we will draw a random sample from the for-profit frame.²

b. Comparison Group

Similar to the for-profit sample, the sampling frame for the comparison group consists of individuals who enrolled in not-for-profit PACE plans in Pennsylvania when the sample is drawn in 2012 and who are alive when the initial survey is administered. The comparison sample will be drawn from a subset of not-for-profit sites in Pennsylvania that have similar population demographics to the for-profit sites. We will use urban/rural status, the racial composition of the local population, and the percentage living in poverty to identify the not-for-profit sites to be included in the study. We will obtain the local population demographics data from the U.S. Census Bureau decennial census and the American Community Survey.

We will use the MARx database (Medicare Advantage Prescription Drug System) to identify all individuals enrolled in PACE plans in Pennsylvania. The number of enrollees in the state as of the fourth quarter of 2011 by for-profit versus not-for-profit plan status is shown in Table B.1.

Table B.1. Pace Enrollees by For-Profit Versus Not-For-Profit Status

	Total Enrollees
For-Profit PACE Enrollees	512
Not-For-Profit PACE Enrollees	2,428
Total	3,078

^a Enrolled in PACE in Pennsylvania and alive as of the 4th quarter of calendar year 2011.

c. Allocation of the Survey Sample

The sample design and allocation is based on the primary analytic objective: to compare access to and quality of care for enrollees in the for-profit PACE plans taking part in the for-profit demonstration to those in not-for-profit PACE plans in Pennsylvania. To make this comparison, the sample of for-profit PACE enrollees needs to match characteristics of the not-for-profit PACE enrollees. For the not-for-profit PACE comparison sample, the sample allocation will be based on strata formed on the basis of the propensity scoring of the for-profit PACE enrollees.

We will target 650 completed interviews, 325 for-profit enrollees and 325 not-for-profit enrollees. Assuming an 80 percent response rate, the initial sample size will be set at 813 enrollees (Table B.2). The universe of for-profit PACE enrollees is large enough (512 from Table B.1) to support the initial target sample of 406 enrollees. In addition, the number of comparison group members available (2,428 from Table B.1) substantially exceeds the sample of for-profit PACE enrollees.

Table B.2. Target PACE Samples

	Target Sample
For-Profit PACE	406
Not-For-Profit PACE	407
Total	813

To match the not-for-profit PACE sample to the for-profit PACE sample, we will distributionally match com­parison group members to be similar to the for-profit PACE enrollees in the sample. We do this matching by computing propensity scores of for-profit PACE enrollees and not-for-profit PACE enrollees using the following variables:

• Age

• Gender

• Medicaid enrollment

• Original reason for entitlement

• HCC score (or other available risk variables)

Using this information, we will compute probability of for-profit PACE entry for enrollees in for-profit plans and enrollees in not-for-profit plans using logistic regression. Next, we will order the for-profit PACE sample by the propensity score to identify levels of the scores to form the sampling strata for matching. For example, assuming all PACE enrollees have a propensity score between 0.95 and 0.50, four strata may be formed based on propensity score values: (1) 0.95 to 0.80, (2) 0.80 to 0.70 (3) 0.70 to 0.60, (4) 0.60 to 0.50.³ We will form 5 strata of not-for-profit enrollees based on their propensity scores with the fifth stratum including not-for-profit enrollees with scores less than 0.50 (these not-for-profit enrollees will be excluded from further sample selection). The assumption is that not-for-profit enrollees who have a score between 0.95 and 0.80 are essentially comparable (based on the model) to the PACE enrollees with a score in that same range. Therefore, a random sample of the not-for-profit enrollees in this propensity score stratum will be comparable to the for-profit sample from the same stratum. We will proportionally allocate the not-for-profit sample across the propensity score strata to match the distribution of the for-profit enrollees in these same strata. That is, if 50 percent of the for-profit enrollee sample is from the propensity score stratum of 0.80 to 0.70 than 50 percent of the not-for-profit enrollee sample will be allocated to this stratum. Ultimately, the not-for-profit enrollee sample will match, in distribution, across propensity score strata to the for-profit enrollee sample.

2. Procedures for the Collection of Information

In this section, we consider several issues relevant to the patient interviews: unusual operation issues and statistical precision and minimum detectable differences.

a. Unusual Operation Issues

We do not expect any unusual operation issues.

b. Statistical Precision and Minimum Detectable Differences

The study will assess differences in the for-profit and not-for-profit PACE plans in Pennsylvania using continuous and discrete outcome measures. In this section, we calculate statistical power for binary outcomes, for example, whether or not enrollees indicated high satisfaction with care. Table B.3 shows the detectable differences of a one-sample test on a binary response along various combinations of power and the design effect. Here, the one-sample test of a proportion compares the sample of not-for-profit enrollees against the population of for-profit enrollees; under the null hypothesis half of the for-profit enrollees in the population (p0=0.5) answer that they have high satisfaction with care. For a fixed level and power of the test and design effect under sampling, the detectable difference suggests the observed response from the 325 not-for-profit enrollees that would lead to rejection of the null hypothesis. For example, at the nominal 5% level with no design effect (=1.0), we would need an affirmative response by 57.7 percent (0.577) of the not-for-profit enrollees in order to correctly reject the null hypothesis with 80% power.

The table below illustrates power and detectable differences under sampling designs that have: no effect on the variance of the estimated proportion (design effect = 1.0); an effect that increases the variance by 50 percent (design effect = 1.5); and an effect that doubles the variance (design effect = 2.0). These hypothetical values for the design effect are considered reasonable under a clustered sampling design in which not-for-profit enrollees will be sampled at the eleven not-for-profit PACE sites. Additionally, while the intraclass correlation coefficient can induce large design effects, these can be captured in logistic regression models at the sacrifice of two or three degrees of freedom and thus trivial loss in power.

Table B.3. Design effects, Statistical Power, and Detectable Differences

Significance level*	Design effect	Statistical power (%)	Detectable difference
5%	1.0	60	0.061
		70	0.069
		80	0.077
	1.5	60	0.075
		70	0.084
		80	0.095
	2.0	60	0.087
		70	0.097
		80	0.109
10%	1.0	60	0.055
		70	0.060
		80	0.069
	1.5	60	0.064
		70	0.074
		80	0.084
	2.0	60	0.074
		70	0.085
		80	0.097

3. Methods to Maximize Response Rates and Deal with Issues of Nonresponse

Access to accurate and complete contact information for the sample is critical to achieving a high response rate. While we will use the MARx database to select the enrollee samples, we do not expect that it can provide reliable contact data. The addresses in the MARx database are typically where the explanation of benefits and reimbursements are sent and might not correspond to the actual residence of the sample member. Moreover, MARx data do not include telephone numbers. To ensure that we have updated, comprehensive contact information for the sample, we will use a multi-pronged approach that draws upon multiple sources of contact data.

Because the PACE sites are likely to have the most accurate contact information, we will first work with the PACE sites to collect contact data for the enrollees and their caregivers. This data will help us identify the caregiver who is most involved with the healthcare of the sample member and therefore the most reliable source of information about the sample member. We anticipate that the PACE sites will cooperate with this effort as the National Pace Association has indicated they will help to encourage the PACE sites to provide this information.

For the sample members for whom we are unable to obtain contact data from the PACE sites, the locating specialists at Mathematica will work with a national telephone and address look-up company and other online databases to conduct multiple rounds of searches for contact data.

Advance letters will also support the locating process. Prior to be start of data collection, Mathematica will mail advance letters on CMS letterhead and information brochures to sample members with confirmed addresses. The letter and brochure will emphasize the importance of their feedback on the PACE program and encourage participation from the sample member (Appendix A). We will ask for ADDRESS SERVICE REQUESTED to obtain up-to-date addresses for those who may have moved. Letters returned with no additional postal information will be sent directly to Mathematica’s locating experts for additional electronic and telephone locating. Mathematica will maintain locating efforts and monitor locating success throughout the survey field period.

To increase participation from a sample that is likely to suffer from physical, emotional, and cognitive impairments that can limit survey response, the interviewer staff, composed of experienced telephone interviewers, will be trained to use procedures developed specifically to interview elderly and functionally impaired individuals. In particular, we will familiarize the interviewers with the range of physical, emotional, and cognitive impairments they might encounter, discuss how these impairments can affect the interview process, and explain what they as interviewers can do to deal with these problems. For example, interviewers might have to adjust the pace of the interview to accommodate the respondent’s health and functional status. Interviewers will receive training on strategies or approaches to encourage individuals to participate and begin the survey as well as ways to support the completion of the survey. Interviewers will be trained to recognize when fatigue affects responses and to apply techniques to handle this situation, such as offering respondents the option to take a break or schedule another time to continue the survey. Finally, with proxy respondents expected to account for a considerable proportion of the survey respondents, interviewers will be trained on how to recognize situations in which the sample member is cognitively unable to participate meaningfully in the survey, identify the appropriate proxy respondent, and gain the proxy respondent’s cooperation.

We will meet the needs of Spanish-speaking sample members by using bilingual interviewers to administer a Spanish translation of the survey. We will train interviewers to offer Spanish speakers the option to speak with a Spanish interviewer and to flag these cases so they will be routed to a Spanish interviewer. Our Spanish interviewers are trained to build rapport and address fears stemming from participation.

4. Test of Procedures or Methods to be Undertaken

The survey developed for this study draws heavily upon the survey instrument that was designed and administered in 2005 for the evaluation of the not-for-profit PACE programs (Appendix B). As the original survey was administered to a comparison group of non-PACE enrollees, many of the questions were more general in nature. We modified several of these questions to refer specifically to the PACE programs. To minimize respondent burden, we removed questions about the source and use of caregivers that were not relevant to the research questions of this study. We added several questions, adapted from the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Adult Commercial Questionnaire and Supplemental Items, to measure satisfaction with other services, including use of rehabilitative care, care delivered by specialist doctors, and transportation services. Prior to data collection, the CATI instruments will be tested by project staff to ensure accuracy of flow, consistency checking, and ranges.

5. Individuals Consulted on Statistical Aspects of Design and Individuals Collecting and/or Analyzing Data

The following staff worked on the statistical aspects of the sample design:

• David Jones, Ph.D., Mathematica Policy Research, Inc., (617) 674-8531

• Robert Schmitz, Ph.D., Mathematica Policy Research, Inc., (617) 301-8976

• Frank Yoon, Ph.D., Mathematica Policy Research, Inc., (212) 554-7518

• Barbara Carlson, Mathematica Policy Research, Inc., (617) 674-8372

• Frank Potter, Ph.D., Mathematica Policy Research, Inc., (293) 558-5956

• Valerie Cheh, Ph.D., Mathematica Policy Research, Inc., (609) 275-2385

The following staff will oversee the data collection:

• Nancy Duda, Ph.D., Mathematica Policy Research, Inc., (609) 945-3340

The following staff will oversee the analysis of the data:

• David Jones, Ph.D., Mathematica Policy Research, Inc., (617) 674-8351

www.mathematica-mpr.com

Improving public well-being by conducting high-quality, objective research and surveys

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Washington, DC

Mathematica^® is a registered trademark of Mathematica Policy Research

1 A fifth for-profit site, Life at Home, terminated on May 1, 2012.

2 An example of a systematic difference that would lead us to take a targeted sample of for-profit enrollees would be if the comparison group was comprised of fee-for-service Medicare beneficiaries that could be systematically different from PACE enrollees. Because both groups are comprised of PACE enrollees, such a targeted sampling of the for-profit enrollees is not required. Other differences between the two groups, such as age, gender, and basic health status, will be accounted for in the matching process described in the following sections.

3 Distributional matching is preferred to nearest neighbor matching in this case since we are not certain which sample members will respond to the survey, i.e., individuals in matched pairs under the nearest neighbor method might not be part of the final sample.