Pcmch_ssb_02122013

Supporting Statement B for
Evaluation of the Multi-Payer Advanced Primary Care Practice Demonstration:
Conduct Beneficiary Experience with Care Surveys

1

B. Statistical Methods (used for collection of information employing statistical methods)

B1. Respondent Universe and Sampling Methods
The target population for the survey is Medicare fee-for-service (FFS) and Medicaid
beneficiaries who were assigned to primary care practices participating in the Multi-Payer
Advanced Primary Care Practice (MAPCP) Demonstration. Assigned beneficiaries are those
who received a plurality of their care from the participating Patient-Centered Medical Homes
(PCMHs). These medical homes are located in eight states (Maine, Michigan, Minnesota, New
York, North Carolina, Pennsylvania, Rhode Island, and Vermont). As of December 2012, more
than 400,000 beneficiaries from 804 medical practices were participating in the MAPCP
Demonstration.
Representative samples will be drawn by simple random sampling from lists of all beneficiaries
assigned to demonstration medical homes in a state. A total of 24 separate samples will be
drawn (eight regions times the three insurance coverage types (Medicare only, Medicaid only,
and dual eligibles). New York State will not be included because a similar survey is being
planned at the same time in that state, but surveys will be done for two separate geographic areas
in Pennsylvania.
Exhibit B-1 details expected response rates to each survey mailing. In each state, we plan to
sample 1,463 beneficiaries to obtain the desired 512 completed surveys. We estimate an overall
35% response rate based on our previous experience administering CAHPS surveys.

Exhibit B-1. Planned Sample Sizes Per Coverage Group in Each State and Expected
Response Rate

Mail Survey Response Stages
Survey Mailing #1

Sample
Size
1,463

Response Rate

Estimated Respondents to 1st Mailing

351

24.0%

Survey Mailing #2

1,112

Estimated Respondents to 2nd Mailing

161

14.5%

Total Completed Surveys

512

35.0%

Exhibit B-2 displays the total number of Medicare and Medicaid beneficiaries we propose to
sample and the expected number of completed surveys within each coverage group in each state.
We assume that 20% of Medicare beneficiaries will be dual enrollees and that these beneficiaries
will do “double duty” for the Medicare/Medicaid duals sample, so we do not need to draw as
large a sample as we do for the Medicare and Medicaid-only samples. We estimate that the
Medicare sample will yield 410 completed surveys from beneficiaries who are Medicare only
2

and an additional 102 completed surveys from beneficiaries with dual status. These additional
surveys will be combined with other dual eligibles to bring the total in the dual status group up to
512. Thus, we will have 1,434 completed surveys per state. Across all eight regions, this requires
an initial sampling of 32,777 beneficiaries to obtain 11,472 completed surveys.

Exhibit B-2. Sampling Plan for MAPCP Beneficiary PCMH-CAHPS Surveys
Initial Number of
Beneficiaries Sampled per
State

Number of Completed
Surveys per State

Medicare (Duals & Non-duals)

1,463

512

Dual Eligibles

1,171

410

Medicaid Only

1,463

512

Total

4,097

1,434

Sample

B2. Procedures for the Collection of Information
Survey Materials. Patient experience will be measured using the 12-month version of the
Patient-Centered Medical Home version of the Consumer Assessment of Healthcare Providers
and Systems (PCMH-CAHPS). The PCMH-CAHPS is a validated, federally developed
instrument that measures patient experience in six domains (access to care, provider
communication, office staff interactions, attention to medical/emotional health, health care
support, and medication decisions). The survey contains 52 close-ended items. We will distribute
the English language version of the survey, although we will make Spanish language versions of
the survey available upon request. The survey will be in scannable form, allowing for easy data
capture of returned surveys. We will develop separate cover letters for each mailing. The first
letter will contain all required elements of informed consent and a toll-free telephone number
that subjects can call if they have questions. The cover letters will be printed on CMS letterhead
and signed by an appropriate official to enhance the survey’s legitimacy.
Survey Schedule. The survey data collection process will consist of an initial mailing to
beneficiaries, followed two weeks later by a second mailing to all nonrespondents. The data
collection period will end approximately 4 weeks after the second mailing.
Power Analysis. Mean scores for key PCMH-CAHPS composites will be contrasted with
medical practice means from the national CAHPS database. The initial target sample size was
441 completed surveys per insurance coverage group to allow us to detect 8% differences from
national criterion values in one-sample statistical tests. This sample size was calculated for a
difference of 63% vs. 55% (the most conservative of the national CAHPS results), assuming
two-sided tests at power = 0.80 for a one-sample t-test with alpha=0.01 to account for multiple
tests. We assume that an absolute difference of 8% or more represents a substantively important
deviation from the average for any PCMH-CAHPS composite.
The sample size target must also account for the fact that beneficiaries are nested within medical
practices. The target was adjusted for potential patient clustering using an intra-class correlation
3

of .02 for the CAHPS Access to Care composite reported by Damman et al. (2009). Assuming an
average of 50 demonstration practices per state, the design effect due to clustering is 1.16. This
increases the desired number of completed surveys to 441*1.16 = 512 per group.
Quality Control. RTI will implement quality control procedures throughout the mail survey
period. Our data preparation staff will match personalized cover letters with surveys using a
unique identification number and will check a portion of all outgoing mail packages to make sure
that the packages contain all required materials (i.e., cover letter, survey, business reply
envelope). Responses for each sampled beneficiary will be tracked and assigned a unique
disposition code in order to compute survey response rates. Our data receipt staff conducts a
manual review of each returned survey to locate any written comments or enclosed materials,
which will be referred to project staff for review. It is through this manual review that we may
learn of sample members who are deceased, physically incapable, or ineligible. At different
points during the data collection period, project staff will also pull a sample of hard-copy surveys
and compare the responses to the scanned data to ensure that data are being accurately captured.

B3. Methods to Maximize Response Rates and Deal with Nonresponse
Mail Survey Practices. A number of “best mail survey practices” have consistently been
shown to be associated with survey response rates (Herberlein and Baumgartner, 1978). These
practices are:
• pre-notification letters
• number of follow-up mailings
• survey sponsorship
• saliency of the survey topics to the target population
• personalization of correspondence
• postage-paid return envelopes.
We have incorporated all of these elements in our protocol except for pre-notification letters,
which are not part of the recommended CAHPS protocol. Response may also be enhanced by
permitting proxies to assist beneficiaries to complete surveys.
Overlap with State Survey Efforts. Some of the MAPCP states are conducting their own
surveys using the PCMH or the Clinician and Group versions of CAHPS. Overlapping survey
periods have the potential to increase respondent burden since the sample frames are likely to be
very similar and the same beneficiaries might be asked to complete the survey twice in the same
year. We will monitor state survey activities to identify any surveys that may be planned for the
same time period as our survey. In cases of overlap, we will collaborate with the state to avoid
duplication. Through our ongoing communication with the MAPCP states, we have already
identified one instance in New York in which the state is planning to administer the CAHPS
survey. After several conversations, we agreed to use their survey data instead of administering
our own separate survey so as to avoid duplication and minimize respondent burden.
Address Changes. Medicare beneficiary addresses will be supplied by CMS and Medicaid
beneficiary addresses will be supplied by state Medicaid staff. The National Change of Address
(NCOA) file will be used to ensure that we have accurate address information prior to the initial
mailing.
4

Nonresponse Weighting. We will analyze the probability that each eligible, sampled
beneficiary completed the survey. This response propensity model will be a logistic regression
model in which the outcome is coded 1 if the sampled beneficiary completed survey and 0 if the
survey was not returned or completed. The explanatory variables will consist of factors that are
available for all MAPCP Demonstration beneficiaries, including Hierarchical Condition Code
risk scores (a measure of expected expenditures based on diagnoses), Charlson co-morbidity
scores, disability status, and demographic characteristics. Separate models will be estimated for
each insurance coverage group in each state. The inverse of the predicted response probabilities
will be used as survey weights.

B4. Test of Procedures or Methods to be Undertaken
The PCMH-CAHPS was developed through a multi-year collaborative effort by several federal
agencies. The survey items have been carefully pre-tested and validated among multiple racial
and ethnic groups. A recent field test (Scholle et al., 2012) supports the reliability and validity of
the instrument. More than 1,000 organizations administer various versions of the CAHPS each
year. As a result, we do not plan to engage in any further pre-testing for the purposes of this
evaluation.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or
Analyzing Data
Kevin W. Smith, M.A. (Survey Methodologist)
RTI International
Telephone: 781-434-1748
Vanessa Thornburg, M.A. (Research Survey Manager)
RTI International
Telephone: 919-541-6880
Vincent Iannachionne, Ph.D. (Statistician)
RTI International
Telephone: 202-728-1960

REFERENCES
Damman OC, Stubbe JH, Hendriks M, et al. Using multilevel modeling to assess casemix adjusters in consumer experience surveys in health care. Medical Care 2009; 47: 496-503.
Herberlein TA, Baumgartner R. Factors affecting response rates to mailed surveys: A
quantitative analysis of the published literature. American Sociological Review 1978; 43:447462.
Scholle SH, Vuong O, Ding L, Fry S, Gallagher P, Brown JA, Hays RD, Cleary PD.
Development of and field test results for the CAPHS PCMH survey. Medical Care 2012; 50: S2S10).
5