CMS-10097.OMB Supporting Statement Part B

Supporting Statement

Request for Clearance
For
Medicare Contractor Provider Satisfaction Survey
(MCPSS)
National Implementation
Part B

January 26, 2007

TABLE OF CONTENTS
PAGE

C.

Collection of Information Employing Statistical Methods...........................................1

C-1

Potential Respondent Universe........................................................................................................................ 1

C-2

Procedures for Collecting Information.............................................................................................................. 1

C-2.1

Study Sample ................................................................................................................................................ 1

C-2.2

Survey Materials........................................................................................................................................... 4

C-2.3

Data Collection ............................................................................................................................................. 5

C-2.4

Processing Returned Surveys ....................................................................................................................... 7

C-2.5

Calculating Satisfaction Scores .................................................................................................................... 7

C-2.6

Contractor Reports........................................................................................................................................ 8

C-3.

Methods to Maximize Response Rates and Deal with Nonresponse ............................................................... 9

C-3.1

Promoting the Survey Project to Increase Saliency.................................................................................... 10

C-3.2

Follow-up with Non-respondents............................................................................................................... 11

C-3.3

Non-response bias analysis ........................................................................................................................ 11

C-3.4

Non-response adjustment ........................................................................................................................... 12

C-4. Tests of Procedures and Methods ........................................................................................................................... 13
C-5.

Individuals Consulted...................................................................................................................................... 13

TABLES AND FIGURES
Table 4

Medicare Provider Sample for National Implementation ......................................1

Figure 1 Power by Sample Size ..............................................................................................3
Figure 2 MCPSS Data Collection Scheme .............................................................................6

ATTACHMENTS
Attachment 1 National Implementation Sample Design
Attachment 2 National Implementation Survey Instrument
Attachment 3 “Redline” version of the 2008 Survey Instrument
Attachment 4 Sample Cognitive Interview Protocol

i

C.

Collection of Information Employing Statistical Methods

C-1 Potential Respondent Universe
The target population for the Survey consists of all Medicare providers served by Medicare
Contractors across the country; CMS will select a sample designed to yield no more than 24,239
completed surveys from providers. The sample of providers will be selected, as shown in Table 4,
from 21 Fiscal Intermediaries Contractors, 17 Medicare Carriers, one Part A and B Medicare
Administrative Contractor (MAC), four Regional Home Health Intermediaries (RHHIs) and four
Durable Medical Equipment Administrative Contractors (DME MACs).
Table 4 Medicare Provider Sample for National Implementation
Provider Types

Sample Size

Hospitals

1,842

Skilled Nursing Facility

3,235

Other Part A providers

3,507

Home Health Agencies

1,541

Hospice facilities

902

Physicians

5,510

Licensed practitioners

3,502

Other Part B providers

1,786

DME suppliers*

2,414

Total

24,239

* DME Suppliers includes physicians who submitted claims for durable medical equipment or supplies.

C-2 Procedures for Collecting Information
C-2.1 Study Sample
The target population for the MCPSS survey consists of all Medicare providers served by all
Medicare Contractors in the nation. These Contractors are currently comprised of 21 Fiscal
Intermediaries Contractors, 17 Medicare Carriers, one Part A and B Medicare Administrative
Contractor (MAC), four Regional Home Health Intermediaries (RHHIs) and four Durable Medical
Equipment Administrative Contractors (DME MACs). The Contractors with multiple service areas
are considered as a single Contractor. With changes in the contracting environment we expect to see

1

fluctuations in the number Contractors from one year to the next.
To meet CMS’ objective of making valid comparisons between Contractors, the sample has
been designed to obtain an equal number of completed questionnaires from each Contractor. We
select a sample to yield 400 completed interviews for each Contractor. For those Contractors with a
provider population size 400 or smaller, all the providers will be selected with certainty. Table 1-1 in
Attachment 1 shows the provider population size for each provider type within each Contractor. The
maximum percent error for estimates of percentages obtained from a simple random sample yielding
400 completed questionnaires will not exceed 5 percent 95 percent of the time. For example, suppose
50 percent of providers responded as satisfied with the service they received. We can be 95 percent
confident that between 45 percent and 55 percent of the providers are satisfied with the service. The
percent error is the largest for the 50 percent proportion and decreases as proportion moves further
away from the 50 percent / 50 percent split. For example, for an 80 percent / 20 percent split, the
error is 4 percent. Thus, 400 completed questionnaires should provide adequate precision for
Contractor-level estimates. Note that several Contractors have multiple service areas. The precision is
provided here for the Contractor-level estimates. The precision of estimates can be much lower for
the service areas within the Contractors.
We considered samples sizes of smaller than 400. The sample sizes smaller than 400 will not
only provide smaller precision, they will also require more oversampling for smaller provider types.
For example, a sample size of 300 will provide an error not exceeding 5.8 percent, which is not
substantially higher than 5 percent, however, it will require more extensive and higher oversampling
rates in smaller provider types. This oversampling can further reduce the precision of the Contractor
level estimates.
The sample size of 400 is allocated proportionately to states and provider types within each
Contractor. In Contractors with multiple service areas, the providers will be first stratified by service
area and within service area by provider type. The proportional allocation provides a representative
sample of providers for Contractors across the service areas and provider types and minimizes the
variance of the Contractor-level estimates. The numbers under the heading “Base sample” in Table 11 in Attachment 1 show the proportionately allocated sample size for each provider type within each
Contractor.
The proportional allocation could result in small sample sizes in several relatively smaller
provider types and states. We oversample these states and provider types to yield a minimum of 30
completed questionnaires. In Attachment 1, the additional number of providers needed is shown

2

under the column with a heading “Oversample.” Thirty responses are adequate to conduct statistical
tests to detect valid differences between provider types within or across the Contractors, or within or
across states.
The satisfaction score has six distinct intervals. The power of a statistical test indicates the
probability of rejecting the null hypothesis in error. If the power is inadequate, we cannot draw
conclusions from the test with confidence. Sample size affects the power of a statistical test. For
example, we could conclude that there is no difference between the scores of two provider types
using small samples when, in fact, the samples are too small to detect the true difference. Assuming a
standard deviation of 1.35 for the satisfaction score within each provider type, 30 completed
questionnaires for each provider type will provide more than 80 percent power (when significance
level is 0.05) to detect a mean satisfaction score difference of 1 between the two provider types.
Figure 1 shows the power function against various sample sizes per provider type with a standard
deviation of 1.35 and a mean score difference of 1 (with equal sample sizes between providers).
Figure 1

Power by Sample Size
Power by Sample Size
1

Power

0.8
0.6
0.4
0.2
0
0

10

20

30

40

50

60

70

80

Sample Size
Pow er function

The target overall response rate for the national survey is 80 percent. The desired precision
level by provider types within Contractors is achieved by 24,239 completed questionnaires. Applying
the estimated response rate of 80 percent and 85 percent eligibility rate, we would need to contact
35,646 (that is, 24,239/ (0.80*0.85)) providers to achieve the desired number of completes. See Table
1-1 in Attachment 1. If the response rate is expected to drop below the OMB target of 80 percent,
additional sample will be released in order to obtain the desired number of completed surveys
(24,239).
3

C-2.2 Survey Materials
Survey materials will follow the same design and format as those used in the first (2006) and
second year (2007) of the national implementation. These include:
The Questionnaire:
The questionnaire includes seven topic areas: provider inquiries, provider outreach &
education, claims processing, appeals, provider enrollment, medical review, and provider audit &
reimbursement. Some of these topics do not pertain to some Contractors and their respective
providers. For example, provider enrollment, medical review, and provider audit & reimbursement do
not apply to DME suppliers and the DME MACs that serve them. Similarly, the topic of provider
audit & reimbursement does not apply to carriers and the providers who work with them. CMS
customizes the questionnaire, so providers receive a questionnaire with topics that are relevant to
their interaction with the Contractor.
Please see Attachment 2 for a copy of the proposed 2008 MCPSS survey instrument.
CMS is committed to improving the survey with each round of data collection and have set
aside dedicated resources to refine the survey. Given the changing contracting environment it is
important to include a core set of measures for trending purposes, but at the same time it is important
to collect data on new and topical initiatives. CMS will therefore be collecting relevant measurement
information from CMS staff and Contractors on a continuous basis.
CMS conducted psychometric and factor analysis using the 2006 data. In addition, CMS
spoke with key area experts in CMS and received unsolicited feedback from the Contractors about
the 2006 survey. As a result of these analyses and discussions, it was determined that some items
were not within the purview of the Contractors, and therefore should be removed from the
questionnaire. At the same time, it was determined that new items would help CMS better understand
the Contractor-provider interaction, and also help CMS understand the impact of some of its own
policy decisions. For this reason, items were either dropped or added from the last OMB approved
version. Attachment 3 includes a “redline” version that shows the changes from the last approved
version. Researchers have taken care to maintain a core set of items that will remain static over time,
to ensure CMS’ ability to monitor trends.
Since continuous improvement of MCPSS is CMS’ goal, it also conducted a small number (7)
of interviews with providers to determine whether the questions were relevant and clear. As a result
of these interviews, CMS revised some of the wording to make the intent of the questions clearer to
4

respondents. CMS intends to complete similar cognitive interviews in 2007 (especially if new content
is determined important for the coming years).
Web Survey: CMS uses the Web as the primary mode of data collection for the MCPSS.
However, to ensure that respondents have the flexibility to respond in the mode that best meets their
needs, CMS also maintains the survey in a paper format, as well as in an interviewer-administered
format. The Web survey includes easy-to-understand instructions and user-friendly navigation features.
The Web survey includes all the instructions included in the paper questionnaire. During past
meetings with providers and provider organization representatives, it was communicated to CMS that
they generally preferred surveys that were available for completion on-line.
As mentioned earlier, CMS has conducted usability testing to improve the functionality and
usability of the Web survey, and we believe no further usability testing is required at this time.
Cover letters: The survey notification package includes two cover letters, one on CMS
letterhead and another from the relevant Contractor. The letters explain the purpose of the study, the
need for the data, a confidentiality clause, and the unique Provider ID and password to access the
Web survey, as well as contact information for questions or to request assistance or a paper
questionnaire (e.g., a toll free phone number, a fax number and an e-mail address).
Web Instructions: A separate flyer is included that provides the study Web site, general
instructions for logging onto the Web site, and the MCPSS toll-free help line.

C-2.3 Data Collection
The data collection steps are as follows:


Screener call to determine most knowledgeable respondent (MKR);



Mail survey notification package (to the address identified during the sample
cleaning/screening process);



10 days after initial mail, send a reminder/thank-you postcard;



Start non-response follow-up (by telephone) 10 days after reminder/thank-you
postcard card; and



Close data collection 16 weeks after initial mailing.

In Figure 2 below, we provide the flow for the current MCPSS data collection scheme (as
each administration of the MCPSS closes, and CMS assesses the “lessons learned,” this scheme is
5

fine-tuned to best meet the needs of this respondent population).
Figure 2

MCPSS Data Collection Scheme
Screen for Most Knowledgeable Respondent
- Locate the MKR
- Ask MKR about facilities handled
- Ask MKR for mailing info
- Ask MKR about Internet access
Does the MKR have Internet Access?
YES

NO
Continue with Telephone
interview/or mail paper survey
(if requested)

Inform MKR of Mailing
Mail survey instructions to MKRs
January 2007

COMPLETE?
YES

NO

Reminder/Thank you postcard
Nonresponse followup by phone

Mail Thank you letter

Nonresponse followup by phone

Close of Data collection
Determined by observed response rate
Latest close by end of April

Providers will be encouraged to complete the survey over the secure Web site. The cover
letter will clearly state options to access the Web site and complete the survey on-line, or the
respondent can print a copy of the questionnaire from the Web site and return it by mail or FAX (so
respondents are able to respond using their preferred delivery method). All providers will be given the
option to request a paper copy of the questionnaire (rather than downloading it from the Web site)
and then submitting their responses via mail or FAX.
The strategy of using the Web as the main mode of data collection worked well during the
first national administration. Telephone contact was, and will continue to be, the primary mode for
following up with non-responders.
The following media have been set up to allow respondents to communicate with CMS during
data collection:


Toll-free Phone: The survey vendor maintains a toll-free telephone number to receive
calls from respondents concerning any issues they have regarding the survey.

6



E-Mail Box: The survey vendor maintains a study e-mail box. This has been a popular
feature and can facilitate communication regarding alternative ways respondents want to
submit survey responses.



FAX Number: A FAX number is available for respondents who wish to respond via this
method. The FAX machine, to which inquiries or responses are sent, is located in a secure
location and only authorized project staff can retrieve these documents.

C-2.4 Processing Returned Surveys
There are three criteria that are used for processing returned surveys:


The submission must contain the pre-coded identification number.



All applicable sections should be completed.



A survey is currently considered a complete if at least one item is completed in the
Claims Processing section, and at least one item in any other survey section is
completed.

C-2.5 Calculating Satisfaction Scores
In order to provide CMS and the Contractors with summary scores with which to monitor
trends and compare success across Contractors, a scoring methodology was developed that allows us
to calculate respondent level scores for Contractors, provider types and each section. Below is an
explanation of how the scores are calculated:
Contractor Score:
The weighted1 sum of ratings for all questions for all business functions across all provider
types related to each Contractor divided by the total number of respondents answering the
questions across all business functions for all provider types related to each Contractor
Business Function Score at the Contractor Level:
The weighted sum of ratings for all questions for a business function across all provider types
related to each Contractor divided by the total number of respondents answering the
questions for that business function related to each Contractor
Provider Score for Each Provider Type under Each Contractor:
The weighted sum of ratings for all questions for all business functions related to a provider
type divided by the total number of respondents answering the questions for all business
functions related to that provider type
1 Because not all providers will be selected for the survey and not all selected providers responded, a sample weight will
be calculated for each responding provider.
7

Business Function Score at the Provider Level:
The weighted sum of ratings for all questions for a business function related to a provider
type divided by the total number of respondents answering the questions for that business
function related to that provider type
Provider Score for Each Provider Type under Each Contractor within a State:
The weighted sum of ratings for all questions for all business functions related to a provider
type divided by the total number of respondents answering the questions for all business
functions related to that provider type within a specified State
Provider Score for Each Provider Type under Each Contractor within a CMS Jurisdiction:
The weighted sum of ratings for all questions for all business functions related to a provider
type divided by the total number of respondents answering the questions for all business
functions related to that provider type within a specified CMS regional Jurisdiction

C-2.6 Contractor Reports
The Contractors have been pleased with the content and level of detail provided in the
final Contractor reports. Contractors have indicated that the reports, particularly the item level
results, are useful to identifying the services that need improvement. Several Contractors have
also stated that the satisfaction scores confirmed what they already thought and/or knew to be
problem service areas. In addition, Contractors have agreed that the timeframe for receiving these
documents (i.e., July) was good because it helped them prepare for the next fiscal year.
The results from the national implementation are available to all Contractors via an
interactive Web based system. Contractors can access the following information via the on-line
reports:


Their scores at the Contractor level, provider level and business function level; as well
as these levels crossed by State or Jurisdiction



Item level weighted frequencies



Verbatim and coded comments; these comments will be sanitized and will not have any
identifiers.

To help identify problem spots, Contractors can view both scores and frequencies by the
following parameters:
8



By state;



By state, by urbanicity (i.e., urban, rural);



By state by provider type;



By state by urbanicity by provider type; and



By provider size.

The summary scores, at all levels, include cell sizes and standard errors. Since providers
may have answered some but not all of the sections or only some of the questions for a particular
section, the cell size for calculating the scores can vary across sections of the survey. A cell size is
presented with each score so Contractors know how many providers responded to that section;
this provides an indication of the stability of the score. If only a few providers answered the
question, then the survey estimate could fluctuate considerably if we happened to survey a
different set of providers. The larger the number of providers who respond to an item, the more
confident we are that the survey estimate is close to the “true” answer we would find had we not
selected a sample, but instead surveyed all providers. The standard errors are intended to help the
Contractor determine how close the Contractor score is to the average Contractor score. If too
few providers answered any given survey section, then the results are suppressed to reduce the
chance of a Contractor identifying a specific provider. The reports will also include information on
key drivers of satisfaction. This information will help Contractors determine which areas within
each business function are key drivers of satisfaction with that business function. They will also
have information on which business functions are key drivers of overall satisfaction. This
information can help Contractors focus their performance improvement efforts.

C-3.

Methods to Maximize Response Rates and Deal with Nonresponse

CMS has explored many issues related to increasing the saliency of the study among the
provider community and using non-response follow-up strategies to maximize response rates.
The target response rate for the national implementation is 80 percent. As a result of efforts to
improve locatability the unweighted response for the 2006 MCPSS was 64.8 percent. Further
improvements were implemented in 2007, including:


use of a new data source for obtaining improved contact information was found;



better screening techniques to ensure we have reached the correct respondent before
mailing the introductory packet;
9



better identification of “duplicate2” sample up-front to reduce respondent frustration;



continued use of the claims history file to only select “active” providers (those
submitting a claim in the past 12 month period); and



an aggressive plan for outreach and dissemination.

However, if the response rate were to fall below 80 percent CMS and their survey vendor will
explore the option of conducting a non-response bias analysis. Please see C-3.3 for a detailed
description of the proposed non-response bias analysis.
C-3.1 Promoting the Survey Project to Increase Saliency
CMS is taking an aggressive approach to achieving the response rate goal of 80 percent. In
addition to obtaining a clean sample, it is essential to create awareness and understanding of the value
and importance of the survey within provider and supplier communities in order to motivate
participation in the survey. In the end, we want providers and suppliers to view the MCPSS as a tool
that will assist CMS and Contractors in identifying and implementing service improvements.
To achieve high saliency for the study, the level of activity between October 2007 and January
2008 will be high. We also propose aggressive outreach campaign between January and March 2008
as well as follow-up activity when results are available in June 2008.
The overall objective of this plan is to create awareness for the Medicare Contractor-Provider
Satisfaction Survey (MCPSS) among financial and business managers employed by Medicare
providers and fee-for-service Contractors. CMS will implement a public relations campaign to
generate broad coverage of the MCPSS initiative through a variety of channels:





The healthcare trade media serving financial and business managers employed by
Medicare providers and fee-for-service Contractors. This includes members of the
print and Web-based media.
Contractor-based communications channels such as list-serves, conferences and
meetings, newsletters, etc.
Professional organizations that serve the provider community
CMS based channels of communications to both the providers and Contractors.

2 The unit of analysis is an individual who submits claims for a health care provider or a group of providers. If more
than one provider is selected for this individual, then they may have “duplicate” records in the sample (since we do not
know, until screening, who the most knowledgeable respondent will be for a given health care provider).
10

C-3.2 Follow-up with Non-respondents
CMS uses, and will continue to use, telephone as the main mode of follow-up with
nonrespondents.
C-3.3 Non-response bias analysis
If response rates fall below 80%, CMS will conduct a nonresponse bias analysis. The purpose
of this analysis is to determine if the non-respondents are significantly different from the respondents.
This will include an analysis of sample frame variables including Contractor, provider type, number of
claims, dollar value of claims, size of facility (bed size and or number of patient days), specialty type
(in the case of physicians, licensed practitioners, and medical equipment providers), ownership type
(for Hospitals and skilled nursing homes). Do note that based on the 2006 response rates of 64.8
percent, CMS had submitted to OMB a non-response bias analysis.
In the event that the response rate falls below 60 percent, CMS will create a sub-sample of
non-respondents to conduct a more detailed non-response bias study. The sub-sample will include
those who refused and facilities that were contacted. Assuming a 60% response (40% non-response),
from among the non-respondents, we will draw a sample to yield 450 follow-up respondents. This
will provide more than 80 percent power to detect mean satisfaction score differences less than 0.3
between the follow-up respondents and respondents to the regular interview. (That is, testing the
difference between the mean scores of 450 follow-up (non) respondents and 15,000 main interview
respondents).
This study will include a follow-up survey to the sub-sample. The follow-up survey will
include only the claims processing section and the overall satisfaction question. We will then compare
the satisfaction scores of the respondents and non-respondents, by Contractor type (FI, Carrier, A/B
MAC, DMERC, DME MAC, RHHI) to determine if there is a significant difference. If significant
differences are found, estimates can be adjusted for nonresponse bias through weighting. This followup survey will be kept to about 6-7 minutes. This follow-up will also include a question on why the
respondent initially refused or did not respond.
The follow-up will be by mail and telephone. The protocol will be as follows:
 First mailing questionnaire, with a revised cover letter from CMS, and Contractors.


One week later-a reminder/thank-you postcard



One week later, a second questionnaire



One week later-telephone interviews, with up to 9 additional callbacks

11

C-3.4 Non-response adjustment
In spite of the best practices, virtually all surveys experience nonresponse. The target response
rate for this survey is 80 percent. This will most likely vary by provider type and by other provider
characteristics.
One consequence of nonresponse is the potential for bias in the survey estimates, making them
larger or smaller than the true statistic for all providers. The extent to which those that do reply differ
in their satisfaction from those that do not reply affects the extent of bias. When response rates vary
among subgroups, such as provider types, as they are likely to do, there is an even greater potential
for bias in survey estimates.
We will adjust the sampling weights to remove potential bias on satisfaction (and on any other
substantive estimates to be produced from the survey) caused by not obtaining responses from all
sampled providers. If response propensity is independent of the satisfaction, then no bias would arise.
Therefore, the objective is, using the known characteristics of the sampled providers, to form
nonresponse adjustment cells so that the response propensity within each cell is independent of
satisfaction. To the extent that this was achieved, the estimates of satisfaction obtained using the
sampling weights that are adjusted for nonresponse within these cells, will have smaller potential bias.
There are several alternative methods of forming the cells to achieve this result. In forming the cells,
we will attempt to minimize the variation in response propensity within the cells.
We plan to use Chi-Square Automatic Interaction Detector (CHAID) software to guide us in
forming the cells. CHAID uses an AID type of algorithm. CHAID partitions data into homogenous
subsets with respect to response propensity. To accomplish this, it first merges values of the
predictors, which are statistically homogeneous with respect to response propensity and maintains all
other heterogeneous values. It then selects the most significant predictor (with the smallest p-value)
as the best predictor of response propensity and thus forms the first branch in the decision tree. It
continues applying the same process within the subgroups (nodes) defined by the "best" predictor
chosen in the preceding step. This process continues until no significant predictor is found or a
specified (about 20) minimum node size is reached. The procedure is stepwise and creates a
hierarchical tree-like structure.
The data on the relevant characteristics of the providers will be available from the sampling
frames for both respondents and nonrespondents. These characteristics include provider type, number
of claims (both volume and dollar value) and MSA/nonMSA status for all providers, number of beds
12

for hospitals and skilled nursing facilities, total patient days for hospitals, ownership type of the
facility, physician/non-physician specialty and age, and specialty for DMACs.
Although nonresponse adjustment should reduce bias, it can also increase the variance of
estimates. Small adjustment classes and/or low response rates (or large nonresponse adjustment
factors) may increase the variance substantially and give rise to unstable estimates. In order to prevent
an excessive increase in variance and thereby an adverse effect on the mean square error of the
estimates, we will limit the size of the classes to a minimum and avoid large adjustment factors.
In June 2008, CMS will provide OMB a supplement with the non-response adjustment
methods used in the 2008 survey.
C-4. Tests of Procedures and Methods
CMS will not test any data collection procedures during the national Implementation.

C-5.

Individuals Consulted

Organization Name

Contact Information

CMS

David C. Clark

410.786.6843/ [email protected]

Alan Constantian

206.615.2306/[email protected]

Dr. Elizabeth
Goldstein

410.786.6665/ [email protected]

Mel Ingber

410.786.1913/ [email protected]

Karen Jackson

410.786.0079/ [email protected]

Rene Mentnech

410.786.6692/ [email protected]

Geraldine Nicholson

410.786.6967/ [email protected]

Colette Shatto

410.786.6932/ [email protected]

Gladys Valentin

410.786.1620/ [email protected]

David Cantor

301.294.2080/ [email protected]

Sherm Edwards

301.294.3993/ [email protected]

Pamela Giambo

240-453-2981/ [email protected]

Huseyin Goksel

301.251.4395/ [email protected]

Terita Jackson

240.314.2479/ [email protected]

Vasudha Narayanan

301.294.3808/ [email protected]

Westat

13