OMB Supporting Statement - Part Bm

The target population for the Survey consists of all Medicare providers served by Medicare Contractors across the country; CMS will select a sample designed to yield no more than 24,239 completed surveys from providers. The sample of providers will be selected, as shown in Table 4, from 21 Fiscal Intermediaries Contractors, 17 Medicare Carriers, one Part A and B Medicare Administrative Contractor (MAC), four Regional Home Health Intermediaries (RHHIs) and four Durable Medical Equipment Administrative Contractors (DME MACs).

Table 4 Medicare Provider Sample for National Implementation

Provider Types	Sample Size
Hospitals	1,842
Skilled Nursing Facility	3,235
Other Part A providers	3,507
Home Health Agencies	1,541
Hospice facilities	902
Physicians	5,510
Licensed practitioners	3,502
Other Part B providers	1,786
DME suppliers*	2,414
Total	24,239

* DME Suppliers includes physicians who submitted claims for durable medical equipment or supplies.

C-2 Procedures for Collecting Information

C-2.1 Study Sample

The target population for the MCPSS survey consists of all Medicare providers served by all Medicare Contractors in the nation. As of the 2008 MCPSS startup, these Contractors were comprised of 20 Fiscal Intermediaries Contractors, 17 Medicare Carriers, one Part A and B Medicare Administrative Contractor (MAC), four Regional Home Health Intermediaries (RHHIs) and four Durable Medical Equipment Administrative Contractors (DME MACs). The Contractors with multiple service areas are considered as a single Contractor. With changes in the contracting environment we expect to see fluctuations in the number Contractors from one year to the next.

To meet CMS’ objective of making valid comparisons between Contractors, the sample has been designed to obtain an equal number of completed questionnaires from each Contractor. We select a sample to yield 400 completed interviews for each Contractor. For those Contractors with a provider population size 400 or smaller, all the providers will be selected with certainty. Table 1-1 in Attachment 1 shows the provider population size for each provider type within each Contractor. The maximum percent error for estimates of percentages obtained from a simple random sample yielding 400 completed questionnaires will not exceed 5 percent 95 percent of the time. For example, suppose 50 percent of providers responded as satisfied with the service they received. We can be 95 percent confident that between 45 percent and 55 percent of the providers are satisfied with the service. The percent error is the largest for the 50 percent proportion and decreases as proportion moves further away from the 50 percent / 50 percent split. For example, for an 80 percent / 20 percent split, the error is 4 percent. Thus, 400 completed questionnaires should provide adequate precision for Contractor-level estimates. Note that several Contractors have multiple service areas. The precision is provided here for the Contractor-level estimates. The precision of estimates can be much lower for the service areas within the Contractors.

We considered samples sizes of smaller than 400. The sample sizes smaller than 400 will not only provide smaller precision, they will also require more oversampling for smaller provider types. For example, a sample size of 300 will provide an error not exceeding 5.8 percent, which is not substantially higher than 5 percent, however, it will require more extensive and higher oversampling rates in smaller provider types. This oversampling can further reduce the precision of the Contractor level estimates.

The sample size of 400 is allocated proportionately to states and provider types within each Contractor. In Contractors with multiple service areas, the providers will be first stratified by service area and within service area by provider type. The proportional allocation provides a representative sample of providers for Contractors across the service areas and provider types and minimizes the variance of the Contractor-level estimates. The numbers under the heading “Base sample” in Table 1-1 in Attachment 1 show the proportionately allocated sample size for each provider type within each Contractor.

The proportional allocation could result in small sample sizes in several relatively smaller provider types and states. We oversample these states and provider types to yield a minimum of 30 completed questionnaires. In Attachment 1, the additional number of providers needed is shown under the column with a heading “Oversample.” Thirty responses are adequate to conduct statistical tests to detect valid differences between provider types within or across the Contractors, or within or across states.

The satisfaction score has six distinct intervals. The power of a statistical test indicates the probability of rejecting the null hypothesis in error. If the power is inadequate, we cannot draw conclusions from the test with confidence. Sample size affects the power of a statistical test. For example, we could conclude that there is no difference between the scores of two provider types using small samples when, in fact, the samples are too small to detect the true difference. Assuming a standard deviation of 1.35 for the satisfaction score within each provider type, 30 completed questionnaires for each provider type will provide more than 80 percent power (when significance level is 0.05) to detect a mean satisfaction score difference of 1 between the two provider types. Figure 1 shows the power function against various sample sizes per provider type with a standard deviation of 1.35 and a mean score difference of 1 (with equal sample sizes between providers).

Figure 1 Power by Sample Size

Shape2

The target overall response rate for the national survey is 80 percent. The desired precision level by provider types within Contractors is achieved by 24,239 completed questionnaires. Applying the estimated response rate of 80 percent and 85 percent eligibility rate, we would need to contact 35,646 (that is, 24,239/ (0.80*0.85)) providers to achieve the desired number of completes. See Table 1-1 in Attachment 1.

C-2.2 Survey Materials

Survey materials will follow the same design and format as those used in the prior administrations of the national MCPSS. These include:

The Questionnaire:

The questionnaire includes seven topic areas: provider inquiries, provider outreach & education, claims processing, appeals, provider enrollment, medical review, and provider audit & reimbursement. Some of these topics do not pertain to some Contractors and their respective providers. For example, provider enrollment, medical review, and provider audit & reimbursement do not apply to DME suppliers and the DME MACs that serve them. Similarly, the topic of provider audit & reimbursement does not apply to carriers and the providers who work with them. CMS customizes the questionnaire, so providers receive a questionnaire with topics that are relevant to their interaction with the Contractor.

Please see Attachment 2 for a copy of the proposed 2009 MCPSS survey instrument.

CMS is committed to improving the survey with each round of data collection and have set aside dedicated resources to refine the survey. Given the changing contracting environment it is important to include a core set of measures for trending purposes, but at the same time it is important to collect data on new and topical initiatives. CMS will therefore be collecting relevant measurement information from CMS staff and Contractors on a continuous basis. In 2008, a question was added to the instrument that asked about CMS outreach activities pertaining to its Physician Quality Reporting Initiative; in 2009, this question will be asked again. Two new questions about CMS outreach will be added in 2009 - one about CMS’ new outreach activities focusing on preventive care services, and one about a new competitive bidding process.

Web Survey: CMS uses the Web as the primary mode of data collection for the MCPSS. However, to ensure that respondents have the flexibility to respond in the mode that best meets their needs, CMS also maintains the survey in a paper format, as well as in an interviewer-administered format. The Web survey includes easy-to-understand instructions and user-friendly navigation features. The Web survey includes all the instructions included in the paper questionnaire. During past meetings with providers and provider organization representatives, it was communicated to CMS that they generally preferred surveys that were available for completion on-line.

CMS has conducted usability testing to improve the functionality and usability of the Web survey, and we believe no further usability testing is required at this time.

Cover letters: The survey notification package includes two cover letters, one on CMS letterhead and another from the relevant Contractor. The letters explain the purpose of the study, the need for the data, a confidentiality clause, and the unique Provider ID and password to access the Web survey, as well as contact information for questions or to request assistance or a paper questionnaire (e.g., a toll free phone number, a fax number and an e-mail address). These letters are delivered by USPS mail or by email (depending on the preference of the provider).

C-2.3 Data Collection

The data collection steps are as follows:

Email survey invitations (where email of a specific respondent has been determined)
Screener call to others determine most knowledgeable respondent (MKR);
Mail/Email survey notification package (to the address identified during the sample cleaning/screening process);
After initial mail, send a reminder/thank-you postcard/email;
Start non-response follow-up (by telephone) to remaining nonrespondents; and
Close data collection 14-16 weeks after initial screening calls.

In Figure 2 below, we provide the flow for the current MCPSS data collection scheme (as each administration of the MCPSS closes, and CMS assesses the “lessons learned,” this scheme is fine-tuned to best meet the needs of this respondent population).

Figure 2 MCPSS Data Collection Scheme

Shape3

Providers will be encouraged to complete the survey over the secure Web site. The cover letter will clearly state options to access the Web site and complete the survey on-line, or the respondent can print a copy of the questionnaire from the Web site and return it by mail or FAX (so respondents are able to respond using their preferred delivery method). All providers will be given the option to request a paper copy of the questionnaire (rather than downloading it from the Web site) and then submitting their responses via mail or FAX.

The strategy of using the Web as the main mode of data collection worked well during the first national administration. Telephone contact was, and will continue to be, the primary mode for following up with non-responders.

The following media have been set up to allow respondents to communicate with CMS during data collection:

Toll-free Phone: The survey vendor maintains a toll-free telephone number to receive calls from respondents concerning any issues they have regarding the survey.
E-Mail Box: The survey vendor maintains a study e-mail box. This has been a popular feature and can facilitate communication regarding alternative ways respondents want to submit survey responses.
FAX Number: A FAX number is available for respondents who wish to respond via this method. The FAX machine, to which inquiries or responses are sent, is located in a secure location and only authorized project staff can retrieve these documents.

C-2.4 Processing Returned Surveys

There are three criteria that are used for processing returned surveys:

The submission must contain the pre-coded identification number.
All applicable sections should be completed.
In previous administrations, a survey was currently considered a complete if at least one item was completed in the Claims Processing section, and at least one item in any other survey section was completed. Moving forward CMS plans a definition based on core items (refer to previous discussion in the Introduction to this submission, and for details refer to Attachment 5).

C-2.5 Calculating Satisfaction Scores

In order to provide CMS and the Contractors with summary scores with which to monitor trends and compare success across Contractors, a scoring methodology was developed that allows us to calculate respondent level scores for Contractors, provider types and each section. Below is an explanation of how the scores are calculated:

Contractor Score:

The weighted^¹sum of ratings for all questions for all business functions across all provider types related to each Contractor divided by the total number of respondents answering the questions across all business functions for all provider types related to each Contractor

Business Function Score at the Contractor Level:

The weighted sum of ratings for all questions for a business function across all provider types related to each Contractor divided by the total number of respondents answering the questions for that business function related to each Contractor

Provider Score for Each Provider Type under Each Contractor:

The weighted sum of ratings for all questions for all business functions related to a provider type divided by the total number of respondents answering the questions for all business functions related to that provider type

Business Function Score at the Provider Level:

The weighted sum of ratings for all questions for a business function related to a provider type divided by the total number of respondents answering the questions for that business function related to that provider type

Provider Score for Each Provider Type under Each Contractor within a State:

Provider Score for Each Provider Type under Each Contractor within a CMS Jurisdiction:

C-2.6 Contractor Reports

The Contractors have been pleased with the content and level of detail provided in the final Contractor reports. Contractors have indicated that the reports, particularly the item level results, are useful to identifying the services that need improvement. Several Contractors have also stated that the satisfaction scores confirmed what they already thought and/or knew to be problem service areas. In addition, Contractors have agreed that the timeframe for receiving these documents (i.e., July) was good because it helped them prepare for the next fiscal year.

The results from the national implementation are available to all Contractors via an interactive Web based system. Contractors can access the following information via the on-line reports:

Their scores at the Contractor level, provider level and business function level; as well as these levels crossed by State or Jurisdiction
Item level weighted frequencies
Verbatim and coded comments; these comments will be sanitized and will not have any identifiers.

To help identify problem spots, Contractors can view both scores and frequencies by the following parameters:

By state;
By state, by urbanicity (i.e., urban, rural);
By state by provider type;
By state by urbanicity by provider type; and
By provider size.

The summary scores, at all levels, include cell sizes and standard errors. Since providers may have answered some but not all of the sections or only some of the questions for a particular section, the cell size for calculating the scores can vary across sections of the survey. A cell size is presented with each score so Contractors know how many providers responded to that section; this provides an indication of the stability of the score. If only a few providers answered the question, then the survey estimate could fluctuate considerably if we happened to survey a different set of providers. The larger the number of providers who respond to an item, the more confident we are that the survey estimate is close to the “true” answer we would find had we not selected a sample, but instead surveyed all providers. The standard errors are intended to help the Contractor determine how close the Contractor score is to the average Contractor score. If too few providers answered any given survey section, then the results are suppressed to reduce the chance of a Contractor identifying a specific provider. The reports will also include information on key drivers of satisfaction. This information will help Contractors determine which areas within each business function are key drivers of satisfaction with that business function. They will also have information on which business functions are key drivers of overall satisfaction. This information can help Contractors focus their performance improvement efforts.

C-3. Methods to Maximize Response Rates and Deal with Nonresponse

CMS has explored many issues related to increasing the saliency of the study among the provider community and using non-response follow-up strategies to maximize response rates.

The target response rate for the national implementation is 80 percent. As a result of efforts to improve locatability the unweighted response for the 2007 MCPSS was 64.8 percent. Further improvements were implemented in 2008, including:

use of a new data source for obtaining improved contact information was found;
better screening techniques to ensure we have reached the correct respondent before mailing the introductory packet;
better identification of “duplicate^²” sample up-front to reduce respondent frustration;
continued use of the claims history file to only select “active” providers (those submitting a claim in the past 12 month period); and
an aggressive plan for outreach and dissemination.

Any year that the MCPSS falls below the OMB target of 80 percent CMS and its survey vendor will explore the option of conducting a non-response bias analysis. Please see C-3.3 for a detailed description of the proposed non-response bias analysis.

C-3.1 Promoting the Survey Project to Increase Saliency

CMS is taking an aggressive approach to achieving the response rate goal of 80 percent. In addition to obtaining a clean sample, it is essential to create awareness and understanding of the value and importance of the survey within provider and supplier communities in order to motivate participation in the survey. In the end, we want providers and suppliers to view the MCPSS as a tool that will assist CMS and Contractors in identifying and implementing service improvements.

To achieve high saliency for the study, the level of outreach activity between October and January will be high. We are also utilizing aggressive outreach campaigns between January and March to low responding groups, and we also conduct follow-up outreach activities when results are available in July.

The overall objective of this plan is to create awareness for the Medicare Contractor-Provider Satisfaction Survey (MCPSS) among financial and business managers employed by Medicare providers and fee-for-service Contractors. CMS has implemented an annual public relations campaign to generate broad coverage of the MCPSS initiative through a variety of channels:

The healthcare trade media serving financial and business managers employed by Medicare providers and fee-for-service Contractors. This includes members of the print and Web-based media.
Contractor-based communications channels such as list-serves, conferences and meetings, newsletters, etc.
Professional organizations that serve the provider community
CMS based channels of communications to both the providers and Contractors.

C-3.2 Follow-up with Non-respondents

CMS uses, and will continue to use, telephone as the main mode of follow-up with nonrespondents.

C-3.3 Non-response bias analysis

If response rates fall below 80%, CMS will conduct a nonresponse bias analysis. The purpose of this analysis is to determine if the non-respondents are significantly different from the respondents. This will include an analysis of sample frame variables including Contractor, provider type, number of claims, dollar value of claims, size of facility (bed size and or number of patient days), specialty type (in the case of physicians, licensed practitioners, and medical equipment providers), ownership type (for Hospitals and skilled nursing homes). CMS has already submitted to OMB results from non-response bias analyses for prior administrations; if response rates to future MCPSS administrations fall between 60 and 80%, additional non-response bias analyses will be provided as necessary.

In the event that the response rate falls below 60 percent, CMS will create a sub-sample of non-respondents to conduct a more detailed non-response bias study. The sub-sample will include those who refused and facilities that were contacted. Assuming a 60% response (40% non-response), from among the non-respondents, we will draw a sample to yield 450 follow-up respondents. This will provide more than 80 percent power to detect mean satisfaction score differences less than 0.3 between the follow-up respondents and respondents to the regular interview. (That is, testing the difference between the mean scores of 450 follow-up (non) respondents and 15,000 main interview respondents).

This study will include a follow-up survey to the sub-sample. The follow-up survey will include only the claims processing section and the overall satisfaction question. We will then compare the satisfaction scores of the respondents and non-respondents, by Contractor type (FI, Carrier, A/B MAC, DME MAC, RHHI) to determine if there is a significant difference. If significant differences are found, estimates can be adjusted for nonresponse bias through weighting. This follow-up survey will be kept to about 6-7 minutes. This follow-up will also include a question on why the respondent initially refused or did not respond.

The follow-up will be by mail and telephone. The protocol will be as follows:

First mailing questionnaire, with a revised cover letter from CMS, and Contractors.
One week later-a reminder/thank-you postcard
One week later, a second questionnaire
One week later-telephone interviews, with up to 9 additional callbacks

C-3.4 Non-response adjustment

In spite of the best practices, virtually all surveys experience nonresponse. The target response rate for this survey is 80 percent. This will most likely vary by provider type and by other provider characteristics.

One consequence of nonresponse is the potential for bias in the survey estimates, making them larger or smaller than the true statistic for all providers. The extent to which those that do reply differ in their satisfaction from those that do not reply affects the extent of bias. When response rates vary among subgroups, such as provider types, as they are likely to do, there is an even greater potential for bias in survey estimates.

We will adjust the sampling weights to remove potential bias on satisfaction (and on any other substantive estimates to be produced from the survey) caused by not obtaining responses from all sampled providers. If response propensity is independent of the satisfaction, then no bias would arise. Therefore, the objective is, using the known characteristics of the sampled providers, to form nonresponse adjustment cells so that the response propensity within each cell is independent of satisfaction. To the extent that this was achieved, the estimates of satisfaction obtained using the sampling weights that are adjusted for nonresponse within these cells, will have smaller potential bias. There are several alternative methods of forming the cells to achieve this result. In forming the cells, we will attempt to minimize the variation in response propensity within the cells.

We plan to use Chi-Square Automatic Interaction Detector (CHAID) software to guide us in forming the cells. CHAID uses an AID type of algorithm. CHAID partitions data into homogenous subsets with respect to response propensity. To accomplish this, it first merges values of the predictors, which are statistically homogeneous with respect to response propensity and maintains all other heterogeneous values. It then selects the most significant predictor (with the smallest p-value) as the best predictor of response propensity and thus forms the first branch in the decision tree. It continues applying the same process within the subgroups (nodes) defined by the "best" predictor chosen in the preceding step. This process continues until no significant predictor is found or a specified (about 20) minimum node size is reached. The procedure is stepwise and creates a hierarchical tree-like structure.

The data on the relevant characteristics of the providers will be available from the sampling frames for both respondents and nonrespondents. These characteristics include provider type, number of claims (both volume and dollar value) and MSA/nonMSA status for all providers, number of beds for hospitals and skilled nursing facilities, total patient days for hospitals, ownership type of the facility, physician/non-physician specialty and age, and specialty for DMACs.

Although nonresponse adjustment should reduce bias, it can also increase the variance of estimates. Small adjustment classes and/or low response rates (or large nonresponse adjustment factors) may increase the variance substantially and give rise to unstable estimates. In order to prevent an excessive increase in variance and thereby an adverse effect on the mean square error of the estimates, we will limit the size of the classes to a minimum and avoid large adjustment factors.

In June 2008, CMS will provide OMB a supplement with the non-response adjustment methods used in the 2008 survey.

C-4. Tests of Procedures and Methods

CMS will not test any data collection procedures during the national Implementation.

C-5. Individuals Consulted

Organization	Name	Contact Information
CMS	David C. Clark	410.786.6843/ [email protected]
	Alan Constantian	206.615.2306/[email protected]
	Dr. Elizabeth Goldstein	410.786.6665/ [email protected]
	Mel Ingber	410.786.1913/ [email protected]
	Karen Jackson	410.786.0079/ [email protected]
	Rene Mentnech	410.786.6692/ [email protected]
	Geraldine Nicholson	410.786.6967/ [email protected]
	Colette Shatto	410.786.6932/ [email protected]
	Gladys Valentin	410.786.1620/ [email protected]
Westat	David Cantor	301.294.2080/ [email protected]
	Sherm Edwards	301.294.3993/ [email protected]
	Pamela Giambo	240-453-2981/ [email protected]
	Huseyin Goksel	301.251.4395/ [email protected]
	Vasudha Narayanan	301.294.3808/ [email protected]

1 Because not all providers will be selected for the survey and not all selected providers responded, a sample weight will be calculated for each responding provider.

2 The unit of analysis is an individual who submits claims for a health care provider or a group of providers. If more than one provider is selected for this individual, then they may have “duplicate” records in the sample (since we do not know, until screening, who the most knowledgeable respondent will be for a given health care provider).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	0000-00-00