SUMMARY OF CHANGES IN THE 2010 MCPSS INSTRUMENT
Rationale for Changes to MCPSS Instrument
CMS is proposing the implementation of a new MCPSS instrument for its 2010 national administration to improve the quality of survey questions. These changes include a new rating scale, wording revisions to some questions in various sections, and the addition or deletion of some questions. The proposed instrument changes are detailed below.
The Rating Scale
The current 2009 MCPSS instrument uses a 6-point anchoring scale where only the two end points are labeled. Respondents are instructed that a ‘1’ represents “not at all satisfied” and a ‘6’ represents “completely satisfied”. CMS has faced challenges in communicating scale results to stakeholders; for example, explaining exactly what an average score of 4.5 indicates. Some stakeholders have attempted to translate the 6-point scale to a 100 point grade, thus a mean of 4.5 would translate to a grade of 70 (or a grade of “C-”). However, this sort of translation is inadequate. For unlabeled scales, the distance between two points may not be equal. For example, the cognitive difference between a score of 2 and 3 may be larger than between 4 and 5.
The change to a fully anchored scale will provide for a clearer message about the performance of Medicare contractors. We will be able to report “percent satisfied” (simply the percent of respondents with a survey mean of “Satisfied” or “Very Satisfied”). Christian, Dillman and Smyth (2006) have indicated that a fully labeled scale has higher reliability than those that are only partially labeled. Furthermore, the new proposed scale is a 5-point fully-labeled Likert scale (simply a linear rescaling of the average rank of the respondent’s ratings across items) with a neutral category:
Very Satisfied
Satisfied
Neither Satisfied nor Dissatisfied
Dissatisfied
Very Dissatisfied
The rationale for including a neutral category is that some respondents may truly have a neutral opinion about a product or service. As such, we believe that a fully labeled scale with a neutral option will pose less of a response burden on respondents than the current version of the scale.
In the past, the 6-point scoring scale was used to maintain historical comparison between administrations of the MCPSS. For the 2010 national implementation CMS will move to a 5-point scoring scale. Our current contracts with the Medicare contractors include a bonus clause for incentive payments derived from their mean score on the 2009 MCPSS. Contractors have an opportunity to earn an incentive payment if the score on this year’s MCPSS is within a set number of standard deviations from the prior year’s score.
We recognize that there is a trade-off for a 50-50 allocation. The current analytic reports include substrata estimates only if there are at least 30 reporting respondents. Assuming a standard deviation of 1.35 for the satisfaction score within each provider type, we found that 30 completed questionnaires for each provider type will provide more than 80 percent power (at a significance level of 0.05) to detect a mean satisfaction score difference of ‘1’ between two groups (such as provider type or states). Since 30 responses are adequate to conduct statistical tests to detect differences among groups, this was used as the guideline for oversampling in the MCPSS. The 5-point scale is necessary to more sufficiently measure the scores of provider responses regarding their satisfaction with FFS Medicare contractors.
The Items
The survey is divided into eight sections; a background section followed by sections addressing each of the seven business functions. The revisions to each section of the survey are outlined below.
Section: Background
Additional background items on:
facility size
identification as a small provider
The rationale: Background items were added to include data on outreach activities to small providers and to enhance reporting of provider communication efforts made to small providers.
Section: Provider Inquiries
This section contains a few additions, two deletions, and a few wording revisions.
Deleted items:
[Satisfaction with] The effort your Contractor makes to make the Provider Inquiries process as easy as possible for you
[Satisfaction with] The professionalism and courtesy of your Contractor’s representatives throughout Provider Inquiries activities
The rationale: We propose deleting these items due to the general nature of the questions and replacing them with questions that focus on specific features of this business function.
New items:
[Satisfaction with] The ease of obtaining information through your Contractor’s automated telephone system (IVR), if accessed?
[Satisfaction with] The information made available through your Contractor’s automated telephone system (IVR) meeting your needs, if accessed?
The rationale: The IVR system has become an integral part of the contractor’s system for handling provider inquiries. Therefore, it should be included in the 2010 survey.
What is your overall satisfaction with your Contractor’s provider inquiry activities?
The rationale: The results from this question are used to assess the key drivers to overall satisfaction. An overall section-level assessment provides us with data relevant to specific business function areas of each contractor.
Revised item wording:
2009 wording: [Satisfaction with] Receiving the correct information
2010 wording: [Satisfaction with] Receiving the correct information over the phone from a representative
The rationale: By narrowing the focus of this item, we can obtain more focused feedback on telephone interactions and provide more specific feedback to the contractors in terms of their performance.
2009 wording: [Satisfaction with] The consistency of responses that you get from different Provider Inquiries representatives
2010 wording: [Satisfaction with] The consistency of written responses
The rationale: By narrowing the focus of this item, we can obtain more focused feedback on written responses and provide more specific feedback to the contractors in terms of their performance.
2010 wording: The addition of “IVR” as a response choice in questions asking about methods of communication
The rationale: Adding this response choice will enable us to collect data on the IVR system, which has become an important tool in this business function.
Section: Provider Outreach and Education
The Provider Outreach and Education section is being restructured so as to provide more specific information to the contractors so they can better target process improvement efforts.
Deleted items:
[Satisfaction with] The topics of the training and education materials are up-to-date
[Satisfaction with] CMS products to educate you on how to bill for preventive services
[Satisfaction with] CMS outreach and education products on how Durable Medical Equipment Prosthetics, Orthotics and Supplies (Competitive Bidding Program) affect you and your Medicare patients1
Do you find CMS listserv messages {@ ADDRESS} an effective method of communication to notify you about new Medicare Fee-for-Service information?
The rationale: Preventive services and the DMEPOS Competitive Bidding Program are no longer research priorities. Additionally, deleting these questions shifts the focus from CMS activities to contractor activities.
New items:
[Satisfaction with] Accessibility of educational materials / information resources
[Satisfaction with] The usefulness of your Contractor’s listserv (e-mail) messages in notifying you about new Medicare program information
[Satisfaction with] The usefulness of your Contractor’s Frequently Asked Questions (FAQs)
The rationale: These items were added to more fully represent and collect data about specific activities within this business function.
What is your overall satisfaction with your Contractor’s outreach and education?
The rationale: The results from this question are used to assess the key drivers to overall satisfaction. An overall section-level assessment provides us with data relevant to specific business function areas of each contractor.
Revised items:
In the last twelve months/ Since {DATE} what education and training resources of (CONTRACTOR) have you used?
In 2010, two response options have been added: electronic mail (E-mail) materials and listserv information
The rationale: The addition of these two choices brings this item in line with activities performed by the contractors.
2009 wording: [Satisfaction with] The amount of training and educational resources available from your Contractor
2010 wording: [Satisfaction with] Availability of training (when asked about training)
2010 wording: [Satisfaction with] Amount of educational materials/information resources (when asked about educational materials)
The rationale: These questions will now be asked about three specific modes of education: face-to-face training, non face-to-face training, and educational materials/information resources. The results will be used to identify and collect data on these specific activities performed by the contractors.
2009 wording: [Satisfaction with] The tailoring of training or education at a level you can understand
2010 wording: [Satisfaction with] Clarity of information presented
The rationale: These questions will now be asked about three specific modes of education: face-to-face training, non face-to-face training, and educational materials/information resources. The results will be used to identify and collect data on these specific activities performed by the contractors.
2009 wording: [Satisfaction with] The detail in which topics are covered
2010 wording: [Satisfaction with] Detail of topics covered
The rationale: This question will now be asked about two specific modes of education, face-to face training and non face-to-face training, to provide more focused feedback on this activity.
2009 wording: [Satisfaction with] The relevance of the training and education material topics to meet your organization’s needs.
2010 wording: [Satisfaction with] The relevance of the training to meet your specific needs
2010 wording: [Satisfaction with] The relevance of the education materials and information resources to meet your specific needs
The rationale: These questions will now be asked about three specific modes of education: face-to-face training, non face-to-face training, and educational materials/information resources. The results will be used to identify and collect data on these specific activities performed by the contractors.
Section: C. Claims Processing
Deleted items:
[Satisfaction with] The accuracy of remittance advices received from your Contractor
[Satisfaction with] The availability of your Contractor’s representatives to address claims-related issues
In the last twelve months how have you submitted claims?
The rationale: The deleted items are replaced with new items that better define the constructs CMS is interested in measuring.
New items:
[Satisfaction with] The promptness of your Contractor in resolving claims–related issues brought to their attention
[Satisfaction with] The correctness of the information provided to you by your Contractor in response to claims-related issues raised by you
The rationale: The new items more closely define the constructs CMS is interested in measuring.
What is your overall satisfaction with your Contractor’s claims processing activities?
The rationale: The results from this question are used to assess the key drivers to overall satisfaction. An overall section-level assessment provides us with data relevant to specific business function areas of each contractor.
Revised item wording:
2009 wording: [Satisfaction with] The ease of correcting claims, such as correcting claims online or asking for a change over the phone
2010 Wording: [Satisfaction with] The ease of correcting claims, such as correcting claims online or requesting a change over the phone
The rationale: The revised wording formalizes the language of the item.
Section: D. Appeals
New items:
[Satisfaction with] Your average telephone hold time before talking to a live person
[Satisfaction with] If leaving a message, the average time before receiving a return call
The rationale: These items were added to more fully represent the specific activities performed by the contractors for this business function.
What is your overall satisfaction with your Contractor’s appeals activities?
The rationale: The results from this question are used to assess the key drivers to overall satisfaction. An overall section-level assessment provides us with data relevant to specific business function areas of each contractor.
Section: E. Provider Enrollment
New items:
What is your overall satisfaction with your Contractor’s provider enrollment?
The rationale: The results from this question are used to assess the key drivers to overall satisfaction. An overall section-level assessment provides us with data relevant to specific business function areas of each contractor.
Section: F. Medical Review
New items:
What is your overall satisfaction with your Contractor’s medical review activities?
The rationale: These questions will now be asked about three specific modes of education: face-to-face training, non face-to-face training, and educational materials/information resources. The results will be used to identify and collect data on these specific activities performed by the contractors.
Section: G. Provider Audit and Reimbursement
New items:
What is your overall satisfaction with your Contractor’s provider audit and reimbursement activities?
The rationale: The results from this question are used to assess the key drivers to overall satisfaction. An overall section-level assessment provides us with data relevant to specific business function areas of each contractor.
Revised item wording:
2009 wording: [Satisfaction with] The timeliness of your Contractor’s audit of your Cost Report, if one is conducted, and the final settlement
2010 wording: [Satisfaction with] The timeliness of your Contractor audit of your Cost Report
2010 wording [Satisfaction with] How satisfied are you with the timeliness of your Contractor’s settlement of your cost report
The rationale: The clarity of the language used will provide more focused feedback to the contractors.
REFERENCES
Christian, Leah Melani, Don A. Dillman and Jolene D. Smyth. 2006. The Effects of Mode and Format on Answers to Scalar Questions in Telephone and Web Surveys.
Draft version of paper presented at the 2nd International Conference on Telephone Survey Methodology, January 12, 2006 in Miami, Florida.
1 Note – while this item was in the previous OMB submission packet it was deleted from the final instrument that went to the field in 2009. We include it here so that OMB is able to compare this submission to the most recent one.
| File Type | application/msword |
| Author | CMS |
| Last Modified By | CMS |
| File Modified | 2009-06-23 |
| File Created | 2009-06-23 |