Part C and D Complaints Resolution Performance Measure
CMS-10308
OMB Supporting Statement – Part A
March 15, 2013
3. Use of Information Technology 4
8. Federal Register/Outside Consultation 7
9. Payments/Gifts to Respondents 12
12. Burden Estimates (Hours & Wages) 13
14. Cost to Federal Government 14
The Health Maintenance Organization Act of 1976, the Balanced Budget Act of 1997 (BBA), and the Medicare Prescription Drug, Improvement, and Modernization Act of 2003 (MMA) introduced private insurers into the Medicare program. With the HMO Act of 1976, health maintenance organizations (HMOs) began to be offered as a Medicare option. The BBA established the Medicare + Choice program, which gave beneficiaries the option of enrolling in a variety of private plans including HMOs, preferred provider organizations (PPOs), provider-sponsored organizations (PSOs), private fee-for-service (PFFS) plans, and medical savings accounts (MSAs) coupled with high-deductible insurance plans. Title I of the MMA established the new prescription drug benefit under Part D of Title XVIII of the Social Security Act (Act). Title II of the MMA modified Part C of the Act to rename the Medicare + Choice program as the Medicare Advantage (MA) program. These programs are administered by the Centers for Medicare & Medicaid Services (CMS).
Part C Sponsors provide medical coverage through at-risk arrangements with CMS. Part C Sponsors include: Local Coordinated Care Plans, which include HMOs, PPOs, and PSO plans; Private fee-for-service plans (PFFS); Special needs plans (SNPs); MSAs; and Regional PPOs. Under Sections 1876 and 1833(a)(1)(A) of the Social Security Act, an HMO or CMP can participate in the Medicare program by receiving “reasonable cost” reimbursement for furnishing covered services to enrolled beneficiaries. 1833 Cost Plans (or Heath Care Prepayment Plans) must either be union- or employer-sponsored and must not provide inpatient hospital services for its enrollees.
Part D Sponsors provide prescription drug benefit coverage through private at-risk prescription drug plans that offer drug-only coverage (Prescription Drug Plans), or through Medicare Advantage (MA) plans that offer integrated prescription drug and health care coverage (MA-PD plans). MA plans that offer this coverage can be risk-based or cost-based plans. A Prescription Drug Plan (PDP) serves one or more PDP regions.
The focus of the current project is to assess the satisfaction of beneficiaries with the process by which their complaints were resolved by the plans in which they were enrolled, and to evaluate the final outcome through an objective exploration of beneficiaries’ complaint resolution experiences.
The agency does not have access to this information through regular administrative or reporting requirement mechanisms. The current data collection effort assists CMS in obtaining this critical information. CMS has the option to use the results from this data collection effort for program monitoring (internal use) or for public reporting purposes via the Medicare Advantage website or other alternative means. The surveys targets complaints—filed by beneficiaries or their representatives—that have been closed in the Complaints Tracking Module (CTM) by a plan during of the plan year. The survey timeframe will be an entire calendar year and the CMS will conduct a census of all beneficiaries that filed a complaint, which will therefore provide the most statistically valid data (further detail regarding the sampling plan is provided in Supporting Statement B). The web-based surveys will occur within 7 to 21 calendar days of the complaint closure and will collect beneficiaries’ opinions on the complaint resolution process and their satisfaction with the final outcome, among other issues.
Several substantive issues are involved in this Complaint Resolution Survey (CRS) data collection request. First, CMS will follow statistical approaches to control for factors affecting satisfaction with the final outcome and the complaint resolution process. Several variables will be considered in the development of a monitoring measure so as not to rely on a single aspect of the beneficiary's experience. This includes the use of beneficiary, health plan, and complaint characteristics recorded in other CMS datasets. Second, CMS will emphasize the primary issue of interest is the "final outcome or decision," to prevent confusion with a beneficiary's opinion of the decision. A focus on the “final outcome or decision” rather than “resolution” puts the focus on the series of actions the plan took; regardless of whether the beneficiary believes his/her complaint was resolved. Third, CMS recognizes the potential for complaints that are outside the scope of the plan (restricted by CMS guidelines), particularly regarding some issues related to enrollment, complaints and will exclude such complaints from the sampling framework.
The premise of the CRS data collection is to conduct a study of the entire population of interests: all complaints that were closed across all plans. CMS will review the results of the survey responses and the analysis and utilize the information to conduct internal monitoring of plans and to identify any potential plan issues. CMS may also conduct a real-time survey of complainants via telephone of specific plans or contracts per CMS’ discretion. The real-time survey will be an excellent monitoring tool to enhance current procedures for tracking and reviewing plans’ adherence to complaints closure guidance and/or following up on faltering plans.
There may be some potential and yet-undefined issues that continue to be addressed in using the survey responses. Some potential issues are listed and described below:
Are beneficiaries responding distinctly to issues of complaint settlement and resolution?
A low response rate for certain contracts and/or low number of complaints during the data collection period (real-time data collection).
Positive response from industry to preliminary results regarding measurement of beneficiary satisfaction of complaint resolution.
It is important to note that CMS has addressed most technical issues with assessing the representativeness of the complaints in the sampling universe and the survey instrument has been vetted with several survey and Medicare experts including CMS staff involved on other CMS surveys (such as the Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey). Thus, any issues that are reviewed during analysis are likely to be outside of these technical aspects. Points (a) through (c) above demonstrate the necessity of this current larger scale data collection effort in order to ensure the high quality of the data.. CMS will review and analyze the responses to the survey and discuss the results with subject-matter experts at CMS and other institutions as well as selected Part C and D contracts to assess its usability and/or representativeness of beneficiary satisfaction with the complaint resolution process.
Beginning in plan year 2013, CMS will administer the beneficiary survey primarily via a website. A paper and pencil self-administered version will also be sent to a sample of beneficiaries that do not respond via the web-based survey. CMS will conduct a pilot test of the web survey with 500 beneficiaries to work out any technical and operational issues with the web-based instrument or study logistics. The pilot test of the web survey will simulate all tasks and activities of the full-scale study from mailing the advance letters and survey sample management to the statistical analysis of the survey responses and contract and beneficiary information. A telephone and paper version of the survey was previously piloted to work out any technical and operational issues with the instrument or study logistics.
All survey responses will be submitted to a rigorous analytical review using multivariate regression analyses to produce monitoring measures or other ad hoc analysis that are controlled by contract characteristics (enrollment, type of contract, etc.) and beneficiary characteristics (age, community risk score). The multivariate regression analyses will support statements of the likelihood of a beneficiary being satisfied or not with a statistical level of certainty. Further information on the analysis is included in Section B.16.a. Tabulations, Analysis.
CMS strives to share information and communicate with plans on issues of performance for public reporting. The results from the current CRS are not intended for performance measurement. CMS will only utilize the data for internal monitoring.
This clearance package seeks approval to conduct a survey as part of the Part C and D Complaints Resolution Performance Measure project. This survey effort is sponsored by the Centers for Medicare & Medicaid Services (CMS) and is being implemented, under contract to CMS, by IMPAQ International, LLC (IMPAQ). The purpose of the project is to develop and support implementation of internal monitoring tools for the Medicare Advantage (Part C) and Prescription Drug (Part D) program that represents, from the beneficiary’s perspective, the way in which plans handle complaints.
The data collection is necessary because a survey is the only way to collect information about the resolution process from the beneficiary’s perspective. Currently, there is no other data source that collects such information for Part C and Part D Medicare plans.
The current survey instrument is attached to this statement. An accompanying document, “Explanations for the Inclusion of Survey Questions,” presents the rationale for the inclusion of each question in the survey.
Data collected from the surveys will be used by CMS to construct monitoring measures of the veracity and effectiveness of plan complaint resolution from the beneficiary’s perspective.
Prior to 2013, CMS used two modes of data collection to assess beneficiaries’ experiences with the complaint resolution process. Computer Assisted Telephone Interviewing (CATI) had been used to conduct telephone surveys. Paper surveys were sent through the mail to beneficiaries who could not be reached by phone. In 2013, CMS will begin collecting the data by primarily utilizing a web-based survey and supplemented on an ad hoc basis with a paper and pencil self-administered version. The inclusion of this mode of data collection – via a website – is an opportunity for CMS to reach out to more beneficiaries at a cost-effective ratio. All beneficiaries who filed a complaint and their complaint has been closed by plan will be contacted to complete the CRS (available in English and Spanish) about the complaint process. For beneficiaries who filed a complaint online, the web-based survey will bring their online complaint process full circle by keeping all complaint-related electronic. The web-based survey will have the same burden to beneficiaries as the current data collection mechanisms (phone and paper).
The original two modes of data collection (telephone and mail-in) included all the same questions and answer options as the web-based survey instrument. In 2013, the phone based survey has been dropped as the primary means of data collection in favor of the web-based survey. The sole difference between the modes is that FAQs will be available to beneficiaries completing the web-based survey. However, this is not anticipated to affect the results of the data collection and the data will not be separated or treated differently in analysis. The FAQs were designed to address any questions the respondents may have regarding the survey sponsorship and purpose of the data collection.
The FAQs will include general information that will also be found in the Advance Letter and the cover page of the survey instrument. For example, FAQs will cover questions beneficiaries may have about why they are being contacted, how their contact information was compiled, whether participation in the survey will affect their current benefits, and IMPAQ’s role in the data collection. As these types of questions will also be covered by the mailed materials, there is little reason to expect the mode of data collection to effect the uniformity of responses.
Both the English and Spanish versions for the online survey have been reviewed by CMS to ensure that they meet the standards of the Plain Language Act (PLA) of 2010. According to the PLA, government documents (including this survey) should use “writing that is clear, concise, well-organized, and follows other best practices appropriate to the subject or field and intended audience1.” The survey materials were updated to align with these standards.
The original two modes of data collection (telephone and mail-in) included all the same questions and answer options as the web-based survey instrument. The sole difference between the telephone and mail-in modes and the web-based survey is that the full list of FAQs will be available to beneficiaries completing the web-based survey. As mentioned before, CMS’s primary data collection mode is the web-based survey. The paper survey will be administered to a sample of 1,000 beneficiaries. CMS anticipates a 50% response rate for the mailed version of the survey resulting in 500 additional paper responses. Beneficiaries from small contracts will make up half of the paper survey sample (n=500) in order to increase responses from these smaller groups. The other 500 cases of the paper survey sample will be spread among contracts or populations targeted by CMS, and beneficiaries who wish to participate in the survey via a collection mode that is easily accessible to them. As stated above, CMS may choose to send paper surveys to specific contracts to increase response rates or for specific populations. CMS will coordinate these priority requests with IMPAQ. The different modes of data collection are not anticipated to affect the results of the study and the data will not be separated or treated differently in analysis.
The FAQs were designed to address any questions the respondents may have regarding the survey sponsorship and purpose of the data collection. The FAQs will include general information that will also be found in the Advance Letter and, in the case of the paper-based survey, the cover page of the survey instrument. For example, FAQs will cover questions beneficiaries may have about why they are being contacted, how their contact information was compiled, whether participation in the survey will affect their current benefits, and IMPAQ’s role in the data collection.
Beneficiaries who filed a complaint will be sent a letter with instructions on how to access the web-based survey. In addition, each beneficiary will be provided with their CTM complaint ID that will survey as their log-in for the web-based survey. The survey is available in both English and Spanish. Beneficiaries will have the option of toggling the survey between English and Spanish by accessing a drop-down menu displayed prominently on the top of the webpage. The instructions for how to toggle appear on the first several screens and beneficiaries are able to switch languages throughout the survey.
The website for the CRS contains an internal link to the FAQs utilized in past waves of the survey. Respondents may access the FAQs at any point during their self-administration of the survey and quickly and easily return to the survey questions. In addition, a section of the website is devoted to beneficiary resources and provides web links to the following resources:
Medicare.gov
HHS.gov
File a Complaint (https://www.medicare.gov/MedicareComplaintForm/home.aspx)
Useful Phone & Websites (http://www.medicare.gov/Contacts/Default.aspx)
The web-based survey will become the primary mode of data collection due to the potential for a larger sample at a lower cost per survey. For beneficiaries who will be filing their initial complaint online, the continuation of examining their satisfaction with the process through a web-based survey may help to increase the survey’s overall response rate2. In addition, the use of self-administered questionnaires such as a web-based or mail survey has been shown to decrease social desirability bias when compared to interviewer-administered survey3. Given that we are surveying beneficiaries who filed a complaint, adding a self-administered web survey option may foster more honest responses regarding beneficiary satisfaction. If any differences arise between the web-based survey and the original modes, the effects will be minimized by the relatively small, if any, number of surveys completed by telephone and paper. Methods to distinguish any differences are described in the analysis overview in section B.16.a. Tabulations, Analysis Plan. The statistical analysis takes into account the different modes of data collection and a control variable identifying the mode of data collection will be included in the multivariate analysis to assess the statistical effect of the mode of data collection on the results regarding beneficiary satisfaction. In summary, we have not seen any statistical differences during the current data collection and do not anticipate and statistical differences between the modes.
This survey will be conducted to collect key information from CMS beneficiaries. No other survey data collection effort has been conducted or has been planned to collect similar information. The study also will use administrative data from the Complaints Tracking Module (CTM) and other CMS datasets, such as the Medicare Beneficiary Database (MBD, Common Tables). CTM data are not sufficient to conduct the study because they do not include the perspective of beneficiaries; hence, survey data are needed to supplement the CTM and other CMS data.
The survey will only involve individual beneficiaries; therefore, it will not pose a burden to small businesses. Members of 800 series contracts will be excluded from the data collection effort for reasons described in Supporting Statement B (Section 1: Respondent Universe and Sampling).
The survey will be the primary source of data for the monitoring of the beneficiaries satisfaction with the complaint resolution process. The survey will collect information about the resolution process and the resolution from the beneficiary’s perspective. Currently, there is no other data source that collects such information for Part C and Part D Medicare plans.
The beneficiaries affiliated with the sampled complaints for this project will only be surveyed once during a year regardless of the number of complaints filed by a beneficiary. Therefore, less frequent data collection is equivalent to not being able to collect any data and would result in an inability to construct the monitoring measures or prepare other ad hoc analysis.
In all respects, the data will be collected in a manner consistent with Federal guidelines. The statistical survey will produce valid and reliable results that can be generalized to the universe of the study, and it will include only statistical data classifications that have been reviewed and approved by OMB. The survey includes a pledge of confidentiality that is supported by authority established in statute or regulation and by disclosure and data security policies that are consistent with the pledge. It will not unnecessarily impede sharing of data with other agencies for compatible confidential use.
Federal Register Notice and Comments
The 60-day Federal Register notice published on May 3, 2013 (78 FR 26053). Comments were received and our response has been added to this PRA package.
Consultation Outside of the Agency
CMS has contracted with IMPAQ to assist in the development, implementation and analysis of the beneficiary surveys. The following individuals help to support the data collection and analysis activities for this effort:
Name |
Affiliation |
Telephone Number |
Oswaldo Urdapilleta |
IMPAQ International |
(202) 696-1003 |
Amy Djangali |
IMPAQ International |
(443) 283-1648 |
Gongmei Yu |
IMPAQ International |
(443) 539-9769 |
Alisu Schoua-Glusberg |
IMPAQ International |
(847) 864 5677 |
There will be no respondent payments for this survey.
IMPAQ follows procedures for ensuring and maintaining confidentiality consistent with provisions of the Privacy Act of 1974. Respondents receive information about confidentiality protection in an advance letter describing the survey (provided as an attachment to this package) and again at the outset of the web-based survey as part of the introductory explanation. Respondents are informed that all information they provide will be treated confidentially. Project staff is trained in confidentiality procedures and is prepared to describe these procedures in full detail, if needed, or to answer any related questions from the respondents who may contact IMPAQ for more information. For example, if asked about confidentiality, the IMPAQ will explain that the answers will be combined with those of others and presented in summary form only, that no identifiable information about participants will be made public, and that the answers will not affect past or future eligibility for any programs.
All data items that identify respondents will be kept only by the contractor, IMPAQ, for use in assembling records data and conducting the interviews. Any data received by CMS does not contain personal identifiers, thus precluding individual identification.
In addition, the following safeguards will be employed to carry out confidentiality assurances:
All employees at IMPAQ sign a confidentiality pledge that emphasizes the importance of confidentiality and sets forth the obligations of staff.
Identifying information is maintained in a separate file from interview data. The files are linked only with a sample identification number.
Access to link-files containing sample identification numbers connecting the research data and the respondents' identification is limited to a few individuals who have a need to know this information.
Access to any hard-copy documents is strictly limited. Physical precautions include use of locked files and cabinets, shredders for discarded materials, and interview control procedures.
The survey of CMS beneficiaries contains a minimal set of items that may be considered sensitive in nature. These questions are related to adverse medical episodes experienced by beneficiaries with complaints. These questions are needed to evaluate the frequency and degree to which beneficiaries suffer as a result of the amount of time spent by plans to determine a final outcome to complaints. As described in item A10, all respondents will be assured of confidentiality in the advance letter as well as within the introduction of the web-based survey. All survey responses will be held in strict confidence and reported in aggregate, summary format, eliminating the possibility of individual identification. IMPAQ will comply with the requirements of the Privacy Act of 1974, in collecting all information.
The total annual hour burden for respondents for the current data collection is shown in Table 1 below. We anticipate 17,710 survey responses (pilot and main data collection) from the web survey assuming a 30% response rate and 500 additional responses from the paper version of the survey with the total universe of CTM complaints (after exclusions) of 59,032 beneficiaries.
Table 1: Annual Hour Burden for Respondents
Cite/reference |
Total Survey Respondents |
Frequency of Data Collection |
Average Time per Respondent |
Annual Hour Burden |
Complaints Resolution: Pilot and Main Survey – Web |
17,710 |
Once |
10 minutes |
2,952 |
Complaints Resolution: Pilot and Main Survey – Mail |
500 |
Once |
10 minutes |
83 |
Total |
18,210 |
Once |
10 minutes |
3,035 |
The total annualized cost to respondents of collecting this information is shown in Table 2 below.
Table 2: Annualized Cost to Respondents
Respondent Category |
Number of Respondents |
Total Number of Hours |
Hourly Rate |
Estimated Data Collection Cost to Respondents |
Estimated Cost per Respondent |
Not working |
15,224* |
2,537 |
$0.00 |
$0.00 |
$0.00 |
Working full-time |
1,693* |
283 |
$22.96** |
$6,498 |
$3.84 |
Working part-time |
1,293* |
215 |
$13.06** |
$2,808 |
$2.17 |
Total |
18,210 |
3,035 |
|
$9,306 |
|
* Based on U.S. Bureau of Labor Statistics figures for labor force participation of workers 65+
** Mean hourly earnings based on the National Compensation Survey, Dec. 2007-Jan. 2009
This is a continuation of a one-time survey. There will be no capital or start-up costs incurred by respondents. There are no record keepers. There will be no costs to respondents for operations, maintenance, or purchase of services.
The cost to the Federal government of conducting the survey is $225,000, which is the total contractor cost of conducting the survey.
We propose to survey all complaints in the universe instead of a sample from the universe.
In 2013, CMS will begin collecting the data by primarily utilizing a web-based survey and supplemented on an ad hoc basis with a paper and pencil self-administered version. The inclusion of this mode of data collection – via a website – is an opportunity for CMS to reach out to more beneficiaries at a cost-effective ratio.
The web-based survey will have the same per response burden to beneficiaries as the current data collection mechanisms (phone and paper).
The original two modes of data collection (telephone and mail-in) included all the same questions and answer options as the web-based survey instrument. In 2013, the phone based survey has been dropped as the primary means of data collection in favor of the web-based survey. The sole difference between the modes is that FAQs will be available to beneficiaries completing the web-based survey.
With the addition of the web-based survey and in order to bring the survey into alignment with the Plain Language Act (PLA), changes/revisions have been made to the survey materials--none of which impact currently approved burden estimates. A Crosswalk of those changes is included in this package.
Tabulations
All survey data will be combined with CMS administrative data, including the Medicare Beneficiary database, HPMS contract information, and the Complaints Tracking Module (CTM) records. The internal monitoring measures are listed below under each respective research domain:
Beneficiary Satisfaction
Beneficiary satisfaction with the complaint handling process
Beneficiary satisfaction with the plan
Resolution Effectiveness
Beneficiary awareness of resolution
Beneficiary satisfaction with final outcome
Veracity of plan’s description of final outcome or decision
Plan Effectiveness
Beneficiary experiences during complaint resolution process
Areas for improvement in the complaint handling process
The research domains are described below and followed by details of how the indicators are linked to the survey instrument questions and how they will be used in analysis.
Beneficiary Satisfaction
In any industry, it is necessary to study the satisfaction of the consumer. There are many ways to define this term. To truly understand satisfaction, the researcher must evaluate consumers’ emotions, which fluctuate within individuals and vary in range across individuals.4 Measuring the consumer’s emotions cannot be accomplished externally; therefore, optimal surveys ask consumers to rate their own emotions. Satisfaction is then the favorability of their subjective assessment of the organization, company, or group.5 In the present context, the rating is a representation of the beneficiary’s sense of fulfillment with the customer service experience.6 Have all his/her needs been met? Have all expectations been met?
It is important to note that a plan may comply with all CMS requirements, yet still have low monitoring scores. This would result because beneficiaries reflect on their subjective evaluation of the plan’s ability to resolve the complaint and adequately address their issues without regard to the plan’s level of compliance with CMS requirements. Thus, the information is analyzed in this context.
The following indicators from the beneficiary survey correspond to the beneficiary satisfaction domain:
Beneficiary satisfaction with the complaint handling process
Beneficiary satisfaction with the plan
For these satisfaction questions and others, CMS is using a 4-point Likert scale rather than a 5-point Likert scale (with a neutral option) to encourage beneficiaries to provide an opinion on each question. The use of a 4-point scale will improve the survey results. However, an N/A answer choice is available to beneficiaries who find the questions not be applicable such as beneficiaries who believe their complaint has not been resolved.
Resolution effectiveness relates to both effectiveness and veracity. Questions of resolution effectiveness assess whether a complaint was resolved satisfactorily from the beneficiary’s perspective. This is a subjective, but clear-cut, outcome measure of the complaint resolution process. However, resolution effectiveness also addresses resolution veracity, by comparing the beneficiary’s knowledge of the complaint resolution with the plan’s stated resolution, which can be found in the CTM. While the plan may have appropriately resolved the complaint and addressed all of the beneficiary’s concerns, the beneficiary may not have been informed of the plan’s decision. Veracity refers to the matching of beneficiary and plan information about the resolution. Moreover, the resolution effectiveness research question separates the plan’s ability to properly address and resolve complaints from its ability to handle complaints well and to the beneficiary’s satisfaction.
The following indicators from the beneficiary survey correspond to the resolution effectiveness domain:
Beneficiary awareness of resolution
Beneficiary satisfaction with final outcome
Veracity of plan’s description of final outcome or decision
Plan effectiveness refers to the complaint resolution process, as evaluated by the beneficiary. Questions about plan effectiveness will demonstrate how the beneficiary believes the complaint was handled by the plan. Regardless of whether the plan resolved the complaint to the beneficiary’s satisfaction (what the outcome was), the perceived difficulty of the process may affect the beneficiary’s subjective evaluation of the plan’s performance. Subjects included in this domain are communication, timeliness, and consequences for the beneficiary.
The following indicators from the beneficiary survey correspond to the plan effectiveness domain:
Beneficiary experiences during complaint resolution process
Areas for improvement in the complaint handling process
“Beneficiary Experiences during the Complaint Resolution Process” is calculated from the percentage of beneficiaries who experienced any potential problems while waiting for a complaint to be resolved. “Contact by Plan” is the percentage of complaints where the beneficiary was contacted by the plan. “Repeat Complaints” highlights the percentage of beneficiaries who contacted the plan more than once before their complaint was resolved.
Indicators:
In principle, the exploratory analysis – described in greater detail in the following section – will include tabulations of survey item responses and beneficiary and plan characteristics. A driver analysis could determine statistically which areas most impact overall customer satisfaction, and it may be possible to estimate the direction and magnitude in which the drivers impact overall satisfaction. We could determine which specific attributes have the most impact on overall customer satisfaction and, therefore, would warrant primary attention and resources for CMS. We will conduct analyses to construct risk-adjusted monitoring measures, using beneficiary characteristics, plan characteristics, and complaint characteristics.
These analyses will minimize measurement bias associated with confounding factors affecting the monitoring measures. For example, beneficiaries with certain characteristics may have higher or lower levels of satisfaction than the average beneficiary. Overall, satisfaction measures will be risk-adjusted to account for beneficiary and plan characteristics.
In all analyses, we will include information regarding whether the respondent is the beneficiary or a representative if there were differences between the satisfaction levels of these two groups. However, during the current survey we have not identified any statistical differences on the satisfaction measures as a function of who is the survey respondent.
Beneficiary satisfaction with the complaint handling process. Monitoring measures will be developed to assess how satisfied complainants are with different aspects of the complaint resolution process. The main aspects include the following: length of the complaint process; courtesy of the plan representative; time your plan took to contact you; amount of time spent handling your complaint; awareness of the complaints process; and explanation of the final outcome. Beneficiary satisfaction on these issues will be measured through the following survey questions:
Q2. How satisfied were you with the following parts of the complaint process? [Specific items follow]
Q5. How would you rate your overall satisfaction with the way your complaint was handled by [PLAN]?
The responses to each item in Question 2 and to Question 5 will together produce measures of the complaint resolution process. The items in Question 2 will highlight the extent to which plans treated the complainant courteously, provided the complainant with understandable explanations of the final outcome, and provided the complainant with enough information about how the complaint was resolved. Question 5 will measure how satisfied beneficiaries are with the overall process undertaken by plans to resolve their complaint.
In responding to these questions (and each item within Question 2), the complainant must answer “very satisfied,” “satisfied,” “dissatisfied,” “very dissatisfied,” or “I don’t know/NA.” These responses will be translated into numerical values, where complainants who are very satisfied will have the greatest value (2), and those who are very dissatisfied will have the smallest value (-2). Each preliminary measure will be calculated as the mean value of the numerically translated responses about the complainants’ satisfaction. Therefore, larger values will indicate better plan performance in handling complaints to complainants’ satisfaction. It is important to note that the survey emphasizes the final outcome rather than the settlement (or resolution).
Final satisfaction with the process by which the complaint was handled (Q2) will be tabulated against satisfaction with each of the main aspects of the process (Q5). The responses to Question 5 will be used to isolate complaints where the beneficiary is satisfied with the final outcome, but has concerns about various aspects of the process.
As with the other internal monitoring measures, this measure will be tabulated against beneficiary and plan characteristics, and we will run risk-adjustment models. No single item will define satisfaction for each aspect of the complaint resolution process. In particular, different complaint categories may affect each aspect of the process in different ways.
Beneficiary satisfaction with the plan. An indicator of overall satisfaction with the plan will be included in the analysis. This indicator will be used to analyze the satisfaction measures listed above as well as allow for further analysis of critical satisfaction issues that may have significant repercussions in a beneficiary’s selection of a plan. The information for overall beneficiary satisfaction will be derived from the following survey questions:
Q7. Based on your recent experience, how satisfied are you with [Plan name]?
Q8. How likely are you to stay with this plan?
Both questions address issues of the beneficiary’s satisfaction with the plan including health care and the complaint resolution process. Question 8 will indicate to what extent a beneficiary’s experience with the complaint resolution process affects his/her opinion of the plan. Being able to tabulate both questions will be insightful since we will learn about the overall satisfaction with the plan. On the one hand, if the complaint resolution process was so unsatisfactory that the beneficiary is willing to switch plans, it will provide context for plans with low “Resolution Handling” indicators. On the other hand, analysis of these responses may show that even beneficiaries with unsatisfactory complaint resolution experiences are not unhappy enough to actually switch plans.
Beneficiary Awareness of Resolution. This measure will capture the percentage of a plan’s complainants who either are aware or agree that a settlement to their complaint has been implemented or reached. Since the complaints in our sample have been closed in the CTM, the expectation is that a complaint has been settled and that the beneficiary is aware of this resolution.
From the survey, the response to the following question will be used to calculate this measure:
Q1. Our records show the complaint you filed about [COMPLAINT] was recently closed by the plan. Was your complaint settled?
Complainants may answer say “yes”, “no”, or “I don’t know.” To calculate this measure, the numerator will be a count of the number of sampled complaints in which the complainant answered “yes.” The denominator will be the total count of sampled complaints in which the complainant indicates either “yes” or “no” regarding the resolution of his/her complaint. Therefore, the exclusion criteria for the denominator will be a response of “I Don’t Know” or those who did not answer that question.
CMS will treat the correlation between satisfaction with the final outcome and the awareness of a resolution as a subjective measure because of the potential that some beneficiaries would respond affirmatively (“yes, the complaint was settled”) only if the complaint had been resolved in their favor.
Beneficiary satisfaction with final outcome. This monitoring measure will relay the extent to which complainants are satisfied with the final outcomes or decisions that plans have provided regarding their complaints. The assessment of satisfaction with the final outcome will be collected through the following question:
Q4. How satisfied are you with the final outcome of your complaint?
Responses to this question will be limited to four options (“very satisfied”, “satisfied”, “dissatisfied”, and “very dissatisfied”). These will be translated into numerical values, where complainants who are very satisfied will have the greatest value (2), and those who are very dissatisfied will have the smallest value (-2).
The raw measure will be calculated as the mean value of the numerically translated responses about the complainants’ satisfaction. Therefore, larger values will indicate better plan performance in resolving complaints to complainants’ satisfaction. As with the other measures, this measure will be tabulated against beneficiary and plan characteristics, and we will run risk-adjustment models. In this way, no single item will define satisfaction with the final outcome.
Veracity of plan’s description of final outcome. This monitoring measure will assist in the review of the accuracy of plans’ descriptions of their complaint resolution (outcome, decisions, etc.) in the CTM. Accuracy will be determined through comparison with the complainants’ descriptions of the final outcome regardless of whether the beneficiary sees it as a resolution. This information will be gathered in the survey through the following question:
Q3. At any time during the complaint process, did you experience any of the following? [specific items follow]
The responses to this question will be used to provide more knowledge about the scale of beneficiaries’ experiences. As with the other measures, this measure will be tabulated against beneficiary and plan characteristics, and we will run risk-adjustment models. The implication is that no single item associated with the risks, difficulties, and problems of a particular complaint will define the monitoring measure.
Areas for improvement in the complaint handling process. This information will not be used in the development of a performance measure. However, information from this open-ended question will allow us to make further recommendations concerning CTM guidelines and identify which complaint categories are most likely to be flagged by beneficiaries or cause strong dissatisfaction by beneficiaries. The open-ended responses will come from the following question:
Q10. Do you have any suggestions or comments about how your plan could handle complaints better?
Analysis:
Several statistical analyses will be implemented to untangle the reasons behind a beneficiary’s assessment of a plan sponsor’s resolution, or a beneficiary’s satisfaction with a plan sponsor’s complaint resolution process. This information is important for two reasons: (1) to define a set of internal monitoring measures that can be used to validate the plan’s resolution of beneficiary complaints closed by plans, from the perspective of the beneficiaries themselves; and (2) to provide critical elements that should be monitored or included in CMS Standard Operating Procedures, CMS guidance on handling of complaints, documentation standards, and other CMS documents on the subject.
For the analysis of survey responses, several administrative datasets will be used. For example, the Common Medicare Enrollment tables from the Medicare Enrollment Database (EDB) will be used to identify (1) beneficiary enrollment at the contract level, (2) beneficiary state and county codes, (3) election periods used for enrollment, (4) Part A and Part B entitlement, and (5) Part D eligibility. The Medicare Beneficiary Database (MBD) will be used to identify contract characteristics and contract service areas, while HPMS will be used to identify contract status and other contract characteristics.
In the analysis of the survey and complementary administrative data, we will look at descriptive statistics, testing, and the development of measures based on beneficiary survey responses. The information provided by beneficiaries will be very valuable in establishing a correlation with a plan’s own assessment of its complaints resolution process.
Our analysis will be both quantitative and qualitative. First, we will use the quantifiable items from the survey to define and calculate measures of beneficiary satisfaction and exploratory analysis of patterns and correlations between beneficiary satisfaction and beneficiary and/or Part C/D sponsor characteristics. We will use information from HPMS CTM and other CMS data sets to explore who the beneficiary respondents are and to develop models for assessing patterns for certain populations (Low-Income-Status beneficiaries) or type of contracts (MA, MA-PD, PDP). Our reason for the exploratory analysis is that certain responses to beneficiary satisfaction have confounding factors that should be controlled in developing MAO and PDP sponsor ratings. For example, it could be the case that certain populations have a higher likelihood of being dissatisfied with Parts C and D sponsors; if a contract has a large proportion of this population, the plan sponsor will have lower ratings than it should have if we were to control for the share of the “prone to be dissatisfied” population. Similar arguments can be made regarding the type of plans.
In the initial analysis, we will prepare descriptions of the survey respondents (gender, age, marital status, health status, dual eligible status) and the plan sponsors (Contract and organization type, enrollment size, complaint type, complaint categories and proportion of subpopulation enrollees) in the study sample. In addition to these exploratory tables, we will test nonresponse rates for those beneficiaries who did not respond to our survey to explore whether they are different from the survey respondents.
In a similar fashion, we will present univariate descriptive statistics for the MAO and Part D sponsors. It is important to emphasize that the survey responses are valid at the contract level. There is no need to adjust the survey results. However, further exploration of the results by contract characteristics would contribute to CMS understanding of the complaint resolution process. For example, are contracts with a large proportion of LIS beneficiaries more likely to have higher rates of disatisfaction, or are PFFS more likely to have more satisfied beneficiaries given the nature of the organization type? In addition, we will compare the data collected at the contract level to the overall universe of complaints.
Following the description of the beneficiaries and the contracts in the study sample, we will proceed to present bivariate descriptive statistics. In an early phase of the analysis, we will assess patterns across populations or Parts C/D sponsor characteristics to the beneficiary satisfaction measures. This information will provide insight on population/contract correlations to beneficiary satisfaction and some of the underlying factors influencing beneficiary satisfaction. We will also conduct cross tabulations of beneficiary satisfaction to consequences or problems associated with the complaint. Such tabulations will provide insights on how certain problems/consequences affect beneficiaries’ perspective on satisfaction. Another table could be developed for satisfaction with the time it took to get a resolution.
After the draft measures have been calculated, we will conduct additional analyses to further refine the measure specifications based on the following criteria:
What are the underlying distributions of performance data, such as the mean, median, standard deviation, and percentile scores?
How much dispersion is there across plans? Is the dispersion random or does it appears to systematically affect certain types of plans?
Are there too many contracts that are subject to the data suppression rules or that have a missing data issue?
We will also conduct a qualitative review of beneficiaries’ responses, which will focus on the open-ended questions. Examples of such monitoring measures might be the following:
Whether the complaint resolution recorded in the CTM data conforms to the beneficiary’s response regarding the final outcome
Whether the resolution provided by the plan conforms to the beneficiary’s original request; What aspects of the plan’s handling of the complaint were unsatisfactory to the beneficiary
We will review a sample of beneficiaries’ responses in order to identify the key themes of their perceived/articulated responses. Once these themes have been identified, the results of these initial reviews will then be captured in keywords (with similar typologies identified as well), which will be used in the next search of the qualitative data. If frequent occurrences of those keywords or similar typologies are found, these issues/recommendations will be summarized as key findings in the Beneficiary Experience Reports. For keywords with less frequency, our summary results will highlight those for which some practical policy adjustments can be offered. Through this qualitative review of beneficiary responses, we can report on aspects of the beneficiary experience that may not have been captured elsewhere in the survey.
Last, we will conduct multivariate analysis that could be developed using data from the survey and other CMS data sources. Logistic and multinomial analysis for several questions of beneficiary satisfaction would provide information on the factors that influence beneficiary responses on a particular aspect of beneficiary satisfaction. These results would further CMS’ knowledge about what is driving beneficiary satisfaction. Overall, these models allow predicting rates of beneficiary satisfaction given certain beneficiary and plan characteristics. On the basis of these results, we will risk-adjust beneficiary responses across contracts.
Publication Plans
The final report on the Part C and Part D Complaint Resolution Measures project will be submitted to CMS in draft form in August and in final form in September of each calendar year. The report will describe the data collection and analysis process and make recommendations for future improvements. The report also will contain summary statistics of the respondents and the satisfaction measures. The measure statistics will be stratified by various plan and complaint characteristics.
Time Schedule
The project began in September 2009 and the instruments were prepared between October and December 2009. OMB approval was obtained in September 2010. The data collection is ongoing at this time. The wave of data collection for this OMB approval request starts in October 2013. The analysis of the survey data and the construction of the monitoring measures will be completed in August 2013. The data collection will continue and run concurrently with the analysis and continue into 2014. The cycle of the data collection, analysis of survey data and construction of the monitoring measures will repeat each calendar year.
The expiration date will be displayed on the advance letter and on the hard copy version of the questionnaire as well as the bottom of each page of the web survey.
There are no exceptions taken to item 19 of OMB Form 83-1.
1 Plain Language Act of 2010, Public Law 111-274, 124 STAT. 2861 (2010)
2 Olson, K., Smyth, J.D. & Wood, H.M. (2012) Does giving people their preferred survey mode actually increase survey participation rates? An experimental examination. Public Opinion Quarterly, 76 (4). 611 – 635.
3 Kreuter, F., Presser, S. & Tourangeau, R. (2008). Social desirability bias in CATI, IVR and web surveys: The effects of mode and question sensitivity. Public Opinion Quarterly, 72 (5). Retrieved from: http://poq.oxfordjournals.org/content/72/5/847.full
4 H. K. Hunt (1977), “CS/D-Overview and Future Research Direction,” in Conceptualization and
Measurement of Customer Satisfaction and Dissatisfaction, ed. H. K. Hunt, L. Hu, and P. M. Bentler.
5 R. A. Westbrook (1980), “A Rating Scale for Measuring Product\Service Satisfaction,” Journal of Marketing, 44: 68-72.
6 R. L. Oliver (1997), Satisfaction: A Behavioral Perspective on the Consumer (New York: McGraw-Hill).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | MEMORANDUM |
| Author | Rosita Turkel |
| File Modified | 0000-00-00 |
| File Created | 2021-01-29 |