Medicare Beneficiary and Family-Centered Satisfaction Survey
Respondent Universe and Sampling Methods
The sampling and data collection methodology used for the Beneficiary Satisfaction survey has to be efficient based on the sample size, minimally burdensome for beneficiary respondents, frequent enough for use in on-going quality improvement efforts, and rigorous enough to permit for scoring and reporting at the QIO-level. To achieve all of the above described goals, CMS will collect and report the data quarterly. While CMS has included two formal evaluations of QIOs during the 11th SOW, providing quarterly data on beneficiary satisfaction with the complaints and appeals processed will permit for interim corrective action to be taken as needed.
The sample for the 11th SOW Beneficiary Satisfaction Survey will include beneficiaries who have filed quality of care complaints and those who have filed appeals about discharge. The proposed approach for the complaints and appeals cases are each described in this section.
Sample of Complaints Cases. The data collection methodology for the 10th SOW relies on obtaining survey response by mail. Based on this data collection methodology, CMS anticipates obtaining a response rate of approximately 60 percent. A 60 percent response rate would be in line with the response rates obtained on other similar surveys conducted by mail, including the Medicare CAHPS survey. During the 9th SOW, Beneficiary Satisfaction Survey data were collected using computer assisted telephone interviewing. Through this data collection, a seventy-five percent response rate was achieved. Because of the larger sample size required to include appeals cases in the data collection, a shift to a mail methodology is proposed for cost efficiency.
Table 1 presents the estimated quarterly number of complaint cases by state. The estimates are based on the average number of complaint cases over a 3-month period during the 9th SOW. Table 1 also presents estimated completes by state, based on a 60 percent response rate. These numbers represent a census of the complaints cases.
Sample of Appeals Cases. The appeals sampling methodology is based on the most efficient means of achieving the analytic goals of providing quarterly state-level scores and a robust data set of annual data analysis. During previous SOW, an average of 25,599 appeals cases was received per quarter. There is no known reason to believe that the volume of appeals cases will change notably in the 11th SOW.
As shown in Table 1, the sample will be drawn by state. Due to the differing volume of appeals by state, in some states with lower volumes, such as Alaska, Wyoming, Arkansas, Washington DC, Delaware and others, CMS will select a census of appeals cases. In other states including Alabama, Connecticut, California, Florida, and New York, CMS will use a simple random sample to draw 50 quarterly cases per state.
Based on a methodology that relies on data collection by mail, CMS anticipates obtaining a response rate of approximately 60 percent. There is no known reason to believe that response to the appeals survey will differ from other similar surveys including the Medicare CAHPS.
State |
Appeals Estimated Quarterly Cases |
Appeals Proposed Quarterly Sample |
Appeals Estimated Quarterly Completes |
Complaints Estimated Quarterly Cases |
Complaints Proposed Quarterly Sample |
Complaints Estimated Quarterly Completes |
AK |
4 |
4 |
2 |
1 |
1 |
0 |
AL |
308 |
50 |
30 |
8 |
8 |
5 |
AR |
31 |
31 |
19 |
9 |
9 |
5 |
AZ |
228 |
50 |
30 |
7 |
7 |
4 |
CA |
2,637 |
50 |
30 |
36 |
36 |
22 |
CO |
82 |
50 |
30 |
14 |
14 |
8 |
CT |
236 |
50 |
30 |
8 |
8 |
5 |
DC |
11 |
11 |
7 |
1 |
1 |
0 |
DE |
23 |
23 |
14 |
4 |
4 |
2 |
FL |
1,473 |
50 |
30 |
47 |
47 |
28 |
GA |
108 |
50 |
30 |
7 |
7 |
4 |
HI |
14 |
14 |
8 |
3 |
3 |
2 |
IA |
60 |
50 |
30 |
3 |
3 |
2 |
ID |
41 |
41 |
25 |
3 |
3 |
2 |
IL |
266 |
50 |
30 |
15 |
15 |
9 |
IN |
131 |
50 |
30 |
7 |
7 |
4 |
KS |
44 |
44 |
26 |
2 |
2 |
1 |
State |
Appeals Estimated Quarterly Cases |
Appeals Proposed Quarterly Sample |
Appeals Estimated Quarterly Completes |
Complaints Estimated Quarterly Cases |
Complaints Proposed Quarterly Sample |
Complaints Estimated Quarterly Completes |
KY |
81 |
50 |
30 |
4 |
4 |
2 |
LA |
28 |
28 |
17 |
5 |
5 |
3 |
MA |
428 |
50 |
30 |
6 |
6 |
4 |
MD |
217 |
50 |
30 |
9 |
9 |
5 |
ME |
28 |
28 |
17 |
2 |
2 |
1 |
MI |
376 |
50 |
30 |
17 |
17 |
10 |
MN |
216 |
50 |
30 |
9 |
9 |
5 |
MO |
187 |
50 |
30 |
9 |
9 |
5 |
MS |
61 |
50 |
30 |
5 |
5 |
3 |
MT |
10 |
10 |
6 |
2 |
2 |
1 |
NC |
147 |
50 |
30 |
2 |
2 |
1 |
ND |
27 |
27 |
16 |
0 |
0 |
0 |
NE |
95 |
50 |
30 |
1 |
1 |
0 |
NH |
28 |
28 |
17 |
1 |
1 |
0 |
NJ |
472 |
50 |
30 |
17 |
17 |
10 |
NM |
30 |
30 |
18 |
5 |
5 |
3 |
NV |
78 |
50 |
30 |
23 |
23 |
14 |
NY |
1,458 |
50 |
30 |
45 |
45 |
27 |
OH |
345 |
50 |
30 |
23 |
23 |
14 |
OK |
36 |
36 |
22 |
14 |
14 |
9 |
OR |
161 |
50 |
30 |
5 |
5 |
3 |
PA |
1,146 |
50 |
30 |
14 |
14 |
9 |
PR |
49 |
49 |
29 |
7 |
7 |
4 |
RI |
51 |
50 |
30 |
3 |
3 |
2 |
SC |
51 |
50 |
30 |
2 |
2 |
1 |
SD |
18 |
18 |
11 |
0 |
0 |
0 |
State |
Appeals Estimated Quarterly Cases |
Appeals Proposed Quarterly Sample |
Appeals Estimated Quarterly Completes |
Complaints Estimated Quarterly Cases |
Complaints Proposed Quarterly Sample |
Complaints Estimated Quarterly Completes |
TN |
187 |
50 |
30 |
18 |
18 |
11 |
TX |
277 |
50 |
30 |
27 |
27 |
16 |
UT |
52 |
50 |
30 |
2 |
2 |
1 |
VA |
251 |
50 |
30 |
5 |
5 |
3 |
VI |
0 |
0 |
0 |
0 |
0 |
0 |
VT |
16 |
16 |
10 |
0 |
0 |
0 |
WA |
331 |
50 |
30 |
14 |
14 |
8 |
WI |
176 |
50 |
30 |
9 |
9 |
5 |
WV |
61 |
50 |
30 |
2 |
2 |
1 |
WY |
2 |
2 |
1 |
0 |
0 |
0 |
National |
12,737 |
50 |
30 |
0 |
||
Total |
25,603 |
2,190 |
1,314 |
478 |
478 |
287 |
Procedures for Data Collection
The data collection methodology used for the Beneficiary Satisfaction Survey flows from the proposed sampling approach and the need for on-going data for quality improvement. Based on recent literature on survey methodology and response rates by mode, including Dillman’s Tailored Design method1, CMS will use a data collection that is primarily mail. A mail-based methodology will achieve the goals of being efficient, effective, and minimally burdensome for beneficiary respondents. A single mode data collection will also reduce the known mode effects seen particularly in satisfaction surveys.2
Data will be collected quarterly during the QIO 11th SOW. That is to say, data will be collected 4 times per year from the time of OMB approval through the end of the contract period. Over the course of this period, CMS anticipates being able to conduct up to
1 Dillman, D. A. (2007). Mail and Internet Surveys; the tailored design method. New Jersey, United States: John Wiley & Sons Inc.
2 Dillman, D. A., Sangster, R. L., Tarnai, J. and Rockwood, T. H. (1996), Understanding differences in people’s answers to telephone and mail surveys. New Directions for Evaluation, 1996: 45–61. doi: 10.1002/ev.1034
10 rounds of data collection. The sample for each round of data collection will include appeals and complaint cases closed in the previous quarter such that no beneficiary should be sampled more than once. Re-appeals cases will not be included in the universe of eligible cases for sampling. In order for CMS and the QIOs to assess the degree to which patient-centered care is being delivered
Since data will be collected and reported for internal quality improvement on a quarterly basis, the data collection methodology must strive to minimize the data collection field period while maximizing the response rate. The desired data collection field period is 8 to 10 weeks. To achieve these goals, CMS will format the survey materials per Dillman’s Tailored Design principles and use a three-staged approach to data collection:
Mailout of a covering letter, the paper survey questionnaire, and a postage-paid return envelope.
Mailout of a post card that thanks respondents and reminds the non-respondents to please return their survey.
Mailout of a follow-up covering letter, the paper survey questionnaire, and a postage-paid return envelope.
First a pilot study will be conducted (described in detail in section 4). Through the pilot test, CMS will determine the response rate that can be achieved using this approach. If it is deemed necessary, a prenotification letter or additional mailout reminders can be added to the protocol, a telephone non-response step can be added to the protocol as needed to achieve the desired response rate. Additional information on maximization of response rates is included in section 3.
Using the 3-step mail approach described above, CMS anticipates that data collection would occur over an 8 to 10 week period. This is to say, if the first survey mailing were dropped on January 1, we would anticipate completing data collection at the end of February or early March. Data would then be cleaned, scores would be generated, and data would be delivered for CMS and QIO quality improvement review. The precise timing required to achieve an acceptable response rate will be determined through the pilot test. The aim is to complete sampling, data collection complete and scoring within a 12-week period.
Survey Material: The Beneficiary Satisfaction survey will capture beneficiary satisfaction with the appeals review process as well as quality of care complaints.
Table 2 provides a summary of the Survey composites and questions. Each of the survey composites represents an important aspect of patient-centeredness. Data from the survey will be used by CMS and the QIOs to support the quality improvement effort, make changes to improve processes, and review overall success of each QIO at executing the complaints and appeals process in a manor aligned with CMS goals for patient and family centeredness.
NOTE: The questionnaire will be available in English and Spanish.
10th SOW and Picker Institute Principles |
Composite Label and Questions |
Promoting effective coordination of care including helping communities support better health; transitions and continuity |
Coordination Q13. The QIO representative talked with you about programs and services in your community that were available to help you with your health and wellbeing. |
Information, communication and education |
Beneficiary-Centered Communication Q7. How satisfied were you that the QIO representative explained things in a way you could understand? Q8. How satisfied were you that the QIO representative spent enough time with you? Q9. How satisfied were you that the QIO representative listened carefully to you? |
|
Communication of Written Materials Q15. How satisfied were you that the forms or letters you got about your [quality of care complaint / appeal] explained things in a way you could understand? Q16. How satisfied were you that the forms or letters you got about your [quality of care complaint / appeal] had all the information you needed? Q17. How satisfied were you that the forms or letters you got about your [quality of care complaint / appeal] showed respect for your concerns? Q18. How satisfied were you that the forms or letters you got about your [quality of care complaint/appeal] were consistent with the information you were told in telephone conversations with the QIO? |
Emotional support and alleviation of fear and anxiety |
Courtesy and Respect Q6. How satisfied were you that the QIO representative was as helpful as you thought he or she should be? Q10. How satisfied were you that the QIO representative showed respect for what you said? |
Access to care |
Access and Responsiveness Q11. The QIO representative was as responsive to your [quality of care complaint / appeal] as you thought he or she should be. Q12. The QIO representative understood the situation related to your [quality of care complaint / appeal]. |
Analysis: On a monthly basis, each QIO will receive interim data results based on data collected to date for the current round of collection. Item level frequencies will be produced for each QIO such that QIOs can review and monitor survey results from beneficiaries in their state on an on-going basis. Both QIOs and CMS will use the data to identify areas of concern with the processes and institute immediate corrective action.
On an annual basis, data collected over the previous four quarters will be used for detailed analysis and reporting. Annual analysis will include univariate analysis for each of the survey variables as well as multivariate analysis to explore drivers of beneficiary satisfaction. Analytic findings will be presented to the QIO community to help QIOs improve the process utilized in processing beneficiary appeals and complaints. Additionally, CMS will use results to help re-shape the infrastructure supporting QIOs.
Satisfaction Scoring: Standardized scoring for the Beneficiary Satisfaction Survey will permit CMS and the QIOs to assess the process used in resolving beneficiary complaints and appeals without undertaking lengthy analysis on a quarterly basis. Satisfaction scoring is linked to the QIO scope of work and on-going evaluation of QIO performance. Based on the parameters of the QIO scope of work, calculations will be performed to determine the percent of beneficiaries who report being satisfied with the complaint or appeal process. Proposed scoring methodology is presented below. All questions proposed for inclusion in the scoring calculation use the satisfaction scale, or the agreement scale. The details of each of these scales are presented below.
Satisfaction Scale:
Very Satisfied
Satisfied
Neither Satisfied nor Dissatisfied
Dissatisfied
Very Dissatisfied
Agreement Scale:
Strongly Agree
Agree
Neither Agree nor Disagree
Disagree
Strongly Disagree
Survey responses of 1 and 2 (very satisfied and satisfied; and strongly agree and agree) will be counted as 1 point. Survey responses of 3, 4 and 5 (neither satisfied nor dissatisfied; neither agree nor disagree dissatisfied and strongly dissatisfied; and disagree and strongly disagree) will be counted as 0 points, as well as missing responses will not be included in the denominator for scoring – with the goal of producing standardized satisfaction scores for QIO evaluation.
Scores will be calculated at the case level for each of the survey composites. The case level scores will then be rolled up to quarterly QIO level scores for each of the survey composites. Per the QIO 10th Scope of Work, QIOs are to achieve the goal of 80 percent satisfied beneficiaries by the mid-point of the contract and 90 percent satisfied beneficiaries by the end of the contract.
Efforts to maximize response rates will take many forms, including multiple contacts and survey design principles.
As described in section 2 on data collection procedures, outreach to respondents will occur over three separate mailouts. All mailouts will be sent via first class mail. Timing of the mailouts will ensure that respondents are reminded of the request for their participation in the survey.
The survey design team will use Dillman’s Tailored Design principles in preparing the survey and mailout materials. These design principles have been shown to increase response rates to mailout surveys using formatting and layout principles. Covering materials will stress the importance of the respondent’s input and the use of survey findings to improve processes and make them more patient- centered, leveraging Dillman’s social exchange theory.
The mail methodology proposed here has been used successfully on other CMS surveys (Medicare CAHPS).
As CMS is requesting an extension of an existing survey, we do not propose any new testing of the survey instrument and are relying on the testing done in association with the development of the existing survey.
At the beginning of the 10th SoW, a round of cognitive testing was completed. CMS did not conduct a full pilot test of the data collection and methodology. We cognitively tested the instruments for clarity of language and responses. At the time we submitted the OMB package for the survey currently in use, we relied on experience regarding response rate on similar surveys of Medicare beneficiaries. Consequently, CMS used the first two years of the survey data to evaluate how QIOs processed cases, including how QIOs encouraged beneficiaries to agree to take the survey. CMS used satisfaction scores collected in the first two years as a process monitoring tool and not a tool used for contract evaluation. The satisfaction score data were used in year 3 for contract evaluation purposes.
The following individuals were consulted in the development of the surveys.
Organization |
Name |
Contact Information |
CMS |
David Russo |
617.565.1310 |
CMS |
Sally Berko |
410.786.6211 |
Attachment 1 - CMS Medicare Beneficiary and Family Centered Care Satisfaction Survey
Quality of Care Complaints
Q1. Our records show that on [DATE] you filed a quality of care complaint about your or another person’s Medicare benefits. Is that right?
Q2. Have you received the results or findings in response to your quality of care complaint?
Q3.
How satisfied are you with the results or findings in response to
your quality of care complaint?
Q4.
Please give us your comments on the results or findings in response
to your quality of care complaint.
Q5. Did you speak to a QIO representative about your quality of care complaint?
Q6. How satisfied were you that the QIO representative was as helpful as you thought he or she should be?
Q7. How satisfied were you that the QIO representative explained things in a way you could understand?
Q8. How satisfied were you that the QIO representative spent enough time with you?
Q9. How satisfied were you that the QIO representative listened carefully to you?
Q10. How satisfied were you that the QIO representative showed respect for what you said?
Q11. The QIO representative was as responsive to your quality of care complaint as you thought he or she should be.
Q12. The QIO representative understood the situation related to your [quality of care complaint.
Q13. The QIO representative talked with you about programs and services in your community that are available to help with your health and wellbeing.
Q15. How satisfied were you that the forms or letters you got about your quality of care complaint explained things in a way you could understand?
Q16. How satisfied were you that the forms or letters you got about your quality of care complaint had all the information you needed?
Q17. How satisfied were you that the forms or letters you got about your quality of care complaint showed respect for your concerns?
Q18. How satisfied were you that the forms or letters you got about your quality of care complaint were consistent with the information you were told in telephone conversations with the QIO?
Q19.
In responding to your appeal, [QIO NAME], the QIO in your state,
gathered information about your quality of care complain, explained
the complaint steps, and gave you the results or findings of your
case. We are referring to this as the “quality of care
complaint process”. Using any number from 0 to 10 where 0 is
the worst quality of care complaint process possible and 10 is the
best quality of care complaint process possible, what number would
you use to rate the overall quality of care complaint process?
Q20.
Please give us your comments on the process that was used in
responding to your quality of care complaint. Include any comments
you have on what worked well, and suggestions you have on ways to
improve the process.
Attachment 2 - CMS Medicare Beneficiary and Family Centered Care Satisfaction Survey
Appeals.
Q1. Our records show that on [DATE] you filed an appeal about your or another person’s Medicare benefits. Is that right?
Q2. Have you received the results or findings in response to your appeal?
Q3.
How satisfied are you with the results or findings in response to
your appeal?
Q4.
Please give us your comments on the results or findings in response
to your appeal.
Q5. Did you speak to a QIO representative about your appeal?
Q6. How satisfied were you that the QIO representative was as helpful as you thought he or she should be?
Q7. How satisfied were you that the QIO representative explained things in a way you could understand?
Q8. How satisfied were you that the QIO representative spent enough time with you?
Q9. How satisfied were you that the QIO representative listened carefully to you?
Q10. How satisfied were you that the QIO representative showed respect for what you said?
Q11. The QIO representative was as responsive to your appeal as you thought he or she should be.
Q12. The QIO representative understood the situation related to your appeal.
Q13. The QIO representative talked with you about programs and services in your community that are available to help with your health and wellbeing.
Q15. How satisfied were you that the forms or letters you got about your appeal explained things in a way you could understand?
Q16. How satisfied were you that the forms or letters you got about your appeal had all the information you needed?
Q17. How satisfied were you that the forms or letters you got about your appeal showed respect for your concerns?
Q18. How satisfied were you that the forms or letters you got about your appeal were consistent with the information you were told in telephone conversations with the QIO?
Q19.
In responding to your appeal, [QIO NAME], the QIO in your state,
gathered information about your appeal, explained the appeal steps,
and gave you the results or findings of your case. We are referring
to this as the “appeal process”. Using any number from 0
to 10 where 0 is the worst appeal process possible and 10 is the best
appeal process possible, what number would you use to rate the
overall appeal process?
Q20.
Please give us your comments on the process that was used in
responding to your appeal. Include any comments you have on what
worked well, and suggestions you have on ways to improve the process.
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
File Title | Medicare Beneficiary and Family-Centered Satisfaction Survey Supporting Statement Part B |
Subject | Supporting Statement Part B |
Author | CMS |
File Modified | 0000-00-00 |
File Created | 2021-01-24 |