CMS-10393 Revised PRA Supporting Stmt-Pt B Passback Clean Final 07 14 2016

Supporting Statement – Part B

Medicare Beneficiary and Family-Centered Satisfaction Survey

2. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The sampling and data collection methodology used for the Beneficiary Satisfaction survey has to be efficient based on the sample size, minimally burdensome for beneficiary respondents, frequent enough for use in on-going quality improvement efforts, and rigorous enough to permit for scoring and reporting at the QIO-level. To achieve all of the above described goals, CMS will collect and report the data quarterly. While CMS has included two formal evaluations of QIOs during the 11^th SOW, providing quarterly data on beneficiary satisfaction with the complaints and appeals processed will permit for interim corrective action to be taken as needed.

The sample for the 11^th SOW Beneficiary Satisfaction Survey will include beneficiaries who have filed quality of care complaints and those who have filed appeals about discharge. The proposed approach for the complaints and appeals cases are each described in this section.

Sample of Complaints Cases. The data collection methodology for the 10^th SOW relies on obtaining survey response by mail. Based on this data collection methodology, CMS anticipates obtaining a response rate of approximately 60 percent. A 60 percent response rate would be in line with the response rates obtained on other similar surveys conducted by mail, including the Medicare CAHPS survey. During the 9^th SOW, Beneficiary Satisfaction Survey data were collected using computer assisted telephone interviewing. Through this data collection, a seventy-five percent response rate was achieved. Because of the larger sample size required to include appeals cases in the data collection, a shift to a mail methodology is proposed for cost efficiency.

Table 1 presents the estimated quarterly number of complaint cases by state. The estimates are based on the average number of complaint cases over a 3-month period during the 9^th SOW. Table 1 also presents estimated completes by state, based on a 60 percent response rate. These numbers represent a census of the complaints cases.

Sample of Appeals Cases. The appeals sampling methodology is based on the most efficient means of achieving the analytic goals of providing quarterly state-level scores and a robust data set of annual data analysis. During previous SOW, an average of 25,599 appeals cases was received per quarter. There is no known reason to believe that the volume of appeals cases will change notably in the 11^th SOW.

As shown in Table 1, the sample will be drawn by state. Due to the differing volume of appeals by state, in some states with lower volumes, such as Alaska, Wyoming, Arkansas, Washington DC, Delaware and others, CMS will select a census of appeals cases. In other states including Alabama, Connecticut, California, Florida, and New York, CMS will use a simple random sample to draw 50 quarterly cases per state.

Based on a methodology that relies on data collection by mail, CMS anticipates obtaining a response rate of approximately 60 percent. There is no known reason to believe that response to the appeals survey will differ from other similar surveys including the Medicare CAHPS.

Table 1: Estimated Quarterly Numbers of Cases and Estimated Quarterly Completes

State	Appeals Estimated Quarterly Cases	Appeals Proposed Quarterly Sample	Appeals Estimated Quarterly Completes	Complaints Estimated Quarterly Cases	Complaints Proposed Quarterly Sample	Complaints Estimated Quarterly Completes
AK	4	4	2	1	1	0
AL	308	50	30	8	8	5
AR	31	31	19	9	9	5
AZ	228	50	30	7	7	4
CA	2,637	50	30	36	36	22
CO	82	50	30	14	14	8
CT	236	50	30	8	8	5
DC	11	11	7	1	1	0
DE	23	23	14	4	4	2
FL	1,473	50	30	47	47	28
GA	108	50	30	7	7	4
HI	14	14	8	3	3	2
IA	60	50	30	3	3	2
ID	41	41	25	3	3	2
IL	266	50	30	15	15	9
IN	131	50	30	7	7	4
KS	44	44	26	2	2	1

State	Appeals Estimated Quarterly Cases	Appeals Proposed Quarterly Sample	Appeals Estimated Quarterly Completes	Complaints Estimated Quarterly Cases	Complaints Proposed Quarterly Sample	Complaints Estimated Quarterly Completes
KY	81	50	30	4	4	2
LA	28	28	17	5	5	3
MA	428	50	30	6	6	4
MD	217	50	30	9	9	5
ME	28	28	17	2	2	1
MI	376	50	30	17	17	10
MN	216	50	30	9	9	5
MO	187	50	30	9	9	5
MS	61	50	30	5	5	3
MT	10	10	6	2	2	1
NC	147	50	30	2	2	1
ND	27	27	16	0	0	0
NE	95	50	30	1	1	0
NH	28	28	17	1	1	0
NJ	472	50	30	17	17	10
NM	30	30	18	5	5	3
NV	78	50	30	23	23	14
NY	1,458	50	30	45	45	27
OH	345	50	30	23	23	14
OK	36	36	22	14	14	9
OR	161	50	30	5	5	3
PA	1,146	50	30	14	14	9
PR	49	49	29	7	7	4
RI	51	50	30	3	3	2
SC	51	50	30	2	2	1
SD	18	18	11	0	0	0

State	Appeals Estimated Quarterly Cases	Appeals Proposed Quarterly Sample	Appeals Estimated Quarterly Completes	Complaints Estimated Quarterly Cases	Complaints Proposed Quarterly Sample	Complaints Estimated Quarterly Completes
TN	187	50	30	18	18	11
TX	277	50	30	27	27	16
UT	52	50	30	2	2	1
VA	251	50	30	5	5	3
VI	0	0	0	0	0	0
VT	16	16	10	0	0	0
WA	331	50	30	14	14	8
WI	176	50	30	9	9	5
WV	61	50	30	2	2	1
WY	2	2	1	0	0	0
National	12,737	50	30	0
Total	25,603	2,190	1,314	478	478	287

2. Procedures for Data Collection

The data collection methodology used for the Beneficiary Satisfaction Survey flows from the proposed sampling approach and the need for on-going data for quality improvement. Based on recent literature on survey methodology and response rates by mode, including Dillman’s Tailored Design method¹, CMS will use a data collection that is primarily mail. A mail-based methodology will achieve the goals of being efficient, effective, and minimally burdensome for beneficiary respondents. A single mode data collection will also reduce the known mode effects seen particularly in satisfaction surveys.²

Data will be collected quarterly during the QIO 11^th SOW. That is to say, data will be collected 4 times per year from the time of OMB approval through the end of the contract period. Over the course of this period, CMS anticipates being able to conduct up to

¹ Dillman, D. A. (2007). Mail and Internet Surveys; the tailored design method. New Jersey, United States: John Wiley & Sons Inc.

² Dillman, D. A., Sangster, R. L., Tarnai, J. and Rockwood, T. H. (1996), Understanding differences in people’s answers to telephone and mail surveys. New Directions for Evaluation, 1996: 45–61. doi: 10.1002/ev.1034

10 rounds of data collection. The sample for each round of data collection will include appeals and complaint cases closed in the previous quarter such that no beneficiary should be sampled more than once. Re-appeals cases will not be included in the universe of eligible cases for sampling. In order for CMS and the QIOs to assess the degree to which patient-centered care is being delivered

Since data will be collected and reported for internal quality improvement on a quarterly basis, the data collection methodology must strive to minimize the data collection field period while maximizing the response rate. The desired data collection field period is 8 to 10 weeks. To achieve these goals, CMS will format the survey materials per Dillman’s Tailored Design principles and use a three-staged approach to data collection:

1. Mailout of a covering letter, the paper survey questionnaire, and a postage-paid return envelope.

2. Mailout of a post card that thanks respondents and reminds the non-respondents to please return their survey.

3. Mailout of a follow-up covering letter, the paper survey questionnaire, and a postage-paid return envelope.

First a pilot study will be conducted (described in detail in section 4). Through the pilot test, CMS will determine the response rate that can be achieved using this approach. If it is deemed necessary, a prenotification letter or additional mailout reminders can be added to the protocol, a telephone non-response step can be added to the protocol as needed to achieve the desired response rate. Additional information on maximization of response rates is included in section 3.

Using the 3-step mail approach described above, CMS anticipates that data collection would occur over an 8 to 10 week period. This is to say, if the first survey mailing were dropped on January 1, we would anticipate completing data collection at the end of February or early March. Data would then be cleaned, scores would be generated, and data would be delivered for CMS and QIO quality improvement review. The precise timing required to achieve an acceptable response rate will be determined through the pilot test. The aim is to complete sampling, data collection complete and scoring within a 12-week period.

Survey Material: The Beneficiary Satisfaction survey will capture beneficiary satisfaction with the appeals review process as well as quality of care complaints.

Table 2 provides a summary of the Survey composites and questions. Each of the survey composites represents an important aspect of patient-centeredness. Data from the survey will be used by CMS and the QIOs to support the quality improvement effort, make changes to improve processes, and review overall success of each QIO at executing the complaints and appeals process in a manor aligned with CMS goals for patient and family centeredness.

NOTE: The questionnaire will be available in English and Spanish.

Table 2: Composite Measures from the Survey of Beneficiary Satisfaction with QIOs

10th SOW and Picker Institute Principles	Composite Label and Questions
Promoting effective coordination of care including helping communities support better health; transitions and continuity	Coordination Q13. The QIO representative talked with you about programs and services in your community that were available to help you with your health and wellbeing.
Information, communication and education	Beneficiary-Centered Communication Q7. How satisfied were you that the QIO representative explained things in a way you could understand? Q8. How satisfied were you that the QIO representative spent enough time with you? Q9. How satisfied were you that the QIO representative listened carefully to you?
	Communication of Written Materials Q15. How satisfied were you that the forms or letters you got about your [quality of care complaint / appeal] explained things in a way you could understand? Q16. How satisfied were you that the forms or letters you got about your [quality of care complaint / appeal] had all the information you needed? Q17. How satisfied were you that the forms or letters you got about your [quality of care complaint / appeal] showed respect for your concerns? Q18. How satisfied were you that the forms or letters you got about your [quality of care complaint/appeal] were consistent with the information you were told in telephone conversations with the QIO?
Emotional support and alleviation of fear and anxiety	Courtesy and Respect Q6. How satisfied were you that the QIO representative was as helpful as you thought he or she should be? Q10. How satisfied were you that the QIO representative showed respect for what you said?
Access to care	Access and Responsiveness Q11. The QIO representative was as responsive to your [quality of care complaint / appeal] as you thought he or she should be. Q12. The QIO representative understood the situation related to your [quality of care complaint / appeal].

Analysis: On a monthly basis, each QIO will receive interim data results based on data collected to date for the current round of collection. Item level frequencies will be produced for each QIO such that QIOs can review and monitor survey results from beneficiaries in their state on an on-going basis. Both QIOs and CMS will use the data to identify areas of concern with the processes and institute immediate corrective action.

On an annual basis, data collected over the previous four quarters will be used for detailed analysis and reporting. Annual analysis will include univariate analysis for each of the survey variables as well as multivariate analysis to explore drivers of beneficiary satisfaction. Analytic findings will be presented to the QIO community to help QIOs improve the process utilized in processing beneficiary appeals and complaints. Additionally, CMS will use results to help re-shape the infrastructure supporting QIOs.

Satisfaction Scoring: Standardized scoring for the Beneficiary Satisfaction Survey will permit CMS and the QIOs to assess the process used in resolving beneficiary complaints and appeals without undertaking lengthy analysis on a quarterly basis. Satisfaction scoring is linked to the QIO scope of work and on-going evaluation of QIO performance. Based on the parameters of the QIO scope of work, calculations will be performed to determine the percent of beneficiaries who report being satisfied with the complaint or appeal process. Proposed scoring methodology is presented below. All questions proposed for inclusion in the scoring calculation use the satisfaction scale, or the agreement scale. The details of each of these scales are presented below.

Satisfaction Scale:

1. Very Satisfied

2. Satisfied

3. Neither Satisfied nor Dissatisfied

4. Dissatisfied

5. Very Dissatisfied

Agreement Scale:

1. Strongly Agree

2. Agree

3. Neither Agree nor Disagree

4. Disagree

5. Strongly Disagree

Survey responses of 1 and 2 (very satisfied and satisfied; and strongly agree and agree) will be counted as 1 point. Survey responses of 3, 4 and 5 (neither satisfied nor dissatisfied; neither agree nor disagree dissatisfied and strongly dissatisfied; and disagree and strongly disagree) will be counted as 0 points, as well as missing responses will not be included in the denominator for scoring – with the goal of producing standardized satisfaction scores for QIO evaluation.

Scores will be calculated at the case level for each of the survey composites. The case level scores will then be rolled up to quarterly QIO level scores for each of the survey composites. Per the QIO 10^th Scope of Work, QIOs are to achieve the goal of 80 percent satisfied beneficiaries by the mid-point of the contract and 90 percent satisfied beneficiaries by the end of the contract.

3. Methods to Maximize Response Rates.

Efforts to maximize response rates will take many forms, including multiple contacts and survey design principles.

As described in section 2 on data collection procedures, outreach to respondents will occur over three separate mailouts. All mailouts will be sent via first class mail. Timing of the mailouts will ensure that respondents are reminded of the request for their participation in the survey.

The survey design team will use Dillman’s Tailored Design principles in preparing the survey and mailout materials. These design principles have been shown to increase response rates to mailout surveys using formatting and layout principles. Covering materials will stress the importance of the respondent’s input and the use of survey findings to improve processes and make them more patient- centered, leveraging Dillman’s social exchange theory.

The mail methodology proposed here has been used successfully on other CMS surveys (Medicare CAHPS).

4. Testing

As CMS is requesting an extension of an existing survey, we do not propose any new testing of the survey instrument and are relying on the testing done in association with the development of the existing survey.

At the beginning of the 10^th SoW, a round of cognitive testing was completed. CMS did not conduct a full pilot test of the data collection and methodology. We cognitively tested the instruments for clarity of language and responses. At the time we submitted the OMB package for the survey currently in use, we relied on experience regarding response rate on similar surveys of Medicare beneficiaries. Consequently, CMS used the first two years of the survey data to evaluate how QIOs processed cases, including how QIOs encouraged beneficiaries to agree to take the survey. CMS used satisfaction scores collected in the first two years as a process monitoring tool and not a tool used for contract evaluation. The satisfaction score data were used in year 3 for contract evaluation purposes.

5. Survey Development Consultants

The following individuals were consulted in the development of the surveys.

Table 3. Individuals Consulted During Development of Surveys

Organization	Name	Contact Information
CMS	David Russo	617.565.1310 [email protected]
CMS	Sally Berko	410.786.6211 [email protected]

Attachment 1 - CMS Medicare Beneficiary and Family Centered Care Satisfaction Survey

Quality of Care Complaints

Q1. Our records show that on [DATE] you filed a quality of care complaint about your or another person’s Medicare benefits. Is that right?

Q2. Have you received the results or findings in response to your quality of care complaint?

Q3. How satisfied are you with the results or findings in response to your quality of care complaint?
Q4. Please give us your comments on the results or findings in response to your quality of care complaint.

Q5. Did you speak to a QIO representative about your quality of care complaint?

Q6. How satisfied were you that the QIO representative was as helpful as you thought he or she should be?

Q7. How satisfied were you that the QIO representative explained things in a way you could understand?

Q8. How satisfied were you that the QIO representative spent enough time with you?

Q9. How satisfied were you that the QIO representative listened carefully to you?

Q10. How satisfied were you that the QIO representative showed respect for what you said?

Q11. The QIO representative was as responsive to your quality of care complaint as you thought he or she should be.

Q12. The QIO representative understood the situation related to your [quality of care complaint.

Q13. The QIO representative talked with you about programs and services in your community that are available to help with your health and wellbeing.

Q15. How satisfied were you that the forms or letters you got about your quality of care complaint explained things in a way you could understand?

Q16. How satisfied were you that the forms or letters you got about your quality of care complaint had all the information you needed?

Q17. How satisfied were you that the forms or letters you got about your quality of care complaint showed respect for your concerns?

Q18. How satisfied were you that the forms or letters you got about your quality of care complaint were consistent with the information you were told in telephone conversations with the QIO?

Q19. In responding to your appeal, [QIO NAME], the QIO in your state, gathered information about your quality of care complain, explained the complaint steps, and gave you the results or findings of your case. We are referring to this as the “quality of care complaint process”. Using any number from 0 to 10 where 0 is the worst quality of care complaint process possible and 10 is the best quality of care complaint process possible, what number would you use to rate the overall quality of care complaint process?
Q20. Please give us your comments on the process that was used in responding to your quality of care complaint. Include any comments you have on what worked well, and suggestions you have on ways to improve the process.

Attachment 2 - CMS Medicare Beneficiary and Family Centered Care Satisfaction Survey

Appeals.

Q1. Our records show that on [DATE] you filed an appeal about your or another person’s Medicare benefits. Is that right?

Q2. Have you received the results or findings in response to your appeal?

Q3. How satisfied are you with the results or findings in response to your appeal?
Q4. Please give us your comments on the results or findings in response to your appeal.

Q5. Did you speak to a QIO representative about your appeal?

Q6. How satisfied were you that the QIO representative was as helpful as you thought he or she should be?

Q7. How satisfied were you that the QIO representative explained things in a way you could understand?

Q8. How satisfied were you that the QIO representative spent enough time with you?

Q9. How satisfied were you that the QIO representative listened carefully to you?

Q10. How satisfied were you that the QIO representative showed respect for what you said?

Q11. The QIO representative was as responsive to your appeal as you thought he or she should be.

Q12. The QIO representative understood the situation related to your appeal.

Q13. The QIO representative talked with you about programs and services in your community that are available to help with your health and wellbeing.

Q15. How satisfied were you that the forms or letters you got about your appeal explained things in a way you could understand?

Q16. How satisfied were you that the forms or letters you got about your appeal had all the information you needed?

Q17. How satisfied were you that the forms or letters you got about your appeal showed respect for your concerns?

Q18. How satisfied were you that the forms or letters you got about your appeal were consistent with the information you were told in telephone conversations with the QIO?

Q19. In responding to your appeal, [QIO NAME], the QIO in your state, gathered information about your appeal, explained the appeal steps, and gave you the results or findings of your case. We are referring to this as the “appeal process”. Using any number from 0 to 10 where 0 is the worst appeal process possible and 10 is the best appeal process possible, what number would you use to rate the overall appeal process?
Q20. Please give us your comments on the process that was used in responding to your appeal. Include any comments you have on what worked well, and suggestions you have on ways to improve the process.

1