Revised PRA Supporting Stmt-Pt B (082812)

_{Supporting Statement – Part B}
Medicare Beneficiary and Family-Centered Satisfaction Survey

B. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The sampling and data collection methodology used for the Beneficiary Satisfaction survey has to be efficient based on the sample size, minimally burdensome for beneficiary respondents, frequent enough for use in on-going quality improvement efforts, and rigorous enough to permit for scoring and reporting at the QIO-level. To achieve all of the above described goals, CMS will collect and report the data quarterly. While CMS has included two formal evaluations of QIOs during the 10^th SOW, providing quarterly data on beneficiary satisfaction with the complaints and appeals processed will permit for interim corrective action to be taken as needed.

The sample for the 10^th SOW Beneficiary Satisfaction Survey will include beneficiaries who have filed quality of care complaints and those who have filed appeals about discharge. The proposed approach for the complaints and appeals cases are each described in this section.

Sample of Complaints Cases. The proposed sample design for beneficiaries who have filed a quality of care complaint is consistent with the design used in the 8^th and 9^th SOWs. During the 9^th SOW, an average of 478 national complaints cases was closed per quarter. A census of complaints cases were fielded for data collection in order to produce a data set of an adequate size for national analysis. There is no known reason to believe that the volume of complaints cases will change notably in the 10^th SOW and as such, a census of complainants cases are recommended for inclusion in data collection for the 10^th SOW.

The data collection methodology for the 10^th SOW relies on obtaining survey response by mail. Based on this data collection methodology, CMS anticipates obtaining a response rate of approximately 60 percent. A 60 percent response rate would be in line with the response rates obtained on other similar surveys conducted by mail, including the Medicare CAHPS survey. During the 9^th SOW, Beneficiary Satisfaction Survey data were collected using computer assisted telephone interviewing. Through this data collection, a seventy-five percent response rate was achieved. Because of the larger sample size required to include appeals cases in the data collection, a shift to a mail methodology is proposed for cost efficiency.

Table 1 presents the estimated quarterly number of complaint cases by state. The estimates are based on the average number of complaint cases over a 3-month period during the 9^th SOW. Table 1 also presents estimated completes by state, based on a 60 percent response rate. These numbers represent a census of the complaints cases.

Sample of Appeals Cases. Standardized national surveys of beneficiary satisfaction with the appeals process was not undertaken in the previous scopes of work. As a result, there is no existing sampling methodology that exists. The new appeals sampling methodology therefore is based on the most efficient means of achieving the analytic goals of providing quarterly state-level scores and a robust data set of annual data analysis. During the 9^th SOW, an average of 25,599 appeals cases was received per quarter. There is no known reason to believe that the volume of appeals cases will change notably in the 10^th SOW. While a census of the 495 quarterly complaints cases is recommended, CMS believes that a census of over 25,000 appeals cases would not be efficient or necessary, and as such, a statistical sample design is recommended.

As shown in Table 1, the sample will be drawn by state. Due to the differing volume of appeals by state, in some states with lower volumes, such as Alaska, Wyoming, Arkansas, Washington DC, Delaware and others, CMS will select a census of appeals cases. In other states including Alabama, Connecticut, California, Florida, and New York, CMS will use a simple random sample to draw 50 quarterly cases per state.

Based on a methodology that relies on data collection by mail, CMS anticipates obtaining a response rate of approximately 60 percent. There is no known reason to believe that response to the appeals survey will differ from other similar surveys including the Medicare CAHPS.

Table 1: Estimated Quarterly Numbers of Cases and Estimated Quarterly Completes

State	Appeals Estimated Quarterly Cases	Appeals Proposed Quarterly Sample	Appeals Estimated Quarterly Completes	Complaints Estimated Quarterly Cases	Complaints Proposed Quarterly Sample	Complaints Estimated Quarterly Completes
AK	4	4	2	1	1	0
AL	308	50	30	8	8	5
AR	31	31	19	9	9	5
AZ	228	50	30	7	7	4
CA	2,637	50	30	36	36	22
CO	82	50	30	14	14	8
CT	236	50	30	8	8	5
DC	11	11	7	1	1	0
DE	23	23	14	4	4	2
FL	1,473	50	30	47	47	28
GA	108	50	30	7	7	4
HI	14	14	8	3	3	2
IA	60	50	30	3	3	2
ID	41	41	25	3	3	2
IL	266	50	30	15	15	9
IN	131	50	30	7	7	4
KS	44	44	26	2	2	1
KY	81	50	30	4	4	2
LA	28	28	17	5	5	3
MA	428	50	30	6	6	4
MD	217	50	30	9	9	5
ME	28	28	17	2	2	1
MI	376	50	30	17	17	10
MN	216	50	30	9	9	5
MO	187	50	30	9	9	5
MS	61	50	30	5	5	3
MT	10	10	6	2	2	1
NC	147	50	30	2	2	1
ND	27	27	16	0	0	0
NE	95	50	30	1	1	0
NH	28	28	17	1	1	0
NJ	472	50	30	17	17	10
NM	30	30	18	5	5	3
NV	78	50	30	23	23	14
NY	1,458	50	30	45	45	27
OH	345	50	30	23	23	14
OK	36	36	22	14	14	9
OR	161	50	30	5	5	3
PA	1,146	50	30	14	14	9
PR	49	49	29	7	7	4
RI	51	50	30	3	3	2
SC	51	50	30	2	2	1
SD	18	18	11	0	0	0
TN	187	50	30	18	18	11
TX	277	50	30	27	27	16
UT	52	50	30	2	2	1
VA	251	50	30	5	5	3
VI	0	0	0	0	0	0
VT	16	16	10	0	0	0
WA	331	50	30	14	14	8
WI	176	50	30	9	9	5
WV	61	50	30	2	2	1
WY	2	2	1	0	0	0
National	12,737	50	30			0
Total	25,603	2,190	1,314	478	478	287

2. Procedures for Data Collection

The data collection methodology used for the Beneficiary Satisfaction Survey flows from the proposed sampling approach and the need for on-going data for quality improvement. Based on recent literature on survey methodology and response rates by mode, including Dillman’s Tailored Design method¹, CMS will use a data collection that is primarily mail. A mail-based methodology will achieve the goals of being efficient, effective, and minimally burdensome for beneficiary respondents. A single mode data collection will also reduce the known mode effects seen particularly in satisfaction surveys.²

Data will be collected quarterly during the QIO 10^th SOW. That is to say, data will be collected 4 times per year from the time of OMB approval through August 2014. Over the course of this period, CMS anticipates being able to conduct up to 10 rounds of data collection. The sample for each round of data collection will include appeals and complaint cases closed in the previous quarter such that no beneficiary should be sampled more than once. Re-appeals cases will not be included in the universe of eligible cases for sampling. In order for CMS and the QIOs to assess the degree to which patient-centered care is being delivered through the redesigned 10^th SOW processes required obtaining on-going data from a small sample of beneficiaries. The on-going survey results will be used to implement mid-stream corrections to processes as needed.

Since data will be collected and reported for internal quality improvement on a quarterly basis, the data collection methodology must strive to minimize the data collection field period while maximizing the response rate. The desired data collection field period is 8 to 10 weeks. To achieve these goals, CMS will format the survey materials per Dillman’s Tailored Design principles and use a three-staged approach to data collection:

1. Mailout of a covering letter, the paper survey questionnaire, and a postage-paid return envelope.

2. Mailout of a post card that thanks respondents and reminds the non-respondents to please return their survey.

3. Mailout of a follow-up covering letter, the paper survey questionnaire, and a postage-paid return envelope.

First a pilot study will be conducted (described in detail in section 4). Through the pilot test, CMS will determine the response rate that can be achieved using this approach. If it is deemed necessary, a prenotification letter or additional mailout reminders can be added to the protocol, a telephone non-response step can be added to the protocol as needed to achieve the desired response rate. Additional information on maximization of response rates is included in section 3.

Using the 3-step mail approach described above, CMS anticipates that data collection would occur over an 8 to 10 week period. This is to say, if the first survey mailing were dropped on January 1, we would anticipate completing data collection at the end of February or early March. Data would then be cleaned, scores would be generated, and data would be delivered for CMS and QIO quality improvement review. The precise timing required to achieve an acceptable response rate will be determined through the pilot test. The aim is to complete sampling, data collection complete and scoring within a 12-week period.

Survey Material: The QIO 10^th SOW includes a strong focus on making all processes beneficiary and family focused, in line with the principles outlined by the Picker Institute. To support that focus, the Beneficiary Satisfaction survey will capture beneficiary satisfaction with the appeals review process as well as quality of care complaints.

Table 2 provides a summary of the Survey composites and questions. Each of the survey composites represents an important aspect of patient-centeredness. Data from the survey will be used by CMS and the QIOs to support the quality improvement effort, make changes to improve processes, and review overall success of each QIO at executing the complaints and appeals process in a manor aligned with CMS goals for patient and family centeredness.

The questionnaire will be available in English and Spanish. See Attachment 1 for a copy of the draft questionnaire as submitted with the 60-day Federal Register Notice. Some modifications were made to the questionnaire as a result of cognitive testing and noted changes in the procedures used by CMS to process appeals and complaints under the 10^th SOW. The revised questionnaire and a crosswalk between the draft and revised versions of the questionnaire are included as attachments 2 and 3 respectively.

Table 2: Composite Measures from the Survey of Beneficiary Satisfaction with QIOs

10th SOW and Picker Institute Principles	Composite Label and Questions
Promoting effective coordination of care including helping communities support better health; transitions and continuity	Coordination Q13. The QIO representative talked with you about programs and services in your community that were available to help you with your health and wellbeing.
Information, communication and education	Beneficiary-Centered Communication Q7. How satisfied were you that the QIO representative explained things in a way you could understand? Q8. How satisfied were you that the QIO representative spent enough time with you? Q9. How satisfied were you that the QIO representative listened carefully to you?
	Communication of Written Materials Q15. How satisfied were you that the forms or letters you got about your [quality of care complaint / appeal] explained things in a way you could understand? Q16. How satisfied were you that the forms or letters you got about your [quality of care complaint / appeal] had all the information you needed? Q17. How satisfied were you that the forms or letters you got about your [quality of care complaint / appeal] showed respect for your concerns? Q18. How satisfied were you that the forms or letters you got about your [quality of care complaint/appeal] were consistent with the information you were told in telephone conversations with the QIO?
Emotional support and alleviation of fear and anxiety	Courtesy and Respect Q6. How satisfied were you that the QIO representative was as helpful as you thought he or she should be? Q10. How satisfied were you that the QIO representative showed respect for what you said?
Access to care	Access and Responsiveness Q11. The QIO representative was as responsive to your [quality of care complaint / appeal] as you thought he or she should be. Q12. The QIO representative understood the situation related to your [quality of care complaint / appeal].

Analysis: On a monthly basis, each QIO will receive interim data results based on data collected to date for the current round of collection. Item level frequencies will be produced for each QIO such that QIOs can review and monitor survey results from beneficiaries in their state on an on-going basis. Both QIOs and CMS will use the data to identify areas of concern with the processes and institute immediate corrective action.

On an annual basis, data collected over the previous four quarters will be used for detailed analysis and reporting. Annual analysis will include univariate analysis for each of the survey variables as well as multivariate analysis to explore drivers of beneficiary satisfaction. Analytic findings will be presented to the QIO community to help QIOs improve the process utilized in processing beneficiary appeals and complaints. Additionally, CMS will use results to help re-shape the infrastructure supporting QIOs.

Satisfaction Scoring: Standardized scoring for the Beneficiary Satisfaction Survey will permit CMS and the QIOs to assess the process used in resolving beneficiary complaints and appeals without undertaking lengthy analysis on a quarterly basis. Satisfaction scoring is linked to the QIO scope of work and on-going evaluation of QIO performance. Based on the parameters of the QIO scope of work, calculations will be performed to determine the percent of beneficiaries who report being satisfied with the complaint or appeal process. Proposed scoring methodology is presented below. All questions proposed for inclusion in the scoring calculation use the satisfaction scale, or the agreement scale. The details of each of these scales are presented below.

Satisfaction Scale:

1. Very Satisfied

2. Satisfied

3. Neither Satisfied nor Dissatisfied

4. Dissatisfied

5. Very Dissatisfied

Agreement Scale:

1. Strongly Agree

2. Agree

3. Neither Agree nor Disagree

4. Disagree

5. Strongly Disagree

Survey responses of 1 and 2 (very satisfied and satisfied; and strongly agree and agree) will be counted as 1 point. Survey responses of 3, 4 and 5 (neither satisfied nor dissatisfied; neither agree nor disagree dissatisfied and strongly dissatisfied; and disagree and strongly disagree) will be counted as 0 points, as well as missing responses will not be included in the denominator for scoring – with the goal of producing standardized satisfaction scores for QIO evaluation.

Scores will be calculated at the case level for each of the survey composites. The case level scores will then be rolled up to quarterly QIO level scores for each of the survey composites. Per the QIO 10^th Scope of Work, QIOs are to achieve the goal of 80 percent satisfied beneficiaries by the mid-point of the contract and 90 percent satisfied beneficiaries by the end of the contract.

3. Methods to Maximize Response Rates.

Efforts to maximize response rates will take many forms, including multiple contacts and survey design principles.

As described in section 2 on data collection procedures, outreach to respondents will occur over three separate mailouts. All mailouts will be sent via first class mail. Timing of the mailouts will ensure that respondents are reminded of the request for their participation in the survey.

The survey design team will use Dillman’s Tailored Design principles in preparing the survey and mailout materials. These design principles have been shown to increase response rates to mailout surveys using formatting and layout principles. Covering materials will stress the importance of the respondent’s input and the use of survey findings to improve processes and make them more patient-centered, leveraging Dillman’s social exchange theory.

The mail methodology proposed here has been used successfully on other CMS surveys (Medicare CAHPS).

4. Testing

Testing for the Beneficiary Satisfaction Survey will occur over a series of steps including a small number of cognitive interviews and a pilot test. Testing will have the following goals:

1. Determine if the survey wording is clear and unambiguous;

2. Verify respondent’s ability to recall interactions pertaining to their complaints and appeals cases;

3. Ensure appropriate and consistent flow question wording and overall survey administration;

4. Ensure data capture and data output are functioning flawlessly;

5. Provide early scoring and data output to CMS for review and consideration.

An initial round of testing interviews was conducted to help understand the language that beneficiaries use to describe their experience with filing a quality of care complaint or appeal. It is important to understand beneficiary’s frame of reference so that questions can appropriately tap into this framing.

A round of cognitive testing was conducted after the beginning of the 10^th SOW. Some changes in how cases are processed were implemented with the start of the 10^th SOW. It was important to conduct the cognitive testing to ensure that the revised survey is capturing all important elements of the 10^th SOW process, from the beneficiary’s perspective. The cognitive testing included 9 beneficiaries.

A pilot test of the survey and full survey operations will be conducted to learn about how the questionnaire and data collection methodologies perform. Through the pilot test, CMS will obtain a data set large enough to assess the psychometric properties of the survey questions. For example, if it is determined that there is little or no variability in how respondents answer a particular question, CMS can consider revising or dropping that question. Additionally, if many respondents use white space in the margin of the questionnaire to insert comments, CMS can determine if additional survey content or response categories should be added.

Through the pilot test, the CMS contractor will determine the actual response rate that should be expected, and will work with CMS to augment the response rate using other modes of contact as necessary.

Finally, the pilot test will permit for a full-scale test of the data capture and data reporting functionality. CMS may choose to use the pilot test data as the first set of data and scores reported to the QIOs if no substantial change in the survey or data collection methodology is required as a result of what is learned through the testing. The proposed sampling methodology and sample size for the pilot testing would be consistent with what is proposed for the quarterly data collection through the rest of the 10^th SOW.

5. Survey Development Consultants

The following individuals were consulted in the development of the surveys.

Table 3. Individuals Consulted During Development of Surveys

Organization	Name	Contact Information
CMS	Robert Kambic	410-786-1515 [email protected]
CMS	Coles Mercier	410-786-2112 [email protected]
Westat	W. Sherman Edwards	301-294-3993 [email protected]
Westat	Vasudha Narayanan	301-294-3808 [email protected]
Westat	Stephanie Fry	301-294-2872 [email protected]

Attachment 1: Draft Questionnaire

CMS Medicare Beneficiary and Family Centered Care Satisfaction Survey

Your Medicare [Quality of Care Complaint / Benefits Appeal]

1. Our records show that on [DATE] you filed [a complaint about the quality of care you or another person received under Medicare / an appeal about your or another person’s Medicare benefits]. Is that right?

• Yes

• N o If No, please return the survey in the postage-paid envelope.

The questions in this survey will refer to the [Medicare quality of care complaint that you filed on the date shown in Question 1 as “your quality of care complaint” / Medicare benefits appeal that you filed on the date shown in Question 1 as “your appeal”.

2. Have you gotten a resolution on your [quality of care complaint / appeal]?

• Yes

• N o If No, please return the survey in the postage-paid envelope.

3. How satisfied are you with the resolution of your [quality of care complaint / appeal]?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

4. Please give us your comments on the resolution of your [quality of care complaint / appeal].

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

Interactions with the Intake Specialist

The next questions are about the way your [quality of care complaint / appeal] was handled from the start. The questions will refer to the person you first spoke with when you called to file your [quality of care complaint / appeal] as the “Intake Specialist”. The Intake Specialist would have collected the details about your [quality of care complaint / appeal].

5. When you spoke with the Intake Specialist, how satisfied were you that he or she was as helpful as you thought they should be?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

6. When you spoke with the Intake Specialist, how satisfied were you that he or she explained things in a way you could understand?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

7. When you spoke with the Intake Specialist, how satisfied were you that he or she spent enough time with you?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

8. When you spoke with the Intake Specialist, how satisfied were you that he or she listened carefully to you?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

9. When you spoke with the Intake Specialist, how satisfied were you that he or she showed respect for what you had to say?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

Interactions with the Case Manager

The next questions will refer to the person who called you back after your [quality of care complaint / appeal] was filed as the “Case Manager”. The Case Manager would have contacted you about the resolution of your [quality of care complaint / appeal].

10. Did you speak to a Case Manager about your [quality of care complaint / appeal] on the phone?

• Yes

• N o If No, go to Q20.

How much do you agree or disagree with the following statements:

11. The Case Manager had all the information that you gave to the Intake Specialist about your [quality of care complaint / appeal].

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

12. The Case Manager was as responsive to your [quality of care complaint / appeal] as you thought they should be.

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

13. The Case Manager understood your situation.

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

14. The Case Manager talked with you about resources that were available to help you.

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

15. When you spoke with the Case Manager, how satisfied were you that he or she was as helpful as you thought they should be?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

16. When you spoke with the Case Manager, how satisfied were you that he or she explained things in a way you could understand?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

17. When you spoke with the Case Manager, how satisfied were you that he or she spent enough time with you?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

18. When you spoke with the Case Manager, how satisfied were you that he or she listened carefully to you?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

19. When you spoke with the Case Manager, how satisfied were you that he or she showed respect for what you had to say?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

Letter(s) about your [Quality of Care Complaint / Appeal]

20. Did you receive any letters about your [quality of care complaint / appeal]?

• Yes

• N o If No, go to Q25.

21. How satisfied were you that the letter(s) you got about your [quality of care complaint / appeal] explained things in a way you could understand?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

22. How satisfied were you that the letter(s) you got about your [quality of care complaint / appeal] contained all the information you needed?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

23. How satisfied were you that the letter(s) you got about your [quality of care complaint / appeal] showed respect for your concerns?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

24. How satisfied were you that the letter(s) you got about your [quality of care complaint / appeal] had the same information that you were told in telephone conversations?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

Overall [Quality of Care Complaint / Appeal] Process

25. Using any number from 0 to 10 where 0 is the worst [quality of care complaint / appeal] process possible and 10 is the best [quality of care complaint / appeal] process possible, what number would you use to rate the overall [quality of care complaint / appeal] process?

• 0 – Worst process possible

• 1

• 2

• 3

• …

• 8

• 9

• 10 – Best process possible

26. Please give us your comments on the process that was used to resolve your [quality of care complaint / appeal]. Include any comments you have on what worked well, and suggestions you have on ways to improve how the process.

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

Thank you: Those are all the questions we have for you now.

Attachment 2: Revised Questionnaires

CMS Medicare Beneficiary and Family Centered Care Satisfaction Survey
Quality of Care Complaint

Your Medicare Quality of Care Complaint

1. Our records show that on [DATE] you filed a complaint about the quality of care you or another person received under Medicare. Is that right?

• Yes

• N o If No, please return the survey in the postage-paid envelope.

The questions in this survey refer to the Medicare quality of care complaint that you filed on the date shown in Question 1 as “your quality of care complaint”.

2. Have you received the results or findings in response to your quality of care complaint?

• Yes

• N o If No, go to #4

3. How satisfied are you with the results or findings in response to your quality of care complaint?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

4. Please give us your comments on the results or findings in response to your quality of care complaint and concerns.

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

Quality of Care Complaint Process

The next questions are about the way your quality of care complaint was handled and the process that the Quality Improvement Organization (QIO) in your state used to get information and coordinate the steps in the process.

The questions will refer to the representative from the QIO in your state as the “QIO representative”. You may have spoken to the QIO representative when you filed your quality of care complaint, or in a follow-up conversation after your quality of care complaint was filed.

5. Did you speak to a QIO representative about your quality of care complaint?

• Yes

• N o If No, go to #14

6. How satisfied were you that the QIO representative was as helpful as you thought he or she should be?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

7. How satisfied were you that the QIO representative explained things in a way you could understand?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

8. How satisfied were you that the QIO representative spent enough time with you?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

9. How satisfied were you that the QIO representative listened carefully to you?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

10. How satisfied were you that the QIO representative showed respect for what you said?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

How much do you agree or disagree with the following statements:

11. The QIO representative was as responsive to your quality of care complaint as you thought he or she should be.

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

12. The QIO representative understood the situation related to your quality of care complaint.

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

13. The QIO representative talked with you about programs and services in your community that are available to help you with your health and wellbeing.

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

Letter(s) about your Quality of Care Complaint

14. Did you get any forms or letters from the Centers for Medicare & Medicaid Services or the QIO about your quality of care complaint?

• Yes

• N o If No, go to #19

15. How satisfied were you that the forms or letters you got about your quality of care complaint explained things in a way you could understand?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

16. How satisfied were you that the forms or letters you got about your quality of care complaint had all the information you needed?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

17. How satisfied were you that the forms or letters you got about your quality of care complaint showed respect for your concerns?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

18. How satisfied were you that the forms or letters you got about your quality of care complaint were consistent with the information you were told in telephone conversations with the QIO?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

• I did not have any telephone conversations with the QIO

Overall Quality of Care Complaint Process

19. In responding to your quality of care complaint the QIO gathered information about your quality of care complaint, explained the complaint steps, and gave you the results or findings of your case. We are referring to this as the “quality of care complaint process”. Using any number from 0 to 10 where 0 is the worst quality of care complaint process possible, and 10 is the best quality of care complaint process possible, what number would you use to rate the overall quality of care complaint process?

• 0 – Worst process possible

• 1

• 2

• 3

• 4

• 5

• 6

• 7

• 8

• 9

• 10 – Best process possible

20. Please give us your comments on the process that was used in responding to your quality of care complaint. Include any comments you have on what worked well, and suggestions you have on ways to improve the process.

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

Thank you for your participation.

CMS Medicare Beneficiary and Family Centered Care Satisfaction Survey
Appeal

Your Medicare Appeal

1. Our records show that on [DATE] you filed an appeal about your or another person’s Medicare benefits. Is that right?

• Yes

• N o If No, please return the survey in the postage-paid envelope.

The questions in this survey will refer to the Medicare appeal that you filed on the date shown in Question 1 as “your appeal”.

2. Have you received the results or findings in response to your appeal?

• Yes

• N o If No, go to #4

3. How satisfied are you with the results or findings in response to your appeal?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

4. Please give us your comments on the results or findings in response to your appeal.

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

Appeal Process

The next questions are about the way your appeal was handled and the process that the Quality Improvement Organization (QIO) in your state used to get information and coordinate the steps in the appeal process.

The questions will refer to the representative from the QIO in your state as the “QIO representative”. You may have spoken to the QIO representative when you filed your appeal, or in a follow-up conversation after your appeal was filed.

5. Did you speak to a QIO representative about your appeal?

• Yes

• N o If No, go to #14

6. How satisfied were you that the QIO representative was as helpful as you thought he or she should be?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

7. How satisfied were you that the QIO representative explained things in a way you could understand?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

8. How satisfied were you that the QIO representative spent enough time with you?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

9. How satisfied were you that the QIO representative listened carefully to you?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

10. How satisfied were you that the QIO representative showed respect for what you said?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

How much do you agree or disagree with the following statements:

11. The QIO representative was as responsive to your appeal as you thought he or she should be.

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

12. The QIO representative understood the situation related to your appeal.

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

13. The QIO representative talked with you about programs and services in your community that are available to help you with your health and wellbeing.

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

Letter(s) about your Appeal

14. Did you get any forms or letters from the Centers for Medicare & Medicaid Services or the QIO about your appeal?

• Yes

• N o If No, go to #19

15. How satisfied were you that the forms or letters you got about your appeal explained things in a way you could understand?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

16. How satisfied were you that the forms or letters you got about your appeal had all the information you needed?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

17. How satisfied were you that the forms or letters you got about your appeal showed respect for your concerns?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

18. How satisfied were you that the forms or letters you got about your appeal were consistent with the information you were told in telephone conversations with the QIO?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

• I did not have any telephone conversations with the QIO

Overall Appeal Process

19. In responding to your appeal the QIO gathered information about your appeal, explained the appeal steps, and gave you the results or findings of your case. We are referring to this as the “appeal process”. Using any number from 0 to 10 where 0 is the worst appeal process possible and 10 is the best appeal process possible, what number would you use to rate the overall appeal process?

• 0 – Worst process possible

• 1

• 2

• 3

• 4

• 5

• 6

• 7

• 8

• 9

• 10 – Best process possible

20. Please give us your comments on the process that was used in responding to your appeal. Include any comments you have on what worked well, and suggestions you have on ways to improve the process.

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

Thank you for your participation.

Attachment 3. CMS Medicare Beneficiary and Family Centered Care Satisfaction Survey Revision Crosswalk

Original Question Number	Revised Question Number	Original Wording	Revised Beneficiary Complaint Survey Wording	Revised Beneficiary Appeals Survey Wording	Rationale for Revision
Q1	Q1	Our records show that on [DATE] you filed [a complaint about the quality of care you or another person received under Medicare / an appeal about your or another person’s Medicare benefits]. Is that right?	Our records show that on [DATE] you filed a complaint about the quality of care you or another person received under Medicare. Is that right?	Our records show that on [DATE] you filed an appeal about your or another person’s Medicare benefits. Is that right?	No change – shown with 2 variations (complaint and appeal)
Intro to Q2	Intro to Q2	The questions in this survey will refer to the [Medicare quality of care complaint that you filed on the date shown in Question 1 as “your quality of care complaint” / Medicare benefits appeal that you filed on the date shown in Question 1 as “your appeal”].	The questions in this survey refer to the Medicare quality of care complaint that you filed on [DATE] as “your quality of care complaint”.	The questions in this survey refer to the Medicare appeal that you filed on [DATE] as “your appeal”.	Simplified language
Q2	Q2	Have you gotten a resolution on your quality of care complaint / appeal? If No, please return the survey in the postage-paid envelope.	Have you received the results or findings in response to your quality of care complaint? If No, go to #4.	Have you received the results or findings in response to your appeal? If No, go to #4.	Improved specificity of question Based on cognitive testing findings, respondent could describe their experience even if they didn’t report having final findings of their case
Q3	Q3	How satisfied are you with the resolution of your [quality of care complaint / appeal]?	How satisfied are you with the results or findings in response to your quality of care complaint?	How satisfied are you with the results or findings in response to your appeal?	Improved specificity of question
Q4	Q4	Please give us your comments on the resolution of your [quality of care complaint / appeal].	Please give us your comments on the results or findings in response to your quality of care complaint and concerns.	Please give us your comments on the results or findings in response to your appeal.	Improved specificity of question
Intro to Q5	Intro to Q5	Interactions with the Intake Specialist The next questions are about the way your [quality of care complaint/appeal] was handled from the start. The questions will refer to the person you first spoke with when you called to file your [quality of care complaint/appeal] as the “Intake Specialist”. The Intake Specialist would have collected the details about your [quality of care complaint/appeal].	Quality of Care Complaint Process These next questions are about the way your quality of care complaint was handled and the process that [QIO NAME], the Quality Improvement Organization (QIO) in your state used to get information and coordinate the steps in the process. The questions will refer to the representative from [QIO NAME], the QIO in your state as the “QIO representative”. You may have spoken to the QIO representative when you filed your quality of care complaint, or in a follow-up conversation after your quality of care complaint was filed.	Appeal Process These next questions are about the way your appeal was handled and the process that [QIO NAME], the Quality Improvement Organization (QIO) in your state used to get information and coordinate the steps in the appeal process. The questions will refer to the representative from [QIO NAME], the QIO in your state as the “QIO representative”. You may have spoken to the QIO representative when you filed your appeal, or in a follow-up conversation after your appeal was filed.	Changed reference to QIO representative “from the start” removed based on cognitive testing. Respondent could consider the “start” to be significantly before s/he filed a complaint/appeal
Q5	-	When you spoke with the Intake Specialist, how satisfied were you that he or she was as helpful as you thought they should be?	REMOVED FROM SURVEY	REMOVED FROM SURVEY	Not measuring interactions with “Intake Specialist” given CMS change in processing of cases
Q6	-	When you spoke with the Intake Specialist, how satisfied were you that he or she explained things in a way you could understand?	REMOVED FROM SURVEY	REMOVED FROM SURVEY	Not measuring interactions with “Intake Specialist” given CMS change in processing of cases
Q7	-	When you spoke with the Intake Specialist, how satisfied were you that he or she spent enough time with you?	REMOVED FROM SURVEY	REMOVED FROM SURVEY	Not measuring interactions with “Intake Specialist” given CMS change in processing of cases
Q8	-	When you spoke with the Intake Specialist, how satisfied were you that he or she listened carefully to you?	REMOVED FROM SURVEY	REMOVED FROM SURVEY	Not measuring interactions with “Intake Specialist” given CMS change in processing of cases
Q9	-	When you spoke with the Intake Specialist, how satisfied were you that he or she showed respect for what you had to say?	REMOVED FROM SURVEY	REMOVED FROM SURVEY	Not measuring interactions with “Intake Specialist” given CMS change in processing of cases
Intro to Q10	-	Interactions with the Case Manager The next questions will refer to the person who called you back after your [quality of care complain / appeal was filed as the “Case Manager”. The Case Manager would have contacted you about the resolution of your [quality of care complaint / appeal].	REMOVED FROM SURVEY	REMOVED FROM SURVEY	No longer distinguishing between Case Manager and Intake Specialist given CMS change in processing of cases
Q10	Q5	Did you speak to a Case Manager about your [quality of care complaint / appeal] on the phone? If No, go to Q20	Did you speak to a QIO representative about your quality of care complaint? If No, go to #14	Did you speak to a QIO representative about your appeal? If No, go to #14	Changed reference to QIO representative Removed “on the phone” as process does not require telephone follow-up in all cases
Q11	-	The Case Manager had all the information that you gave to the Intake Specialist about you [quality of care complaint/appeal].	REMOVED FROM SURVEY	REMOVED FROM SURVEY	No longer distinguishing between Case Manager and Intake Specialist given CMS change in processing of cases
Q12	Q11	The Case Manager was as responsive to your [quality of care complaint / appeal] as you thought they should be.	The QIO representative was as responsive to your quality of care complaint as you thought he or she should be.	The QIO representative was as responsive to your appeal as you thought he or she should be.	Changed reference to QIO representative Improved grammar
Q13	Q12	The Case Manager understood your situation.	The QIO representative understood the situation related to your quality of care complaint.	The QIO representative understood the situation related to your appeal.	Changed reference to QIO representative Improved specificity of question
Q14	Q13	The Case Manager talked with you about resources that were available to help you.	The QIO representative talked with you about programs and services in your community that are available to help you with your health and wellbeing.	The QIO representative talked with you about programs and services in your community that are available to help you with your health and wellbeing.	Changed reference to QIO representative Improved specificity of question. Based on cognitive testing findings, beneficiaries did not have a clear and consistent understanding of the term “resources”
Q15	Q6	When you spoke with the Case Manager, how satisfied were you that he or she was as helpful as you thought they should be?	How satisfied were you that the QIO representative was as helpful as you thought he or she should be?	How satisfied were you that the QIO representative was as helpful as you thought he or she should be?	Changed reference to QIO representative
Q16	Q7	When you spoke with the Case Manager, how satisfied were you that he or she explained things in a way you could understand?	How satisfied were you that the QIO representative explained things in a way you could understand?	How satisfied were you that the QIO representative explained things in a way you could understand?	Changed reference to QIO representative
Q17	Q8	When you spoke with the Case Manager, how satisfied were you that he or she spent enough time with you?	How satisfied were you that the QIO representative spent enough time with you?	How satisfied were you that the QIO representative spent enough time with you?	Changed reference to QIO representative
Q18	Q9	When you spoke with the Case Manager, how satisfied were you that he or she listened carefully to you?	How satisfied were you that the QIO representative listened carefully to you?	How satisfied were you that the QIO representative listened carefully to you?	Changed reference to QIO representative
Q19	Q10	When you spoke with the Case Manager, how satisfied were you that he or she showed respect for what you had to say?	How satisfied were you that the QIO representative showed respect for what you said?	How satisfied were you that the QIO representative showed respect for what you said?	Changed reference to QIO representative Simplified language
Q20	Q14	Did you receive any letters about your [quality of care complaint / appeal]? If No, go to Q25	Did you get any forms or letters from the Centers for Medicare & Medicaid Services or the QIO about your quality of care complaint? If No, go to #19	Did you get any forms or letters from the Centers for Medicare & Medicaid Services or the QIO about your appeal? If No, go to #19	Simplified language Improved specificity of question Added “forms” based on cognitive testing findings: beneficiaries referred to written materials they received as letter, form letters, and forms
Q21	Q15	How satisfied were you that the letter(s) you got about your [quality of care complaint / appeal] explained things in a way you could understand?	How satisfied were you that the forms or letters you got about your quality of care complaint explained things in a way you could understand?	How satisfied were you that the forms or letters you got about your appeal explained things in a way you could understand?	Added “forms” based on cognitive testing findings: beneficiaries referred to written materials they received as letter, form letters, and forms
Q22	Q16	How satisfied were you that the letter(s) you got about your [quality of care complaint / appeal] contained all the information you needed?	How satisfied were you that the forms or letters you got about your quality of care complaint had all the information you needed?	How satisfied were you that the forms or letters you got about your appeal had all the information you needed?	Added “forms” based on cognitive testing findings: beneficiaries referred to written materials they received as letter, form letters, and forms Simplified language
Q23	Q17	How satisfied were you that the letter(s) you got about your [quality of care complaint / appeal] showed respect for your concerns?	How satisfied were you that the forms or letters you got about your quality of care complaint showed respect for your concerns?	How satisfied were you that the forms or letters you got about your appeal showed respect for your concerns?	Added “forms” based on cognitive testing findings: beneficiaries referred to written materials they received as letter, form letters, and forms
Q24	Q18	How satisfied were you that the letter(s) you got about your [quality of care complaint / appeal] had the same information that you were told in telephone conversations? Very satisfied Satisfied Neither satisfied nor dissatisfied Dissatisfied Very dissatisfied	How satisfied were you that the forms or letters you got about your quality of care complaint were consistent with the information you were told in telephone conversations with the QIO? Very satisfied Satisfied Neither satisfied nor dissatisfied Dissatisfied Very dissatisfied I did not have any telephone conversations with the QIO.	How satisfied were you that the forms or letters you got about your appeal were consistent with the information you were told in telephone conversations with the QIO? Very satisfied Satisfied Neither satisfied nor dissatisfied Dissatisfied Very dissatisfied I did not have any telephone conversations with the QIO.	Added “forms” based on cognitive testing findings: beneficiaries referred to written materials they received as letter, form letters, and forms Improved specificity of question Added a tailored inapplicable response based on cognitive testing findings.
Q25	Q19	Using any number from 0 to 10 where 0 is the word [quality of care complaint / appeal] process possible and 10 is the best [quality of care complaint / appeal] process possible, what number would you use to rate the overall [quality of care complaint / appeal] process?	In responding to your quality of care complaint [QIO NAME], the QIO in your state gathered information about your quality of care complaint, explained the complaint steps, and gave you the results or findings of your case. We are referring to this as the “quality of care complaint process”. Using any number from 0 to 10 where 0 is the word quality of care complaint process possible and 10 is the best quality of care complaint process possible, what number would you use to rate the overall quality of care complaint process?	In responding to your appeal [QIO NAME], the QIO in your state gathered information about your appeal, explained the appeal steps, and gave you the results or findings of your case. We are referring to this as the “appeal process”. Using any number from 0 to 10 where 0 is the word appeal process possible and 10 is the best appeal process possible, what number would you use to rate the overall appeal process?	Added clarification on definition of process to improve specificity of question
Q26	Q20	Please give us your comments on the process that was used to resolve your [quality of care complaint / appeal]. Include any comments you have on what worked well, and suggestions you have on ways to improve how the process.	Please give us your comments on the process that was used in responding to your quality of care complaint. Include any comments you have on what worked well, and suggestions you have on ways to improve the process.	Please give us your comments on the process that was used in responding to your quality of care complaint. Include any comments you have on what worked well, and suggestions you have on ways to improve the process.	Simplified language
Thank you	Thank you	Thank you: Those are all the questions we have for you now	Thank you for your participation.	Thank you for your participation.	Simplified language

1Dillman, D. A. (2007). Mail and Internet Surveys; the tailored design method. New Jersey, United States: John Wiley & Sons Inc.

2Dillman, D. A., Sangster, R. L., Tarnai, J. and Rockwood, T. H. (1996), Understanding differences in people’s answers to telephone and mail surveys. New Directions for Evaluation, 1996: 45–61. doi: 10.1002/ev.1034

47