Bene Satis Survey-PRA-Supporting Stmt Pt B

Supporting Statement – Part B

Medicare Beneficiary and Family-Centered Satisfaction Survey

The sampling and data collection methodology used for the Beneficiary Satisfaction survey has to be efficient based on the sample size, minimally burdensome for beneficiary respondents, frequent enough for use in on-going quality improvement efforts, and rigorous enough to permit for scoring and reporting at the QIO-level. To achieve all of the above described goals, CMS will collect and report the data quarterly. While CMS has included two formal evaluations of QIOs during the 10^th SOW, providing quarterly data on beneficiary satisfaction with the complaints and appeals processed will permit for interim corrective action to be taken as needed.

The sample for the 10^th SOW Beneficiary Satisfaction Survey will include beneficiaries who have filed quality of care complaints and those who have filed appeals about discharge. The proposed approach for the complaints and appeals cases are each described in this section.

Sample of Complaints Cases. The proposed sample design for beneficiaries who have filed a quality of care complaint is consistent with the design used in the 8^th and 9^th SOWs. During the 9^th SOW, an average of 478 national complaints cases was closed per quarter. A census of complaints cases were fielded for data collection in order to produce a data set of an adequate size for national analysis. There is no known reason to believe that the volume of complaints cases will change notably in the 10^th SOW and as such, a census of complainants cases are recommended for inclusion in data collection for the 10^th SOW.

The data collection methodology for the 10^th SOW relies on obtaining survey response by mail. Based on this data collection methodology, we anticipate obtaining a response rate of approximately 60 percent. A 60 percent response rate would be in line with the response rates obtained on other similar surveys conducted by mail, including the Medicare CAHPS survey. During the 9^th SOW, Beneficiary Satisfaction Survey data were collected using computer assisted telephone interviewing. Through this data collection, a seventy-five percent response rate was achieved. Because of the larger sample size required to include appeals cases in the data collection, a shift to a mail methodology is proposed for cost efficiency.

Table 1 presents the estimated quarterly number of complaint cases by state. The estimates are based on the average number of complaint cases over a 3-month period during the 9^th SOW. Table 1 also presents estimated completes by state, based on a 60 percent response rate. These numbers represent a census of the complaints cases.

Sample of Appeals Cases. Standardized national surveys of beneficiary satisfaction with the appeals process was not undertaken in the previous scopes of work. As a result, there is no existing sampling methodology that exists. The new appeals sampling methodology therefore is based on the most efficient means of achieving the analytic goals of providing quarterly state-level scores and a robust data set of annual data analysis. During the 9^th SOW, an average of 25,599 appeals cases was received per quarter. There is no known reason to believe that the volume of appeals cases will change notably in the 10^th SOW. While a census of the 495 quarterly complaints cases is recommended, we believe that a census of over 25,000 appeals cases would not be efficient or necessary, and as such, a statistical sample design is recommended.

As shown in Table 1, the sample will be drawn by State. Due to the differing volume of appeals by state, in some states with lower volumes, such as Alaska, Wyoming, Arkansas, Washington DC, Delaware and others, we will select a census of appeals cases. In other states including Alabama, Connecticut, California, Florida, and New York, we will use a simple random sample to draw 50 quarterly cases per state.

Based on a methodology that relies on data collection by mail, we anticipate obtaining a response rate of approximately 60 percent. There is no known reason to believe that response to the appeals survey will differ from other similar surveys including the Medicare CAHPS.

Procedures for Data Collection

The data collection methodology used for the Beneficiary Satisfaction Survey flows from the proposed sampling approach and the need for on-going data for quality improvement. Based on recent literature on survey methodology and response rates by mode, including Dillman’s Tailored Design method¹, we recommend using a data collection that is primarily mail. A mail-based methodology will achieve the goals of being efficient, effective, and minimally burdensome for beneficiary respondents. A single mode data collection will also reduce the known mode effects seen particularly in satisfaction surveys².

Data will be collected quarterly during the QIO 10^th SOW. That is to say, data will be collected 4 times per year from the time of OMB approval through August 2014. Over the course of this period, we anticipate being able to conduct 10 rounds of data collection. The sample for each round of data collection will include appeals and complaint cases closed in the previous quarter such that no beneficiary should be sampled more than once. Re-appeals cases will not be included in the universe of eligible cases for sampling. In order to CMS and the QIOs to assess the degree to which patient-centered care is being delivered through the redesigned 10^th SOW processes required obtaining on-going data from a small sample of beneficiaries. The on-going survey results will be used to implement mid-stream corrections to processes as needed.

Since data will be collected and reported for internal quality improvement on a quarterly basis, the data collection methodology must strive to minimize the data collection field period while maximizing the response rate. The desired data collection field period is 8 to 10 weeks. To achieve these goals, we would format the survey materials per Dillman’s Tailored Design principles and use a three-staged approach to data collection:

1. Mailout of a covering letter, the paper survey questionnaire, and a postage-paid return envelope.

2. Mailout of a post card that thanks respondents and reminds the non-respondents to please return their survey.

3. Mailout of a follow-up covering letter, the paper survey questionnaire, and a postage-paid return envelope.

We will first conduct a pilot study (described in detail in section 4). Through the pilot test, we will determine the response rate that can be achieved using this approach. If it is deemed necessary, a prenotification letter or additional mailout reminders can be added to the protocol, a telephone non-response step can be added to the protocol as needed to achieve the desired response rate. Additional information on maximization of response rates is included in section 3.

Using the 3-step mail approach described above, we anticipate that data collection would occur over an 8 to 10 week period. This is to say, if the first survey mailing were dropped on January 1, we would anticipate completing data collection at the end of February or early March. Data would then be cleaned, scores would be generated, and data would be delivered to CMS for CMS and QIO quality improvement review. Through the pilot test, we will determine the precise timing required to achieve an acceptable response rate, but we are aiming to complete sampling, data collection complete and scoring within a 12-week period.

Survey Material: The QIO 10^th SOW includes a strong focus on making all processes beneficiary and family focused, in line with the principles outlined by the Picker Institute. To support that focus, the Beneficiary Satisfaction survey will capture beneficiary satisfaction with the appeals review process as well as quality of care complaints.

Below is a summary of the Survey composites and questions. Each of the survey composites represents an important aspect of patient-centeredness. Composites are made up for two or more questions. Composite level scores will be provided to CMS and the QIOs along with item-level data to support the quality improvement effort.

The questionnaire will be available in English and Spanish. Please see Attachment 1 for a copy of the draft questionnaire.

10th SOW and Picker Institute Principles	Beneficiary Survey Composite/Score	Questions
Promoting effective coordination of care including helping communities support better health; transitions and continuity	Coordination: Up to date about interactions you had with other members of the case management team Talked with you about resources that were available to help you	Q11—QIO, had information Q14—QIO gave resources
Information, communication and education	Beneficiary-Centered Communication: Intake Specialist: spent enough time with you listened carefully to you explained things in a way you could understand Case Manager: spent enough time with you listened carefully to you explained things in a way you could understand Communication – written materials (letters) letters explained things in a way you could understand letters contained as much info as needed letters showed respect for your concerns letters reflected information conveyed in telephone conversations	Q7—Intake, enough time Q8--Intake, listened carefully Q6—Intake, understandable Q17—QIO, enough time Q18—QIO, listened carefully Q15—QIO, understandable Q20—letter understandable Q21—letter had needed information Q22—letter showed respect Q23—letter accurate
Emotional support and alleviation of fear and anxiety	Courtesy & Respect Case manager/ team as helpful as they could be Case manager/ team treated you with courtesy and respect Courtesy & Respect – intake process Intake Specialist as helpful as they could be Intake Specialist treated you with courtesy and respect	Q15—QIO, helpful Q19 —QIO, showed respect Q5—Intake, helpful Q9—Intake, showed respect
Access to care	Access & Responsiveness understood situation responsive to your complaint	Q13—QIO understood Q12—QIO responsive

Analysis and Scoring: Standardized scoring for the Beneficiary Satisfaction Survey will permit CMS and the QIOs to assess the process used in resolving beneficiary complaints and appeals without undertaking lengthy analysis on a quarterly basis. Proposed scoring methodology is presented below. All questions proposed for inclusion in the scoring calculation use the satisfaction scale, or the agreement scale. The details of each of these scales are presented below.

Satisfaction scale:

1. Very satisfied

2. Satisfied

3. Neither Satisfied nor Dissatisfied

4. Dissatisfied

5. Very Dissatisfied

Agreement Scale:

1. Strongly agree

2. Agree

3. Neither Agree nor Disagree

4. Disagree

5. Strongly Disagree

Survey responses of 1 and 2 (very satisfied and satisfied; and strongly agree and agree) will be counted as 1 point. Survey responses of 4 and 5 (dissatisfied and strongly dissatisfied; and disagree and strongly disagree) will be counted as 0 points, responses of 3 (neither satisfied nor dissatisfied; and neither agree nor disagree) as well as missing responses will not be included in the denominator for scoring.

Scores will be calculated at the case level for each of the survey composites. The case level scores will then be rolled up to quarterly QIO level scores for each of the survey composites.

In addition to producing quarterly scores at the QIO-level, an annual analytic report will be produced including univariate and multivariate analysis of data at the national and QIO-levels. Analytic reporting will focus on the 4 survey composites: coordination; beneficiary-centered communication; courtesy and respect; and access to care.

Methods to Maximize Response rates.

Efforts to maximize response rates will take many forms, including multiple contacts, survey design principles, and use of pre-paid incentives if needed.

As described in section 2 on data collection procedures, outreach to respondents will occur over three separate mailouts. All mailouts will be sent via first class mail. Timing of the mailouts will ensure that respondents are reminded of the request for their participation in the survey.

The survey design team will use Dillman’s Tailored Design principles in preparing the survey and mailout materials. These design principles have been shown to increase response rates to mailout surveys using formatting and layout principles. Covering materials will stress the importance of the respondent’s input and the use of survey findings to improve processes and make them more patient-centered, leveraging Dillman’s social exchange theory.

The mail methodology proposed here has been used successfully on other CMS surveys (Medicare CAHPS). To reach a response rate of 70% or higher we are proposing the use of apre-paid incentive. Since it is not clear how many percentage points the incentive will add, we propose conducting an experiment in the pilot.

If the incentive does not provide the needed boost in response rate, we will consider adding telephone non-response follow-up.

Testing:

Testing for the Beneficiary Satisfaction Survey will occur over a series of steps including a small number of cognitive interviews and a pilot test. Testing will have the following goals:

1. Determine if the survey wording is clear and unambiguous;

2. Verify respondent’s ability to recall interactions pertaining to their complaints and appeals cases;

3. Ensure appropriate and consistent flow question wording and overall survey administration;

4. Ensure data capture and data output are functioning flawlessly;

5. Provide early scoring and data output to CMS for review and consideration.

An initial round of testing is proposed to help understand the language that beneficiaries use to describe their experience with filing a quality of care complaint or appeal. It is important to understand beneficiary’s frame of reference so that questions can appropriately tap into this framing. As the 9^th SOW sample did not include appeals cases, cognitive testing will be our first opportunity to learn from beneficiaries about how they describe the event. The initial round of cognitive testing will include up to 9 beneficiaries.

A second round of cognitive testing is proposed to take place after the beginning of the 10^th SOW. Some changes in how cases are processed will be implemented with the start of the 10^th SOW. These changes will impact who interacts with the beneficiary during various stages of the complaint and appeals processes. It will be important to conduct the second round of testing to ensure that the survey is capturing all important elements of the process, from the beneficiary’s perspective. For example, the draft questionnaire includes wording to ask about interactions with the “intake specialist”. We anticipate that “intake specialist” is not the best wording to use to have the beneficiary think about the person with whom they initiated their case. Through the cognitive testing, we will learn from the beneficiaries about the best term to use in these questions. The second round of cognitive testing will include 5-9 beneficiaries with finalized appeals cases, and 5-9 beneficiaries with finalized complaint cases. After each round of cognitive testing, the questionnaire will be revised as necessary.

A pilot test of the survey and full survey operations is recommended in order to learn about how the questionnaire and data collection methodologies perform. Through the pilot test, we will obtain a data set large enough to assess the psychometric properties of the survey questions. For example, if we determine that there is little or no variability in how respondents answer a particular question, we can consider revising or dropping that question. Additionally, if many respondents use white space in the margin of the questionnaire to insert comments, we can determine if additional survey content or response categories should be added.

The pilot test will also permit us to assess the data collection methodology. During the pilot, we will conduct an experiment by sending a $5, prepaid incentive in the first mailing to a random sub-sample of the cases drawn for pilot data collection. Based on current research a pre-paid incentive will provide an increased response rate.³ Because the population of Medicare beneficiaries who file appeals or complaints cases varies from the general population, including in age, it is not know the precise degree to which a pre-paid incentive will increase response rates, but it is estimated that an 8-10 percent increase may be seen with use of the incentive. If the experiment is successful, the cost of a $5 incentive is far less than the cost of adding non-response follow-up by telephone.

Through the pilot test, we will determine the actual response rate that should be expected, and we will work with CMS to augment the response rate using other modes of contact as necessary.

Finally, the pilot test will permit for a full-scale test of the data capture and data reporting functionality. CMS may chose to use the pilot test data as the first set of data and scores reported to the QIOs if no substantial change in the survey or data collection methodology is required as a result of what is learned through the testing. The proposed sampling methodology and sample size for the pilot testing would be consistent with what is proposed for the quarterly data collection through the rest of the 10^th SOW.

The following individuals were consulted in the development of the surveys

Organization	Name	Contact Information
CMS	Robert Kambic Coles Mercier	410-786-1515 [email protected] 410-786-2112; [email protected]
Westat	W. Sherman Edwards Vasudha Narayanan Stephanie Fry	301-294-3993; [email protected] 301-251-2257 [email protected] 301-294-2872 [email protected]

Table 1: Estimated Quarterly Numbers of Cases and Estimated Quarterly Completes

	Appeals			Complaints
State	Est'd Quarterly Cases	Proposed Quarterly Sample	Est'd Quarterly Completes	Est'd Quarterly Cases	Proposed Quarterly Sample	Est'd Quarterly Completes
AK	4	4	2	1	1	0
AL	308	50	30	8	8	5
AR	31	31	19	9	9	5
AZ	228	50	30	7	7	4
CA	2,637	50	30	36	36	22
CO	82	50	30	14	14	8
CT	236	50	30	8	8	5
DC	11	11	7	1	1	0
DE	23	23	14	4	4	2
FL	1,473	50	30	47	47	28
GA	108	50	30	7	7	4
HI	14	14	8	3	3	2
IA	60	50	30	3	3	2
ID	41	41	25	3	3	2
IL	266	50	30	15	15	9
IN	131	50	30	7	7	4
KS	44	44	26	2	2	1
KY	81	50	30	4	4	2
LA	28	28	17	5	5	3
MA	428	50	30	6	6	4
MD	217	50	30	9	9	5
ME	28	28	17	2	2	1
MI	376	50	30	17	17	10
MN	216	50	30	9	9	5
MO	187	50	30	9	9	5
MS	61	50	30	5	5	3
MT	10	10	6	2	2	1
NC	147	50	30	2	2	1
ND	27	27	16	0	0	0
NE	95	50	30	1	1	0
NH	28	28	17	1	1	0
NJ	472	50	30	17	17	10
NM	30	30	18	5	5	3
NV	78	50	30	23	23	14
NY	1,458	50	30	45	45	27
OH	345	50	30	23	23	14
OK	36	36	22	14	14	9
OR	161	50	30	5	5	3
PA	1,146	50	30	14	14	9
PR	49	49	29	7	7	4
RI	51	50	30	3	3	2
SC	51	50	30	2	2	1
SD	18	18	11	0	0	0
TN	187	50	30	18	18	11
TX	277	50	30	27	27	16
UT	52	50	30	2	2	1
VA	251	50	30	5	5	3
VI	0	0	0	0	0	0
VT	16	16	10	0	0	0
WA	331	50	30	14	14	8
WI	176	50	30	9	9	5
WV	61	50	30	2	2	1
WY	2	2	1	0	0	0
National	12,737	50	30			0
Total	25,603	2,190	1,314	478	478	287

Attachment 1: Draft Questionnaire

Your Medicare [Quality of Care Complaint / Benefits Appeal]

1. Our records show that on [DATE] you filed [a complaint about the quality of care you or another person received under Medicare / an appeal about your or another person’s Medicare benefits]. Is that right?

• Yes

• N o If No, please return the survey in the postage-paid envelope.

The questions in this survey will refer to the [Medicare quality of care complaint that you filed on the date shown in Question 1 as “your quality of care complaint” / Medicare benefits appeal that you filed on the date shown in Question 1 as “your appeal”.

2. Have you gotten a resolution on your [quality of care complaint / appeal]?

• Yes

• N o If No, please return the survey in the postage-paid envelope.

3. How satisfied are you with the resolution of your [quality of care complaint / appeal]?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

4. Please give us your comments on the resolution of your [quality of care complaint / appeal].

_____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Interactions with the Intake Specialist

The next questions are about the way your [quality of care complaint / appeal] was handled from the start. The questions will refer to the person you first spoke with when you called to file your [quality of care complaint / appeal] as the “Intake Specialist”. The Intake Specialist would have collected the details about your [quality of care complaint / appeal].

5. When you spoke with the Intake Specialist, how satisfied were you that he or she was as helpful as you thought they should be?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

6. When you spoke with the Intake Specialist, how satisfied were you that he or she explained things in a way you could understand?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

7. When you spoke with the Intake Specialist, how satisfied were you that he or she spent enough time with you?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

8. When you spoke with the Intake Specialist, how satisfied were you that he or she listened carefully to you?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

9. When you spoke with the Intake Specialist, how satisfied were you that he or she showed respect for what you had to say?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

Interactions with the Case Manager

The next questions will refer to the person who called you back after your [quality of care complaint / appeal] was filed as the “Case Manager”. The Case Manager would have contacted you about the resolution of your [quality of care complaint / appeal].

10. Did you speak to a Case Manager about your [quality of care complaint / appeal] on the phone?

• Yes

• No If No, go to Q20.

How much do you agree or disagree with the following statements:

11. The Case Manager had all the information that you gave to the Intake Specialist about your [quality of care complaint / appeal].

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

12. The Case Manager was as responsive to your [quality of care complaint / appeal] as you thought they should be.

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

13. The Case Manager understood your situation.

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

14. The Case Manager talked with you about resources that were available to help you.

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

15. When you spoke with the Case Manager, how satisfied were you that he or she was as helpful as you thought they should be?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

16. When you spoke with the Case Manager, how satisfied were you that he or she explained things in a way you could understand?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

17. When you spoke with the Case Manager, how satisfied were you that he or she spent enough time with you?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

18. When you spoke with the Case Manager, how satisfied were you that he or she listened carefully to you?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

19. When you spoke with the Case Manager, how satisfied were you that he or she showed respect for what you had to say?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

Letter(s) about your [Quality of Care Complaint / Appeal]

20. Did you receive any letters about your [quality of care complaint / appeal]?

• Yes

• No If No, go to Q25.

21. How satisfied were you that the letter(s) you got about your [quality of care complaint / appeal] explained things in a way you could understand?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

22. How satisfied were you that the letter(s) you got about your [quality of care complaint / appeal] contained all the information you needed?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

23. How satisfied were you that the letter(s) you got about your [quality of care complaint / appeal] showed respect for your concerns?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

24. How satisfied were you that the letter(s) you got about your [quality of care complaint / appeal] had the same information that you were told in telephone conversations?

• Very satisfied

• Satisfied

• Neither satisfied nor dissatisfied

• Dissatisfied

• Very dissatisfied

Overall [Quality of Care Complaint / Appeal] Process

25. Using any number from 0 to 10 where 0 is the worst [quality of care complaint / appeal] process possible and 10 is the best [quality of care complaint / appeal] process possible, what number would you use to rate the overall [quality of care complaint / appeal] process?

• 0 – Worst process possible

• 1

• 2

• 3

…

• 8

• 9

• 10 – Best process possible

26. Please give us your comments on the process that was used to resolve your [quality of care complaint / appeal]. Include any comments you have on what worked well, and suggestions you have on ways to improve how the process.

_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Thank you: Those are all the questions we have for you now

1 Dillman, D. A. (2007). Mail and Internet Surveys; the tailored design method. New Jersey, United States: John Wiley & Sons Inc.

2 Dillman, D. A., Sangster, R. L., Tarnai, J. and Rockwood, T. H. (1996), Understanding differences in people's answers to telephone and mail surveys. New Directions for Evaluation, 1996: 45–61. doi: 10.1002/ev.1034

3 Petrolia, D. R., Bhattacharjee, S. “Revisiting incentive effects; evidence from a random-sample mail survey on consumer preferences for fuel ethanol.” Public Opinion Quarterly Volume 73, issue 3, (2011): 537-550.