Beneficiary Cognitive testing Report.
Beneficiary Cognitive testing Report.docx
Medicare Beneficiary and Family-Centered Satisfaction Survey
Beneficiary Cognitive testing Report.
OMB: 0938-1177
CENTERS FOR MEDICARE & MEDICAID SERVICES
Survey Management, Analysis, Reporting, and Technical Support
Cognitive Testing Report
For Beneficiary 10th SOW Survey
Contract #: HHSM-500-2009-00040C
January 12, 2012
Submitted to:
Centers for Medicare & Medicaid Services
Baltimore, MD
Prepared by:
QualMed Surveys
1700 Research Blvd., Suite 101
Rockville, MD 20850
TABLE OF CONTENTS
Page
INTRODUCTION 1
METHODS 1
FINDINGS 1
Outcome vs. Process 1
Intake Specialist and Case Manager Processes 2
Item-Specific Findings and Recommended Changes 2
Recommended Changes for Quality of Care Complaint Survey 3
Recommended Changes for Benefits Appeal Survey 9
LIST OF APPENDIXES
A CMS Medicare Beneficiary and Family Centered Satisfaction Survey Quality of Care Complaint Cognitive Interviewing Protocol A-1
B CMS Medicare Beneficiary and Family Centered Satisfaction Survey Quality of Care Complaint Cognitive Interviewing Protocol B-1
INTRODUCTION
Under the Quality Improvement Organization (QIO) 10th Scope of Work (SOW), new systems and processes were put in place to handle complaints from Medicare beneficiaries about quality of care, as well as appeals about discharge. In order to stay current with how the complaints and appeals processes are taking place, the survey which measures beneficiary satisfaction with those processes must be updated. The revisions to the beneficiary satisfaction survey are designed to address the following:
The 10th SOW includes a strong focus on making all processes beneficiary and family focused, in line with the principles outlined by the Picker Institute.
The survey which was used to measure beneficiary satisfaction in the 8th and 9th SOW measured beneficiary satisfaction with the quality of care complaint review process only. In the 10th SOW, the survey will also capture beneficiary satisfaction with the appeals review process.
The intake functions are being completed by the National Coordinating Center (NCC) and the follow-up case management is being completed by the QIOs.
To address the changes in systems and process describe above, QualMed developed draft surveys to measure beneficiary satisfaction with complaints and appeals processes. The content, structure, and wording of those surveys were tested with Beneficiaries using cognitive testing methods. A report of the methods and findings from cognitive testing follows.
METHODS
To be eligible, a participant in the interviews, subjects had to be at least 18 years old. Adult subjects had to have filed a quality of care complaint or a discharge appeal after August 1 2011. That complaint or appeal case had to have been closed, indicating the full set of interactions regarding that complaint or appeal was completed. Subjects were recruited from a sample list provided by CMS.
Cognitive interviews were conducted one-on-one via the telephone in English. Interviews lasted approximately 30 minutes. A supplement of mail surveys were sent to identify potential problems that may arise with self-administration of the questionnaire. A total of 7 survey interviews were completed between November and December 2011.
The cognitive interviewing protocols for both complaint and appeals instruments are included as appendices.
FINDINGS
This section of the report describes the findings and recommendations for the Beneficiary Satisfaction Survey based on the cognitive testing completed in December 2011. Some general findings are described followed by specific findings and recommendations at the questions item level.
Outcome vs. Process
The Beneficiary Satisfaction Survey was designed to measure Beneficiary satisfaction with the process of filing an appeal or a quality of care complaint as well as satisfaction with the follow-up steps to complete and close that case. The Survey was not designed to measure Beneficiary satisfaction with the outcome of the complaint or appeal. Since the outcome of the case is highly salient to the Beneficiaries, it is challenging to get Beneficiaries to focus on reporting their experiences related only to the elements of process. To help achieve the CMS goal of obtaining valid measures of Beneficiary satisfaction with the process, QualMed relied on previous testing completed with the population under the 8th and 9th SOWs and developed a survey which included a small number of questions about the outcome of the complaint or appeal. The goal was to get the Beneficiary to tell his or her story about the case outcome, and thereafter ask the Beneficiary to turn his or her attention to the elements of process. As with previous testing findings, the 10th SOW testing found that this strategy was successful at getting Beneficiaries to tease apart elements of process without undue reference to the outcome.
Intake Specialist and Case Manager Processes
Historically, the QIOs were responsible for all interactions with Beneficiaries in their state who filed complaints or appeals. Under the 10th SOW, responsibility for the complaint and appeal intake functions and initial interactions with Beneficiaries was shifted to the newly created NCC. Under the new 10th SOW processes, after all case data are collected and the case is triaged by the NCC, follow-up functions are typically handled by the QIO. With this multi-step process in place, it is important to ask Beneficiaries to report separately about their interactions during the intake process and their interactions with case managers who may have followed-up after intake. Beneficiary experience can then be appropriately attributed to the organization with whom the Beneficiary was interacting. Additionally, the NCC and the QIOs can receive feedback about Beneficiary satisfaction related to their interactions, and quality improvement strategies can be implemented as appropriate.
In order to capture the needed Beneficiary experience data, two terms of art were introduced into the Beneficiary Satisfaction Survey: Intake Specialist and Case Manager. Each of the two terms of art was introduced with a definition to describe the function performed and how the beneficiary would have interacted with the individual. Through testing we determined that the combination of term of art and definition functioned as expected to direct the Beneficiary to think about the appropriate set of interactions either intake or case manager follow up. In both telephonic and mail surveys, Beneficiaries were clearly able to differentiate intact interactions from case manager follow-up. This was demonstrated by Beneficiary ratings which varied across similar measures for the two entities – finding true in telephonic and mail surveys. It was further demonstrated by Beneficiary feedback in the cognitive testing when they would reference individuals by name and describe the different functions that each served in the process. Further, Beneficiary representatives and those who did not place initial calls themselves were also found to be able to differentiate and attribute experiences to the appropriate individuals either at intake or at case management follow-up.
It should be noted, that in the case of some appeals cases, Beneficiaries reported having no follow-up interactions with a case manager. The Beneficiary received a form/letter to indicate the outcome of the appeal, but no telephone interaction was noted after intake. In such cases, survey skip patterns will direct Beneficiaries to respond only to questions that apply to their circumstance and no QIO-level data or scores will be generated.
Item-Specific Findings and Recommended Changes
The following section shows the draft complaint and appeal surveys that were used in the cognitive testing. Where testing findings suggested the need for a change to item wording, these changes are noted. Where testing indicated that items were performing as expected, no notation is made. The full survey instruments are provided to assist readers in following the flow of the instruments.
CMS Medicare Beneficiary and Family Centered Satisfaction Survey
Quality of Care Complaint
Recommended Edits based on Cognitive Testing
Your Medicare Quality of Care Complaint
1. Our records show that on [DATE] you filed a complaint about the quality of care you or another person received under Medicare about your or another person’s Medicare benefits. Is that right?
Yes
N
o
If No, please return
the survey in the postage-paid envelope.
The questions in this survey will refer to the Medicare quality of care complaint that you filed on the date shown in Question 1 as “your quality of care complaint”.
2. Have you received the results or findings in response to your quality of care complaint?
Yes
N
o
If No, please return the survey in the postage-paid envelope.
Bene who was initiating another form of follow-up reported that he had not received findings that were satisfying to him. Recommend dropping this skip instruction and having Beneficiaries continue to respond to the survey regardless of their response.
3. How satisfied are you with the results or findings in response to your quality of care complaint?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
4. Please give us your comments on the results or findings in response to your quality of care complaint.
________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Interactions with the Intake Specialist
The next
questions are about the way your quality of care complaint was
handled from
the start. The
questions will refer to the person you first
spoke with when you called to file your complaint as the “Intake
Specialist”. The Intake Specialist would have collected the
details about your complaint. Bene
believed that the “start” of his complaint was
significantly before he placed his call to the NCC. The second and
third sentences functioned as intended to introduce the term of art
“Intake Specialist”. Recommend removing “from the
start” to improve overall clarity.
5. When you spoke with the Intake Specialist, how satisfied were you that he or she was as helpful as you thought they should be?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
6. When you spoke with the Intake Specialist, how satisfied were you that he or she explained things in a way you could understand?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
7. When you spoke with the Intake Specialist, how satisfied were you that he or she spent enough time with you?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
8. When you spoke with the Intake Specialist, how satisfied were you that he or she listened carefully to you?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
9. When you spoke with the Intake Specialist, how satisfied were you that he or she showed respect for what you had to say?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
Interactions with the Case Manager
The next questions will refer to the person who called you back after your quality of care complaint was filed as the “Case Manager”. The Case Manager would have contacted you about the results or findings in response to your quality of care complaint.
10. Did you speak to a Case Manager about your quality of care complaint on the phone?
Yes
No
If No, go to Q20.
How much do you agree or disagree with the following statements:
11. The Case Manager had all the information that you gave to the Intake Specialist about your quality of care complaint.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
12. The Case Manager was as responsive to your quality of care complaint as you thought they should be.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
13. The Case Manager understood your situation.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
14. The
Case Manager talked with you about resources
programs
and services in your community that
were available to help with
your
health and wellbeing.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
Complaint and appeals Beneficiaries did not have a clear and consistent understanding of the term “resources”. Alternate item wording tested with Beneficiaries and found to be consistently understood by Beneficiaries: The Case Manager talked with you about programs and services in your community that were available to help with your health and wellbeing.
15. When you spoke with the Case Manager, how satisfied were you that he or she was as helpful as you thought they should be?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
16. When you spoke with the Case Manager, how satisfied were you that he or she explained things in a way you could understand?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
17. When you spoke with the Case Manager, how satisfied were you that he or she spent enough time with you?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
18. When you spoke with the Case Manager, how satisfied were you that he or she listened carefully to you?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
19. When you spoke with the Case Manager, how satisfied were you that he or she showed respect for what you had to say?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
Letter(s) about your Quality of Care Complaint
20. Have you received any forms or letters from the Centers for Medicare & Medicaid Services about your quality of care complaint?
Yes
No
If No, go to Q25.
Beneficiaries referred to written materials they received in a variety of ways: letters, form letters, and forms. In order to capture all such written materials, recommend adding reference to forms in addition to the reference to letters.
21. How
satisfied were you that the forms
or letter(s)
you got about
your quality of care complaint explained things in a way you could
understand?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
22. How
satisfied were you that the forms
or letter(s)
you got about your quality of care complaint contained all the
information you needed?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
23. How
satisfied were you that the forms
or letter(s)
you got about your quality of care complaint showed respect for your
concerns?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
24. How
satisfied were you that the forms
or letter(s)
you got about your quality of care complaint had the same information
that you were told in telephone conversations?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
Was not given any information in telephone conversations
Based on test findings for appeals case, recommend adding a tailored inapplicable response: Was not given any information in telephone conversations.
Overall Quality of Care Complaint Process
In responding to your quality of care complaint a process was used to gather information about your quality of care complaint, explain the complaint steps, and give you the results or findings of your case.
25. Using any number from 0 to 10 where 0 is the worst quality of care complaint process possible and 10 is the best quality of care complaint process possible, what number would you use to rate the overall quality of care complaint process?
0 – Worst process possible
1
2
3
4
5
6
7
8
9
10 – Best process possible
26. Please give us your comments on the process that was used in responding to your quality of care complaint. Include any comments you have on what worked well, and suggestions you have on ways to improve the process.
________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Thank you for your participation.
CMS Medicare Beneficiary and Family Centered Satisfaction Survey
Benefits Appeal
Recommended Edits based on Cognitive Testing
Your Medicare Benefits Appeal
1. Our records show that on [DATE] you filed an appeal about your or another person’s Medicare benefits. Is that right?
Yes
N
o
If No, please return
the survey in the postage-paid envelope.
The questions in this survey will refer to the Medicare benefits appeal that you filed on the date shown in Question 1 as “your appeal”.
2. Have you received the results or findings in response to your appeal?
Yes
N
o
If No, please return the survey in the postage-paid envelope.
Recommend dropping this skip instruction and having Beneficiaries continue to respond to the survey regardless of their response. Based on testing findings from complaint case.
3. How satisfied are you with the results or findings in response to your appeal?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
3a. Thinking about the period of time from when you first called to file your appeal to the time when you received the results or findings, how long did the appeal process take?
Less than 12 hours
12-23 hours
24-47 hours
48-71 hours
72 hours or more
4. Please give us your comments on the results or findings in response to your appeal.
________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Interactions with the Intake Specialist
The next
questions are about the way your appeal was handled from
the start. The
questions will refer to the person you first
spoke with when you called to file your appeal as the “Intake
Specialist”. The Intake Specialist would have collected the
details about your appeal.
Recommend removing “from the start” to improve overall clarity based on testing finding from complaint case.
5. When you spoke with the Intake Specialist, how satisfied were you that he or she was as helpful as you thought they should be?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
6. When you spoke with the Intake Specialist, how satisfied were you that he or she explained things in a way you could understand?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
7. When you spoke with the Intake Specialist, how satisfied were you that he or she spent enough time with you?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
8. When you spoke with the Intake Specialist, how satisfied were you that he or she listened carefully to you?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
9. When you spoke with the Intake Specialist, how satisfied were you that he or she showed respect for what you had to say?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
Interactions with the Case Manager
The next questions will refer to the person who called you back after your appeal was filed as the “Case Manager”. The Case Manager would have contacted you about the results or findings in response to your appeal.
10. Did you speak to a Case Manager about your appeal on the phone?
Yes
No
If No, go to Q20.
How much do you agree or disagree with the following statements:
11. The Case Manager had all the information that you gave to the Intake Specialist about your appeal.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
12. The Case Manager was as responsive to your appeal as you thought they should be.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
13. The Case Manager understood your situation.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
14. The
Case Manager talked with you about resources
programs
and services in your community that
were available to help with
your
health and wellbeing.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
Complaint and appeals Beneficiaries did not have a clear and consistent understanding of the term “resources”. Alternate item wording tested with Beneficiaries and found to be consistently understood by Beneficiaries: The Case Manager talked with you about programs and services in your community that were available to help with your health and wellbeing.
15. When you spoke with the Case Manager, how satisfied were you that he or she was as helpful as you thought they should be?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
16. When you spoke with the Case Manager, how satisfied were you that he or she explained things in a way you could understand?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
17. When you spoke with the Case Manager, how satisfied were you that he or she spent enough time with you?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
18. When you spoke with the Case Manager, how satisfied were you that he or she listened carefully to you?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
19. When you spoke with the Case Manager, how satisfied were you that he or she showed respect for what you had to say?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
Letter(s) about your Appeal
20. Have you received any forms or letters from the Centers for Medicare & Medicaid Services about your appeal?
Yes
No
If No, go to Q25.
Beneficiaries referred to written materials they received in a variety of ways: letters, form letters, and forms. In order to capture all such written materials, recommend adding reference to forms in addition to the reference to letters.
21. How
satisfied were you that the forms
or letter(s)
you got about your appeal explained things in a way you could
understand?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
22. How
satisfied were you that the forms
or letter(s)
you got about your appeal contained all the information you needed?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
23. How
satisfied were you that the forms
or letter(s)
you got about your appeal showed respect for your concerns?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
24. How
satisfied were you that the forms
or letter(s)
you got about your appeal had the same information that you were told
in telephone conversations?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
Was not given any information in telephone conversations
Beneficiary filed appeal with Intake Specialist and had no further telephone conversations with QIO or Case Managers. As such, she could not respond to this question. Recommend adding a tailored inapplicable response: Was not given any information in telephone conversations.
Overall Appeal Process
In responding to your appeal a process was used to gather information about your appeal, explain the appeal steps, and give you the results or findings of your case.
25. Using any number from 0 to 10 where 0 is the worst appeal process possible and 10 is the best appeal process possible, what number would you use to rate the overall appeal process?
0 – Worst process possible
1
2
3
4
5
6
7
8
9
10 – Best process possible
26. Please give us your comments on the process that was used in responding to your appeal. Include any comments you have on what worked well, and suggestions you have on ways to improve the process.
________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Thank you for your participation.
Appendix A
CMS Medicare Beneficiary and Family Centered Satisfaction Survey
Quality of Care Complaint
Cognitive Interviewing Protocol
CMS Medicare Beneficiary and Family Centered Satisfaction Survey
Quality of Care Complaint
Draft Instrument for Cognitive Testing
Introduction
Hello May I speak with {Name}.
My
name is {name}. I am calling for the Centers for Medicare and
Medicaid Services (CMS). We are conducting a brief survey with
beneficiaries who have filed a complaint about the quality of care
they
or another person received under Medicare. Thank you for
agreeing to speak with us today.
I am joined today by my colleague {NAME} who is going to be assisting me by making some notes to ensure we capture what you have to say. Ask if this is OK with the interviewee.
We have been contracted by CMS to develop a survey for beneficiaries who have filed a complaint about the quality of care they or another person received under Medicare. The survey will ultimately be mailed to beneficiaries such as yourself. We mailed you a draft of the survey and we are looking to fine-tune the draft and make sure that it is clear and thorough, and that it accurately captures your experience.
Your participation in this interview is voluntary. You may choose to skip any questions you do not wish to answer or decline to participate at any time. No individual data collected will ever be reported. The interview should only take about 30 minutes.
Because the information you provide is so important, we would like to audiotape our discussion. We would like to do this so we can review the conversation to better capture your responses. Again, no one other than project staff will have access to these tapes and the information you provide will be kept strictly confidential.
Do I have your permission to tape our conversation? (Yes or No)
IF YES, turn on recorder
We are required to record on tape your consent to participate in this interview. I would like to confirm that you have given your permission both for this interview and for the interview to be recorded.
Are you willing to participate in this interview? (Yes / No)
Do I have your permission to record the interview? (Yes / No)
Please remember that you may ask me to turn off the recorder at any time.
I would like to begin by asking how clear, or unclear the survey questions were to you. As we go along, please let me know if you have questions or anything is difficult to answer.
Do you have the survey in front of you? Doesn’t really matter if they do or don’t, just useful to know.
Do you have any questions before we get started?
What are your reactions to the survey we sent you?
Was anything difficult to answer?
Now I am going to go through and ask you some of the questions from the survey. I will follow-up by asking about what you though of to answer the question, what a particular team may mean to you, or why you answered the way that you did. There are no right or wrong answers, we are simply looking to get an understanding of how you are understanding the questions.
Your Medicare Quality of Care Complaint
1. Our records show that you filed a complaint about the quality of care you or another person received under Medicare about you or another person’s Medicare benefits. Is that right?
Yes
No
[Probe: Can you tell me the approximate date you filed the complaint.] ____________________.
The questions in this survey will refer to the Medicare quality of care complaint that you filed on the date shown in Question 1 as “your quality of care complaint”.
This survey refers to your “quality of care complaint”. When you see or hear that term, what do you think of? How would you refer to that process?
2. Have you received the results or findings in response to your quality of care complaint?
Yes
No
In this question, what does “results or findings” mean to you?
3. How satisfied are you with the results or findings in response to your quality of care complaint?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
4. Please give us your comments on the results or findings in response to your quality of care complaint.
________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Interactions with the Intake Specialist
The next questions are about the way your quality of care complaint was handled from the start. The questions will refer to the person you first spoke with when you called to file your complaint as the “Intake Specialist”. The Intake Specialist would have collected the details about your complaint.
5. When you spoke with the Intake Specialist, how satisfied were you that he or she was as helpful as you thought they should be?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
Who did you think about when answering this question? How many peopld did you speak with as you were filing your complaint? How would you refer to that person/those people? What does “Intake Specialist” mean to you -- which person comes to mind?
What does being “helpful” mean here?
6. When you spoke with the Intake Specialist, how satisfied were you that he or she explained things in a way you could understand?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
What did you think about to answer this question? What “things” did he/she explain?
7. When you spoke with the Intake Specialist, how satisfied were you that he or she spent enough time with you?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
What does it mean to spend enough time with you?
8. When you spoke with the Intake Specialist, how satisfied were you that he or she listened carefully to you?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
9. When you spoke with the Intake Specialist, how satisfied were you that he or she showed respect for what you had to say?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
Using your own words, what is this question asking?
Interactions with the Case Manager
The next questions will refer to the person who called you back after your quality of care complaint was filed as the “Case Manager”. The Case Manager would have contacted you about the results or findings in response to your quality of care complaint.
10. Did you speak to a Case Manager about your quality of care complaint on the phone?
Yes
No
If No, go to Q20.
What did you think about to answer this question? Tell me about the person who called you back to follow-up. How would you refer to that person? Did more than one person follow up with you? What does the term “Case Manager” mean to you?
How much do you agree or disagree with the following statements:
11. The Case Manager had all the information that you gave to the Intake Specialist about your quality of care complaint.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
What did you think about to answer this question? How did you know what information the Case Manager had?
12. The Case Manager was as responsive to your quality of care complaint as you thought they should be.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
What does this question mean to you? What does it mean to be “responsive”?
13. The Case Manager understood your situation.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
What do you think this question is asking?
14. The Case Manager talked with you about resources that were available to help you.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
What did you think about to answer this question? What “resources” came to mind? What would “resources” mean in this context?
15. When you spoke with the Case Manager, how satisfied were you that he or she was as helpful as you thought they should be?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
What did you think about to answer this question?
16. When you spoke with the Case Manager, how satisfied were you that he or she explained things in a way you could understand?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
17. When you spoke with the Case Manager, how satisfied were you that he or she spent enough time with you?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
18. When you spoke with the Case Manager, how satisfied were you that he or she listened carefully to you?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
19. When you spoke with the Case Manager, how satisfied were you that he or she showed respect for what you had to say?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
Letter(s) about your Quality of Care Complaint
20. Have you received any letters from the Centers for Medicare & Medicaid Services about your quality of care complaint?
Yes
No
If No, go to Q25.
How many letters did you receive? Do you remember who the letters came from? What information did the various letters provide?
21. How satisfied were you that the letter(s) you got about your quality of care complaint explained things in a way you could understand?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
22. How satisfied were you that the letter(s) you got about your quality of care complaint contained all the information you needed?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
[IF MORE THAN 1 LETTER] Were you equally satisfied with all the letters in terms of the degree to which they contained the information you needed? How did you come up with your response to this question?
23. How satisfied were you that the letter(s) you got about your quality of care complaint showed respect for your concerns?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
What does it mean for a letter to “show respect for your concerns”?
24. How satisfied were you that the letter(s) you got about your quality of care complaint had the same information that you were told in telephone conversations?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
What did you think about to answer this question? What if I asked a slightly different question: How satisfied were you that the letter you got about your quality of care complaint matched what you were told in telephone conversations? Is that question asking the same thing or something different? Which of those questions do you prefer?
Overall Quality of Care Complaint Process
In responding to your quality of care complaint a process was used to gather information about your quality of care complaint, explain the complaint steps, and give you the results or findings of your case.
25. Using any number from 0 to 10 where 0 is the worst quality of care complaint process possible and 10 is the best quality of care complaint process possible, what number would you use to rate the overall quality of care complaint process?
0 – Worst process possible
1
2
3
4
5
6
7
8
9
10 – Best process possible
How did you come up with your rating? What did you think about to answer?
26. Please give us your comments on the process that was used in responding to your quality of care complaint. Include any comments you have on what worked well, and suggestions you have on ways to improve the process.
________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Do you have any other comments about the survey?
Thank you very much for your time. You have done a lot to help improve CMS’ ability to understand the experiences of beneficiaries like yourself.
Appendix B
CMS Medicare Beneficiary and Family Centered Satisfaction Survey
Appeal
Cognitive Interviewing Protocol
CMS Medicare Beneficiary and Family Centered Satisfaction Survey
Appeal
Draft Instrument for Cognitive Testing
Introduction
Hello May I speak with {Name}.
My name is {name}. I am calling for the Centers for Medicare and Medicaid Services (CMS). We are conducting a brief survey with beneficiaries who have filed an appeal about their or another person’s Medicare benefits. Thank you for agreeing to speak with us today.
I am joined today by my colleague {NAME} who is going to be assisting me by making some notes to ensure we capture what you have to say. Ask if this is OK with the interviewee.
We have been contracted by CMS to develop a survey for beneficiaries who have filed an appeal about their or another person’s Medicare benefits. The survey will ultimately be mailed to beneficiaries such as yourself. We mailed you a draft of the survey and we are looking to fine-tune the draft and make sure that it is clear and thorough, and that it accurately captures your experience.
Your participation in this interview is voluntary. You may choose to skip any questions you do not wish to answer or decline to participate at any time. No individual data collected will ever be reported. The interview should only take about 30 minutes.
Because the information you provide is so important, we would like to audiotape our discussion. We would like to do this so we can review the conversation to better capture your responses. Again, no one other than project staff will have access to these tapes and the information you provide will be kept strictly confidential.
Do I have your permission to tape our conversation? (Yes or No)
IF YES, turn on recorder
We are required to record on tape your consent to participate in this interview. I would like to confirm that you have given your permission both for this interview and for the interview to be recorded.
Are you willing to participate in this interview? (Yes / No)
Do I have your permission to record the interview? (Yes / No)
Please remember that you may ask me to turn off the recorder at any time.
I would like to begin by asking how clear, or unclear the survey questions were to you. As we go along, please let me know if you have questions or anything is difficult to answer.
Do you have the survey in front of you? Doesn’t really matter if they do or don’t, just useful to know.
Do you have any questions before we get started?
What are your reactions to the survey we sent you?
Was anything difficult to answer?
Now I am going to go through and ask you some of the questions from the survey. I will follow-up by asking about what you thought of to answer the question, what a particular team may mean to you, or why you answered the way that you did. There are no right or wrong answers, we are simply looking to get an understanding of how you are understanding the questions.
Your Medicare Appeal
1. Our records show that you filed an appeal about your or another person’s Medicare benefits. Is that right?
Yes
No
[Probe: Can you tell me the approximate date you filed the appeal.]
The questions in this survey will refer to the Medicare benefits appeal that you filed as “your appeal”.
This survey refers to your “your appeal”. When you see or hear that term, what do you think of? How would you refer to that process?
2. Have you received the results or findings in response to your appeal?
Yes
No
In this question, what does “results or findings” mean to you?
3. How satisfied are you with the results or findings in response to your appeal?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
3a. Thinking about the period of time from when you first called to file your appeal to the time when you received the results or findings, how long did the appeal process take?
Less than 12 hours
12-23 hours
24-47 hours
48-71 hours
72 hours or more
4. Please give us your comments on the results or findings in response to your appeal.
________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Interactions with the Intake Specialist
The next questions are about the way your appeal was handled from the start. The questions will refer to the person you first spoke with when you called to file your appeal as the “Intake Specialist”. The Intake Specialist would have collected the details about your appeal.
5. When you spoke with the Intake Specialist, how satisfied were you that he or she was as helpful as you thought they should be?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
Who did you think about when answering this question? How many people did you speak with as you were filing your appeal? How would you refer to that person/those people? What does “Intake Specialist” mean to you -- which person comes to mind?
What does being “helpful” mean here?
6. When you spoke with the Intake Specialist, how satisfied were you that he or she explained things in a way you could understand?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
What did you think about to answer this question? What “things” did he/she explain?
7. When you spoke with the Intake Specialist, how satisfied were you that he or she spent enough time with you?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
What does it mean to spend enough time with you?
8. When you spoke with the Intake Specialist, how satisfied were you that he or she listened carefully to you?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
9. When you spoke with the Intake Specialist, how satisfied were you that he or she showed respect for what you had to say?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
Using your own words, what is this question asking?
Interactions with the Case Manager
The next questions will refer to the person who called you back after your appeal was filed as the “Case Manager”. The Case Manager would have contacted you about the results or findings in response to your appeal.
10. Did you speak to a Case Manager about your appeal on the phone?
Yes
No
If No, go to Q20.
What did you think about to answer this question? Tell me about the person who called you back to follow-up. How would you refer to that person? Did more than one person follow up with you? What does the term “Case Manager” mean to you?
How much do you agree or disagree with the following statements:
11. The Case Manager had all the information that you gave to the Intake Specialist about your appeal.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
What did you think about to answer this question? How did you know what information the Case Manager had?
12. The Case Manager was as responsive to your appeal as you thought they should be.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
What does this question mean to you? What does it mean to be “responsive”?
13. The Case Manager understood your situation.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
What do you think this question is asking?
14. The Case Manager talked with you about resources that were available to help you.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
What did you think about to answer this question? What “resources” came to mind? What would “resources” mean in this context?
How would you answer the following question: The Case Manager talked with you about programs and services in your community that were available to help with your health and wellbeing? [Strongly agree, Agree, Neither agree nor disagree, Disagree, Stongly disagree]
In your own words, what is that question asking?
15. When you spoke with the Case Manager, how satisfied were you that he or she was as helpful as you thought they should be?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
What did you think about to answer this question?
16. When you spoke with the Case Manager, how satisfied were you that he or she explained things in a way you could understand?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
17. When you spoke with the Case Manager, how satisfied were you that he or she spent enough time with you?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
18. When you spoke with the Case Manager, how satisfied were you that he or she listened carefully to you?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
19. When you spoke with the Case Manager, how satisfied were you that he or she showed respect for what you had to say?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
Letter(s) about your Appeal
20. Have you received any letters from the Centers for Medicare & Medicaid Services about your appeal?
Yes
No
If No, go to Q25.
How many letters did you receive? Do you remember who the letters came from? What information did the various letters provide?
Have you received any forms from the Centers for Medicare & Medicaid Services about your appeal?
Did you get anything in writing that told you about the decision on your appeal? What was that? What information did it contain? What would you call that, e.g. a letter, a form?
INTERVIEWER – SUB IN CORRECT WORD FOR THIS SERIES BASED ON RESPONSE ABOVE.
21. How satisfied were you that the letter(s) you got about your appeal explained things in a way you could understand?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
22. How satisfied were you that the letter(s) you got about your appeal contained all the information you needed?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
[IF MORE THAN 1 LETTER] Were you equally satisfied with all the letters in terms of the degree to which they contained the information you needed? How did you come up with your response to this question?
23. How satisfied were you that the letter(s) you got about your appeal showed respect for your concerns?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
What does it mean for a letter to “show respect for your concerns”?
24. How satisfied were you that the letter(s) you got about your appeal had the same information that you were told in telephone conversations?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
What did you think about to answer this question? What if I asked a slightly different question: How satisfied were you that the letter you got about your appeal matched what you were told in telephone conversations? Is that question asking the same thing or something different? Which of those questions do you prefer?
Overall Appeal Process
In responding to your appeal a process was used to gather information about your appeal, explain the steps, and give you the results or findings of your case.
25. Using any number from 0 to 10 where 0 is the worst appeal process possible and 10 is the best appeal process possible, what number would you use to rate the overall appeal process?
0 – Worst process possible
1
2
3
4
5
6
7
8
9
10 – Best process possible
How did you come up with your rating? What did you think about to answer?
26. Please give us your comments on the process that was used in responding to your appeal. Include any comments you have on what worked well, and suggestions you have on ways to improve the process.
__________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Do you have any other comments about the survey?
Thank you very much for your time. You have done a lot to help improve CMS’ ability to understand the experiences of beneficiaries like yourself.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Trish |
| File Modified | 0000-00-00 |
| File Created | 2021-01-30 |
© 2024 OMB.report | Privacy Policy