Pilot Survey Findings Report

1979 Marcus Avenue, Suite 105  Lake Success, NY 11042-1072
Phone: 516-209-5253  Fax: 516-326-7805  [email protected]
www.esrdncc.org  www.kcercoalition.com

ESRD Grievant Satisfaction Survey
Pilot Survey Findings Report

Table of Contents
Executive Summary ........................................................................................................ 2
Background..................................................................................................................... 3
Purpose .......................................................................................................................... 3
Methodology ................................................................................................................... 3
Survey Design ........................................................................................................... 3
Survey Method .......................................................................................................... 4
Sample ...................................................................................................................... 4
Survey Tools ............................................................................................................. 5
Pre-Notification Letter .......................................................................................... 5
Survey Script ....................................................................................................... 5
Survey Questions ................................................................................................ 5
Validity ...................................................................................................................... 6
Reliability................................................................................................................... 6
Survey Administration ..................................................................................................... 7
Process – Initial Survey ............................................................................................. 7
Process – Retest ....................................................................................................... 7
Scoring ........................................................................................................................... 7
Key Findings and Interpretation ...................................................................................... 8
Satisfaction Results ................................................................................................... 8
Validity ...................................................................................................................... 9
Survey Tools ............................................................................................................. 9
Surveyor Qualifications ........................................................................................... 10
Cognitive Testing .................................................................................................... 10
Reliability................................................................................................................. 11
Participant Data....................................................................................................... 12
Average Time to Administer .................................................................................... 13
Recommendations ........................................................................................................ 14
References ................................................................................................................... 15
Appendix A: Pilot Survey Participant Demographics ..................................................... 16
Appendix B: Pilot Survey Tools ..................................................................................... 18
Pre-Notification Letter.............................................................................................. 18
Initial Survey - Script and Questions........................................................................ 19
Cognitive Test Questions ........................................................................................ 24

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 1 of 24

Executive Summary
From December 2013 to July 2014, the End Stage Renal Disease (ESRD) Network
Coordinating Center (NCC), in collaboration with CMS, developed and piloted a survey
to assess grievant satisfaction with the ESRD Network grievance process. This supports
the requirements stated in Subtask 1.B. of the NCC Statement of Work (SOW): Support
for Clinical AIM 1 – Improved Patient Experience of Care and Improved Access to
Patient-Appropriate Care – Grievances, Failures to Place, Involuntary Discharges,
Involuntary Transfers.
Telephone survey instruments and administration processes were developed by CMS and the
NCC, including NCC survey experts1, and then assessed over the course of 4 meetings by an
ESRD Pilot Survey Review Board, comprised of ESRD beneficiaries and ESRD Network
representatives. This review board was created with the goal of obtaining patient input,
validating survey instruments, assessing and identifying the best methodology, initiate
discussion on the purpose of the instrument and types of data CMS is seeking, and obtaining
the members’ recommendations for establishing a robust survey process. Detailed summaries
from the review board meetings were provided to CMS to assess and incorporate participant
feedback.
The NCC obtained the pilot sample from grievant data submitted by 15 ESRD Networks that
included patient name, demographic information, contact information, treatment modality, area
of concern, and grievance close date. Pre-notification letters were mailed to 41 participants
identified by the Networks. Of those, 18 participants were randomly selected for the pilot
sample and 9 of those participants completed the survey. The pilot survey was administered
twice over a 14 to 15 day period in June and July to assess reliability. A brief cognitive test was
administered at the conclusion of the retest to obtain feedback about the participant’s
experience while taking the survey.
Findings reveal that the survey tool is both a valid and reliable tool to measure grievant
satisfaction with the Network grievance process. Feedback from cognitive testing and review
board meetings supported the validation of the survey materials and administration process.
Results from analyzing differences in participant response from the initial survey to the retest
show that response variation is not statistically significant and demonstrated the survey tool
consistently measures grievant satisfaction.
Based on an analysis of the findings of the pilot survey and feedback from the review board
meetings, the NCC identified the following recommendations to enhance the final survey tool:
(1) use relevant feedback from the review board meetings and the cognitive test findings to
create valid and reliable survey materials and finalize a standardized administration process
that meets community survey administration standards; and
(2) develop tools and a process to be used by the Networks to standardize Network data entry in
the Patient Contact Utility (PCU) in order to obtain accurate grievance data.2
1

NCC survey experts who supported the design and development of methodologies related to reliability,
validity, and scoring; these experts are (1) MA, CHCA, senior director, certified HEDIS compliance auditor,
more than 15 years of experience in healthcare quality improvement and performance assessment,
survey research, and statistical analysis; and (2) MA, PhD, 21 years of experience in research, analysis,
and preparation of reports; manages Medicaid quality improvement research studies and survey research
projects, has performed sophisticated statistical analyses and survey development.
2
This recommendation is outside of the NCC’s current Statement of Work.

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 2 of 24

The purpose of this report is to (1) describe the methodologies used to design and administer
the ESRD NCC Grievant Satisfaction Pilot Survey, (2) report findings from the pilot survey, and
(3) provide recommendations to enhance the final survey administration process, including the
survey tools.

Background
As specified by the federal regulations at 42 CFR §405.2112(g) and §§1881(c)(2)(D) of the
Social Security Act, each ESRD Network is required to evaluate and resolve beneficiary
grievances. The sources of grievances may include ESRD beneficiaries, their representatives,
other family members/caregivers, facility employees, physicians and other practitioners, federal
or state agencies, Quality Improvement Organizations (QIOs), and State Survey Agencies
(SAs). In addition, Network responsibilities under the core contract include conducting quality
improvement efforts relative to the grievance process.
The NCC is contracted by CMS to develop a satisfaction survey relative to the grievance
process, create the survey instruments, pilot test the survey process, and analyze and report
the findings. In response, the ESRD Grievant Satisfaction Survey was created to measure the
grievant’s satisfaction with the process created by CMS to be used by the Networks in
addressing grievances and non-grievance access to care issues and a pilot survey was
conducted. Prior to this period, there had been no formal measurement by CMS of patient
satisfaction with the ESRD Network grievance process.

Purpose
The purpose of the pilot survey was to (1) test the effectiveness of the survey administration
methodology, (2) test the validity and reliability of the survey questions, and (3) make
modifications to the process and tools to improve the final survey.

Methodology
Survey Design
The survey was designed to measure satisfaction with several components of the Network
grievance process, including customer service, the participant’s personal experience with filing a
grievance, overall satisfaction with the interaction with the Network in filing, investigating, and
resolving the grievance, and knowledge and understanding of the Network grievance resolution
process. The target survey population was anyone who filed a grievance in 2013 that has been
closed.
Per guidelines from the Office of Management and Budget, a maximum of 9 participants can
complete a survey without approval of a formal Paperwork Reduction Act (PRA) Information
Collection Request Package. To comply with the requirements of PRA, the target number of
completed surveys was limited to 9. The administration of the survey was limited to less than 15
minutes per CMS guidance.

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 3 of 24

Survey Method
Initially, the method of administering the survey was determined by CMS to be a written survey
distributed via mail. Advantages of a mail survey include (1) they are generally less expensive,
and (2) they are more convenient to complete for the participant as they can be answered at the
participant’s leisure. However, CMS subject matter experts3 reported prior experience with low
response rates from mail surveys, and CMS concluded that administering a telephone survey
would increase the overall survey participation rate. Other advantages of a telephone survey
include (1) participants are contacted directly, and (2) the timeframe in which results are
captured is much shorter with real-time data capture.
A test-retest method was also employed, in which the survey was administered two times in a
14 to 15-day period to the same participants to measure response variance for each question.

Sample
Each of the 18 ESRD Networks was asked to identify the survey candidates, and was
responsible for contacting and obtaining consent to participation from three individuals and
submitting the requested information to the NCC. This information included contact information,
demographic information, grievance number, treatment modality, and the primary and
secondary areas of concern. A total of 15 Networks4 submitted grievant information5, resulting
in 45 candidates. However, 4 candidates’ information could not be validated in CROWNWeb
and consequently these candidates were excluded from the sample, resulting in 41 final survey
candidates.
After compiling the information provided by the Networks, the NCC used a simple random
sample methodology to select the pilot sample. Simple random sampling was recommended by
the NCC survey experts6 and endorsed by available research findings as the most effective
method to prevent bias by ensuring that all candidates have an equal chance of being surveyed
(Davis, 2012; Schwarz, 2014), and that the sample is representative of the overall grievant
population (Davis, C. 2012; Schwarz, 2014; Black, 1999; The University of Hawaii, n.d).
To determine the sample, the following steps were taken:
 The NCC randomly selected four participants from each of the 4 CMS regions (Boston,
Kansas City, Dallas, and Seattle) using a lottery method, producing 16 participants.
 The NCC combined the remaining participants from all regions and randomly selected 2
participants.
 These 18 participants made up the pilot sample. While the target number of completed
surveys was 9, the sample size was increased to 18 in order to have a larger pool of
available participants in the event some participants could not be contacted.

3

Steven Preston, PhD, Science Officer, CMS; and Coles Mercier, MBA, CCSQ, Quality Improvement
Three Networks reported not having any data that met survey participant criteria.
5
Demographics of the participants in the pilot survey can be found in Appendix A.
6
(1) MA, CHCA, senior director, certified HEDIS compliance auditor; (2) MA, PhD
4

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 4 of 24

Survey Tools
The tools7 developed to support the pilot survey administration included:
 the pre-notification letter,
 the survey script,
 the survey questions.
As directed by the CMS subject matter expert8 and supported by a 2009 study regarding
readability and patient education materials used for low income populations (Wilson 2009), all
survey materials were developed at a fifth grade reading level. All materials were edited by the
NCC technical writer.
Pre-Notification Letter
The pre-notification letter9 was mailed to each of the 41 possible participants identified in the
survey sample. This letter described the purpose of the survey, the process for administering
the survey, the expected duration of the survey, and the timeframe in which calls will be placed
to the participants. It also included information about the confidentiality of the participant’s
responses and assurance that the survey will not change the participant’s benefits.
Survey Script
The survey script10 was developed to provide standard wording for the NCC surveyor to use
when administering the survey and included (1) a scripted introduction, (2) the survey questions,
and (3) a scripted closing statement. The introduction included the information communicated
in the pre-notification letter, introduced the NCC, its relationship to CMS and its role in the
patient satisfaction survey, confirmed the grievance close date, and asked permission from the
participant to administer the survey. It also included directions on how to answer the survey
questions and provided contact information for CMS in the event the participant had questions
related to the survey or being contacted. In the conclusion, the participant was thanked for their
time and the contact information for CMS was repeated.
Survey Questions
The survey consisted of 10 rating scale questions11 that used a scoring system to compute an
overall satisfaction rating. The Likert scale was used to format the possible responses to the
questions, as it is the most widely used rating scale and includes responses that are balanced
on both sides of a neutral option, creating a less biased measurement. Responses for rating
scale question numbers 1, 3C, 5, 6, 7, 7B, 8, and 9B ranged from very satisfied to very
dissatisfied. Responses for rating scale question number 2 ranged from very good
understanding to very poor understanding. Responses for question number 4 ranged from very
respected to very disrespected.
In addition, questions 3A, 3B, 7A, and 9A used a yes/no format to determine if the next question
applied to the participant or if the participant met the criteria for not being asked. Question 10
was an open-ended question used to obtain additional comments from the participant.

7

Each of the survey tools can be found in Appendix B.
Steven Preston, PhD, Science Officer, CMS
9
Each of the survey tools can be found in Appendix B.
10
Ibid.
11
Ibid.
8

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 5 of 24

Validity
Validity is the extent to which a survey measures what it intends to measure, and is a
requirement for any study which aims to generalize its findings. Two procedures to establish
validity were performed during survey development to ensure the survey tool accurately
measured grievant satisfaction with the Network grievance process. Each procedure focused
on collecting feedback about the survey materials and administration process from key
stakeholders, ESRD beneficiaries and ESRD Networks.
The first procedure involved a series of 2 review board meetings with ESRD beneficiaries and 2
review board meetings with the ESRD Networks to obtain feedback on the pilot survey materials
and administration process. Beneficiaries and Network representatives from each CMS region
attended separate forums to maintain beneficiary confidentiality, as well as to minimize any
chance of dominant response bias. This bias can occur when one participant influences other
participants, particularly when participants in different roles are in the same group. One of each
forum occurred prior to pilot survey administration, and one of each forum took place after the
pilot concluded. The survey tools were supplied to the review board members in advance of the
forums, with instructions to review the materials and to be prepared to provide feedback to a
predefined list of questions in a focus group format.
The next step in determining validity was to administer a cognitive test12 consisting of 5
questions at the conclusion of the survey retest. The cognitive test collected participant
feedback about the effectiveness and readability of the survey and the accompanying materials,
as well as the participant’s experience of taking the survey. Findings from the cognitive test
were presented to the review board for feedback. CMS and the NCC used feedback from both
the review board meetings and the cognitive test findings to revise the survey materials and to
refine the administration process.

Reliability
Reliability is a measure that confirms the survey tool can produce consistent results. Much like
validity, reliability is a requirement to verify that a measurement can be used in real life
scenarios. The NCC implemented two processes to establish reliability for the pilot survey.
The first was the test-retest method, as described in the Survey Method section. A low
response variance between the first survey and the second survey is an indicator of reliability.
The second process involved a review of the pre-notification letter, survey script, and survey
questions by the NCC technical writer to ensure readability, comprehension, and appropriate
use of language, formatting, and sentence structure. Following this review, the technical writer
revised the survey materials to a 5th grade reading level, edited the materials to remove passive
voice, and revised the grammar and formatting.

12

See Appendix B for the cognitive test questions.

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 6 of 24

Survey Administration
Process – Initial Survey
The pre-notification letter was mailed to the 41 possible participants 14 days prior to
administering the initial survey. Using the sample, the NCC surveyor contacted one participant
at a time from each of the 4 CMS regions until the survey had been administered to 9
participants. The surveyor contacted the participants via telephone between 9:00 AM and 7:00
PM over the course of 5 business days in June and early July, making up to 3 attempts at
different times of the day. The survey script and questions were read to the participants and
responses were recorded using an electronic Microsoft Access data collection tool developed by
the NCC specifically for the pilot survey. At the conclusion of each survey, the NCC surveyor
scheduled a time with the participant 14 to 15 days after the initial survey to administer the
retest.

Process – Retest
Over the course of 6 business days in July, the NCC surveyor contacted the participants who
completed the initial survey via telephone during the times scheduled at the conclusion of the
initial survey. The survey script and questions were read to the participants and responses were
recorded using the electronic data collection tool in the same manner as the initial survey. In
addition, a 5-question cognitive test was administered at the conclusion of the retest to obtain
feedback about the participant’s experience while taking the survey.

Scoring
The responses to the rating scale questions were assigned a value of 0 to 4 and used to
calculate the final score. The following Likert-style scale was applied to 8 of the satisfaction
questions: Very satisfied (4); Somewhat satisfied (3); Neither satisfied nor dissatisfied (2);
Somewhat dissatisfied (1); Very dissatisfied (0). In addition, 1 satisfaction question was scored
using the following Likert-style scale: Very good understanding (4); Good understanding (3);
Neither good nor poor understanding (2); Poor understanding (1); Very poor understanding (0);
Did not understand at all (-1). One question used the following Likert-style scale: Very
respected (4); Somewhat respected (3); Neither respected nor disrespected (2); Somewhat
disrespected (1); Very disrespected (0).
Points from Likert-style questions were added together and divided by the total number of
satisfaction questions answered (out of ten possible questions) to determine the composite
score for overall satisfaction. A threshold of 80% percent was recommended by a CMS subject
matter expert13 as an indicator of overall satisfaction.
For the single open-ended question, key comments were identified in each response and
grouped into two categories: (1) Network Approval, and (2) Other. Yes/no questions were not
scored, as they were used to identify if a participant met the criteria to respond to a satisfaction
question.

13

Steven Preston, PhD, Science Officer, CMS

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 7 of 24

Key Findings and Interpretation
Satisfaction Results
In accordance with the defined scoring methodology, 6 participants reported satisfaction (≥ 80%
overall satisfaction) with the Network grievance process, while 2 participants expressed
dissatisfaction (< 80% overall satisfaction) during both the initial survey and retest14. Of the
participants who expressed satisfaction during the initial survey, 1 participant had an overall
satisfaction score of 87%, whereas 5 participants had overall satisfaction scores between 93%
and 100%. Although the only measurement of overall satisfaction was a score of ≥80%, the
majority of satisfaction scores for the initial survey were extremely high. Of the 2 participants
who expressed overall dissatisfaction during the initial survey, 1 score was 75%, just below the
80% satisfaction threshold, and the other was exceptionally low at 32% due to dissatisfaction
from a lack of good customer service.
For the retest, overall satisfaction scores changed slightly. The total number of participants
reporting an overall satisfaction score of 80% or higher remained the same at 6, however 3
participants reported overall satisfaction scores between 83% and 87%, and 3 participants
reported scores between 97% and 100%. The biggest change involved 1 participant who had
an improved overall satisfaction score from 32% on the initial survey to 64% on the retest. This
may be attributed to the result of research showing that customers are more satisfied when they
believe that their opinion is important.15
Table 1 below provides a summary of the responses received for the open-ended question
during pilot administration. The NCC grouped responses into two different categories: (1)
Network Approval, and (2) Other. While the sample size of this pilot survey was small, the
comment regarding a concern for possible retaliation confirms a hypothesis identified by CMS
and the NCC at the beginning of this project. During the survey development process, 2
questions about possible retaliation that were originally included were removed from the survey
for inclusion in an environmental scan. This comment from the initial pilot supports the assertion
to add these 2 questions back into the survey.

14

These satisfaction results are for the 8 participants who completed both the initial survey and the retest;
one participant who completed the initial survey could not be reached at the time the retest was
administered.
15
West, Ken. (2014). 10 Factors That Affect Customer Satisfaction. National Business Research Institute.
Retrieved from http://www.nbrii.com/customer-survey-white-papers/10-factors-that-affect-customersatisfaction/

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 8 of 24

Table 1. Summary of Open-ended Question
Network Approval
“The Network was very conscientious and they knew the situation I was in. The
grievance was resolved that day.”
“The Network provides a useful service that provides an avenue for patients to resolve
grievances when they occur.”
The Network did an outstanding job as far as coordinating and contacting the right
people to resolve my grievance and making sure things were taken care of.”
Other
Additional Concern: Me and other patients are afraid to file a concern about a staff
because everyone is family and we feel that the staff will become hostile to us in
retaliation.”
Criticism for Dialysis Facilities: “The nurses and the personnel at the dialysis facility in
general should not treat their job like an assembly line; they need to treat patients like
people. The facility dehumanizes people.”

Validity
By obtaining feedback and approval from ESRD beneficiaries and ESRD Networks through the
review board meeting format, the NCC was able to validate both the survey materials and the
survey administration process. Responses collected through cognitive testing also supported
survey validation and were used for further discussion with the review board to identify ways in
which the survey process could be improved. Based on these findings, the following changes
were made:
 Highlighted the purpose of the survey in both the survey script and pre-notification letter.
 Identified a procedure to officially translate survey materials into Spanish.
 Emphasized the number of contacts in the pre-notification letter that will be made to
administer the survey.

Survey Tools
Feedback from the review board meeting with the Networks held prior to the pilot survey on
April 28, 2014 included the following recommendations to the survey tools:



Emphasize the purpose of the survey in both the survey script and pre-notification letter
to prevent the chance of the participant conflating the grievance process with the
grievance outcome.
Include an additional question asking if the participant would recommend the Network
grievance process to a friend on dialysis.

Feedback from the review board meeting with beneficiaries held prior to the pilot survey on May
5, 2014 included the following recommendations to the survey tools:



Officially translate Spanish language survey tools.
Emphasize the number of contacts that will be made to administer the survey.

At the conclusion of the first series of review board meetings, the NCC provided CMS with a
summary of participant feedback. CMS and the NCC used applicable feedback to revise the
survey script, questions, and pre-notification letter prior to pilot survey administration.

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 9 of 24

Two additional review board meetings were held after the pilot survey concluded. Feedback
from the review board meeting with the Networks on July 24, 2014 did not yield any substantive
findings. The meeting with beneficiaries on July 22, 2014 included the following
recommendations to the survey tools:






Change the survey script to be more conversational.
Reorganize the survey script to be more clear.
Add a question asking if the participant feels comfortable enough with the Network
grievance process to file another grievance.
Restate assurances at numerous points throughout the survey script and pre-notification
letter about the confidentiality of the participant’s responses and that the survey will not
change the participant’s benefits.
Clearly state in the survey script and the pre-notification letter that the surveyor is not
employed by the Network or the dialysis facility.

Surveyor Qualifications
Feedback from every review board meeting revealed a shared concern for skilled surveyors. In
addition, CMS expressed concern about the use of NCC surveyors as IPRO, the NCC’s parent
company, also operates two ESRD Networks. CMS identified this as a potential conflict of
interest. Therefore, a subcontract with a 3rd party survey administrator is recommended and will
require expertise in survey administration, data collection, and analysis. The use of skilled
surveyors to administer the survey will likely increase the overall response rate, as trained
interviewers will be able to employ professional survey administration techniques.

Cognitive Testing
Cognitive testing yielded a great deal of substantive feedback. The findings are presented in
Figure 1.0. Each of the participants (1) reported having enough information to easily answer
each question, (2) did not feel the survey questions were too long, and (3) were unable to
identify any questions that could be added to the survey. Interestingly, 50% of participants
believed the survey questions differed from initial survey to retest, although there were no
differences in the questions. A participant’s inability to remember survey content may be
attributed to the 14 to 15-day gap between both tests, in addition to the number of questions
included in the survey, which could have been difficult to commit to memory during the brief
survey administration period.
Participant feedback for the question, “Is there anything we can do to improve the survey?”
revealed a number of useful recommendations for consideration. In particular, participant
feedback underscored the need to provide additional confidentiality assurances in survey
materials, as well as assurances that beneficiary care and benefits will not be affected as a
result of participation in the survey. Other recommendations included the option of providing
participants with a printed copy of the survey, adding questions about grievance follow-up or
grievance outcome, as well as the suggestion to administer the survey as soon as possible after
the Network closes the grievance.

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 10 of 24

Figure 1.0. Cognitive Test Findings

Cognitive Test - Participant Response
100%
80%
60%
Yes
No

40%

20%
0%

Did you think Did you have
Did my
Are there any
Is there
my questions enough info to questions
questions you anything we
today were the easily answer today or the would add to
can do to
same as last
each
first time take the survey?
improve the
time?
question?
too long to
survey?
answer?

Reliability
Overall findings reveal that the survey questions are reliable, as the variation in responses to
individual survey questions from the initial survey to the retest did not produce a significant
difference.
The statistical analysis of the test-retest results for each question is presented in Figure 2.0
below. Correlation coefficients, using both the Spearman’s rho correlation and the Pearson
correlation, were calculated to assess the strength and direction of the relationship between the
first and second survey scores. There were 8 participants who completed both the initial survey
and the retest, as 1 participant could not be reached for the retest.
Correlation coefficients could not be computed for questions 3C and 7B, as there was no
variation in responses from the initial survey to the retest. A correlation coefficient could not be
analyzed for question 9B since no responses were received during both tests due to all
participants qualifying to not administer this question.

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 11 of 24

Figure 2.0. Test-Retest Statistical Analysis
Survey Question
N
#
1
8
2
6
3C
4
4
8
5
8
6
8
7
8
7B
2
8
8
9B
0
Overall
8

Correlation
Coefficient
.69
.42
N/A
.99
.52
.47
.57
N/A
.42
N/A
.88

Statistical
Significance
N.S.
N.S.
N/A
.0001
N.S.
N.S.
N.S.
N/A
N.S.
N/A
.004

N = number who completed both initial survey and retest
N.S. = not significant
N/A = not applicable

Participant Data
Three participants selected in the sample were ineligible to participate in the survey due to
criteria that were not disclosed at the time the Networks submitted candidate data. These
criteria included incorrect contact information, a grievance close date of 2012, and death of a
participant. The lack of accurate data can most likely be attributed to incorrect data entry in the
Patient Contact Utility (PCU), the database the Networks use to capture grievance data. CMS
has acknowledged that the Networks have not been given a standard process to follow when
entering grievances, and as a result, grievance data is not being entered in a consistent or
standardized way by the Networks into the PCU.
Any individual who filed a grievance in 2013 and had the grievance closed was eligible to be
included in the pilot sample. Pilot participants represented three age groups (40-49; 50-59; 6069), which are characteristic of the majority of ESRD patients as reported in the CMS End Stage
Renal Disease Network Organization Program 2011 Summary Annual Report (2011 SAR, p.
80)16. Participant gender was also representative of the figures reported in the 2011 SAR, with
56% male and 44% female participants (p. 81), while participant race was nearly characteristic
with 56% White and 44% Black or African American participants (p. 82). Ethnicity reporting
showed that 89% of participants were categorized as Not Hispanic or Latino, whereas 11%
were categorized as Hispanic or Latino. Although there was not equal participant
representation from across the four CMS regions, participants from each region were included
in the sample (Region 1/Boston = 33.3%; Region 2/Kansas City =22.2%; Region 3/Dallas =
33.3%; and Region 4/Seattle = 11.1%).17

16

Centers for Medicare and Medicaid Services. End Stage Renal Disease Network Organization Program
2011 Summary Annual Report. Baltimore, MD: CMS; 2012.
17
Complete demographics of the participants in the pilot survey can be found in Appendix A.

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 12 of 24

In June 2014, CMS convened the PCU Data Committee18, comprised of CMS, NCC, and
Network representatives, to address issues related to the standardization of PCU data entry and
reporting by identifying (1) inconsistencies in data entry, (2) opportunities for standardization, (3)
training needs, (4) reporting requirements, and (5) changes to the PCU extract query.
Identification of these items by the PCU Data Committee and the process of developing
solutions will ensure that the data entered into the PCU at the time of full survey administration
(estimated April 2015) will be of higher quality, minimizing the likelihood of extracting incorrect
participant information.

Average Time to Administer
The average time to administer the initial survey was 15:22. Two outliers (24:08; 38:23) were
responsible for the average time to administer exceeding 15 minutes in length, as the remainder
of administration times were between 7:31 and 14:47. Conversely, the average administration
time of the retest was 8:48. Retest times included 7 participants with an administration time
between 6:40 and 9:32, and 1 outlier whose call was 15:32 due to an extensive answer for the
wrap-up question.
The difference in the average time to administer between the initial survey (15:22) and the retest
(8:48) can be credited to the majority of participants (1) electing not to restate the response
provided for the wrap-up question (originally provided during the initial survey) for the retest, and
(2) submitting the response to the survey question prior to the NCC surveyor providing the full
list of response options. The latter can be attributed to participants becoming more familiar with
the survey response options (majority of satisfaction question response options are alike) after
similar response options were provided in previous questions. The projected duration of the full
survey is expected to be in between the 2 averages: (1) the average time to administer the initial
survey, and (2) the average time to administer the retest. This projected time is 12 minutes.

18

The PCU Data Committee was suspended pending CMS direction on contractor responsibility.

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 13 of 24

Recommendations
Based on the findings of the pilot survey, the NCC recommends the following actions to
strengthen the survey process prior to full administration.




Implement applicable feedback from the review board and cognitive test findings to
improve the quality of the survey materials and to standardize the administration
process.
o Feedback from the review board
 Change the survey script to be more conversational.
 Group related subject matter together to make the survey script clearer.
 Add a question asking if the participant feels comfortable enough with the
Network grievance process to file another grievance.
 Restate assurances at defined points throughout the survey script and in
the pre-notification letter about the confidentiality of the participant’s
responses as well as the assurance that there will be no change in the
care and services the participant receives.
 Clearly state in the survey script and the pre-notification letter that the
surveyor is not employed by the Network or the dialysis facility.
 Develop an education plan to ensure survey administrators are trained to
employ professional survey administration techniques.19
o Feedback from the cognitive test
 Add additional confidentiality assurances to the survey tools.
 Add assurances that beneficiary care and benefits will not be affected by
the survey responses or participation.
 Provide participants with a printed copy of the survey.
 Administer the survey as soon as possible following the time the
grievance is closed.
Develop a process to standardize Network grievance data entry in the PCU in order to
obtain accurate grievance data.20

Implementing these recommendations will further validate the survey by ensuring both the tools
and processes used to collect participant responses incorporate the feedback from the pilot
survey findings and stakeholder review board. Validating the survey in this way increases the
likelihood that the fully developed Grievant Satisfaction Survey will achieve its objectives.
Moreover, the survey results will assist in identifying areas for improvement in (1)
standardization of the grievance process, (2) actionable quality improvement activities at each
ESRD Network and the ESRD Network program as a whole, and (3) increased assurance of
meeting the requirements of the PRA package approval process.

19
20

rd

The use of a 3 party survey administrator will address this recommendation.
This recommendation is outside of the NCC’s current Statement of Work.

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 14 of 24

References
Black, T. R. (1999). Doing quantitative research in the social sciences: An integrated approach
to research design, measurement, and statistics. Thousand Oaks, CA: SAGE Publications, Inc.
Davis, C.S., Gallardo, H.P., & Lachlan, K. (2012). Straight talk about communication research
methods (1st. Ed.). Dubuque, IA: Kendall-Hunt Publishing.
Schwartz, C.J. (2014). Sampling. In Course Notes for Beginning and Intermediate Statistics.
Retrieved from http://www.stat.sfu.ca/~cschwarz/CourseNotes.
The University of Hawaii (n.d.). Sampling strategies and their advantages and disadvantages.
Retrieved August 28, 2014, from
http://www2.hawaii.edu/~cheang/Sampling%20Strategies%20and%20their%20Advantages%20
and%20Disadvantages.htm
Wilson, M . (2009). Readability and patient education materials used for low-income. Clinical
Nurse Specialist, 23(1), 33-40.

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 15 of 24

Appendix A: Pilot Survey Participant Demographics21

CMS Region
11% 33%
33%
22%

Boston
Kansas City
Dallas
Seattle

Gender
44%
56%

21

Male
Female

Sum of percentages less than 100% are due to rounding.

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 16 of 24

Age
33%

22%
44%

0 - 19
20 - 29
30 - 39
40 - 49
50 - 59
60 - 69
70 - 79
80 - 89

Race
44%
55%

Black or
African
American
White

Ethnicity
11%

89%

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Not Hispanic
or Latino
Hispanic or
Latino

Version 2.0 / Revision Date: 10/8/14
Page 17 of 24

Appendix B: Pilot Survey Tools
Pre-Notification Letter
June X, 2014

Dear [Mr./Ms./Dr. Name of Grievant]
I work for the End Stage Renal Disease (ESRD) Network Coordinating Center (NCC). We work with the
Centers for Medicare & Medicaid Services (CMS). Part of our job is to make sure patients are happy with
the way their ESRD Network handles their grievance.
We are now choosing ESRD patients who have filed a grievance to take part in a survey. We will use
what we learn from this survey to help CMS support ESRD Networks and improve the way they handle
the grievance process.
Our records show you filed a grievance with [Network Name] on or around [Complete Date]. If this is true,
we might want to ask you some questions about your contact with [Name of Network] during 2013. If we
choose you to take part, a surveyor from the ESRD NCC will call you to set up an interview. You can
decide then whether you want to talk to the surveyor.
If you did not file a grievance, please tell the NCC surveyor if he or she calls you. We will try to call you
three times to set up the first interview. At the end of the first interview, we will set up a time for the
second telephone interview. We will choose a time that is good for you. We will have the second
interview two weeks after the first one.
If we choose you, one of our ESRD NCC surveyors will call you on the following dates:


mm/dd/yyyy and mm/dd/yyyy [between 9:00 am and 7:00 pm]; and



mm/dd/yyyy and mm/dd/yyyy [between 9:00 am and 7:00 pm]

If you get a phone call but do not have time to talk, you can choose a better time for your interview.
The surveyor will ask different types of questions. Each call should last 10 or 15 minutes.
Your thoughts and feedback are very important to us. But If you choose not to take this survey, there will
be no change in the care and services you receive. The questions will be about your contact with
[Network Name] during the time you filed your grievance. They will not be about what happened with
your grievance. We will keep your answers private. Your dialysis facility or [ESRD Network] will not see
your answers. Your answers will not change your Medicare benefits.
If you have any questions, please call Renee Dupee at CMS (410-786-6747). You can also send an email
to [email protected].
Thank you for your time.
Sincerely,

Walter Linney, MA
Patient Services Project Manager
End Stage Renal Disease Network Coordinating Center

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 18 of 24

Initial Survey - Script and Questions
Introduction
Hello, may I please speak with [participant name]?
If the participant is unavailable, the surveyor will thank the person on the phone and end the call
by saying, “Thank you for your time.”
If the participant answers the phone, state the following:
Good Morning/Evening [participant name], this is [surveyor name], I’m calling from the End
Stage Renal Disease Network Coordinating Center. We call it the ESRD NCC for short. I want
to ask you some questions about your contact with the ESRD Network when you filed your
grievance. We would like to talk about how you felt during your contact with [Network Name].
We will not talk about the result.
You might want to take some notes while we talk. If you want to get a pencil and paper before
we begin, I can wait. Wait for respondent to get pencil and paper.
The ESRD NCC contracts with CMS. We are responsible for leading this survey. We will use
what we learn today to help CMS improve the process.
What we talk about today is private; we will not share what you say with your ESRD Network or
with any dialysis centers. You do not have to take the survey if you do not want to. The survey
will not change your Medicare benefits. We will talk for about 10 or 15 minutes.
I will give you a number to call in case you have questions about the survey. You can also ask
about the NCC and its role in the survey. If you have questions, please call Renee Dupee at
410-786-6747 at CMS.
Our records show you contacted [Network Name] around [complete date] to file a grievance.
Is this correct?
If yes, continue with survey. If no, conclude survey and thank the respondent.
Is this a good time for you to take the survey?
If yes, continue with the survey. If no, the surveyor will schedule a follow-up call.
Grievance/ Process: The next few questions are about the way [Network Name] handled your
grievance. Please consider only the question I ask. Try not to think about whether your
grievance turned out the way you wanted. I will give you a list of answers, and you can choose
the best one.

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 19 of 24

1. How satisfied were you with the customer service [Network Name] provided when you first
contacted them to talk about your grievance?
Very satisfied
Somewhat satisfied
Neither satisfied nor dissatisfied
Somewhat dissatisfied
Very dissatisfied

(4)
(3)
(2)
(1)
(0)

No answer/Don’t know

(9)

2. After speaking with the Network, did you have a good understanding of your right to file a
grievance?
Very good understanding
Good understanding
Neither good nor poor understanding
Poor understanding
Very poor understanding
Did not understand at all
Network did not explain grievance rights

(4)
(3)
(2)
(1)
(0)
(-1)
(-2)

No answer/Don’t know

(9)

3A. Did you talk more than once with [Network Name] while your grievance was in process?
Yes (go to 3B)
No (go to 4)
3B. Did a patient representative or someone who works with patients at your dialysis facility help
you with your grievance?
Yes (go to 3C)
No (go to 3C)
3C. How satisfied were you with the customer service [Network Name] provided in follow-up
talks during your grievance?
Very satisfied
Somewhat satisfied
Neither satisfied nor dissatisfied
Somewhat dissatisfied
Very dissatisfied

(4)
(3)
(2)
(1)
(0)

No answer/Don’t know

(9)

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 20 of 24

4. Did you feel respected while [Network Name] processed your grievance?
Very respected
Somewhat respected
Neither respected nor disrespected
Somewhat disrespected
Very disrespected

(4)
(3)
(2)
(1)
(0)

No answer/Don’t know

(9)

5. How satisfied were you that the Network listened to your concerns and understood them?
Very satisfied
Somewhat satisfied
Neither satisfied nor dissatisfied
Somewhat dissatisfied
Very dissatisfied

(4)
(3)
(2)
(1)
(0)

No answer/Don’t know

(9)

6. How satisfied were you with the Network’s effort to process your grievance?
Very satisfied
Somewhat satisfied
Neither satisfied nor dissatisfied
Somewhat dissatisfied
Very dissatisfied

(4)
(3)
(2)
(1)
(0)

No answer/Don’t know

(9)

7. How satisfied were you with the way the Network acted in your best interest
Very satisfied
Somewhat satisfied
Neither satisfied nor dissatisfied
Somewhat dissatisfied
Very dissatisfied

(4)
(3)
(2)
(1)
(0)

No answer/Don’t know

(9)

7A. Did you get a letter from [Network Name] with results of their work to resolve your
grievance? (Item is not scored.)
Yes (if yes, go to 7B)
No (if no, go to 8)

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 21 of 24

7B. How satisfied were you with the letters you received from the Network?
Very satisfied
Somewhat satisfied
Neither satisfied nor dissatisfied
Somewhat dissatisfied
Very dissatisfied

(4)
(3)
(2)
(1)
(0)

No answer/Don’t know

(9)

8. Overall, how satisfied were you with the help [Network Name] offered you to resolve your
grievance?
Very satisfied
Somewhat satisfied
Neither satisfied nor dissatisfied
Somewhat dissatisfied
Very dissatisfied

(4)
(3)
(2)
(1)
(0)

No answer/Don’t know

(9)

9A. Did you file an appeal based on the Network’s decision about your grievance?
No (go to question 10)
Yes (go to 9B)
9B. How satisfied were you with the Network appeal process?
Very satisfied
Somewhat satisfied
Neither satisfied nor dissatisfied
Somewhat dissatisfied
Very dissatisfied

(4)
(3)
(2)
(1)
(0)

No answer/Don’t know

(9)

Wrap-Up Question
10. Would you like to add any thoughts about your contact with [Network Name] during the
time you filed your grievance?
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 22 of 24

Conclusion
On behalf of the ESRD NCC, I want to thank you for your time today. Again, if you have any
questions or concerns about this survey or the way I asked questions, please contact CMS at:
Renee Dupee, CMS
410-786-6747
[email protected]

For Pilot Survey Process Only:
Someone from the NCC will call you in two weeks on [exact date] to complete the second part
of our survey. The second call will be questions like those we asked this time. You will also
have a chance to tell us what you thought about the survey. You can always choose whether
you want to take the survey.
Is [exact time] the best time to reach you?
And is [exact phone number] the best phone number?
Thank you again. I look forward to talking to you on [exact date].
Have a good day/evening. Goodbye.

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 23 of 24

Cognitive Test Questions
1. Did you think my questions today were the same as last time?
Yes
No
2.

Did I give you enough information for you to easily answer each question?
Yes
No

3. Did my questions today or the first time take too long to answer?
Yes
No
4. Are there any questions you would add to this survey?
Yes
No
If yes:
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
5.

Is there anything we can do to improve the survey?

Yes
No
If yes:
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
Conclusion
On behalf of the ESRD NCC, I want to thank you for your time today. Again, if you have any
questions or concerns about this survey or the way I asked questions, please contact CMS at:
Renee Dupee, CMS
410-786-6747
[email protected]

ESRD Network Coordinating Center (NCC)
Grievant Satisfaction Pilot Survey Findings Report

Version 2.0 / Revision Date: 10/8/14
Page 24 of 24