The National Implementation of the ICH CAHPS Survey

Status: Original and derived artifacts are available for this document.

Download: docx | pdf | html

Supporting Statement Part B 
 
The In-Center Hemodialysis Consumer Assessment of Healthcare Providers 
and Systems (CAHPS) Survey Mode Experiment 
 
	B. 	COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS 
 
The Centers for Medicare & Medicaid Services (CMS) is requesting clearance from the Office of Management and Budget (OMB) to test a web mode as an alternative to the existing approved modes of administration and to test a survey that includes home dialysis patients in addition to in-center hemodialysis (ICH) patients. The In-Center Hemodialysis Consumer Assessment of Healthcare Providers and Systems (ICH CAHPS) Survey measures patients’ experience of care with in-center hemodialysis facilities under Contract Number GS-00F354CA - 75FCMC25FJ128. 
 
	B.1 	Potential Respondent Universe and Sample Selection Method 
 
B.1.1. Sampling Patients for the Mode Experiment  
 
The ICH CAHPS Survey is administered twice annually to ICH patients who are at least 18 years of age or older, have received hemodialysis from their current facility for 3 months or longer, and are not currently receiving hospice care or living in a nursing home/skilled nursing facility or other long-term facility such as a prison or jail. For this mode experiment, we will use the same eligibility criteria but also include patients who receive home hemodialysis or peritoneal dialysis at home (both referred to as home dialysis patients). 
 
For the mode experiment, we will select a sample using data from the End Stage Renal Disease Quality Reporting System (EQRS), choosing a mix of types of facilities and patient-mix factors. Each patient will be randomly assigned to one of the four data collection arms: mail-only, telephone-only, mail with telephone follow-up, web with mail and telephone follow-up. Exhibit B.1 below presents our proposed sample sizes and anticipated response rates by mode. Response rates are based on current ICH CAHPS response rates for the 3 modes implemented, and our best guess at a mixed-mode web option.  
 
 	 
Table 1. 	Proposed Mode Experiment Sampling Metrics 
Metric 
Mail 
Phone 
Mail/Phone 
Web/Mail/Phone 
Sample Size 
5335 
6277 
4355 
4268 
Completed Surveys: 
Mail 
1067 
NA 
740 
403 
Completed Surveys: 
Phone 
NA 
1067 
325 
241 
Completed Surveys: 
Web 
NA 
NA 
NA 
427 
Total Completed 
Surveys 
1067 
1067 
1065 
1071 
Response Rate 
20% 
17% 
24.5% 
25.1% 
 
	B.2 	Information Collection Procedures 
 
Sample patients in the mode experiment will be randomly assigned into one of four modes (mail-only, telephone-only, mail with telephone follow-up, or web with mail and telephone follow-up.  These four modes are described below: 
    • Mail-only Mode: All sample patients are sent a prenotification letter, explaining the purpose of the survey, and letting patients know that a hardcopy survey will soon be sent to them via mail. A survey package consisting of a cover letter, the ICH CAHPS survey, and a pre-addressed, postage-paid return envelope is sent to all sample patients two weeks following the prenotification letter. A second mailing containing a survey and cover letter is mailed to all sample patients who do not respond to the first mailing approximately four weeks after the first survey package is mailed. Data collection ends ten weeks after the first survey package is mailed and twelve weeks after the prenotification letter is mailed. 
    • Telephone-only Mode: All sample patients are sent a prenotification letter letting them know that a professional interviewer working on the ICH CAHPS Survey will soon be contacting them via telephone. All sample patients are then contacted by professional telephone interviewers who are trained on ICH CAHPS survey administration procedures, including procedures for working with dialysis patients. Telephone interviewers are trained on the appropriate response to common questions and concerns that dialysis patients might have about survey participation, and are required to offer to administer the interview in different call- backs if the sample patient indicates that he or she cannot complete the interview in one call. A maximum of 10 telephone contact attempts per patient are attempted to complete the survey. Data collection ends ten weeks after the initial telephone contact begins and twelve weeks after the prenotification letter is mailed. 
    • Mail with Telephone Follow-Up Mode: All sampled patients are sent a a prenotification letter letting them know that we will soon be contacting them via mail. We then send an initial mailing of a survey, cover letter, and postage-paid return envelope. Sample patients assigned to this mode who do not respond to the mail survey within four weeks after the survey is mailed will be assigned to the telephone follow-up. Telephone interviewers make up to 10 attempts to complete the interview by phone with all mail survey non-respondents. Data collection ends ten weeks after the first survey package is mailed and twelve weeks after the prenotification letter is mailed. 
    • (NEW – Mode Experiment) Web with Mail and Telephone Follow-Up Mode: All sampled patients in the experimental web mode will be sent a prenotification letter letting them know that we will soon be contacting them via email and/or mail. For those with email addresses available, a web survey invitation with a hyperlink to complete the online survey will be emailed a week after the prenotification letter. Non-respondents will receive a reminder email one week later. The sampled patients assigned to this experimental mode who do not have an email address will receive a letter in the mail with a short hyperlink to the survey and a QR code that they can scan with a smartphone or tablet; this will be mailed one week after the prenotification letter. Approximately two weeks after the initial web invitations are emailed/mailed, a mail survey package with cover letter will be sent to nonrespondents; this will contain a push to the web with the short hyperlink and QR code. Sampled patients who do not respond to the mailed survey within four weeks will receive telephone calls from interviewers to complete the interview by phone. The telephone interviewers will make up to 10 attempts to complete the interview by phone. Data collection ends ten weeks after the initial email/mail contact begins and twelve weeks after the prenotification letter is mailed.  
	B.3 	Methods to Maximize Response Rate 
 
This data collection effort will test the addition of a web mode to the survey administration procedures, as described above. All surveys including the ICH CAHPS survey have faced declining response rates, as shown in Exhibit B.2. The ICH CAHPS survey response rate is highest for the existing mixed mode (mail with telephone follow-up) at 28.6% versus 24.2% for mail-only and 23.% for telephone-only. CMS believes a web with mail and telephone follow-up mode will likely increase response rates.  
 
Exhibit B.2. Response Rates for ICH CAHPS Semiannual Surveys 
 
 
 
Survey Period 
 
# 
Sampled 
 
# Eligible 
# Completed 
Surveys 
 
RR 
 
2014 Fall 
 
329,493 
 
305,590 
 
113,935 
 
37.3% 
 
2015 Spring 
 
363,181 
 
337,316 
 
114,847 
 
34.0% 
 
2015 Fall 
 
356,721 
 
324,139 
 
103,808 
 
32.0% 
 
2016 Spring 
 
363,670 
 
339,092 
 
107,582 
 
31.7% 
 
2016 Fall 
 
347,879 
 
323,386 
 
100,184 
 
31.0% 
 
Survey Period 
 
# 
Sampled 
 
# Eligible 
# Completed 
Surveys 
 
RR 
 
2017 Spring 
 
348,024 
 
323,185 
 
111,851 
 
34.6% 
 
2017 Fall 
 
351,700 
 
321,818 
 
105,120 
 
32.7% 
 
2018 Spring 
 
332,183 
 
307,078 
 
98,611 
 
32.1% 
 
2018 Fall 
 
344,827 
 
316,093 
 
102,757 
 
32.5% 
 
2019 Spring 
 
353,703 
 
325,690 
 
98,868 
 
30.4% 
 
2019 Fall 
 
356,127 
 
324,732 
 
96,255 
 
29.6% 
 
2020 Spring 
 
29,676 
 
26,827 
 
8,571 
 
31.9% 
 
2020 Fall 
 
350,949 
 
322,607 
 
96,805 
 
30.0% 
 
2021 Spring 
 
345,448 
 
316,014 
 
82,987 
 
26.3% 
2021 Fall 
            
354,025  
                   
330,102  
              61,930 
18.8% 
2022 Spring 
            
366,899  
                   
346,823  
              90,198 
26.0% 
2022 Fall 
            
340,354  
                   
320,760  
              76,154 
23.7% 
2023 Spring 
            
378,513  
                   
355,613  
              91,914 
25.8% 
2023 Fall 
            
383,116  
                   
362,718  
              86,566 
23.9% 
2024 Spring 
            
389,982  
                   
369,304  
              89,306 
24.2% 
2024 Fall 
397,897 
374,576 
91,591
 24.5% 
2025 Spring 
428,172 
404,279 
98,757
 24.4% 
2025 Fall 
425,445 
402,896 
92,277
 22.9% 
 
All Survey Periods 
         
8,037,984 
                
7,480,638  
        2,120,874 
28.4% 
 
Each prenotification letter and survey package envelope will contain the CMS Logo. Because some dialysis patients may be reluctant to participate because of fear of retribution from their dialysis centers, the mail survey materials will contain assurances that the patients’ survey responses will be kept private and cannot be linked to their names.. 
We will use best practices in the survey materials to enhance response rates: using a simple font no smaller than 12 point size in the survey cover letters, allowing ample white space between questions in the survey, avoiding a format that displays the questions as a matrix, using a unique subject identification number on the survey rather than printing the sample member’s name, and displaying the OMB number and expiration date on the survey.0 
For sample patients included in the mail-only data collection for the ICH CAHPS Survey, the second survey mailing is expected to increase the response rate. The cover letter included in the second survey package to mail survey nonrespondents contains a stronger appeal for the sample patient’s help on this survey, including indicating that the survey is an opportunity for them to provide input on the quality of dialysis care dialysis patients receive. To maximize response rates for the telephone-only mode and the telephone follow-up of the mail/telephone and web/mail/telephone modes, we will make up to 10 attempts to reach each sample patient, with those attempts varying by day of the week and time of day. Telephone interviewers are trained on how to answer the questions that are most frequently asked by sample patients, and to address any concerns that they may have about participating in the survey. Because some dialysis patients may not feel well on the day that they receive dialysis treatments, telephone interviewers are instructed to offer to call back at a time that is better for the sample patient, and offer to conduct the telephone interview on two or more different calls.  	For the new web-based mode, we will test the use of a 3rd party email address search. We will also use short URL hyperlinks and unique codes in both email and mailed materials, so that patients can easily reach the web survey. In addition, we will use QR codes on the mailed materials so that patients can quickly be routed to the survey via a smartphone or tablet.  
	B.4 	Tests of Procedures 
The proposed mode experiment will test a new web-based mode of survey administration along with a revised survey that will include home dialysis patients. The following analyses will be conducted using the mode experiment data: psychometric analysis, mode and patient-mix analyses, and non-response analysis. These analyses are described in detail below:  
Psychometric Analyses:  
    • Confirm that public reporting metrics continue to be based on solid patient experience. Leverage exploratory analyses to determine item placement in composites or uncover a new composite structure.  
    • Assess revised survey items for reliability and validity in mixed-method format, including itemlevel analysis (Rasch measurement) and classical psychometrics (internal consistency and factor analysis).  
    • Examine item-level measurement assessments to rigorously evaluate composite dimensionality, continued and newly developed response option functioning, items not producing useful measurement into the existing composite constructs, and item-level bias for key demographic and mode groups. 
Mode Effects and Patient-Mix Analyses:  
    • Conduct patient-mix analyses to determine whether modifications should be made for patient-mix adjustments and mode effects, based on revised surveys.  
    • Examine top-box, bottom-box, and linear means for all global ratings and composites.  
    • Replicate psychometric analyses for mode to determine whether the experimental web-based mode provides the same psychometric quality as mail and phone modes.  
Nonresponse Analyses:  
    • Establish whether patient characteristics affecting nonresponse should also be used in creating patient-mix adjusted scores by using nonresponse-adjusted weights.  
○ If no statistically significant correlations exist between nonresponse weights and residuals of our regression models, including patient-mix and mode adjusters, there will be no need to adjust the data.  
○ To date, nonresponse-adjusted weights have not been needed for ICH CAHPS.  
 
	B.5 	Statistical Consultation and Independent Review 
 
This sampling and statistical plan for the mode experiment was prepared by RTI 
International. The mode experiment statistical design was provided by Amy Couzens, the Sampling Task Leader at RTI and can be reached by telephone at 919-541-5910 or by email at [email protected].