The National Implementation of the ICH CAHPS Survey

Status: Original and derived artifacts are available for this document.

Download: docx | pdf | html

Supporting Statement Part B 
 
National Implementation of the In-Center Hemodialysis CAHPS Survey
(CMS-10105, OMB 0938-0926)
 
	B. 	COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS 
 
The Centers for Medicare & Medicaid Services (CMS) is requesting clearance from the Office of Management and Budget (OMB) to continue implementing nationally the In-center Hemodialysis CAHPS (ICH CAHPS) Survey to measure patients’ experience of care with in- center hemodialysis (ICH) facilities under Contract Number GS-00F-354CA - 75FCMC20F0078. 
 
	B.1 	Potential Respondent Universe and Sample Selection Method 
 
B.1.1. Sampling Patients for the National Implementation 
 
Medicare-certified dialysis facilities that serve more than 30 survey-eligible sample members during a calendar year are required to contract with a CMS-approved survey vendor to collect and submit ICH CAHPS Survey data on their behalf. All approved ICH CAHPS Survey vendors are required to use standardized survey administration protocols and specifications provided by CMS. The national implementation of the ICH CAHPS survey is conducted on a semiannual basis, with sampling and data collection activities conducted as shown in Exhibit B.1. 
 
Exhibit B.1. Typical Sampling Window for the ICH CAHPS Semiannual Surveys 
 
 
Spring survey 
Fall survey 
Typical sampling window (months in which patients received ICH care) 
October-December 
April-June 
Sample selected 
March 
September 
Data collection period 
April-July 
October–January 
 
 
The national implementation of the survey is fielded on a rolling semiannual basis. The results for each semiannual survey are merged with data from the immediately preceding semiannual survey for developing composite measures for public reporting. A primary issue is obtaining sufficient sample size within a facility to produce confidence intervals for point estimates that are sufficiently narrow. Approximately 200 observations are needed per year to produce a confidence interval that has a bound of +/−0.07. Approximately 92% of Medicare- certified ICH facilities serve 100 or fewer unique patients a year; 7.5% serve between 101 and 200 patients a year, and less than 1.0% serve more than 200 patients each year. 
For each semiannual wave, patients who received care during the sampling window and who meet survey eligibility criteria will either be chosen randomly or selected with certainty depending on the number of survey-eligible patients the ICH facility served during the preceding 12-month period. If a facility’s patient volume is large enough, the number of patients sampled for that facility for each semiannual survey will be sufficient to yield a minimum of 200 completed surveys over the two semiannual surveys. If a facility does not serve enough survey- eligible patients over a given 12-month period to yield 200 completed surveys from the two semiannual surveys, a census of all survey-eligible patients will comprise the sample. Most of the ICH facilities need to survey all of their eligible patients at least once during the course of a calendar year and most patients are sampled twice within a given year. 
Facilities with 1–200 Unique Patients. A census of all ICH patients is conducted for facilities with fewer than 201 eligible ICH patients at each semiannual sampling wave. Thus, patients at these smaller ICH facilities are sampled twice in a given year. 
Facilities with 201–400 Patients. For dialysis centers that have between 201 and 400 eligible ICH patients at the first semiannual sampling period (the Spring Survey), a simple random sample of 200 patients is selected for that sampling period to obtain 100 completed responses. For the Fall Survey, the goal is to obtain an additional 100 completed interviews while attempting to minimize patient overlap of patients between the first and second semiannual waves of sampling. To achieve this goal, we first identify all eligible patients from that facility who were not selected for the Spring Survey. If the number of eligible patients not selected in the Spring Survey is equal to or exceeds 200 then we select a simple random sample of 200 from these patients for the Fall Survey. If there are 200 or fewer patients then all of these patients are selected for the Fall Survey. To obtain 200 completed surveys, we also select a simple random sample of the appropriate size from the patients who were selected in the Spring Survey, provided that they are still receiving treatment at that facility and still meet all of the survey eligibility requirements. 
	B.2 	Information Collection Procedures 
 
Three modes of survey administration are allowed during the national implementation of the ICH CAHPS Survey to give ICH facilities options for their preferred survey administration modes, based on their goals and resources. These three modes are described below: 
    • Mail-only Mode: ICH CAHPS Survey data collection for the mail-only mode consists of mailing a prenotification letter, explaining the purpose of the survey, and letting patients know that a hardcopy questionnaire will soon be sent to them via mail. A questionnaire package consisting of a cover letter, the ICH CAHPS questionnaire, and a pre-addressed, postage-paid return envelope is sent to all sample patients two weeks following the pre notification letter. A second mailing containing a questionnaire and cover letter is mailed to all sample patients who do not respond to the first mailing within four weeks after the first questionnaire package is mailed. Data collection ends ten weeks after the first questionnaire package is mailed. 
    • Telephone-only Mode: In this mode, all sample patients are first sent a pre notification letter letting them know that a professional interviewer working on the ICH CAHPS Survey will soon be contacting them via telephone. All sample patients are then contacted by professional telephone interviewers who are trained on ICH CAHPS survey administration procedures, including procedures for working with dialysis patients. Telephone interviewers are trained on the appropriate response to common questions and concerns that dialysis patients might have about survey participation, and are required to offer to administer the interview in different call- backs if the sample patient indicates that he or she cannot complete the interview in one call. A maximum of 10 telephone contact attempts per patient are attempted to complete the survey. Data collection ends ten weeks after the initial telephone contact begins. 
    • Mixed Mode: All sampled patients included in the mixed mode data collection sample receive a prenotification letter letting them know that we will soon be contacting them via mail. We then send an initial mailing of a questionnaire, cover letter, and postage-paid return envelope that patients included in the mail-only sample receive. Sample patients assigned to this mode who do not respond to the mail survey within four weeks after the questionnaire is mailed will be assigned to the telephone follow- up. Telephone interviewers make up to 10 attempts to complete the interview by phone with all mail survey non-respondents included in the mixed-mode sample. Data collection ends ten weeks after the first questionnaire package is mailed. 
Survey vendors who wish to become “approved” to conduct the ICH CAHPS Survey on behalf of ICH facilities must submit an application and complete the ICH CAHPS survey vendor training, which provides detailed guidance on the protocols and guidelines for all aspects of survey implementation, from sample selection to data collection and data submission. 
	B.3 	Methods to Maximize Response Rate 
Response rates range average 27.8% for mail-only surveys, 23.9% for phone-only surveys and 32.2% for mixed mode surveys, for an overall average response rate of 31.9%, as shown in Exhibit B.2. Every effort will be made to maximize patient response rates while retaining the voluntary nature of the ICH CAHPS Survey. 
 
Exhibit B.2. Response Rates for ICH CAHPS Semiannual Surveys 
 
 
 
Overall 
 
Survey Period 
 
# 
Sampled 
 
# Eligible 
# Completed 
Surveys 
 
RR 
 
 
2014 Fall 
 
329,493 
 
305,590 
 
113,935 
 
37.3% 
 
 
2015 Spring 
 
363,181 
 
337,316 
 
114,847 
 
34.0% 
 
 
2015 Fall 
 
356,721 
 
324,139 
 
103,808 
 
32.0% 
 
 
2016 Spring 
 
363,670 
 
339,092 
 
107,582 
 
31.7% 
 
 
2016 Fall 
 
347,879 
 
323,386 
 
100,184 
 
31.0% 
 
 
2017 Spring 
 
348,024 
 
323,185 
 
111,851 
 
34.6% 
 
 
2017 Fall 
 
351,700 
 
321,818 
 
105,120 
 
32.7% 
 
 
2018 Spring 
 
332,183 
 
307,078 
 
98,611 
 
32.1% 
 
 
2018 Fall 
 
344,827 
 
316,093 
 
102,757 
 
32.5% 
 
 
2019 Spring 
 
353,703 
 
325,690 
 
98,868 
 
30.4% 
 
 
2019 Fall 
 
356,127 
 
324,732 
 
96,255 
 
29.6% 
 
 
2020 Spring 
 
29,676 
 
26,827 
 
8,571 
 
31.9% 
 
 
2020 Fall 
 
350,949 
 
322,607 
 
96,805 
 
30.0% 
 
 
2021 Spring 
 
345,448 
 
316,014 
 
82,987 
 
26.3% 
 
 
All Survey Periods 
 
4,573,581 
 
4,213,567 
 
1,342,181 
 
31.9% 
 
After the sample file is downloaded, survey vendors must verify mailing addresses using a commercial address update service. Each prenotification letter envelope contains the CMS logo along with the survey vendor’s return address. Furthermore, each questionnaire mailing includes a cover letter containing the CMS logo and information about the survey, including sponsorship and objectives, a description of how survey results will be used, and the vendor’s toll-free telephone number that sampled patients can contact if they have questions or need additional information about the survey. Because some dialysis patients may be reluctant to participate because of fear of retribution from their dialysis centers, the mail survey materials contain assurances that the patients’ survey responses are kept private and cannot be linked to their names. In addition, CMS requires that the mail survey cover letters include the CMS logo and signature of a CMS representative. 
We require that all mail survey vendors use current best practices in the survey materials to enhance response rates. These best practices include using a simple font no smaller than 12 point size in the survey cover letters, allowing ample white space between questions in the questionnaire, avoiding a format that displays the questions as a matrix, using a unique subject identification number on the questionnaire rather than printing the sample member’s name, and displaying the OMB number and expiration date on the questionnaire.01 
For sample patients included in the mail data collection for the ICH CAHPS Survey, the second questionnaire mailing is expected to increase the response rate. The cover letter included in the second questionnaire package to mail survey nonrespondents contains a stronger appeal for the sample patient’s help on this survey, including indicating that the survey is an opportunity for them to provide input on the quality of dialysis care dialysis patients receive. To maximize response rates for the telephone-only mode and the telephone follow-up of the mixed- mode survey, we require that up to 10 attempts be made to reach each sample patient, with those attempts varying by day of the week and time of day. Telephone interviewers are trained on how to answer the questions that are most frequently asked by sample patients, and to address any concerns that they may have about participating in the survey. Because some dialysis patients may not feel well on the day that they receive dialysis treatments, telephone interviewers are instructed to offer to call back at a time that is better for the sample patient, and offer to conduct the telephone interview on two or more different calls. 
 
 
	B.4 	Tests of Procedures 
CMS used data from the mode experiment to assess the effects of data collection mode, patient characteristics, and nonresponse on survey results. We used the data from the mode experiment to develop models that will be used to statistically adjust survey results from the national survey to control for factors that are beyond the control of the ICH facilities. The following analyses were conducted on mode experiment data: 
    • Analyses of individual survey items to assess missing data and item distributions • 	Statistical analysis of patient mix effects and nonresponse patterns on survey results 
    • Hypothesis testing to detect differences in key variables between modes. 
 
	B.5 	Statistical Consultation and Independent Review 
This sampling and statistical plan was prepared by RTI International. The primary statistical design was provided by Gordon Brown and refined by Scott Scheffler. Amy Couzens now serves as the Sampling Task Leader at RTI and can be reached by telephone at 919-541- 5910or by email at [email protected].