National Implementation of the In-Center Hemodialysis CAHPS Survey
CMS-10105, OCN 0938-0926
The Centers for Medicare & Medicaid Services (CMS) is requesting clearance from the Office of Management and Budget (OMB) to implement nationally the In-center Hemodialysis CAHPS (ICH CAHPS) Survey to measure patients’ experience of care with in-center hemodialysis (ICH) facilities under Contract Number HHSM-500-2012-00151G. Under this information collection request, CMS is also seeking review and approval for a mode experiment that will be conducted to determine the impact of using different data collection modes in the national implementation of the ICH CAHPS Survey.
In the national implementation of the ICH CAHPS Survey, Medicare-certified ICH facilities and their selected ICH CAHPS Survey vendor can choose to implement the ICH CAHPS Survey using one of three modes of data collection: mail only, phone only, and mixed mode (mail with telephone follow-up of mail survey nonrespondents). Because the data collection mode can have a substantial impact on survey results, CMS will conduct a mode experiment to determine whether patients rate the quality of care they receive from their ICH facility differently based on data collection mode. Data from the mode experiment also will determine whether patients with different demographic or health characteristics (that is, characteristics beyond the facility’s control) rate their care differently. Mode experiment data will determine which of those factors affect patients’ assessment of their care.
The ICH CAHPS mode experiment will be a one-time (cross-sectional) survey of currently dialyzing ICH patients at selected facilities who are 18 years old and older with at least three months of ICH at their current center. The goal of the sampling process for the mode experiment is to obtain approximately 1,570 respondents for each of the three modes. CMS will draw a stratified simple random sample from the population of ICH patients using the CROWNWeb database to construct the sample frame. The sampling frame will include the following information: name, address, telephone number, gender, race/ethnicity, total time on in-center hemodialysis, total time at current dialysis facility, diagnosis, dual Medicare-Medicaid eligibility, and age at which dialysis began.
One of the primary purposes of stratification is to ensure that certain subsets of the population are not overlooked when selecting the sample. More specifically, it is possible that there will be geographic variations in responses rates and patients’ assessment of their ICH care in different parts of the country. If, in the course of sampling, certain regions are excluded because of chance then we will have no ability to detect potential regional variation or may obtain biased results because of lack of sufficient coverage of all ICH patients in the country. Should there be regional heterogeneity or responses, this type of stratification can reduce the sampling variability, thereby increasing the power of all statistical tests and reducing the standard error of all statistics produced during analysis of mode experiment data.
The first stage of the sampling process for the mode experiment will involve creating strata based on geographic regions using combinations of the 50 states and U.S. territories in which Medicare-certified ICH facilities are located. We will determine the total number of ICH patients in each state and use this information to create the first-stage strata. In the second stage of the sampling process we will select a simple random sample of patients from each stratum with sample size proportional to the number of survey-eligible ICH patients in that stratum.
Based on the results from the ICH CAHPS field test that was conducted in 2005 and recent surveys of other patient populations, we are assuming that we will receive a 35% response rate from sample patients in the mail-only mode, 45% response rate from those in the telephone-only data collection mode, and a 50% response rate from those in the mixed mode. To that end, we will select a total of 11,115 ICH patients from the CROWNWeb database. After the sample is selected, we will randomly assign patients to the three modes using the inverse of the estimated response rates so that we have the desired sample size in each mode (see Exhibit B.1). The number of respondents shown in Exhibit B.1 for each mode will allow us to detect a 5 percentage point difference with 80% power with an alpha level of 0.05.
Exhibit B.1 Sample Sizes by Data Collection Mode
|
Mail Only |
Phone Only |
Mixed Mode |
Sample size |
4,486 |
3,489 |
3,140 |
Completed surveys: mail |
1,569 |
NA |
1,098 |
Completed surveys: phone |
NA |
1,570 |
472 |
Total completed surveys |
1,569 |
1,570 |
1,570 |
Response rate |
35% |
45% |
50% |
To determine whether patients rate the quality of care they receive from their ICH facility differently based on data collection mode and patient characteristics, CMS will use the data from the mode experiment to determine whether the method of administering the survey influences the results and whether patients with varying demographic or health characteristics affect their assessment of the dialysis care they receive. The information from the sampling frame will also determine which patients are less likely to respond to the survey. Nonresponse adjustments will become very important as we approach national implementation and public reporting. Models predicting the propensity to respond will be developed to adjust respondent weights.
Medicare-certified dialysis facilities that serve more than 30 survey-eligible sample members during a calendar year will be required to contract with a CMS-approved survey vendor to collect and submit ICH CAHPS Survey data on their behalf. All approved ICH CAHPS Survey vendors will be required to use standardized survey administration protocols and specifications provided by CMS. The national implementation of the ICH CAHPS survey will be conducted on a semiannual basis, with sampling and data collection activities conducted as shown in Exhibit B.2.
Exhibit B.2 Sampling Window for the ICH-CAHPS Semiannual Surveys
|
Spring survey |
Fall survey |
Sampling window (months in which patients received ICH care) |
January–April |
June–September |
Sample selected |
May |
October |
Data collection period |
June–August |
November–January |
The national implementation of the survey will be fielded on a rolling semiannual basis. The results for each semiannual survey will be merged with data from the immediately preceding semiannual survey for developing composite measures for public reporting. A primary issue will be obtaining sufficient sample size within a facility to produce confidence intervals for point estimates that are sufficiently narrow. Approximately 200 observations will be needed per year to produce a confidence interval that has a bound of +/−0.07. Approximately 81.4% of Medicare-certified ICH facilities serve 99 or fewer unique patients a year; 17.6% serve between 100 and 199 patients a year, and 1.0% serve more than 200 patients each year.1
For each semiannual wave, patients who received care during the sampling window and who meet survey eligibility criteria will either be chosen randomly or selected with certainty depending on the number of survey-eligible patients the ICH facility served during the preceding 12-month period. If a facility’s patient volume is large enough, the number of patients sampled for that facility for each semiannual survey will be sufficient to yield a minimum of 200 completed surveys over the two semiannual surveys. If a facility does not serve enough survey-eligible patients over a given 12-month period to yield 200 completed surveys from the two semiannual surveys, a census of all survey-eligible patients will comprise the sample. Depending on the data collection mode the ICH facility decides to use, most of the ICH facilities will need to survey all of their eligible patients at least once during the course of a calendar year and most patients will be sampled twice within a given year.
Facilities with 1–200 Unique Patients. A census of all ICH patients will be conducted for facilities with fewer than 201 eligible ICH patients at each semiannual sampling wave. Thus, patients at these smaller ICH facilities will be sampled twice in a given year.
Facilities with 201–400 Patients. For dialysis centers that have between 201 and 400 eligible ICH patients at the first semiannual sampling period (the Spring Survey), a simple random sample of 200 patients will be selected for that sampling period to obtain 100 completed responses. For the Fall Survey, the goal will be to obtain an additional 100 completed interviews while attempting to minimize patient overlap of patients between the first and second semiannual waves of sampling. To achieve this goal, we will first identify all eligible patients from that facility who were not selected for the Spring Survey. If the number of eligible patients not selected in the Spring Survey is equal to or exceeds 200 then we will select a simple random sample of 200 from these patients for the Fall Survey. If there are 200 or fewer patients then all of these patients will be selected for the Fall Survey. To obtain 200 completed surveys, we will also select a simple random sample of the appropriate size from the patients who were selected in the Spring Survey, provided that they are still receiving treatment at that facility and still meet all of the survey eligibility requirements.
Three modes of survey administration will be allowed during the national implementation of the ICH CAHPS Survey to give ICH facilities options for their preferred survey administration modes, based on their goals and resources. These three modes are described below:
Mail-only Mode: ICH CAHPS Survey data collection for the mail-only mode will consist of mailing a pre-notification letter, explaining the purpose of the survey, and letting patients know that a hardcopy questionnaire will soon be sent to them via mail. A questionnaire package consisting of a cover letter, the ICH CAHPS questionnaire, and a pre-addressed, postage-paid return envelope will be sent to all sample patients 1 week following the pre notification letter. A second mailing containing a questionnaire and cover letter will be mailed to all sample patients who do not respond to the first mailing within three weeks after the first questionnaire package is mailed. Data collection will end six weeks after the first questionnaire package is mailed.
Telephone-only Mode: In this mode, all sample patients will first be sent a pre-notification letter letting them know that a professional interviewer working on the ICH CAHPS Survey will soon be contacting them via telephone. All sample patients will then be contacted by professional telephone interviewers who will be trained on ICH CAHPS survey administration procedures, including procedures for working with dialysis patients. Telephone interviewers will be trained on the appropriate response to common questions and concerns that dialysis patients might have about survey participation, and will be required to offer to administer the interview in different call-backs if the sample patient indicates that he or she cannot complete the interview in one call. A maximum of 10 telephone contact attempts per patient will be attempted to complete the survey. Data collection will end six weeks after the initial telephone contact begins.
Mixed Mode: All sampled patients included in the mixed-mode data collection sample will receive a pre-notification letter letting them know that we will soon be contacting them via mail. We will then send an initial mailing of a questionnaire, cover letter, and postage-paid return envelope that patients included in the mail-only sample will receive. Sample patients assigned to this mode who do not respond to the mail survey within three weeks after the questionnaire is mailed will be assigned to the telephone follow-up. Telephone interviewers will make up to 10 attempts to complete the interview by phone with all mail survey nonrespondents included in the mixed-mode sample. Data collection will end six weeks after the first questionnaire package is mailed.
Survey vendors who wish to become “approved” to conduct the ICH CAHPS Survey on behalf of ICH facilities must complete the ICH CAHPS survey vendor training, which will provide detailed guidance on the protocols and guidelines for all aspects of survey implementation, from sample selection to data collection and data submission.
The same information collection procedures described above for the national implementation will also be used on the ICH CAHPS mode experiment.
As indicated previously, we expect that the response rate will vary based on data collection mode, with a 35% response rate from mail-only sample patients, 45% for telephone-only surveys, and 50% for mixed-mode data collection efforts, for an overall response rate of approximately 43%. Every effort will be made to maximize patient response rates while retaining the voluntary nature of the ICH CAHPS Survey. Each questionnaire mailing will include a cover letter containing information about the survey, including sponsorship and objectives, a description of how survey results will be used, and the name and toll-free telephone number of a survey staff member that sampled patients can contact if they have questions or need additional information about the survey. Because some dialysis patients may be reluctant to participate because of fear of retribution from their dialysis centers, the mail survey materials will contain assurances that the patients’ dialysis facility will not have access to their survey responses linked to their names or any other information that can identify the patients. In addition, CMS will require that the mail survey cover letters be printed on the survey vendor’s letterhead and signed by the survey vendor’s ICH CAHPS project manager.
We will require that all mail survey vendors use current best practices in the survey materials to enhance response rates. These best practices include using a simple font no smaller than 12-point size in the survey cover letters, allowing ample white space between questions in the questionnaire, avoiding a format that displays the questions as a matrix, using a unique subject identification number on the questionnaire rather than printing the sample member’s name, and displaying the OMB number and expiration date on the questionnaire (Dillman, 2009).
For sample patients included in the mail and mixed-mode data collection ICH CAHPS Survey, the second questionnaire mailing is expected to increase the response rate. The cover letter included in the second questionnaire package to mail survey nonrespondents will contain a stronger appeal for the sample patient’s help on this survey, including indicating that the survey is an opportunity for them to provide input on the quality of dialysis care dialysis patients receive. To maximize response rates for the telephone-only mode and the telephone follow-up of the mixed-mode survey, we will require that up to 10 attempts be made to reach each sample patient, with those attempts varying by day of the week and time of day. Telephone interviewers will be trained on how to answer the questions that are most frequently asked by sample patients, and to address any concerns that they may have about participating in the survey. Because some dialysis patients may not feel well on the day that they receive dialysis treatments, telephone interviewers will be instructed to offer to call back at a time that is better for the sample patient, and will offer to conduct the telephone interview on two or more different calls.
CMS will use data from the mode experiment to assess the effects, if any, of data collection mode, patient characteristics, and nonresponse on survey results. We will use the data from the mode experiment to develop models that will be used to statistically adjust survey results from the national survey to control for factors that are beyond the control of the ICH facilities. The following analyses will be conducted on mode experiment data:
Analyses of individual survey items to assess missing data and item distributions
Statistical analysis of patient mix effects and nonresponse patterns on survey results
Hypothesis testing to detect differences in key variables between modes
This sampling and statistical plan was prepared by RTI International. The primary statistical design was provided by Gordon Brown of RTI International. Dr. Brown can be reached by telephone at (919) 485-5647 or by e-mail at [email protected].
Dillman, Don A. (1999). Mail and Other Self-Administered Surveys in the 21st Century: The Beginning of a New Era. The Gallup Research Journal, 2(1): 121–140.
Dora, M., Zalai, M. D., & Novak, M. (2008). Depressive disorders in patients with chronic kidney disease. Primary Psychiatry, 15(1), 66–72.
Kurella Tamura, M., Wadley, V., Yaffe, K., McClure, L. A., Howard, G., Go, R., Allman, R. M., Warnock, D. G., & McClellan, W. (2008). Kidney function and cognitive impairment in US adults: The Reasons for Geographic and Racial Differences in Stroke (REGARDS) study. American Journal of Kidney Diseases, August 2008.
Medicare Payment Advisory Commission (MedPac). (2003, October). Report to the Congress: Modernizing the outpatient dialysis payment system. Retrieved from http://www.medpac.gov/documents/oct2003_Dialysis.pdf
U.S. Bureau of Labor Statistics (2010). National Compensation Survey: Occupational wages in the United States, July 2010. U.S. Department of Labor, Bureau of Labor Statistics. Available at http://www.bls.gov/ncs/ncswage2010.pdf. Based on average wages.
1 Based on data from the 2011 End Stage Renal Disease Facility Survey.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | The National Implementation of the Home Health Care CAHPS Survey |
| Subject | OMB Supporting Statement for Home Health Care CAHPS Survey |
| Author | Centers for Medicare & Medicaid Services |
| File Modified | 0000-00-00 |
| File Created | 2021-01-29 |