Supporting Statement for the National Implementation of the Hospital CAHPS Survey
Supporting Statement – Part B:
COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
C 1.0 Respondent Universe and Sampling Methods
National Implementation
HCAHPS is designed to collect data on care from the patient’s perspective for general acute care hospitals. Pediatric, psychiatric, and other specialty hospitals are not included (additional/different aspects of care need to be examined for these specialty settings).
Within general acute care hospitals, HCAHPS scores are designed to reflect the care received by patients of all payers (not just Medicare patients). Specifically, this includes the population of non-psychiatric and non-pediatric patients who had an overnight stay in the hospital and were alive when discharged.
Pediatric patients (under age 18 at admission) and psychiatric patients are excluded because the current HCAHPS instrument is not designed to address the behavioral health issues pertinent to psychiatric patients or the situation of pediatric patients and their families. Patients who did not have an overnight stay are excluded because we don’t want to include patients who had very limited inpatient interaction in the hospital (e.g., patients who were admitted for a short period for purely observational purposes; patients getting only outpatient care are not included in HCAHPS). Patients who died in the hospital are excluded because HCAHPS is designed to capture information from the perspective of the patient, not the family. For logistical reasons, we also exclude several other categories of patients from eligibility: those who were prisoners, who were discharged to hospice care, who had a foreign home address, or who at admission had requested the hospital not to survey them (“no publicity” patients). Some states have further restrictions on patients who may be contacted. Hospitals/vendors should exclude other patients as required by law or regulation in the state in which they operate.
The monthly sampling frame that defines this population for a given hospital includes all discharges between the first and last days of the month, with the exclusions noted above.
To keep sampling as simple as possible, the hospital/vendor draw a simple random sample each month from the sampling frame of relevant discharges, as noted above. Sampling can be done at one time after the end of the month or continuously throughout the month as long as a simple random sample is generated for the month (the hospital/vendor can chose what works best with their current survey activities for internal improvement).
As noted above, monthly data are cumulated to produce a rolling 12-month data file for each hospital that will be used to produce hospital-level scores for the HCAHPS measure. A target for statistical precision of these scores is based on a reliability criterion. Higher reliability means higher “signal to noise” in the data. The target is that the reliability for the global ratings and most composites be 0.8 or more. Based on this, it is necessary for each hospital to obtain 300 completed HCAHPS questionnaires over a 12 month period.
In order to calculate the number of discharges that need to be sampled each month to reach this target, it is necessary to take into account the proportion of sampled patients expected to complete the survey (not all sampled patients who are contacted to complete the survey actually complete it; the target is 300 completes). This is a function of the proportion of sampled patients who turn out to be ineligible for the survey and the survey response rate among eligible respondents. The calculation of the sample size needed each month can be summarized as follows:
Step 1 – Identify the number of completes needed over 12 months
C = number of completes needed = 300
Step 2 – Estimate the proportion of sampled patients expected to complete the survey
Let:
I = expected proportion of ineligible sampled patients
R = expected survey response rate among eligible respondents
P = proportion of sampled patients expected to complete the survey
= (1 – I) x R
Step 3 – Calculate the number of discharges to sample
N12 = sample size needed over 12 months = C / P
N1 = Sample size needed each month = N12 / 12
Some small hospitals will not be able to reach the target of 300 completes in a given 12-month period. In such cases, the hospital should sample all discharges and attempt to obtain as many completes as possible.
Response rate
A great deal of public input regarding response rate was received. We seriously considered all of this input; as a result we’ve chosen an administration protocol to achieve at least on average a 40% response rate. CMS is encouraging hospitals/vendors to strive for higher response rates. As part of the reporting displays for HCAHPS, we publicly report response rates.
The raw response rate is the total number of completed surveys divided by the total number of respondents sampled. We consider a questionnaire to be complete if half or more of the items applicable to everyone (i.e., excluding items that apply to only a subset of hospitalized patients) are completed. In addition, the denominator of the raw response rate should be adjusted for any patients who were included in the sample but were ineligible (e.g. were deceased during the data collection period). Vendors should make every effort to obtain complete surveys from all eligible respondents. Bad contact information (out-of-date addresses or phone numbers) and other reasons (respondent out of town / on vacation) are eligible cases that should not be removed from the denominator.
Flexibility in the mode of administering the survey is permitted. The hospitals/vendors may use any of the following modes: telephone only, mail only, a mixed methodology of mail with telephone follow-up, or active interactive voice response (IVR). All modes of administration require following a standardized protocol. These four survey modes were chosen to achieve, on average, at least a 40 percent response rate. CMS conducted a large-scale mode experiment to detect for the presence of, then develop adjustments for, survey mode effects, as described earlier. Before publicly reporting HCAHPS results, CMS will adjust the hospital-level scores for differences across hospitals in terms of the mode of administration. The algorithm for the patient-mix adjustment can be found in Appendix B.
Modes of Survey Administration
For the mail only option, the hospital/vendor is required to send the HCAHPS questionnaire alone or combined with hospital-specific questions with a cover letter between 48 hours and 6 weeks following discharge. CMS provides sample cover letters to hospitals/vendors in their training program for HCAHPS; all survey materials are also provided in the current version of the HCAHPS Quality Assurance Guidelines, http://www.hcahpsonline.org/qaguidelines.aspx. The hospitals/vendors may tailor their letters, but the letters must contain information about the purpose of the survey, and state that participation in the survey is voluntary and will not affect their patients’ health care benefits.
The hospital/vendor is required to send a second questionnaire with a reminder/thank you letter to those not responding after the first mailing. Data collection is closed out for a particular respondent 6 weeks following the mailing of the first questionnaire.
For the telephone only option, the hospital/vendor is required to begin data collection between 48 hours and 6 weeks following discharge. The hospital/vendor must attempt to contact respondents up to 5 times unless the respondent explicitly refuses to complete the survey. These attempts must be made on different days of the week and different times of the day, in different weeks to ensure that as many respondents are reached as feasible. Data collection is closed out for a particular respondent 6 weeks following the first telephone attempt.
For the HCAHPS portion of the survey, CMS provides scripts to follow for the telephone interviewing. The interviewers conducting the survey must be trained before beginning interviewing. In their training program for HCAHPS, CMS provides guidance on how to train interviewers to conduct HCAHPS. The training program ensures that interviewers are reading questions as worded, interviewers are using non-directive probes, interviewers are maintaining a neutral and professional relationship with the respondent, and interviewers are recording only the answers that the respondents themselves choose.
A third option is a combination of mail and telephone. In this mixed mode of administration, there is one wave of mailing (cover letter and questionnaire) and up to five telephone call-back attempts for non-respondents. The first survey must be sent out between 48 hours and six weeks following discharge. Telephone follow-up occurs for non-respondents to the questionnaire mailing. The telephone attempts must be made on different days of the week and different times of the day, in different weeks to ensure that as many respondents are reached as feasible. Telephone interviewing ends 6 weeks after the first survey mailing. Similar to the telephone only mode, CMS provides telephone scripts for the hospitals/vendors to follow.
For active IVR, hospitals/vendors must initiate data collection by phone between 48 hours to 6 weeks following discharge. A live interviewer first asks the respondent if she/he is willing to complete the survey using the IVR system. Through the IVR system the respondent completes the survey using their touch-tone keypad on their phone. The hospital/vendor is required to provide an option for the respondent to opt out of the system and return to a live interviewer.
Similar to the telephone mode, the hospital/vendor must call each respondent up to 5 times unless the respondent refuses to complete the survey. These attempts must be made on different days of the week and different times of the day and in different weeks to ensure that as many respondents are reached as feasible. Data collection is closed out for a particular respondent 6 weeks following the first telephone attempt.
Ways to avoid problems and increase response rates have been integrated into the current data collection protocol; among these are:
To avoid undeliverable mail, the lag time between the date of discharge and the mailing of the questionnaire has been shortened.
The most specific and locally used hospital name is used to increase patient recognition in the cover letter.
Response rates will be published on the website displays.
4. We require two waves of mailing of the survey (mail only protocol) to boost response rates.
Many hospitals currently do a one-wave mailing.
5. We require 5 call-back attempts for the telephone only protocol, the mixed mode protocol, and the IVR protocol.
Most of the questions on the survey instrument are modifications of existing survey questions from hospital satisfaction surveys conducted by private survey vendors or modifications of current CAHPS questions. Some questions have been newly developed by CAHPS grantees that have had many years of experience developing these types of questions. The HCAHPS instrument and data collection protocol have been modified to include input from consumers, hospitals and vendors.
CMS carried out a three state pilot test with adult medical, surgical, and obstetric patients at 132 hospitals. Data collection took place between June 2003 and October 2003 with a sample size of 49,812 inpatient events. In the pilot, sampled patients were sent an advance letter, which was then followed about one week later by another mailing that contained the questionnaire. After another 10 days, a postcard was mailed. Approximately four weeks after the mailing of the postcard the non-responding patients from core hospitals were then followed up by telephone; a maximum of five follow-up phone attempts was used to complete the interview with core responders. Four weeks following the first mailing of the postcard the non-core patients were followed-up by a second mailing of the questionnaire. The final response rate was 35% for the non-core sample that used mail only administration. There was an almost 11 percentage point increase in the response rate due to the second mailing of the questionnaire. The final response rate was 45.6% for the core sample that included following up by telephone. The telephone follow-up accounted for an almost 18 percentage point increase in the response rate.
Connecticut state legislation requires that all licensed hospitals in the state be compared on patient satisfaction measures. In order to meet this mandate, Connecticut administered the 66-item version of the HCAHPS survey to patients from all 30 acute care hospitals in Connecticut. The data collection protocol included the same patient service areas covered in the three-state pilot areas (medical, surgical, and ob/gyn) and administered in two languages, English or Spanish. Data collection strategies included two survey mailings without a reminder/thank you postcard or letter. Replication of patient-level psychometric analysis of the HCAHPS instrument using the 3-state pilot sample (Arizona, Maryland, and New York) and the Connecticut sample was conducted.
The revised HCAHPS measure is compiled into the following seven composites: communication with nurses (n = 3), communication with doctors (n=3), communication about medicine (n = 2), nursing services (n = 2), discharge information (n = 2), pain management (n = 2), and cleanliness and quiet of the physical environment (n = 2). Within both samples, the reliability of the seven composites was estimated using the internal consistency method (Cronbach’s alpha coefficient). The construct validity of the composites was evaluated with regard to their relationships to an overall rating of the hospital (Hospital Rating) and whether the patient would recommend the hospital to others (Hospital Recommendation). The results of these analyses indicate that the revised HCAHPS measure performed similarly in the CT and three-state pilot data sets. See Appendix C for details.
Note; Subsequent to the above analysis we decomposed the “cleanliness and quiet of the physical environment” composite into two individual items: “Cleanliness of the hospital environment,” and “Quietness of the hospital environment.”
Ongoing statistical consultation is provided by:
Marc N. Elliot, Ph.D.
RAND Corporation
1776 Main Street
Santa Monica, CA 90401-3208
Tel: 310-393-0411
| File Type | application/msword |
| File Title | Justification of the Hospital CAHPS Survey |
| Author | CMS |
| Last Modified By | CMS |
| File Modified | 2010-10-07 |
| File Created | 2010-10-07 |