Generic Supporting Statement

Application to Use Burden/Hours from Generic PRA Clearance:

Testing of Web Survey Design and Administration for CMS Experience of Care Surveys

(CMS-10694, OMB 0938-1370)

Generic Information Collection (GenIC) #2: HCAHPS Mode Experiment

Division of Consumer Assessment & Plan Performance (DCAPP)

Center for Medicare (CM)

Centers for Medicare & Medicaid Services (CMS)

1. Background

The HCAHPS (Hospital Consumer Assessment of Healthcare Providers and Systems) Survey is the first national, standardized, publicly reported survey of patients’ perspectives of their hospital care. Since 2008, HCAHPS has allowed valid comparisons to be made across hospitals locally, regionally and nationally.

HCAHPS currently offers four approved modes of survey administration: Mail Only, Telephone Only, Mixed Mode (mail with telephone follow-up of non-respondents), and Interactive Voice Response. The Mode experiment will allow CMS to test the effect of adding a web-based mode to the HCAHPS Survey and assess the impact of using a web-based mode on survey response rates and scores. The mode experiment will include some new and modified survey items.

In order to promote survey participation, improve response rates (in particular among younger patients), and keep current with common modes of survey administration CMS will test administration of the HCAHPS survey via web as part of a mode experiment. Under this mode experiment CMS will administer HCAHPS via web followed by either mail, telephone, or mail and telephone mixed modes in order to assess response rate, characteristics of patients who respond via web, and performance of HCAHPS measures. The results of the experiment will inform CMS decision making about future implementation of HCAHPS via web and survey mode adjustments necessary for web administration.

CMS has received feedback from hospitals, hospital associations, survey vendors, and other stakeholders requesting the option to administer the survey using a web mode as an alternative to the existing approved modes. The stakeholders’ suggestions are generally aimed at improving the survey’s response rate, obtaining a more representative set of survey respondents, and reducing the cost of survey administration.

CMS believes that the addition of a web-based survey in mixed mode formats has the potential to improve response rates, particularly among patients who have the lowest response rates to HCAHPS, such as younger patients. In 2019, while the overall response rate for HCAHPS was 22%, the response rate for patients ages 65 and older was above 30%, while that for patients 18-54 was below 15%. There is reason to believe that offering a web-based survey as the first mode in a mixed mode format could increase the participation of younger patients. In the hospital-based emergency department setting, using a survey similar in design and content to HCAHPS, field research showed that, (1) a higher survey response rate could be achieved by using a web-based survey as the initial mode followed by mail and telephone compared to mail and telephone survey arms without web,¹ and (2) characteristics (e.g., age) of telephone and web respondents were more representative of the sampled population than mail respondents.²

B. Description of Information Collection

Data from CMS Experience of Care Surveys are publicly reported and many survey results are linked to payment to health and drug plans and to providers and facilities in the effort to improve the quality of care. As such, these survey data come under close scrutiny by the public and the regulated community. The design, testing, and implementation of these surveys is held to the strictest of statistical survey methodologies and standards. This generic clearance request is to support methodological research designed to improve the quality and reduce the burden of the suite of CMS Experience of Care Surveys. This generic clearance request encompasses an array of research activities to support decisions about whether and how to add web administration protocols to a series of surveys conducted by CMS.

We are requesting clearance under the Testing of Web Survey Design and Administration for CMS Experience of Care Surveys generic clearance package that supports an array of research activities to support the decisions about whether and how to add the web administration to CMS patient experience of care surveys. This request focuses on the HCAHPS Survey of hospital inpatients.

The HCAHPS Survey is administered to a random sample of adult inpatients between 48 hours and six weeks after discharge. Patients admitted in the medical, surgical and maternity care service lines are eligible for the survey; HCAHPS is not restricted to Medicare patients. The proposed mode experiment design will apply all HCAHPS eligibility criteria, which are detailed in the HCAHPS Quality Assurance Guidelines, V15.0, pp. 57-64 (https://hcahpsonline.org/globalassets/hcahps/quality-assurance/2020_qag_v15.0.pdf).

CMS has recruited a group of hospitals with enough HCAHPS-eligible patients to enable them to contribute a specified number of patients each month for six months to the mode experiment without jeopardizing the hospitals’ participation in CMS programs that employ HCAHPS, such as the Hospital Inpatient Quality Reporting and Hospital Value-Based Purchasing programs. The 50 hospitals included in the mode experiment are broadly representative of all participating hospitals in terms of location, bed size, and HCAHPS performance.

The patients submitted each month for this mode experiment by each participating hospital will be randomly assigned in equal proportion to the six arms of the mode experiment: three existing modes (Mail Only, Telephone Only, Mixed Mode (mail with telephone follow-up)), and three experimental modes that include an initial web-based survey (E-mail-Mail, E-mail-Telephone, E-mail–Mail-Telephone. Across the experimental mixed modes CMS will vary the duration of the e-mail phase and the timing of the e-mail reminders. Patients will be randomly assigned to the arms irrespective of the availability of an e-mail address. We will not test an e-mail-only mode because of evidence that it yields both low response rate and unrepresentative respondents.^{3 4}

In the three web-based study arms invitations to the web survey will be sent by email to patients with email addresses. The email will be personalized to the discharged patient and will include a link to the web version of the survey, which can be completed on either a computer or a mobile device such as a smartphone or tablet. A reminder invitation will be sent approximately three days after the initial invitation. If the patient does not complete the web survey approximately one week after the first email invitation or does not have a valid email address to send an email, the secondary mode (mail or telephone) will be initiated. The table below summarizes the contact attempts by study arm.

Study Arm	Number of Email Invitations	Number of Survey Mailings	Number of Telephone Attempts
Web+mail	Up to 3	Up to 2	None
Web+telephone	Up to 3	None	Up to 5
Web+mail+telephone	Up to 2	1	Up to 5
Mail only	None	Up to 2	None
Telephone only	None	None	Up to 5
Mail+telephone	None	1	Up to 5

Across the six study arms, CMS plans collect information from 7,801 eligible patients discharged during approximately April through September 2021 from over 60 hospitals. Discharged patients will be randomized by discharge month within each hospital to one of the six study arms.

The proposed design is of sufficient scale to examine response rates and the effect of the mode on survey scores, and to provide precise adjustment estimates for survey items and composites on survey. Specifically, this design provides 80% power to detect (1) differences in response rate of 1.7-2.5% between any pair of the six mode protocols, (2) very small to small differences (Cohen’s d =0.11) in response patterns between and two pairs of the six mode protocols, and (3) very-small-to-small differences (Cohen’s d = 0.15-0.17) in response patterns by study arm (mode protocol within email address availability stratum).

The mode test survey will contain 43 questions: 25 existing HCAHPS items, plus 18 new survey items that assess content not currently included in the survey (e.g., care coordination within the hospital, communication with family or caregivers) or to update existing survey items (e.g., care transition, discharge experience).

C. Deviations from Generic Request

No deviations are requested.

D. Burden Hour Deduction (HCAHPS)

The total approved burden ceiling of the “Generic PRA Clearance: Testing of Web Survey Design and Administration for CMS Experience of Care Surveys (CMS-10694, OMB 0938-1370)” is 17,000 hours. For the HCAHPS mode experiment we are requesting a total deduction of 1,398 hours from the approved burden ceiling.

The HCAHPS mode experiment will be conducted among a maximum of 7,801 patients. All patients in the mode experiment will receive the mode experiment survey instrument, which contains 43 items. It is estimated that the survey will require an average administration time of 10.75 minutes, or 0.179 hours (0.25 minutes per item, or 4 items per minute). These burden estimates are based on CMS’s experience with the HCAHPS Survey. As shown in Table 1, based on these assumptions, the one-time total burden hours for survey participants are estimated to be 1,398 for participation in the mode experiment.

Table 1. Estimated Burden Hours for Respondents: HCAHPS Survey Mode Experiment

Survey Version	Number of Respondents	Number of Responses per Respondent	Hours per Response	Total Burden Hours
HCAHPS mode experiment survey instrument	7,801	1	0.179	1,398.0

Because not all sampled patients will complete the survey, this estimate reflects the maximum burden possible.

The total approved burden ceiling of the generic ICR is 17,000 hours. We are requesting a total deduction of 1,398 hours from the approved burden ceiling.

E. Timeline

The following table provides an overview of the HCAHPS mode experiment project schedule, including data collection and analysis.

Project Month

Tasks

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

OMB Approval

X

Conduct survey

X

X

X

X

X

X

Collect data

X

X

X

X

X

X

X

X

X

Data Analysis

X

X

X

X

X

X

X

X

X

X

Final Report

X

The following attachment is provided for this information collection: HCAHPS Survey.

1 Mathews M., Parast L., Tolpadi A., Elliott M.N., Flow-Delwiche E., Becker K. (2019). Methods for Improving Response Rates in an Emergency Department Setting - A Randomized Feasibility Study. Survey Practice, 12(1): 10.29115/SP-2019-0007.

2 Parast L., M. Mathews, M. Elliott, A. Tolpadi, E. Flow-Delwiche, W.G. Lehrman, D. Stark, K. Becker. (2019). Effects of Push-to-Web Mixed Mode Approaches on Survey Response Rates: Evidence from a Randomized Experiment in Emergency Departments. Survey Practice, 12(2). https://doi.org/10.29115/SP-2019-0008. Published online, 7-29-19: https://www.surveypractice.org/article/9772-effects-of-push-to-web-mixed-mode-approaches-on-survey-response-rates-evidence-from-a-randomized-experiment-in-emergency-departments.

3 Parast L., M. Mathews, M. Elliott, et al., 2019.

4 M.N. Elliott, J.A. Brown, W.G. Lehrman, M.K. Beckett, K. Hambarsoomian, L.A. Giordano and E. Goldstein. “A Randomized Experiment Investigating the Suitability of Speech-Enabled IVR and Web Modes for Publicly Reported Surveys of Patients’ Experience of Hospital Care.” Medical Care Research and Review, 70 (2): 165-184. 2013.

1