201502-0938-011_CMS-10542 - HCAHPS OMB Supporting Statement B 20150224_C 20150420 (2)

Supporting Statement B

for HCAHPS Mode Experiment

Contract Number: HHSM-500-2014-00421G February 24, 2015

Prepared for CMS

Elizabeth Flow-Delwiche, Project Officer

RAND Corporation

^{1776 Main Street}

P.O. Box 2138

^{Santa Monica, CA 90407-2138}

TABLE OF CONTENTS

SUPPORTING STATEMENT ..................................................................................................... 4

B. Collection of Information Employing Statistical Methods ..................................................... 4

B1. Respondent Universe and Respondent Selection ........................................................... 4

B2. Data Collection Procedures ........................................................................................ 5

B3. Response Rates and Non-Response ............................................................................. 6

B4. Tests of Procedures or Methods .................................................................................. 6

B5. Statistical and Data Collection Consultants .................................................................. 7

SUPPORTING STATEMENT

HCAHPS MODE EXPERIMENT

B. Collection of Information Employing Statistical Methods

B1. Respondent Universe, Hospital Recruitment, and Respondent Selection

A total of 50 hospitals will be recruited to participate in the HCAHPS mode experiment. 

For hospital recruitment, RAND will partner with Health Services Advisory Group (HSAG), the organization responsible for HCAHPS national implementation as well as hospital recruitment for the EDPECS field test. RAND and HSAG have a long history of working together and will ensure that current HCAHPS implementation and hospital reporting are not affected by the mode experiments. RAND will rely on a two-tier system with HSAG to recruit hospitals using RAND’s successful system of establishing randomized recruitment calling queues. The strategy ensures that selective non-participation does not distort the composition of the hospitals from an initially representative design.

RAND will use the American Hospital Association database to identify the full universe of hospitals meeting basic inclusion criteria. Children’s hospitals and other specialty hospitals will be excluded. Remaining hospitals will be limited to hospitals that collect HCAHPS data and have their data reported on Hospital Compare in order to make sure hospitals meet minimum sample size requirements to ensure adequate power. Once the pool of hospitals meeting the baseline inclusion criteria for participation in the mode experiments are identified, the pool will be divided into two subsamples: one for the HCAHPS mode experiment and one for the EDPEC mode experiments (CMS 10543). The smallest hospitals that would be unable to provide sufficient sample for the EDPEC experiment with admitted patients will be assigned to the HCAHPS mode experiment subsample. All remaining hospitals will be randomly assigned. After the subsamples have been identified, each respective subsample will be stratified into queues. Because queues are defined by factorial combinations of designated characteristics, we are limited to 2 or 3 characteristics of 2 or 3 categories each to inform our queues.

A total of 50 hospitals of varied bed size and geographic region will be recruited to

participate in the HCAHPS mode experiment. To secure 50 participating hospitals for the HCAHPS mode experiment, HSAG will recruit approximately 100 hospitals from 12 queues. Each queue will have different targets, with an average of 8-9 per queue recruited.  RAND proposes the following hospital characteristics for stratification: hospital bed size (<200 beds/200+ beds), HCAHPS HVBP score (below median/above median), and geographic region (Northeast/Southeast/Midwest and West). Exhibit 3 below delineates the queues:

Exhibit 3. HCAHPS mode experiment hospital recruitment queues

Hospital bed Size/HCAHPS HBVP score	Northeast Region	South Region	Midwest/West Region
<200 beds, below median
<200 beds, above median
200+ beds, below median
200+ beds, above median

The mode experiment will use 24,000 HCAHPS-eligible adult hospital discharges over a 3-month period. The specific three- month period of discharges to be used will depend upon the timing of completion of OMB review. We will follow the HCAHPS procedures, which prohibit the start of data collection any sooner than 48 hours post-discharge and no later than 42 days post- discharge. The pool of eligible discharges from each hospital will be randomized within hospital to survey version and mode of administration. Exhibit 4 details how the sample of 24,000 discharges will be randomized to survey version and mode of administration.

_{Exhibit 4. Total sample size and sample assigned to each mode of administration}

Survey Version		Total Sample Size	Mail Only Sample	Telephone Only Sample	Mixed Mode Sample	Touch Tone IVR Sample
HCAHPS1		12,000	3,000	3,000	3,000	3,000
HCAHPS+9		3,000	0	1,500	0	1,500
HCAHPS+16		3,000	0	1,500	0	1,500
HCAHPS+27		4,500	1,500	0	3,000	0
HCAHPS+44		1,500	1,500	0	0	0
	Totals	24,000	6,000	6,000	6,000	6,000

Based on our familiarity with HCAHPS and the response rates observed for each of the four HCAHPS data collection modes we anticipate response rates of 32% for Mail only mode, 34% for Telephone only mode, 45% for Mixed Mode (Mail-Telephone), and 25% for Touch Tone IVR. Exhibit 5 details the anticipated number of completed surveys by survey version and mode of administration.

¹Please note that while the burden estimate and analysis plan is included in this application for respondents randomized to the HCAHPS-alone survey version, CMS is not seeking approval for the HCAHPS survey instrument. The HCAHPS Survey has been approved under OMB Control # 0938-0981(7/2/2012) as part of the National Implementation of Hospital Consumer Assessment of Health Providers and Systems (HCAHPS) CMS-10102.

Exhibit 5. Total completes and number of completes by mode of administration

Survey Version		Total Completes	Mail Only Completes	Telephone Only Completes	Mixed Mode Completes	Touch Tone IVR Completes
HCAHPS*		4,080	960	1,020	1,350	750
HCAHPS+9		885	0	510	0	375
HCAHPS+16		885	0	510	0	375
HCAHPS+27		1,830	480	0	1,350	0
HCAHPS+44		480	480	0	0	0
	Totals	8,160	1,920	2,040	2,700	1,500

* This survey instrument was previously approved under (OMB# 0938-0981)

The sample design provides sufficient power² to address the main goals of this experiment. These include (a) accurate estimation of mode effects on response propensity and response patterns, and (b) accurate estimation of the effects of supplemental items on response propensity and response patterns. This design provides power to detect differences of 2.3 to 4.8 percent in response rate by survey modes or by number of supplemental items. Similarly, when comparing responses to CAHPS items by mode and by supplemental item instrument arms (pooled across modes and within each mode), we will be able to detect very-small-to-small patient-level differences (Cohen’s d=0.062-0.204 SD); because of smaller hospital-level standard deviations, these translate into small-to-medium effect sizes at the hospital level, which is the relevant metric when a hospital may choose one survey mode for all of its patients. If one were not able to detect such small effects at the person level, these differences could translate to large

differences at the hospital level in terms of rankings and have negative consequences; this consideration is applied to the design of the mode experiment.

In addition, we have powered the experiments to be consistent with previous HCAHPS mode experiments that form the current basis for adjusting publicly reported scores for CMS’ Hospital Value-Based Purchasing program.

B2. Data Collection Procedures

We replicate the HCAHPS survey administration procedures followed by hospitals and their approved HCAHPS vendors by following the procedures that are publicly available at hcahpsonline.org, including:

Mail only administration. The HCAHPS mail protocol provides for a maximum of two mail contacts. An initial survey and cover letter will be mailed no sooner than 48 hours post-discharge and no later than 42 days post-discharge. Twenty-one days after the initial survey is mailed, the second or non-response survey is mailed. Data collection is closed

_{42 days after the initial mailing.}

² 80% power with 2-sided test at alpha=0.05.

Telephone only administration. The HCAHPS telephone only protocol provides for the conduct of the survey by a live interviewer using computer-assisted telephone interview (CATI) technology, and a maximum of five survey attempts. The first survey attempt may occur no sooner than 48 hours post-discharge and no later than 42 days post- discharge. Data collection is closed 42 days after the first call attempt mailing.

Mixed-mode administration (Mail-Telephone). HCAHPS marries mail and telephone procedures for conduct of the mixed mode protocol. Mail is the initial mode of administration and telephone is the secondary or non-response mode. An initial survey and cover letter will be mailed no sooner than 48 hours post-discharge and no later than

42 days post-discharge. Twenty-one days after the initial survey is mailed, the case is eligible for telephone interview via CATI. The CATI portion of the protocol is limited to

a maximum of five survey attempts. Data collection is closed 42 days after the initial mailing.

Touch Tone IVR (TT-IVR) administration. The HCAHPS TT- IVR protocol provides for outbound calling by a live interviewer, who administers the introductory script and transfers the sampled individual to a TT-IVR system. The survey participant listens to an audio recording of the survey questions and responses and uses the touch-tone keypad of his or her telephone to respond to the questions. Participants can exit the system or request return to a live interviewer at any time. As with telephone only administration, this protocol the first survey attempt may occur no sooner than 48 hours post-discharge and no later than 42 days post-discharge, and the number of call attempts is capped at

_{five. Data collection is closed 42 days after the initial mailing.}

B3. Response Rates and Non-Response

Exhibit 6 summarizes information from Exhibits 4 and 5 to provide the total sample size, the total estimated number of respondents and the response rate by mode for each version of the survey. Overall, we anticipate that across all survey versions and modes of administration we will achieve 8,160 completed surveys from the sample of 24,000 discharges (an overall response rate of 34.0%).

_{Exhibit 6. Sample size and estimated response rates}

Survey Version		Total Sample Size	Total Completes	Mail Only Response Rate	Telephone Only Response Rate	Mixed Mode Response Rate	Touch Tone IVR Response Rate
HCAHPS*		12,000	4,080	32%	34%	45%	25%
HCAHPS+9		3,000	885	N/A	34%	N/A	25%
HCAHPS+16		3,000	885	N/A	34%	N/A	25%
HCAHPS+27		4,5040	1,830	32%	N/A	45%	N/A
HCAHPS+44		1,500	480	32%	N/A	N/A	N/A
	Totals	24,000	8,160	32%	34%	45%	25%

* This survey instrument was previously approved under (OMB# 0938-0981)

Response rates used in power calculations discussed in section B1 are based on recently observed HCAHPS mode experiments. The design of this experiment adheres to the design principles of the original HCAHPS mode experiments and will produce highly reliable, precise mode effect estimates that are generalizable to the discharged inpatient population and which will have standard errors of 0.02-0.07 patient-level standard deviations.

B4. Tests of Procedures or Methods

No tests of procedures or methods other than the effect of survey length on unit-level response and patterns of response to “About You” survey items will be undertaken as part of this data collection.

B5. Statistical and Data Collection Consultants

The survey, sampling approach, and data collection procedures were designed by the

RAND Corporation under the leadership of:

Robin Weinick, Ph.D. Kirsten Becker, M.S. RAND Corporation RAND Corporation

1200 South Hayes Street 1776 Main Street

Arlington, VA 22202-5050 Santa Monica, CA 90407

Key input to the statistical and/or methodological aspects of the design was received from the following individuals:

Marc Elliott, Senior Statistician

Julie Brown, Senior Survey Director