201502-0938-005_EDPEC_Supporting_Statement_B (2015-2-24)_2015-04-06 Comm emfd 20150429 CLEAN

Supporting Statement B for the Emergency Department Patient Experience of Care Survey Mode Experiments

Contract Number: HHSM-500-2014-00421G

April 29, 2015

Prepared for CMS

Elizabeth Flow-Delwiche, Project Officer

RAND Corporation

1776 Main Street

P.O. Box 2138

Santa Monica, CA 90407-2138

TABLE OF ContentS

B. Collection of Information Employing Statistical Methods 4

B1. Respondent Universe, Hospital Recruitment, and Respondent Selection 4

B2. Data Collection Procedures 9

B3. Response Rates and Non-Response 10

B4. Tests of Procedures or Methods 11

B5. Statistical and Data Collection Consultants 11

LITERATURE CITED 12

ATTACHMENTS 13

SUPPORTING STATEMENT

Emergency department patient experiences with care survey MODE EXPERIMENTS

B. Collection of Information Employing Statistical Methods

B1. Respondent Universe, Hospital Recruitment, and Respondent Selection

Data collection will occur in 2015.

Selecting and Recruiting Hospitals

A total of 50 hospitals will be recruited to participate in the HCAHPS mode experiment. 

For hospital recruitment, RAND will partner with Health Services Advisory Group (HSAG), the organization responsible for HCAHPS national implementation as well as hospital recruitment for the EDPEC field test. RAND will rely on a two-tier system with HSAG to recruit hospitals using RAND’s successful system of establishing randomized recruitment calling queues. The strategy ensures that selective non-participation does not distort the composition of the hospitals from an initially representative design.

We will sample hospitals, subject to minimum sample size constraints that ensure adequate power, by proportionately stratifying by select hospital characteristics. We will not recruit freestanding emergency departments (EDs). While freestanding EDs and urgent care centers have become more common in recent years, they tend to see a lower-acuity population with fewer ambulance transports and admissions, provide different services, and have faster patient throughput – all of which could affect patient experiences with care and therefore require different survey approaches (California HealthCare Foundation 2009). As a result, we will use a fully hospital-based sample. 

RAND will use the American Hospital Association database to identify the full universe of hospitals meeting basic inclusion criteria. Children’s hospitals and other specialty hospitals will be excluded. Remaining hospitals will be limited to hospitals that collect HCAHPS data and have their data reported on Hospital Compare in order to make sure hospitals meet minimum sample size requirements to ensure adequate power. Once the pool of hospitals meeting the baseline inclusion criteria for participation in the mode experiments are identified, the pool will be divided into two subsamples: one for the HCAHPS mode experiment (CMS 10542) and one for the EDPEC mode experiment. The smallest hospitals that would be unable to provide sufficient sample for the EDPEC experiment with admitted patients will be assigned to the HCAHPS mode experiment subsample. All remaining hospitals will be randomly assigned. After the subsamples have been identified, each respective subsample will be stratified into queues. Because queues are defined by factorial combinations of designated characteristics, we are limited to 2 or 3 characteristics of 2 or 3 categories each to inform our queues.

For the ED PEC mode experiments with inpatient and discharged patients, HSAG will recruit approximately 150 hospitals across the 12 queues of hospitals with 1000 or more discharges per quarter to secure 50 hospitals that will participate in the arms of the EDPEC mode experiments. Hospitals in this subsample will be stratified by 3 sizes of annual ED visits (medium, large, and extra-large) and four geographic regions (Northeast, South, Midwest, and West).  Exhibit 3 delineates the 12 queues.

Exhibit 3: EDPEC mode experiment hospital recruitment queues

ED Visits Annually	Northeast Region	South Region	Midwest Region	West Region
Medium
Large
Extra Large

We will try to ensure representation across urban/rural location, hospital trauma level, and staffing model (physician services outsourced to a contracted ED management group or not). We will be using a stratified random sample approach as we have successfully used on a number of other hospital recruitments for mode experiments. A simple random sample would on average ensure representation on these characteristics, given the hospital sample size. Our approach that uses stratified random sampling will guarantee better representation of all hospital characteristics than would occur by chance alone, and that would include these characteristics in particular, which though not explicitly used in our strata, are correlated with our strata characteristics (e.g. urban/rural, trauma level, etc.). It is thus unlikely that our sample will deviate substantially from the national distribution of these characteristics.

The selection of a distinct set of 50 hospitals for the EDPEC mode experiments will follow a parallel and coordinated recruitment strategy with the HCAHPS mode experiment, working closely with HSAG to ensure compliance with the recruitment and sampling strategy.

Employing the model used to recruit hospitals for participation prior EDPEC Field Test and Hospital CAHPS data collection, we will conduct initial outreach to hospitals to secure an initial agreement of participation. Once hospitals have initially agreed to participation, CMS’ contractor will follow-up to discuss details of data transmission requirements and secure fully executed business associate agreements/data use agreements.

We plan to sample patients from 50 hospitals, targeting 320 inpatient surveys and 48 completed discharged to community surveys in each hospital. In order to enable an examination of mode effects, proxy response, and comparison of two different supplemental survey versions for the inpatients, each hospital emergency room included in the sample must see both (a) at least 160 patients per month who are admitted to the hospital through the emergency room and (b) at least 24 patients per month who are discharged to the community. With approximately 13 percent of all emergency room visits resulting in an admission, this will eliminate the smallest one-third of hospitals in the U.S. from the potential sample. Smaller hospitals cannot be accommodated while maintaining the field period CMS requires, since adequate time is available to sample from only three months of emergency room visits in each hospital.

Inpatient EDPEC Mode Experiment (English Only)

Assuming that approximately half of admitted cases are admitted via the ED (Pines 2013), we will sample 640 eligible discharges from each of 50 representative hospitals that have at least 1000 total eligible discharges per quarter,¹ resulting in approximately 320 patients admitted via the ED in each hospital. This will leave at least 200 representative HCAHPS-eligible cases to return for that quarter’s official HCAHPS submission. Our design ensures that no patient will receive both HCAHPS and EDPEC surveys.

We will randomly assign the 16,000 patients admitted via the ED such that 4,480 (28%) will be assigned to HCAHPS alone, 5,760 (36%) will be assigned to the EDPEC for Admitted Patients: HCAHPS Add-on Version A instrument (Attachment 1) and 5,760 (36%) will be assigned to the EDPEC for Admitted Patients: HCAHPS Add-on Version B instrument (Attachment 2). Analysis of the EDPEC instruments as compared with the HCAHPS-alone arm will provide quantitative information about the extent to which the presence of ED-specific questions impacts HCAHPS response rates and patterns. Maintaining an HCAHPS-only arm provides a test of the feasibility of simultaneously administering HCAHPS and the EDPEC Survey in the same set of 50 hospitals, enabling a thorough test of the ability to coordinate the two surveys and to identify and mitigate threats to later coordination in national implementation.

In the HCAHPS-alone arm, we will randomize equally to the four modes, yielding between 280 and 448 completes per mode, applying response rates observed in HCAHPS nationally. Fewer cases are randomized to HCAHPS alone than the two EDPEC instruments in keeping with differences in expected response rates. The inclusion of the four modes in the HCAHPS-alone arm will allow unbiased comparison to the EDPEC HCAHPS Add-on instrument overall and within each mode.

Please note that while the burden estimate and analysis plan is included in this application for respondents randomized to the HCAHPS-alone survey version, CMS is not seeking approval for the HCAHPS survey instrument. The HCAHPS Survey has been approved under OMB Control # 0938-0981(7/2/2012) as part of the National Implementation of Hospital Consumer Assessment of Health Providers and Systems (HCAHPS) CMS-10102.

We will also test the effect of allowing proxy reports on responses patterns and response rate, and will examine whether allowing proxies has different effects by mode. Within both EDPEC instruments, we will randomly assign 3,840 (2/3) to a protocol that allows for proxy assistance and 1,920 (1/3) to a protocol that does not allow for proxy assistance, and then within each proxy arm to randomize in equal numbers to the four data collection modes. More cases are assigned to the arm allowing proxy assistance to achieve higher power of the experiment overall by increasing the response rate for each instrument. We have conservatively assumed a 2 percentage point smaller response rate in the proxy not-allowed arm compared to the proxy allowed arm. In total, we anticipate 988 completed surveys in the proxy allowed arm and 466 completed surveys in the proxy not-allowed arm for each instrument.

Table 1 details the instruments, modes, proxy status, and expected completes described above.

Table 1: Instruments and Modes for Mode Experiments

Patient Characteristic	Survey Instrument	Is Proxy Response Allowed?	Survey Mode	Number of respondents anticipated
Admitted to hospital following ED visit	HCAHPS alone*	Not applicable	Mail only	358
Admitted to hospital following ED visit	HCAHPS alone*	Not applicable	Telephone only	381
Admitted to hospital following ED visit	HCAHPS alone*	Not applicable	Mixed mode	448
Admitted to hospital following ED visit	HCAHPS alone*	Not applicable	Interactive voice response	280
Admitted to hospital following ED visit	HCAHPS Add-on Version A	Yes	Mail only	230
Admitted to hospital following ED visit	HCAHPS Add-on Version A	Yes	Telephone only	259
Admitted to hospital following ED visit	HCAHPS Add-on Version A	Yes	Mixed mode	288
Admitted to hospital following ED visit	HCAHPS Add-on Version A	Yes	Interactive voice response	211
Admitted to hospital following ED visit	HCAHPS Add-on Version A	No	Mail only	106
Admitted to hospital following ED visit	HCAHPS Add-on Version A	No	Telephone only	120
Admitted to hospital following ED visit	HCAHPS Add-on Version A	No	Mixed mode	144
Admitted to hospital following ED visit	HCAHPS Add-on Version A	No	Interactive voice response	96
Admitted to hospital following ED visit	HCAHPS Add-on Version B	Yes	Mail only	230
Admitted to hospital following ED visit	HCAHPS Add-on Version B	Yes	Telephone only	259
Admitted to hospital following ED visit	HCAHPS Add-on Version B	Yes	Mixed mode	288
Admitted to hospital following ED visit	HCAHPS Add-on Version B	Yes	Interactive voice response	211
Admitted to hospital following ED visit	HCAHPS Add-on Version B	No	Mail only	106
Admitted to hospital following ED visit	HCAHPS Add-on Version B	No	Telephone only	120
Admitted to hospital following ED visit	HCAHPS Add-on Version B	No	Mixed mode	144
Admitted to hospital following ED visit	HCAHPS Add-on Version B	No	Interactive voice response	96
Discharged to community	Discharged to Community	Not applicable	“Walk-away” paper instrument with call-in option	288
Discharged to community	Discharged to Community	Not applicable	Post-ED visit mixed mode	288

* This survey instrument was previously approved under (OMB# 0938-0981)

The design provides sufficient power to address the main goals of this experiment, including accurate estimation of mode effects on response propensity and response patterns in the EDPEC Survey among admitted ED patients; testing the impact of adding ED items to the HCAHPS survey; and assessment of the impact of allowing proxy reports on the responses patterns and response rate. This design provides power to detect differences of 1.5 to 4.9 percent in response rate by survey mode, instrument, and proxy arm. When comparing responses to CAHPS items by mode, instrument, or proxy, we will be able to detect very-small-to-small patient-level differences (Cohen’s d=0.104-0.231 SD); because of smaller hospital-level standard deviations, these translate into small-to-medium hospital effect sizes. A detailed power analysis is shown in Table 2.

Table 2. Power Analysis of 4b Sample Design*

Comparison	Detectable difference in response rate near 31%	Detectable difference in CAHPS measure (standard deviations)
Mode Effect
Any 2 modes of survey administration, pooling EDPEC instruments	3.5%	0.139 - 0.156
Any 2 modes of survey administration, within either EDPEC instrument	4.9%	0.197 - 0.221
Instrument
EDPEC HCAHPS Add-on Version A instrument vs. EDPEC HCAHPS Add-on Version B instrument	2.5%	0.104
HCAHPS alone vs. either EDPEC HCAHPS Add-on instrument	2.6%	0.104
EDPEC HCAHPS Add-on Version A instrument vs. EDPEC HCAHPS Add-on Version B instrument, within mode	4.9%	0.191 - 0.226
HCAHPS alone vs. either EDPEC HCAHPS Add-on instrument, within mode	5.3%	0.189 - 0.231
Proxy
Proxy allowed vs. proxy not allowed, pooling EDPEC instruments (excludes HCAHPS alone)	1.5%	0.111
Proxy allowed vs. proxy not allowed (excludes HCAHPS alone), within either EDPEC instrument	3.7%	0.157
* All power calculations are for 80% power with 2-sided test at a 0.05 significance level. Estimates shown here (a) assume that we will observe sizeable differences in response rates by mode in the EDPEC instruments, as has been seen in HCAHPS, with lower response rates by mail and IVR and (b) incorporate publicly reported information that the EDPEC field test response rates for these instruments overall was approximately 30-31% to obtain a realistic estimate of the number of completes in each arm.

Discharged-to-Community EDPEC Mode Experiment (English only)

To examine survey mode effects for EDPEC discharged-to-community patients and investigate novel approaches to increasing the response rate, the design will include sampling 48 eligible discharged-to-community patients from each hospital and randomize them in approximately equal numbers to one of two modes using the EDPEC Discharged to Community instrument (Attachment 3). The first mode will be a post-ED mixed mode, similar to the mixed mode in the prior EDPEC field test. The second mode will use the same instrument administered as a “walk-away” paper instrument with a call-in option. The walk-away instrument will be administered by ED staff to patients as they are being discharged from the ED, with instructions to return the completed survey by mail. Hospitals will use separate time periods for the two mode arms (e.g., 1 week with 24 walk-away surveys and the next week with 24 post-ED surveys). Half of hospitals would administer one mode first and the others would administer the other mode first. Small hospitals can be accommodated within this design, expanding the block of time during which each survey is administered.

Our approach will result in 1,200 patients assigned to each mode and 288 completes per mode (conservatively assuming a 24 percent response, slightly higher than that observed in the EDPEC field test).

Our design provides sufficient power to compare response rates and response patterns by mode among discharged-to-community patients. This design will allow us to detect differences of 4.9 percent in response rate when comparing the 2 survey modes and to detect small differences (Cohen’s d=0.233) when comparing responses to CAHPS items by mode.

Determining Patient Eligibility

Patient eligibility criteria have been determined in consultation with CMS and are largely based on Hospital CAHPS eligibility criteria. In general, all adult patients are eligible for inclusion in the sampling universe, with the exception of the following ineligible groups:

• Patients under the age of 18

• Patients with a primary mental health or substance use diagnosis

• Patients who were discharged to hospice care, nursing homes, and skilled nursing facilities

• Patients who were transferred to another hospital

• Patients who died in the ED or who were admitted to the hospital from the ED and died during the inpatient stay

• Patients who request that they not be contacted (those who sign “no publicity” requests while hospitalize or otherwise directly request not to be contacted)

• Court/Law enforcement patients (i.e., prisoners)

• Patients with a foreign (Non-US or US Territory address) home address

• Patients who are excluded because of state regulations that place further restrictions on which patients may be contacted after discharge

• Patients who left without being seen and did not receive a billing code

Identification of patients for exclusion will be based on hospital administrative data.

B2. Data Collection Procedures

EDPEC Survey Inpatient Experiment

There will be four modes of data collection, with sampled individuals randomized to the four modes:

1. Mail only: We will mail an initial letter, survey, and a business reply envelope. See Attachment 4 for copies of the initial letter of the survey. All non-responders will be sent a second mailing three weeks after the initial mailing.

2. Telephone only: We will make five attempts to reach each sampled individual over a six week maximum period using computer assisted telephone interviewing (CATI). See Attachment 5 for the phone script for these calls. Calls will be made only between the hours of 9:00 am and 9:00 pm respondent local time (unless a respondent specifically requests a callback outside of this range).

3. Mail with telephone follow up: Individuals will receive an initial mailing mailed simultaneous to the mail only group and containing an initial letter, survey, and business reply envelope. For sampled individuals randomized to this mode who have not responded by mail by a cutoff date three weeks after the first survey mailing, we will begin attempts to complete surveys by phone using CATI. As with the telephone only mode, we will make five attempts to reach each sampled individual. These attempts will be made over a period of three weeks such that the entire field period is no longer than six weeks.

4. Interactive Voice Response - IVR cases will be introduced to the IVR system by a live operator, and operators will be available to provide support and complete a survey when a patient does not wish to continue with IVR. IVR cases will receive up to 5 call attempts over more than one week.

Procedures for all modes parallel the mail, telephone, mixed mode, and IVR procedures recommended for Hospital CAHPS data collection.

EDPEC Survey Discharged-to-Community Experiment

There will be two modes of data collection, with sampled individuals randomized to the modes as described above in the Respondent Selection section above.

1. Mail with telephone follow up: Following the same procedure as that used for the admitted mixed mode patients, individuals will receive an initial mailing containing an initial letter (See Attachment 6), survey, and business reply envelope. For sampled individuals randomized to this mode who have not responded by mail by a cutoff date three weeks after the first survey mailing, we will begin attempts to complete surveys by phone using CATI (Attachment 5). We will make five attempts to reach each sampled individual. These attempts will be made over a period of three weeks such that the entire field period is no longer than six weeks.

2. In-ED distribution (“walk-away”) with call-in option: At the time of discharge, eligible patients will be given a survey packet containing a letter (See Attachment 7), survey, and business reply envelope. The letter will ask the patient to return the survey via mail (or locked drop box) in the provided envelope. Additionally, the letter will provide a call in option where patients may call in to complete the survey with a live telephone interviewer. Sites will be asked to track distribution of survey materials.

B3. Response Rates and Non-Response

In the HCAHPS-alone arm of the EDPEC Survey Inpatient experiment, we will randomize equally to the four modes, and we anticipate response rates observed in HCAHPS nationally ranging from 25 percent to 40 percent depending on mode. For admitted patients receiving the HCAHPS Add-on instruments, on average across arms and modes we anticipate a response rate of 30 percent based on recent experience with similar surveys administered in the ED CAHPS field test. We will employ multiple mail and phone contacts to minimize non-response.

Within the discharged-to-community patient population, we anticipate a response rate of at least 24 percent based on recent experience with similar surveys administered in the EDPEC field test in 2014. We hope the new walk-away mode we are testing will increase response propensity and overall response rates for this patient population.

We will also plan for survey unit and item non response analyses to evaluate response propensity across modes, instruments, discharge status, and proxy status. We will compute these statistics overall, and separately by mode of administration (mail; telephone; mixed; IVR), Admitted Add-on Version A vs. Admitted Add-on Version B, and discharged to community in-ED distribution vs. mixed mode administration, discharged to community vs admitted patients, and proxy allowed vs. not-allowed.

B4. Tests of Procedures or Methods

This data collection effort includes

1. A test of mode of data collection (admitted patients): mail-only; telephone-only; mail with telephone follow up, and IVR. This mode experiment is described above. Procedures for all modes parallel mail, telephone, and IVR procedures recommended for Hospital CAHPS data collection.

2. A test of two different versions of EDPEC instruments where emergency room survey items are administered as a supplement to Hospital CAHPS for those patients admitted to the hospital.

3. A test of HCAHPS only vs. HCAHPS Add-on survey responses to evaluate the impact of the addition of ED-specific items to the HCAHPS survey responses.

4. A test of the impact of allowing proxy reports on the responses patterns and response rate.

5. A feasibility test of an in-ED survey distribution (walk-away) survey option and a test of mode of data collection (discharged to community patients): mail with telephone follow up, and in-ED distribution. This mode experiment is described above.

B5. Statistical and Data Collection Consultants

The survey, sampling approach, and data collection procedures were designed by the RAND Corporation under the leadership of:

Robin Weinick, Ph.D.

RAND Corporation

1200 South Hayes Street

Arlington, VA 22202-5050

Kirsten Becker, MS

RAND Corporation

1776 Main Street

Santa Monica, CA 90407

LITERATURE CITED

Pines JM, Mutter RL, Zocchi MS. (2013). “Variation in Emergency Department Admission Rates Across the United States.” Medical Care Research and Review (published online 6 January 2013, http://mcr.sagepub.com/content/early/2013/01/03/1077558712470565).

ATTACHMENTS

• Attachment 1- HCAHPS

• Attachment 2- EDPEC for Admitted Patients: HCAHPS Add-on Version A

• Attachment 3 -EDPEC for Admitted Patients: HCAHPS Add-on Version B

• Attachment 4 -EDPEC Discharged to Community

• Attachment 5- HCAHPS Cover Letter

• Attachment 6 - Initial Letter for EDPEC admitted to hospital patients

• Attachment 7 – EDPEC Telephone script – all versions

• Attachment 8 - Initial Letter for EDPEC discharged to community (mixed mode)

• Attachment 9 - Initial Letter for EDPEC discharged to community (on-site distribution)

1 Thirty-one percent of all HCAHPS hospitals have fewer than 900 sampled HCAHPS visits per year, ensuring that a large proportion of U.S. hospitals will meet the eligible discharge criteria.