HHCAHPSSupportingStatementB

The Home Health Care CAHPS® Survey

Mode Experiment and Survey Analysis Part B

Collection of Information Employing Statistical Methods

i

TABLE OF CONTENTS

Section Page

2. Collection of Information Employing Statistical Methods 1

   1. Potential Respondent Universe and Sample Selection Method 1

   2. Information Collection Procedures 3

   3. Methods to Maximize Response Rate 3

   4. Tests of Procedures 4

   5. Statistical Consultation and Independent Review 6

2. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

This information collection request seeks OMB approval to conduct a mode experiment to determine the feasibility of adding Web as a new data collection mode for the Home Health Care Consumer Assessment of Healthcare Providers and Systems (HHCAHPS^®) Survey. This collection request has been extracted from the currently approved OMB Umbrella Generic Clearance (OMB Control Number: 0938-1370).

The sampling plan for the mode experiment is described below.

1. Potential Respondent Universe and Sample Selection Method

The HHCAHPS respondent universe is patients 18 years old and older who received at least one skilled visit in the sample month and two skilled visits over the look-back period of 2 months, from a Medicare-certified home health agency (HHA) and who are not discharged to hospice.

We will use a two-stage sample selection method where stage 1 is selection of 100 HHAs and stage 2 is selection of patients within each agency. A total of 23,150 home health patients will be sampled for the mode experiment. These will be divided into four modes: mail only, telephone only, mail with telephone follow-up, and Web with mail follow-up of nonrespondents to the Web survey. We anticipate completing 6,280 surveys across all study arms, or 1,570 surveys per mode.

RTI International, CMS’s federal contractor for the proposed mode experiment, will carefully balance the mode experiment sample to mirror the national distribution of HHAs on key agency characteristics and ensure that the sample is distributed over different regions. The three key agency characteristics we will track during recruitment to ensure that they reflect the universe of HHAs participating in the national HHCAHPS Survey are agency ownership, agency type, and location (urban/rural). Additionally, we will ensure that agency size and geographic location mirror the distribution of agencies that participate in the national implementation as much as possible.

• Agency Ownership: For-profit, Nonprofit

• Agency Type: Institutional (i.e., hospital or skilled nursing facility-based), Other

• Agency Location: Urban, Rural

• Agency Size: Large, Medium, Small

• Geographic Location: Northeast, Midwest, South, West, and Puerto Rico

We will select 23,150 patients from the 100 selected HHAs, which should yield 6,280 interviews based on our estimates of expected response rates. Since we propose to use leftover eligible sample obtained from each HHA’s current HHCAHPS Survey vendor, the sample we receive will contain all the patient data needed for both fielding the survey and analyzing the data. The HHA’s survey vendor will have reviewed the initial sample and will exclude any patients who are not eligible to participate in the HHCAHPS Survey. Patients ineligible for the survey are those who:

• are receiving hospice or are discharged to hospice,

• are deceased when the sample is drawn,

• are under 18 years of age at the end of the sample month,

• did not have at least one skilled home health visit in the sample month and at least two home health care visits during a 2-month look-back period covering the sample month and the prior month,

• are maternity patients only,

• are “no publicity” patients, (i.e., Patients who requested that the HHA not release their name and contact information to anyone other than agency personnel),

• are receiving only nonskilled (aide) care, or

• are state-regulated patients.

Mode Experiment Patient Sampling Specifics

After HHAs are recruited, we will work with their authorized HHCAHPS Survey vendor(s) to obtain unused sample for each month during the proposed 3-month data collection period. We will use existing HHCAHPS eligibility criteria and construct a sampling frame of all eligible patients for each participating agency.

Stratification. No explicit stratification will be used, but to decrease the likelihood of a random sample ending up by chance with a significantly different demographic distribution than the frame, age group and gender will be used as sorting variables.

Experimental Design and Sample Sizes. Our goal is to complete 1,570 interviews for each mode, which will allow us to have at least 80% power to detect a 5% difference for a binary survey outcome at the 0.05 significance level. We expect approximately a 27% response rate for the mail-only mode, 25% for the phone-only mode, 32% for the mail with telephone follow-up mode, and 26% for the Web with mail follow-up mode (based on the current HHCAHPS national implementation response rates and anticipated participation rates for web administration based on the 2019 OAS CAHPS mode experiment). We will need samples of 5,810 for mail, 6,280 for phone, 4,910 for mail with phone follow-up, and 6,150 for Web with mail to achieve 1,570 completed interviews for each mode. We will select a random sample of approximately 7,717 patients per month for each of the 3 months in the mode experiment, using a systematic random sampling method. The systematic sample is a leapfrog type sample where we take every n^th patient on the list. We will sort the eligible patient list within HHA by age group and gender, as noted above, and then sample systematically within HHA to achieve a representative sample at the agency level. We will then randomly assign sampled patients within each HHA to each of the four experimental modes using the inverse of the estimated response rates. The approximate sample sizes, expected number of completed interviews, and expected response rates are summarized by mode in Exhibit 1.

Exhibit 1. Approximate sample sizes, expected number of completed interviews, and expected response rates by mode

Characteristic	Mail only	Phone only	Mail with Phone	Web with Mail
Sample size	5,810	6,280	4,910	6,150
Completed surveys: mail	1,570	NA	982	1,144
Completed surveys: phone	NA	1,570	589	NA
Completed surveys: web	NA	NA	NA	431
Total completed surveys	1,570	1,570	1,570	1,570
Response rate	27%	25%	32%	26%

2. Information Collection Procedures

CMS proposes to test four modes of survey administration. Three of these modes are currently approved for the national implementation of the HHCAHPS Survey and are shown first in the list below. The fourth, which is Web with mail follow-up, is the proposed new mode. The fielding period for each mode will be 42 days, for three consecutive sample months. Each of these modes is described below:

• Mail-only mode

  • Mailing of the questionnaire and cover letter to all sampled patients.

  • Second mailing of the questionnaire with a cover letter to sample patients who do not respond to the first mailing within 3 weeks after the first questionnaire package is mailed.

• Telephone-only mode

  • A maximum of five telephone contact attempts per sampled patient to complete the survey.

• Mail with telephone follow-up mode

  • Mailing of the questionnaire and cover letter to all sampled patients.

  • Telephone follow-up with all sampled patients who do not respond to the questionnaire mailing by the third week. A maximum of five telephone contact attempts per patient will be made to complete the survey.

• Web with mail follow-up mode

  • Mailing and emailing Web survey invitation to all sampled patients.

  • After 1 week, send a second email invitation to all sampled patients with email addresses who do not respond to the first invitation.

  • At the start of the third week, send a hardcopy mail questionnaire with cover letter to all sampled patients who have not responded to the Web survey invitations.

  • At the start of the fifth week, send a final email reminder to all sampled patients with email addresses who have not responded to either the Web survey invitations or the mail questionnaire.

1. Methods to Maximize Response Rate

Every effort will be made to maximize patient response rates while retaining the voluntary nature of the revised HHCAHPS Survey. Each questionnaire mailing will include a cover letter explaining what the survey is about, who is conducting it and why, and the name and toll-free telephone number of a survey staff member who patients can contact if they have questions or need additional information about the survey.

For the mail-only mode of administration, we will use best practices in designing survey materials to enhance response rates. These include using a simple font no smaller than 12-point size in the survey cover letters, allowing ample white space between questions in the questionnaire, avoiding a format that displays the questions as a matrix, using a unique subject identification number on the questionnaire rather than printing the sample member’s name, and displaying the OMB number and expiration date on the questionnaire. The second mailing for the mail-only implementation is expected to increase the response rate, as is the telephone follow-up portion (or the mail survey portion) of the mixed-mode implementation.

For the telephone-only mode and follow-up of mail survey nonrespondents for mixed mode, we will make up to five attempts to reach each sample patient, with those attempts varying by day of the week and time of day. Telephone interviewers will be trained on how to answer questions that are most frequently asked by sample patients and to address any concerns that they may have about participating in the survey.

For the Web mode, the instrument will be designed to be optimized for both desktop/laptop computers and mobile devices. The Web survey design will be Section 508- compliant. Up to three email invitations will be sent to sample members with email addresses, to maximize response rates without burdening sample members with survey requests. Sample members without email addresses will receive an initial letter and the follow-up mail survey.

Additionally, we will conduct analyses to assess nonresponse bias using the following two approaches.

Compare Distributions of Variables from All Selected Cases and Respondents. For the person-level variables available for both respondents and non-respondents, such as age, gender, survey language, and Activities of Daily Living information, we will test whether the distributions are statistically different. A significantly different distribution between all selected patients and respondents may be an indication of nonresponse bias if there is a meaningful/substantial difference.

Conduct Statistical Analyses on Nonresponse Bias. We will conduct a logistic regression analysis to model survey response propensity using all administrative variables feasible. The logistic regression analysis identifies the independent variables that significantly affected the binary dependent variable that represented survey response or nonresponse status. If the variable identified was not one we recommended for inclusion in the patient-mix adjustment we will determine a statistical course of action to adjust for it to completely account for nonresponse bias.

4. Tests of Procedures

For CMS to assess the results of this mode experiment and ensure objective comparisons between HHAs, it must be able to identify and account for significant sources of variation outside the control of the agencies. Known sources of variation include variability in patient-mix and response propensity across patients and HHAs, one of which is data collection mode. The following analyses will be conducted using the mode experiment data: psychometric analysis, mode and patient-mix analyses, and nonresponse analysis. These analyses are described in detail below.

Analyze Data to Assess Psychometric Properties

We will evaluate the psychometric properties of the survey items and composites at the patient and agency levels. We will also finalize and report on the item and composite scores derived from the survey data, based on the inter-unit reliability and validity of the measures.

We will compute survey item frequencies and percentages and conduct outlier analyses to identify “ceiling” or “floor” effects, insufficient variance, severe bimodality, or high amounts of missing data. We will fit confirmatory factor models to determine whether the items cluster into composites as expected. We will then conduct item response theory analyses to assess the performance of the individual items and response options. We will also check for consistency in

item performance by patient subgroup (e.g., gender, race/ethnicity, age group) within composites.

At the respondent level, we will compute Cronbach’s alphas to assess internal consistency reliability; at the agency level, we will assess composite reliability using intraclass correlations. We will assess discriminant validity by computing correlations of each item with scores for its own composite relative to other composites. We will assess construct validity by examining correlations of each composite with the two current HHCAHPS Survey global items of rating of care and recommending the agency to family or friends and potentially with other survey measures.

Conduct Mode and Patient-mix Analyses

We will use descriptive statistics, correlation analysis, and multivariate regression analysis, conducted in the following order:

• Define dependent variable. For global rating variables, we will use the “top box” approach to code them as binary variables. For the Overall Rating variable, we will code as follows: 1 if responses are 9 or 10 and 0 if responses are 0–8. For the Willingness to Recommend variable, we will code as follows: 1 if responses are “Definitely yes” and 0 for any other valid response. For composite measures, the score for all composite measures will be calculated as the proportion of times the patient responded in the most positive response category for HHCAHPS Survey questions that asked about related topics or domains of care included in that composite. For individual rating questions, the most positive response category will be used.

• Propose independent variable candidate for regression analysis. Independent variables are those hypothesized to affect the outcome measures and can thus be used as patient-mix adjusters for HHAs. The independent variables will include mode of survey administration (mail only, telephone only, mail with telephone follow-up, and Web with mail follow-up), self-reported characteristics (e.g., health status, mental health status, education, language spoken at home), and administrative data (e.g., age, gender, ICD-10 diagnosis codes).

Conduct Descriptive Analysis. We will produce descriptive statistics for all dependent and independent variables to check the mean value and percent of missing values for each variable, the sufficiency of sample sizes in each response category for the categorical variables, and the amount of variation for continuous variables. Descriptive statistics also provide a more detailed understanding of the dependent and independent variables to aid in interpreting the results of the multivariate regression analyses.

Conduct Correlation Analysis Among Independent Variables. We will conduct correlation analysis on all candidate-independent variables. Highly correlated independent variables can cause problems for estimating regression models when two or more correlated variables are included in the models. This analysis will include calculating both Pearson correlation coefficients and variance inflation factor statistics.

Develop Multivariate Regression Models for Each Dependent Variable. We will develop multivariate regression analysis models to assess the degree to which the HHCAHPS outcome measures are affected by the mode and patient-mix characteristics. If the effects of the independent variables are of substantial size and statistically significant, we can calculate

adjustments for the ratings of HHAs for those independent variables that will affect their scores on the HHCAHPS outcome measures. The coefficient estimates from the multivariate regression models quantify the change in the dependent variables related to the individual characteristics represented by the independent variables. A separate model will be fit for each dependent variable.

We will use an ordinary least squares model to fit the data. We will set mail-only mode as the reference mode. In the regression model for each outcome, the coefficients of the mode dummy variables are the mode effects. In the 2009 HHCAHPS Survey mode experiment (reference 0938-1066), we found that mode did not appreciably provide a consistent, sizeable effect for use as an adjustor. We will assess potential mode effects anew, controlling for patient characteristics, to see if results change from the first mode experiment, particularly because we are including a new web mode and a revised, shorter, survey. If an adjustment for mode is needed, then we will implement it sequentially, after we adjust for patient-mix.

Based on the results and findings from the analyses outlined above, CMS will make a decision on what patient-mix adjustors should be included in the model for national implementation with this survey instrument, if CMS elects to propose this revised survey for national implementation, and whether nonresponse bias needs to be addressed further after the patient-mix adjustment.

5. Statistical Consultation and Independent Review

This sampling and statistical plan was prepared by RTI International and reviewed by CMS. RTI’s primary statistical point of contact is Mr. Harper Gordek, MPH, RTI International, Research Triangle Park, NC (919) 541-1231.