HomeHealthCAHPS_OMB_PartB 5-18-17

As noted in Section A in the 2014 PRA package, it is necessary to consider data collection and analysis for two components of the Home Health Care CAHPS® Survey. The first component involves conducting a mode experiment to develop adjusters for survey administration mode and patient mix. The second component is the national implementation, conducted by approved HHCAHPS survey vendor doing the survey on behalf of Medicare home health agencies (HHAs). Typically, there is a survey mode effect but none was found in the Mode Experiment. It could be due to the numerous “yes, no” questions. However, there were differences for patient characteristics, particularly does the patient live alone, does the patient self-report fair to poor health, does the patient report poor emotional health, the age of the patient, and reported language barriers. Every time we update our public reporting file we adjust it for the patient mix in that file, so the patient mix adjusters change with every file. We post the patient mix adjusters on our website, https://homehealthcahps.org and we have downloadable databases of HHCAHPS data.

The sampling plans for both the mode experiment and national implementation are described below. To maintain consistency between these two related phases of data collection, the same overall sampling methodology and data collection protocols were used in both data collections.

B.1.1 Sampling HHAs for the Mode Experiment

For the mode experiment, RTI used a two-stage sampling process, starting with selection of HHAs that have voluntarily agreed to participate, followed by selection of patients from the sampled volunteer HHAs. RTI used the most recent cost reports in the Medicare Healthcare Cost Report Information System (HCRIS) as a sample frame to select a target sample of HHAs for the mode experiment. The HCRIS contains cost reports from Medicare-certified HHAs. Because most HHAs are Medicare certified, the HCRIS includes the complete universe of Medicare-certified HHAs and essentially the universe of HHAs. A cost report is available for each Medicare provider number. There were about 9,500 HHAs in the sample frame at the time of the Mode Experiment.

The data in the HCRIS include counts of patients and visits for home health care patients whose care is reimbursed by Medicare, Medicaid, and private insurance. RTI project staff reviewed these data for their suitability as a sample frame. Generally, the data approximately matched various metrics of Medicare home health care concerning the number of patients, visits, and episodes for Medicare patients, although the episode measure was unavailable for non-Medicare patients. The list of HHAs, counts of patients, and counts of visits were used in structuring a target sample frame of all Medicare-certified HHAs. Based on RTI’s analysis of the cost reports in HCRIS files, approximately 30 HHAs contained potentially incorrect visit data. RTI delete these HHAs from the frame for the mode experiment.

There were limitations to the use of cost report data for this purpose. Cost report data may lag current operations by a year or more. This lag may exist because some HHAs have been more recently certified, have gone out of business, or have changed ownership or characteristics. After forming the target sample frame and selecting the sample, RTI project staff validated the information about the provider when they recruited HHAs to voluntarily participate in the mode experiment. The project staff used the home health patient assessments, the Outcome and Assessment Information Set (OASIS), to determine whether certain providers who have submitted cost reports were not currently billing. Inactive providers were identified and removed from the frame prior to sampling for HHAs. When the project staff discovered that a provider’s characteristics changed significantly, the sample frame sorting process, described below, allowed the project statistician to choose an adjacent substitute HHA from the frame.

Given the purposes of the mode experiment (to determine potential necessary adjustments in reporting of the national comparative data for survey mode and selected patient characteristics), we included a representative sample of HHAs as participants in the mode experiment. The methods used to select target HHAs for the mode experiment ensured representation of home health agencies with a number of different characteristics for a range of patient types. Project staff used combination of stratification by size (based on number of patient visits) and sorting by other HHA characteristics within strata to achieve the representative mix of HHA characteristics. The characteristics accounted for (in addition to size) were:

geographic location of the HHA, using a broad definition based on similarity in per capita use of home health care in contiguous states;
whether an HHA is freestanding or hospital-based;
whether an HHA is for-profit or not-for-profit; and
whether the HHA is in an urban or rural area, based on county designation.

Some HHAs did not agree to participate in the mode experiment, so the project staff drew a similar nearest neighbor substitute from the sorted frame.

The project staff stratified by size using visit counts by HHA. Based on the visit count, they created categories of size. After sorting on size, project staff created strata based on HHA size. For example, if four strata were used, the largest HHAs (representing one quarter of the visits) would be the first stratum and the smallest HHAs (also with one quarter of the visits) would be the fourth. This method produced increasing numbers of HHAs in each stratum, since fewer HHAs are large (first stratum) and more agencies are small (fourth stratum). Within strata the data were sorted on the other HHA characteristics listed above. To avoid sampling a very large number of small HHAs that contributed few patients to a sample, the strata were sampled disproportionately. With large numbers of agencies in each stratum, systematic sampling selected a representative sample across HHA types.

As an example of this sampling approach, using approximately 9,000 HHAs as the universe, the number of HHAs in each stratum might be 100 (the largest HHAs), 1,900, 2,500, and 4,500 (the smallest HHAs). One hundred HHAs could be selected by systematic sampling of 20, 30, 30, 20, HHAs respectively, from each of the 4 strata. In this example, 22% of the HHAs in the sample frame would be relatively large and 78% relatively small, but the larger HHAs would contribute a much larger proportion of the sample for the mode experiment—in this example, 50% of the sample would be in the two strata of larger HHAs (50 of the total 100 selected).

Within size strata, the agencies were sorted across the other characteristics so that the systematic sampling covered a range of the geographic and organizational characteristics mentioned above. As described below, the sampling rate was adjusted for each stratum so that a sufficient number of patients can be drawn for the experiment.

RTI recruited approximately 100 HHAs to participate in the mode experiment. The number of agencies was chosen as a compromise between the need to cover a variety of agency types across the dimensions listed above, the intensity of effort needed to recruit HHAs to participate in the mode experiment, and recognition of the additional burden to HHAs of participating voluntarily in this additional step toward national implementation. For the mode experiment to be valid statistically it was more important that the number of patients in the sample would be adequate than the number of participating HHAs. By adjusting the sampling rate within agencies the patient sample was achieved.

B.1.2 Sampling Patients for the Mode Experiment and the National Implementation

For both the mode experiment and the national implementation, HHAs assemble a census of their patients (both current and discharged) for the sampling window, defined as a calendar month. Every month, each HHA submits a file containing patient information for all patients to whom the HHA provided home care during the sampling month to its contracted survey vendor (for the national implementation) or to RTI (for the mode experiment). The mode experiment involved sampling and data collection for three months. The national survey was fielded on a rolling basis, and the results for each quarter merged with data from the three immediately preceding quarters and analyzed. The sample frame for the national implementation is assembled at the level of the CMS Certification Number (CCN), and the CCNs (Medicare home health agencies) comprise the units of comparison for HHCAHPS survey results reported on the Home Health Compare website every three months, in the months of January, April, July, and October.

The HHA’s sampling frame contains all the patient data needed for fielding the survey and data needed for analysis. The HHA’s survey vendor (in national implementation) reviews the frame and exclude any patients who are not eligible to participate in the Home Health Care CAHPS® Survey. Patients ineligible for the survey are those who

are receiving hospice or are discharged to hospice,
are deceased when the sample is drawn,
are under 18 years of age at any time during their stay,
did not have at least one skilled home health visit in the sample month and at least two home health care visits during a 60-day look-back period starting with the last day of the sampled month,
are maternity patients,
are “no publicity” patients,
are receiving only nonskilled (aide) care,
are state-regulated patients, or
are patients who were sampled in HHCAHPS during the last 5 months.

The requirement that a patient not be sampled more than twice a year is intended to reduce burden on individual patients and to increase the probability of response.

B.1.2a Mode Experiment Patient Sampling Specifics

For the mode experiment, the patient sample size required was computed on the basis of power to detect a difference in proportions of 0.05, from a proportion of 0.5, with 80% confidence and an alpha of 0.05. When regressions were run, predicting the proportion of patients choosing a particular answer to a survey item, the estimated coefficient of the variable indicating a particular mode is the increment in the model prediction that would arise from the reference mode in the equation. This is the difference in proportions that is targeted.

The sample size (total number of completed interviews) needed for each mode being tested in the mode experiment is about 1,570. With three data collection modes being tested, with equal sample size, the targeted number of completed surveys is 4,710. The following are estimates of response rates associated with surveys of similar patient populations and with different modes after adjustment for possible ineligibles found during the survey.

Data Collection Mode	Response Rate
Mail	30.0%
Phone	28.0%
Mixed	34.5%

Assuming an average response rate of about 30% based on other surveys, the fielded sample should be 15,700. However, given the nature of the HHA patient population, particularly the large proportion of aged Medicare or Medicaid patients (including many patients in a long recovery or requiring more maintenance health care), there are some concerns about achieving this response rate. We have therefore assumed in our sampling approach for a “safety margin” that allows the response to be as low as 20% while still achieving statistical power. This requires a sample of about 23,000. With a sample of this size we should be able to cover a wide variety of patients with random sampling.

This sample size can be achieved with 100 HHAs as the primary sampling units by adjusting the sampling rates on the larger and smaller HHAs recruited. The size of HHA client populations at any one time varies from fewer than 100 to many thousands. The particular mix of HHAs in different size strata may need to be adjusted depending on recruiting success but it will be easier to adjust the sampling rate. We will determine the actual stratification into size classes when the HCRIS data are analyzed, but a simple example, assuming only two size classes, shows the feasibility of this approach. Assume there are large agencies with 1,000 clients at a point in time and small agencies with 100. A target of 23,000 patients can be achieved with the following combinations of numbers and sampling rates:

50 large HHAs at 36% = 18,000 and 50 small at 100% = 5,000
25 large HHAs at 62% = 15,500 and 75 small at 100% = 7,500

RTI will aim for a distribution that is closer to the proportions of HHAs in the size categories but the more important patient target can be met with disproportionate distributions.

For the mode experiment, patients were selected using random sampling. Within each HHA, each sampled patient was assigned randomly to one of three data collection modes. The project staff did not assign an entire HHA sample to one mode to avoid correlation of mode with HHA characteristics.

B.1.2b National Implementation Sampling Specifics

For the national implementation of the Home Health Care CAHPS® Survey, each participating HHA sends to its contracted survey vendor each month a patient sample frame containing information about each patient who received home health care during the sample month, with sufficient information for the vendor to determine exclusions and with information needed for both fielding the survey and for patient-mix adjustment. The survey vendor removes from the sample frame patients who do not meet survey eligibility requirements and then draws a random sample of the remaining patients.

Survey vendors working under contract with HHAs are instructed to use a reliable program to generate random numbers for sampling. The Centers for Medicare & Medicaid Services (CMS) has continually recommended to survey vendors that they use the free program RATSTATS, available from the Department of Health and Human Services, Office of Inspector General website, or some other validated sample selection program such as SAS to select the sample. The sampling procedure recommended is simple random sampling, but disproportionate and proportional stratified random sampling may be allowed since some HHAs may want to analyze their own data and view survey results for individual branches. HHAs that deviate from simple random sampling (using disproportionate sampling) are required to request an exception and to obtain approval from CMS. An exception is permitted if the minimum sample is 10 per strata and the information needed to increase weights is reported to RTI.

Although the national implementation sampling is conducted on a monthly basis (with the survey initiated for each monthly sample within 3 weeks after the sample month ends), data from four quarters are accrued, aggregated, analyzed, and publicly reported on a quarterly basis, with the data from the most current quarter replacing data from the oldest of the four quarters, when the data is posted on Home Health Compare on www.medicare.gov. For four calendar quarters, a minimum of 300 completed surveys is the target for each participating HHA. If an HHA’s patient population is too small to yield 300 completed surveys, a census will be surveyed. The 300 completed surveys needed for analysis is derived from the formula for the precision of a proportion with the estimate at .5, the confidence interval of about +/- 0.05, and a confidence level of 95%. (Many agencies, with a substantial sampling fraction, can achieve a higher precision because of the finite population correction factor.)

In the national implementation of the Home Health Care CAHPS Survey, the number of patients needed for selection each month to yield a minimum of 300 completed surveys per year (25 per month) is determined by each HHA and its survey vendor. The mode of administration of the survey may be an important determining factor in response rates. Using the estimated response above, the estimated sample sizes for HHAs participating in the national implementation of the Home Health Care CAHPS Survey are the following:

Mode	Response Rate	Sample Size for 25 Responses/Month
Mail	30.0%	84
Phone	28.0%	90
Mixed	34.5%	73

Each agency survey vendor uses its experience on other surveys with home health patients and/or other similar populations, the data collection mode, and expected response rates as guides for calculating the monthly sample sizes that will be needed for the Home Health Care CAHPS Survey.

About 80% of HHAs in HHCAHPS use the Mail mode.

The sampling rate to achieve these sample sizes indicates that HHAs with monthly frame sizes of 90 or below should start with a sample equal to the sample frame. That is, all patients who meet the eligibility criteria will be included in the survey sample. For HHAs with larger sampling frames the sampling rate can be reduced, although it clearly will be higher than 50% until the frame exceeds about 180 eligible patients per month. Monthly sample size rates should be based on the number of patients who meet survey eligibility criteria in the frames after the first test month, since that month does not have any patients who are ineligible for the survey because they would be sampled during the first month of the test file.

B.2 Information Collection Procedures

Three modes of survey administration are allowed during the national implementation of the Home Health Care CAHPS Survey to give HHAs options in how they would like to administer the survey, based on their goals and resources. These three modes are described below:

Mail-only mode

Mailing of the questionnaire and cover letter to all sampled patients.
Second mailing of the questionnaire with a cover letter to sample patients who do not respond to the first mailing within 3 weeks after the first questionnaire package is mailed.

Telephone-only mode

A maximum of five telephone contact attempts per patient to complete the survey.

Mixed-mode

Mailing of the questionnaire and cover letter to all sample patients.
Telephone follow-up with all sample patients who do not respond to the questionnaire mailing. A maximum of five telephone contact attempts per patient will be made to complete the survey.

Data collection for each sampled patient must be initiated no later than 3 weeks (21 days) after the close of the sample month. We do allow HHAs to apply for a late fielding request up to the 14^th day in the following month. For example, if an HHA was late sending its April 2017 patient list to the vendor (between May 21st-26^th), then the HHA and the HHA’s vendor can apply for a late fielding request, which CMS usually accepts up to June 14^th (in this example). The fielding period is 42 days and there are not extensions on that time frame. Again, once data collection begins, it must be closed out within 6 weeks.

Survey vendors who wish to become “approved” to conduct the Home Health Care CAHPS Survey on behalf of HHAs complete the Home Health Care CAHPS survey vendor training, which will provide detailed guidance on the protocols and guidelines for all aspects of survey implementation, from sample selection to data collection and data submission. We post the list of approved HHCAHPS survey vendors on our website, https://homehealthcahps.org.

B.3 Methods to Maximize Response Rate

Every effort is made to maximize patient response rates, while retaining the voluntary nature of the Home Health Care CAHPS Survey. Each questionnaire mailing includes a cover letter explaining what the survey is about, who is conducting it and why, and the name and toll-free telephone number of a survey staff member that sampled patients can contact if they have questions or need additional information about the survey. For the mail-only mode of administration, our approved HHCAHPS survey vendors must use best practices in survey materials to enhance response rates. These best practices include using a simple font no smaller than 10 point size in the survey cover letters, allowing ample white space between questions in the questionnaire, avoiding a format that displays the questions as a matrix, using a unique subject identification number on the questionnaire rather than printing the sample member’s name, and displaying the OMB number and expiration date on the questionnaire. The second mailing for the mail only implementation is expected to increase the response rate, as is the telephone follow-up portion of the mixed-mode implementation.

B.4 Tests of Procedures

These analyses were done in the Mode Experiment:

Analyses of individual survey items will assess missing data and item distributions.
Hypothesis testing will detect differences in key variables between modes.
The analysis of individual items and the hypothesis testing will form the basis for constructing an adjustor to be used for telephone and mixed-mode surveys.

We did not find a mode effect for HHCAHPS. Data from the field test confirmed that self-reported overall health status, education, and age should be included in the patient mix models. The mode experiment also showed that whether the patient lived alone, did the patient have language barriers, and did the patient report poor emotional health, were also important case mix adjusters.

B.5 Statistical Consultation and Independent Review

This sampling and statistical plan was prepared by RTI International and reviewed by CMS. RTI continues to evaluate the data in many different analyses. Recently, RTI has evaluated the outliers in the HHCAHPS survey data results, and they have analyzed how the questions contribute to our third composite that concerns specific health issues. RTI is additionally evaluating HHAs that receive footnotes in our publicly reported data.

File Type	application/msword
File Title	The National Implementation of the Home Health Care CAHPS Survey
Subject	OMB Supporting Statement for Home Health Care CAHPS Survey
Author	Centers for Medicare & Medicaid Services
Last Modified By	Lori Teichman
File Modified	2017-05-18
File Created	2017-05-18