CMS-10500 - Supporting Statement B [rev 05-04-2015 by OSORA PRA]

Supporting Statement B

Consumer Assessment of Healthcare Providers and Systems

Outpatient and Ambulatory Surgery (OAS CAHPS) Survey

CMS-10500, OCN 0938-1240

B. Collection of Information Employing Statistical Methods

The Centers for Medicare & Medicaid Services (CMS) is requesting clearance from the Office of Management and Budget (OMB) to conduct the Outpatient/Ambulatory Surgery Patient Experience of Care Survey (O/ASPECS). The purpose of O/ASPECS is to measure patients’ experience of care with outpatient and ambulatory surgery centers in the United States. As with most large-scale data collections, CMS plans to test the survey prior to national implementation. This information collection request seeks OMB approval for both the testing and the national implementation. The testing component involves conducting a mode experiment with the goal of piloting procedures and developing statistical adjustments for survey administration mode and patient mix. It will be sponsored by CMS under Contract Number HHSM-500-2014-00426G. The national implementation will be voluntarily sponsored by outpatient surgery facilities. Each participating facility will contract with an independent CMS- approved survey vendor to conduct the survey.

The sampling plans for the mode experiment and national implementation are described below.

B.1 Potential Respondent Universe and Sample Selection Method

The O/ASPECS respondent universe is patients 18 years old and older who received an outpatient surgery or a procedure in a HOPD or ASC and who are not discharged to hospice. For the mode experiment, we will use a two-stage sample selection method where stage 1 is selection of 100 facilities (50 HOPDs and 50 ASCs) and stage 2 is selection of patients within each facility. A total of 13,737 outpatient surgery patients will be sampled for the mode experiment. These will be divided into three modes: mail only, telephone only, and mail with telephone follow-up of non-respondents to the mail survey (“mixed mode”). We anticipate completing 4,710 surveys or 1,570 surveys per mode.

For the national implementation, sample selection is single-stage. ASCs and HOPDs that choose to participate will work directly with a survey vendor of their choice. The vendors will be responsible for selecting adequate patient sample needed to complete 300 complete surveys for each outpatient surgery facility annually.

B.1.1 Sampling HOPDs and ASCs for the Mode Experiment

For the mode experiment, RTI will use a two-stage sample design in which the outpatient surgery facilities are selected first, and patients are then selected from these outpatient surgery facilities. Our target is to recruit 100 outpatient surgery facilities, comprising 50 HOPDs and 50 ASCs. The facility sampling frame will be all eligible outpatient surgery facilities obtained from CMS’s POS file. RTI will carefully balance the outpatient surgery facility sample to represent key facility characteristics and ensure that the outpatient surgery facility sample is also distributed over different regions. The four key facility characteristics are:

• Facility Specialty: Single, Multiple

• Facility Size¹: Large, Medium, Small

• Facility Location: Urban, Rural

• Facility Ownership: Hospital, Management Company, Health System/Managed Care, Physician, Government

The 100 selected facilities will yield 13,737 sampled patients

B.1.2 Sampling Patients for the Mode Experiment and the National Implementation

For both the mode experiment and the national implementation, outpatient surgery facilities will assemble a census of their patients who received a surgery during the previous month. Each facility will submit a file containing patient information for all eligible patients to RTI (for mode experiment) or to its contracted survey vendor (for national implementation). The mode experiment is a one-time survey which will involve sampling and data collection for the patients in the reference period. The national implementation will be sampled monthly. To be eligible to be selected for the O/ASPECS mode experiment or national implementation sample, patients must meet the following eligibility requirements:

• have had an outpatient surgery or procedure from the facility in the prior calendar month;

• the surgery or procedure is included in the surgical codes which are being studied by O/ASPECS (see B.1.2a);

• were at least 18 years old when they received their outpatient surgery or procedure;

• were not included in the sample for pre-specified period; and

• were not discharged to hospice or to a hospital for an inpatient stay.

B.1.2a Mode Experiment Patient Sampling Specifics

For the mode experiment, the patient sample size was calculated on the basis of power to detect a 0.05 difference in the proportion estimates when the proportion estimate is 0.5, with 80% confidence and the 0.05 significance level. The sample size, the total number of completed interviews needed for each mode being tested in the mode experiment is 1,570. With three data collection modes being tested, the targeted number of completed surveys is 4,710. Exhibit B‑1 shows the response rates and the number of patients need to be selected for each mode.

Exhibit B-1. Proposed Sample Size, Expected Number of Completed Interviews, and Expected Response Rate

Mode	Sample Size	Number of Respondents	Response Rate (%)
Mail Only	4,906	1,570	32
Telephone Only	4,906	1,570	32
Mail + Telephone Mixed Mode	3,925	1,570	40
Total	13,737	4,710	34.3

After facilities are recruited, RTI will work with each participating facility to obtain a list of patients receiving a surgery during the 3-month reference period. RTI will work with CMS to set up eligibility criteria and construct a sampling frame of all eligible patients for each participating facility. All eligible patients in the patient sampling frame will be sorted by sex, age, and procedure type, prior to selecting a systematic random sample of patients.

RTI expects a 32% response rate for mail only mode, 32% for phone only, and 40% for mixed mode, indicating that we need samples of 4,906 for mail only, 4,906 for phone, and 3,925 for mixed mode to achieve 1,570 completed interviews for each mode. RTI will select a random sample of approximately 13,800 patients, or 138 patients per facility (46 patients per facility per month for each of the three months) using a systematic random sampling method. RTI will sort the patient sampling frame by age group, gender, and procedure type to balance the sample. RTI will then randomly assign patients to the three modes using the inverse of the estimated response rates.

B.1.2b National Implementation Sampling Specifics

For the national implementation of O/ASPECS, each participating facility will send to its contracted survey vendor patient sample frames containing information about each patient who received an outpatient surgery or procedure during the sample month. The survey vendor will remove patients who do not meet survey eligibility requirements and then draw a random sample of the remaining patients.

Survey vendors working under contract with O/ASPECS will be instructed to use a reliable program to generate a random patient sample. CMS will recommend that survey vendors use the free program RATSTATS, available from the DHHS, Office of Inspector General website, or some other validated sample selection programs such as SAS to select the sample. The recommended sampling procedure is simple random sampling, but disproportionate and proportional stratified sampling may be allowed subject to CMS’s approval.

A minimum of 300 completed surveys annually is the target for each participating outpatient facility. If a facility patient volume is too small to yield 300 completed surveys per year, a census will be surveyed. The 300 completed surveys needed for analysis is derived from the formula for the precision of a proportion with the estimate at 0.5, the confidence interval of about +/- 0.05, and a confidence level of 95%. The number of patients needed to be selected each month to yield a minimum of 300 completed surveys per year will ultimately be determined by each facility and its survey vendor. Exhibit B‑2 shows a general guide on the sample size for different mode.

Exhibit B-2. Expected Annual Sample Size to Achieve 300 Completed Surveys

Mode	Sample Size	Number of Respondents	Response Rate (%)
Mail Only	938	300	32
Telephone Only	938	300	32
Mail + Telephone Mixed Mode	750	300	40

CMS recommends that prior to starting the national implementation, survey vendors acquire from client facilities sample frame information for each of the 3 or 6 months prior to the beginning of the national implementation. These test files will be used to determine an appropriate sampling rate to use during the national implementation. Sampling rates should be based on the number of patients who meet survey eligibility criteria in the frames of months 2 through 6. The frame of month 1 will not have any patients who are ineligible for the survey because they were previously sampled.

B.2 Information Collection Procedures

Three modes of survey administration will be tested in the mode experiment and allowed during the national implementation to give facilities options in how they would like to administer the survey, based on their goals and resources. These three modes are described below:

• Mail-only Mode

• Mailing of the questionnaire and cover letter to all sampled patients.

• Second mailing of the questionnaire with a cover letter to sample patients who do not respond to the first mailing within 3 weeks after the first questionnaire package is mailed.

• Telephone-only Mode

• A maximum of five telephone contact attempts per patient to complete the survey.

• Mixed Mode (Mail with Telephone Follow-up )

• Mailing of the questionnaire and cover letter to all sample patients.

• Telephone follow-up with all sample patients who do not respond to the questionnaire mailing. A maximum of five telephone contact attempts per patient will be made to complete the survey.

Data collection for each sampled patient will be initiated no later than 3 weeks after the close of the sample month. Once data collection begins, it must be completed within 6 weeks.

Survey vendors who wish to become approved to conduct the national implementation of O/ASPECS on behalf of outpatient facilities must complete the O/ASPECS vendor training, which will provide detailed guidance on the protocols and guidelines for all aspect of survey implementation, from sample selection to data collection and data submission. As of the date of this submission, CMS anticipates that the first training sessions for vendors will be offered in October 2015. The national implementation is expected to start in early 2016.

B.3 Methods to Maximize Response Rate

To reduce nonresponse bias, every effort will be made to maximize the patient response rate while retaining the voluntary nature of the survey. RTI estimates achieving a response rate of approximately 32% for mail only and telephone only, 40% for mail with telephone mixed mode in the mode experiment based on the field test and other CAHPS studies.

With respect to both the mode experiment and national implementation, the questionnaire mailing will include a cover letter containing information about the survey, including sponsorship and objectives, a description of how survey results will be used, and the name and toll-free telephone number of a survey staff member that sampled patients can contact if they have questions or need additional information about the survey. Mailings will also include a statement that assures patients that their survey responses will not be linked to their names or any other information that can identify them.

For the mail-only mode, both RTI (for the mode experiment) and survey vendors (for the national implementation) will use best practices in survey materials to enhance response rates. These best practices include using a simple font no smaller than 10-point size in the survey cover letters, allowing ample white space between questions in the questionnaire, avoiding a format that displays the questions as a matrix, using a unique subject identification number on the questionnaire rather than printing the sample member’s name, and displaying the OMB number and expiration date on the questionnaire. The second mailing for the mail-only mode is expected to increase the response rate, as is the telephone follow-up portion of the mixed-mode implementation.

For the telephone-only mode and follow-up of mail survey nonrespondents for mixed-mode, RTI (for the mode experiment) and survey vendors (for the national implementation) will make up to five attempts to reach each sample patient, with those attempts varying by day of the week and time of day. Telephone interviewers will be trained on how to answer questions that are most frequently asked by sample patients and to address any concerns that they may have about participating in the survey.

B.4 Tests of Procedures

To achieve the purpose of the mode experiment the following analyses will be conducted:

• Analyses of individual survey items will assess missing data and item distributions.

• Hypothesis testing will detect differences in key variables between modes.

• The analysis of individual items and the hypothesis testing will form the basis for constructing an adjustor to be used for telephone and mixed-mode surveys.

RTI will conduct regression analyses for key survey outcomes, including individual rating questions or composite measures, to determine the patient-mix adjustors which may be necessary for reporting of the national survey results. RTI will evaluate whether the ranking of outpatient facilities differs for adjusted and unadjusted O/ASPECS results.

B.5 Statistical Consultation and Independent Review

This sampling and statistical plan was prepared by RTI International. The primary statistical design was provided by Patrick Chen of RTI International. Mr. Chen can be reached by telephone at (919) 541-6309 or by e-mail at [email protected].

References

Ambulatory Surgery Center Association (ASCA). (2011). Ambulatory Surgery Centers: A Positive Trend in Health Care, October 2011. Retrieved from http://www.ascassociation.org/Resources/ViewDocument/?DocumentKey=7d8441a1-82dd-47b9-b626-8563dc31930c

Medicare Payment Advisory Commission (MedPAC). (2014, March). Report to the Congress: Medicare Payment Policy, p. 129. Washington, DC: MedPac. Retrieved from http://medpac.gov/documents/Mar14_EntireReport.pdf.

U.S. Bureau of Labor Statistics. (2013). May 2013 National Occupational Employment and Wage Estimates United States. U.S. Department of Labor, Bureau of Labor Statistics. Retrieved from http://www.bls.gov/oes/current/oes_nat.htm#00-0000. Based on average wages.

1Ideally facility size can be defined according to the number of patients a facility served from the POS file. Facilities will be sorted by patient volume; the top 1/3 of facilities will be classified as ‘Large’, the middle 1/3 of facilities as ‘Medium’, and the bottom 1/3 of facilities as ‘Small’. However, if the information about the number of patients a facility served is unavailable, we could use other measures as a surrogate, such as the number of beds or operating rooms, etc.

22