CMS-10500 - Supporting_Statement_B [07-31-2018]

Supporting Statement Part B:

Collection of Information Employing Statistical Methods

Consumer Assessment of Healthcare Providers and Systems Outpatient and Ambulatory Surgery Survey (OAS CAHPS)

CMS-10500, OMB 0938-1314

The Centers for Medicare & Medicaid Services (CMS) is requesting clearance from the Office of Management and Budget (OMB) to conduct the Outpatient and Ambulatory Surgery CAHPS (OAS CAHPS). The purpose of OAS CAHPS is to measure patients’ experience of care with outpatient and ambulatory surgery centers in the United States. As with most large-scale data collections, CMS tested the survey prior to national implementation. This information collection request seeks OMB approval for continued national implementation of the OAS CAHPS Survey, which began in January 2016. The national implementation will be voluntary and sponsored by outpatient surgery facilities. Each participating facility will contract with an independent CMS-approved survey vendor to conduct the survey.

This information collection request also seeks OMB approval to conduct a mode experiment to determine the feasibility of adding Web surveys as a new data collection mode.

The sampling plan for both the national implementation of OAS CAHPS and the mode experiment are described below.

B.1 Potential Respondent Universe and Sample Selection Method

The OAS CAHPS respondent universe is patients 18 years old and older who received an outpatient surgery or a procedure in a HOPD or ASC and who are not discharged to hospice.

For the national implementation, sample selection is single-stage. ASCs and HOPDs that choose to participate work directly with a survey vendor of their choice. The vendors are responsible for selecting adequate patient sample needed to complete 300 complete surveys for each outpatient surgery facility annually.

Mode Experiment.

For the mode experiment, will use a two-stage sample selection method where stage 1 is selection of 70 facilities (35 HOPDs and 35 ASCs) and stage 2 is selection of patients within each facility. A total of 23,312 outpatient surgery patients will be sampled for the mode experiment. These will be divided into five modes: mail only, telephone only, Web only, Web with mail follow-up, and Web with telephone follow-up of non-respondents to the Web survey. We anticipate completing 7,850 surveys or 1,570 surveys per mode.

B.1.1 Sampling Patients

For national implementation, outpatient surgery facilities assemble a census of their patients who received a surgery during the previous month. Each facility will submit a monthly file containing patient information for all eligible patients to its contracted survey vendor. To be eligible to be selected for the OAS CAHPS sample, patients must meet the following eligibility requirements:

• have had an outpatient surgery or procedure from the facility in the prior calendar month;

• the surgery or procedure is included in the surgical codes which are being studied by OAS CAHPS (see B.1.2a);

• were at least 18 years old when they received their outpatient surgery or procedure;

• were not included in the sample for pre-specified period; and

• were not discharged to hospice or to a hospital for an inpatient stay.

Mode Experiment.

For the mode experiment, RTI will use a two-stage sample design in which the outpatient surgery facilities are selected first, and patients are then selected from these outpatient surgery facilities. Our target is to recruit 70 outpatient surgery facilities, comprising 35 HOPDs and 35 ASCs. The facility sampling frame will be all eligible outpatient surgery facilities obtained from CMS’s Provider of Service file. RTI will carefully balance the outpatient surgery facility sample to represent key facility characteristics and ensure that the outpatient surgery facility sample is also distributed over different regions. The four key facility characteristics are:

• Facility Specialty: Single, Multiple

• Facility Size¹: Large, Medium, Small

• Facility Location: Urban, Rural

• Facility Ownership: Hospital, Management Company, Health System/Managed Care, Physician, Government

23,312 patients will be selected from the 70 selected facilities, which will yield 7,850 interviews.

B.1.2 Sampling Specifics

For the national implementation of OAS CAHPS, each participating facility sends to its contracted survey vendor patient sample frames containing information about each patient who received an outpatient surgery or procedure during the sample month. The survey vendor removes patients who do not meet survey eligibility requirements and then draws a random sample of the remaining patients.

Survey vendors working under contract with OAS CAHPS are instructed to use a reliable program to generate a random patient sample. CMS recommends that survey vendors use the free program RATSTATS, available from the DHHS, Office of Inspector General website, or some other validated sample selection programs such as SAS to select the sample. The recommended sampling procedure is simple random sampling, but stratified systematic sampling, disproportionate and proportional stratified sampling may be allowed subject to CMS’s approval.

A minimum of 300 completed surveys annually is the target for each participating outpatient facility. If a facility patient volume is too small to yield 300 completed surveys per year, a census is surveyed. The 300 completed surveys needed for analysis is derived from the formula for the precision of a proportion with the estimate at 0.5, the confidence interval of about +/- 0.05, and a confidence level of 95%. The number of patients needed to be selected each month to yield a minimum of 300 completed surveys per year will ultimately be determined by each facility and its survey vendor. Exhibit B‑1 shows a general guide on the sample size for different mode.

Exhibit B-1. Expected Annual Sample Size to Achieve 300 Completed Surveys

Mode	Sample Size	Number of Respondents	Response Rate (%)
Mail Only	938	300	32
Telephone Only	938	300	32
Mail + Telephone Mixed Mode	750	300	40

CMS recommends that before initiating data collection for the first time (for a client facility), the survey vendor acquires from the client facility sample frame information for 3 or 6 months prior to the first sample month of administration. These test files are used to determine an appropriate sampling rate to use for implementation. Sampling rates should be based on the number of patients who meet survey eligibility criteria in the frames of months 2 through 6. The frame of month 1 does not have any patients who are ineligible for the survey because they were not previously sampled.

Mode Experiment.

Outpatient surgery facilities will assemble a census of their patients who received a surgery during the previous month. Each facility will submit a monthly file containing patient information for all eligible patients to RTI (for the mode experiment) The mode experiment is a one-time survey which will involve sampling and data collection for the patients in the reference period. To be eligible to be selected for the mode experiment sample, patients must meet the following eligibility requirements:

• have had an outpatient surgery or procedure from the facility in the prior calendar month;

• the surgery or procedure is included in the surgical codes which are being studied by OAS CAHPS (see B.1.2a);

• were at least 18 years old when they received their outpatient surgery or procedure;

• were not included in the sample for pre-specified period; and

• were not discharged to hospice or to a hospital for an inpatient stay.

Mode Experiment Patient Sampling Specifics

For the mode experiment, the patient sample size was calculated on the basis of power to detect a 0.05 difference in the proportion estimates when the proportion estimate is 0.5, with 80% confidence and the 0.05 significance level. The sample size, the total number of completed interviews needed for each mode being tested in the mode experiment is 1,570. With five data collection modes being tested, the targeted number of completed surveys is 7,850. Exhibit B‑2 shows the response rates and the number of patients need to be selected for each mode.

Exhibit B-2. Proposed Sample Size, Expected Number of Completed Interviews, and Expected Response Rate

Mode	Sample Size	Number of Respondents	Response Rate (%)
Mail	4,245	1,570	37
Telephone	4,620	1,570	34
Web	6,280	1,570	25
Web with Mail	3,925	1,570	40
Web with Telephone	4,242	1,570	37
Total	23,312	7,850	35

After facilities are recruited, RTI will work with each participating facility to obtain a list of patients receiving a surgery during the 3-month reference period. RTI will work with CMS to set up eligibility criteria and construct a sampling frame of all eligible patients for each participating facility. All eligible patients in the patient sampling frame will be sorted by sex, age, and procedure type, prior to selecting a systematic random sample of patients.

RTI expects a 37% response rate for mail mode, 34% for phone, 25% for Web mode, 40% for Web with mail follow-up mixed mode, and 37% for Web with telephone follow-up mixed mode indicating that we need samples of 4,245 for mail only, 4,620 for phone, 6,280 for web only, 3,925 for Web with mail mixed mode and 4,242 for Web with telephone follow-up to achieve 1,570 completed interviews for each mode. RTI will select a random sample of approximately 23,312 patients, or 333 patients per facility (111 patients per facility per month for each of the three months) using a systematic random sampling method. RTI will sort the patient sampling frame by age group, gender, and procedure type to balance the sample. RTI will then randomly assign patients to the five modes, allocating using the inverse of the estimated response rates.

B.2 Information Collection Procedures

Three modes of survey administration are allowed during the national implementation to give facilities options in how they would like to administer the survey, based on their goals and resources. These three modes are described below:

• Mail-only Mode

• Mailing of the questionnaire and cover letter to all sampled patients.

• Second mailing of the questionnaire with a cover letter to sampled patients who do not respond to the first mailing within 3 weeks after the first questionnaire package is mailed.

• Telephone-only Mode

• A maximum of five telephone contact attempts per sampled patient to complete the survey.

• Mixed Mode (Mail with Telephone Follow-up )

• Mailing of the questionnaire and cover letter to all sampled patients.

• Telephone follow-up with all sampled patients who do not respond to the questionnaire mailing. A maximum of five telephone contact attempts per sampled patient will be made to complete the survey.

Data collection for each sampled patient should be initiated no later than 3 weeks after the close of the sample month. Once data collection begins, it must be completed within 6 weeks.

Survey vendors who wish to become approved to conduct the national implementation of OAS CAHPS on behalf of outpatient facilities must complete the OAS CAHPS vendor training, which provides detailed guidance on the protocols and guidelines for all aspect of survey implementation, from sample selection to data collection and data submission. As of the date of this submission, CMS anticipates that the first training sessions for vendors will be offered in October 2015. The national implementation is expected to start in early 2016.

Mode Experiment.

CMS proposes to test five modes of survey administration. The information collection procedures for mail and telephone mode are the same as described above for national implementation. The information collection procedures for the other three modes are described below:

• Web Only Mode

• Email invitations and send mail Web survey invitation to all sampled patients

• After two weeks, send a second email invitation and mail a second web survey invitation to all sampled patients who do not respond to the first invitation.

• At the start of the third week, send a third email invitation and mail a third web survey invitation to all sampled patients who do not respond to the second invitation.

• At the start of the fourth week, send a fourth email invitation and mail a fourth web survey invitation to all sampled patients who do not respond to the third invitation.

• At the start of the fifth week, send a fifth email reminder to all sampled patients who do not respond to the fourth invitation.

• Web with Mail Follow-up Mode

• Email invitations and send mail web survey invitation to all sampled patients

• After two weeks, send a second email invitation and mail a second web survey invitation to all sampled patients who do not respond to the first invitation.

• At the start of the third week, send a mail questionnaire with cover letter to all sampled patients who do not respond to the second invitation.

• At the start of the fifth week, send a third email reminder to all sampled patients who do not respond to the mail questionnaire.

• Web with Telephone Follow-up Mode

• Email invitations to all sampled patients.

• After two weeks, send a second email invitation to all sampled patients who do not respond to the first invitation.

• At the start of the third week, start telephone contact with all sampled patients who do not respond to the second invitation.

• At the start of the fifth week, send a third email reminder to all sampled patients who do not respond to the telephone follow-up.

Data collection for each sampled patient will be initiated no later than 3 weeks after the close of the sample month. Once data collection begins, it must be completed within 6 weeks.

B.3 Methods to Maximize Response Rate

To reduce nonresponse bias, every effort will be made to maximize the patient response rate while retaining the voluntary nature of the survey. RTI estimates achieving a response rate of approximately 32% for mail only and telephone only, 40% for mail with telephone mixed mode in the mode experiment based on the mode experiment and voluntary national implementation data received to date.

The questionnaire mailing includes a personalized cover letter containing information about the survey, including sponsorship and objectives, a description of how survey results are used, and the name and toll-free telephone number of a survey staff member that sampled patients can contact if they have questions or need additional information about the survey. Mailings also include a statement that assures patients that their survey responses will not be linked to their names or any other information that can identify them.

For the mail-only mode, survey vendors use best practices in survey materials to enhance response rates. These best practices include using a simple font no smaller than 10-point size in the survey cover letters, allowing ample white space between questions in the questionnaire, avoiding a format that displays the questions as a matrix, using a unique sample identification number on the questionnaire rather than printing the sample member’s name, and displaying the OMB number and expiration date on the questionnaire. The second mailing for the mail-only mode is expected to increase the response rate, as is the telephone follow-up portion of the mixed-mode implementation.

For the telephone-only mode and follow-up of mail survey nonrespondents for mixed-mode, survey vendors make up to five attempts to reach each sample patient, with those attempts varying by day of the week and time of day. Telephone interviewers are trained on how to answer questions that are most frequently asked by sample patients and to address any concerns that they may have about participating in the survey.

Mode Experiment.

Informed in part by the 2015 mode experiment, we expect to achieve a response rate of approximately 37% for the mail mode, 34% for the telephone mode, 25% for Web, 40% Web with mail follow-up and 37% for Web with telephone follow-up.

For the mode experiment in 2018, the questionnaire mailings for mail-only mode will follow the same protocols outlined above for national implementation. The telephone mode protocols will also be the same as national implementation protocols.

For the Web mode, the instrument will be designed to be optimized for both desktop/laptop computers and for mobile devices. The Web survey design will be Section 508-compliant. Up to four e-mail and mail invitations will be sent to maximize response rates without burdening sample members with survey requests.

For the Web with mail follow-up mode, the same instruments developed for the Web mode and for the mail mode will be used. A maximum of three e-mail and three mail contacts will be made.

For the Web with telephone follow-up mode, the same instruments developed for the Web mode and for the telephone mode will be used. A maximum of three e-mail and five telephone contacts will be made.

B.4 Tests of Procedures

During a mode experiment, conducted in 2015, the following analyses were conducted:

• Analyses of individual survey items to assess missing data and item distributions;

• Hypothesis testing to detect differences in key variables between modes; and

• The analysis of individual items and the hypothesis testing to form the basis for constructing an adjustor to be used for telephone and mixed-mode surveys.

RTI conducted regression analyses for key survey outcomes, including individual rating questions or composite measures, to determine the patient-mix adjustors which are necessary for reporting of the national survey results. RTI evaluated whether the ranking of outpatient facilities differed for adjusted and unadjusted results.

Mode Experiment.

The mode experiment in 2018 will follow the same test of procedures outlined above for the 2015 mode experiment.

B.5 Statistical Consultation and Independent Review

This sampling and statistical plan was prepared by RTI International. The primary statistical design was provided by Patrick Chen of RTI International. Mr. Chen can be reached by telephone at (919) 541-6309 or by e-mail at [email protected].

References

Ambulatory Surgery Center Association (ASCA). (2011). Ambulatory Surgery Centers: A Positive Trend in Health Care, October 2011. Retrieved from http://www.ascassociation.org/Resources/ViewDocument/?DocumentKey=7d8441a1-82dd-47b9-b626-8563dc31930c

Medicare Payment Advisory Commission (MedPAC). (2014, March). Report to the Congress: Medicare Payment Policy, p. 129. Washington, DC: MedPac. Retrieved from http://medpac.gov/documents/Mar14_EntireReport.pdf.

U.S. Bureau of Labor Statistics. (2013). May 2013 National Occupational Employment and Wage Estimates United States. U.S. Department of Labor, Bureau of Labor Statistics. Retrieved from http://www.bls.gov/oes/current/oes_nat.htm#00-0000. Based on average wages.

1 Ideally facility size can be defined according to the number of patients a facility served from the Provider of Service file. Facilities will be sorted by patient volume; the top 1/3 of facilities will be classified as ‘Large’, the middle 1/3 of facilities as ‘Medium’, and the bottom 1/3 of facilities as ‘Small’. However, if the information about the number of patients a facility served is unavailable, we could use other measures as a surrogate, such as the number of beds or operating rooms, etc.

24