Supporting Statement B 2019 12 16

Supporting Statement – Part B

Commercial Consumer Assessment of Healthcare Providers and Systems

Introduction

The Consumer Assessment of Healthcare Providers and Systems (CAHPS) is developed by the Agency for Healthcare Research and Quality (AHRQ), a division of the Department of Health and Human Services (HHS). It is the industry standard for measuring patient experience and satisfaction. CAHPS surveys are utilized by a wide variety of public programs, including Medicare, Medicaid, TRICARE, Veterans Affairs (VA), and states. In addition, health plan accrediting bodies all require commercial health plans to conduct a CAHPS survey of their enrollees in each line of business as part of the accreditation process. The National Quality Forum is the endorsement body for healthcare quality and customer experience measures and has endorsed CAHPS through its rigorous process.

There is no alternative customer experience survey in health care that is as thoroughly vetted, methodologically sound, widely known and utilized as the existing CAHPS survey for health plans.

Obtaining OMB approval and a PRA number for CAHPS surveys is done by all the federal entities listed in the previous paragraph, for example the Centers for Medicare and Medicaid Services’ (CMS) Medicare Advantage and Prescription Drug Plan CAHPS^® Survey MA PDP CAHPS OMB number 0938-0732 (expires 4/30/2021).

AHRQ CAHPS surveys provide a uniform and industry accepted survey vehicle to assess member satisfaction. From the AHRQ website (https://www.ahrq.gov/cahps/surveys-guidance/hp/index.html):

The CAHPS Health Plan Survey is a tool for collecting standardized information on enrollees' experiences with health plans and their services. It was designed to support consumers in assessing the performance of health plans and choosing the plans that best meet their needs. Health plans can also use the survey results to identify their strengths and weaknesses and target areas for improvement. Survey results can be used to:

• Support consumers in assessing the performance of health plans and choosing the plans that best meet their needs.

• Identify the strengths and weaknesses of health plans and target areas for improvement.

As part of its FEHB Plan Performance Assessment (PPA), OPM is requiring all FEHB Carriers to utilize the CAHPS commercial survey using a certified CAHPS vendor (more on certification below). The PPA ties FEHB Carriers profit to performance on key metrics (48 CFR 1615.404-70).

OPM is using the Commercial survey. The survey utilizes a sampling methodology as required by the survey licensee, the National Committee for Quality Assurance (NCQA), outlined in extensive detail in the HEDIS Volume 3: Specifications for Survey Measures includes surveys and protocols for the CAHPS 5.0H Survey. Information can also be found on the AHRQ website here https://www.ahrq.gov/sites/default/files/wysiwyg/cahps/surveys-guidance/hp/fielding-the-survey-hp50-2013.pdf. Recommended data collection modes include mail and telephone, and

protocols have been developed for each mode.

AHRQ versions of the survey are utilized by CMS as part of their public reporting and reimbursement programs, as OPM is also doing. CAHPS is also utilized by many private health insurance purchasers, and is part of the accreditation process for all health plan accrediting bodies: Accreditation Association for Ambulatory Health Care, National Committee for Quality Assurance and URAC.

B1.  Development of Concepts, Methods, and Design

AHRQ first launched the CAHPS Survey in 1995 to develop standardized surveys that organizations can use to collect comparable information on patients’ experience with care, and to generate tools and resources to support the dissemination and use of comparative survey results to inform the public and improve health care quality.

From AHRQ’s website (https://www.ahrq.gov/sites/default/files/wysiwyg/cahps/about-cahps/cahps-program/14-p004_cahps.pdf.):

Development of the CAHPS Surveys

The development process for CAHPS surveys emphasizes scientific rigor and frequent input from patients and other key stakeholders. Major steps include literature reviews and environmental scans, focus groups with patients, input from health care providers and other key stakeholders, cognitive testing of survey questions and reporting measures, and field testing. This process is designed to ensure that the survey will generate valid and reliable data to meet the information needs of health care consumers, care providers, health plans, purchasers, and policymakers.

To achieve the program’s goals, AHRQ funds private research organizations with proven expertise in survey design and evaluation, public reporting, and quality assessment and improvement. The current organizations are RAND and the Yale School of Public Health. These research organizations work with AHRQ and other Federal agencies to develop, test, and maintain the surveys and to produce resources that support and evaluate their use in reports and improvement initiatives.

Collectively, these organizations are known as the CAHPS Consortium.

Uses of CAHPS Survey Results

Once AHRQ’s CAHPS Consortium releases a CAHPS survey into the public domain, it is available for any organization to use for its own purposes. The results of CAHPS surveys are typically used to monitor and drive improvements in patient experience with care and to better inform consumers about health care providers in their area. Some organizations incorporate the survey results into programs that reward or recognize health care providers for providing high-quality care.

The target population is FEHB members who are eighteen and older without Medicare and were enrolled in the health plan. The estimated target population is 2.55 million people. This sample frame is the eligible population and is required by AHRQ and NCQA. The anchor date to be eligible is December 31 of the measurement year (i.e., the calendar year before the survey is administered).

AHRQ and NCQA require a systematic sampling method. The sample is drawn by the certified survey vendor. The steps, as shown in Section 3 of NCQA’s HEDIS 2020 Volume 3, Specifications for Survey Measures, are:

Step 1

Determine the minimum required sample size (MRSS) from Table S-3. If the number of eligible members in the sample frame is ≤ MRSS, include the health plan’s entire eligible population in the sample (steps 2–7 are not needed in this case).

Table S-3: Minimum Required Sample Sizes

Survey Type	Required Sample Size
Adult Commercial	1,100
Adult Medicaid	1,350
Child Medicaid	1,650

Step 2	Determine the oversampling rate based on anticipated disenrollment and response rate (refer to Oversampling section).
Step 3	Determine the final sample size (FSS). The FSS includes the MRSS (from step 1) plus oversample (from step 2) and is calculated by the following formula: FSS = MRSS + (MRSS  oversampling rate) (round up to the next whole number), where MRSS = the minimum required sample size. For example, if the MRSS is 1,100 and a 10 percent oversample is needed, FSS = 1,100 + (1,100  0.10) = 1,210.
Step 4	Using the validated sample frame, sort the list of eligible members (EM) in alphabetical order by last name, first name, date of birth and address. Sort EMs from Z to A. Note: Sort order applies to all components. Sort all fields by descending order (i.e., last name descending, first name descending, date of birth descending, address descending).
Step 5	Calculate N = EM/FSS. Round down to a whole number. Determine N, which is used in the formula to determine which member will start your sample. N is calculated using the equation: N = EM/FSS where EM = the eligible member population (the number of eligible members in the sample frame) and FSS = the final sample size (step 3).
Step 6	Calculate START = (RAND  N). Before choosing members, determine the member to start with (START). It is important that the sample be selected from a single pass through the member list. START can have many values and still allow only one pass.
	NCQA provides survey vendors with list of Random Numbers (RAND) each with a value between 0 and 1. Survey vendors use the RAND to calculate the starting point from which to draw the final sample. Calculate the number from which to start drawing the final sample as follows: START = (RAND  N) (round per the .5 rule to the nearest whole number greater than 0), where RAND = the random number.
Step 7	Select the sample, choosing every ith member using the formula: ith member = START + [(i-1) x (EM/FSS)] (rounding [(i-1) x (EM/FSS)] per the .5 rule to the nearest whole number greater than 0). For i = 2,3,4, …, FSS where EM = the eligible member population (e.g., the number of members in the sample frame). FSS = the final sample size (step 3). Starting with the member corresponding to the number START, choose every ith member until the FSS is met. If the oversample was underestimated and the sample falls below the MRSS, the oversampling rate must be adjusted and a new sample must be pulled.

Additional information is included addressing small numbers and oversampling.

Two modes of survey administration are allowed during the national implementation to give health plans options in how they would like to administer the survey, based on their goals and resources. These two modes are described below:

• Mail-only Mode

  • Mailing of the questionnaire and cover letter to all sampled members.

  • Reminder postcard mailing

  • Second mailing of the questionnaire and cover letter to sampled members who do not respond to the first questionnaire mailing within 5 weeks

  • Second reminder postcard mailing

  • Third mailing of the questionnaire and cover letter to sampled members who do not respond to the first or second questionnaire mailing within 3 weeks of the second questionnaire mailing.

• Mixed Mode (Mail with Telephone Follow-up)

  • Mailing of the questionnaire and cover letter to all sampled members.

  • Reminder postcard mailing

  • Second mailing of the questionnaire and cover letter to sampled members who do not respond to the first questionnaire mailing within 5 weeks

  • Second reminder postcard mailing

  • Telephone follow-up with all sampled patients who do not respond to one of the questionnaire mailings. A maximum of six telephone contact attempts per sampled patient will be made to complete the survey.

Under the FEHB contract, FEHB Carriers must administer CAHPS consistent with the AHRQ guidelines required by NCQA. These can be found here https://www.ahrq.gov/sites/default/files/wysiwyg/cahps/surveys-guidance/hp/fielding-the-survey-hp50-2013.pdf, and in Volume 3.

B2.  Collection of Data

To collect the data, health plans contract with NCQA-Certified survey vendors to administer the CAHPS survey on their behalf. To become an NCQA-Certified survey vendor, “an organization must demonstrate that it has the capabilities, experience and expert personnel to collect and report survey results accurately. NCQA issues an annual request for proposal (RFP) for new organizations to apply for certification.”

In volume 3, NCQA provides sample cover letters and postcards to use with the CAHPS survey. As noted above, survey administration procedures are also detailed in Volume 3 and the surveying must be done by certified CAHPS vendors. These strict requirements ensure the survey is consistently administered, allowing for the development of industry wide benchmarks and the comparison of survey results across health plans.

The surveys are collected and processed by NCQA-Certified survey vendors, and submitted through the NCQA Portal, access to which is restricted by NCQA. NCQA validates the submissions, determines benchmarks for each measure, and provides results to health plans and purchasers. Respondents are informed about the data collection in the instructions for the survey. OPM contracts with an NCQA-Certified vendor to further analyze and validate the aggregated results and produce a dataset for OPM’s use.

FEHB Carriers are required to report their CAHPS results, and many report the results for their entire book of business of which FEHB is a part. Consumer response to the survey is entirely voluntary. The burden on consumers is not considered onerous as the estimated time to complete the survey is fifteen minutes.

OPM receives results that are aggregated at the plan level and used as part of the formula for the determination of FEHB Plans profit factor. Aggregated CAHPS survey results are also provided on two OPM webpages as categorical results to help inform consumer selection of FEHB health plans.

NCQA sets minimum response rates, and the required number of responses to obtain a reportable result, which requires a denominator of 100. NCQA provides detailed procedures for oversampling for survey measures that it does not expect to achieve a denominator of 100 for most survey calculations, and a formula to calculate the estimated number of responses for individual survey questions.

B3.  Processing and Editing of Data

Standard data cleaning and analysis procedures are performed by the NCQA-Certified vendor. The vendor is responsible for validating the results, including making sure that the sample protocol and all other requirements are followed. Only then is the data submitted to NCQA thru the NCQA Secure Interactive Data Submission System (IDSS). OPM’s NCQA-Certified vendor does additional validation and analytics under a separate contract with OPM.

B4.   Production of Estimates and Projections

The CAHPS sampling protocols are set by AHRQ and NCQA and applied by the NCQA-Certified Vendor for each health plan. This ensures that they are statistically valid, can be used to represent population results, and that comparisons of results can be made across all Carriers. NCQA-Certified vendors identify any biased results, and those results are excluded from use. If a specific measure is identified as biased, the results are not used in the analysis or for the calculation of profit factor, and those results are also not made available to consumers.

B5.  Data Analysis

Respondent-level data analysis is conducted by the NCQA-Certified vendor before submission to NCQA. Plan-level basic survey analysis techniques (e.g., comparing results to commercial benchmarks produced by NCQA for CAHPS, calculating frequencies, patterns) will be used to interpret and explain the data provided by respondents. The results will be used in the FEHB Plan Performance Assessment, and categorical results for each FEHB Plan posted on the OPM website (https://www.opm.gov/healthcare-insurance/healthcare/plan-information/compare-plans/quality) and included in the results for the FEHB Plan Comparison Tool (https://www.opm.gov/healthcare-insurance/healthcare/plan-information/compare-plans/).

B6.  Review and Evaluation Procedures

As noted above, the survey tool was developed by AHRQ and is the industry standard, used for multiple government healthcare programs including Medicare and Medicaid, by NCQA for accreditation, by states for their healthcare programs, and throughout the industry. PRA approval has been granted on numerous occasions for CAHPS in CMS programs.

As noted in B1, the development process for CAHPS surveys emphasizes scientific rigor, input from patients and other key stakeholders, focus groups, cognitive testing of survey questions and reporting measures, and field testing. AHRQ funds private research organizations with proven expertise in survey design and evaluation, public reporting, and quality assessment and improvement, currently RAND and the Yale School of Public Health. AHRQ contracts with Westat to support the work of the researchers, assist users of CAHPS products, and manage the CAHPS Database. Collectively, these organizations are known as the CAHPS Consortium. Information on the CAHPS Consortium and the organizations involved in the different phases of the CAHPS program can be found at https://cahps.ahrq.gov/aboutcahps/cahps-program/index.html.

As a part of HHS, AHRQ produces CAHPS and is subject to OMB and agency Information Quality Guidelines. Other federal programs use CAHPS and have obtained PRA approval and numbers from OMB. AHRQ has obtained PRA approval for the testing of CAHPS, and CMS, TRICARE and VA have also obtained OMB approval and PRA numbers, as previously noted.

B7.   Data Dissemination

Aggregated reporting on the CAHPS survey results are provided on two OPM webpages to help inform selection of FEHB health plans. Specific Carrier scores are reported categorically.

No PII is reported on the CAHPS results to OPM. The results are aggregated to the health plan level and no individual responses are received by OPM.

B8.  Contact Person(s)

Joseph Kahn

Federal Employee Insurance Operations

U.S. Office of Personnel Management

(202) 606-0971

[email protected]

Page 6 of 6