Supporting Statement – Part B
Consumer Assessment of Healthcare Providers and Systems (CAHPS) Survey
for the Merit-Based Incentive Payment System (MIPS)
CMS-10450, OMB Control Number 0938-1222
Collections of Information Employing Statistical Methods
Introduction
CMS is submitting updates to two ICRs associated with the CAHPS for MIPS survey to OMB for approval under the PRA as a revision of the previously approved CAHPS for MIPS package (0938-1222; CMS-10450). The CAHPS for MIPS survey (version 2.0) is used in the Quality Payment Program (QPP) to collect data on fee-for-service Medicare beneficiaries’ experiences of care with eligible clinicians participating in MIPS and is designed to gather only the necessary data that CMS needs for assessing physician quality performance, and related public reporting on physician performance, and should complement other data collection efforts. The CAHPS for MIPS survey version 2.0 consists of the core Agency for Healthcare Research and Quality (AHRQ) CAHPS Clinician & Group Survey, version 3.0, plus additional survey questions to meet CMS’s information and program needs. The survey information is used for quality reporting, the Physician Compare website, and annual statistical experience reports describing MIPS data for all MIPS eligible clinicians.
CMS will be requesting approval for one additional PRA package associated with the CY 2019 Physician Fee Schedule (PFS) final rule. The collection of information associated with the CY 2019 PFS final rule (other than virtual group election and CAHPS-related data collection) will be submitted as a revision of the currently approved MIPS PRA package (0938-1314; CMS-10621). OMB has approved the information collection associated with the virtual group election process, which was submitted as a separate PRA package (0938-1343; CMS-10652).
Because historical participation rates for quality data submission under Physician Quality Reporting System (PQRS) have never reached 100 percent, we anticipate that MIPS will not achieve full participation.
Based on participation data from the 2017 MIPS performance period and 2018 MIPS eligibility data, we assume that 964,246 clinicians who participated in MIPS and who are not QPs in Advanced APMs in the 2017 MIPS performance period will continue to submit quality data in the 2019 MIPS performance period. We assume that 100 percent of APM Entities in MIPS APMs will submit quality data to CMS as required under their models. Groups and virtual groups can elect to contract with a CMS-approved survey vendor to collect and submit the CAHPS for MIPS survey version 2.0 as one of their quality performance category measures and also as a high-weighted activity under the improvement activities performance category. Virtual groups are subject to the same requirements as groups, therefore we will only refer to groups as an inclusive term for both unless otherwise noted.
For the 2019 MIPS performance period, we assume that 282 groups will register to conduct the CAHPS for MIPS survey based on the number of groups which elected to register during the CY 2018 registration period; a decrease of 179 compared to the number of groups currently approved by OMB under the aforementioned control number (82 FR 53917). With regard to beneficiary respondents, we assume that 143 groups will elect to report on the CAHPS for MIPS survey, which is equal to the number of groups that have registered and have a sufficient beneficiary sample size to conduct the CAHPS for MIPS survey in the 2018 MIPS performance period; a decrease of 318 from the 461 groups currently approved by OMB. Based on the number of complete and partially complete surveys for groups participating in CAHPS for MIPS survey administration for the 2017 MIPS performance period, we assume that an average of 273 beneficiaries will respond per group for the 2019 MIPS performance period. Therefore, the CAHPS for MIPS survey will be administered to approximately 39,039 beneficiaries per year (143 groups x an average of 273 beneficiaries per group responding). This is a decrease of 93,268 from our currently approved 132,307 beneficiary estimate.
2 Describe the procedures for the collection of information including:
- Statistical methodology for stratification and sample selection,
- Estimation procedure,
- Degree of accuracy needed for the purpose described in the justification,
- Unusual problems requiring specialized sampling procedures, and
- Any use of periodic (less frequent than annual) data collection cycles to reduce burden.
Groups submitting quality measures data using a CMS-approved survey vendor to report the CAHPS for MIPS survey would need to meet the data submission requirements on the sample of the Medicare Part B fee-for-service (FFS) beneficiaries. Based on the number of groups electing to register during the 2018 registration period that were found to have a sufficient beneficiary sample size, we anticipate that 143 groups will contract with CMS-approved survey vendors to collect CAHPS for MIPS survey version 2.0 data. Groups that elect to participate in the CAHPS for MIPS survey must submit using a CMS-approved survey vendor and must still also submit their other quality measures to ensure that the group meets the requirement for the minimum number of measures. Groups that do not elect to participate in the CAHPS for MIPS survey version 2.0 may just choose to submit the minimum number of required measures through one or more other submission types.1
TABLE 1: Summary of Quality Data Submission Criteria Finalized for the CAHPS for MIPS Survey Year 3
Performance Period |
Measure Type |
Submission Criteria, including Sampling |
Data Completeness |
Jan 1 – Dec 31, 2019 |
Groups |
CMS-approved survey vendor would have to be paired with another collection type to ensure the minimum number of measures are reported. The CAHPS for MIPS survey would fulfill the requirement for a high priority measure (if no outcome measure is available) towards the MIPS quality performance category data submission criteria. The CAHPS for MIPS survey will only count for one quality measure. The CAHPS for MIPS survey will count as a high-weighted activity for the improvement activities performance category. |
Sampling requirements for the group’s Medicare Part B patients |
For groups that elect to contract with a CMS-approved survey vendor, CMS will identify beneficiaries eligible for the survey from the pool of Medicare fee-for-service (FFS) beneficiaries assigned to the group.2 CMS uses retrospective beneficiary assignment determined at the end of the registration period to identify beneficiaries eligible to receive the CAHPS for MIPS survey. If a beneficiary receives at least one primary care service by a primary care clinician who is part of the group or virtual group, the beneficiary is eligible to be assigned to the group or virtual group based on a two-step process3:
The first step assigns a beneficiary to the group or virtual group if the beneficiary receives the plurality of his or her primary care services from primary care clinicians who are part of the group or the virtual group. Primary care clinicians are defined as those with one of seven specialty designations: internal medicine, general practice, family practice, geriatric medicine, nurse practitioner, clinical nurse specialist, and physician assistant.
The second step only considers beneficiaries who have not had any primary care service furnished by a primary care clinician, including primary care clinicians external to the group or the virtual group. Under this second step, we assign a beneficiary to the group or the virtual group if the beneficiary receives the plurality of his or her primary care services from clinicians who are not primary care clinicians within the group or virtual group.
A plurality means a greater proportion of primary care services was provided from clinicians who are part of the group or the virtual group than any other entity, measured in terms of allowed charges. A plurality may be less than the majority of services.
CMS assigns Medicare FFS beneficiaries to a group and then randomly samples from those assigned beneficiaries to create the sample for the CAHPS for MIPS survey. The sample will be limited to beneficiaries age 18 or older, who are not known to be institutionalized or deceased, and who had at least two visits for care to the group. The sample is drawn at the group level, not at the individual clinician level. The survey names a specific clinician, who delivered primary care to the beneficiary over multiple visits in the performance year to help orient the beneficiary to the care he or she received. The named provider can be a physician, specialist, nurse practitioner, physician assistant, or clinical nurse specialist.
CMS will oversample high utilizers of care, defined as beneficiaries who accounted for the highest 10 percent of total primary care charges within each practice. High utilizers represent 25 percent of the survey sample. For practices where the top 10 percent of users comprise fewer than 215 beneficiaries, we will sample all users in the top 10 percent. The reasons for oversampling are two-fold. First, oversampling increases the likelihood that the survey items that measure less-common experiences receive adequate numbers of responses to enable analysis. Second, because one of the goals of the MIPS program is to incentivize high quality and efficient service delivery, it is particularly useful to capture the patient experiences of those with high levels of health care utilization.
The number of beneficiaries sampled may vary based on the size of the group.
For large groups of 100 or more MIPS eligible clinicians:
CMS will draw a sample of 860 beneficiaries
If the group has fewer than 860 beneficiaries, but more than 415 beneficiaries, all eligible beneficiaries will be surveyed in Performance Year (PY) 2018
If the group has fewer than 416 beneficiaries, the survey cannot be conducted
For groups with 25 to 99 MIPS eligible clinicians:
CMS will draw a sample of 860 beneficiaries
If the group has fewer than 860 beneficiaries, but more than 254 beneficiaries, all eligible beneficiaries will be surveyed in PY 2018
If the group has fewer than 255 beneficiaries, the survey cannot be conducted
For groups with 2 to 24 MIPS eligible clinicians:
CMS will draw from a sample of 860 beneficiaries
If the group has fewer than 860 beneficiaries, but more than 124 beneficiaries, all eligible beneficiaries will be surveyed in PY 2018
If the group has fewer than 125 beneficiaries, the survey cannot be conducted
The sample sizes recommended above are based on analysis of 2012 CAHPS for Accountable Care Organizations (ACOs) survey data. Specifically, we set a target number of completed questionnaires for each group practice to obtain a desired level of interunit reliability (IUR) for most survey measures. The IUR is defined as 1-V/(V+t2), where V is the variance of the estimate for a specific unit and t2 is the between-unit variance of population means. For CAHPS for ACOs and CAHPS for PQRS, IUR=0.75 is regarded as adequate reliability for public reporting; IUR between 0.60 and 0.75 and in the lowest 12% of reliability for ACOs or practices is considered low, while IURs below 0.60 are deemed very low. Measure scores with lower than adequate IURs can still provide practices with useful information about patient experience and potential areas for improvement. The target sample for group practices (regardless of their number of eligible clinicians) is 860, as a sample of this size is anticipated to produce measure scores meeting the adequate reliability threshold for most measures; a minimum sample size threshold is set for each practice size category to ensure that practices do not pursue the survey if they have so few beneficiaries that most measures would be expected to have very low reliability. These recommendations reflect a conservative approach that suggests sampling the same sample size for medium and small groups as is recommended for large groups when it is feasible, but lowers the minimum sample size threshold.
The historical response rate for beneficiaries invited to participate in the CAHPS for PQRS survey has ranged from 47 percent in RY 2013 to 37 percent in RY 2016. Factors that contribute to the lower response rate over time include a switch from survey administration by a single vendor to using multiple vendors, and time of year of survey administration.
3 Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, OMB guidance requires that a non-response bias assessment be conducted to determining if the results are generalized to the universe studied.
The CAHPS for MIPS survey version 2.0 will be collected via a mixed-mode data collection protocol that uses a pre-notification letter alerting sample members that a survey will be mailed to them shortly, a first mailing of the full questionnaire booklet, followed by a second mailing to those who do not respond to the earlier mailing of the questionnaire. For those who also do not respond to the second mailing of the questionnaire, CMS-approved survey vendors employ a telephone follow-up through which it offers sample members the opportunity to complete the survey by phone. The mailing materials to all sample members also include a toll-free telephone number that allows recipients to call in to ask questions about the survey. CMS-approved survey vendors are supplied with mail and telephone versions of the survey in electronic form, and text for beneficiary pre-notification and cover letters. Further, CAHPS for MIPS surveys can be administered in English, Spanish, Cantonese, Mandarin, Korean, Russian and/or Vietnamese. Across reporting years 2013-2016, CAHPS for PQRS has achieved a 42 percent response rate on average, slightly higher than some other CAHPS surveys of Medicare beneficiaries.
CMS-approved survey vendors will continue to be required to administer the survey according to established protocols to ensure valid and reliable results. Survey vendors will be required to use appropriate quality control, encryption, security and backup procedures to maintain survey response data. The data would then be securely sent back to CMS for scoring and validation in accordance with applicable law. To ensure that a survey vendor possesses the ability to transmit survey measures data for a particular performance period, we have proposed to require survey vendors to undergo this approval process for each year in which the survey vendor seeks to transmit survey measures data to us. The approval process includes submitting an application, meeting minimum business requirements, participation in training(s), passing post-training evaluation(s), submitting a Quality Assurance Plan, and following the schedule and procedures for survey administration. Additional details about the vendor approval process can be found at https://www.cms.gov/Research-Statistics-Data-and-Systems/Research/CAHPS/mips.html.
With regard to assessment of potential non-response bias, we will compare the characteristics of survey respondents with the characteristics of the sample frame using the standardized mean difference, which compares the mean of a beneficiary characteristic among respondents to the mean among the sample frame. A standardized mean difference of greater than 0.2 indicates that the respondents differ from the sample frame and there is potential for non-response bias. We will also fit a logistic regression model predicting beneficiary-level response using fixed effects for beneficiary characteristics. Any characteristics that are strongly associated with response are potential drivers of nonresponse bias. Differential non-response across practices is directly addressed using case-mix adjustment, recalculated every year ensuring that the comparison of practices is valid in the presence of potential nonresponse bias for that year. The case-mix model includes various demographic and health questions that are used to adjust the summary survey measure (SSM) scores at the practice level. Using case-mix adjustments allows for fair comparisons across practices and across reporting years.
4 Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.
We do not anticipate any testing to occur under this PRA. The current version 2.0 of the CAHPS for MIPS survey was tested prior to the 2018 MIPS performance period and we are not proposing any revisions to the survey instrument. The text below is a summary of that testing.
The CAHPS for MIPS survey version 2.0 reflect two sets of tests. Several of the changes reflect testing done by AHRQ to the core Clinician & Group CAHPS Survey (CG-CAHPS) from version 2.0 to version 3.0. Version 3.0 CG-CAHPS reflects wording improvements (e.g., use of “contact” instead of “phone” to reflect all the ways beneficiaries communicate with providers) and a shorter survey.
The second set of testing was a pilot test done under the auspices of the CAHPS for ACOs pilot survey, which was identical to the proposed CAHPS for MIPS survey version 2.0. The CAHPS for ACOs pilot field testing was conducted from November 2016 through February 2017 with a specific goal of determining whether a shorter survey affects SSM scores, response rates, and reliability. The pilot study participation included 18 ACOs served by seven vendors. The vendor and ACO participants were selected to represent ACOs with high and low CAHPS scores in 2015, ACOs with high and low response rates in 2015, and vendors with many and few ACO clients. Vendors followed standard CAHPS for ACOs data collection protocols and specifications to administer the ACO pilot survey.
Results from the pilot study suggest that administration of the shortened version of the survey (i.e., the pilot survey) is likely to result in improvements in overall response rates. Findings show that the response rate to the pilot survey was 3.4 percentage points higher than the response rate to the RY2016 CAHPS for ACOs survey among ACOs participating in the pilot study. Increases in response rates tended to be larger among ACOs that had lower response rates in the prior year.
In addition, after accounting for survey questions that were removed from the pilot survey, the average survey responses for ACOs who participated in the pilot study were mostly similar across the two survey versions (pilot and RY2016).
5 Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
The CAHPS for MIPS survey version 2.0 consists of the core Clinician & Group CAHPS Survey (CG-CAHPS), version 3.0, which was developed by the Agency for Healthcare Quality Research (AHRQ) and additional supplemental items covering the information needs of CMS and MIPS.
The survey administration, sampling approach, and data collection procedures were designed by the RAND Corporation.
1 In the CY 2017 Quality Payment Program final rule (81 FR 77091), we finalized that MIPS eligible clinicians and groups submitting on behalf of MIPS eligible clinicians could submit information via one submission mechanism or, for groups that elect to include the CAHPS for MIPS survey as a quality measure, one submission mechanism and a CMS-approved survey vendor. However, in the CY 2018 Quality Payment Program final rule, we establish a policy that allows MIPS eligible clinicians and groups submitting on behalf of MIPS eligible clinicians to submit quality data via one or more submission mechanisms (other than a CMS-approved survey vendor) beginning in the 2019 MIPS performance period.
2 https://www.cms.gov/Medicare/Quality-Payment-Program/Resource-Library/CAHPS-for-MIPS-Fact-Sheet.pdf
3 https://www.cms.gov/Medicare/Quality-Payment-Program/Resource-Library/2018-CMS-Web-Interface-and-CAHPS-for-MIPS-Survey-assignment-methodology.pdf
Page
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-16 |