CMS-10488 - Goals of the Psychometric and Beta Test Goals

Appendix A: QHP Enrollee Survey Psychometric Report Findings & Beta Test Findings

Goals of Psychometric Test

• Test if response rates differ by mode

• Inform changes in data collection methods to improve response rates and minimize potential non-response bias

• Verify the internal and inter-unit reliability and validity of existing CAHPS 5.0 composite measures

• Evaluate the reliability and validity of new questions and new composite measures

• Revise composites if necessary

• Test the equivalence of measurement properties across 3 languages (English, Spanish, Chinese) and 3 modes (mail, phone, web)

• Identify survey items that can be dropped from the instrument

• Implement and evaluate the process for sampling frame construction and for sampling

• Estimate sample size requirements for future data collections

• Identify potential case mix adjusters

• Determine if additional cognitive testing is necessary

Goals of Beta Test

• Test the survey vendor system

• Repeat analyses from psychometric test to verify findings and test the effect of any changes to methods or survey content

• If possible, calculate QHP-level assessment scores to provide initial feedback to CMS and states to inform quality improvement efforts and to be used in the beta test of the Quality Rating System (QRS)

Lessons from the Psychometric & Beta Tests

From the Psychometric Test, we learned that:

• Response rates varied substantially by language and survey mode. Chinese-language samples had the highest response rate, followed by Spanish- and English-language samples. Within the English-language mode experiment, mail with phone follow-up had the highest response rate, followed by mail with FedEx follow-up, telephone only, first class mail, and online. Thus, we recommended using a mail with phone follow-up mode for all languages in subsequent rounds of the QHP Enrollee Survey.

• There was substantial variation in composite scores among reporting units, which resulted in good inter-unit reliability. Based on these findings, we projected that the standard CAHPS sample size target for CAHPS Health Plan Surveys—300 completes per sample unit—would achieve enough usable responses to obtain good inter-unit reliability in the Beta Test.

• Because of low item response rates for some items and composites in the Psychometric Test, our full hypothesized factor structure could not be analyzed. Thus, psychometric analysis of data from the larger 2015 Beta Test sample was required to evaluate some components of the hypothesized structure. Nevertheless, the components of the hypothesized structure that were evaluated using the 2014 Psychometric Test, including the entire CAHPS Health Plan 5.0 factor structure and some of the supplemental composites needed for the QRS, had excellent fit to the data.

From the Beta Test, we learned that:

• Despite achieving the desired 30 percent response rate overall, most reporting units did not achieve the desired 300 completes per reporting unit, because there were more ineligible sample members than anticipated. This occurred because QHP issuers were unable to exclude 2014 enrollees who had not re-enrolled for 2015. These disenrollees were deemed ineligible for the 2015 Beta Test survey in an effort to make eligibility rules for the QHP Enrollee Survey consistent with eligibility for the Medicare and HEDIS CAHPS health plan surveys. As a result, CMS plans to increase the initial sample size for the 2016 QHP Enrollee Survey to 1,300 enrollees from the 1,000 enrollees used for the Beta Test.

• QHP issuers were unable to provide information about QHP enrollee’s language preference, which meant that some sample members who received survey materials in English would have preferred them in Spanish or Chinese. CMS has worked with QHP issuers and believes that this information will be available for the 2016 QHP Enrollee Survey; however, CMS will continue to monitor this issue.

• Allowing more time for stakeholders to prepare for survey operations would be beneficial. CMS aims to publish the Technical Guidance for the QHP Enrollee Survey earlier to allow QHP issuers, HEDIS Compliance Auditors, and HHS-approved survey vendors additional time to better prepare for administering the survey.

• For the 2015 Beta Test, QHP issuers were confused about how to handle “off-Marketplace” products, because the definition of “off-Marketplace” was not clear. For the 2016 QHP Survey and beyond, CMS will provide further clarification regarding “off-Marketplace” products.