Change Request Memo

Justification of Non-material Change

September 10, 2014

(0938-1221)

We request approval of the beta test component of the QHP Enrollee Survey with the following revisions. We request approval ASAP because the QHP Enrollee Survey beta test procedures and materials must be prepared and provided to external, approved survey vendors by September 15, 2014, to enable January 2015 beta testing.

• The purpose of the QHP Enrollee Survey beta test originally was to test the distributed vendor system to be used for the QHP Enrollee Survey and to provide early quality metrics using a shorter questionnaire that had been revised based on the psychometric analysis of the field test data. The unanticipated lack of sampling frame data in CCIIO data systems delayed the psychometric testing of the QHP Enrollee Survey. The results of the psychometric analyses needed to shorten the questionnaire are now expected in April 2015, a full six months after we have to provide issuers and survey vendors with the survey technical specifications and questionnaires. The QHP Enrollee Survey beta test cannot be delayed because the survey results are needed for the beta test of the Quality Rating System. Our solution is to shorten the questionnaire to the minimum set of items that will not be cut as a result of the psychometric testing.

• The field test questionnaire includes five kinds of items: core CAHPS 5.0 Health Plan Survey items, supplemental CAHPS items required by the QRS, items required by the Affordable Care Act section 4302 standards for data collection, items covering topics requested by the public during the 30- and 60-day public comment periods, and items potentially relevant to this new population. The CAHPS 5.0 Health Plan Survey core items are required so that the survey can be identified as a CAHPS survey which is a requirement for health plan accreditation and QHP certification. The CAHPS 5.0 Health Plan Survey core items and supplemental items must be available to the QRS by July 2015 for the QRS to meet its beta test schedule. The remaining items are not vital to the beta test, because a decision about including or excluding them in the 2016 survey can be based on the psychometric analysis of the 2014 field test data. Deleting these items from the beta test questionnaire reduces the number of items from 107 to 76. These 76 items are the minimum needed to adequately test the QRS in late 2015 and can only be supplied by the QHP Enrollee Survey beta test.

• We have reduced the estimated number of QHP sampling units from 2,000 to 600. Two thousand was an early estimate received from CCIIO, but the decision to define sampling and reporting units as unique issuer-state-product type (HMO, PPO, POS, or EPO) combinations has greatly reduced the number. Based on current estimates of total number of QHP issuers, there will be approximately 600 QRS and QHP Enrollee Survey sampling and reporting units.

• For the 2015 beta test, the reporting unit has been defined at the level of product type (i.e., EPO, HMO, PPO, POS) offered by a QHP issuer through the Marketplace in a particular state. For example, Aetna’s HMOs offered through the Marketplace in Florida would be considered a single reporting unit. Depending on the way a QHP issuer packages its plan offerings, the reporting unit might include anywhere from a single QHP to dozens of QHPs spanning all categories of coverage (i.e., bronze, silver, gold, platinum, catastrophic). Stand-alone dental plans and child-only plans are not included. The reporting unit must have more than 500 enrollees to be included in the survey. These QHP issuer/state/product-type combinations were chosen as reporting units for the 2015 beta test in an attempt to balance the value of the information, enrollment patterns, and QHP issuer and respondent burden. CMS will explore data collection at a more granular level of QHP issuer coverage (e.g., HMO Bronze level) in the future based on the beta test results.

• Testing the distributed survey vendor system requires the original sample size of 300 completions per reporting unit and a starting sample of 1,000 per reporting unit, assuming a 30 percent completion rate. The QRS metrics to be reported to the public in 2016 are critical information that stakeholders have been anticipating. Consumers will use them to choose QHPs, Marketplaces will use them to monitor the performance of the QHPs they offer, and issuers will use them to invest in quality improvement activities and as the basis for provider compensation. It is essential to stress the system at all possible points to prepare for public reporting in 2016. This is accomplished best by conducting the beta test, in 2015, as a dry-run using the same specifications planned for national implementation. In this way all stakeholders responsible for making the system work will have a chance to test and refine their operations under the same conditions that they will face in national implementation. This includes operations conducted by survey vendors, the QHP Enrollee Survey contractor, CMS IT contractors working on QRS-related systems, and the QRS measure development contractor. Example stress points include:

  • QHP issuers will contract with HHS-approved survey vendors to collect and submit QHP Enrollee Survey data on their behalf. For the national implementation survey vendors must be prepared to use a standardized mixed mode of survey administration involving mail, telephone, and internet options in multiple languages. It is important to recognize that most survey vendors will be conducting the QHP Enrollee Survey under contract with multiple QHP issuer clients involving potentially many QRS sampling/reporting units. The number of their sampling/reporting units and sample size per unit will represent their overall survey load. In the beta test, survey vendors need to stress their systems under the same volume-related conditions as they will face in national implementation to ensure that they have adequate resources and systems to handle the survey load. For example, survey vendors will want to test the adequacy of their printing and mailing facilities to ensure they can produce survey materials in the fast and accurate manner that will be required for national implementation; they will want to ensure that their data receipt and survey management systems can keep up with the volume of responses that will be coming in; they will want to ensure that they have a sufficient number of high quality interviewers who can deal with required telephone follow-up with what may be hard-to-reach populations, including non-English speaking populations, in a short window of time; and they will want to ensure that their quality control systems and resources can keep up with the volume of survey activities to ensure the overall accuracy of the data. Based on the beta test, survey vendors will have actual experience under the conditions they will face in national implementation so they can make any needed adjustments to their systems and resources to ensure that they are ready to effectively conduct the survey in 2016.

  • Ten vendors applied and nine were approved to contract with issuers. To determine if nine vendors are sufficient to handle the total survey load expected in 2016, the beta test must expose them to the total survey load. (Note: the total survey load to be managed by the nine survey vendors is estimated to be 600 QHP reporting units, with a target of 300 completed questionnaires per reporting unit.)

  • Once survey vendors have collected the QHP Enrollee Survey data, they will submit it on behalf of their client QHP issuers into a web-based data submission application being developed by the QHP Enrollee Survey contractor. This is a new system being created for the QHP Enrollee Survey. The system needs to be able to accept large volumes of survey data from multiple survey vendors in a short window of time. The system will perform automatic validation checks on the files submitted by the survey vendors, and if errors are found, will notify the vendor of the errors and require that they be fixed and the file(s) re-submitted. Because this system is newly developed, it needs to be fully stressed in the beta test to be sure it can efficiently and accurately handle the large volume of survey responses it will be responsible for ingesting. Based on the beta test experience, the new system will be adjusted and modified as needed so it can efficiently deal with the volumes of data that will be submitted for national implementation in 2016.

  • The QHP Survey contractor will aggregate the data submitted by the multiple survey vendors and conduct needed analyses and calculate survey scores for the QRS reporting units. All survey-related data and scores then will be submitted into the Marketplace Quality Module (MQM) that CMS is developing for QRS processing as part of the HIOS system. The MQM is also a new system and needs to be fully stressed with the large volume of survey data that is similar to what will be submitted for national implementation to ensure that data ingestion and editing functions are sufficient.

  • Once the QHP Enrollee Survey data have been ingested into the MQM, they will be combined with QRS clinical data by the QRS measure developer to create the QRS hierarchical reporting system. A key part of the development process in the beta test is to assess the properties of the QRS measurement system. Thus it is important for the survey and clinical measures that form the basis of the QRS to have statistical properties similar to those expected in national implementation (e.g., to assess statistical precision, likely patterns of missing measures due to inadequate sample size, etc.). Based on this beta test analysis with comparable data to what would be expected in national implementation, the QRS can be refined to more effectively support public reporting in 2016.

• We have reduced the estimated number of survey vendors to reflect the ten vendors that have applied to be HHS-approved survey vendors for the QHP Enrollee Survey beta test in 2015. Although nine were approved, the tenth still has the opportunity to appeal, so we have assumed ten vendors as the basis of this burden estimate.

• We have reduced the total burden for the beta test component of the QHP Enrollee Survey to reflect the changes above (i.e., removal of survey questions for the beta test, decrease in the number of survey vendor respondents, decrease in the number of sampling and reporting units). See table below.

Exhibit A4. Estimated Burden Hours for QHP Enrollee Survey by Round

Source	Total Completes	Hours per response	Total burden hours
Psychometric Test Round 1, 2014
English	3,000	0.45	1,350
Spanish	600	0.45	270
Chinese	600	0.45	270
TOTAL Psychometric Test	4,200		1,890
Beta Test Round 2, 2015
Survey Respondents (600 QHPs)	180,000	0.32*	57,600
Survey Vendors	10	1	10
Total Beta Test	180,010		57,610
TOTAL QHP ENROLLEE SURVEY	184,210		59,500

*Represents 32% of an hour (or about 19 minutes). It is based on the common assumption that four CAHPS-like items typically can be completed in one minute. Under this assumption, it would take 19 minutes to complete the 76 item questionnaire (i.e., 76 / 4 = 19).  The 19 minutes were converted into a percentage of an hour for the burden table by dividing by 60 (i.e., 19 / 60 = .32).

2