Health Insurance Marketplace Consumer Experience Surveys: Qualified Health Plan Enrollee Experience Survey
Supporting
Statement—Part B
Collections of Information Employing
Statistical Methods
July 13, 2015
Centers for Medicare & Medicaid Services
TABLE OF CONTENTS
Section Page
1. Potential Respondent Universe and Sampling Methods 1
1.2 Sample Frame & Respondent Universe 1
2. Information Collection Procedures 3
Exhibit B-1. Data Collection Protocol for 2016 QHP Enrollee Survey 4
3. Methods to Maximize Response Rates and Address Non-Response Bias 5
3.2 Evaluating Non-Response Bias 5
4. Tests of Procedures 6
5. Statistical Consultants 6
This supporting statement includes information in support of the Qualified Health Plan Enrollee Experience Survey (“QHP Enrollee Survey”).
As outlined in 45 CFR § 156.1125(b), Qualified Health Plan (QHP) issuers are required to conduct the QHP Enrollee Survey at the level specified by HHS for all QHPs that had more than 500 enrollees as of the previous year. CMS has established the sampling/reporting unit as the product type (i.e., Exclusive Provider Organization [EPO], Health Maintenance Organization [HMO], Preferred Provider Organization [PPO], Point of Service [POS]) offered by a QHP issuer through the Marketplace in a particular state. For example, XYZ issuer’s HMOs offered through the Marketplace in Florida would be considered a single sampling unit. Depending on the way a QHP issuer packages its plan offerings, the sampling unit might include anywhere from a single QHP to many QHPs spanning all categories of coverage (i.e., bronze, silver, gold, platinum, catastrophic). QHP issuers will create a sample frame for each product type they offer through the Health Insurance Marketplace within a particular state. Child-only QHPs as well as Standalone Dental Plans (SADPs) are excluded from the QHP Enrollee Survey at this time.
The same structure was used to define sampling units for the 2015 Beta Test QHP Enrollee Survey and resulted in 15,317 QHPs being rolled up to 517 reporting units. Of the 517 sampling units, 298 sampling units conducted the 2015 Beta Test QHP Enrollee Survey. The remaining 219 sampling units did not conduct the survey for a variety of reasons including not meeting the minimum enrollment threshold or withdrawing the QHPs from the Marketplace. The exact number of sampling units is not currently available; however, a January 8, 2015 report by the HHS Office of the Assistant Secretary for Planning and Evaluation (ASPE) reported a 25 percent increase in the number of issuers offering coverage through the Health Insurance Marketplace. Based on this report and the higher enrollment numbers during the 2014-2015 Open Enrollment Period, CMS estimates that 400 sampling units will conduct the 2016 QHP Enrollee Survey.
The sample frame for each sample unit is generated by the QHP issuer and then provided to the issuer’s CMS-approved survey vendor who is responsible for drawing a simple random sample. Eligible sample units for the 2016 QHP Enrollee Survey, are those that meet both of the following criteria:
The sample unit was offered through a Marketplace in 2016 and
The sample unit had more than 500 enrollees regardless of age on July 1, 2015.
Issuers must collect and report survey data for all eligible sample units to comply with the regulation.
Eligible sample units are comprised of eligible QHPs. Individual QHPs must also meet eligibility criteria to be included in a sample unit. An eligible QHP must meet all three of the following criteria:
An eligible QHP has a 14-digit Standard Component ID number (SCID) assigned in the Health Insurance Oversight System (HIOS) database and
An eligible QHP is offered on an Individual Marketplace and/or a SHOP Marketplace and
Provides family and/or adult-only medical coverage, regardless of whether or not it also covers dental care or vision care.
By this definition, an eligible QHP is synonymous with an eligible SCID in the HIOS system. QHPs with all HIOS 2-digit variant suffixes of eligible 14-digit SCIDs are eligible to be included in the sample unit. Once the eligible SCIDs are identified, issuers group them by product type (i.e., HMO, PPO, POS, or EPO) within each state. Each state-issuer-product type combination is a potential sample unit. Eligible sample units are those potential sample units that meet the two criteria listed above (i.e., offered through a Marketplace in 2016 and more than 500 enrollees on July 1, 2015).
The sample frame for the survey includes all enrollees of each eligible sample unit who meet the following three criteria:
Eligible sample frame members must be 18 years or older on December 31, 2015, and
Eligible sample frame members must have been enrolled in the eligible QHP from July 1, 2015 through Decmber 31, 2015 with no more than one 31-day break in enrollment during those 6 months, and
Eligible sample frame members must have coverage for primary health care through the eligible QHP.
The sample frame is audited by a NCQA Licensed HEDIS®1 Compliance Organization (NCQA Certified HEDIS Compliance Auditor) to ensure that the sample frame conforms to all established specifications. A random sample is drawn from each audited eligible sample frame by the issuer’s survey vendor and inferences are made to the sample frame.
Beginning with the 2016 QHP Enrollee Survey, CMS is proposing to sample 1,300 eligible enrollees from each sample frame, which we estimate will result in approximately 300 eligible responses per sample unit. This sample size reflects a standard recommendation in CAHPS for achieving an interunit reliability of at least 0.70 for all consumer-reported composites across all sample units. The initial sample size used for the 2015 QHP Enrollee Survey was 1,000 enrollees per sample unit; however, based on the results of the 2015 beta test, CMS and its contractor have determined that a higher sample size is necessary in order to achieve the desired 300 responses per reporting unit because of a lower than anticipated response rate and a higher ineligibity rate. The higher rate of ineligibes stems from the fact that QHP issuers need to draw their sample frame before the end of the Open Enrollment Period. Unfortunately, the timing for drawing the sample frame cannot be delayed in order to ensure that the survey results are ready in time for display during the subsequent year’s Open Enrollment Period.
Additionally, the desired number of completes, 300, is consistent with the number recommended by the CAHPS Consortium for the CAHPS Health Plan 5.0 (Adult Medicaid) Survey, which is used as the core of the QHP Enrollee Survey questionnaire.
The QHP Enrollee Survey will be conducted by HHS-approved survey vendors who meet minimum business requirements. A similar system is currently used for other CMS surveys, including Medicare CAHPS, Hospital CAHPS (H-CAHPS), Home Health CAHPS (HH-CAHPS), the CAHPS Survey for Accountable Care Organizations, and the Health Outcomes Survey. Under this model, all issuers that are required to conduct the QHP Enrollee Survey must contract with a HHS-approved survey vendor to collect the data and submit it to CMS on their behalf (45 CFR § 156.1125(a) ). CMS is responsible for approving and training vendors, providing technical assistance to vendors, and overseeing vendors to ensure that they are following the data collection protocols.
The data collection protocol for the 2016 QHP Enrollee Survey will use a mixed-mode methodology that combines web, mail, and telephone surveys. First, all sampled enrollees will receive a pre-notification letter informing them that they have been sampled for the survey and providing them with information about the survey and how the data collected will be used. The pre-notification letter also provides information on completing the survey online, including the website URL and the sample member’s user ID and password, which are unique to each sample member. One week after the pre-notification letter, individuals will receive a mail questionnaire, which is then followed-up by a reminder letter one week after the questionnaire is mailed. Three weeks after the first questionnaire is mailed, nonrespondents will receive a second questionnaire. Finally, three weeks after the second questionnaire is mailed, survey vendors will initiate telephone follow-up calls making six attempts on varying days of the week and at differing times of the day.
The data collection protocol for the 2016 Survey differs from the data collection protocol used for the 2015 QHP Beta Test in two ways: (1) the amount of time between the pre-notification letter and the first survey mailing will be one week in 2016 instead of two weeks used in 2015 and (2) the reminder contact in between the first and second mailings in 2016 will be a letter instead of the postcard used in 2015. These minor modifications have been made to ensure that survey data is ready for public reporting during the 2016-2017 Open Enrollment Period.
Task |
Date |
Survey vendors sample enrollees according to sampling protocols. |
January 2016 –February 2016 |
Mail prenotification letter to sampled enrollees.
|
Day 0 |
Customer support phone center opens (toll-free phone number required). |
Day 1 |
Mail first questionnaire with survey cover letter to nonrespondents 1 week (7 calendar days) after the prenotification letter is mailed. |
Day 7 |
|
Day 17 |
Mail second questionnaire with survey cover letter to nonrespondents 3 weeks (21 calendar days) after the first questionnaire is mailed. |
Day 28 |
Initiate telephone follow-up contact for nonrespondents 3 weeks (21 calendar days) after the second questionnaire is mailed.
|
Days 49–70 |
End data collection activities. |
Day 71 |
All survey estimates generated from the QHP Enrollee Survey are weighted to adjust for the unequal probability of selection across sample units. Weights are generated for each case by multiplying the inverse of the enrollee’s probability of selection by the enrollee’s probability of response.
Additionally, case-mix adjustment is used when comparing sample units to the overall national benchmark scores. Case-mix adjustment accounts for the demographics of a particular sample unit. Case-mix variables include education, age, and self-reported health status, mental health rating, receiving help completing the survey, having a chronic condition, the language in which the survey is completed, and survey mode.
Every effort will be made to maximize the response rate, while retaining the voluntary nature of the effort. The mixed-mode methodology of the QHP Enrollee Survey provides sampled individuals with numerous methods for completing the survey across a 71 day time period. This methodology has been developed using best practices within the survey research field, including the Tailored Design Method.2
the 2014 Psychometric Test of the QHP Enrollee Survey achieved a 37.2 percent response rate while the 2015 Beta Test achieved a response rate of 30.9 percent. Response rates for the QHP Enrollee Survey are calculated using the Response Rate 3 (RR3) formula established by the American Association for Public Opinion Research (AAPOR). A response is counted as a completed interview if the respondent completes 50 percent of the questions that are applicable to all respondents, excluding the questions included in the “About You” section of the survey.
Given that CMS expects that the response rate will be below 80 percent, CMS plans to conduct a nonresponse bias analysis to determine whether nonrespondents systematically differ from respondents based on their experience with their health plan. In the 2014 Psychometric Test, CMS found that older enrollees and those who received the Advance Premium Tax Credit (APTC) were significantly more likely to respond.
Thus far, the response rates discussed have been at the unit level, where respondents either completed or did not complete the survey. There is also item-level nonresponse where a respondent answers some, but not all of the questions they are eligible to answer in the survey. Although highly unlikely, if the item response rate is less than 70% for any survey questions, CMS will conduct an item nonresponse analysis similar to that discussed above for unit nonresponse as required by Guideline 3.2.10 of the Office of Management and Budget’s Standards and Guideline for Statistical Surveys.
All reporting websites under CMS’ control will provide Marketplace consumers with the overall response rate and the minimum and maximum response rates obtained by reporting units nationwide. This information will also include a statement of findings from the nonresponse bias analysis and CMS’ assessment of the potential implications of those findings for use of the response rates by consumers in choosing a QHP. CMS will report back to OMB before posting results publicly regarding how it intends to communicate these concepts to consumers within the context of the Quality Rating System (QRS).
The QHP Enrollee Survey uses the CAHPS® Health Plan 5.0 Survey, which was developed and is maintained by the Agency for Healthcare Research & Quality (AHRQ), as the core of the QHP Enrollee Survey. Additional items included in the survey are from other CAHPS surveys and supplemental item sets. These items have undergone extensive testing and practical use by AHRQ, CMS, NCQA, and other users since they were first developed nearly 20 years ago. In 2014, CMS conducted a Psychometric Test to evaluate the psychometric properties of the QHP Enrollee Survey in the Marketplace population. This test resulted in numerous questions being dropped from and revised for the questionnaire for the 2015 Beta Test. More specifically, in 2014, CMS conducted the Psychometric Test of the QHP Enrollee Survey with 30 sampling units (defined by state, issuer, and product type (i.e., HMO, PPO, POS, or EPO)). The sample unit definition matches the reporting unit definition CMS developed for the Quality Reporting System (QRS). Because the Psychometric Test took place in the second half of 2014, the results were not available in time to influence the content of the questionnaire for the 2015 Beta Test, which had to be finalized for distribution to the Beta Test survey vendors by November 2014. As a result, CMS, in consultation with OMB, reduced the questionnaire to include only the survey items that were part of the CAHPS Health Plan 5.0 core items and the items that were needed for the QRS. Thus, the Beta Test questionnaire contained 31 fewer items than the Psychometric Test questionnaire. When the results of the Psychometric Test became available in early 2015, CMS determined that nine of the questions that had been removed for the Beta Test were vital for understanding enrollee experiences with their QHPs. These questions have been returned to the questionnaire for the 2016 and subsequent National Implementation Surveys for which clearance is now being requested. These restored items address enrollees’ experiences with out-of-pocket costs for covered services, health insurance literacy, and health insurance coverage during the previous year, and whether respondents would recommend their QHPs to their friends and family.
The 2015 Beta Test was mainly intended to test the survey vendor system but also provided additional information about the questionnaire items and data collection procedures. As a result of the Psychometric and Beta Tests, CMS has identified the changes in the questionnaire items, the data collection procedures, and the sampling specifications that we propose to use in the 2016 and future national implementations for public reporting (See Appendix).
This sampling and statistical plan was prepared and reviewed by staff of CMS and by the American Institutes for Research. The primary statistical design was provided by Chris Evensen, MS, of the American Institutes for Research at (919) 918-2310; Michael P. Cohen, PhD, of the American Institutes for Research at (202) 403-6453; Steven Garfinkel, PhD, of the American Institutes for Research at (919) 918-2306, and HarmoniJoie Noel, PhD, of the American Institutes for Research at (202) 403-5779.
1 Healthcare Effectiveness Data and Information Set (HEDIS®) is a registered trademark of the National Committee for Quality Assurance (NCQA).
2 Dillman, D., Smyth, J., & Christian, L. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Wiley.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |