Health Insurance Marketplace Consumer Experience Surveys: Qualified Health Plan Enrollee Experience Survey
Supporting
Statement—Part A
Supporting Statement for Information
Collection the Enrollee Satisfaction Survey and Marketplace Survey
Data Collection
July 13, 2015
Centers for Medicare & Medicaid Services
TABLE OF CONTENTS
A. Background 2
B. Justification 3
3. Use of Information Technology 3
9. Payments/Gifts to Respondents 6
12. Burden Estimates (Hours & Wages) 7
14. Cost to Federal Government 8
16. Publication/Tabulation Dates 9
The Affordable Care Act (ACA) authorized the creation of Health Insurance Marketplaces (Marketplaces) to help individuals and small employers shop for, select, and enroll in high quality, affordable private health plans beginning October 2013. By 2020, more than 27 million individuals and employees of small firms are expected to obtain their health insurance through Marketplaces. Section 1311(c)(4) of the ACA requires the Department of Health and Human Services (HHS) to develop an enrollee satisfaction survey system that assesses consumer experience with qualified health plans (QHPs) offered through a Marketplace. It also requires public display of enrollee satisfaction information by the Marketplace to allow individuals to easily compare enrollee satisfaction levels between comparable plans. In 2014, HHS established two surveys that assess consumer experience with the Marketplaces and the QHPs offered through the Marketplaces: the Health Insurance Marketplace Survey and the Qualified Health Plan (QHP) Enrollee Experience Survey. The main purpose of the Marketplace Survey is to evaluate consumers’ experiences with the Marketplace website, its telephone call centers and in person support. Results from the survey would be used by Marketplaces for internal quality improvement of their services. The main purpose of the QHP Enrollee Survey is to assess enrollee experience with their QHP around areas such as access to care, access to information, care coordination, cultural competence, doctor communication and plan administration. Under OMB Control Number 0938-1221, psychometric tests of both surveys were performed in 2014, which were followed by beta tests of revised surveys in 2015. At this time, the Centers for Medicare & Medicaid Services (CMS) is only seeking approval for the revisions to the information collection related to conducting the QHP Enrollee Experience Survey. The Health Insurance Marketplace Survey will be implemented via a separate revision to this ICR in the future. Thus, the remainder of this Supporting Statement addresses only the QHP Enrollee Survey.
QHP Enrollee Survey
As required by section 1311(c) (4) of the Affordable Care Act, CMS established the QHP Enrollee Survey with the goals of (1) informing consumer decision making in choosing a QHP, (2) providing actionable information that the QHP issuers can use to improve performance, and (3) providing information that state and federal regulators and Marketplaces need for effective oversight.
The development of the QHP Enrollee Survey included a comprehensive review of the literature and related surveys, consumer focus groups, stakeholder discussions, and input from a technical expert panel (TEP). As a result of this formative research, CMS decided that the QHP Enrollee Survey would use the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Health Plan 5.0 Adult Medicaid Survey as its core, with supplemental items drawn from the CAHPS Health Plan 5.0 Adult Supplemental Item Set, the CAHPS Health Plan 4.0 Supplemental Item Set, and the CAHPS 5.0 HEDIS Survey to provide the full set of information needed to evaluate QHP performance. Also, a few additional items were developed specifically for the QHP Enrollee Survey to fill CAHPS in the topics covered by existing CAHPS items. All selected items underwent two rounds of cognitive testing and revisions before being further evaluated with a Psychometric Test and a Beta Test. Appendix A provides the purposes, goals of the Psychometric and Beta test phases as well as describing how the results influenced the design of the first full implementation of the design, planned for 2016.
The questionnaire submitted for clearance is available in English, Spanish, and Traditional Chinese for use in a mixed-mode methodology that includes mail, telephone, and web survey modes.
The QHP Enrollee Survey will be conducted by HHS-approved survey vendors who meet minimum business requirements. A similar system is currently used for other CMS surveys, including Medicare CAHPS, Hospital CAHPS (HCAHPS), Home Health CAHPS (HHCAHPS), the CAHPS Survey for Accountable Care Organizations, and the Health Outcomes Survey. Under this model, all QHPs that are required to conduct the QHP Enrollee Survey must contract with a HHS-approved survey vendor to collect the data and submit it to CMS on the issuer’s behalf (45 CFR § 156.1125(a) ). CMS is responsible for approving and training vendors, providing technical assistance to vendors, overseeing vendors to ensure that they are following the data collection protocols, collecting and analyzing the data from vendors, and producing reports that QHP issuers can use for quality improvement. The Survey Vendor process was tested in the 2015 Beta Test
Section 1311(c)(4) of the Affordable Care Act (ACA) requires HHS to establish an enrollee satisfaction survey to be administered to members of each QHP offered through a Marketplace. The QHP Enrollee Survey meets the goal of measuring enrollee satisfaction with their health plan. Additionally, in accordance with section 1311(c)(4), the results of this survey will be available on each State Marketplace’s web portal, as well as the Federally-facilitated Marketplace’s web portal, in a manner that allows applicants for coverage to compare plans.
Beginning with the 2016 national implementation of the QHP Enrollee Survey, the data collected from the survey will be publicly reported on the Marketplace websites to aid consumers in choosing a QHP. The QHP Enrollee Survey data will also be used by QHP issuers to improve their performance and better tailor efforts to the QHP enrollee population. Additionally, the data will be used by HHS, State Based Marketplaces, and state insurance commissioners to aid in effective regulatory and oversight efforts. Finally, a de-identified, public-use dataset will be made available for use by health services researchers.
The current data collection protocol for the QHP Enrollee Survey includes the use of an online survey as well as the use of Computer Assisted Telephone Interviewing (CATI). Beginning with the 2016 QHP Enrollee Survey, the survey vendors will have the option of offering the web survey in Spanish and Chinese. CMS will continue to evaluate methods to increase the use of online surveys.
In addition to the actual data collection methodologies used by survey vendors, survey vendors are required to submit the final data files to CMS for analysis and scoring through a secure portal on the QHP Enrollee Survey website. This process ensures that the data files meet established specifications. Additionally, after analysis the survey data will be submitted into the Marketplace Quality Module (MQM) within CMS’ Health Insurance Oversight System (HIOS) for use in the Quality Rating System (QRS) and for public reporting.
There is no duplication of efforts. The QHP Enrollee Survey is the only survey being conducted by HHS to measure patient experiences with QHPs offered through the Marketplaces.
The survey population for the QHP Enrollee Survey includes individuals who enrolled in QHPs through an individual Marketplace, a Small Business Health Options Program (SHOP) Marketplace, or directly with the issuer. The sample frame is developed by issuers, few if any of whom are small businesses. CMS expects that this will not have an impact on small businesses. Some survey vendors who will apply to field the QHP Enrollee Survey will be small businesses, but conducting CAHPS surveys is their business and the decision to apply for approval as a vendor for the QHP Enrollee Survey is voluntary. Furthermore, the survey vendor application process imposes a minimal burden on any applicant, including small businesses. Thus, there is no reason to expect that the survey will burden small businesses; it offers them a business opportunity if they choose to apply for participation.
Annual data collection of the QHP Enrollee Survey is needed to meet the objectives of providing feedback to Marketplaces, issuers, and regulators for quality improvement; providing information for consumers’ choice; and to track performance.
There are no special circumstances associated with this data collection.
This is a revised collection to 0938-1221. As required by 5 CFR 1320.8(d), CMS solicited comments on the revisions to the QHP Enrollee Survey, through a Federal Register Notice which was posted on April 28, 2015.
There were six comments received from the following organizations:
1. Association for Community Affiliated Plans (ACAP)
2. America’s Health Insurance Plans (AHIP)
3. Anthem, Inc.
4. Blue Cross Blue Shield of Michigan
5. Neighborhood Health Plan of Rhode Island (Neighborhood)
6. Tufts Health Plan & Tufts Health Plan – Network Health
We appreciate the commenters’ feedback, have considered each comment and made applicable revisions to the survey. Although we have not made revisions to questions that are from the CAHPS Health Plan 5.0 questionnaire or from Healthcare Effectiveness Data and Information Set (HEDIS) because these are based on previously tested, national, health plan industry measures, we have addressed the comments received by providing feedback to the CAHPS Consortium as well as to the QRS measures steward, as appropriate. We intend to continually monitor the QHP Enrollee Survey as well as the QRS and make necessary adjustments to ensure that the measures remain consistent with the intended goals and principles of the QHP Enrollee Survey and QRS. As advancements in consumer experience surveys and health plan quality measurement and reporting are made, we will consider ways in which the survey may evolve.
A top-line summary of the comments and how we have addressed them is below.
Revise cover letter. Several comments suggested revisions to the cover letter. There was concern about the instructions for those sample members who switched health plans and the introduction to the intent of the survey. CMS has made wording updates to clarify both points.
Cognitive test all new survey questions and make results publically available. Many groups highlighted the importance of cognitive testing of questions, especially those that are new to CAHPS. The core CAHPS Health Plan questions were cognitively tested with the Marketplace population before the Psychometric Test and have been repeatedly tested by the CAHPS Consortium over the last 20 years. In addition, we have conducted three rounds of cognitive testing of all new items and all items were also included in the psychometric testing after the field test.
Revise question wording. There were multiple recommendations for revisions to item wording for core CAHPS questions that have been previously validated in cognitive testing and field testing. We are not making changes to items from the CAHPS Health Plan core questionnaire in order to allow CMS, policy makers, and health plans to make direct comparisons between their QHP populations and other product-line populations. Furthermore, changes to the CAHPS Health Plan 5.0 questionnaire would invalidate our use of the CAHPS trademark; use of an approved CAHPS-trademarked survey is required by the ACA. Comments regarding items contained in the CAHPS Health Plan questionnaire have been shared with members for the CAHPS Consortium, who will consider them when deciding about future revisions.
Allow the option for additional questions. Stakeholders advocated for the opportunity to add a small number of custom questions to assist in cultural and linguistic reporting. CMS will consider these questions and other revisions in the future; however because of the rounds of psychometric and beta testing already done we will not include additional questions at this time to ensure that we are able to implement the 2016 QHP Enrollee Survey in appropriate timeframes.
Revise provider questions to assess quality of provider network. Commenters recommended that questions about a personal doctor and provider communication be replaced with questions about the plan’s provider network. CMS is not making these changes to the current QHP Enrollee Survey because these items are all standard CAHPS Health Plan questionnaire items. CMS may consider questions about provider networks in future QHP surveys. Comments regarding items contained in the CAHPS Health Plan questionnaire have been shared with members for the CAHPS Consortium.
Revise the response scales. Some commenters recommended changing the question “Would you recommend this plan to your friends and family?” to “How likely is it you would recommend your health plan to a friend or family member?” with a scale from 0 to 10 to allow for calculation of a Net Promoter Score. CMS has made this change because this item is not part of the CAHPS Health Plan 5.0 Survey. In addition, some commenters objected to the use of the Never/Sometimes/Usually/Always (NSUA) scale and recommended a yes/no scale instead. CMS did not make this change, because the response scales are standardized elements of the CAHPS Health Plan 5.0 Survey items.
Revision of invasive questions. One stakeholder group noted that prevention questions about smoking, aspirin use, and cholesterol may be too invasive and respondents may not answer honestly. These questions are National Committee for Quality Assurance (NCQA) HEDIS items and have been previously tested. They are also required for the QRS measures and so must be included in this implementation. Many surveys have asked these and similar questions for many years and they are not considered by survey experts in general to be unusually invasive or sensitive when used in a voluntary survey.
Survey length. Stakeholders expressed their concern about the length of the survey; in particular they recommended shortening the “About You” section. CMS recognizes that the survey is relatively long, but there are many factors that affect response rates. Previous research has shown that the effect of additional questions up to even 95 questions had minimal effect on response rates. Additionally, many of the items included in the “About You” section are required for the aspirin use, tobacco cessation, and flu shot measures that are included in the Quality Rating System. Furthermore, other CAHPS surveys are of similar length. For example, the ACO CAHPS questionnaire is currently 80 questions.
Clarify the survey methodology and ensure participants do not get both the Marketplace and the QHP Enrollee Survey. There was a recommendation to ensure the same person is not selected for both the Marketplace and the QHP surveys. The issue is moot for 2016, because the Marketplace Survey will not be conducted. If the Marketplace survey is administered in the future, CMS will address overlapping samples at that time. Nevertheless, there is no way to ensure that the same persons are not selected for both surveys, because the samples are drawn by different organizations and no organization has access to the identities of sample members selected by other organizations to protect privacy.
Metal-level analysis. Stakeholders recommended the inclusion of analysis by metal level. CMS does have this information and will include it in appropriate analyses.
Assess if experience differences exist for those who were previously uninsured. Stakeholders suggested that CMS assess whether experience differs across previous insurance status. CMS plans to perform this analysis using a question on the survey that asks whether the enrollee had insurance during the previous year.
Paperless survey option. It was recommended to include a paperless survey option. CMS included an internet survey option in the 2014 Psychometric Test and 2015 Beta Test. CMS plans to continue offering this option and will expand internet surveys to include Spanish in the 2016 Implementation.
CMS is working with a variety of outside organizations and individuals to aid in the development and implementation of the QHP Enrollee Survey. Chief among these organizations is the American Institutes of Research (AIR) and the National Committee for Quality Assurance (NCQA).. In addition, a Technical Expert Panel composed of consumer advocates, health plan representatives, Marketplace administrators, survey design experts, state regulators, and providers provides ongoing feedback on technical issues. The panel meets approximately three times a year to provide guidance.
No payments or gifts will be made to any respondents.
Individual survey respondents will be told the purposes for which the information is collected and that, in accordance with section 934(c) of the Public Health Service Act, 42 USC 299c-3(c), any identifiable information about them will not be used or disclosed for any purpose beyond conducting the survey. The confidentiality of individual’s replies is further assured under 5 U.S.C.552 (Freedom of Information Act), 5 U.S.C.552a (Privacy Act of 1974), and OMB Circular No.A-130.
There are no sensitive questions associated with this information collection.
Estimated burden hours for the QHP Enrollee Survey in 2016-2018 are presented in Exhibit A2 and are based on the following assumptions and definitions..
Units. The sampling/reporting unit has been defined at the level of product type (i.e., Exclusive Provider Organization [EPO], Health Maintenance Organization [HMO], Preferred Provider Organization [PPO], Point of Service [POS]) offered by a QHP issuer through the Marketplace in a particular state. For example, XYZ issuer’s HMOs offered through the Marketplace in Florida would be considered a single sampling unit. Depending on the way a QHP issuer packages its plan offerings, the sampling unit might include anywhere from a single QHP to many QHPs spanning all categories of coverage (i.e., bronze, silver, gold, platinum, catastrophic). QHP issuers will create a sample frame for each product type they offer through the Marketplace within a particular state.
CMS is exploring collecting data at a more granular level of QHP issuer coverage (e.g., HMO bronze level) in the future, keeping in mind the need to balance the value of this information for consumers with enrollment volume and QHP issuer data collection, validation, and reporting burden. CMS will revise this Information Collection Request if changes are implemented.
Respondents per unit. Based on the results of the 2014 Psychometric Test, CMS plans to collect 300 responses per reporting unit. As this survey program continues, CMS will explore whether the number of responses can be reduced.
Total respondents. The total number of respondents equals the product of the completed surveys per sampling unit and the current estimate of the number of QHP sampling units.
Hours per response. Based on testing of the QHP Enrollee Survey it takes 20 minutes to complete.
Survey vendors. Survey vendors who want to participate in collecting QHP Enrollee Survey data must complete a Survey Vendor Participation Form. CMS anticipates that approximately 15 survey vendors will apply to field the QHP Enrollee Survey annually. The Survey Vendor Participation Form is designed to be completed in 90 minutes.
Exhibit A2. Estimated Burden Hours for 2016-2018 National Implementation of QHP Enrollee Survey
Source |
Num. of Units |
Completes per Unit |
Total Sample |
Burden Hours |
Total burden hours |
2016 Survey Respondents |
400 |
300 |
120,000 |
0.33 |
39,600 |
2016 Survey Vendors |
15 |
1 |
15 |
1.5 |
22.5 |
2016 TOTAL |
415 |
|
|
|
39,623 |
2017 TOTAL |
415 |
|
|
|
39,623 |
2018 TOTAL |
415 |
|
|
|
39,623 |
3-year TOTAL |
1,245 |
|
|
|
118,869 |
In 2016, the total annual burden hours for the 2016 QHP Enrollee Survey are estimated to be 39,623 hours. Because only minimal adjustments to the questionnaire are expected for 2017 and 2018, we estimate an annual burden of 39,623 hours for 2017 and 2018. We estimate a total burden of 118,869 hours over three years.
The Bureau of Labor Statistics reported the average hourly wage for civilian workers in the United States was $24.78 as of February 2015. To estimate the burden costs for survey vendors, CMS used the average hourly wage for employees in the business and professional services sector as of February 2015. See exhibit A3 for estimated burden costs.
Exhibit A3. Estimated Burden Costs
Source |
Number of Respondents |
Total Burden Hours |
Average Hourly Wage Rate |
Total Cost Burden |
2016 Survey Respondents |
120,000 |
39,600 |
$24.78 |
$981,288.00 |
2016 Survey Vendors |
15 |
22.5 |
$29.78 |
$670.05 |
2016 TOTAL |
120,015 |
39,623 |
|
$981,958.05 |
2017 TOTAL |
120,015 |
39,623 |
|
$981,958.05 |
2018 TOTAL |
120,015 |
39,623 |
|
$981,958.05 |
3-Year TOTAL |
360,045 |
118,869 |
|
2,945,874.10 |
There are no direct capital costs to respondents other than their time to participate in the survey.
The only cost to the Government of these data collections that would not otherwise have been incurred is the cost of the American Institutes for Research (AIR) contract. The portion of the AIR contract attributable to the QHP Enrollee Survey is approximately $1.7 million for the 2016 national implementation and $1.7 million for the 2017 national implementation. We expect the 2018 national implementation to also be approximately $1.7 million as well for a 3-year total of $5.1 million. This cost includes soliciting and approving survey vendors, developing quality assurance guidelines and technical specifications for survey vendors, providing technical assistance and training to survey vendors, conducting oversight of approved survey vendors, providing technical assistance to QHP issuers, scoring and analyzing the survey data, and development of final reports for QHP issuers.
There is a reduction of burden, of 3150 hours, because the 2014 psychometric testing of the surveys has ended. The increased annual burden of 39,623 hours (or 118,869 hours over three years) is being requested for the future administrations of the QHP Enrollee Survey and due to the estimated increased number of QHPs offered through the Marketplaces. The questionnaire length has been reduced by 22 items since the original approval. Therefore, the net increase of burden hours is 36,473.
Publication of the QHP Enrollee Survey results will occur in the fall of 2016, following the data collection. Reporting of the survey results will include distribution of survey reports for each sampling unit to QHP issuers, summary reports to Marketplaces, and the Office of Personnel Management (OPM), as well as public reporting of survey results through Marketplace websites. CMS also publishes updates about the survey through its Marketplace Quality Initiatives webpage1 and through the QHP Enrollee Survey project webpage.2
All reporting websites under CMS’ control will provide Marketplace consumers with the overall response rate and the minimum and maximum response rates obtained by reporting units nationwide. This information will also include a statement of findings from the nonresponse bias analysis and CMS’ assessment of the potential implications of those findings for use of the response rates by consumers in choosing a QHP. CMS will report back to OMB before posting results publicly regarding how it intends to communicate these concepts to consumers within the context of the Quality Rating System (QRS).
Additionally, CMS anticipates annually releasing a de-identified public use microdata sample (PUMS) file through an HHS website in the fall of the year following the year in which data were collected (e.g. the 2016 QHP Enrollee Survey PUMS File would become available in fall 2017).
The expiration date will be displayed.
There are no exceptions to the certification statement identified in Item 19, "Certification for Paperwork Reduction Act Submissions," of OMB Form 83-I.
1 https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/QualityInitiativesGenInfo/Health-Insurance-Marketplace-Quality-Initiatives.html
2 https://qhpcahps.cms.gov/
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |