CMS-10488 - ESS OMB Supporting Statement Part A_121614 508 clean
CMS-10488 - ESS OMB Supporting Statement Part A_121614 508 clean.docx
Health Insurance Marketplace Consumer Experience Surveys: Enrollee Satisfaction Survey and Marketplace Survey Data Collection (CMS-10488)
OMB: 0938-1221
Health Insurance Marketplace Consumer Experience Surveys: Enrollee Satisfaction Survey and Marketplace Survey
Supporting
Statement—Part A
Supporting Statement for Information
Collection the Enrollee Satisfaction Survey and Marketplace Survey
Data Collection: Testing and Early Feedback Phases
December 16, 2014
Centers for Medicare & Medicaid Services
TABLE OF CONTENTS
Section Page
A. Background 3
Exhibit A1. Surveys Included in this Application and Timing 5
Previous Testing of CAHPS Surveys on which the Marketplace and QHP Enrollee Surveys are Based 6
Cognitive Testing for the Marketplace and QHP Surveys 8
Decision-making Criteria Related to Methodological Goals 10
B. Justification 14
3. Use of Information Technology 16
6. Less Frequent Collection 16
Exhibit A2: Question Topics added to surveys based on public comments 21
9. Payments/Gifts to Respondents 22
12. Burden Estimates (Hours & Wages) 23
14. Cost to Federal Government 26
16. Publication/Tabulation Dates 26
18. Certification Statement 26
Appendix A: Draft Minimum Business Requirements for 2014 QHP Enrollee Survey Vendors 27
A. Background
The Affordable Care Act (ACA) authorized the creation of Health Insurance Marketplaces (Marketplaces) to help individuals and small employers shop for, select, and enroll in high quality, affordable private health plans beginning October 2013. By 2020, more than 27 million individuals and employees of small firms are expected to obtain their health insurance through Marketplaces. Section 1311(c)(4) of the ACA requires the Department of Health and Human Services (HHS) to develop an enrollee satisfaction survey system that assesses consumer experience with qualified health plans (QHPs) offered through a Marketplace. It also requires public display of enrollee satisfaction information by the Marketplace to allow individuals to easily compare enrollee satisfaction levels between comparable plans. HHS intends to establish two surveys that assess consumer experience with the Marketplaces and the QHPs offered through the Marketplaces.
Project Goals
The purpose of this project is to develop, test, and implement two assessment surveys:
A Health Insurance Marketplace Survey (Marketplace Survey) - Survey of health insurance marketplace consumers
A Qualified Health Plan Enrollee Experience Survey (QHP Enrollee Survey) - Survey of adult enrollees in Qualified Health Plans (QHPs).
Marketplace Survey
The Marketplace Survey will provide (1) actionable information that the Marketplaces can use to improve performance, (2) information that CMS and state regulatory organizations can use for quality improvement and oversight, and (3) a longitudinal database for future Marketplace research. This data can only be collected through a representative sample of the users of each Marketplace and is a vital oversight tool for the Marketplaces.
QHP Enrollee Survey
CMS intends to establish the QHP Enrollee Survey to (1) help consumers choose among competing health plans, (2) provide actionable information that the QHPs can use to improve performance, (3) provide information that Marketplaces, regulatory and accreditation organizations can use to regulate and accredit plans, and (4) provide a longitudinal database for consumer research. CMS is required by section 1311(c) (4) of the Affordable Care Act to provide information about Qualified Health Plans’ to consumers and a random sample of QHP members is the best method to collect this information.
The project goals for the initial psychometric testing and early feedback phases of these surveys are summarized in Exhibit A1. CMS is seeking immediate clearance for the Marketplace beta test round, so that Marketplace Survey beta testing operations can begin on February 16, 2015. CMS has previously obtained clearance for the psychometric testing of both surveys and the QHP Enrollee Survey beta test. The survey instruments are shorter for second phase and all survey changes are outlined in the change request form.
Additionally, we plan to request clearance for two subsequent rounds of full-scale national implementation with public reporting of scores for each survey in 2016 and 2017, but we will not request clearance for those rounds until the testing in 2014 and the early feedback phase in 2015 have been completed. A summary of findings from the testing rounds will be included when requesting clearance for the additional two rounds of national implementation with public reporting in 2016 and 2017. Given the statutory requirements for these surveys included in section 1311(c) (4) of the Affordable Care Act, CMS plans to conduct both surveys on an annual basis beyond 2017.
Exhibit A1. Surveys Included in this Application, Goals, and Timing
Round |
|
Marketplace Survey |
|
QHP Enrollee Survey |
|
Timing |
Goals |
Timing |
Goals |
Psychometric Test Phase |
April - August 2014 |
|
August - December 2014 |
|
FeedbaBeta Beta Test Phase |
Feb- July JulyJuly – Apr 2015 |
|
Jan – Apr 2015 |
|
Survey Development
Previous Testing of CAHPS Surveys on which the Marketplace and QHP Enrollee Surveys are Based
A comprehensive review of the literature and related surveys, focus groups, stakeholder discussions, and input from the technical expert panel (TEP) informed the development of the survey. It is important that all Marketplaces and QHPs offered through the Marketplaces be assessed using a reliable and valid survey, administered and scored according to standards developed and monitored by independent organizations. CMS is using the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Principles (About CAHPS) for developing the surveys (see background information below).
The Marketplace Survey was developed using CAHPS Principles, CAHPS question structure, CAHPS response sets, and, in a few cases, CAHPS items. However, there was no precedent for Marketplaces in the U.S. health care system (other than in Massachusetts), so there was no existing consumer assessment survey of entities similar to the Marketplaces. Thus, almost all the Marketplace Survey questions were developed for this project; however, some of the survey items were drawn from existing surveys including: CAHPS Health Plan 5.0 Adult Medicaid core; CAHPS Health Plan 5.0, Adult Supplemental Item Set (these are new CAHPS questions that are not in public documentation yet); CAHPS Clinician & Group 2.0, Adult Supplemental; Hospital CAHPS Adult; HHS Office of Minority Health ACA section 4302 Data Collection Standards; American Community Survey (ACS) – Person section; 2010 National HIV Behavioral Surveillance System; and 2014 Medicare Provider Satisfaction Survey (Items for ACOs Participating in Medicare Initiatives).
CMS tested the Marketplace Survey in 2014 to understand its psychometric and functional properties, develop composite measures and case mix adjusters, and reduce the number of items in the questionnaire. In addition to the beta test goals, the administration of the Marketplace Survey to be conducted in 2015 will establish a baseline measure of Marketplace performance and provide early feedback to States and QHPs. The goals for the 2014 psychometric testing and 2015 beta test/early feedback phases of this ICR are listed above in Exhibit A1.
The QHPs are very similar to the health plans that are assessed by the original CAHPS Health Plan Survey, thus CMS adopted the current version, the CAHPS 5.0 Health Plan Survey as the core of the QHP Enrollee Survey. Supplemental item sets from other CAHPS surveys, many of which have never been formally tested or used were added to the draft of the questionnaire that were used in the psychometric testing phase in order to address specific issues found to be important to consumers in formative research. Finally, a few new questions were added to the draft survey instrument when existing questions were inadequate to provide the information sought.
More specifically, given its reputation as the national standard for measuring consumer experiences with health plans and endorsement from the NQF, the CAHPS Health Plan 5.0 Medicaid Survey served as the starting point for the QHP Enrollee Survey. By maintaining consistency with the CAHPS Health Plan 5.0 Survey, CMS and the public will be able to make comparisons between consumers’ experiences with QHPs purchased through the Health Insurance Marketplaces and consumers’ experiences with private and Medicaid health plans. The adult survey includes all of the CAHPS® Health Plan 5.0 (Adult Medicaid) items, and adds several new domains and items based on a comprehensive review of the literature and related surveys, focus groups, stakeholder discussions, and input from the TEP. These additional topics included shared decision making between patients and providers, mental health services, habilitative services, care coordination, information provided by the health plan, and cost of medical care. In order to add these topics for the survey, questions from other CAHPS surveys were used including the Health Plan 5.0 Adult Supplemental item set, the Adult Supplemental Items from the CAHPS Clinician and Group 2.0 Survey, and the Patient Centered Medical Home (PCMH) Supplement from the CAHPS Clinician and Group 2.0 Survey. Additionally, a few of the survey questions were drafted by CMS consultants after determining that no suitable items were available from other CAHPS surveys. The questionnaires submitted for review include indicators to identify the source of each survey question.
The items that were added to the survey were drawn from: CAHPS Health Plan 5.0, Adult Medicaid; CAHPS Health Plan 5.0 HEDIS, Adult Medicaid; CAHPS Health Plan 5.0 HEDIS, Adult Commercial; CAHPS Health Plan 4.0, Adult Supplemental; CAHPS Health Plan 5.0, Adult Supplemental (these are new CAHPS questions that are not in public documentation yet); CAHPS Clinician & Group 2.0, Adult Supplemental; CAHPS Clinician & Group 2.0, Adult Supplemental; CAHPS Clinician & Group 2.0, Patient-Centered Medical Items; HHS Office of Minority Health ACA Section 4302 Data Collection Standards; American Community Survey (ACS) – Person Section; 2010 National HIV Behavioral Surveillance System; 2014 Medicare Provider Satisfaction Survey – Items for ACOs Participating in Medicare Initiatives. The goals for the 2014 test phase and 2015 beta test phase are listed above in Exhibit A1.
Background Information on CAHPS: The various CAHPS® surveys currently in the public domain ask consumers and patients to report on their experiences with health plans and health care providers. These surveys cover topics that are important to consumers and focus on aspects of quality that consumers are best qualified to assess, such as the communication skills of providers and ease of access to health care services. Rather than asking if consumers are satisfied, CAHPS surveys ask if the consumer observed certain provider behaviors that are believed by consumers and generally acknowledged by providers to represent good quality care.
Over the past 19 years, the CAHPS Consortium has established a set of principles to guide the development of CAHPS surveys and related tools. These principles include identifying and supporting the consumer's or patient's information needs, conducting thorough scientific testing, ensuring comparability of data, maintaining an open development process, and keeping products in the public domain. CAHPS surveys are developed by following a set of design principles to ensure that the data are psychometrically sound, scientifically credible, and useful for research, quality reporting, and accountability.
The core set of CAHPS® principles include the following:
Use the best scientific evidence available
Measure only those things for which the respondent is the best or only source of information
Base the assessment on respondents’ experiences with specific provider behaviors
Incorporate stakeholder input throughout the development process
Design the survey so that the results are easily communicated to consumer audiences
Place products in the public domain.1
The development of the original CAHPS Health Plan 1.0 Survey was described in a special issue of the journal Medical Care2. In preparation for submission to the National Quality Foundation (NQF) for their endorsement of the Health Plan Survey’s measures, the CAHPS Consortium engaged with the National Committee for Quality Assurance (NCQA) and the public to receive comments on the Health Plan 4.0 Survey. Additionally, the consortium conducted a field test of the questionnaire with six health plans. The CAHPS Health Plan 4.0 Survey was submitted to NQF and received their endorsement in 2007. In order to receive NQF endorsement, measures must meet a number of criteria3 including proving that a measure has been thoroughly tested for reliability and validity. The CAHPS Health Plan 5.0 Survey used as the basis for development of the QHP Enrollee Survey included only minor revisions from the 4.0 version4.
Cognitive Testing for the Marketplace and QHP Surveys
CMS conducted two rounds of cognitive testing in three languages (English, Spanish, and Chinese) for both surveys. The testing focused on the Marketplace Survey, which was almost entirely new. To avoid duplication, CMS relied heavily on the testing that had already been done on the CAHPS questions used in the QHP Enrollee Survey and only tested new or modified questions.
The first round of testing was conducted with proxy Marketplace users from the Massachusetts Health Connector because it had to be done before Marketplaces became operational. Nine interviews were conducted in each language to understand respondents’ experiences with the Massachusetts Health Connector. The second round of testing was conducted in the first weeks of Marketplace open enrollment when people had varying experiences with the Marketplaces. The 9 respondents in each language were used to provide a perspective of positive and negative experiences interacting with the Marketplace in a variety of ways such as on the website, over the phone, and in person. The short timeline between the opening of the Health Insurance Marketplaces and the psychometric testing of both surveys, combined with the iterative approach utilized, drove this approach to evaluating the surveys. In addition to the cognitive testing, the survey team coordinates and consults on an ongoing basis with CMS/CCIIO regarding alignment with CCIIO information collection activities to minimize duplication of burden and environmental scan information about likely enrollee characteristics to inform this project. The psychometric testing will verify and validate the cognitive testing and identify any additional testing needs.
Survey Translation
CMS chose Spanish and Chinese for translation because they are the two most commonly spoken languages in the United States after English according to data collected by the 2011 American Community Survey. Additionally, 56.3 percent of individuals who speak Spanish at home said that they spoke English “very well” and 44.3 percent of individuals who speak Chinese at home said they spoke English “very well”5. Thus about half of Spanish and Chinese speakers do not speak English very well and so could benefit from the availability of Spanish and Chinese versions.
The Spanish translation is designed for Latinos from different countries of origin living in the United States whose primary or sole language is Spanish. The Chinese translation was designed for Chinese living in the United States whose primary or only spoken language is Mandarin Chinese. For Chinese translation, the survey and accompanying materials have been translated into Traditional Chinese to ensure that the broadest population possible is able to understand the survey. Generally, individuals from Mainland China as well as Hong Kong and Taiwan are able to read Traditional Chinese. The use of Simplified Chinese was considered; however, populations outside of Mainland China are generally unable to read Simplified Chinese. That said, for both the psychometric and beta test phases, we are only sending Chinese surveys to two types of respondents: 1) those who specifically request a survey in Chinese, by calling a toll-free number, and 2) those who explicitly indicated a preference for Chinese in written materials when they filled out their Marketplace application. In the beta test phase, a mail with phone follow-up mode will be used for Spanish and Chinese and spoken language preferences will be incorporated. For the QHP Enrollee Survey beta test, conducting Chinese surveys are optional with the decision made by the QHP Issuer.
Survey Vendor Participation
There will be a single survey vendor for all rounds of administration for the Marketplace Survey and for the QHP Enrollee Survey testing in 2014 operating under a contract from CMS. However, for the 2015 QHP Enrollee Survey beta test and in publically reported QHP surveys in 2016 and 2017, 10 vendors have applied for approval to collect data under contract to the QHP issuers. They have applied using the Survey Vendor Participation form. The form is included in this clearance package and is modeled on the Medicare Part C and D survey vendor participation form. A list of approved vendors is posted on the QHP survey vendor website and will be updated annually in the same manner as is done on the Medicare Part C and D survey vendor website (CMS-Approved Survey Vendors). As a part of the 2015 beta test, we will be testing the survey vendor system. This test includes evaluating all of the systems including the survey vendor recruitment and application processes, training of vendors, technical assistance to vendors, and collection of data from survey vendors. Given that the survey will be administered by multiple vendors, it is essential that the results from each vendor are analyzed after the beta test to ensure that there are no anomalies or variations resulting in deviations from the survey administration protocol established by CMS. Appendix A includes a draft version of the minimum business requirements that survey vendors will have to meet in order to be approved as a survey vendor for the 2015 QHP Enrollee Survey. Vendors’ approval must be renewed annually. CMS may withdraw a survey vendor’s approval after three instances of non-compliance in a 24 month period. Additional criteria and procedures for evaluating a vendors’ performance are currently under development. This includes each vendor developing a Quality Assurance Plan (QAP) outlining how the vendor will comply with the program specifications; a combination of in-person and teleconference oversight activities during the 2015 beta test to ensure that vendors are complying with the specifications; and a review of all survey materials (Survey questionnaires, letters, postcards, CATI programming, and login credentials for the web survey) by the project team.
Decision-making Criteria Related to Methodological Goals
Psychometric Test Phase
The goals shown in Exhibit A1 all reflect the exploratory nature of these tests in that these goals are designed to inform revisions to the survey content, the sampling methodology, and the data collection protocols. The results of the test phase will be used to develop a set of guidelines and protocols reflecting what CMS learns from the psychometric testing and will include recommendations based on those findings. These recommendations will be aligned with CAHPS standards to the extent possible. See the following documents for examples of these guidelines:
Fielding the CAHPS Clinician & Group Surveys
Fielding the CAHPS Health Plan Survey 4.0: Commercial Version
As these two examples illustrate, some recommendations provide potential users with options (e.g., in choosing the mode of survey administration), while others are made more ardently (e.g., the recommendation to obtain a response rate of 40% or higher). Thus, the overarching goal for the psychometric tests of both surveys is to inform the development of a set of guidelines, one for each survey.
Survey content
For both surveys, there were two goals with respect to survey content: 1) identify survey items that can be dropped in order to shorten the survey, and 2) identify a set of internal and inter-unit reliable and valid composites to be used for scoring, analysis, and reporting.
Goal 1:
Substantive assessment items were dropped for the following reasons:
They did not load strongly on any measurement domains (i.e., factor loadings < 0.40).
The content of an item significantly overlapped with other items that have better psychometric properties.
A very small number of respondents were eligible to respond to the item (e.g., < 10%). Note that this criterion may conflict with other valid reasons to include the items in the survey. For example, very few respondents may report that they needed language interpretation services, but the case for retaining the item might supersede the statistical evidence for dropping it.
An item had a high level of non-response (e.g., > 10%), possibly indicating that respondents either did not understand the item or did not want to answer it. Note that item-level non-response rates do not include items that are legitimately skipped based on respondents’ answers to screening questions, only items that are truly missing.
An item had low covariance coverage with other items in the factor analysis.
The item had unit-level reliability < 0.70 and/or would require an effective sample size that is prohibitively large (more than 300 completed responses from each sampling and reporting entity) in order to obtain unit-level reliability ≥ 0.70.
CMS, the TEP, or other stakeholders determined that the item is inappropriate or otherwise not useful in meeting the goals of the survey data collection effort.
Items hypothesized to be case mix adjusters failed to meet the criteria for case mix adjusters.
An item was not a strong driver of the global ratings.
If substantive items controlled by screening questions were dropped, those screening questions were dropped as well. Demographic questions (‘About You’ items) were dropped if they are either found not useful for substantive subgroups analyses or they are not selected as case-mix adjusters. Global rating questions were dropped if they are either highly correlated with other items (indicating that they provide little unique information) or if the unit-level reliability of the items was unacceptable (see item 5 above). See change request form for detailed list of changes to Marketplace Survey for beta test.
Goal 2:
In the psychometric analysis of the test data, composites were evaluated based on the following criteria:
Results of a confirmatory factor analysis (CFA) show a good fit between the hypothesized composites and the observed psychometric test data. The fit is considered acceptable based on the following criteria: root mean square error of approximation (RMSEA) < 0.05; comparative fit index ≥ 0.95, and the Tucker-Lewis index ≥ 0.95.
Observed unit-level reliability should be ≥ 0.70. It can be less than 0.70, but in that case sample size projections based on unit-level reliability should indicate that an effective sample size of 300 or less per reporting unit will be needed to obtain unit-level reliability ≥ 0.70.
Internal-consistency reliability (Cronbach’s alpha) should be ≥ 0.70.
Scaling success should be 100%. Scaling success is an indicator of the extent to which items correlate more highly with their own composites than with competing composites.
Ceiling effects should be no higher than 75%, and preferably would be no higher than 50%. The ceiling effect indicates the proportion of respondents who give the highest possible rating for all items in a composite.
The items grouped together in a composite should make sense, substantively. This requirement is subjective rather than statistical, and relies on the mutual agreement of various stakeholders.
In some cases, where there is a compelling reason to retain a composite (a composite measures a concept that is very important to consumers), some of these criteria may be relaxed in order to allow the retention of that composite. Alternatively, such items may be retained as single item indicators. Although the psychometric properties of the CAHPS 5.0 Survey are well established, it will be important to learn if they hold for the QHP population given that most will have been uninsured and, thus, not previously included in the CAHPS 5.0 psychometric testing.
Psychometric analyses will be conducted separately by mode and language. Fit estimates from a CFA will be calculated separately for each mode and language group and statistical tests performed to estimate the probability that the fit, factor loadings, and error matrices are equal across groups. For example, using factor analysis we would test the following hypothesis:
Where Λ (lambda) represents a matrix of factor loadings estimated separately for respondents in each language. Failure to reject this null hypothesis would indicate that the pattern of factor loadings, and thus the construct validity, was consistent across the three languages. Similar tests would be done with error matrices and matrices of factor variances.
Sampling Methodology
The test results informed recommendations regarding sampling frame construction, sampling methods, and sample size. Since the tests were the first attempt at constructing a sampling frame using CMS administrative data, CMS maintained a log of challenges and decisions made with respect to identifying eligible individuals to include in the frames, the data fields required to properly identify accountable units (for the QHP survey), and differences or limitations related to the data available from the SBMs. Guidelines were developed based on these experiences to implement for the beta tests, and further refined as necessary between each subsequent round of data collection. See change request form for detailed update to sampling methodology for the Marketplace Survey beta test.
Response rates and non-response bias. Observed response rates from the tests, along with the unit-level reliability analysis, will inform future sample size recommendations. As part of testing the performance of the surveys in the test, CMS will determine if the goal of 30 percent response can be achieved. The response rates obtained in the psychometric test were used to adjust the response rate goals for the beta test and subsequent rounds. If 30 percent is not achieved in the psychometric test, the reliability of the surveys as determined at the national level will not be affected, but the ability to conduct subgroup analyses might be affected. The mode experiments will provide evidence of the feasibility of administering the surveys in different modes such as mail, phone, or web by comparing the response rates among sample members randomly assigned to each mode. In addition, the mode experiments will provide evidence for or against enhanced forms of follow-up (e.g., using FedEx for follow-up mailings) or mixed mode (e.g., mail with phone follow-up) compared to single mode approaches (e.g., mail-only or web-only). If response rates vary by mode in the psychometric test, CMS will compute a cost per complete for each mode and relate the response rate for that mode to its unit cost to determine if the benefit in terms of better response is worth any additional cost that might be required.
Response rate results from the Marketplace Survey psychometric test mode experiment show that the mail with phone follow-up mode produced the highest response rates (32%) compared to mail with FedEx follow-up (27%), mail with all first class (20%), phone only (22%), and web only (9%). On a cost per complete basis, mail with phone follow-up was less expensive than first class mail, mail with FedEx follow-up, and phone-only. Only the web-only mode was less expensive than mail with phone follow-up. There is also evidence from the psychometric test that there are differences in the characteristics of people who respond by phone, mail, or web. The phone respondent distribution is generally more comparable to the frame distribution than mail or web. The phone captures a higher percentage of potential enrollees and participants who are younger, Black, have lower incomes, and who have a disability compared to mail and web. For all of these reasons, we plan to administer the Marketplace Survey using a mail with phone follow-up mode in the beta test for all languages. For the English sample only, we will continue to offer the online mode in the pre-notification letter as a way to get early responses.
CMS conducted a nonresponse bias analysis to determine if there are systematic differences between respondents and nonrespondents in terms of their demographic characteristics that are related to the study outcomes. If bias is found, CMS will explore the feasibility of applying nonresponse bias adjustments to the results. CMS will also consider the possibility of conducting non-English surveys by telephone if the results of these analyses suggest that there is a significant bias associated with limiting non-English surveys to mail only.
For the Marketplace Survey psychometric test, CMS has conducted a nonresponse bias analysis and found that response rates varied across a number of demographic variables and other attributes (i.e., availability of e-mail address or phone number on the frame). A number of significant predictors were identified and the predictor that was most relevant to the survey measures was the application status of the individual. Individuals who successfully enrolled in a QHP were more likely to respond to the survey than those who were potential enrollees (those who completed the application, but had not chosen a health plan) and potential applicants (those who did not complete the application). As a result of this analysis, CMS has applied a nonresponse adjustment to the sampling weights in order to account for an individual’s propensity to respond to the survey. CMS will perform additional analyses for the QHP Enrollee Survey psychometric test and the 2015 beta test in order to develop procedures that should be utilized in future implementations of the Marketplace Survey.
The results of all of these analyses will be summarized in reports. For future implementations of the Marketplace survey, this assessment will be made by CMS once we see the variation in costs and response rates among the modes. There is no a priori assumption about an acceptable benefit-cost tradeoff and CMS also wants to remain consistent with standard CAHPS procedures to the extent possible. For the QHP Enrollee Survey, CMS will make guidelines available to survey sponsors and vendors.
Data collection protocols
CMS will make recommendations regarding choice of modes based on previous literature and the findings regarding response rates, nonresponse bias, and cost-benefit tradeoffs associated with the different modes tested, as described above.
Beta Test Phase
For the Marketplace Survey, psychometric analyses will be repeated using the beta test data and applying the same set of criteria described above. Also, disparities will be examined by estimating differences in assessment scores by race, ethnicity, income, and disability using regression models. Statistically significant parameter estimates from models regressing assessment scores on these respondent characteristics will indicate the possible presence of disparities (e.g., if respondents with lower income levels report more difficulties with the application process). CMS will also test if respondent characteristics vary across the four Marketplace application milestones. For example, if consumers start the application process but then never enroll, and this happens at a higher rate among consumers with lower incomes or with disabilities, such findings would suggest potential disparities that should be explored further.
For the QHP beta test, psychometric analyses will be repeated using the beta test data and applying the same set of criteria described above. In addition, CMS will be testing the survey vendor system. This test includes evaluating all of the systems including the survey vendor recruitment and application processes, training of vendors, technical assistance to vendors, and collection of data from survey vendors. Given that the survey will be administered by multiple vendors, it is essential that the results from each vendor are analyzed after the beta test to ensure that there are no anomalies or variations resulting in deviations from the survey administration protocol established by CMS. If such problems are detected, CMS will inform vendors of these problems and indicate to them what steps need to be taken to correct them.
Finally, for the QHP beta test, CMS will test for disparities in enrollee experiences across QHPs using the approach described above for the Marketplace beta test.
Project Tasks and Results
CMS is requesting approval for information collections associated with the 2015 beta test and generation of early feedback to the states and QHPs. CMS intends to update this request to include data collections associated with 2016 and 2017 national implementation with reporting of the Marketplace Survey to States and with public reporting of the QHP Enrollee Survey in the future. The questionnaires and sampling approaches included in this application are intended for use in the 2015 beta test phase. We expect small refinements for the 2016 national implementation with reporting based on the experience in the 2015 early feedback/beta test phase. Thus, we will submit amended questionnaires and sampling designs to this information collection each year, based on results from the previous year. In the future, CMS may also request an amendment to cover a Child QHP Enrollee Survey, a Dental survey, and a survey for disenrollees currently under development.
B. Justification
1. Need and Legal Basis
Section 1311(c) (4) of the Affordable Care Act (ACA) requires HHS to establish an enrollee satisfaction survey to be administered to members of each QHP offered through a Marketplace. In accordance with section 1311(c)(4), the results of this survey will be available on each State Marketplace’s web portal, as well as the Federally-facilitated Marketplace’s web portal, in a manner that allows applicants for coverage to compare plans.
CMS determined that since the Marketplace survey data elements do not align with CAHPS® as closely as the QHP enrollee survey and because the sampling requirements and reference periods differ, the Marketplace survey would not wholly align with the requirements under section 1311(c)(4). In 45 CFR § 155.1200(b)(3)6, we direct State Exchanges to submit performance monitoring data (which would include information on consumer satisfaction and other data from the Marketplace Survey) on an annual basis and so therefore, CMS intends to establish the Marketplace survey pursuant to sections 1313 and 1321(A) of the Affordable Care Act. Therefore, based on the authority in section 1313 and the authority of section 1321(A) which provides general standards for operating Marketplaces, CMS is now establishing and administering the Marketplace survey distinct from the section 1311(c)(4) authority of the ESS.
In the 2016 and 2017 full-scale national implementation, CMS plans to use these survey results to assess the potential for disparities in the use of the Marketplace and QHPs by race, ethnicity, income, and disability status. Direct information from consumers’ about their experiences over time is essential to detecting and correcting disparities through the quality improvement process. In order to meet this obligation, CMS requires survey-based performance scores for racial, ethnic, and income subgroups within each state from the beginning of Marketplace operations in October 2013 and QHP operations in 2014. This is consistent with section 3101(a) (1) of the Public Health Service Act (as amended by section 4302 of the Affordable Care Act), which requires HHS to make survey estimates by race, ethnicity, and other personal characteristics at the smallest practicable geographic level.7
2. Information Users
Psychometric Test Phase of the Marketplace and QHP Enrollee Surveys. We conducted a one-time psychometric test of the Marketplace and QHP Enrollee Survey instruments. The Marketplace Survey psychometric test took place from April–August 2014. The QHP Enrollee survey psychometric took place in August–December 2014. CMS has used the information from the psychometric test to develop reliable and valid questionnaires, survey methods and composite scores for future use in an annual survey of Marketplace and QHP performance
Beta Test Phase of the Marketplace and QHP Enrollee Survey. The beta test will be used by CMS to evaluate the survey vendor system for the QHP Enrollee Survey. The beta test will also provide feedback to Marketplaces and QHPs regarding the operations of the data collection, submission and scoring processes for the QHP Enrollee Survey and the QRS.
2016 and 2017 Full-Scale National Implementation of the Marketplace and QHP Enrollee Survey. The national implementation is expected to involve administration of surveys to the population of Marketplace users by a single contractor and to QHP enrollees by HHS-approved enrollee satisfaction survey vendors. It is anticipated that the QHP scores will be reported to the public for the first time in 2016 and again in 2017. The information is designed to be used by CMS and state governments to monitor the performance of the Marketplaces and QHPs. The QHP Enrollee Survey scores are designed to be used by the public to help choose QHPs.
3. Use of Information Technology
Marketplace Survey and QHP Enrollee Survey 2014 Psychometric Test Phase; Marketplace 2015 Beta Test Phase; and Marketplace 2016 and 2017 National Implementation. The Marketplace survey testing and implementation and the QHP Enrollee Survey testing will be administered by a single contractor using three modes of administration—mail, telephone, and online. Telephone interviews will be conducted using computer assisted telephone interviewing (CATI). The sample for the English language version of the surveys will have the option of completing the survey online through an emailed survey link.
QHP Enrollee Survey 2015 Vendor Beta Test and QHP Enrollee Survey/ Marketplace Survey 2016 and 2017 National Implementation. The survey administration will be similar to what is described above. The survey vendors will upload survey data to a secure electronic data warehouse operated by a CMS-approved contractor.
4. Duplication of Efforts
There is no duplication of efforts. This is a new collection and the only surveys of Marketplaces and QHPs planned by CMS. We have worked closely with the Center for Consumer Information and Insurance Oversight (CCIIO) in the development of these surveys to ensure that there is no duplication of data collection and that these surveys would complement other information obtained from issuers, consumers and marketplaces. Although CCIIO is collecting various Marketplace and QHP issuer performance metrics, the data collected from the Marketplace survey and QHP Enrollee survey are different and unique from those collected by CCIIO. As mentioned previously in this Supporting Statement, the QHP Enrollee Experience Survey is based on the CAHPS® Health Plan Survey and modeled on the Medicare Part C and D CAHPS survey (OMB Control Number 0938‐0732) system.
5. Small Businesses
No small businesses will be affected by this collection of information. The samples for all surveys comprise individuals. Sampling frames might be required from the Marketplaces and the issuers (insurance companies) that operate QHPs, but the Marketplaces are operated by government and the issuers are large businesses.
6. Less Frequent Collection
The Marketplace and QHP surveys sample different populations for different, although related, purposes. Rounds 1, 2, and 3 of each survey—the psychometric test, beta test, and 2016 reporting of results—have different purposes. The fourth round—2017 reporting—has the same purpose as the 2016 round, but annual data collection is needed to meet the objectives of providing feedback to Marketplaces, issuers, and regulators and providing information to consumers for choice, because Marketplaces and issuers will attempt to improve their performance each year.
7. Special Circumstances
There are no special circumstances associated with this data collection.
8a. Federal Register
This is a new collection. As required by 5 CFR 1320.8 (d), the Center for Consumer Information and Insurance Oversight (CCIIO) solicited comments on this information collection prior to submission to OMB, in the Federal Register (June 27 2013).
There were eleven comments received from the following organizations:
Access Living, Center for Service, Advocacy and Social Change for People with Disabilities
Agency for Healthcare Research and Quality (AHRQ)
America’s Health Insurance Plans (AHIP)
Anonymous, NY 12205
Association for Community Affiliated Plans (ACAP)
BlueCross BlueShield Association (BCBSA)
Center for the Study of Services (CSS)
Coalition for Disability Health Equity (CDHE) (this letter is identical to the Access Living letter)
Consumer-Purchaser Disclosure Project
Disability Rights Education & Defense Fund (DREDF)
Family Voices-NJ, Statewide Parent Advocacy Network
A top-line summary of the comments and how we have addressed them is below.
Addition of Items to Assess Consumer Experience for Persons with Disabilities. Multiple advocacy organizations highlighted the issue of health and health care disparities for people with disabilities, gaps in data collection for this population, and the potential for the Marketplace Survey and QHP Enrollee Survey data collection efforts to address those gaps to better assess the quality of services for these consumers. CMS has consulted with the HHS Commissioner on Disability and her staff at the Administration for Community Living. We have added a set of items currently used by the American Community Survey and other federal statistical agencies to capture disability status to both surveys. We can use this variable to understand the characteristics of those who respond to this survey and stratify responses by subgroup, in the same way that we do with income and language spoken at home. Additionally, we have added items related to habilitative services, such as physical, occupational, and speech therapy as well as home health care. CMS realizes that this is a longer-term issue, and that it will impact all CAHPS® surveys and will further consider how to address this in future data collection efforts.
Addition of Cultural Competence Items. The recommendation was made to add items from the CAHPS® Cultural Competence Item Set. These changes have been made to the QHP Enrollee Survey.
Recommended Additional Items. There were multiple recommendations for additional survey items and revisions to item wording. We have added questions about after-hours care and shared decision making from the CAHPS® supplemental item set.
Questionnaire Length. There were multiple comments on concerns over the length of the QHP Enrollee Survey. The version of the QHP Enrollee Survey that will undergo psychometric testing is longer than we expect the final questionnaire to be. We plan to eliminate items that do not perform well in the test.
Advance Letter and Cover Letters. A stakeholder group recommended omitting the advanced letter. Research on the benefits of sending an advanced letter is mixed and this is still current protocol for all CMS surveys. We will retain the advanced letter for the psychometric test and consider this recommendation for future survey administrations. We have revised the text in the cover letters to emphasize how respondent participation will help others and to further clarify confidentiality.
Online Survey Administration. A stakeholder group recommended testing the online mode of administration as many current CAHPS® surveys are utilizing this mode. We have added an experiment to the psychometric test to test online administration versus mail and phone administration. This is described in detail in supporting statement B.
Additional Pediatric Survey. Multiple stakeholder groups have recommended the creation of a separate survey to assess child services. The child-only QHP Enrollee Survey is currently under development.
Clarify case mix adjusters for QHP Enrollee and Marketplace Surveys. We plan to case-mix adjust for age, education, and general health status (standard case-mix adjusters). We will conduct a case mix analysis on other candidate adjusters and make recommendations based on those results.
Report QHP Enrollee Survey results by metal level. We have considered reporting at more granular levels such as the metal level, but based on feasibility concerns due to the number of metal level plans offered, we intend to have QHP Enrollee Survey reporting be at the product level in the initial years so as to align with the Quality Rating System under separate development.
Include Consumers That Do Not Enroll Through the Marketplace. It has always been the plan to collect data from consumers that begin the enrollment process and do not purchase coverage through the Marketplace. We will include all consumers that enter contact information in the sample for the Marketplace psychometric test. See supporting statement B for more details.
Additional Languages. It was recommended that surveys be developed in additional languages. We are initially testing surveys in English, Spanish, and Traditional Chinese. Future consideration will be given to additional languages.
A second Federal Register Notice was published on November 1, 2013, to solicit public comments. Six comments were received from the following groups:
America’s Health Insurance Plans (AHIP)
BlueCross BlueShield Association (BCBSA)
Families USA
Montana & Wyoming Tribal Leaders Council
National Indian Health Board
Tribal Technical Advisory Group to the Centers for Medicare & Medicaid Services
A top level summary of the comments and how they were addressed is below.
Addition of Items to Assess Consumer Experience for Persons with Disabilities. In response to comments that the survey results should be used to address disparities among individuals with disabilities, we have added the set of six disability questions, to both surveys, (Q22, Q29, Q38, Q40, Q65-66) that is used by the American Community Survey and other Federal statistical collections to estimate the portion of the population responding to the questionnaire that is disabled. We can use this variable to understand the characteristics of those who respond to this survey and stratify responses by subgroup, in the same way that we do with income and language spoken at home.
Questionnaire Length. A number of comments expressed concern about the length of both surveys. As noted previously, we have eliminated multiple poor performing questions based on the results of the psychometric testing.
Addition of Items Related to American Indians. A number of comments were received with suggestions on how to improve the surveys and data collection methods to better include the American Indian and Alaskan Native populations. We have added two questions on both surveys to determine whether an individual is eligible to use Indian Health Service facilities and whether they have ever done so. We will continue to consider improvements to these surveys and to addressing these populations.
Removed Items Related to Health Plan Quality. Based on comments we received, we removed items asking whether a respondent considered quality ratings of a health plan when they were selecting a health plan through the Health Insurance Marketplace. These questions have been removed given that information from the Quality Rating System (QRS) will not be publicly available until 2016. We plan to add these questions to the Marketplace Survey once this data is available.
Added Question about Health Insurance Literacy. A number of comments were received requesting additional questions be added to both surveys regarding health literacy. Based on these comments, we have added a potential measure of an individual’s health insurance literacy as we believe this data will be more meaningful than health literacy given the topic of the surveys.
Revised Questions about Wait Time for Health Plan’s Customer Service. We received several comments about the questions that were included on the QHP Enrollee Survey regarding the amount of time an individual waited to speak with their health plan’s customer service staff. Based on these comments, we removed two questions and instead will test a question that was included in a previous draft of the survey, as recommended by one of the comments.
Advance Letter and Cover Letters. A few comments were received regarding the advance and cover letters. One comment suggested deleting the advance letter based on a study that found that advance letters did not increase response rates in a telephone survey. We have decided to keep the advance letter as a part of the survey methodology based on previous research that has found that advance letters can increase response rates in a mail survey, particularly when the letter comes from an official source such as a government agency. Additionally, the advance letter for both surveys will inform respondents about the ability to complete the survey online, which will result in cost savings. Another comment was received recommending that some of the language in both the advance and cover letters may inadvertently result in individuals failing to complete the survey. Based on this comment, we have revised this language as suggested. A final comment suggested clarifying language in the cover and advance letters for the QHP Survey to explicitly state that QHPs would not receive identifiable information from these surveys. Based on this comment, we have added this language to both the advance and cover letters. Furthermore, in order to ensure that this identifiable information is not provided to health plans, we have added this as specific criteria for survey vendors.
In response to both public comment periods fourteen questions were added to the test version of the Marketplace Survey and eighteen questions were added to the test version of the QHP Enrollee Survey. These additions were partially offset by removing one question from the test version of Marketplace Survey and five questions from the test version of the QHP Enrollee Survey based on public comments that CMS received. A summary of the questions that were added to the surveys is shown in Exhibit A2.
Exhibit A2: Question Topics added to surveys based on public comments
Question Number(s) |
Question Topic |
|
Marketplace Survey |
20 |
Determine whether an individual had technical issues using the Health Insurance Marketplace website |
38 |
Determine whether an individual was unable to seek in person assistance with the Marketplace due to a building being inaccessible |
53 |
Determine whether an individual attempted to determine whether a specific medication was covered by a QHP |
54 |
How often it was easy for an individual to determine whether a specific medication was covered by a QHP |
65-66 |
Determine whether an individual needed forms in a different format, such as large print or braille and if the individual needed them how often forms were available in a different format |
75-80 |
Disability screening questions as required by Section 4302 of the ACA (Disability questions from the American Community Survey) |
88-89 |
Determine whether an individual was eligible to use the Indian Health Services, and if they were eligible determine if they have used these services |
93 |
Determine an individual’s confidence about understanding health insurance terminology |
|
QHP Enrollee Survey |
7-8 |
Determine whether an individual needed after-hours medical care, and if they did, determine how easy it was for the individual to obtain these services |
12-15 |
Determine whether the individual engaged in shared decision making with their health care provider regarding prescription medications |
59 |
Determine whether the individual knew the purpose of a form from their health plan before they filled it out |
61-62 |
Determine whether an individual needed forms in a different format, such as large print or braille and if the individual needed them how often forms were available in a different format |
64 |
Determine whether an individual would recommend their health plan to their friends and family |
86-91 |
Disability screening questions as required by Section 4302 of the ACA (Disability questions from the American Community Survey) |
99-100 |
Determine whether an individual was eligible to use the Indian Health Services, and if they were eligible determine if they have used these services |
104 |
Determine an individual’s confidence about understanding health insurance terminology |
105 |
Determine how confident an individual is that they understand how their health plan works |
All of the above changes were made for the psychometric testing of the QHP Enrollee and Marketplace surveys. For the QHP Enrollee Survey beta test, we shortened the survey for the purpose of testing the survey vendor process and data operations for the QRS. We kept items that are part of the CAHPS Health Plan 5.0 Core Survey (which is the basis for calling the QHP Enrollee Survey a CAHPS survey); incorporated in the QRS; required by the Affordable Care Act section 4302 standards for data collection; and newly written questions that would benefit from additional psychometric testing. We removed items that did not meet our criteria for inclusion discussed above including survey questions related to health promotion, shared decision-making, specialized services, and some respondent characteristics (e.g. employment status, eligibility for health services from Indian Health Service, tribal or urban Indian health program).
Changes to the Marketplace Survey for the beta test are outlined in detail in the change request form.
8b. Outside Consultation
CMS is working with a variety of outside organizations and persons to develop the Marketplace and QHP Enrollee Surveys. Chief among these organizations is the American Institutes of Research (AIR) for development of the survey, National Committee for Quality Assurance (NCQA), and Ipsos. In addition, a Technical Expert Panel composed of consumer advocates, health plan representatives, Marketplace administrators, survey design experts, state regulators, and providers provided feedback on technical issues. The panel met in December 2012, February 2013, April 2013, December 2013, April 2014, and October 2014.
9. Payments/Gifts to Respondents
No payments or gifts will be made to any respondents.
10. Confidentiality
Individual survey respondents will be told the purposes for which the information is collected and that, in accordance with section 934(c) of the Public Health Service Act, 42 USC 299c-3(c),, any identifiable information about them will not be used or disclosed for any purpose beyond conducting the survey.
Individual respondents contacted will be further assured their replies will be protected consistent with 20 CFR 401 and 4225 U.S.C.552a (Privacy Act of 1974). The advance and cover letters for the Marketplace Survey state, “Of course, what you have to say is private. Your answers will be part of a pool of information from others like you. What you say will be used only by this study. As required by federal law, the Centers for Medicare & Medicaid Services will not share it with anyone without your O.K., except where required by law, for example, if you indicate that you intend to harm yourself or others.” An identical statement will be used in the QHP Survey with the addition of a clause explicitly stating that a respondent’s health plan will not receive individually identifiable data. Additionally, similar statements will be reiterated before respondents complete either the telephone or online surveys. In instances where respondent identity is needed, the information collection will fully comply with all respects of the Privacy Act. The systems of records, maintained by CMS, that are associated with this information collection are the ‘‘Master Demonstration, Evaluation, and Research Studies (DERS)’’ System No. 09–70–0591 and the “Health Insurance Exchanges Program” System Number 09-70-0560.
Marketplaces and QHPs that supply sampling frames for the psychometric testing and beta test will have their organization’s results reported to themselves, CMS, and state regulators. QHPs that provide sampling frames for the 2016 and 2017 national implementations of the QHP Enrollee Surveys will make aggregate results available to the general public by insurance product type offered by a health plan issuer. Marketplaces that provide sampling frames for 2016 and 2017 will receive aggregate results for their Marketplace.
AIR will apply for HIPAA waivers of authorization from the AIR Institutional Review Board (IRB) to enable the Marketplaces and QHPs to share contact information as part of the sampling and survey operations processes without obtaining prior permission from Marketplace consumers and QHP enrollees. The AIR IRB typically grants a waiver for the surveys of health insurance beneficiaries, because it is impracticable to contact each sample member before the survey is administered.
11. Sensitive Questions
There are no sensitive questions associated with this information collection.
12. Burden Estimates (Hours & Wages)
The estimated burden for the Marketplace Survey is shown in Exhibit A3 and for the QHP Enrollee Survey in Exhibit A4.
Marketplace Survey
Units For the psychometric test, the sample was drawn from the 36 states with Federally Facilitated Marketplace (FFM) or State Partnership Marketplaces (SPM).
For the beta test, respondents will be sampled from each individual state, including states with a State-based Marketplace that wish to participate for a total of 44 states. In Exhibit A3 below we have included all states that use Healthcare.gov (including 7 SBMs and 3 SSBMs) as part of the FFM.
Respondents per unit. See burden table.
Total Respondents. The total respondents were calculated by summing the product of the number of Units by the Respondents per Unit and total Spanish and Chinese Respondents.
Number of responses per respondent. Respondents will only be asked to respond once.
Hours per response. The psychometric test version of the Marketplace Survey was 96 items with an estimated completion time of 24minutes (.40 hours). This estimate is based on the length of previous CAHPS® surveys of comparable length that have been administered to similar populations.
The revised Marketplace Survey for the beta test is 83 items with an estimated completion time of 14.4 minutes (0.24 hours) (approximately 5.75 questions per minute). This estimate is based on the time it took to complete the Marketplace Survey in the psychometric test: 19 minutes for the phone administration and 14 minutes for the online administration for an average of 16.5 minutes to answer 95 questions. This is approximately 5.75 questions per minute.
Survey vendors. There will be a single survey vendor for all rounds of administration for the Marketplace Survey who will be a CMS contractor. Thus, no vendor burden is associated with the Marketplace Survey.
Exhibit A3. Estimated Burden Hours for Marketplace Survey by Round
Data Collection |
Total Completes |
Hours per Response |
Total Burden Hours |
Psychometric Test (Round 1, 2014) |
Psychometric Test (Round 1, 2014) |
Psychometric Test (Round 1, 2014) |
Psychometric Test (Round 1, 2014) |
English |
2,250 |
0.4 |
900 |
Spanish |
450 |
0.4 |
180 |
Chinese |
450 |
0.4 |
180 |
TOTAL Psychometric Test |
3,150 |
0.4 |
1,260 |
Beta Test (Round 2, 2015) |
Beta Test (Round 2, 2015) |
Beta Test (Round 2, 2015) |
Beta Test (Round 2, 2015) |
Type of Marketplace |
|
|
|
SBMs (7) |
8,400 |
0.24 |
2,016 |
FFMs (36) |
43,200 |
0.24 |
10,368 |
Total Beta Test |
51,600 |
0.24 |
12,384 |
TOTAL MARKETPLACE SURVEY |
54,750 |
|
13,644 |
QHP Enrollee Survey
Units. For the QHP Enrollee Survey psychometric test (Round 1), we want to make estimates of QHP performance at the state level by sampling a predetermined number of QHP enrollees across all QHPs operating in the state. It is not necessary to have reliable estimates for every QHP operating in the state to confirm that the CAHPS Health Plan Survey is reliable and valid for the QHP population, so we can minimize the burden by making estimates at the state level. Beginning with the beta test of the vendor system, however, it is necessary to make estimates for all QHPs, so that they can establish relationships with vendors and so that CMS can receive initial feedback on the scoring and reporting methods. Thus, the units for the beta test, 2016, and 2017 surveys will be the QHPs. In calculating the burden, the number of units is the current estimate of the number of QHPs that will be approved by all Marketplaces. The burden estimates will be revised prior to survey rounds 3–4.
Respondents per unit. See burden table.
Total respondents. The total respondents are equal to the product of the completed surveys per plan and the current estimate of the number of QHPs.
Number of responses per respondent. Respondents will only be asked to respond once.
Hours per response. The length of the QHP Enrollee Survey for the round 1 psychometric test is 107 items with an estimated completion time of 26.75 minutes (.45hours) based on the length of previous CAHPS® surveys of comparable length that have been administered to similar populations. The QHP Enrollee survey for the round 2 beta test is reduced to 93 items with an estimated completion time of 23.25 (.39 hours) minutes.
Survey vendors. Survey vendors who want to participate in collecting QHP Enrollee Survey data must complete a Survey Vendor Participation Form. The estimated survey vendor burden to complete the Survey Vendor Participation Form is 1 hour per vendor. There will be a single survey vendor for the QHP Enrollee Survey psychometric test in 2014 operating under contract to CMS, so no vendor burden is imposed for the psychometric test. Ten vendors have applied to be survey vendors for the Beta Test in 2015. These vendors will not be under Federal contract so burden was incurred.
Exhibit A4. Estimated Burden Hours for QHP Enrollee Survey by Round
Source |
Total Completes |
Hours per response |
Total burden hours |
Psychometric Test (Round 1, 2014) |
Psychometric Test Round 1, 2014 |
Psychometric Test Round 1, 2014 |
Psychometric Test Round 1, 2014 |
English |
3,000 |
0.45 |
1,350 |
Spanish |
600 |
0.45 |
270 |
Chinese |
600 |
0.45 |
270 |
TOTAL Psychometric Test |
4,200 |
|
1,890 |
Beta Test (Round 2, 2015) |
Beta Test Round 2, 2015 |
Beta Test Round 2, 2015 |
Beta Test Round 2, 2015 |
Survey Respondents (2,000 QHPs) |
600,000 |
0..39 |
234,000 |
Survey Vendors |
10 |
1 |
10 |
Total Beta Test |
600,010 |
|
234,010 |
TOTAL QHP ENROLLEE SURVEY |
604,210 |
|
235,900 |
In 2014, the total annual burden hours for the Marketplace and QHP Enrollee Survey psychometric tests are estimated to be 3,150 hours with 7,350 responses. In 2015, the total annual burden hours for the Marketplace and QHP Enrollee Survey beta tests are estimated to be 249,544 hours with 658,960 responses. The total average annualized burden over three years for this requested information collection is 83,181 hours and the total average annualized number of responses is 219,653 responses.
The Bureau of Labor Statistics reported the average hourly wage for civilian workers in the United States was $24.10 in October 2013 and $24.66 in November 2014. See exhibit A5 for estimated burden costs.
Exhibit A5. Estimated Burden Costs
Survey |
Number of Respondents |
Total Burden Hours |
Average Hourly Wage Rate |
Total Cost Burden |
Psychometric Test (Round 1,2014) |
Psychometric Test, 2014 |
Psychometric Test, 2014 |
Psychometric Test, 2014 |
Psychometric Test, 2014 |
Marketplace Survey |
3,150 |
1,260 |
$24.10 |
$30,366.00 |
QHP Survey |
4,200 |
1,890 |
$24.10 |
$45,549.00 |
Total |
7,350 |
3,150 |
$24.10 |
$75,915.00 |
Beta Test (Round 2, 2015) |
Beta Test, 2015 |
Beta Test, 2015 |
Beta Test, 2015 |
Beta Test, 2015 |
Marketplace Survey |
51,600 |
12,384 |
$24.66 |
$305,389.44 |
QHP Survey |
600,010 |
234,010 |
$24.66 |
$5,770,686.60 |
Total |
651,610 |
246,394 |
$24.66 |
$6,076,076.04 |
TOTAL OVERALL |
658,960 |
249,544 |
|
$6,151,991.04 |
13. Capital Costs
There are no direct capital costs to respondents other than their time to participate in the survey.
14. Cost to Federal Government
The only cost to the Government of these data collections that would not otherwise have been incurred is the cost of the AIR contract. Those costs for all four rounds of both surveys will be approximately $3,707,278 annually over 4 years. This includes the following for both surveys: survey development; cognitive testing and survey revisions; psychometric test design, administration, data analysis, and survey revisions; survey vendor certification; beta test design, administration, data analysis, and revisions; design and administration of the first national implementation for public reporting; and design and administration of the second national implementation for public reporting.
15. Changes to Burden
There are updates to the estimated burden based on this change request. For the QHP Enrollee Survey beta test, we have updated the number of items to 93 changing the burden for each respondent from .45 hours to .39 hours. We have also updated the burden for the survey vendors for the QHP Enrollee Survey beta test to 10 hours based on the number of survey vendors that applied. For the Marketplace Survey beta test, we have updated the number of items to 83 and updated the length of time to .24 hours based on the psychometric test.
16. Publication/Tabulation Dates
Publication for this project constitutes the preparation and distribution of scores for the relevant unit for each survey (states or QHPs). Exhibit A6 summarizes the planned reporting and publication activities.
Exhibit A6. Schedule of Reporting Activities
Activity |
Marketplace Survey |
QHP Enrollee Survey |
2014 Psychometric Test Report |
December 2014 |
February 2015 |
2015 Beta Test Report |
April 2015 |
April 2015 |
2016 National Implementation Report |
April 2016 |
April 2016 |
2016 Scores Delivered for QRS Website |
NA |
June 2016 |
2016 Scores on QRS Website for Reporting |
NA |
October 2016 |
Publication of Survey Results |
December 2016 |
December 2016 |
2017 National Implementation Report |
April 2017 |
April 2017 |
17. Expiration Date
The expiration date will be displayed.
18. Certification Statement
There are no exceptions to the certification statement identified in Item 19, "Certification for Paperwork Reduction Act Submissions," of OMB Form 83-I.
Appendix A: Draft Minimum Business Requirements for 2014 QHP Enrollee Survey Vendors
DCAPP = Division of Consumer Assessment & Plan Performance
MA & PDP CAHPS = Medicare Advantage and Prescription Drug Plan CAHPS
HCAHPS = Hospital CAHPS
A survey vendor must meet all of the Minimum Business Requirements listed below in order to apply to administer the Qualified Health Plan Enrollee Experience Survey (QHP Enrollee Survey).
Please note that these requirements have not yet finalized, and will likely undergo further revisions by CMS.
Relevant Survey Experience
Demonstrated recent experience in fielding health care patient experience surveys.
Criteria |
Survey Vendor Requirements |
Rationale |
Number of Years in Business |
|
|
Organization Survey Experience |
|
|
Number of Years Conducting Surveys |
|
|
Experience with Multiple Survey Languages |
|
|
Organization Survey Capacity
Capacity to handle a required volume of mail questionnaires or conduct standardized telephone interviewing in a specified time frame.
Criteria |
Survey Vendor Requirements |
Rationale |
Capacity to Handle Estimated Workload |
|
|
Criteria |
Survey Vendor Requirements |
Rationale |
Personnel |
|
|
System Resources |
|
|
Use of Subcontractors (Subject to Approval) |
|
|
Mode Administration |
|
|
Sampling Experience |
Applicant organization must document statistical approach to drawing a sample. Demonstrate ability to work with individual QHPs to electronically obtain sample frame for sampling within specified timeframe. Conduct quality checks on sample frame file received from QHP issuer to verify accuracy and completeness of sample frame information. |
|
Data Submission |
|
|
Data Security |
|
|
Data Retention |
|
|
Confidentiality |
|
|
Technical Assistance/ Customer Support |
|
|
Quality Control Procedures
Personnel training and quality control mechanisms employed to collect valid, reliable survey data.
Criteria |
Survey Vendor Requirements |
Rationale |
Demonstrated Quality Control Procedures |
|
|
Training Requirements |
|
|
Training Participants |
|
|
Approval Term
An approved survey vendor may administer the QHP Enrollee Survey for the specified amount of time.
Criteria |
Survey Vendor Requirements |
Rationale |
Approval Term |
|
|
1 Goldstein E, Farquhar M, Crofton C, Darby C, Garfinkel S. Measuring hospital care from the patients' perspective: an overview of the CAHPS® Hospital Survey development process. Health Serv Res. 2005 Dec; 40(6 Pt 2):1977-95. PubMed PMID: 16316434; PubMed Central PMCID: PMC1361247.
2 "Consumer Assessment of Health Plans Study (CAHPS)."Medical Care. 37.Supplement 3 (1999): MS1 - MS115. <Consumer Assessment of Health Plans Study (CAHPS)>
3 A full list of NQF’s endorsement criteria can be found at: NQF: Measure Evaluation Criteria
4 A crosswalk of the changes between the 4.0 and 5.0 can be found at: CAHPS Health Plan Surveys: Overview of the Questionnaires
5 Ryan, Camille. United States. Census Bureau. Language Use in the United States: 2011. Suitland, MD: 2013. <Language Use in the United States: 2011>.
6 Patient Protection and Affordable Care Act; Program Integrity: Exchange, Premium Stabilization Programs, and Market Standards; Amendments to the HHS Notice of Benefit and Payment Parameters for 2014; Final Rule, 78 FR 65046 (Oct. 30, 2013)
7 ‘‘TITLE XXXI—DATA COLLECTION, ANALYSIS, AND QUALITY
SEC. 3101. DATA COLLECTION, ANALYSIS, AND QUALITY.
(a) DATA COLLECTION.
(1) IN GENERAL.—The Secretary shall ensure that, by not later than 2 years after the date of enactment of this title, any federally conducted or supported health care or public health program, activity or survey (including Current Population Surveys and American Community Surveys conducted by the Bureau of Labor Statistics and the Bureau of the Census) collects and reports, to the extent practicable—
(A) data on race, ethnicity, sex, primary language, and disability status for applicants, recipients, or participants;
(B) data at the smallest geographic level such as State, local, or institutional levels if such data can be aggregated;
(C) sufficient data to generate statistically reliable estimates by racial, ethnic, sex, primary language, and disability status subgroups for applicants, recipients or participants using, if needed, statistical oversamples of these subpopulations; and
(D) any other demographic data as deemed appropriate by the Secretary regarding health disparities.”
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-25 |
© 2024 OMB.report | Privacy Policy