Supporting Statement Part B

Universe of health plans and representativeness of the data. While there are many survey vendors that collect Consumer Assessment of Healthcare Providers and Systems (CAHPS) Health Plan Survey data and may maintain a database of their own clients’ data, the Agency for Healthcare Research and Quality (AHRQ) is the only entity that serves as a comprehensive repository of CAHPS survey data. However, the CAHPS Health Plan Database is comprised of data that are voluntarily submitted by health plans that have administered the CAHPS Health Plan Survey and is not a statistically selected sample, nor is it a representative sample of all health plans in the U.S. Voluntary participants include public and private employers, State Medicaid agencies, State Children's Health Insurance Programs (SCHIP), the Centers for Medicare & Medicaid Services (CMS), individual health plans, and the Department of Defense. AHRQ collaborates with CMS and the National Committee for Quality Assurance (NCQA) to combine their CAHPS Health Plan Survey data with the CAHPS Health Plan Database.

The CAHPS Health Plan Survey Database is a voluntary, comprehensive database of CAHPS Health Plan Survey responses gathered directly from Medicaid plans and CMS. NCQA is the managed care plan accreditor and requires yearly CAHPS results for accreditation. Medicare conducts a yearly survey of all Medicare plans and provides the CAHPS Health Plan Database these results. Many Medicaid program health plans do not seek NCQA accreditation, but submit instead directly to the CAHPS Health Plan Database.

Because the organizations that voluntarily contribute data to the CAHPS Database are not from a statistically representative sample of all U.S. health plans, and a limited number of plans may choose to participate, the submitting organizations are not representative of all U.S. health plans or enrollee populations¹. Estimates based on these voluntarily submitted data sets may produce biased estimates of the U.S. health plan and enrollee populations; it is not possible to compute estimates of precision from these data. In addition, the number and mix of sponsors contributing data vary slightly from year to year, and therefore comparisons over time should be made with these limitations and variations in mind. Comparisons of results across populations should also take into account that variations in benefit design and other factors might affect survey responses across populations.

Table B-1 presents the number of Medicaid, SCHIP, and Medicare survey respondents and health plan submissions included in the CAHPS Health Plan Database for 2017 and 2018. The number of health plan submissions is indicated in parentheses.

Table B-1. Number of Survey Respondents and Health Plan Submissions: 2017 and 2018

Year	Medicaid		CHIP	Medicare
(Version)	Adult	Child	Child	Adult
2018 (5.0)	54,362 (146)	79,736 (150)	13,933 (25)	190,838 (388)
2017 (5.0)	65,053 (152)	103,283 (169)	15,221 (23)	194,916 (397)

Medicaid Data: The 5.0 CAHPS Health Plan Survey results for the Medicaid sector were obtained from data submitted directly to the CAHPS Health Plan Database by State Medicaid agencies and individual health plans. The 2018 database consists of submissions from 36 states, of which a total of 19 Medicaid State Agencies submitted data, compared to 36 States, 18 of which were State Medicaid agencies in 2017. Medicaid results are based on survey data collected from October through June.

Medicare Data: Each year, the CAHPS Health Plan Database receives the CAHPS Medicare Managed Care survey data collected by the Centers for Medicare & Medicaid Services (CMS). CMS collected survey information from beneficiaries enrolled in managed care health plans that provide a prescription drug benefit. For 2018, 388 managed care plans participated in the survey. Survey participants included both enrollees receiving prescription drug coverage through their health plan and those that don’t receive prescription drug coverage through their health plan. Beneficiaries enrolled in traditional Medicare (Medicare fee-for-service) as well as fee-for-service enrollees who selected a prescription drug plan were also surveyed in 2018, but these data are not represented in the Chartbook. The results in the Chartbook include only beneficiaries who were enrolled in a managed care health plan. The Medicare Survey data are collected from March through June. The Medicare results presented in the CAHPS online reporting system may differ from other reports due to the inclusion or exclusion of certain beneficiary groups and/or the use of case-mix adjustment variables.

Tables B-2 and B-3 present the number of survey respondents and health plan submissions obtained for the 5.0 results within each major sector by State, including U.S. territories and the District of Columbia.

Table B-2. 2018 Survey Respondents and Health Plan Submissions by State (5.0 Results)

State	Medicaid Adult	Medicaid Child	CHIP	Medicare Adult
Alabama	515	592	885	2,532
Arizona	-	-	-	5,478
Arkansas	-	-	412	1,391
California	4,339	4,515	-	12,023
Colorado	584	412	1,953	2,528
Connecticut	389	-	1,677	1,715
Delaware	416	385	-	-
District Of Columbia	450	1,155	-	-
Florida	1,517	3,551	-	11,443
Georgia	650	1,393		4,193
Hawaii	2,503	-	603	2,172
Idaho	-	-	-	1,766
Illinois	-	2,462	-	3,878
Indiana	1,064	1,248	-	2,016
Iowa	-	-	-	837
Kansas	1,372	4,228	1,246	-
Kentucky	1,019	1,281	852	2,063
Louisiana	1,694	4,069	-	3,289
Maine	-	-	-	2,645
Maryland	2,689	7,986	-	2,394
Massachusetts	-	-	-	4,739
Michigan	5,168	4,066	-	9,791
Minnesota	-	-	-	4,880
Mississippi	-	-	-	1,738
Missouri	-	-	-	3,551
Montana	-	-	-	647
Nebraska	-	-	-	379
Nevada	282	576	-	2,619
New Hampshire	316	712	-	2,060
New Jersey	849	1,105	601	4,838
New Mexico	868	1,719	-	2,904
New York	6,950	1,679	-	13,953
North Carolina	-	-	-	4,005
Ohio	2,062	4,215	-	6,422
Oklahoma	475	-	426	2,477
Oregon	5,407	6,541	-	7,103
Pennsylvania	3,429	3,779	1,653	12,368
Puerto Rico	-	-	-	3,709
Rhode Island	893	1,368	-	1,010
South Carolina	828	1,926	-	1,210
Tennessee	1,226	3,225	-	4,504
Texas	1,440	5,826	1,849	12,209
Utah	864	2,133	459	3,493
Vermont	497	4,65	-	-
Virginia	1,456	2,854	805	1,686
Washington	1,028	2,947	512	6,441
West Virginia	474	552	-	1,055
Wisconsin	649	771	-	8,684
Total	54,362	79,736	13,933	190,838

Table B-3. 2017 Survey Respondents and Health Plan Submissions by State (5.0 Results)

State	Medicaid Adult	Medicaid Child	CHIP	Medicare Adult
Alabama	446	445	922	1,857
Arizona	-	-	-	4,293
Arkansas	410	404	479	2,107
California	3,752	5,116	-	13,440
Colorado	613	364	2,412	3,256
Connecticut	-	-	-	2,350
Delaware	584	421	-	-
District Of Columbia	790	2,312	-	-
Florida	1,723	4,083	-	10,514
Georgia	1,387	2,847	-	5,076
Hawaii	652	2,510	659	3,006
Idaho	-	-	-	2,140
Illinois	1,591	4,131	-	4,451
Indiana	3,320	3,336	-	4,420
Iowa	-	-	-	1,286
Kansas	1,590	4,684	1,294	-
Kentucky	1,619	1,806	1,226	2,385
Louisiana	1,692	2,885	1,416	2,947
Maine	-	-	-	2,878
Maryland	4,895	9,952	-	2,361
Massachusetts	1,931	314	-	5,861
Michigan	5,937	4,881	-	6,007
Minnesota	5,093	-	-	4,481
Mississippi	-	-	-	602
Missouri	-	1,439	-	3,192
Montana	-	-	-	663
Nebraska	-	-	-	653
Nevada	481	1,274	-	3,021
New Hampshire	374	926	-	1,972
New Jersey	1,353	1,583	758	2,882
New Mexico	1,300	2,606	-	2,498
New York	2,260	8,974	-	15,264
North Carolina	-	-	-	2,601
Ohio	2,372	4,688	-	6,661
Oklahoma	-	-	501	2,945
Oregon	4,906	5,149	-	8,424
Pennsylvania	3,771	4,365	1,817	13,777
Puerto Rico	-	-	-	4,524
Rhode Island	1,050	1,274	-	972
South Carolina	1,160	2,712	-	1,803
Tennessee	1,045	2,791	-	4,576
Texas	1,387	3,154	1,491	11,630
Utah	-	-	-	3,031
Vermont	-	-	-	-
Virginia	1,971	3,091	908	2,614
Washington	1,793	6,195	1,338	6,730
West Virginia	1,111	1,273	-	1,112
Wisconsin	694	1,298	-	7,653
Total	65,053	103,283	15,221	194,916

The CAHPS Health Plan Database currently contains 20 years of data from the CAHPS Health Plan Survey. Table B-4 shows data submissions to the CAHPS Database from 1998 to 2018. The total number of respondents is presented by population sector, with the number of health plan submissions given in parentheses.

Table B-4. Data Submissions to the CAHPS Health Plan Database From 1998-2018

Year (CAHPS Version)	Commercial		Medicaid		CHIP	Medicare
Year (CAHPS Version)	Adult	Child	Adult	Child	Child	Adult
2018 (5.0)	N/A2	N/A	54,362 (146)	79,736 (150)	13,933 (25)	190,838 (388)
2017 (5.0)	N/A2	N/A	65,053 (152)	103,283 (169)	15,221 (23)	194,916 (397)
2016 (5.0)	N/A2	N/A	73,155 (157)	79,058 (132)	14,999 (21)	147,908 (382)
2015 (5.0)	N/A	N/A	61,369 (133)	91,049 (136)	13,466 (19)	155,095 (431)
2014 (5.0)	N/A	N/A	68,234 (149)	60,153 (100)	11,762 (15)	195,748 (443)
2013 (5.0)	N/A	N/A	60,249 (124)	66,804 (105)	9,149 (12)	198,350 (451)
2011 (4.0)	168,341 (376)	900 (1)	73,820 (148)	85,003 (129)	26,232 (41)	163,182 (445)
2010 (4.0)	139,156 (288)	1,474 (2)	97,626 (132)	88,694 (132)	N/A	221,120 (431)
2009 (4.0)	179,528 (405)	751 (2)	63,391 (126)	68,697 (107)	N/A	206,647 (405)
2008 (4.0)	174,307 (410)	0 (0)	59,840 (120)	9,755 (29)	0 (0)	207,366 (343)
2008 (3.0)	0 (0)	1,882 (4)	0 (0)	37,347 (64)	0 (0)	0 (0)
2007 (4.0)	106,811 (239)	0 (0)	45,979 (109)	4,647 (16)	0 (0)	115,910 (296)
2007 (3.0)	0 (0)	1,659 (4)	0 (0)	64,039 (103)	0 (0)	0 (0)
2006 (3.0)	124,585 (271)	2,400 (7)	43,174 (119)	50,204 (95)	9,303 (30)	97,955 (273)
2005 (3.0)	123,272 (254)	2,661 (4)	32,115 (76)	40,204 (65)	1,252 (3)	127,930 (276)
2004 (3.0)	111,680 (223)	7,024 (12)	59,515 (149)	86,159 (128)	16,657 (29)	132,420 (288)
2003 (3.0)	114,063 (216)	1,866 (4)	39,275 (112)	31,081 (69)	19,061 (49)	141,421 (295)
2002 (2.0)	94,546 (219)	5,600 (10)	48,109 (136)	60,534 (122)	18,910 (43)	153,172 (321)
2001 (2.0)	165,500 (266)	9,913 (24)	45,127 (142)	36,940 (124)	0 (0)	179,451 (381)
2000 (2.0)	135,479 (270)	2,760 (8)	49,327 (156)	41,400 (140)	0 (0)	166,072 (367)
1999 (2.0)	168,234 (307)	42,879 (149)	28,420 (77)	14,106 (66)	0 (0)	0 (0)
1998 (1.0)	34,965 (54)	0 (0)	23,519 (31)	9,871 (33)	0 (0)	0 (0)
TOTALS	1,840,467	81,769	1,091,659	1,208,764	169,945	2,995,501

Most of the CAHPS Health Plan Survey questions ask respondents to report on their experiences with different aspects of their care. These reporting questions are combined into groups that address the same aspect of care or service to arrive at a broader assessment. The 5.0 version of the CAHPS Adult and Child Health Plan Surveys reporting questions fall into four major “composites” that summarize consumer experiences in the following areas: 1) Getting needed care, 2) Getting care quickly, 3) How well doctors communicate, and 4) Health plan information & customer service.

The CAHPS Health Plan Survey collects four separate global ratings to distinguish between important aspects of care. The four questions ask plan enrollees to rate their experiences in the past 6 months with: 1) their personal doctor, 2) the specialist they saw most often, 3) health care received from all doctors and, 4) their health plan. Ratings are scored on a 0 to 10 scale, where 0 is the “worst possible” and 10 is the “best possible.” The ratings are analyzed and presented in the three-part bar chart display used in the CAHPS Health Plan Survey reports: the percentage of respondents who gave a rating of either 0-6, 7-8, or 9-10. This three-part scale is used because testing by the CAHPS team determined that these cut-points improve the ability to discriminate among plans while simplifying the presentation of results.

Case-Mix Adjustment. Case-mix refers to the respondents’ health status and other socio-demographic characteristics that have been shown to affect enrollee reports and ratings of health plans. Characteristics used to case-mix adjust CAHPS Health Plan scores, where applicable, are respondent age, education, and self-reported physical and mental health status.

Testing for Statistical Differences. The individual participant’s comparative reports test for statistically significant differences between mean scores and ratings of individual health plans and the mean of all plan means in the CAHPS Health Plan Database using the t-test. A significance level of 0.05 or less is considered statistically significant. As described in the previous sections, the mean scores are adjusted for case-mix differences before the statistical tests are applied. To compute the means, reports and rating responses are grouped into three categories and assigned a score of 1, 2, or 3. Then, significance tests for both the reports and ratings are conducted on the mean scores. Individual plan results that differ significantly from the overall mean are denoted by arrows, either pointing up (significantly higher than the overall mean) or down (significantly lower than the overall mean).

2. Information Collection Procedures

Information collection for the AHRQ CAHPS Health Plan Survey Database occurs in a regular data collection cycle each year in June/July. The information collection procedure for submitting and processing data for the database is shown in Figure B-1. Each of the steps is described below. Screen shots of each step are provided in Attachment E.

Figure B-1. CAHPS Health Plan Database Data Submission Process

Step 1: Call for Data Submission. State Medicaid agencies and health plans that have administered the CAHPS Health Plan survey are recruited through multiple outlets and asked to submit to the database. The call for data submission is done through various publicity sources such as AHRQ’s electronic newsletters, GovDelivery messages, and communication with prior year’s participants. Organizations that have administered the Health Plan Survey and are interested in participating in the CAHPS Health Plan Database communicate with Westat through a dedicated email address ([email protected]) that routes directly to Westat and a toll-free phone number (1-888-808-7108) to indicate their interest in participation.

Step 2: Complete Registration Form. A database submission extranet has been set up so that interested parties such as, state Medicaid agencies, coalitions, vendors, and health plans register for the data submission process. Information about eligibility requirements, benefits of participation, data use agreement, and data file specifications regarding how to prepare their data file for inclusion in the CAHPS Health Plan Database are posted on the extranet. The data file specifications ensure that data files received from users are standardized and consistent in the way variables are coded and formatted. Potential participants’ online registration information will be reviewed by Westat staff. Upon approval of the registrant, an automated email that contains a username and password to the data submission extranet is sent to the registered user. The automatic email informs registrants of the information needed in the next steps of the data submission process.

Step 3: Complete Health Plan Information Form. This step requires each health plan that administered the CAHPS Health Plan survey, submit the requested characteristics including such details as the name of the plan, product type (e.g., HMO, PPO), the population surveyed (e.g., adult Medicaid, child Medicaid) and plan state.

Step 4: Upload Data Use Agreement. To protect the confidentiality of all respondents and entities that are included in any CAHPS Database, all participating institutions must sign a data use agreement (DUA) that has been reviewed and approved by AHRQ. The data use agreement specifies how the submitted data will be used, provides assurance that the identity of the participating institution will be protected and ensures the confidentiality of the data. Data are not included in the database without this signed data use agreement. Users must upload a copy of the signed agreement. Data collection vendors may not sign and submit the DUA on behalf of an institution (even if they have been given permission by the entity to handle the actual submission of data). Only a duly appointed representative may sign the DUA.

Step 5: Upload Questionnaire. Each health plan must upload a copy of the questionnaire used. The CAHPS Database reviews the questionnaire to ensure that it meets CAHPS Health Plan Survey standards (the survey instrument must include all core questions, not alter the wording of any core questions, and must not omit any of the survey items related to respondent characteristics that are used for case mix adjustment.) Once the questionnaire is reviewed by CAHPS Database staff, an email notification is sent to the registrant within three business days with an approval or rejection. Only health plans that receive questionnaire approval may submit data files.

Step 6: Secure Online Data Submission. To enable participants to transmit their CAHPS survey data to Westat in a secure manner, an online data submission extranet has been developed. The online system will be expanded and adapted to include data submission for all CAHPS surveys. Data are accepted in csv format. Data files must conform to the Data File Layout Specifications provide by the CAHPS Database. Since the unit of analysis is at the health plan level, users must upload one data file per health plan.

Data File Approval. Once a data file is successfully uploaded, a separate load program developed in Visual Basic (VB) reads the submitted files and loads them into the SQL database that stores the data. Upon submission, a data file status report is produced and made available to the participant. This report displays item frequencies and flags out-of-range values. If there are any out of range values or problems with the data file the submitter may review the Data File Status Report for further detail. Participants are expected to fix any errors and resubmit their data file(s) for processing. If the data have been properly received, a CAHPS Database staff member then reviews the report to conduct data quality checks. If any data problems are discovered, users will be notified immediately along with a description of the problem. If there are no problems with the data file the CAHPS Database staff review all aspects of the submission for an account final approval status and an email will be sent to the participant contact via the database submission extranet indicating their data will be included in the CAHPS Health Plan Database.

Step 7: Account Status. Participants have the opportunity to check the status of their account at any time during the submission process. Only accounts that receive the CAHPS Database Final Approval status will be included in the CAHPS Health Plan Database.

3. Methods to Maximize Response Rates

AHRQ promotes the voluntary participation in the CAHPS Health Plan Survey Database using several methods to different target audiences. We continually conduct general marketing through existing CAHPS channels and targeted outreach to existing and previous health plan participants. The CAHPS Database staff also contact national quality initiatives to promote the Database and have sought data partners that result in the yearly data contributions from NCQA and CMS.

Ongoing general marketing includes:

CAHPS Database Web Site: Announcements regarding data submission and reporting timetables;
CAHPS Database Annual Report and Related Press Release(s): Announcements for release of Annual Report that includes CAHPS Database contact information for plans and purchasers interested in participating; and
AHRQ Web Site: Brief summary of CAHPS Database products and benefits of participation, with link to the CAHPS Database website.

GovDelivery Messages: Send GovDelivery messages to participants who signed up for CAHPS News and Events.

In addition to the direct contact of health plans themselves, the CAHPS Database staff contact many national leaders and programs and direct them to the annual chartbook and references to the AHRQ National Healthcare Quality and National Healthcare Disparities Reports. These organizations and programs often cite and use CAHPS Database information. These include:

National Quality Initiatives
National Forum on Health Care Quality Measurement and Reporting (board members)
Quality Interagency Coordinating Committee (e.g., federal agencies such as HHS, Labor, Defense, Veterans Affairs, Federal Trade Commission, etc.)
Institute of Medicine Quality of Health Care in America Project

Federal and State Health Policy Leaders
Appropriate Federal and State Agency Administrators (including public health)
Federal and State Congressional Staffs
National Governors Association
National Conference of State Legislatures
National Association of Health Data Organizations
National Association for State Health Policy
State Medicaid Directors Association

Consumer Advocacy Groups
American Association of Retired Persons
Consumer Coalition for Quality Health Care
Families USA
Family Voices

Business Leaders on Health
National Business Coalition on Health
Managed Health Care Association
Leapfrog Group
Washington Business Group on Health
Midwest Business Group on Health
National Health Care Purchasers Institute

Health Care Industry Leaders
American Association of Health Plans
Health Insurance Association of America
American Health Quality Association
National Association of Insurance Commissioners

CAHPS Users and Researchers
CAHPS Database Advisory Group and Participants
CAHPS Survey Users Network (SUN)
CAHPS Consortium
CAHPS Advisory Committee
Medicare Managed Care CAHPS Technical Expert Panel
Medicare Health Outcomes Study Technical Expert Panel

Health Policy and Health Services Researchers
Grant Makers in Health
Academy for Health Services Research and Health Policy

4. Tests of Procedures

The CAHPS Database staff talks with submitters about their experience and use their feedback to improve the collection process.

5. Statistical Consultants

Ron Hays, PhD

Professor of Medicine

UCLA School of Medicine/RAND

[email protected]

1 The Medicare results are based on a representative sample of all Medicare Advantage health plans.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	SUPPORTING STATEMENT
Author	wcarroll
File Modified	0000-00-00
File Created	2021-01-14

Supporting Statement Part B