Supporting Statement Part B -- Collection of Information for CAHPS Health Plan Survey Comparative Database 2016_DRAFT_9-2-16

Supporting Statement Part B -- Collection of Information for CAHPS Health Plan Survey Comparative Database 2016_DRAFT_9-2-16.docx

Collection of Information for Agency for Healthcare Research and Qualitys (AHRQ) Consumer Assessment of Healthcare Providers and Systems (CAHPS) Health Plan Survey Comparative Database

OMB: 0935-0165

Document [docx]
Download: docx | pdf




SUPPORTING STATEMENT


Part B







Collection of Information for

Agency for Healthcare Research and Quality’s (AHRQ)

Consumer Assessment of Healthcare Providers and Systems (CAHPS)

Health Plan Survey Comparative Database





Version 9-2-2016









Agency of Healthcare Research and Quality (AHRQ)


Table of contents

B. Collections of Information Employing Statistical Methods


1. Respondent Universe and Sampling Methods


Universe of health plans and representativeness of the data. While there are many survey vendors that collect Consumer Assessment of Healthcare Providers and Systems (CAHPS) Health Plan Survey data and may maintain a database of their own clients’ data, the Agency for Healthcare Research and Quality (AHRQ) is the only entity that serves as a comprehensive repository of CAHPS survey data. However, the CAHPS Health Plan Database is comprised of data that are voluntarily submitted by health plans that have administered the CAHPS Health Plan Survey and is not a statistically selected sample, nor is it a representative sample of all health plans in the U.S. Voluntary participants include public and private employers, State Medicaid agencies, State Children's Health Insurance Programs (SCHIP), the Centers for Medicare & Medicaid Services (CMS), individual health plans, and the Department of Defense. AHRQ collaborates with CMS and the National Committee for Quality Assurance (NCQA) to combine their CAHPS Health Plan Survey data with the CAHPS Health Plan Database.

The CAHPS Health Plan Survey Database is a voluntary, comprehensive database of CAHPS Health Plan Survey responses gathered directly from Medicaid plans and CMS. NCQA is the managed care plan accreditor and requires yearly CAHPS results for accreditation. Medicare conducts a yearly survey of all Medicare plans and provides the CAHPS Health Plan Database these results. Many Medicaid program health plans do not seek NCQA accreditation, but submit instead directly to the CAHPS Health Plan Database.


The CAHPS Health Plan Survey Chartbook contains a section entitled “Data Sources and Comparison Limitations” which outlines the limitations of the data. Because of differences in the source of data submissions to the CAHPS Health Plan Database from year to year, it is not possible to directly compare results for the 2 years. Comparison of results across sectors also should be made with caution, since significant variations may exist in benefit design and other factors that might affect survey responses across sector.


The number of health plans in the U.S. is estimated to be as follows:

  • 470 HMO

  • 712 Medicare Advantage (MA and MA-PD)

  • 953 Medicaid plans


(HMO Source: http://kff.org/other/state-indicator/number-of-hmos/?currentTimeframe=0&sortModel=%7B%22colId%22:%22Location%22,%22sort%22:%22asc%22%7D

Medicaid Source: https://www.medicaid.gov/medicaid-chip-program-information/by-topics/delivery-systems/managed-care/medicaid-managed-care-enrollment-report.html

Medicare Source: https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/MCRAdvPartDEnrolData/index.html?redirect=/mcradvpartdenroldata.

The AHRQ CAHPS Health Plan Survey results for 2015 consists of 532 health plans covering 568,035 respondents which represents about 25% of the total estimated population of U.S. health plans.

Table B-1 presents the number of Medicaid, SCHIP, and Medicare survey respondents and health plan submissions included in the CAHPS Health Plan Database for 2014 and 2015. The number of health plan submissions is indicated in parentheses.


Table B-1. Number of Survey Respondents and Health Plan Submissions: 2014 and 2015

Year

Medicaid

CHIP

Medicare

(Version)

Adult

Child

Child

Adult

2015 (5.0)

61,369 (133)

91,049 (136)

13,466 (19)

155,095 (431)

2014 (5.0)

68,234 (149)

60,153(100)

11,762 (15)

195,748 (443)


Medicaid Data: The 5.0 CAHPS Health Plan Survey results for the Medicaid sector were obtained from data submitted directly to the CAHPS Health Plan Database by State Medicaid agencies and individual health plans. The 2015 database consists of submissions from 36 states, of which a total of 16 Medicaid State Agencies submitted data, compared to 31 States, 17 of which were State Medicaid agencies in 2014. Medicaid results are based on survey data collected from October through June.


Medicare Data: Each year, the CAHPS Health Plan Database receives the CAHPS Medicare Managed Care survey data collected by the Centers for Medicare & Medicaid Services (CMS). CMS collected survey information from beneficiaries enrolled in managed care health plans that provide a prescription drug benefit. For 2015, 431 managed care plans participated in the survey. Survey participants included both enrollees receiving prescription drug coverage through their health plan and those that don’t receive prescription drug coverage through their health plan. Beneficiaries enrolled in traditional Medicare (Medicare fee-for-service) as well as fee-for-service enrollees who selected a prescription drug plan were also surveyed in 2015, but these data are not represented in the Chartbook. The results in the Chartbook include only beneficiaries who were enrolled in a managed care health plan. The Medicare Survey data are collected from February through June. The Medicare results presented in the CAHPS online reporting system may differ from other reports due to the inclusion or exclusion of certain beneficiary groups and/or the use of case-mix adjustment variables.


Tables B-2 and B-3 present the number of survey respondents and health plan submissions obtained for the 5.0 results within each major sector by State, including U.S. territories and the District of Columbia.

Table B-2. 2015 Survey Respondents and Health Plan Submissions by State (5.0 Results)

State

Medicaid Adult

Medicaid Child

CHIP

Medicare Adult

Alabama

-

-

849 (1)

1,596 (4)

Arizona

-

-

-

4,516 (16)

Arkansas

400 (1)

395 (1)

437 (1)

864 (3)

California

3,504 (7)

3,902 (4)

-

10,136 (26)

Colorado

3,152 (10)

9,222 (10)

2,190 (5)

3,184 (8)

Connecticut

-

-

-

1,447 (6)

Delaware

-

-

-

363 (1)

District of Columbia

1,006 (2)

1,584 (2)

-

326 (1)

Florida

642 (2)

770 (2)

-

8,198 (28)

Georgia

840 (2)

1,740 (3)

-

3,600 (14)

Hawaii

-

2,534 (5)

653 (1)

1,872 (6)

Idaho

-

-

-

1,775 (5)

Illinois

507 (2)

2,541 (2)

-

3,713 (13)

Indiana

2,102 (3)

930 (1)

-

2,843 (8)

Iowa

-

-

-

1,167 (3)

Kansas

1,757 (3)

4,993 (3)

3,223 (2)

344 (1)

Kentucky

405 (1)

408 (1)

237 (1)

790 (3)

Louisiana

1,135 (3)

1,337 (2)

1,119 (1)

1,800 (4)

Maine

-

-

-

2,500 (4)

Maryland

4,290 (8)

8,481 (8)

-

1,288 (3)

Massachusetts

1,469 (4)

-

-

4,605 (10)

Michigan

7,362 (14)

6,074 (14)

-

5,692 (12)

Minnesota

8,664 (9)

-

-

3,888(10)

Mississippi

-

-

-

2,448 (2)

Missouri

-

-

-

3,838 (11)

Montana

-

-

-

621 (1)

Nebraska

331 (1)

652 (1)

-

969 (3)

Nevada

431 (1)

854 (1)

-

3,318 (7)

New Hampshire

328 (1)

742 (1)

-

489 (2)

New Jersey

954 (2)

1,583 (2)

-

2,293 (8)

New Mexico

1,367 (3)

1,553(2)

-

2,029 (6)

New York

1,654 (4)

8,156 (17)

-

13,227 (39)

North Carolina

-

-

-

1,341 (6)

Ohio

2,500 (5)

6,432 (5)

-

6,051 (18)

Oklahoma

434 (1)

492 (1)

514 (1)

1,338 (4)

Oregon

5,326 (17)

5,255 (17)

-

6,511 (18)

Pennsylvania

3,964 (9)

4,435 (9)

2,137 (3)

12,478 (23)

Puerto Rico

-

-

-

2,767 (10)

Rhode Island

491 (1)

-

-

637 (2)

South Carolina

743 (2)

1,036 (2)

-

811 (3)

Tennessee

412 (1)

1,067 (1)

-

2,922 (8)

Texas

823 (2)

2,608 (5)

1,039 (2)

7,401 (27)

Utah

379 (1)

-

-

2,530 (7)

Vermont

-

-

-

-

Virginia

2,006 (5)

2,689 (5)

1,068 (1)

1,730 (6)

Washington

1,249 (3)

6,680 (6)

-

3,778 (12)

West Virginia

245 (1)

579 (1)

-

3,626 (4)

Wisconsin

497 (2)

1,325 (2)

-

5,435 (15)

Totals

61,369 (133)

91,049 (136)

13,466 (19)

155,095 (431)

Table B-3. 2014 Survey Respondents and Health Plan Submissions by State (5.0 Results)

State

Medicaid Adult

Medicaid Child

CHIP

Medicare Adult

Alabama

-

-

970 (1)

2,247 (5)

Arizona

-

-

-

7,171 (17)

Arkansas

-

-

507 (1)

283 (1)

California

2,812 (6)

4,408 (4)

-

16,749 (33)

Colorado

1,028 (3)

963 (3)

2,572 (5)

3,921 (9)

Connecticut

-

-

-

1,970 (5)

Delaware

-

-

-

400 (1)

District of Columbia

935 (2)

1,671 (2)

-

-

Florida

451 (2)

-

-

11,547 (30)

Georgia

457 (1)

1,124 (2)

-

5,265 (14)

Hawaii

3,495 (7)

-

814 (1)

3,752 (8)

Idaho

-

-

-

2,197 (5)

Illinois

364 (1)

495 (1)

-

4,435 (10)

Indiana

1,578 (3)

1,523 (2)

-

2,346 (6)

Iowa

-

-

-

2,143 (5)

Kansas

1,727 (3)

3,076 (2)

3,765 (2)

394 (1)

Kentucky

329 (1)

340 (1)

255 (1)

1,296 (3)

Louisiana

1,870 (5)

3,314 (4)

1,497 (1)

2,702 (6)

Maine

-

-

-

2,832 (5)

Maryland

3,465 (7)

6,589 (7)

-

1,753 (4)

Massachusetts

1,703 (4)

-

-

4,856 (10)

Michigan

5,381 (11)

5,407 (13)

-

6,022 (10)

Minnesota

9,721 (9)

-

-

4,303 (11)

Mississippi

-

-

-

2,937 (3)

Missouri

-

-

-

5,273 (13)

Montana

-

-

-

632 (1)

Nebraska

-

-

-

1,233 (3)

Nevada

-

-

-

3,018 (8)

New Hampshire

-

-

-

756 (2)

New Jersey

2,850 (12)

3,332 (9)

-

2,715 (7)

New Mexico

1,368 (3)

741 (1)

-

2,826 (7)

New York

8,620 (19)

-

-

14,778 (33)

North Carolina

-

-

-

2,997 (8)

Ohio

2,719 (5)

6,764 (5)

-

8,523 (21)

Oklahoma

319 (1)

875 (1)

-

2,539 (6)

Oregon

5,465 (17)

5,958 (17)

-

9,523 (18)

Pennsylvania

4,071 (9)

4,418 (9)

534 (1)

10,823 (20)

Puerto Rico

-

-

-

4,113 (12)

Rhode Island

478 (1)

-

-

730 (2)

South Carolina

871 (2)

1,068 (2)

-

972 (3)

Tennessee

-

-

-

3,581 (8)

Texas

402 (1)

1,407 (3)

495 (1)

9,347 (25)

Utah

-

382 (1)

353 (1)

3,317 (8)

Vermont

254 (1)

308 (1)

-

-

Virginia

2,573 (6)

3,395 (6)

-

2,776 (7)

Washington

2,355 (5)

1,266 (2)

-

5,727 (10)

West Virginia

303 (1)

513 (1)

-

1,060 (3)

Wisconsin

270 (1)

916 (1)

-

6,968 (16)

Totals

68,234 (149)

60,153 (100)

11,762 (15)

195,748 (443)

The CAHPS Health Plan Database currently contains 17 years of data from the CAHPS Health Plan Survey. Table B-4 shows data submissions to the CAHPS Database from 1998 to 2015. The total number of respondents is presented by population sector, with the number of health plan submissions given in parentheses.

Table B-4. Data Submissions to the CAHPS Health Plan Database From 1998-2015

Year
(CAHPS Version)

Commercial

Medicaid

CHIP

Medicare

Adult

Child

Adult

Child

Child

Adult

2015 (5.0)

N/A

N/A

61,369 (133)

91,049 (136)

13,466 (19)

155,095 (431)

2014 (5.0)

N/A

N/A

68,234 (149)

60,153 (100)

11,762 (15)

195,748 (443)

2013 (5.0)

N/A

N/A

60,249

(124)

66,804

(105)

9,149

(12)

198,350 (451)

2011 (4.0)

168,341 (376)

900 (1)

73,820 (148)

85,003 (129)

26,232 (41)

163,182 (445)

2010 (4.0)

139,156 (288)

1,474 (2)

97,626 (132)

88,694 (132)

N/A

221,120 (431)

2009 (4.0)

179,528 (405)

751 (2)

63,391 (126)

68,697 (107)

N/A

206,647 (405)

2008 (4.0)

174,307

(410)

0

(0)

59,840

(120)

9,755

(29)

0

(0)

207,366

(343)

2008 (3.0)

0

(0)

1,882

(4)

0

(0)

37,347

(64)

0

(0)

0

(0)

2007 (4.0)

106,811

(239)

0

(0)

45,979

(109)

4,647

(16)

0

(0)

115,910

(296)

2007 (3.0)

0

(0)

1,659

(4)

0

(0)

64,039

(103)

0

(0)

0

(0)

2006 (3.0)

124,585

(271)

2,400

(7)

43,174

(119)

50,204

(95)

9,303

(30)

97,955

(273)

2005 (3.0)

123,272

(254)

2,661

(4)

32,115

(76)

40,204

(65)

1,252

(3)

127,930

(276)

2004 (3.0)

111,680
(223)

7,024

(12)

59,515

(149)

86,159

(128)

16,657

(29)

132,420

(288)

2003 (3.0)

114,063
(216)

1,866
(4)

39,275
(112)

31,081
(69)

19,061
(49)

141,421
(295)

2002 (2.0)

94,546
(219)

5,600
(10)

48,109
(136)

60,534
(122)

18,910
(43)

153,172
(321)

2001 (2.0)

165,500
(266)

9,913
(24)

45,127
(142)

36,940
(124)

0
(0)

179,451
(381)

2000 (2.0)

135,479
(270)

2,760
(8)

49,327
(156)

41,400
(140)

0
(0)

166,072
(367)

1999 (2.0)

168,234

(307)

42,879

(149)

28,420

(77)

14,106

(66)

0

(0)

0

(0)

1998 (1.0)

34,965

(54)

0

(0)

23,519

(31)

9,871

(33)

0

(0)

0

(0)

TOTALS

1,840,467

81,769


899,089

946,687

125,792

2,461,839


Most of the CAHPS Health Plan Survey questions ask respondents to report on their experiences with different aspects of their care. These reporting questions are combined into groups that address the same aspect of care or service to arrive at a broader assessment. The 5.0 version of the CAHPS Adult and Child Health Plan Surveys reporting questions fall into four major “composites” that summarize consumer experiences in the following areas: 1) Getting needed care, 2) Getting care quickly, 3) How well doctors communicate, and 4) Health plan information & customer service.


The CAHPS Health Plan Survey collects four separate global ratings to distinguish between important aspects of care. The four questions ask plan enrollees to rate their experiences in the past 6 months with: 1) their personal doctor, 2) the specialist they saw most often, 3) health care received from all doctors and, 4) their health plan. Ratings are scored on a 0 to 10 scale, where 0 is the “worst possible” and 10 is the “best possible.” The ratings are analyzed and presented in the three-part bar chart display used in the CAHPS Health Plan Survey reports: the percentage of respondents who gave a rating of either 0-6, 7-8, or 9-10. This three-part scale is used because testing by the CAHPS team determined that these cut-points improve the ability to discriminate among plans while simplifying the presentation of results.


Case-Mix Adjustment. Case-mix refers to the respondents’ health status and other socio-demographic characteristics that have been shown to affect enrollee reports and ratings of health plans. Characteristics used to case-mix adjust CAHPS Health Plan scores, where applicable, are respondent age, education, and self-reported physical and mental health status.


Testing for Statistical Differences. The individual participant’s comparative reports test for statistically significant differences between mean scores and ratings of individual health plans and the mean of all plan means in the CAHPS Health Plan Database using the t-test. A significance level of 0.05 or less is considered statistically significant. As described in the previous sections, the mean scores are adjusted for case-mix differences before the statistical tests are applied. To compute the means, reports and rating responses are grouped into three categories and assigned a score of 1, 2, or 3. Then, significance tests for both the reports and ratings are conducted on the mean scores. Individual plan results that differ significantly from the overall mean are denoted by arrows, either pointing up (significantly higher than the overall mean) or down (significantly lower than the overall mean).


2. Information Collection Procedures

Information collection for the AHRQ CAHPS Health Plan Survey Database occurs in a regular data collection cycle each year in June/July. The information collection procedure for submitting and processing data for the database is shown in Figure B-1. Each of the steps is described below. Screen shots of each step are provided in Attachment F.


Figure B-1. CAHPS Health Plan Database Data Submission Process

Shape1







Step 1: Call for Data Submission. State Medicaid agencies and health plans that have administered the CAHPS Health Plan survey are recruited through multiple outlets and asked to submit to the database. The call for data submission is done through various publicity sources such as AHRQ’s electronic newsletters, GovDelivery messages, and communication with prior year’s participants. Organizations that have administered the Health Plan Survey and are interested in participating in the CAHPS Health Plan Database communicate with Westat through a dedicated email address ([email protected]) that routes directly to Westat and a toll-free phone number (1-888-808-7108) to indicate their interest in participation.


Step 2: Complete Registration Form. A database submission extranet has been set up so that interested parties such as, state Medicaid agencies, coalitions, vendors, and health plans register for the data submission process. Information about eligibility requirements, benefits of participation, data use agreement, and data file specifications regarding how to prepare their data file for inclusion in the CAHPS Health Plan Database are posted on the extranet. The data file specifications ensure that data files received from users are standardized and consistent in the way variables are coded and formatted. Potential participants’ online registration information will be reviewed by Westat staff. Upon approval of the registrant, an automated email that contains a username and password to the data submission extranet is sent to the registered user. The automatic email informs registrants of the information needed in the next steps of the data submission process.


Step 3: Complete Health Plan Information Form. This step requires each health plan that administered the CAHPS Health Plan survey, submit the requested characteristics including such details as the name of the plan, product type (e.g., HMO, PPO), the population surveyed (e.g., adult Medicaid, child Medicaid), plan State, total enrollment at time sample frame was generated, mode of survey administration and how the sample was selected.


Step 4: Submit Data Use Agreement. To protect the confidentiality of all respondents and entities that are included in any CAHPS Database, all participating institutions must sign a data use agreement (DUA) that has been reviewed and approved by AHRQ. The data use agreement specifies how the submitted data will be used, provides assurance that the identity of the participating institution will be protected and ensures the confidentiality of the data. Data are not included in the database without this signed data use agreement. Users can fax and/or mail a copy of the signed agreement. Data collection vendors may not sign and submit the DUA on behalf of an institution (even if they have been given permission by the entity to handle the actual submission of data). Only a duly appointed representative may sign the DUA.


Step 5: Upload Questionnaire. Each health plan must upload a copy of the questionnaire used. The CAHPS Database reviews the questionnaire to ensure that it meets CAHPS Health Plan Survey standards (the survey instrument must include all core questions, not alter the wording of any core questions, and must not omit any of the survey items related to respondent characteristics that are used for case mix adjustment.) Once the questionnaire is reviewed by CAHPS Database staff, an email notification is sent to the registrant within three business days with an approval or rejection. Only health plans that receive questionnaire approval may submit data files.


Step 6: Secure Online Data Submission. To enable participants to transmit their CAHPS survey data to Westat in a secure manner, an online data submission extranet has been developed. The online system will be expanded and adapted to include data submission for all CAHPS surveys. Data are accepted in ascii/flat format. Data files must conform to the Data File Layout Specifications provide by the CAHPS Database. Since the unit of analysis is at the health plan level, users must upload one data file per health plan.


Data File Approval. Once a data file is successfully uploaded, a separate load program developed in Visual Basic (VB) reads the submitted files and loads them into the SQL database that stores the data. Upon submission, a data file status report is produced and made available to the participant. This report displays item frequencies and flags out-of-range values. If there are any out of range values or problems with the data file the submitter may review the Data File Status Report for further detail. Participants are expected to fix any errors and resubmit their data file(s) for processing. If the data have been properly received, a CAHPS Database staff member then reviews the report to conduct data quality checks. If any data problems are discovered, users will be notified immediately along with a description of the problem. If there are no problems with the data file the CAHPS Database staff review all aspects of the submission for an account final approval status and an email will be sent to the participant contact via the database submission extranet indicating their data will be included in the CAHPS Health Plan Database.


Step 7: Account Status. Participants have the opportunity to check the status of their account at any time during the submission process. Only accounts that receive the CAHPS Database Final Approval status will be included in the CAHPS Health Plan Database.


3. Methods to Maximize Response Rates


AHRQ promotes the voluntary participation in the CAHPS Health Plan Survey Database using several methods to different target audiences. We continually conduct general marketing through existing CAHPS channels and targeted outreach to existing and previous health plan participants. The CAHPS Database staff also contact national quality initiatives to promote the Database and have sought data partners that result in the yearly data contributions from NCQA and CMS.



Ongoing general marketing includes:


  • CAHPS Database Web Site: Announcements regarding data submission and reporting timetables;

  • CAHPS Database Annual Report and Related Press Release(s). Announcements for release of Annual Report that includes CAHPS Database contact information for plans and purchasers interested in participating; and

  • AHRQ Web Site. Brief summary of CAHPS Database products and benefits of participation, with link to the CAHPS Database website.

On a yearly basis, current and prior participants are contacted by phone and e-mail to determine their plans to participate in the upcoming cycle. These steps include:

  1. Send GovDelivery messages to participants who signed up for CAHPS News and Events.


Attachment G contains the e-mails that are used to solicit participation.


In addition to the direct contact of health plans themselves, the CAHPS Database staff contact many national leaders and programs and direct them to the annual chartbook and references to the AHRQ National Healthcare Quality and National Healthcare Disparities Reports. These organizations and programs often cite and use CAHPS Database information. These include:


National Quality Initiatives
National Forum on Health Care Quality Measurement and Reporting (board members)
Quality Interagency Coordinating Committee (e.g., federal agencies such as HHS, Labor, Defense, Veterans Affairs, Federal Trade Commission, etc.)
Institute of Medicine Quality of Health Care in America Project

Federal and State Health Policy Leaders
Appropriate Federal and State Agency Administrators (including public health)
Federal and State Congressional Staffs
National Governors Association
National Conference of State Legislatures
National Association of Health Data Organizations
National Association for State Health Policy
State Medicaid Directors Association

Consumer Advocacy Groups
American Association of Retired Persons
Consumer Coalition for Quality Health Care
Families USA
Family Voices

Business Leaders on Health
National Business Coalition on Health
Managed Health Care Association
Leapfrog Group
Washington Business Group on Health
Midwest Business Group on Health
National Health Care Purchasers Institute

Health Care Industry Leaders
American Association of Health Plans
Health Insurance Association of America
American Health Quality Association
National Association of Insurance Commissioners

CAHPS Users and Researchers
CAHPS Database Advisory Group and Participants
CAHPS Survey Users Network (SUN)
CAHPS Consortium
CAHPS Advisory Committee
Medicare Managed Care CAHPS Technical Expert Panel
Medicare Health Outcomes Study Technical Expert Panel

Health Policy and Health Services Researchers
Grant Makers in Health
Academy for Health Services Research and Health Policy


4. Tests of Procedures


The CAHPS Database staff talks with submitters about their experience and use their feedback to improve the collection process.

5. Statistical Consultants

Ron Hays, PhD

Professor of Medicine

UCLA School of Medicine/RAND

[email protected]

File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File TitleSUPPORTING STATEMENT
Authorwcarroll
File Modified0000-00-00
File Created2021-01-22
© 2024 OMB.report | Privacy Policy