Supporting Statement Part B -- CAHPS Clinician & Group Survey Comparative Database

SUPPORTING STATEMENT

Part B

Collection of Information for

Agency for Healthcare Research and Quality’s (AHRQ)

Consumer Assessment of Healthcare Providers and Systems (CAHPS)

Clinician and Group Survey Comparative Database

May 2015

Agency of Healthcare Research and Quality (AHRQ)

Table of Contents

B. Collections of Information Employing Statistical Methods 1

1. Respondent Universe and Sampling Methods 1

2. Information Collection Procedures 5

3. Methods to Maximize Response Rates 8

4. Tests of Procedures 9

5. Statistical Consultants 9

List of Attachments 9

B. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The AHRQ Consumer Assessment of Healthcare Providers and Systems (CAHPS) Clinician and Group (CG) Survey Database serves as a central repository, the largest database of the CAHPS CG survey results. Currently, comparative results from over 2,100 clinicians and groups that voluntarily submitted data to the CAHPS Database are included. These participating organizations do not constitute a representative sample of all clinicians and groups in the U.S. however; users can compare their results to similar organizations. Similar to the CAHPS Health Plan Database, the CAHPS CG Survey results can be viewed on the CAHPS web site at https://www.cahpsdatabase.ahrq.gov/CAHPSIDB/Public/about.aspx. A section on data limitations is included in the CG Comparative Reports that outline the limitations of the data.

Universe of medical offices and representativeness of the data. Quantifying the universe of clinicians and groups is not straightforward. Clinician practices and groups are typically characterized as either medical practices with 1 or 2 physicians or medical groups consisting of 3 or more physicians. Participation in the Clinician and Group comparative database is open to all practices, clinicians, provider care services, and groups that administer the surveys according to the CAHPS specifications. According to the U.S. Census Bureau's 2012 Economic Census, (2012 NAICS code 6211 “Offices of physicians”), there were 221,470 physicians’ offices in the U.S.

(http://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?pid=ECN_2012_US_62I1&prodType=table)

A 2005 Health Affairs article examining group medical practices, whose lead author is from the Medical Group Management Association, states: “The total number of U.S. group practices is not known, but we estimate it to be somewhat larger than the 34,490 practices we identified, perhaps in the range of 40,000–50,000” (David Gans, John Kralewski, Terry Hammons, and Bryan Dowd, “Medical Groups’ Adoption Of Electronic Health Records And Information Systems,” Health Affairs, 2005, Vol 24 (5), pp. 1323-1333.)

The most relevant and thorough source of data on the population of medical group practices in the U.S. is the American Medical Association’s (AMA) 1999 edition of “Medical Group Practices in the U.S.: A Survey of Practice Characteristics.” This report is the only (and most recent) source that reports information about the characteristics of medical group practices in the U.S., with groups defined as those with 3 or more physicians. A total of 34,066 eligible medical groups were identified in this census conducted by the AMA in 1996. The problem with comparing these group practices to individual medical offices is that a single group practice can comprise several different medical office locations. The AMA report only includes data from the parent or primary location of group practices, and therefore is also an underestimate of the actual number of group medical offices.

Table 1 shows the number of CAHPS Clinician & Group Survey database medical practices that participated in the 2013 CG Database compared to the number of offices of physicians based on the 2012 U.S. Economic Census (http://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?pid=ECN_2012_US_62I1&prodType=table).

Table 1. Total of CAHPS CG Database Medical Practices (2013) and
U.S. Economic Census, Offices of Physicians (2012)

	2013 CAHPS Clinician & Group Comparative Database	2012 Census
	Number	Number
TOTAL	2,172	221,470

Because there is not a recent and comprehensive source of data describing the population of group medical offices in the U.S. by specific characteristics, we do not present comparisons of the CAHPS CG database to any other population statistics. Only descriptive statistics about the database medical practices are provided.

Statistics from the 2013 CAHPS CG Survey Comparative Results Report. The following tables provide medical practice characteristics data for the total of 2,172 participating organizations included in the 2013 CAHPS CG Database report. Medical practice characteristics were obtained from each participating medical practice included in the 2013 CAHPS CG Survey database. The database contains characteristics such as region, physician specialty, practice ownership and affiliation, and number of visits per practice. Tables 2 and 3 show the distribution of practices by physician specialty and by practice ownership and affiliation.

Physician Specialty. Table 2 includes distributions of the 16 most reported physician specialties. The largest concentration of practice sites across all the CG-CAHPS survey versions was “family practice,” followed by “internal medicine.” Since more than one specialty can be attributed to a given practice site, the total number of practice sites included across all specialty categories may be greater than the absolute number of practice sites.

Table 2. Distribution of Practice Site Counts by Physician Specialty, 2013¹,

Physician Specialty	12-Month Adult/ PCMH Combined		Visit Adult		12-month Child/ PCMH Combined
	Practice Site Count	Percent	Practice Site Count	Percent	Practice Site Count	Percent
Cardiology	214	7%	29	3%	-	-
Dermatology	177	6%	22	2%	-	-
Endocrinology	176	6%	21	2%	-	-
Family Practice	404	13%	341	34%	37	47%
Gastroenterology	187	6%	28	3%	-	-
Hematology/ Oncology	179	6%	15	1%	-	-
Internal Medicine	323	11%	171	17%	17	22%
Neurology	184	6%	33	3%	-	-
OB/GYN	220	7%	82	8%	-	-
Ophthalmology	172	6%	31	3%	-	-
Orthopedics	83	3%	37	4%	-	-
Pediatrics	-	-	29	3%	25	32%
Pulmonary Medicine	146	5%	27	3%	-	-
Rheumatology	163	5%	21	2%	-	-
Surgical	230	8%	94	9%	-	-
Urology	181	6%	27	3%	-	-
TOTAL	3,039	100%	1,008	100%	79	100%

Note: Column percent totals may not add exactly to 100% because of rounding

Practice Ownership and Affiliation. The distribution of the practice ownership and affiliation categories is shown in Table 3. The largest number of practice sites was “hospital/health system” across all survey versions, followed by “provider/physician.” In addition, some of the categories are not mutually exclusive and could therefore misrepresent the true distribution among the practice sites included in the database.

Table 3. Distribution of Practices by Ownership and Affiliation, 2013²,

Practice Ownership and Affiliation	12-Month Adult/ PCMH Combined		Visit Adult		12-month Child/ PCMH Combined
	Practice Site Count	Percent	Practice Site Count	Percent	Practice Site Count	Percent
Provider/ Physician	177	21%	291	24%	-	-
Hospital/ Health System	496	60%	893	73%	73	71%
University/ Academic Med Center	73	9%	-	-	30	29%
Community Health Center	-	-	47	4%	-	-
Other	80	10%	-	-	-	-
TOTAL	826	100%	1,231	100%	103	100%

Note: Column percent totals may not add exactly to 100% because of rounding

Comparative results and explanation of how results are calculated.

The CAHPS Database adjusts the survey results in order to account for factors that may affect scores for the practice, clinician, or other entity that are beyond the control of the entity. Without an adjustment, differences between entities could be due to differences in these exogenous factors rather than to true differences in performance. CAHPS data are most commonly adjusted for respondent characteristics (i.e. case mix adjustments), but can also be adjusted for other factors such as the mode of survey administration (telephone, interactive voice response, or Web/Internet). The adjusted results are reported in the online reporting system.

Case-mix adjustments. Case mix refers to the respondents’ health status and other socio-demographic characteristics that have been shown to affect patient reports and ratings of practice sites, clinicians, or other entities. Characteristics used to case-mix adjust CAHPS CG scores are respondent age, education, and self-reported health status.

Testing for Statistical Differences. Statistical tests (t-tests) are used to determine whether a participating organization's mean item or composite score is significantly above or below the overall mean. These statistical tests are based on a participating organization's mean item score or composite score rather than top box scores. Top box scores are the percent of respondents who choose the most positive score for a given item. These scores are case-mix adjusted by patient characteristics. If an organization's mean item or composite score is significantly higher or lower than the overall mean, an ‘up’ or ‘down’ arrow is assigned respectively. If there is no significant difference between the organization and the overall mean, no arrow is assigned.

2. Information Collection Procedures

Information collection for the AHRQ CAHPS CG Survey Database occurs in an annual data collection cycle. Information collection procedures for submitting and processing data are shown in Figure B-1.

Figure B-1. CAHPS Clinician and Group Database Data Submission Process

Step 1

Call for Data Submission

Step 2

Registration for Potential Participants

Step 3

Verify Account by Email

Step 4

Upload or Submit Data Use Agreement

Step 5

Upload Questionnaire

Step 6

Upload Data File(s)

CAHPS Database Review and Approval of Questionnaire

CAHPS Database Review and Approval of the Data Use Agreement

CAHPS Database Review and Approval of Data File(s)

Step 7

Final Review of All Submission Items and Final Approval

Email Notification of Final CAHPS Database Approvals Sent to Registrant.

Step 1: Call for Data Submission. Announcements about the opening of data submission go out through various publicity sources. AHRQ’s electronic newsletter and communications target approximately 50,000 subscribers. In addition, the AHRQ CAHPS News and Events listserv targets approximately 38,000 subscribers. An example of an email announcement calling for data submission is shown in Attachment E, Email # 1: Call for Data Submission. Reminder emails are sent one and two weeks after the initial email announcing the call for data submission. In addition, the AHRQ CAHPS Web site posts public information about the yearly timeline and instructions for data submission. Through these efforts, U.S. medical offices are made aware of and invited to voluntarily submit their survey data to the database. As the administrator of the database and under contract with AHRQ, Westat provides free technical assistance to medical offices and their vendors through a dedicated email address ([email protected] that routes to Westat) and toll-free phone number (888-808-7108).

Step 2: Registration for Potential Participants. A secure data submission Web site allows interested parties such as medical groups and offices to register and submit data. Registrants are asked to provide contact and other basic information and create a unique ID and password. See Attachment F: CAHPS Clinician and Group Data Submission Registration Form.

Step 3: Verify Account by Email. Once a submitter has registered and is deemed eligible to submit data, an automated email is sent to provide them with the username and password and information needed to activate their account. See Attachment E, Email # 2: Notice to Activate Account. Once users have a username and password and have activated their account, they can enter the main page menu of the Web site. Information about eligibility requirements, data use agreements, and data file specifications regarding how to prepare their data for inclusion in the CAHPS CG database is posted and can be reviewed.

Step 4: Submit Data Use Agreement. To protect the confidentiality of all participating medical offices, a duly authorized representative from the group or medical office must sign a data use agreement (DUA). The DUA language was reviewed and approved by AHRQ’s general counsel. The DUA states that the group or medical office’s data are handled in a secure manner using necessary administrative, technical and physical safeguards to limit access to it and maintain its confidentiality. In addition, the DUA states the data are used for the purposes of the database, that only aggregated results are publicly reported, and that the medical office is not identified by name. Data are not included in the database without this signed data use agreement. Users fax, email, mail or upload a signed copy of the DUA.

Step 5: Upload Questionnaire. Each registered participating organization must upload a copy of the questionnaire used. See Attachment F: Questionnaire Upload Form. The CAHPS Database reviews the questionnaire to ensure that it meets CAHPS CG Survey standards (the survey instrument must include all core questions, not alter the wording of any core questions, and must not omit any of the survey items related to respondent characteristics that are used for case mix adjustments.) Once the questionnaire is reviewed, an email notification is sent to the registrant within three business days with an approval or rejection. See Attachment E: Email #3 Questionnaire File Approved and Email #4 Questionnaire File Rejected. Only participants that receive questionnaire approval may submit data files.

Step 6: Enter Survey Administration Information and Upload Data Files. Each registered submitter must enter survey administration information (mode of administration, response rate, field period, number sampled) and upload their CAHPS CG survey data file(s). Data are submitted through a secure data submission Web site to ensure confidential transmission of the survey data. See Attachment F Survey Administration Information and Data File Upload Form. If a user has multiple medical practices within a medical group, users can upload one file that identifies all of the medical practices in their group. Data files must conform to the Data File Layout Specifications provided on the AHRQ CAHPS Web site.

Once a data file is uploaded, a Visual Basic program reads the submitted files and loads them into the SQL database that stores the data. A data file status report is then produced and made available to the submitter. This report displays item frequencies and flags out-of-range values and any possible errors. If there are any problems, the participating organization is notified by email and may review the Data File Status Report for further detail. See Attachment E: Email # 5 Data File Rejected. Submitters are expected to correct any errors and resubmit. Once there are no problems with the data file(s), an email is sent to the participating organization's point of contact indicating their data has been approved. See Attachment E: Email # 6 Data File Approved.

Step 7: Review of Submission and Final Approval. When all of the information required for submission is submitted and approved, an email is sent to the participating organization indicating that their submission has received final acceptance into the CG database. See Attachment E: Email #7 CAHPS CG Database Final Approval.

3. Methods to Maximize Response Rates

AHRQ makes a number of toolkit materials available to assist medical offices with the CAHPS CG surveys. The CAHPS Survey and Reporting Kit explains how to prepare and field a CAHPS questionnaire, analyze the results, and produce consumer-friendly reports. These kits include: survey instruments, protocols and related guidance, sample documents to help administer the survey, analysis programs, instructions for using analysis programs and reporting composites. A reporting resources kit provides sample materials and documents that provide guidance to Medical offices who wish to produce public reports of CAHPS data.

Announcements about the opening of data submission go out through various publicity sources as a way to boost medical office participation in the database. AHRQ’s electronic newsletter and communications target approximately 50,000 subscribers and the AHRQ CAHPS News and Events listserv targets approximately 38,000 subscribers. In addition, email announcements are sent to survey users who have at some point requested technical assistance or who have used the CG survey. AHRQ, through its contractor Westat, provides free technical assistance to users through a dedicated email box and toll-free phone number. In addition, reminders are sent to database registrants to remind them of the deadline for data submission.

A training video on the data submission process is also available on the Web site at https://www.cahpsdatabase.ahrq.gov/files/CG/CG%20Training.swf. This video is designed for new and previous participants and anyone that wants a more comprehensive orientation to the Clinician and Group data submission process.

4. Tests of Procedures

Input and Feedback for the Development of the CAHPS Database Submission System. The CAHPS CG Database has modeled its data submission processes after those utilized by the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Health Plan Database that has been in operation for many years. In addition, each year the CAHPS Database staff talks with submitters about their experience and use their feedback to improve the collection process. Information compiled from the CAHPS CG Survey Database, as well as feedback obtained during provision of technical assistance for both the Health Plan Survey Database and the Clinician and Group Survey Database each year the databases have been running has been used to improve the CAHPS CG online data submission and reporting system and process over time.

5. Statistical Consultants

Ron Hays, PhD

Professor of Medicine

UCLA School of Medicine/RAND

[email protected]

List of Attachments:

Attachment A: Clinician and Group Data Submission System Registration Form

Attachment B: Clinician and Group Database Data Use Agreement

Attachment C: 60 Day Federal Register Notice

Attachment D: CAHPS Technical Expert Panel Members

Attachment E: CAHPS Clinician and Group Database Data Submission Emails to Users

Attachment F: Data Submission Secure Web Site and Information Collection Forms

1 For all CG Database reports, when reporting comparison scores by medical practice site characteristic categories, a category’s results are suppressed if there are fewer than five practices and/or fewer than 300 completed surveys available for that category. For more information see “CAHPS Clinician and Group Database; How Results Were Calculated” https://www.cahpsdatabase.ahrq.gov/CAHPSIDB/Public/Files/Doc4_How_Results_are_Calculated_2013.pdf

2 For all CG Database reports, when reporting comparison scores by medical practice site characteristic categories, a category’s results are suppressed if there are fewer than five practices and/or fewer than 300 completed surveys available for that category. For more information see “CAHPS Clinician and Group Database; How Results Were Calculated” https://www.cahpsdatabase.ahrq.gov/CAHPSIDB/Public/Files/Doc4_How_Results_are_Calculated_2013.pdf

2

DRAFT 05-2015