Supporting Statement Part B -- CAHPS Clinician & Group Survey Database

SUPPORTING STATEMENT

Part B

Collection of Information for

Agency for Healthcare Research and Quality’s (AHRQ)

Consumer Assessment of Healthcare Providers and Systems (CAHPS)

Clinician and Group Survey Database

April 2018

Agency of Healthcare Research and Quality (AHRQ)

Table of Contents

B. Collections of Information Employing Statistical Methods 1

1. Respondent Universe and Sampling Methods 1

2. Information Collection Procedures 5

3. Methods to Maximize Response Rates 7

4. Tests of Procedures 8

5. Statistical Consultants 8

List of Attachments 8

B. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The AHRQ Consumer Assessment of Healthcare Providers and Systems (CAHPS) Clinician and Group (CG) Survey Database serves as a central repository, the largest database of the CAHPS CG survey results. Currently, survey results from approximately 14,000 practice sites and 3 million respondents that voluntarily submitted data to the CAHPS Database are included. These do not constitute a representative sample of all clinicians and groups in the U.S. however; users can compare their results to similar organizations. Similar to the CAHPS Health Plan Database, the CAHPS CG Survey results can be viewed on the CAHPS web site at https://www.cahpsdatabase.ahrq.gov/CAHPSIDB/Public/about.aspx. A section on data limitations is included in the CG Survey Reports that outline the limitations of the data.

Universe of medical offices and representativeness of the data. Quantifying the universe of clinicians and groups is not straightforward. Clinician practices and groups are typically characterized as either medical practices with 1 or 2 physicians or medical groups consisting of 3 or more physicians. Participation in the Clinician and Group survey database is open to all practices, clinicians, provider care services, and groups that administer the surveys according to the CAHPS specifications. According to the U.S. Census Bureau's 2012 Economic Census, (2012 NAICS code 6211 “Offices of physicians”), there were 220,890 physicians’ offices in the U.S.

(https://www.census.gov/EconomicCensus).

A 2005 Health Affairs article examining group medical practices, whose lead author is from the Medical Group Management Association, states: “The total number of U.S. group practices is not known, but we estimate it to be somewhat larger than the 34,490 practices we identified, perhaps in the range of 40,000–50,000” (David Gans, John Kralewski, Terry Hammons, and Bryan Dowd, “Medical Groups’ Adoption Of Electronic Health Records And Information Systems,” Health Affairs, 2005, Vol 24 (5), pp. 1323-1333.)

The most relevant and thorough source of data on the population of medical group practices in the U.S. is the American Medical Association’s (AMA) 1999 edition of “Medical Group Practices in the U.S.: A Survey of Practice Characteristics.” This report is the only (and most recent) source that reports information about the characteristics of medical group practices in the U.S., with groups defined as those with 3 or more physicians. A total of 34,066 eligible medical groups were identified in this census conducted by the AMA in 1996. The problem with comparing these group practices to individual medical offices is that a single group practice can comprise several different medical office locations. The AMA report only includes data from the parent or primary location of group practices, and therefore is also an underestimate of the actual number of group medical offices. Table 1 shows the number of CAHPS Clinician & Group Survey database medical practices that participated in the 2016 CG Database compared to the number of offices of physicians based on the 2012 U.S. Economic Census (https://www.census.gov/EconomicCensus).

Table 1. Total of CAHPS CG Database Medical Practices (2016) and
U.S. Economic Census, Offices of Physicians (2012)

	2016 CAHPS Clinician & Group Survey Database	2012 Census
	Number	Number
TOTAL	1,587	220,890

Because there is not a recent and comprehensive source of data describing the population of group medical offices in the U.S. by specific characteristics, we do not present comparisons of the CAHPS CG database to any other population statistics. Only descriptive statistics about the database medical practices are provided.

Statistics from the 2016 CAHPS CG Survey Results Report. The following tables provide medical practice characteristics data for the total of 1,587 participating organizations included in the 2016 CAHPS CG Database report. Medical practice characteristics were obtained from each participating medical practice included in the 2016 CAHPS CG Survey database. The database contains characteristics such as region, physician specialty, practice ownership and affiliation, and number of visits per practice. Tables 2 and 3 show the distribution of practices by physician specialty and by practice ownership and affiliation.

Physician Specialty. Table 2 includes distributions of the 21 most reported physician specialties. The largest concentration of practice sites across the Adult Survey 2.0 was “hematology/oncology” followed by “family practice” and the Adult Survey 3.0 was “family practice” followed by “internal medicine” and Child Survey 3.0 was “pediatrics” followed by “family practice”. Since more than one specialty can be attributed to a given practice site, the total number of practice sites included across all specialty categories may be greater than the absolute number of practice sites.

Table 2. Distribution of Practice Site Counts by Physician Specialty, 2016¹,

Physician Specialty	Adult Survey 2.0		Adult Survey 3.0		Child Survey 3.0
	Practice Site Count	Percent	Practice Site Count	Percent	Practice Site Count	Percent
Allergy/Immunology	127	5.4%	9	1.0%	NA	NA
Cardiology	162	6.9%	35	3.8%	NA	NA
Dermatology	155	6.6%	20	2.2%	NA	NA
Endocrinology	137	5.9%	26	2.8%	NA	NA
Family Practice	202	8.7%	276	30.0%	21	26.3
Gastroenterology	152	6.5%	19	2.1%	NA	NA
Hematology/Oncology	239	10.2%	28	3.0%	NA	NA
Internal Medicine	173	7.4%	154	16.8%	8	10%
Neurology	151	6.5%	32	3.5%	NA	NA
OB/GYN	164	7.0%	50	5.4%	NA	NA
Ophthalmology	156	6.7%	19	2.1%	0	0%
Orthopedics	64	2.7%	42	4.6%	5	6.3
Pediatrics	0	0%	12	1.3%	46	57.5
Physical Medicine & Rehabilitation	NA	NA	13	1.4%	0	0%
Podiatry	NA	NA	24	2.6%	NA	NA
Psychiatry	NA	NA	7	0.8%	0	0%
Pulmonary Medicine	NA	NA	23	2.5%	NA	NA
Radiology	141	6.0%	NA	NA	0	0%
Rheumatology	150	6.4%	32	3.5%	NA	NA
Surgery	159	6.8%	76	8.3%	NA	NA
Urology	NA	NA	22	2.4%	0	0%
Total	2,332	100%	919	100%	80	100%

Note: NA = fewer than five practices to report

Practice Ownership and Affiliation. The distribution of the practice ownership and affiliation categories is shown in Table 3. The largest number of practice sites was “provider/physician” for Adult Survey 2.0 and Child Survey 3.0, followed by “hospital/health system.” The largest number of practice sites was “hospital/health system” for Adult 3.0 followed by “provider/physician”. In addition, some of the categories are not mutually exclusive and could therefore misrepresent the true distribution among the practice sites included in the database.

Table 3. Distribution of Practices by Ownership and Affiliation, 2016²,

Practice Ownership and Affiliation	Adult Survey 2.0		Adult Survey 3.0		Child Survey 3.0
	Practice Site Count	Percent	Practice Site Count	Percent	Practice Site Count	Percent
Provider/ Physician	257	30.2%	232	35.4%	47	58.0%
Hospital/ Health System	85	10.0%	257	39.2%	21	25.9%
University/ Academic Med Center	0	0.0%	145	22.1%	11	13.6%
Community Health Center	6	0.7%	8	1.2%	0	0.0%
Other	502	59.1%	14	2.1%	2	2.5%
TOTAL	850	100%	656	100%	81	100%

Survey results and explanation of how results are calculated.

The CAHPS Database adjusts the survey results in order to account for factors that may affect scores for the practice, clinician, or other entity that are beyond the control of the entity. Without an adjustment, differences between entities could be due to differences in these exogenous factors rather than to true differences in performance. CAHPS data are most commonly adjusted for respondent characteristics (i.e. case mix adjustments), but can also be adjusted for other factors such as the mode of survey administration (telephone, interactive voice response, or Web/Internet). The adjusted results are reported in the private Excel^® report.

Case-mix adjustments. Case mix refers to the respondents’ health status and other socio-demographic characteristics that have been shown to affect patient reports and ratings of practice sites, clinicians, or other entities. Characteristics used to case-mix adjust CAHPS CG scores are respondent age, education, and self-reported health status.

Testing for Statistical Differences. Statistical tests (t-tests) are used to determine whether a participating organization's mean item or composite score is significantly above or below the overall mean. These statistical tests are based on a participating organization's mean item score or composite score rather than top box scores. Top box scores are the percent of respondents who choose the most positive score for a given item. These scores are case-mix adjusted by patient characteristics. If an organization's mean item or composite score is significantly higher or lower than the overall mean, an ‘up’ or ‘down’ arrow is assigned respectively. If there is no significant difference between the organization and the overall mean, no arrow is assigned.

2. Information Collection Procedures

Information collection for the AHRQ CAHPS CG Survey Database occurs in an annual data collection cycle. Information collection procedures for submitting and processing data are shown in Figure B-1.

Figure B-1. CAHPS Clinician and Group Database Data Submission Process

Step 1

Call for Data Submission

Step 2

Registration for Potential Participants

Step 3

Verify Account by Email

Step 4

Upload or Submit Data Use Agreement and Excel^® Practice Site File

Step 5

Upload Questionnaire

CAHPS Database Review and Approval of Questionnaire

CAHPS Database Review and Approval of the Data Use Agreement and Excel^® Practice Site File

Step 6

Upload Data File(s)

CAHPS Database Review and Approval of Data File(s)

Step 7

Final Review of All Submission Items and Final Approval

Email Notification of Final CAHPS Database Approvals Sent to Registrant.

Step 1: Call for Data Submission. Announcements about the opening of data submission go out through various publicity sources. AHRQ’s electronic newsletter and communications target approximately 50,000 subscribers. In addition, the AHRQ CAHPS News and Events listserv targets approximately 40,000 subscribers. Reminder emails are sent two weeks after the initial email announcing the call for data submission. In addition, the AHRQ CAHPS Web site posts public information about the yearly timeline and instructions for data submission. Through these efforts, U.S. medical offices are made aware of and invited to submit their survey data voluntarily to the database. As the administrator of the database and under contract with AHRQ, Westat provides free technical assistance to medical offices and their vendors through a dedicated email address ([email protected] that routes to Westat) and toll-free phone number (888-808-7108).

Step 2: Registration for Potential Participants. A secure data submission Web site allows interested parties such as medical groups and offices to register and submit data. Registrants are asked to provide contact and other basic information and create a unique ID and password. See Attachment F: Figure 2. Registration Form.

Step 3: Verify Account by Email. Once a submitter has registered and is deemed eligible to submit data, an automated email is sent to provide them with the username and password and information needed to activate their account. See Attachment E, Email # 1: Notice to Activate Account. Once users have a username and password and have activated their account, they can enter the main page menu of the Web site. Information about eligibility requirements, data use agreements, and data file specifications regarding how to prepare their data for inclusion in the CAHPS CG database is posted and can be reviewed.

Step 4: Submit Data Use Agreement. To protect the confidentiality of all participating medical offices, a duly authorized representative from the group or medical office must sign a data use agreement (DUA). The DUA language was reviewed and approved by AHRQ’s general counsel. The DUA states that the group or medical office’s data are handled in a secure manner using necessary administrative, technical and physical safeguards to limit access to it and maintain its confidentiality. In addition, the DUA states the data are used for the purposes of the database, that only aggregated results are publicly reported, and that the medical office is not identified by name. Data are not included in the database without this signed data use agreement. Users fax, email, mail or upload a signed copy of the DUA. See Attachment F: Figure 4. DUA Upload Form. Once the DUA is reviewed, an email notification is sent to the participating organization within three business days with an approval or rejection. See Attachment E: Email # 3 DUA Approval and Email #4 DUA Rejected. Only participants who receive DUA approval may submit data files.

Step 5: Upload Questionnaire. Each registered participating organization must upload a copy of the questionnaire used. See Attachment F: Figure 5. Questionnaire Upload Form. The CAHPS Database reviews the questionnaire to ensure that it meets CAHPS CG Survey standards (the survey instrument must include all core questions, not alter the wording of any core questions, and must not omit any of the survey items related to respondent characteristics that are used for case mix adjustments.) Once the questionnaire is reviewed, an email notification is sent to the registrant within three business days with an approval or rejection. See Attachment E: Email #6 Questionnaire File Approved and Email #7 Questionnaire File Rejected. Only participants that receive questionnaire approval may submit data files.

Step 6: Upload Group, Practice Site and Sample Data Files. Each registered submitter must upload a Practice Site file that includes survey administration information such as mode of administration, response rate, field period and number sampled for each practice site. A registered submitter must also upload their CAHPS CG survey data file(s) related to those practices listed in the Practice Site file. If a submitter has multiple medical practices within a medical group, submitters can upload a Group file that identifies the medical groups. Data are submitted through a secure data submission Web site to ensure confidential transmission of the survey data. See Attachment F Figure 6. Group, Practice Site and Sample Data File Upload Form. Data files must conform to the Data File Layout Specifications provided on the AHRQ CAHPS Web site.

Once a data file is uploaded, a Visual Basic program reads the submitted files and loads them into the SQL database that stores the data. A data file status report is then produced and made available to the submitter. This report displays item frequencies and flags out-of-range values and any possible errors. Submitters are expected to correct any errors and resubmit.

Step 7: Review of Submission and Final Approval. When all of the information required for submission is submitted and approved, an email is sent to the participating organization indicating that their submission has received final acceptance into the CG database. See Attachment E: Email #8 CAHPS CG Database Final Approval.

3. Methods to Maximize Response Rates

AHRQ makes a number of toolkit materials available to assist medical offices with the CAHPS CG surveys. The CAHPS Survey and Reporting Kit explains how to prepare and field a CAHPS questionnaire, analyze the results, and produce consumer-friendly reports. These kits include: survey instruments, protocols and related guidance, sample documents to help administer the survey, analysis programs, instructions for using analysis programs and reporting composites. A reporting resources kit provides sample materials and documents that provide guidance to Medical offices who wish to produce public reports of CAHPS data.

Announcements about the opening of data submission go out through various publicity sources as a way to boost medical office participation in the database. AHRQ’s electronic newsletter and communications target approximately 50,000 subscribers and the AHRQ CAHPS News and Events listserv targets approximately 40,000 subscribers. In addition, email announcements are sent to survey users who have at some point requested technical assistance or who have used the CG survey. AHRQ, through its contractor Westat, provides free technical assistance to users through a dedicated email box and toll-free phone number. In addition, reminders are sent to database registrants to remind them of the deadline for data submission.

4. Tests of Procedures

Input and Feedback for the Development of the CAHPS Database Submission System. The CAHPS CG Database has modeled its data submission processes after those utilized by the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Health Plan Database that has been in operation for many years. In addition, each year the CAHPS Database staff talks with submitters about their experience and use their feedback to improve the collection process. Information compiled from the CAHPS CG Survey Database, as well as feedback obtained during provision of technical assistance for both the Health Plan Survey Database and the Clinician and Group Survey Database each year the databases have been running has been used to improve the CAHPS CG online data submission and reporting system and process over time.

5. Statistical Consultants

Ron Hays, PhD

Professor of Medicine

UCLA School of Medicine/RAND

[email protected]

List of Attachments

Attachment E: CAHPS Clinician and Group Database Submission Emails to Users

Attachment F: Data Submission Secure Web Site and Information Collection Forms

1 For all CG Database reports, when reporting comparison scores by medical practice site characteristic categories, a category’s results are suppressed if there are fewer than five practices and/or fewer than 300 completed surveys available for that category. For more information see “CAHPS Clinician and Group Database; How Results Were Calculated” https://cahpsdatabase.ahrq.gov/cahpsidb/Public/Files/Doc6_How_Results_are_Calculated_CG_2016.pdf

2 For all CG Database reports, when reporting comparison scores by medical practice site characteristic categories, a category’s results are suppressed if there are fewer than five practices and/or fewer than 300 completed surveys available for that category. For more information see “CAHPS Clinician and Group Database; How Results Were Calculated” https://cahpsdatabase.ahrq.gov/cahpsidb/Public/Files/Doc6_How_Results_are_Calculated_CG_2016.pdf

2

DRAFT 04-2018