Supporting Statement Part B -- CAHPS Clinician Group Survey Comparative Database 5-15-2012

SUPPORTING STATEMENT

Part B

Agency for Healthcare Research and Quality’s (AHRQ)

Consumer Assessment of Healthcare Providers and Systems (CAHPS)

Clinician and Group Survey Comparative Database

February 2012

Agency of Healthcare Research and Quality (AHRQ)

Table of Contents

B. Collections of Information Employing Statistical Methods 1

1. Respondent Universe and Sampling Methods 1

2. Information Collection Procedures 4

3. Methods to Maximize Response Rates 7

4. Tests of Procedures 8

5. Statistical Consultants 8

B. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The AHRQ Consumer Assessment of Healthcare Providers and Systems (CAHPS) Clinician and Group (CG) Survey Database serves as a central repository, the largest database of the survey’s results. Currently, comparative results from 1,095 clinicians and groups that voluntarily submitted data to the CAHPS Database are included. These participating organizations do not constitute a representative sample of all clinicians and groups in the U.S. however; users can compare their results to similar organizations. Similar to the CAHPS Health Plan Database the CAHPS CG Survey results can be viewed on the CAHPS web site at https://www.cahps.ahrq.gov/CAHPSIDB/default.aspx. A section on data limitations is included in the CG Comparative Reports that outline the limitations of the data.

Universe of medical offices and representativeness of the data. Quantifying the universe of clinicians and groups is not straightforward. Clinician practices and groups are typically characterized as either medical practices with 1 or 2 physicians or medical groups consisting of 3 or more physicians. According to the U.S. Census Bureau's 2007 Economic Census, (2007 NAICS code 6211 “Offices of physicians”), there were 220,131 physicians’ offices in the U.S. (http://www.census.gov/econ/industry/geo/g6211.htm). These offices consist of those with only 1 or 2 physicians as well as offices from medical groups with 3 or more physicians. Participation in the Clinician and Group comparative database is open to all practices, clinicians, provider care services, and groups that administer the surveys according to the CAHPS specifications.

A 2005 Health Affairs article examining group medical practices, whose lead author is from the Medical Group Management Association, states: “The total number of U.S. group practices is not known, but we estimate it to be somewhat larger than the 34,490 practices we identified, perhaps in the range of 40,000–50,000” (David Gans, John Kralewski, Terry Hammons, and Bryan Dowd, “Medical Groups’ Adoption Of Electronic Health Records And Information Systems,” Health Affairs, 2005, Vol 24 (5), pp. 1323-1333.)

The most relevant and thorough source of data on the population of medical group practices in the U.S. is the American Medical Association’s (AMA) 1999 edition of “Medical Group Practices in the U.S.: A Survey of Practice Characteristics.” This report is the only (and most recent) source that reports information about the characteristics of medical group practices in the U.S., with groups defined as those with 3 or more physicians. A total of 34,066 eligible medical groups were identified in this census conducted by the AMA in 1996. The problem with comparing these group practices to individual medical offices is that a single group practice can comprise several different medical office locations. The AMA report only includes data from the parent or primary location of group practices, and therefore is also an underestimate of the actual number of group medical offices.

A more recent report from the National Center for Health Statistics in 2008 (http://www.cdc.gov/nchs/data/series/sr_13/sr13_166.pdf) presents estimates of the number and characteristics of medical practices with which physicians are associated. These data, from the 2005-2006 National Ambulatory Medical Care Surveys (NAMCS), are physician-based rather than office-based, and do not allow direct comparisons with the CAHPS Clinician and Group Survey database. However, we do present the NAMCS geographic region data in Table 1. The NAMCS report estimates that during 2005-2006 there were 163,700 medical practices in the United States, which is considerably lower than the 220,131 physicians’ offices in the U.S. Census Bureau 2007 Economic Census.

Table 1 shows the geographic distribution of the CAHPS Clinician & Group Survey database medical practices compared to the distribution of offices of physicians based on the 2007 U.S. Economic Census (http://www.census.gov/econ/industry/geo/g6211.htm) and the NAMCS estimates of the number of office-based medical practices in 2005-2006. The table shows that the 1,095 CAHPS Clinician & Group Survey database medical offices represent less than 1 percent of the estimated population of medical offices. In addition, the database overrepresents medical practices in the Midwest and Northeast and underrepresents medical practices in the South.

Table 1. Distribution of CAHPS CG Database Medical Practices (2010), U.S. Economic Census, Offices of Physicians (2007), and NAMCS Office-Based Medical Practices (2005-2006) by Region

Region	CAHPS Clinician & Group Comparative Database		U.S. Economic Census Offices of Physicians (2007)		2005 – 2006 NAMCS
	Number	Percent	Number	Percent	Number	Percent
Midwest	508	46%	38,951	18%	30,100	18%
Northeast	329	30%	44,605	20%	36,300	22%
South	35	4%	84,424	38%	60,700	37%
West	223	20%	52,151	24%	36,000	22%
TOTAL	1,095	100%	220,131	100%	163,700	100%

Note: Column percent totals may not add to exactly 100% because of rounding.

Because there is not a recent and comprehensive source of data describing the population of group medical offices in the U.S.by specific characteristics, we do not present comparisons of the CAHPS CG database to any other population statistics. Only descriptive statistics about the database medical practices are provided.

Statistics from the 2010 CAHPS CG Survey Comparative Results Report. The following tables provide medical practice characteristics data for the total of 1,095 participating organizations included in the 2010 CAHPS CG Database report. Medical practice characteristics were obtained from each participating medical practice included in the 2010 CAHPS CG Survey database. The database contains characteristics such as region, physician specialty, practice ownership and affiliation, and number of visits per practice. Tables 2 and 3 show the distribution of practices by physician specialty and practice ownership and affiliation.

Physician Specialty. Table 2 presents the distribution by physician specialty. The largest concentration of specialty type is internal medicine followed by missing specialty type, then family practice and other specialty. The data specifications allowed participating organizations to identify 38 different physician specialties including an option for other specialty. The 38 different physician specialties are then combined into eight categories for reporting.

Table 2. Distribution of Practices by Physician Specialty, 2010¹,

Physician Specialty	CAHPS Clinician & Group Comparative Database
	Number	Percent
Missing	543	33%
Family Practice	346	21%
Internal Medicine	226	14%
OB/GYN	93	6%
Pediatrics	63	4%
Other Primary Care	23	1%
Surgical	68	4%
Other Specialty*	272	17%
TOTAL	1,634**	100%

* Other Specialty is categorized as follows: Allergy/Immunology, Anesthesiology, Cardiology, Child & Adolescent Psychiatry, Dermatology, Emergency Medicine, Endocrinology/Metabolism, Gastroenterology, Hematology/Oncology, Nephrology, Neurology, Ophthalmology, Orthopedics, Pathology, Physical Medicine & Rehabilitation, Psychiatry, Public Health & Rehabilitation, Pulmonary Medicine, Radiology, Rheumatology, Urology, and Vascular Medicine.

**A practice can have more than one specialty and may be included in more than one specialty category.

Practice Ownership and Affiliation. The distribution of participating organizations by ownership and affiliation category is presented in Table 3. Because of difficulties with defining the categories, and given the high percentage of ‘missing’ data, the definitions will need to be revised for upcoming rounds of data submissions to more accurately define the practice’s ownership and affiliation.

Table 3. Distribution of Practices by Ownership and Affiliation, 2010

Practice Ownership and Affiliation	CAHPS Clinician & Group Comparative Database
	Number	Percent
Missing	70	6%
Commercial	47	4%
Hospital or Integrated Delivery System	431	39%
University or Medical School	88	8%
County	8	1%
Other	451	41%
TOTAL	1,095	100%

Note: Column percent totals may not add to exactly 100% because of rounding.

Comparative results and explanation of how results are calculated.

The CAHPS Database adjusts the survey results in order to account for factors that may affect scores for the practice, clinician, or other entity that are beyond the control of the entity. Without an adjustment, differences between entities could be due to differences in these exogenous factors rather than to true differences in performance. CAHPS data are most commonly adjusted for respondent characteristics (i.e. case mix adjustments), but can also be adjusted for other factors such as the mode of survey administration (telephone, interactive voice response, or Web/Internet). The adjusted results are reported in the online reporting system.

Case-mix adjustments. Case mix refers to the respondents’ health status and other socio-demographic characteristics that have been shown to affect patient reports and ratings of practice sites, clinicians, or other entities. Without an adjustment, differences between entities could be due to case-mix differences rather than true differences in quality. The CAHPS Consortium recommends adjusting the survey data for respondent age, education, and general health status. This makes it more likely that reported differences are due to real differences in performance, rather than differences in the characteristics of enrollees or patients. Individuals in better health and older individuals tend to rate their care, plans, and providers higher. There is also evidence from a number of studies that education affects ratings, with more educated individuals giving lower ratings.

Data are analyzed using the CAHPS Analysis Program. The goal of the CAHPS Analysis Program – often referred to as the CAHPS macro – is to provide a flexible way to analyze CAHPS survey data in order to make valid comparisons of performance. Written in SAS, the CAHPS Analysis Program is designed to assist CAHPS survey users in implementing two kinds of statistical adjustments.

For details on the CAHPS Analysis program see Instructions for Analyzing CAHPS Data at
https://www.cahps.ahrq.gov/surveys-guidance/cg/~/media/files/surveydocuments/cg/12%20month/prep_analyze/2015_instructions_for_analyzing_data.pdf.

Testing for Statistical Differences. Statistical tests (t-tests) are used to determine whether a participating organization's mean item or composite score is significantly above or below the overall mean. These statistical tests are based on a participating organization's mean item score or composite score rather than top box scores. Top box scores are the percent of respondents who choose the most positive score for a given item. These scores are case-mix adjusted by patient characteristics. If an organization's mean item or composite score is significantly higher or lower than the overall mean, an ‘up’ or ‘down’ arrow is assigned respectively. If there is no significant difference between the organization and the overall mean, no arrow is assigned.

2. Information Collection Procedures

Information collection for the AHRQ CAHPS CG Survey Database occurs in a periodic data collection cycle bi-annually. Information collection procedures for submitting and processing data are shown in Figure B-1.

Figure B-1. CAHPS Clinician and Group Database Data Submission Process

Step 1.

Call for Data Submission

Step 2.

Registration for Potential Participants

Step 3.

Verify Account by Email

Step 4.

Submit Data Use Agreement

Step 5.

Upload Questionnaire

Step 6.

Enter Survey Administration
and Upload Data File(s)

Email Sent to Registrant with Link to Verify Account Information

CAHPS Database Review and Approval of Questionnaire

Data File Status Report Generated. Email Notification Sent to Registrant if Any Data File Issues.

CAHPS Database Review and Approval of Data File(s) Approval Email Notification Sent to Registrant.

Step 7.

Final Review of All Submission Items and Final Approval

Email Notification of Final CAHPS Database Approval Sent to Registrant.

Step 1: Call for Data Submission. Announcements about the opening of data submission go out through various publicity sources. AHRQ’s electronic newsletters, the CAHPS Connection, and communications target approximately 27,000 subscribers. In addition, email announcements are sent to approximately 200 survey users who have at some point submitted CAHPS CG data or requested technical assistance. An example of an email announcement calling for data submission is shown in Attachment E, Email # 1: Call for Data Submission. Reminder emails are sent one and two weeks after the initial email announcing the call for data submission. In addition, the AHRQ CAHPS Web site posts public information about the yearly timeline and instructions for data submission. Through these efforts, U.S. medical offices are made aware of and invited to submit their survey data to the database. As the administrator of the database and under contract with AHRQ, Westat provides free technical assistance to submitting medical offices and their vendors through a dedicated email address ([email protected] that routes to Westat) and toll-free phone number (888-852-8277).

Step 2: Registration for Potential Participants. A secure data submission Web site allows interested parties such as medical groups and offices to register and submit data. Registrants are asked to provide contact and other basic information and create a unique ID, password and security question. See Attachment F: CAHPS Clinician and Group Data Submission Registration Form.

Step 3: Verify Account by Email. Once a submitter has registered and is deemed eligible to submit data, an automated email is sent to provide them with the username and password and information needed to activate their account. See Attachment E, Email # 2: Notice to Activate Account. Once users have a username and password and have activated their account, they can enter the main page menu of the Web site. Information about eligibility requirements, data use agreements, and data file specifications regarding how to prepare their data for inclusion in the CAHPS CG database is posted and can be reviewed.

Step 4: Submit Data Use Agreement. To protect the confidentiality of all participating medical offices, a duly authorized representative from the group or medical office must sign a data use agreement (DUA). The DUA language was reviewed and approved by AHRQ’s general counsel. The DUA states that the group or medical office’s data are handled in a secure manner using necessary administrative, technical and physical safeguards to limit access to it and maintain its confidentiality. In addition, the DUA states the data are used for the purposes of the database, that only aggregated results are publicly reported, and that the medical office is not identified by name. Data are not included in the database without this signed data use agreement. Users fax, email and/or mail a signed copy of the DUA.

Step 5: Upload Questionnaire. Each registered participating organization must upload a copy of the questionnaire used. See Attachment F: Questionnaire Upload Form. The CAHPS Database reviews the questionnaire to ensure that it meets CAHPS CG Survey standards (the survey instrument must include all core questions, not alter the wording of any core questions, and must not omit any of the survey items related to respondent characteristics that are used for case mix adjustments.) Once the questionnaire is reviewed, an email notification is sent to the registrant within three business days with an approval or rejection. See Attachment E: Email #3 Questionnaire File Approved and Email #4 Questionnaire File Rejected. Only participants that receive questionnaire approval may submit data files.

Step 6: Enter Survey Administration Information and Upload Data Files. Each registered submitter must enter survey administration information (mode of administration, response rate, field period, number sampled) and upload their CAHPS CG survey data file(s). Data are submitted through a secure data submission Web site to ensure confidential transmission of the survey data. See Attachment F Survey Administration Information and Data File Upload Form. If a user has multiple medical practices within a medical group, users can upload one file that identifies all of the medical practices in their group. Data files must conform to the Data File Layout Specifications provided on the AHRQ CAHPS web site.

Once a data file is uploaded, a Visual Basic program reads the submitted files and loads them into the SQL database that stores the data. A data file status report is then produced and made available to the submitter. This report displays item frequencies and flags out-of-range values and any possible errors. If there are problems, the participating organization is notified by email and may review the Data File Status Report for further detail. See Attachment E: Email # 5 Data File Rejected. Submitters are expected to correct any errors and resubmit. Once there are no problems with the data file(s), an email is sent to the participating organization's point of contact indicating their data has been approved. See Attachment E: Email # 6 Data File Approved.

Step 7: Review of Submission and Final Approval. When all of the information required for submission is submitted and approved, an email is sent to the participating organization indicating that their data have received final acceptance into the CG database. See Attachment E: Email #7 CAHPS CG Database Final Approval.

3. Methods to Maximize Response Rates

AHRQ makes a number of toolkit materials available to assist medical offices with the CAHPS CG surveys. The CAHPS Survey and Reporting Kit explains how to prepare and field a CAHPS questionnaire, analyze the results, and produce consumer-friendly reports. These kits include: survey instruments, protocols and related guidance, sample documents to help administer the survey, analysis programs, instructions for using analysis programs and reporting composites. A reporting resources kit provides sample materials and documents that provide guidance to Medical offices who wish to produce public reports of CAHPS data.

As noted earlier in this document under Information Collection Procedures, announcements about the opening of data submission go out through various publicity sources as a way to boost medical office participation in the database. AHRQ’s CAHPS newsletters target approximately 125,000 subscribers. In addition, email announcements are sent to survey users who have at some point requested technical assistance or who have used the CG survey. AHRQ, through its contractor Westat, provides free technical assistance to users through a dedicated email box and toll-free phone number. In addition, reminders are sent to database registrants to remind them of the deadline for data submission.

4. Tests of Procedures

Input and Feedback for the Development of the CAHPS Database Submission System. The CAHPS CG Database has modeled its data submission processes after those utilized by the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Health Plan Database that has been in operation for many years. Every year the CAHPS Database staff hold training sessions for health plans and their survey vendors to provide an overview of the submission process with an emphasis on any changes since the previous year. In addition, each year the CAHPS Database staff talks with submitters about their experience and use their feedback to improve the collection process. Information from the CAHPS Health Plan Survey Database, as well as feedback obtained during the provision of technical assistance each year the database has been running has been used to improve the CAHPS CG online data submission system and process over time.

5. Statistical Consultants

Ron Hays, PhD

Professor of Medicine

UCLA School of Medicine/RAND

[email protected]

Michael Wilson

Sr. Study Director

Westat

[email protected]

1 For all CG Database reports, when reporting comparison scores by medical practice site characteristic categories, a category’s results are suppressed if there are fewer than five practices and/or fewer than 300 completed surveys available for that category. For more information see “CAHPS Clinician and Group Database; How Results Were Calculated” https://www.cahps.ahrq.gov/CAHPSIDB/Public/Files/Doc6_How_Results_are_Calculated_2010.pdf.

2