Supporting Statement Part B -- Establishing Comparative Data for the MOSOPS 05-06-09

SUPPORTING STATEMENT

Part B:

Collections of Information Employing

Statistical Methods

Establishing Comparative Data for the Medical Office

Survey on Patient Safety

Version: May 6, 2009

TABLE OF CONTENTS

B.1. Respondent Universe and Sampling Methods 1

B.2. Procedures for the Collection of Information 3

B.3. Methods to Maximize Response Rates and Deal with Non-Response 4

B.4. Tests of Procedures or Methods to be Undertaken 5

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting

and/or Analyzing Data 5

B.1. Respondent Universe and Sampling Methods

The potential respondent universe consists of all physicians and staff members in ambulatory medical practices that are members of one or more of 14 practice-based research networks (PBRNs). The 14 PBRNs are part of a consortium assembled by the Oregon Rural Practice-based Research Network (ORPRN), which was awarded a contract by AHRQ to conduct this project. PBRNs are groups of ambulatory practices whose principal purpose is to provide health care for patients but who are affiliated with each other and academic researchers to study issues related to the delivery of health care services and the quality of ambulatory health care.

The 14 networks include 1,311 practices located in 12 states (New York, Connecticut, Pennsylvania, Kentucky, Indiana, Iowa, Michigan, Oklahoma, Wisconsin, Minnesota, Texas and Oregon) with a total of 3,312 physicians and 8, 970 non-physician staff. Sampling will be performed at the practice level, with all physicians and staff in each selected practice being asked to complete the survey. Physicians involved in the project may care for patients at multiple physical locations, or offices. If the same practice group submits survey responses from more than one office, we will consider each office as a separate practice. To be eligible, practices must include two or more physicians. Practices with only 1 physician will be excluded because it would be difficult to protect the confidentiality of their staff with respect to their survey responses.

The response target is to obtain 320 practice-level responses, defined as receiving the Practice Characteristics questionnaire and a completed MO-SOPS survey from at least one member of the practice. The expected response rate is 80 percent. In order to achieve 320 practice-level responses, 400 offices will be sampled. Practice sampling will be stratified by three key variables: multi-specialty vs. single specialty practice; size (2-3 physicians vs. 4 or more physicians); and whether or not the practice has enabled health information technology for patient records. The three stratification variables yield a total of 8 sampling cells. Within each cell, eligible practices will be selected by simple random sampling. The target is to achieve an approximately equal number of practice-level responses in each cell; thus, 50 practices will be selected from each cell, with an expectation of obtaining 40 practice-level responses per cell. This is summarized in Table 1 and Table 2.

Table 1. Practice Sampling Plan

	MULTI-SPECIALTY		SINGLE SPECIALTY		TOTAL PRACTICES
	HIT-enabled	Not HIT-enabled	HIT-enabled	Not HIT-enabled
SMALL PRACTICE (2-3 MDs)	50	50	50	50	200
LARGE PRACTICE (4+ MDs)	50	50	50	50	200
TOTAL	100	100	100	100	400

Table 2. Practice-Level Response Target

	MULTI-SPECIALTY		SINGLE SPECIALTY		TOTAL PRACTICES
	HIT-enabled	Not HIT-enabled	HIT-enabled	Not HIT-enabled
SMALL PRACTICE (2-3 MDs)	40	40	40	40	160
LARGE PRACTICE (4+ MDs)	40	40	40	40	160
TOTAL	80	80	80	80	320

It is possible that the number of available practices may be less than the sampling target in some cells. If this occurs, the shortfall within those cells will be equally distributed over the remaining cells to achieve a total sample selection of 400 practices.

We will estimate percentile rankings using data from surveyed practices in each cell shown in Table 2. These percentiles will be calculated for percent of positive responses for each questionnaire item and composite. Data from 40 practices within a cell will provide estimates of these percentile rankings with 95% confidence intervals of no more than approximately  5.0 percentage points, assuming that the percentile is between the 10th and 90th; however, the confidence intervals become larger as percentiles move outside this range (e.g., toward either the 1st or 99th percentile).

The sample percentile has an asymptotic normal distribution with an asymptotic variance given by this formula:

AV(_p) =

where n is the sample size, p is the proportion corresponding to the percentile (e.g., P(X  _p) = p), and f represents the density of the underlying random variable X.

To calculate confidence intervals for percentiles, we first calculated the standard error for percentiles of the standard normal distribution using the formula shown above, with n = 40. Table 1 gives the proportion (0.10, 0.25, etc), the corresponding percentile from the standard normal distribution (1.282, 0.674, etc), the standard normal density function evaluated at the percentile value (0.1755, 0.3178, etc), and the corresponding standard error (0.270, 0.215, etc), which was calculated as the square root of the formula shown above. The half-widths of the confidence intervals for the standard normal percentiles is also shown in Table 1 (0.530, 0.422, etc), which was calculated was 1.96 times the standard error.

Table 1: Calculation of confidence intervals for percentiles

Proportion	Percentile	Density function	Standard error	Confidence interval	 0.10
0.90	1.282	0.1755	0.270	0.530	0.053
0.75	0.674	0.3178	0.215	0.422	0.042
0.50	0.000	0.3989	0.198	0.388	0.039
0.25	-0.674	0.3178	0.215	0.422	0.042
0.10	-1.282	0.1755	0.270	0.530	0.053

To transform these confidence intervals for percentile rankings for the Medical Office Survey on Patient Safety, we used information on standard deviations on percentiles from the Hospital Survey on Patient Safety Culture 2008 Comparative Database Report¹. Table 6-3 in that document shows standard deviations for the items used in the hospital survey. These standard deviations range from roughly 6% to 14%, so we took the midpoint, 10%, as an approximate standard deviation for the distribution of proportions of positive responses among medical offices.

Since the standard deviation of the standard normal distribution is 1.0, we multiplied it by 0.10 was to transform the confidence intervals from the standard normal distribution to confidence intervals for the distribution of proportions reported by medical offices. This step is shown in the last column of Table 1. Note that all values round to 0.05 or less.

The justification for multiplying by the standard deviation (i.e., 0.10) is as follows. First, the percentile from the standard normal distribution is defined by this equation:

which implies

where and are the mean and standard deviation of X. Thus, , the p-th percentile of the distribution of X, is given by

.

Now, if the confidence interval for has the form

then the confidence interval for has the form

which shows that has a confidence interval with half-width of , i.e., that of the standard normal percentile multiplied by the standard deviation of the transformed distribution.

This degree of precision is desired for benchmarking safety practices within each type of practice, as defined by our 8 strata. Ambulatory practices that use this database will be able to compare themselves to the practices that are in the appropriate cell of this comparative database. Because this is a purposive sample from the 14 PBRNs, we do not expect to use the comparative data to compare strata or to weight the sample up to a national estimate of all primary care practices.

B.2. Procedures for the Collection of Information

Data collection will be performed by the research staff of the ORPRN. The procedures will include these steps:

• Each sampled practice will be asked to designate a primary point of contact (POC) for the office (or offices, if more than one for the practice group), preferably the office manager for the practice;

• Hard-copy questionnaires for both the Practice Characteristics and MO-SOPS surveys (including a cover letter) will be mailed to the POC;

• The POC will personally distribute survey packets to each physician and staff member;

• Respondents will mail their completed questionnaires directly to the ORPRN research office in order to maintain confidentiality at the office level;

• A reminder postcard will be sent to the POC if not all questionnaires are received by the ORPRN; and

• The POC will request second questionnaires for any non-responders in the practice who indicate a willingness to respond but no longer have the initial questionnaire.

B.3. Methods to Maximize Response Rates and Deal with Non-Response

The practices that will be selected for this survey are members of PBRNs that formed a consortium to conduct this project. Thus, we expect the practice-based response rate to be high, as noted in Section B.1. Steps that we will employ to maximize response rates to the MO-SOPS and Practice Characteristics surveys include:

• If a particular network’s practices are not responding as expected, we will enlist the assistance of the research staff of that network to contact their practices to encourage participation;

• Use of a POC will give us one person within each practice who is committed to coordinating the survey effort for us in that practice and encouraging response;

• The average length of each individual questionnaire has been limited to 15 minutes;

• Our contact procedures include a follow-up contact and mailing for non-responders.

Practice level non-response is expected to be relatively low. This will be dealt with by over-sampling of practices at the outset to achieve our target number of responses. No adjustments are anticipated for non-response.

B.4. Tests of Procedures or Methods to be Undertaken

The questionnaires and procedures were tested during the pilot test, which was conducted under OMB clearance # 0935-0131. The instruments and procedures to be used in this survey were modified and improved based on the findings of the pilot.

We also intend to ask a representative from each medical office that responded to the surveys to complete an internet-based post-survey evaluation that assesses barriers and facilitators to survey participation and also collects information on the ways in which practices have used (or intend to use) the results of the survey. Respondents will provide their responses directly to the ORPRN research office.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Input on statistical aspects was provided by:

• James Bethel, PhD, Senior Statistician, Westat, 301-294-2067

Persons responsible for data collection and analysis issues are:

• James Battles, PhD, senior program officer, Center for Quality Improvement and Patient Safety, AHRQ;

• Lyle Fagnan, MD, Director, Oregon Rural Practice-based Research Network;

• Nancy Rollins, MS, project coordinator, Oregon Rural Practice-based Research Network.

1 See Sorra J, Famolaro T, Dyer N, et al. Hospital Survey on Patient Safety Culture 2009 comparative database report. (Prepared by Westat, Rockville, MD, under Contract No. HHSA 290200710024C). Rockville, MD: Agency for Healthcare Research and Quality; March 2009. AHRQ Publication No. 09-0030.

ix