Rollow et al annals

Rollow et al annals.pdf

Program Evaluation of the Ninth Scope of Work Quality Improvement Organization Program (CMS-10294)

Rollow et al annals

OMB: 0938-1104

Document [pdf]

Download: pdf | pdf

Annals of Internal Medicine

Improving Patient Care

Assessment of the Medicare Quality Improvement
Organization Program
William Rollow, MD, MPH; Terry R. Lied, PhD; Paul McGann, SM, MD; James Poyer, MS, MBA; Lawrence LaVoie, PhD;
Robert T. Kambic, MSH; Dale W. Bratzler, DO, MPH; Allen Ma, PhD; Edwin D. Huff, PhD; and Lawrence D. Ramunno, MD, MPH

Background: Studies have shown improvement in quality of health
care in the United States. However, the factors responsible for this
improvement are largely unknown.
Objective: To evaluate the effect of the Medicare Quality Improvement Organization (QIO) Program in 4 clinical settings by using
performance data for 41 quality measures during the 7th Scope of
Work.
Design: Observational study in which differences in quality measures were compared between baseline and remeasurement periods
for providers that received different levels of QIO interventions.
Setting: Nursing homes, home health agencies, hospitals, and physician offices in the 50 U.S. states, the District of Columbia, and 2
U.S. territories.

Results: For nursing home, home health, and physician office measures, providers recruited specifically by QIOs for receipt of assistance showed greater improvement in performance on 18 of 20
measures than did providers who were not recruited; similar improvement was seen on the other 2 measures. Nursing homes and
home health agencies improved more in all measures on which
they chose to work with the QIO than in other measures. Nineteen
of 21 hospital measures showed improvement; in this setting, QIOs
were contracted for improvement initiatives solely at the statewide
level. Overall, improvement was seen in 34 of 41 measures from
baseline to remeasurement in the 7th Scope of Work.
Limitations: As in any observational study, selection bias, regression to the mean, and secular trends may have influenced the
results.

Participants: Providers receiving focused QIO assistance related to
quality measures and providers receiving general informational assistance from QIOs.

Conclusions: These findings are consistent with an impact of the
QIO Program and QIO technical assistance on the observed improvement. Future evaluations of the QIO Program will attempt to
better address the limitations of the design of this study.

Measurements: 5 nursing home quality measures, 11 home health
measures, 21 hospital measures, and 4 physician office measures.

Ann Intern Med. 2006;145:342-353.
For author affiliations, see end of text.

evaluated the effect of the QIO Program in 4 clinical settings by using performance data for 41 quality measures
and explored the implications of these findings for future
Program evaluations.

ecent reports have highlighted deficiencies in quality of
health care in the United States (1, 2). Several reports
of nationwide improvements have also been published by
such organizations as the Agency for Healthcare Research
and Quality, the National Committee for Quality Assurance, the Joint Commission on Accreditation of Healthcare Organizations, and the Medicare Quality Improvement Organization (QIO) Program. The extent to which
these improvements are attributable to the efforts of health
plans, accreditors, or QIOs is unclear, given the absence of
comparison groups (3–11).
The Centers for Medicare & Medicaid Services
(CMS), the federal agency responsible for administering
Medicare, Medicaid, and several other health care–related
programs, seeks to improve the quality of health care for
Medicare beneficiaries through contracts with QIOs
(12)—state-based organizations staffed with clinicians, analysts, and others with expertise in case review and quality
improvement. The 53 QIO contracts cover the 50 U.S.
states, the District of Columbia, Puerto Rico, and the Virgin Islands. A single organization can hold more than 1
QIO contract. Appendix Figure 1 (available at www
.annals.org) shows the locations of QIO lead offices.
The most recently concluded QIO contract period,
the 7th Scope of Work, began in 2002. At various points
during this period, the CMS began public reporting of
provider performance on quality measures for nursing
homes, home health agencies, and hospitals (13–15). We
342 5 September 2006 Annals of Internal Medicine Volume 145 • Number 5

www.annals.org

METHODS
Participants

For the 7th Scope of Work, the CMS expanded the
QIO contract, which was previously limited to hospitals
and physician offices, to include nursing homes and home
health agencies. For each of the 4 settings, the CMS required QIOs to offer assistance to all interested providers
in their state or jurisdiction. In the nursing home, home
health agency, and physician office settings, QIOs were
also required to recruit subsets of providers, known as iden-

See also:
Print
Editorial comment. . . . . . . . . . . . . . . . . . . . . . . . . . 388
Web-Only
Appendix
Appendix Tables
Appendix Figures
Conversion of figures and tables into slides
www.annals.org

Assessment of the Medicare Quality Improvement Organization Program

tified participant groups (IPGs), that would receive focused
assistance related to clinical quality measures. There was no
IPG requirement for the hospital setting.
The 53 QIOs recruited voluntary IPGs among nursing homes, home health agencies, and physician offices
during the initial months of the 7th Scope of Work. The
CMS used outcome measures and clinical process measures
to evaluate the performance of each QIO as part of a performance-based service contract. The QIOs were evaluated
on improving statewide and IPG clinical process measures
and provider satisfaction with their QIO.
Each QIO was required to recruit an IPG of at least
10% but no more than 15% of the nursing homes in its
state or jurisdiction and at least 5% but no more than
7.5% of primary care practitioners (physicians, nurse practitioners, and physicians’ assistants). For the home health
setting, the minimum number of providers in the IPG was
30% of the agencies in the state; there was no maximum
number.
The CMS provided general guidance to the QIOs on
the selection of IPG providers but did not control the
selection process. The QIOs shared information among
themselves about appropriate factors, such as willingness to
commit resources to quality improvement and baseline
performance, for which there were opportunities for improvement.
In this study, we classified providers not participating
in an IPG as non-IPG providers. Nursing homes and home
health agencies that were participants in an IPG were required to select 1 or more quality measures to target for
improvement. For these 2 settings, we subdivided providers in IPGs by measure into 2 subgroups: IPG-select and
IPG-other (Appendix Figure 2, available at www.annals
.org). For a given measure, the IPG-select subgroup consists
of IPG providers that elected to focus on that measure, and
the IPG-other subgroup consists of IPG providers that selected other measures.
QIO Interventions

We collected information on the intensity of QIO assistance for nursing homes but not the other provider settings so that we could classify the non-IPG and IPG providers according to 4 levels of QIO intervention (high,
medium, low, and none). The highest level of activity involved on-site visits or planned multicontact educational
interventions in a group setting to the provider by QIO
staff; low-level activity was often limited to sending written
or electronic material to the nursing home. Overall, there
was a strong relationship between participant status and
level of QIO intervention. Only 32.5% of the non-IPG
facilities received a high level of QIO intervention, whereas
97.3% of the IPG facilities received this level of intervention. We did not collect information on non-QIO quality
improvement programs in which non-IPG providers may
have participated during the 7th Scope of Work.
At the statewide level, QIOs promoted quality imwww.annals.org

Improving Patient Care

provement in the 4 settings through such activities as partnerships with provider organizations, work with business
and consumer groups, regional educational meetings, and
direct QIO communication with providers (10, 16, 17).
Development and dissemination of information on best
practices and improvement tools gave providers resources
that were useful in improvement work (6, 7, 18 –26). With
the IPG providers, QIOs conducted more focused activities.
Quality Measures

Quality measures selected by the IPGs were driven by
different factors in each setting, including contractual direction and limitations, baseline performance, and method
of improvement. Data are reported on 5 nursing home
measures, 11 home health agency measures, 21 hospital
measures, and 4 outpatient measures. One measure (infection) was not reported because it was measured in more
than 1 way, and another measure (delirium) was not reported because very few providers specifically worked to
improve performance in this area. The Appendix (available
at www.annals.org) provides details on the selection and
reporting of measures by setting.
Data Sources

We used data from nursing homes and home health
agencies that were reported to CMS through the systems
required for Medicare payment: the Minimum Data Set
(27) and the Outcomes Assessment and Information Set
(28). Medicare- and Medicaid-certified nursing homes are
required to conduct Minimum Data Set assessments of all
residents on admission and at mandated intervals. The
Outcomes Assessment and Information Set provides a
comprehensive assessment of adult home care patients; like
the Minimum Data Set assessment, its use is required on
admission and at mandated intervals.
The hospital data were abstracted by clinical data abstraction contractors, who provide data support to the
CMS. As in the 6th Scope of Work evaluation (8), we used
random samples of 125 inpatient records per state per
quarter for Medicare patients with a diagnosis of acute
myocardial infarction, heart failure, or pneumonia or who
had undergone surgery. Sample cases were weighted according to their probability of selection in the national
quarterly sample. We analyzed CMS National Claims History data to determine assignment, based on majority of
care, of Medicare beneficiary to practitioner for the physician office setting. Information on assignment of beneficiary to practitioner and performance on physician office
quality measures was compiled quarterly.
For nursing homes, home health agencies, hospitals,
and physician offices, we report only baseline and remeasurement data, because of space limitations and because the
CMS evaluated QIO performance on the basis of improvement from baseline to remeasurement. The baseline and
remeasurement periods were separated by about 2 years for
5 September 2006 Annals of Internal Medicine Volume 145 • Number 5 343

Improving Patient Care

Assessment of the Medicare Quality Improvement Organization Program

Table 1. Characteristics of Nursing Homes, Home Health
Agencies, and Physician Offices*
Characteristic

Nursing homes
Number studied
Beds
Mean, n
Median, n
⬍50, %
50–99, %
100–149, %
150–199, %
ⱖ200, %
Urban location, %
Located in hospital, %
Medicaid residents, %
Home health agencies
Number studied
Size, %†
Very small (10–30)
Small (31–150)
Medium (151–500)
Large (ⱖ501)
Physician offices
Number studied
Mean age of physicians, y
Physician specialty code, %
General practice
Family practice
Internal medicine
All other

Non–Identified
Participant
Group Providers
11 076
117.6
106.0
4.2
38.1
36.7
13.0
8.0
65.2
3.6
68.1

2445
28.2
32.1
26.2
13.5

Identified
Participant
Group Providers
2236
127.6
116.0
2.5
33.6
38.3
14.8
10.7
68.6
6.1
63.5

4251
5.7
24.5
36.4
33.4

325 634
46

15 263
47

5
26
29
40

5
41
39
15

* Identified participant group providers volunteered to receive focused attention
from the Quality Improvement Organizations.
† The proxy for size is the number of utilization outcome episodes for the acute
hospitalization quality measure during the fourth quarter of 2002.

nursing homes, home health agencies, and hospitals and by
3 years for physician offices.
For nursing homes, we used the second quarter of
2002 as the baseline period and the second quarter of 2004
as the remeasurement period. For home health, 1 May
2001 to 30 April 2002 was the baseline period and 1 February 2004 to 31 January 2005 was the remeasurement
period. For the hospital setting, the first quarter of 2002
was the baseline period and the fourth quarter of 2004 was
the remeasurement period. For the physician office setting,
the baseline and remeasurement periods varied depending
on the QIO contract cycle. The selection of baseline and
remeasurement periods varied by setting because of contractual reasons and data set limitations.
Statistical Analysis

For the nursing home setting, QIO contracts and publicly reported data required a minimum denominator of 30
for the chronic care measures and 20 for the acute care
measures to create stable rates (qualifying providers). Approximately 13 000 of 16 000 nursing facilities, or about
80%, were included for each long-stay measure. The 1
short-stay measure, pain, had approximately 3100 qualify344 5 September 2006 Annals of Internal Medicine Volume 145 • Number 5

ing providers. For nursing homes, the percentage of providers with excluded data because they did not meet the
denominator requirements (exclusion rate) varied by measure from 17.8% to 44.9% for non-IPG providers and
from 5.4% to 34.9% for IPG providers. For contractual
and reporting reasons, in the home health setting, agencies
had to have at least 30 episodes of care in their denominator for a particular measure to be included in the calculations. For the 11 home health agency measures in our
study, approximately 6000 home health agencies (about
80%) were included in the rate calculations. The exclusion
rate varied by measure from 10.4% to 17.2% for non-IPG
providers and from 0.5% to 1.8% for IPG providers.
In the hospital setting, the only missing information
was from an occasional chart that was not sent to the clinical data abstraction contractors. The physician office setting measurement was based on claims data, and we have
no measure of claims that were not submitted.
For nursing homes and home health agencies, the national mean baseline and remeasurement rates were calculated, by quality measure, for participant groups (non-IPG,
IPG-other, and IPG-select). National means were calculated as means of individual facility rates. The sample for
both nursing homes and home health agencies consisted of
qualifying providers that reported quality measure rates at
both baseline and remeasurement. For the hospital setting,
we report data on the number of patients sampled nationwide with primary diagnoses of acute myocardial infarction, heart failure, and pneumonia, as well as those eligible
for surgical infection prevention measures. Descriptive statistics for the physician office setting are the number of
eligible patients at baseline and baseline and remeasurement rates for Medicare beneficiaries who received most
care from practitioners in IPG and non-IPG physician offices.
We also analyzed time-trend data for nursing homes
and home health agencies. These data included all providers who reported at any of the time intervals and met the
denominator requirement for at least 1 period. These data
included all reporting nursing homes and home health
agencies regardless of the number of periods in which they
reported data; therefore, the denominator counts are not
the same as those in the previous analyses.

RESULTS
Table 1 shows provider characteristics across non-IPG
and IPG nursing homes, home health agencies, and physician offices. Compared with non-IPG nursing homes, IPG
nursing homes tended to have a lower proportion of Medicaid residents and were larger, more often found in urban
locations, and more likely to be located in a hospital. The
IPG home health agencies tended to be larger than nonIPG home health agencies. In the physician office setting,
the predominant specialty codes were similar for the 2
groups: general practice, family practice, internal medicine,
www.annals.org

Assessment of the Medicare Quality Improvement Organization Program

Improving Patient Care

Table 2. Nursing Home Performance on Selected Quality Measures at Baseline and Remeasurement*
Quality Measure

Mean Rate, %

Nursing
Homes,
n†

Baseline‡

Remeasurement§

Change in
Performance,
percentage
points

Decline in activities of daily living since the last Minimum Data
Set administration
Non-IPG
IPG
IPG-other
IPG-select

10 438
2251
1817
434

15.2
16.5
15.6
20.4

15.4
16.0
15.5
17.8

0.2
⫺0.5
⫺0.1
⫺2.6

Moderate pain daily or severe pain any time in the past 7 days
Non-IPG
IPG
IPG-other
IPG-select

10 892
2305
496
1089

10.5
11.7
6.7
13.0

6.8
6.0
5.2
6.2

⫺3.7
⫺5.7
⫺1.5
⫺6.8

Physically restrained daily in the past 7 days
Non-IPG
IPG
IPG-other
IPG-select

11 076
2326
1717
609

9.8
9.6
7.3
16.5

7.7
6.4
5.7
8.4

⫺2.1
⫺3.2
⫺1.6
⫺8.1

Pressure ulcers in the past 7 days
Non-IPG
IPG
IPG-other
IPG-select

11 075
2326
946
1380

8.4
9.0
7.4
10.1

8.6
8.8
8.2
9.3

0.2
⫺0.2
0.8
⫺0.8

7425
1600
932
668

24.9
26.3
24.0
29.7

22.5
21.5
20.4
23.0

⫺2.4
⫺4.8
⫺3.6
⫺6.7

Short-stay residents with moderate pain daily or severe pain
any time in the past 7 days
Non-IPG
IPG
IPG-other
IPG-select

* IPG ⫽ identified participant group. For a given measure, the IPG-select subgroup consists of IPG providers that elected to focus on that measure, and the IPG-other
subgroup consists of IPG providers that selected other measures.
† Number of nursing homes that reported at both baseline and remeasurement for the specific measure.
‡ Mean of rates from individual facilities across all U.S. states or jurisdictions at baseline in the second quarter of 2002. Rates from individual facilities are the number of
patients as a percentage of eligible patients.
§ Mean of rates from individual facilities across all U.S. states or jurisdictions at remeasurement in the second quarter of 2004. Lower values at remeasurement indicate
improved performance.

and obstetrics/gynecology. However, IPG physician offices
had a higher proportion of family practice and internal
medicine specialists compared with non-IPG physician offices.
Appendix Table 1 (available at www.annals.org)
shows the total number of nursing homes and the number
of IPG nursing homes for the second quarters of 2002 and
2004, by state or jurisdiction. Appendix Table 2 (available
at www.annals.org) shows similar data for home health
agencies. Appendix Table 3 (available at www.annals.org)
shows the number of primary care physicians at baseline,
IPG practitioners, and IPG practitioners as a percentage of
total primary care physicians, by state or jurisdiction.
Nursing Homes

Table 2 shows quality measures for IPG and non-IPG
nursing homes. For all 5 quality measures, IPG nursing
homes experienced greater improvement from baseline to
remeasurement than did non-IPG nursing homes. NonIPG nursing homes improved in 3 measures, whereas IPG
www.annals.org

nursing homes improved in all 5 measures. The IPG-select
nursing homes showed greater improvement than did IPGother nursing homes in all 5 measures. The most substantial improvements for the IPG-select nursing homes were
in the chronic care pain measure (from 13.0% of residents
at baseline to 6.2% at remeasurement), the short-stay
(post–acute care) pain measure (from 29.7% to 23.0%),
and the restraints use measure (16.5% to 8.4%). Figures
1 to 3 show time-trend results for the nursing home
measures.
Home Health Agencies

Table 3 shows mean data on quality measures at baseline and remeasurement for IPG and non-IPG home
health agencies. Both groups showed improvement from
baseline to remeasurement in mean facility rates for 10 of
11 measures. In addition, for 10 of 11 measures, improvement was greater for IPG agencies than non-IPG agencies.
For the 1 measure with an overall decline in performance
(acute care hospitalizations), IPG and non-IPG agencies
5 September 2006 Annals of Internal Medicine Volume 145 • Number 5 345

Improving Patient Care

Assessment of the Medicare Quality Improvement Organization Program

had the same rate of hospitalization at baseline and remeasurement. Comparisons within IPG agencies show that the
IPG-select agencies improved more than IPG-other agencies on all 11 quality measures. Appendix Figures 3 to 8
(available at www.annals.org) show time trends for the
home health agency quality measures.

period. The 2 measures for which no improvement was
seen were use of angiotensin-converting enzyme inhibitors
for left ventricular systolic dysfunction in heart failure and
selection of prophylactic antibiotics for surgical patients.
Physician Offices

Table 5 shows baseline and remeasurement performance for IPG and non-IPG physician offices. The IPG
offices showed improvement in all 4 measures, whereas the
non-IPG offices showed improvement in 2 of the 4 measures but slight worsening in the screening mammography
and diabetic retinal eye examination measures. Baseline
performance was generally better for IPG offices than nonIPG offices. The greatest improvements for IPG offices

Hospitals

There were no comparison groups for the hospital setting; therefore, overall trends in hospital performance were
examined. Table 4 shows the nationally weighted mean
performance rates for the 21 hospital quality measures
from the first quarter of 2002 and the fourth quarter of
2004. Nineteen of the 21 measures improved during this

Figure 1. Time-trend data for nursing home facilities: pressure ulcer (top) and daily physical restraints (bottom).
IPG-other (n = 954)

IPG-select (n = 1401)
Non-IPG (n = 11 623)

National (n = 13 979)
Pressure Ulcer, %

5
2

2002

2003

2004

Quarter

18
17

IPG-other (n = 1736)

Daily Physical Restraints, %

IPG-select (n = 620)

Non-IPG (n = 11 623)

National (n = 13 979)

13
12
11
10
9
8
7
6
5
4
2

2002

2003

2004

Quarter

Quarterly means were calculated from facilities with a reportable score in that quarter. Lower scores indicate better performance. Numbers in parentheses
are the average numbers of facilities across quarters. IPG⫽ identified participant group. For a given measure, the IPG-select subgroup consists of IPG
providers that elected to focus on that measure, and the IPG-other subgroup consists of IPG providers that selected other measures.
346 5 September 2006 Annals of Internal Medicine Volume 145 • Number 5

www.annals.org

Assessment of the Medicare Quality Improvement Organization Program

Improving Patient Care

Figure 2. Time-trend data for nursing home facilities: chronic care pain (top) and post–acute care pain (bottom).
15

IPG-other (n = 508)

Chronic Care Pain, %

IPG-select (n = 1834)

Non-IPG (n = 11 477)

National (n = 13 819)

11
10
9
8
7
6
5
4
3
2

2002

2003

2004

Quarter

Post–Acute Care Pain, %

32
31

IPG-other (n = 1054)

IPG-select (n = 694)

Non-IPG (n = 8454)

National (n = 10 201)

27
26
25
24
23
22
21
20
19
18
2

2002

2003

2004

Quarter

occurred in the diabetic hemoglobin A1c testing measure
(improvement of 8.7 percentage points) and the diabetic
lipid profile determination measure (improvement of 11.2
percentage points). Procedures performed outside of primary care practices (diabetic retinal eye examination and
screening mammography) had more modest improvements.

DISCUSSION
We assessed whether national clinical quality measures
had improved in the QIO 7th Scope of Work and whether
QIOs contributed to this improvement. We found that
www.annals.org

clinical quality improved for Medicare beneficiaries on 34
of 41 measures. These findings are consistent with published findings for a previous contract period for the inpatient hospital and outpatient (physician office) settings (8).
However, for the first time, they now include the clinical
performance results for nursing homes and home health
agencies.
Assessment of the contribution of the QIO Program
to quality improvement is challenging. Two types of contributions are possible. One type derives from the work
that the Program does in partnership with stakeholder organizations and in support of CMS quality initiatives. This
5 September 2006 Annals of Internal Medicine Volume 145 • Number 5 347

Improving Patient Care

Assessment of the Medicare Quality Improvement Organization Program

work includes the convening of partnerships and other activities that focus provider attention on the need to improve in a given area, the development of quality measures
and support for the infrastructure by which provider performance data are publicly reported, and the development
of improvement methods and tools. The other contribution is direct provision of technical assistance to providers.
As an example of the first type of contribution, in
preparation for and during the 7th Scope of Work, the
Program played a leading national role in creating the measurement, reporting, and improvement infrastructure for
efforts to prevent surgical infection in hospitals. In 2000,
there were no nationally available measures of surgical infection prevention, nor were there reliable estimates of the
extent to which preventable infections occur. Through a
QIO developmental project, in partnership with the Centers for Disease Control and Prevention, national performance measures were formulated that were broadly endorsed by major surgical and medical specialty societies
(29). In another QIO project, a retrospective medical
record review demonstrated a substantial opportunity to
improve the quality of care for delivery of prophylactic
antibiotics (30). A QIO-sponsored national collaborative
that included 56 hospitals and 43 QIOs representing 50
states tested change ideas to prevent surgical site infections
and to facilitate the spread of improvement methods between hospitals and QIOs across the United States (31,
32). These activities created the foundation for the inclusion of surgical infection prevention measures in the QIO
7th Scope of Work. The substantial improvement that occurred during the contract period preceded the adoption of

surgical infection measures by the Joint Commission on
Accreditation of Healthcare Organizations and public reporting of hospital performance on these measures.
Although it is difficult to distinguish the unique contribution of the QIO Program through such activities, the
Program has played a leading role with other stakeholders
in national quality improvement initiatives. Increasingly,
however, reviewers have focused on the second type of
contribution that the Program may make: technical assistance by QIOs to providers.
Snyder and Anderson (9) attempted to assess the contribution of QIOs to the improvement in hospital measures in the QIO 6th Scope of Work by comparing the
amount of improvement achieved by hospitals receiving
different levels of QIO assistance. They did not find evidence of such a contribution. Their study, however, was
criticized for its small and unrepresentative sample of 5
U.S. states, the short interval assessed (half of the contract
period), and use of retrospective assignment by QIOs of
the intensity of assistance that hospitals received in which
the reliability of the assignment was not tested (33). Furthermore, in the context of a contract that required statewide improvement in which QIOs may have directed their
resources away from hospitals that had internal capacity for
improvement, a comparison among hospitals receiving different levels of assistance would not be a true test of the
effects of QIOs (34). In another study of the effect of
QIOs on hospitals in the 6th Scope of Work, some hospital quality improvement directors reported QIO assistance
as being helpful, and others said that they did not need it
(10).

Decline in Activities of Daily Living, %

Figure 3. Time-trend data for nursing home facilities: decline in activities of daily living.
23

IPG-other (n = 1859)

IPG-select (n = 445)
Non-IPG (n = 11 069)

National (n = 13 373)

20
19
18
17
16
15
14
13
2

2002

2003

2004

Quarter

Quarterly means were calculated from facilities with a reportable score in that quarter. Lower scores indicate better performance. Numbers in parentheses
are the average numbers of facilities across quarters. IPG⫽ identified participant group. For a given measure, the IPG-select subgroup consists of IPG
providers that elected to focus on that measure, and the IPG-other subgroup consists of IPG providers that selected other measures.
348 5 September 2006 Annals of Internal Medicine Volume 145 • Number 5

www.annals.org

Assessment of the Medicare Quality Improvement Organization Program

Improving Patient Care

Table 3. Home Health Agency Performance on Selected Quality Measures at Baseline and Remeasurement*
Quality Measure

Home Health Agencies, n

Mean Rate, %

Change in Performance,
percentage points

Baseline†

Remeasurement‡

1788
4187
3855
332

29.5
31.5
31.6
29.9

32.3
35.6
35.4
37.6

2.8
4.1
3.8
7.7

Improved transferring
Non-IPG
IPG
IPG-other
IPG-select

1751
4177
3738
439

42.7
46.0
46.8
39.3

44.4
49.8
49.8
49.4

1.7
3.8
3
10.1

Improved toileting
Non-IPG
IPG
IPG-other
IPG-select

1686
4127
4074
53

51.7
56.7
56.8
50.9

56.3
61.3
61.3
62.7

4.6
4.6
4.5
11.8

Improved pain interfering with activity
Non-IPG
IPG
IPG-other
IPG-select

1779
4181
3510
671

50.6
53.0
54.1
47.0

54.7
58.6
58.5
59.0

4.1
5.6
4.4
12

Improved bathing
Non-IPG
IPG
IPG-other
IPG-select

1789
4186
3913
273

48.1
53.1
53.4
49.0

52.8
58.2
58.3
56.5

4.7
5.1
4.9
7.5

Improved oral medication management
Non-IPG
IPG
IPG-other
IPG-select

1758
4178
3675
503

29.7
31.9
32.2
29.3

32.5
36.1
36.0
36.7

2.8
4.2
3.8
7.4

Improved upper-body dressing
Non-IPG
IPG
IPG-other
IPG-select

1619
4114
4029
85

55.4
58.6
58.7
52.9

60.0
64.1
64.2
60.1

4.6
5.5
5.4
7.2

Improved frequency of confusion
Non-IPG
IPG
IPG-other
IPG-select

1720
4160
4067
93

33.2
36.1
36.3
28.4

36.5
40.8
40.8
40.4

3.3
4.7
4.5
12

Stabilization in bathing
Non-IPG
IPG
IPG-other
IPG-select

1805
4184
4113
71

91.1
91.6
91.7
87.1

92.5
93.0
93.1
91.2

1.4
1.4
1.4
4.1

Any emergent care provided
Non-IPG
IPG
IPG-other
IPG-select

1856
4202
3637
565

27.0
25.4
24.3
32.5

26.5
23.8
23.1
28.6

⫺0.5
⫺1.6
⫺1.2
⫺3.9

Received acute care hospitalization
Non-IPG
IPG
IPG-other
IPG-select

1859
4202
3746
456

33.4
31.7
30.8
39.3

34.8
31.7
30.9
38.4

1.4
0
0.1
⫺0.9

Improved ambulation or locomotion
Non-IPG
IPG
IPG-other
IPG-select

* IPG ⫽ identified participant group. For a given measure, the IPG-select subgroup consists of IPG providers that elected to focus on that measure, and the IPG-other
subgroup consists of IPG providers that selected other measures.
† Mean of rates at individual agencies across all U.S. states or jurisdictions at baseline, 1 May 2001–30 April 2002. Rates at individual agencies are the number of patients
as a percentage of eligible patients.
‡ Mean of rates at individual agencies across all U.S. states or jurisdictions at remeasurement, 1 February 2004 –31 January 2005. Higher values at remeasurement indicate
improved performance, except for emergent care and acute hospitalization, for which lower values indicate improvement.
www.annals.org

5 September 2006 Annals of Internal Medicine Volume 145 • Number 5 349

Improving Patient Care

Assessment of the Medicare Quality Improvement Organization Program

Table 4. Hospital Performance on Selected Quality Measures at Baseline and Remeasurement
Quality Measure

Acute myocardial infarction
Aspirin given at arrival
Aspirin prescribed at discharge
Angiotensin-converting enzyme inhibitor given for left ventricular
systolic dysfunction‡
Adult smoking cessation advice or counseling provided
␤-Blocker prescribed at discharge
␤-Blocker given at arrival

Mean Rate, %

Change in
Performance,
percentage
points

Baseline*

Remeasurement†

83.6
86.3
69.5

86.8
91.4
72.0

3.2
5.1
2.5

43.2
80.2
73.2

71.6
90.1
82.0

28.4
9.9
8.8

Heart failure
Discharge instructions provided
Left ventricular failure assessment
Angiotensin-converting enzyme inhibitor given for left ventricular
systolic dysfunction‡
Adult smoking cessation advice or counseling provided

5.5
78.3
64.0

20.8
85.5
63.8

15.3
7.2
–0.2

25.6

61.6

Pneumonia
Initial antibiotic given within 4 hours of arrival
Initial antibiotic given consistent with guidelines
Blood culture done within 24 hours
Blood culture done before first antibiotic received
Influenza immunization or screening
Pneumococcal immunization or screening
Adult smoking cessation advice or counseling provided
Oxygenation assessment

61.8
68.5
62.1
81.1
24.8
22.9
32.1
98.0

69.0
74.9
73.6
81.9
43.2
50.2
57.4
99.0

7.2
6.4
11.5
0.8
18.4
27.3
25.3
1.0

Surgical infection prevention
Antibiotic given within 1 hour before incision
Prophylactic antibiotic consistent with guidelines
Prophylactic antibiotics withdrawn within 24 hours

46.2
91.4
37.1

68.5
91.2
51.6

22.3
–0.2
14.5

* Mean of rates from individual hospitals across all U.S. states or jurisdictions at baseline, first quarter of 2002. The rates from individual hospitals are the number of patients
as a percentage of eligible patients.
† Mean of rates from individual hospitals across all U.S. states or jurisdictions at remeasurement, fourth quarter of 2004. Higher values at remeasurement indicate improved
performance.
‡ The definitions for this quality measure changed between the first quarter of 2001 and the fourth quarter of 2004.

The 2005 National Healthcare Quality Report recently released by the Agency for Healthcare Research and
Quality reported 4-fold greater improvement on measures
on which QIOs worked than on other measures included
in the report (35). The Institute of Medicine, however, in
a statutorily mandated review of the QIO Program whose
results were released in February 2006, concluded that “the
quality of health care received by Medicare beneficiaries
has improved over time” but “the existing evidence is inadequate to determine the extent to which the QIO program has contributed directly to those improvements”
(36).
We assessed the effect of technical assistance by comparing results among providers that received greater or
lesser amounts of assistance from QIOs. Such a comparison was not possible in the hospital setting because QIOs
were responsible for working with all hospitals in the state
and we did not have reliable data about the relative intensity of assistance in this setting. In other settings, however,
we found that providers that were recruited specifically by
the QIO for receipt of assistance (those in an IPG) improved more on 18 of 20 measures than did providers who
350 5 September 2006 Annals of Internal Medicine Volume 145 • Number 5

were not recruited, and improvement on the other 2 measures was similar. Nursing homes and home health agencies improved more on all measures on which they chose to
work with the QIO (IPG-select) than on other measures
(IPG-other).
Although there are potential limitations to the data
used to assess trends in clinical measure results, evidence
suggests that our findings are generally valid and reliable.
Data for nursing homes and home health agencies are selfreported and therefore are subject to reporting bias that
may be heightened by public reporting. However, these
data are linked to payment, and providers may be penalized if they report incorrect information. According to
Sangl and colleagues (37), the reliability of the Outcomes
Assessment and Information Set (home health agencies)
and the Minimum Data Set (nursing homes) data are acceptable, although evidence for the validity of the quality
measures themselves is mixed. Kinatukara and associates
(38) demonstrated low interrater reliability among experts
using the Outcomes Assessment and Information Set in
test situations. The clinical hospital data are abstracted
through independent review of medical records by the clinwww.annals.org

Assessment of the Medicare Quality Improvement Organization Program

ical data abstraction contractors, who use standardized data
collection procedures with rigorous internal quality control
procedures (8). The trends in hospital measures are consistent with those recently reported by the Joint Commission
on Accreditation of Healthcare Organizations in which
self-reported data by hospitals seeking accreditation during
this period were used (5). Physician office claims data
sometimes understate utilization (39).
Another potential limitation is the dearth of quantitative information on the intensity of assistance received by
the participant groups. By contract, QIOs were required to
improve statewide rates for the 7th Scope of Work quality
measures and, more specifically, to improve them within
an identified subset of providers, the IPG. Contract monitoring showed that IPG providers received greater assistance than did non-IPG providers, and we further confirmed this finding retrospectively for nursing homes (but
not for other settings). Because non-IPG providers also
received some QIO assistance, the observed difference between the 2 groups may be less than it would be if nonIPGs had received no assistance.
Our findings are consistent with an effect of QIO
technical assistance on performance. For many programs,
identifying recipients of assistance, helping them, and finding improvement might be a sufficient demonstration of
program impact. However, there are alternative explanations for our findings. Chance is an unlikely explanation,
given that we are working with population data sets and all
of the trends are in the same direction. Secular trends,
particularly public reporting, may explain some of the observed improvement, but not all of the measures improved,
and the greater improvement for the IPG-select providers
argues against this as the sole explanation.

Improving Patient Care

Our findings could be influenced by selection bias and
regression to the mean. The IPG was presumably selected
in part by the QIOs because they viewed these providers as
likely to improve on the basis of their baseline performance
and the QIO’s assessment of their internal capacity and
motivation for improvement. The baseline performance for
the IPG was generally similar to that of the non-IPG; the
finding of greater improvement by this group for virtually
all measures is therefore inconsistent with regression to the
mean as a sufficient explanation for our findings. Selection
bias could account for some of the difference between the
IPG and non-IPG providers, but within the group of providers that the QIO recruited—the IPG—the greater improvement among IPG-select providers than IPG-other
providers is not explainable simply by a selection effect.
The IPG-select providers, however, had worse baseline
performance than the IPG-other providers, and there may
be interactive effects with other factors, such as provider
size and differences among QIOs at the state level. We
attempted to explore these relationships through more detailed analysis. We stratified the various comparison groups
into quartiles by baseline performance and further stratified
by provider size. We also used inferential statistics to control for differences among these groups in baseline performance, size, and other factors. Ultimately, we decided not
to report such analyses, given the lack of an a priori experimental design that would have allowed unbiased estimates
of QIO effect.
In summary, improvements on most measures in the
7th Scope of Work were greater for the providers with
which the QIO worked closely and were greater for the
measures for which providers requested and received QIO
technical assistance. These findings are consistent with an

Table 5. Physician Office Performance on Quality Measures at Baseline and Remeasurement*
Quality Measure

Chronic disease: diabetes
Biennial retinal eye examination by an eye care
professional
Non-IPG
IPG
Annual hemoglobin A1c testing
Non-IPG
IPG
Biennial lipid profile determination
Non-IPG
IPG
Preventive services: breast cancer
Biennial screening mammography
Non-IPG
IPG

Patients at
Risk at
Baseline,
n†

Mean Rate, %

Change in
Performance,
percentage
points

Baseline

Remeasurement

2 062 173
192 233

69.1
69.6

69.0
71.3

⫺0.1
0.7

2 062 173
192 233

76.6
76.7

82.8
85.4

6.2
8.7

2 062 173
192 233

75.1
76.1

84.8
87.3

9.7
11.2

2 697 029
210 573

59.2
64.9

58.6
65.5

0.6
⫺0.1

* IPG ⫽ identified participant group. For a given measure, the IPG-select subgroup consists of IPG providers that elected to focus on that measure, and the IPG-other
subgroup consists of IPG providers that selected other measures.
† Three different sets of patients, 1 per quality improvement organization “wave” of baseline and remeasurement, were included.
www.annals.org

5 September 2006 Annals of Internal Medicine Volume 145 • Number 5 351

Improving Patient Care

Assessment of the Medicare Quality Improvement Organization Program

effect of the QIO Program and an effect of QIO technical
assistance. The Program’s effect is conjoined with that of
other CMS initiatives and those of other stakeholders. Secular trend, regression to the mean, and selection bias may
also have contributed to our findings related to QIO technical assistance.
Our study demonstrates the difficulty in evaluating a
program that aims to offer assistance to those who request
it and to achieve the maximum possible improvement nationally and within each U.S. state. To mitigate this difficulty, program interventions and related data collection
techniques must be prospectively designed. Evidence of effect is likely to be most difficult to distinguish from other
factors when historical controls are used, more distinguishable when comparison groups are used that are well
matched to the intervention group, and best when selection bias is eliminated through randomized selection of the
intervention group.
To improve our ability to assess program impact in the
QIO 8th Scope of Work, which was launched in August
2005, we have made 3 changes. First, we have reduced the
contract requirements for statewide improvement and will
prospectively collect information on the level of assistance
received by IPG and non-IPG providers. Second, we will
seek to match IPG providers with controls in the non-IPG
group. Finally, we will use an independently administered
survey that will ask providers to assess the extent to which
they would have achieved improvement without QIO assistance.
For the QIO 9th Scope of Work, which will begin in
August 2008, we are in the process of convening an evaluation workgroup that will make recommendations on
program design for that contract period, on the basis of the
work of a contractor currently in place and on advice from
a group of independent technical experts. Through that
process, we will consider the potential for more fundamental redesign, such as randomized selection of the IPG, delayed implementation of assistance for a subset of providers, or both.
From the Centers for Medicare & Medicaid Services, Baltimore, Maryland, Kansas City, Missouri, and Boston, Massachusetts; Oklahoma
Foundation for Medical Quality, Oklahoma City, Oklahoma; and
Northeast Health Care Quality Foundation, Dover, New Hampshire.
Disclaimer: The opinions herein are those of the authors and are not
necessarily those of the Centers for Medicare & Medicaid Services.
Disclosure: William Rollow, MD, MPH, had full access to all of the

data in the study and takes responsibility for the integrity of the data and
the accuracy of the data analysis.
Acknowledgments: The authors acknowledge Judith B. Kaplan, MS,

Centers for Medicare & Medicaid Services; Mark Gottlieb, PhD, New
Mexico Medical Review Association; Meghan B. Harris, MS, Ohio
KePRO; Michael J. McInerney, PhD, Mountain Pacific Quality Health
Foundation; and Rodney J. Presley, PhD, Georgia Medical Care Foundation, for their insights and helpful suggestions. They also thank the
352 5 September 2006 Annals of Internal Medicine Volume 145 • Number 5

data team at the Colorado Foundation for Medical Care (Kris Mattivi,
Beth Stevens, Steve Anderson, and Laura Palmer) and Sean Hunt of
Quality Insights of Pennsylvania for expert assistance in trending the
nursing home and home health measures.
Grant Support: All funding for this work was provided by the Centers
for Medicare & Medicaid Services.
Potential Financial Conflicts of Interest: All authors work for the

federal government on administering the Medicare Quality Improvement Organization Program or work in a Quality Improvement Organization as one of the program contractors.
Requests for Single Reprints: William Rollow, MD, MPH, Centers for

Medicare & Medicaid Services, Mail Stop S3-02-01, 7500 Security Boulevard, Baltimore, MD 21244-1850; e-mail, [email protected]
.gov.
Current author addresses are available at www.annals.org.

References
1. Institute of Medicine, Committee on Quality of Health Care in America.
Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Pr; 2001.
2. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al.
The quality of health care delivered to adults in the United States. N Engl J Med.
2003;348:2635-45. [PMID: 12826639]
3. Agency for Healthcare Research and Quality. 2004 National Healthcare
Quality Report. Rockville, MD: U.S. Department of Health and Human Services; 2004.
4. National Committee for Quality Assurance. 2004 State of Health Care Quality Report. Washington, DC: National Committee for Quality Assurance; 2004.
5. Williams SC, Schmaltz SP, Morton DJ, Koss RG, Loeb JM. Quality of care
in U.S. hospitals as reflected by standardized measures, 2002-2004. N Engl J
Med. 2005;353:255-64. [PMID: 16034011]
6. Marciniak TA, Ellerbeck EF, Radford MJ, Kresowik TF, Gold JA, Krumholz
HM, et al. Improving the quality of care for Medicare patients with acute myocardial infarction: results from the Cooperative Cardiovascular Project. JAMA.
1998;279:1351-7. [PMID: 9582042]
7. Chu LA, Bratzler DW, Lewis RJ, Murray C, Moore L, Shook C, et al.
Improving the quality of care for patients with pneumonia in very small hospitals.
Arch Intern Med. 2003;163:326-32. [PMID: 12578513]
8. Jencks SF, Huff ED, Cuerdon T. Change in the quality of care delivered to
Medicare beneficiaries, 1998-1999 to 2000-2001. JAMA. 2003;289:305-12.
[PMID: 12525231]
9. Snyder C, Anderson G. Do quality improvement organizations improve the
quality of hospital care for Medicare beneficiaries? JAMA. 2005;293:2900-7.
[PMID: 15956635]
10. Bradley EH, Carlson MD, Gallo WT, Scinto J, Campbell MK, Krumholz
HM. From adversary to partner: have quality improvement organizations made
the transition? Health Serv Res. 2005;40:459-76. [PMID: 15762902]
11. Pace KB, Johnson KE. Quality improvement in home health care. Am J
Nurs. 2006;106:38.
12. Sprague L. Contracting for quality: Medicare’s quality improvement organizations. NHPF Issue Brief. 2002:1-15. [PMID: 12061386]
13. Medicare. Nursing Home Compare. Available at www.medicare.gov/NH
Compare/. Accessed 11 May 2006.
14. Centers for Medicare & Medicaid Services. Home Health Quality Initiatives. Overview. Available at www.cms.hhs.gov/homehealthqualityinits/. Accessed
11 May 2006.
15. U.S. Department of Health and Human Services. Hospital Compare. Available at www.hospitalcompare.hhs.gov. Accessed 11 May 2006.
16. Lee D, Lopez L. An invitational workshop on collaboration between quality
improvement organizations and business coalitions. Jt Comm J Qual Improv.
1997;23:334-41. [PMID: 9234075]
17. Hertz KT, Fabrizio N. The many faces of QIOs, and what their latest
incarnation means for your practice. MGMA Connex. 2005;5:42-5, 1. [PMID:
www.annals.org

Assessment of the Medicare Quality Improvement Organization Program
16173178]
18. Simpson RJ Jr, Weiser RR, Naylor S, Sueta CA, Metts AK. Improving care
for unstable angina patients in a multiple hospital project sponsored by a federally
designated quality improvement organization. Am J Cardiol. 1997;80:80H-84H.
[PMID: 9373005]
19. Ellerbeck EF, Kresowik TF, Hemann RA, Mason P, Wiblin RT, Marciniak
TA. Impact of quality improvement activities on care for acute myocardial infarction. Int J Qual Health Care. 2000;12:305-10. [PMID: 10985268]
20. Hayes R, Bratzler D, Armour B, Moore L, Murray C, Stevens BR, et al.
Comparison of an enhanced versus a written feedback model on the management
of Medicare inpatients with venous thrombosis. Jt Comm J Qual Improv. 2001;
27:155-68. [PMID: 11242721]
21. Kiefe CI, Allison JJ, Williams OD, Person SD, Weaver MT, Weissman
NW. Improving quality improvement using achievable benchmarks for physician
feedback: a randomized controlled trial. JAMA. 2001;285:2871-9. [PMID:
11401608]
22. Mehta RH, Montoye CK, Gallogly M, Baker P, Blount A, Faul J, et al.
Improving quality of care for acute myocardial infarction: The Guidelines Applied in Practice (GAP) Initiative. JAMA. 2002;287:1269-76. [PMID:
11886318]
23. Kissam S, Gifford D, Parks P, Patry G, Palmer L, Wilkes L, et al. Approaches to quality improvement in nursing homes: lessons learned from the
six-state pilot of CMS’s Nursing Home Quality Initiative. BMC Geriatr. 2003;
3:2. [PMID: 12753699]
24. Weingarten JP Jr, Fan W, Peacher-Ryan H, Brittman S, FitzGerald D,
Jowers S, et al. Hospital quality improvement activities and the effects of interventions on pneumonia: a multistate study of Medicare beneficiaries. Am J Med
Qual. 2004;19:157-65. [PMID: 15368780]
25. Kilo CM. A framework for collaborative improvement: lessons from the
Institute for Healthcare Improvement’s Breakthrough Series. Qual Manag Health
Care. 1998;6:1-13. [PMID: 10339040]
26. Allen BL, Burt P, Roychoudhury C, Chen B. Analysis of OBQI outcomes in
participating Michigan home health agencies. J Nurs Care Qual. 2004;19:14955. [PMID: 15077832]
27. Centers for Medicare & Medicaid Services. Minimum Data Sets 2.0. Overview. Available at www.cms.hhs.gov/minimumdatasets20/. Accessed 11 May
2006.

www.annals.org

Improving Patient Care

28. Centers for Medicare & Medicaid Services. Home Health Quality Initiatives. OASIS Data Set. Available at www.cms.hhs.gov/HomeHealthQualityInits
/12_HHQIOASISDataSet.asp#TopOfPage. Accessed 11 May 2006.
29. Bratzler DW, Houck PM; Surgical Infection Prevention Guidelines Writers Workgroup; American Academy of Orthopaedic Surgeons; American Association of Critical Care Nurses; American Association of Nurse Anesthetists; et
al. Antimicrobial prophylaxis for surgery: an advisory statement from the National Surgical Infection Prevention Project. Clin Infect Dis. 2004;38:1706-15.
[PMID: 15227616]
30. Bratzler DW, Houck PM, Richards C, Steele L, Dellinger EP, Fry DE, et
al. Use of antimicrobial prophylaxis for major surgery: baseline results from the
National Surgical Infection Prevention Project. Arch Surg. 2005;140:174-82.
[PMID: 15724000]
31. Dellinger EP, Hausmann SM, Bratzler DW, Johnson RM, Daniel DM,
Bunt KM, et al. Hospitals collaborate to decrease surgical site infections. Am J
Surg. 2005;190:9-15. [PMID: 15972163]
32. Polk HC Jr. Keeping score and the application of known principles [Editorial]. Am J Surg. 2005;190:16-7. [PMID: 15972164]
33. Jencks SF. Quality improvement organizations and hospital care [Letter].
JAMA. 2005;294:2028. [PMID: 16249410]
34. Sugarman JR, Baumgardner GA. Quality improvement organizations and
hospital care [Letter]. JAMA. 2005;294:2029. [PMID: 16249412]
35. Agency for Healthcare Research and Quality. 2005 National Healthcare
Quality Report. Rockville, MD: U.S. Department of Health and Human Services; 2005.
36. Institute of Medicine. Medicare’s Quality Improvement Organization Program: Maximizing Potential. Washington, DC: Institute of Medicine; March
2006.
37. Sangl J, Saliba D, Gifford DR, Hittle DF. Challenges in measuring nursing
home and home health quality: lessons from the First National Healthcare Quality Report. Med Care. 2005;43:I24-32. [PMID: 15746587]
38. Kinatukara S, Rosati RJ, Huang L. Assessment of OASIS reliability and
validity using several methodological approaches. Home Health Care Serv Q.
2005;24:23-38. [PMID: 16203688]
39. Mouchawar J, Byers T, Warren M, Schluter WW. The sensitivity of Medicare billing claims data for monitoring mammography use by elderly women.
Med Care Res Rev. 2004;61:116-27. [PMID: 15035859]

5 September 2006 Annals of Internal Medicine Volume 145 • Number 5 353

Annals of Internal Medicine
Current Author Addresses: Drs. Rollow, Lied, and McGann, Mr.

Poyer, and Mr. Kambic: Centers for Medicare & Medicaid Services,
Mail Stop S3-02-01, 7500 Security Boulevard, Baltimore, MD 212441850.
Dr. LaVoie: Centers for Medicare & Medicaid Services, Kansas City
Regional Office, 601 East 12th Street, Suite 235, Kansas City, MO
64106.
Drs. Bratzler and Ma: Oklahoma Foundation for Medical Quality,
14000 Quail Springs Parkway, Suite 400, Oklahoma City, OK 73134.
Dr. Huff: Centers for Medicare & Medicaid Services, Boston Regional
Office, JFK Federal Building, Boston, MA 02203.
Dr. Ramunno: Northeast Health Care Quality Foundation, 15 Old Rollinsford Road, Suite 302, Dover, NH 03820.

APPENDIX: CLARIFICATION ADDENDUM
AND REPORTING OF MEASURES

FOR

SELECTION

Performance data for nursing home and home health agency
measures were publicly reported for the first time in 2002 and
2003, respectively. Criteria for selection of a particular measure
for quality improvement were discussed among QIOs in “community of practice” national telephone conferences, e-mail listservs, and national meetings. For nursing homes, the CMS had
no specific requirements for measure selection; the principal advice was to choose a measure for which there was significant
“room for improvement.” In practice, this meant avoiding measures for which a facility’s current performance was well above
the national or state average or for which the provider had
achieved the best possible performance. The guidance for home
health agencies was found in the formal Outcome-Based Quality
Improvement training system. Every home health agency in the
IPG was encouraged to use the Outcome-Based Quality Improvement system, which includes a data-driven procedure for
identifying measures for which an agency had significant room
for improvement. In the outpatient setting, QIOs were evaluated
on the relative improvement of all IPG practices on all 4 measures.
Nursing Homes
We report data on 5 nursing home measures. Four of these
measures applied to long-term residents: the percentage of residents whose need for help with daily activities had increased from
the previous assessment, the percentage with moderate pain daily
or severe pain in the past 7 days, the percentage with pressure
ulcers in the past 7 days, and the percentage who were physically
restrained daily in the past 7 days. The fifth measure was the
percentage of short-stay (post–acute care) residents with moderate or severe pain. We do not report data for 3 nursing home
quality measures: percentage of short-stay residents with delirium
(because few nursing homes selected this measure), percentage of
chronic care residents with an infection (because of varying definitions of this measure across states), and percentage of residents

www.annals.org

whose ability to move about in and around their room got worse
(because of concerns about the reliability of the measure). Four
hundred thirty-four nursing homes targeted decline in activities
of daily living for improvement during the 7th Scope of Work,
1809 targeted pain in chronic care residents, 609 targeted use of
physical restraints, 1380 targeted development of pressure ulcers,
and 668 targeted post–acute care pain.
Home Health Agencies
We report data for 11 of 41 publicly reported measures.
These 11 measures account for more than 75% of all home
health agencies that selected measures for improvement. Four
measures are related to mobility: improved ambulation, improved transferring, improved toileting, and improvement in
pain interfering with activity. Another 4 measures are related to
daily needs: improved upper-body dressing, improved bathing,
improved management of oral medications, and stabilization in
bathing. Two measures are related to medical emergencies: acute
hospital admission and emergent care. One measure, improvement in confusion, is related to mental status. Improved toileting, upper-body dressing, confusion frequency, stabilization in
bathing, and emergent care were each targeted by 50 to 100
home health agencies. Improved ambulation, transferring, pain,
bathing, and medication management and acute care hospitalization were each targeted by 250 to 700 home health agencies.
Hospitals
We report national rates for 21 of 23 publicly reported measures: 6 acute myocardial infarction measures, 4 heart failure
measures, 8 pneumonia measures, and 3 surgical infection prevention measures. Two measures were excluded owing to very
low numbers of eligible cases: timely thrombolysis and timely
percutaneous coronary intervention.
Physician Offices
We report national rates for all 4 physician office measures,
based on analysis of Medicare claims for beneficiaries enrolled in
traditional fee-for-service Medicare. Three of these measures pertain to patients with diabetes: biennial retinal eye examination by
an eye care professional, annual testing of hemoglobin A1c, and
biennial lipid profile. The fourth measure was biennial mammography for women 52 to 69 years of age at the end of the 2-year
period. The QIOs were also assessed on the statewide performance of these 4 measures on the basis of analysis of Medicare
claims. In addition, QIOs were required to improve their state’s
rates of influenza and pneumococcal vaccination of Medicare
beneficiaries 65 years of age or older as assessed by the Consumer
Health Plans Survey. The QIOs were not required to improve
vaccination rates for their IPG practitioners because we did not
have a reliable measure of vaccination rates for each IPG practitioner.

5 September 2006 Annals of Internal Medicine Volume 145 • Number 5 W-87

Appendix Table 1. Number of Nursing Homes at Baseline and Remeasurement, Number of Nursing Homes in an Identified
Participant Group at Baseline and Remeasurement, and Proportion of Identified Participant Group Nursing Homes, by U.S. State or
Jurisdiction
U.S. State or Jurisdiction

Alabama
Alaska
Arizona
Arkansas
California
Colorado
Connecticut
Delaware
District of Columbia
Florida
Georgia
Hawaii
Idaho
Illinois
Indiana
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Puerto Rico
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Vermont
Virgin Islands
Virginia
Washington
West Virginia
Wisconsin
Wyoming
All states or jurisdictions

All Nursing Homes, n

Identified Participant Group
Nursing Homes, n

First
Quarter,
2002

Second
Quarter,
2004

Second
Quarter,
2002

Second
Quarter,
2004

230
15
137
251
1349
223
254
42
21
713
362
46
83
854
548
464
377
301
321
121
246
502
435
426
204
542
103
229
43
83
359
82
671
414
85
996
373
144
759
7
97
177
112
343
1144
91
44
1
282
268
141
406
39
16 560

228
14
134
237
1311
215
246
42
20
692
362
46
80
820
516
458
367
294
309
118
240
475
431
415
205
523
101
228
43
81
355
81
667
420
83
985
361
138
725
6
95
177
112
335
1136
90
41
1
280
249
133
401
39
16 161

35
15
21
25
191
33
31
13
14
74
56
15
13
90
86
70
57
45
48
18
33
75
64
68
38
81
15
35
15
15
54
15
100
63
15
149
56
20
113
6
16
23
17
52
172
15
15
1
35
46
20
73
15
2480

35
14
21
25
188
32
31
13
13
74
56
15
13
90
84
70
57
45
48
18
33
74
64
67
37
81
15
35
15
15
54
15
100
63
15
149
56
20
112
5
16
23
17
51
168
15
14
1
35
43
20
73
15
2458

Identified Participant
Group Nursing
Homes at Baseline,
%*

15
100
15
10
14
15
12
31
67
10
15
33
16
11
16
15
15
15
15
15
13
15
15
16
19
15
15
15
35
18
15
18
15
15
18
15
15
14
15
86
16
13
15
15
15
16
34
100
12
17
14
18
38
15

* Proportion of nursing homes in a U.S. state or jurisdiction that were included in the nursing home identified participant group for that state or jurisdiction.

W-88 5 September 2006 Annals of Internal Medicine Volume 145 • Number 5

www.annals.org

Appendix Table 2. Number of Home Health Agencies at Baseline and Remeasurement, Number of Home Health Agencies in the
Identified Participant Group at Baseline and Remeasurement, and Proportion of Identified Participant Group Home Health
Agencies at Baseline, by U.S. State or Jurisdiction
U.S. State or Jurisdiction

All Home Health Agencies, n*

1 August 2002

31 July 2005

1 August 2002

31 July 2005

Identified Participant
Group Home Health
Agencies as of 1
August 2002, %†

140
16
62
175
541
119
81
14
12
354
87
17
48
274
172
172
129
108
229
33
44
110
190
223
60
160
47
64
35
35
51
61
198
162
30
316
169
61
271
46
23
72
46
139
818
38
12
2
146
59
63
119
29
6682

141
16
75
173
653
130
83
16
21
583
96
19
49
339
185
171
132
102
224
30
47
113
275
209
57
154
38
68
57
36
50
68
188
165
26
394
200
60
283
47
22
69
42
138
1339
54
12
2
161
59
60
118
24
7873

110
6
41
88
283
86
46
13
6
279
77
16
17
124
101
149
81
74
161
31
38
89
124
94
54
113
22
52
25
35
42
26
178
147
28
152
156
47
157
42
20
46
40
107
417
25
12
1
122
35
45
94
22
4396

109
6
41
88
266
80
46
12
6
268
77
16
17
121
97
146
76
72
159
29
38
84
120
88
51
111
22
50
25
34
41
24
168
145
25
145
148
42
148
42
19
45
39
105
412
25
12
1
118
35
41
91
19
4245

79
38
66
50
52
72
57
93
50
79
89
94
35
45
59
87
63
69
70
94
86
81
65
42
90
71
47
81
71
100
82
43
90
91
93
48
92
77
58
91
87
64
87
77
51
66
100
50
84
59
71
79
76
66

Identified Participant Group Home
Health Agencies, n*

* Pediatric home health agencies were excluded.
† Proportion of home health agencies in a U.S. state or jurisdiction that were included in the home health agency identified participant group for that state or jurisdiction.

www.annals.org

5 September 2006 Annals of Internal Medicine Volume 145 • Number 5 W-89

Appendix Table 3. Number of Primary Care Practitioners at Baseline, Number of Physicians in an Identified Participant Group at
Baseline, and Proportion of Identified Participant Group Primary Care Practitioners, by U.S. State or Jurisdiction
U.S. State or Jurisdiction

All Primary Care
Practitioners, n*

Identified Participant
Group Primary Care
Practitioners, n

Identified Participant
Group Primary Care
Practitioners at
Baseline, %†

2828
393
2886
1705
24 705
3044
3306
485
906
10 636
5016
1023
723
10 304
3914
1686
1915
2666
3150
969
4837
6350
6936
4190
1479
3641
585
1347
1050
900
7128
1201
19 662
5534
529
8395
1859
2496
10 276
728
972
2733
562
4116
12 237
1251
598
116
5183
4507
1369
3987
335
209 349

214
29
213
109
3074
223
163
52
64
798
245
89
57
752
233
126
159
146
236
72
348
472
521
216
110
205
44
98
62
51
486
65
1004
308
36
447
117
208
783
43
70
185
43
201
918
78
37
19
302
338
70
300
24
15 263

8
7
7
6
12
7
5
11
7
8
5
9
8
7
6
7
8
5
7
7
7
7
8
5
7
6
8
7
6
6
7
5
5
6
7
5
6
8
8
6
7
7
8
5
8
6
6
16
6
7
5
8
7
7

* Defined as physicians or midlevel providers (MP, PA) who had a specialty code of general practice, family practice, obstetrics/gynecology, or internal medicine and
subspecialties that predominantly performed primary care.
† Percentage of primary care practitioners in a U.S. state or jurisdiction who were included in the physician office identified participant group for that state or jurisdiction.

W-90 5 September 2006 Annals of Internal Medicine Volume 145 • Number 5

www.annals.org

Appendix Figure 1. Locations of quality improvement organizations (QIOs).

The size of each QIO is represented by the size of each star. Maine and Vermont QIO coverage is directed from the New Hampshire QIO office (Maine,
New Hampshire, and Vermont are similar in QIO size).

Appendix Figure 2. Stratification of providers.

All nursing homes, home health agencies, and
practitioners in a state or jurisdiction

Nursing homes, home health

agencies, and practitioners

agencies, and practitioners that did

recruited to IPGs

not agree to be, or were not
recruited as, identified participants
(non-IPG)*

For a given measure, all

providers (nursing homes

or home health agencies)

that targeted the specific

that targeted other

measure (IPG-select

measure(s) (IPG-other

for a given measure)

Quality improvement organizations (QIOs) were required to recruit a limited number of nursing homes, home health agencies, and physician offices into
identified participant groups (IPGs) for focused quality improvement interventions. Facilities not participating in the IPG for a given setting are labeled
“non-IPG” for that setting. For a given quality measure, IPG nursing homes and home health agencies are subdivided into those focusing on a specified
quality measure (IPG-select) and those not focusing on the specified measure (IPG-other). *Low-volume nursing homes and home health agencies and
non–primary care physicians were not eligible for the IPG for contractual reasons.
www.annals.org

5 September 2006 Annals of Internal Medicine Volume 145 • Number 5 W-91

Appendix Figure 3. Time-trend data for home health agencies: acute care hospitalization (top) and any emergent care (bottom).

IPG-other (n = 3660)

Non-IPG (n = 1610)

IPG-select (n = 441)

National (average n = 6802)

Acute Care Hospitalization, %

34
32

28
26
0

Months from Start of 7th SOW Contract

Any Emergent Care, %

IPG-other (n = 3548)

Non-IPG (n = 1604)

IPG-select (n = 553)

National (average n = 6788)

28
26

22
20
0

Months from Start of 7th SOW Contract

All rates are calculated as 12-month averages, ending with the last day of the month shown. “0” represents the 12-month period ending the month before
the start of the contract. For the national group, the average number of facilities across all periods is given. All other groups include only home health
agencies with valid data in the first and last time points shown. For these quality measures, lower scores indicate better performance. IPG ⫽ identified
participant group; SOW ⫽ Scope of Work. For a given measure, the IPG-select subgroup consists of IPG providers that elected to focus on that measure,
and the IPG-other subgroup consists of IPG providers that selected other measures.

W-92 5 September 2006 Annals of Internal Medicine Volume 145 • Number 5

www.annals.org

Appendix Figure 4. Time-trend data for home health agencies: improvement in ambulation or locomotion (top) and improvement in
bathing (bottom).

Improvement in Ambulation/Locomotion, %

IPG-other (n = 3664)

Non-IPG (n = 1420)

IPG-select (n = 311)

National (average n = 6442)

38
36
34
32
30
28
26
24
22
20
0

Months from Start of 7th SOW Contract

Improvement in Bathing, %

IPG-other (n = 3721)

Non-IPG (n = 1428)

IPG-select (n = 259)

National (average n = 6456)

61
59
57
55
53
51
49
47
45
0

Months from Start of 7th SOW Contract

All rates are calculated as 12-month averages, ending with the last day of the month shown. “0” represents the 12-month period ending the month before
the start of the contract. For the national group, the average number of facilities across all periods is given. All other groups include only home health
agencies with valid data in the first and last time points shown. For these quality measures, higher scores indicate better performance. IPG ⫽ identified
participant group; SOW ⫽ Scope of Work. For a given measure, the IPG-select subgroup consists of IPG providers that elected to focus on that measure,
and the IPG-other subgroup consists of IPG providers that selected other measures.

www.annals.org

5 September 2006 Annals of Internal Medicine Volume 145 • Number 5 W-93

Appendix Figure 5. Time-trend data for home health agencies: improvement in confusion frequency (top) and improvement in
management of oral medications (bottom).

Improvement in Confusion Frequency, %

45
43
41
39
37
35
33
31
29
27

IPG-other (n = 3676)

Non-IPG (n = 1243)

IPG-select (n = 86)

National (average n = 6032)

25
0

Improvement in Management of Oral Medications, %

Months from Start of 7th SOW Contract

40
38
36
34
32
30
28
26
24

IPG-other (n = 3383)

Non-IPG (n = 1315)

IPG-select (n = 476)

National (average n = 6218)

20
0

Months from Start of 7th SOW Contract

All rates are calculated as 12-month averages, ending with the last day of the month shown. “0” represents the 12-month period ending the month before
the start of the contract. For the national group, the average number of facilities across all periods is given. All other groups include only home health
agencies with valid data in the first and last time points shown. For these quality measures, higher scores indicate better performance. IPG ⫽ identified
participant group; SOW ⫽ Scope of Work. For a given measure, the IPG-select subgroup consists of IPG providers that elected to focus on that measure,
and the IPG-other subgroup consists of IPG providers that selected other measures.

W-94 5 September 2006 Annals of Internal Medicine Volume 145 • Number 5

www.annals.org

Improvement in Pain Interfering with Activity, %

Appendix Figure 6. Time-trend data for home health agencies: improvement in pain interfering with activity (top) and improvement
in toileting (bottom).
65
63

IPG-other (n = 3301)

Non-IPG (n = 1374)

IPG-select (n = 633)

National (average n = 6365)

61
59
57
55
53
51
49
47
45
0

Months from Start of 7th SOW Contract

Improvement in Toileting, %

63
61
59
57
55
53
51
49

IPG-other (n = 3444)

Non-IPG (n = 1125)

IPG-select (n = 51)

National (average n = 5605)

45
0

Months from Start of 7th SOW Contract

All rates are calculated as 12-month averages, ending with the last day of the month shown. “0” represents the 12-month period ending the month before
the start of the contract. For the national group, the average number of facilities across all periods is given. All other groups include only home health
agencies with valid data in the first and last time points shown. For these quality measures, higher scores indicate better performance. IPG ⫽ identified
participant group; SOW ⫽ Scope of Work. For a given measure, the IPG-select subgroup consists of IPG providers that elected to focus on that measure,
and the IPG-other subgroup consists of IPG providers that selected other measures.

www.annals.org

5 September 2006 Annals of Internal Medicine Volume 145 • Number 5 W-95

Improvement in Transferring, %

Appendix Figure 7. Time-trend data for home health agencies: improvement in transferring (top) and improvement in upper-body
dressing (bottom).
55

IPG-other (n = 3478)

Non-IPG (n = 1341)

IPG-select (n = 423)

National (average n = 6276)

51
49
47
45
43
41
39
37
35
0

Months from Start of 7th SOW Contract

Improvement in Upper-Body Dressing, %

70
68

IPG-other (n = 3728)

Non-IPG (n = 1317)

IPG-select (n = 80)

National (average n = 6170)

66
64
62
60
58
56
54
52
50
0

Months from Start of 7th SOW Contract

All rates are calculated as 12-month averages, ending with the last day of the month shown. “0” represents the 12-month period ending the month before
the start of the contract. For the national group, the average number of facilities across all periods is given. All other groups include only home health
agencies with valid data in the first and last time points shown. For these quality measures, higher scores indicate better performance. IPG ⫽ identified
participant group; SOW ⫽ Scope of Work. For a given measure, the IPG-select subgroup consists of IPG providers that elected to focus on that measure,
and the IPG-other subgroup consists of IPG providers that selected other measures.

W-96 5 September 2006 Annals of Internal Medicine Volume 145 • Number 5

www.annals.org

Appendix Figure 8. Time-trend data for home health agencies: stabilization in bathing.
100

Stabilization in Bathing, %

IPG-other (n = 3928)

Non-IPG (n = 1455)

IPG-select (n = 66)

National (average n = 6515)

96
94
92
90
88
86
84
82
80
0

Months from Start of 7th SOW Contract

All rates are calculated as 12-month averages, ending with the last day of the month shown. “0” represents the 12-month period ending the month before
the start of the contract. For the national group, the average number of facilities across all periods is given. All other groups include only home health
agencies with valid data in the first and last time points shown. For this quality measure, higher scores indicate better performance. IPG ⫽ identified
participant group; SOW ⫽ Scope of Work. For a given measure, the IPG-select subgroup consists of IPG providers that elected to focus on that measure,
and the IPG-other subgroup consists of IPG providers that selected other measures.

www.annals.org

5 September 2006 Annals of Internal Medicine Volume 145 • Number 5 W-97

File Type	application/pdf
File Modified	2006-08-11
File Created	2006-08-09