CMS-10294.Supporting Statment Part A (8-23-10)

Contract No.: HHSM-500-2005-00025I (0010)
Mathematica Reference No.: 6514-180

Evaluation of the
Ninth Scope of
Work Quality
Improvement
Organization
Program:
Supporting
Statement for
Paperwork
Reduction Act
Submission

August 19, 2010
Martha Kovac
Sue Felt-Lisk
Arnold Chen
John Hall

Submitted to:
U.S. Department of Health and Human Services
Centers for Medicare & Medicaid Services
Office of Research, Development, and Information
C3-23-04 Central Bldg.
7500 Security Blvd.
Baltimore, MD 21244-1850
Project Officer:
Robert Kambic

Submitted by:
Mathematica Policy Research, Inc.
P.O. Box 2393
Princeton, NJ 08543-2393
Telephone: (609) 799-3535
Facsimile: (609) 799-0005

Project Director:
Myles Maxfield

CONTENTS

Page
A. BACKGROUND .............................................................................................................1
1.  Rationale for Evaluation of the QIO Program...........................................................1 
2.  Evaluation Design for the Ninth Scope of Work QIO Program................................6 
B.  JUSTIFICATION ..........................................................................................................17
1.  Need and Legal Basis ..............................................................................................17 
2.  Information Users ....................................................................................................19 
3.  Use of Information Technology ..............................................................................19 
4.  Duplication of Effort ...............................................................................................20 
5.  Small Businesses .....................................................................................................20 
6.  Less-Frequent Collection ........................................................................................21 
7.  Special Circumstances .............................................................................................21 
8.  Federal Register/Outside Consultation ....................................................................21 
9.  Payments/Gifts to Respondents ...............................................................................22 
10.  Confidentiality .........................................................................................................22 
11.  Sensitive Questions .................................................................................................23 
12.  Burden Estimates (Hours and Wages).....................................................................24 
13.  Capital Costs............................................................................................................28 
14.  Cost to Federal Government....................................................................................28 
15.  Changes to Burden ..................................................................................................39 
16.  Publication/Tabulation Dates ..................................................................................29 
17.  Expiration Date........................................................................................................30 
18.  Certification Statement ............................................................................................31 
C.  SUPPORTING STATEMENT PART B COLLECTION OF INFORMATION
EMPLOYING STATISTICAL METHODS .................................................................33
1. 
2. 
3. 
4. 
5. 

Respondent Universe and Sampling Methods.........................................................33 
Procedures for the Collection of Information..........................................................45 
Methods to Maximize Response Rates ...................................................................52 
Tests of Procedures or Methods ..............................................................................56 
People Involved in Design ......................................................................................57 

REFERENCES ..............................................................................................................................59 

iii

CONTENTS (continued)
Page
APPENDIX A: LEGISLATION FOR THE NINTH SCOPE OF WORK QUALITY
IMPROVEMENT PROGRAM EVALUATION ...................................... A.1
APPENDIX B: FEDERAL REGISTER NOTICE ............................................................. B.1
APPENDIX C: QIO SURVEY AND INVITATION LETTER ......................................... C.1
APPENDIX D: PROJECT DESCRIPTION FOR ADVANCE MAILINGS .....................D.1
APPENDIX E: PROVIDER SURVEYS AND LETTERS ................................................ E.1
APPENDIX F: CASE STUDY DISCUSSION GUIDES AND LETTERS ...................... F.1
APPENDIX G: PARTNER DISCUSSION GUIDE AND LETTER .................................G.1

iv

TABLES

Table

Page

A.1

LIST OF OUTCOME MEASURES FOR THE EVALUATION
OF THE QIO 9TH SOW .........................................................................................10

A.2

QIO SURVEY KEY TOPICS .................................................................................12

A.3

HOSPITAL AND NURSING HOME SURVEYS KEY TOPICS .........................13

A.4

CASE STUDY DISCUSSION TOPICS BY TYPE OF RESPONDENT ..............14

B.1

CONSULTANTS ....................................................................................................22

B.2

QIO SURVEY RESPONSE BURDEN ..................................................................25

B.3

HOSPITAL SURVEY RESPONSE BURDEN ......................................................25

B.4

NURSING HOME SURVEY RESPONSE BURDEN ...........................................26

B.5

CASE STUDY DISCUSSIONS RESPONSE BURDEN .......................................27

B.6

DISCUSSIONS WITH QIO PARTNER ORGANIZATIONS RESPONSE
BURDEN.................................................................................................................28

B.7

QIO PROGRAM EVALUATION DELIVERY SCHEDULE OF REPORTS.......29

C.1

COUNTS OF HOSPITALS IN EACH STRATUM AND ALLOCATION OF
SURVEY SAMPLE ACROSS STRATA ...............................................................38

C.2

COUNTS OF NURSING HOMES COUNTS OF NURSING HOMES IN
EACH STRATUM AND ALLOCATION OF SURVEY SAMPLE ACROSS
STRATA .................................................................................................................39

C.3

EXPECTED SURVEY COMPLETES BY SURVEY TYPE .................................49

v

A. BACKGROUND

The Centers for Medicare & Medicaid Services (CMS) contracted with Mathematica Policy
Research, Inc. (Mathematica) to conduct an evaluation of the Ninth Scope of Work Quality
Improvement Organization (QIO) Program, which is a high priority of the administration and
secretary. The purpose of the study is to design and conduct an analysis evaluating the impact on
national and regional health care processes and outcomes of the Ninth Scope of Work QIO
Program. The QIO Program is national in scope and scale and affects the quality of healthcare of
43 million elderly and disabled Americans. Mathematica will conduct an analysis of (1) primary
data collected via in-depth interviews and surveys of QIOs, health care providers, and other
stakeholders; (2) secondary data reported by QIOs through CMS systems; and (3) CMS
administrative data. Mathematica will also prepare a final report to CMS as well as develop
documents and reports suitable for publication in peer-review journals.
To support the evaluation, we are requesting OMB approval of three surveys—a survey of
QIOs, a survey of hospitals, and a survey of nursing homes—and approval of telephone and inperson discussion guides for use with QIOs, their partner organizations, health care providers,
and community health leaders. All data collection plans pertain to the portion of the evaluation
that focuses on the Ninth SOW Quality Improvement Organization Program.
1.

Rationale for Evaluation of the QIO Program
The QIO Program is one of CMS’s most important tools for monitoring and improving the

quality of care for Medicare beneficiaries as well as for ensuring the integrity of Medicare
expenditures. In fact, it is the largest federal investment in quality improvement infrastructure for
the nation in general—one of the nation’s best hopes for “crossing the quality chasm.” Though
the QIO Program’s purpose is central to the mission of the Office of Clinical Standards and
Quality and to the mission of CMS in general, the cost of the QIO Program is $1.1 billion over
1

the current three-year contract cycle. Although its cost pales beside the cost of the Medicare
program itself, it nonetheless represents a significant expenditure by CMS and the Department of
Health and Human Services (HHS). The centrality of its purpose and its high cost make
evaluating the effectiveness of the QIO Program essential. Ultimately, understanding how
effective the QIO Program is and what approaches are most effective within it will lead to its
improvement, which in turn will lead to improvements in the quality of care received by
beneficiaries and Americans in general.
The statutory mission of the QIO Program, as set forth in section 1862(g) of the Social
Security Act, is to promote the effectiveness, efficiency, economy, and quality of services
delivered to Medicare beneficiaries and to ensure that those services are reasonable and
necessary. The quality strategies of the Medicare QIO Program are carried out by state and
territory-specific QIO contractors working with health care providers in their state or territory, or
in the District of Columbia.
The HHS annual report to Congress on the state of health care quality for all Americans, The
National Healthcare Quality Report, found that improvements in hospital care may have resulted
from public reporting of health care quality measure data, focused quality improvement
programs including the activities of the QIO Program, and policies that supported improvement
initiatives. 1 Examples of improvement in hospital care reported in 2006 include the following:

1

Agency for Healthcare Research and Quality, 2006 National Healthcare Quality Report (Rockville, MD:
AHRQ; December 2006). Available at: http://www.ahrq.gov/qual/nhqr06/nhqr06report.pdf

2

• “The Centers for Medicare & Medicaid Services (CMS) Quality Improvement
Organization (QIO) measures for good heart attack care showed the greatest
improvement of all core measures at 15.0 percent per year. This rate of improvement
is markedly better than the 9.2 percent rate reported last year (2005) and more than 5
times the 2.6 percent overall rate of improvement for all non-hospital core measures.”
• “QIO measures of the quality of hospital care for pneumonia care and for heart failure
also showed high rates of improvement compared with all other measures—
11.7 percent and 8.4 percent, respectively.”
In the Sixth SOW, August 2001 to August 2003, QIOs worked for statewide improvement
on hospital and ambulatory patient quality measures. Jencks et al. evaluated this contract and
showed improvement in 20 of 22 measures nationally, with most states also showing
improvement. 2 However, the extent to which improvement was attributable to the efforts of
QIOs could not be assessed, although the breadth of the improvement was not likely due to
chance alone. CMS maintains that the QIO Program was instrumental in the creation of an
environment that led to improvement.
Two studies of QIO impact during this period were subsequently published. Snyder and
Anderson assessed whether improvement in hospital measures was attributable to the
intervention of the QIOs. 3 In an October 25, 2005, letter to JAMA the authors admitted to
concerns raised about their study methods, which called into question their results. Bradley et al.
assessed QIO impact on hospitals by asking 105 hospital quality improvement directors to
evaluate the assistance provided by the QIOs. 4 One third of respondents said that their
performance would have been different if the QIO had not intervened. About 40 percent said that
2

Jencks et al., “Change in the Quality of Care Delivered to Medicare Beneficiaries, 1998–1999 to 2000–
2001.” JAMA, vol. 289, no. 3, pp. 305–312.
3

Snyder, C., Anderson, G. “Do Quality Improvement Organizations Improve the Quality of Hospital Care for
Medicare Beneficiaries?” JAMA, vol. 293, 2005, pp. 2900–2907.
4

Bradley, E.H. et al. “From Adversary to Partner: Have Quality Improvement Organizations Made the
Transition?” HSR, vol. 40, no. 2, pp. 459–476.

3

their performance would not have been different. This suggests that there is a group of hospitals
that does not require QIO assistance and a group that is helped by the QIO.
In the Seventh SOW, CMS widened the scope of QIO activities to include three additional
settings beyond hospitals: physician offices, nursing homes, and home health agencies. In
addition to promoting improvement at the statewide level, CMS required QIOs to offer more
intensive assistance to a subset of providers; in CMS parlance, this subgroup was called an
identified participant group (IPG). The Seventh SOW evaluation summarized the results in each
of the four settings. Rollow et al. demonstrated that there was improvement in most, but not all,
measures for each setting. 5 This finding is consistent with the view that the program has a broad
impact on quality of care. Additionally, Rollow et al. showed that providers inside the IPG
subgroup improved more than providers outside of the intensive assistance subgroup, the nonIPGs. This finding is consistent with the view that the direct provision of assistance to providers
by QIOs has an impact on quality. It also is consistent with selection bias: that is, those with
greater motivation or capacity to improve might have preferentially cooperated with the QIO and
joined their IPG. The design of the Seventh SOW was such that CMS could not fully address this
alternative explanation.
Reflecting the high priority that it has placed on assessing and improving the QIO Program,
Congress mandated in the Medicare Prescription Drug, Improvement and Modernization Act of
2003 (PL-108-73) that the Institute of Medicine (IOM) conduct a review of the Program. The
IOM’s extensive review of the QIO Program’s effect on quality in the Seventh and early Eighth
SOW concluded:

5

Rollow, W., Lied, T.R., McGann, P., Poyer, J., LaVoie, L., Kambic, R.T., Bratzler, D.W., Ma, A., Huff, E.D.,
and Ramuno, L.D. “Assessment of the Medicare Quality Improvement Organization Program.” Annals of Internal
Medicine, vol. 145, no. 5, Sept 5, 2006, pp. 342.

4

• “The quality of the health care received by Medicare beneficiaries has improved over
time.
• The existing evidence is inadequate to determine the extent to which the QIO
Program has contributed directly to those improvements.
• The QIO Program provides a potentially valuable nationwide infrastructure dedicated
to promoting quality health care.
• The value of the program could be enhanced through the use of strategies designed to
focus the QIOs’ attention on the provision of the technical assistance in support of
quality improvement, to broaden their governance base and structure, and to improve
CMS’s management of data system and program evaluations.” 6
Furthermore, the IOM explicitly recommended that HHS periodically commission
“independent, external evaluations of the QIO program’s overall contributions,” including
systems and program management as well as impact. In his Report to Congress responding to the
IOM’s study, the Secretary of HHS acknowledged the importance of program evaluation and
listed the Department’s plans for an independent, external evaluation of the Program.
The Eighth SOW contract focused on quality improvement through organizational
“transformations” intended to produce more rapid, measurable improvements in care. QIOs work
with health care providers such as physicians, hospitals, nursing homes, and home health
agencies; consumers; and other partners and stakeholders (including national medical
professional, nonprofit, and public health organizations; state-based affiliates of the national
organizations; and state and local departments of health). The Eighth SOW required the QIOs
both to work with IPGs that had defined quality improvement objectives (hospitals, nursing
homes, home health agencies, and physicians) and to perform statewide quality improvement
activities. Mathematica submitted a draft evaluation report on the Eighth SOW to CMS on
March 18, 2009 (under review). Legal constraints prevented Mathematica from accessing data
6

Institute of Medicine. “Medicare’s Quality Improvement Organization Program: Maximizing Potential.”
Washington, DC: IOM, March 2006.

5

that would support the strongest possible methodologies, so the results were not definitive
regarding QIO impact.
The Ninth SOW, which began in August 2008, aims to improve the quality of care and
protect Medicare beneficiaries through several national and sub-national themes:
• Beneficiary Protection
• Patient Safety
• Prevention
• Prevention–Disparities (sub-national)
• Care Transitions (sub-national)
• Prevention–Chronic Kidney Disease (sub-national)
In the Patient Safety, Prevention, and Prevention–Disparities themes, the QIO recruits and
works with participating providers to improve quality and safety on areas of care associated with
specific measures (similar to the IPGs of the Eighth SOW). In the Care Transitions and
Prevention–Chronic Kidney Disease themes, a major emphasis is for the QIO to work in
partnership with other interested organizations (including but not limited to provider
organizations) in one or more communities to achieve improvement. There is much less
statewide activity required in the Ninth SOW relative to the Eighth SOW; this gives the
evaluation better potential to isolate effects of the QIOs on the providers and communities they
work with.
2.

Evaluation Design for the Ninth Scope of Work QIO Program
The purpose of the evaluation is to design and conduct an analysis evaluating the impact of

the Ninth SOW of the QIO Program on regional and national health outcomes and processes. In
keeping with the prior evaluations and consistent with recommendations of the IOM, the current

6

evaluation will address not only program impact but also the mechanisms whereby this occurs.
The Eighth SOW Program evaluation was used to pilot test and build a foundation for the Ninth
SOW evaluation.
The following issues will be addressed for Ninth SOW Program evaluation:
a.

Impact
1. What is the relative impact of the QIO on the quality of care for Medicare
beneficiaries in its “jurisdiction”?
2. What is the program impact on the quality of care for Medicare beneficiaries
nationwide?
3. What evidence is there that the technical assistance methods and quality
improvement activities of the QIOs are having an impact on the quality of care for
Medicare beneficiaries?
4. Has the QIO Program improved health care for the underserved; has it adequately
addressed the health care disparities issue?
5. What are the costs and benefits of the QIO Program?
6. What is the overall cost-benefit ratio of the QIO Program?
7. What factors mediate the cost-benefit ratios across states, across regions, and
nationally
8. What is the utility (Quality-Adjusted Life Years [QALYs]) of the various
improvements?

b. Mechanisms
1. What is the best method for demonstrating QIOs have a direct effect on quality
improvement?
2. What mechanisms generate program impacts?
3. Are some methods and activities more useful than others; do they create greater
effect than others? What factors mediate the utility of these activities?
4. What are the systems and relationships that are in place to support and enhance the
goals of the QIO Program?
5. How does the state-specific provider environment enhance or deter QIO activities
and impact?

7

Within each state 7 served by a QIO, there may be multiple health care quality improvement
efforts. CMS sponsors some of these efforts such as quality reporting (for physicians offices, the
Physician Quality Reporting Initiative [PQRI] and for other providers, the Medicare Compare
websites) as well as public interest groups such as the Institute for Healthcare Improvement
(IHI), the Joint Commission on Accreditation of Healthcare Organizations (JCAHO), the Agency
for Healthcare Research and Quality (AHRQ), the Leapfrog Group for Patient Safety, and other
such public and private groups. The current evaluation will seek to determine the extent to which
the QIO Program is responsible for improvement in the quality indicators specified in the QIO
contract and the contribution to this improvement that can be attributed to the QIO Program. The
evaluation will account for the other stakeholder efforts noted above in making this
determination. We plan to separate the impacts of QIOs from that of the many other quality
improvement programs and policies by comparing the quality of care delivered by recipients of
QIO services (the treatment group) to statistically identical providers who do not (the
comparison group). Since both groups will have benefited from non-QIO programs and policies,
comparing the two groups should isolate the impact of the QIO program. The details of how we
plan to identify the comparison groups may be found in the evaluation design report (Chen et al.
2010).
QIOs use numerous tools, methods, and interventions to assist providers to improve care
quality and patient safety. Which methods are used varies but often includes one-on-one
consultations and site visits, group educational meetings including teleconferences and web-ex’s,
provision of information about measurement and improvement tools, and performance data
feedback with corresponding discussion. The evaluation will seek to determine which of these
7

We will use “state” for simplicity but also mean to include territories and the District of Columbia.

8

quality improvement methods, tools, and interventions are the most effective in producing
positive change in the QIO contract metrics, how this change is produced, and what is it that
makes interventions either work or not work. The effectiveness of quality improvement methods,
tools, and interventions may vary by provider type, beneficiary demographics, geography, QIO
staff, and other factors. The evaluation will identify the most effective and least effective quality
interventions, quality improvement methods, and the issues that contribute to their success or
failure. A report of these findings and recommendations for enhancements will be prepared and
submitted to CMS.
The evaluation will also seek to address environmental questions such as what role does
provider leadership play in mediating QIO success? Can quality improvement occur without the
presence of a project “champion”? Does state legislation mediate QIO effectiveness? How does
the regulatory environment in-state interact with QIO quality improvement activities?
A critical part of the assessment of the QIO Program is the systematic evaluation of the QIO
Program infrastructure. There is a network of organizations that provides numerous supportive
products and services to the program. Without this network of support functions, the program
cannot be expected to function effectively. The supportive infrastructure products and services
include measurement development, end-to-end data systems, information technology services,
contract administration, and developmental studies. These supportive products and services are
internal to CMS as well as contractor-based activities.
Part of the evaluation will be to map this interdependent system of support organizations and
their activities in order to determine their impact and contribution to the overall success of the
QIO Program. Specifically, the evaluation will determine if the supportive interventions,
products, and services may be linked to outcomes and to the success of the QIO contractors and
the QIO Program in general. The evaluation will assess the extent to which infrastructure

9

processes adequately support the QIO Program and will make recommendations for
enhancement when less than optimal results are found.
Finally, the evaluation will seek to assess the overall effect of the program on a set of
critical measures specified in the QIO SOW on a national basis. It will seek to identify how
much quality improvement in health care nationally can be attributed to the QIO Program and
the program’s impact on ongoing efforts to eliminate disparities (including ethnic, racial, socioeconomic, and geographic) in care. Table A.1 lists the outcome measures to be analyzed in the
evaluation.
TABLE A.1.

LIST OF OUTCOME MEASURES FOR THE EVALUATION OF THE QIO 9TH SOW
Measure

Data Source
PATIENT SAFETY THEME

Nursing Home Subtheme
Percentage of high-risk long-stay residents who have pressure sores
Percentage of long stay residents who were physically restrained
Frequency of contact with QIO
Presence of internal quality improvement efforts in specific measures
Whether has ever analyzed performance data in specific measures to identify
underlying causes (“root cause analysis”)
Whether has undertaken various quality improvement strategies for specific
measures
Whether reports adequate leadership and resources for quality improvement in
specific measures
Whether faces barriers to quality improvement
Hospital Subtheme
Surgery patients on a beta blocker prior to arrival who received a beta blocker
during the perioperative period
Prophylactic antibiotic received on time (INF-1)
Percent who received prophylactic antibiotics recommended for their specific
surgical procedure (INF-2)
Prophylactic antibiotics discontinued within 24 hours after surgery end time(INF-3)
Cardiac surgery patients with controlled 6 a.m. postoperative serum glucose(INF-4)
Surgery patients with appropriate hair removal (INF-6)
Surgery patients with recommended VTE prophylaxis ordered (VTE-1)
Surgery patients who received appropriate vte prophylaxis within 24 hours prior to
surgery to 24 hours after surgery (VTE-2)
Heart failure patients with left ventricular systolic dysfunction without ACEI and
ARB contraindications who are prescribed ACEI/ARB at discharge (HF3)
Risk-adjusted 30-day heart failure mortality rate PIHOEM
Postoperative sepsis (AHRQ PSI-13)
Postoperative wound dehiscence in abdominopelvic surgical patients (AHRQ PSI-

10

MDSa
MDS
Nursing Home Surveyb
Nursing Home Survey
Nursing Home Survey
Nursing Home Survey
Nursing Home Survey
Nursing Home Survey
Hospital Comparec
Hospital Compare

Hospital Compare
Hospital Compare
Hospital Compare
Hospital Compare
Hospital Compare
Hospital Compare
PIHOEMd
PIHOEM
PIHOEM

Measure
14)
Frequency of contact with QIO
Receipt of educational materials or tools from QIO
If received materials, perceived value
Presence of internal quality improvement efforts for specific measures
Presence of internal quality improvement efforts in specific measuresd
Whether has ever analyzed performance data in specific measures to identify
underlying causes (“root cause analysis”)
Whether has undertaken various quality improvement strategies for specific
measuresd
Whether reports adequate leadership and resources for quality improvement in
specific measures
Whether faces barriers to quality improvement Hospital Survey

Data Source
Hospital Surveye
Hospital Survey
Hospital Survey
Hospital Survey
Hospital Survey
Hospital Survey
Hospital Survey
Hospital Survey
Hospital Survey

Nursing Homes in Need (NHIN) Subtheme
Percentage of high-risk long-stay residents who have pressure sores
Percentage of long stay residents who were physically restrained
Deficiencies in resident behavior and facility practicesg
Deficiencies in quality of care

MDS
MDS
OSCARf
OSCAR

CORE PREVENTION THEME
Colorectal cancer screening in specified time frame (varies by screening test)
Breast cancer screening within past two years
Influenza vaccination during September through March
Pneumococcal vaccination since January 1, 199

PPRg
PPR
PPR
PPR

PREVENTION-CHRONIC KIDNEY DISEASE (CKD) THEME
Urinary testing for urine microalbumin
CKD Analytic Filesh
Initial hemodialysis through AV fistula or with maturing AV fistula present
CKD Analytic Files
PREVENTION-DISPARITIES THEME
Hemoglobin A1c testing within past 12 months
Dilated eye exam within past 12 months
Lipid testing within past 12 months

Diabetes Analytic Filesh
Diabetes Analytic Files
Diabetes Analytic Files

CARE TRANSITIONS THEME
All cause 30 day readmission after hospitalization for acute myocardial infarction
All cause 30 day readmission after hospitalization for congestive heart failure
All cause 30 day readmission after hospitalization for pneumonia

PIHOEM
PIHOEM
PIHOEM

a

MDS: the nursing home Minimum Data Set is a national CMS dataset of information submitted by nursing homes
on their residents [http://www.cms.gov/NursingHomeQualityInits/25_NHQIMDS30.asp]. MDS data underlie CMS’
Nursing Home Compare public reporting system [http://www.medicare.gov/nhcompare/]

b

Nursing Home Survey: the proposed survey of nursing homes by Mathematica, part of the data collection currently
under review by OMB for PRA clearance.

c

Hospital Compare is CMS’ public reporting system on hospital quality of care [www.hospitalcompare.hhs.gov]

d

PIHOEM: Production and Implementation of Hospital Outcome and Efficiency Measures. Mathematica is using
Medicare claims and enrollment data to produce the PIHOEM measures for CMS under a different contract than the
QIO Evaluation contract, in part for public reporting as Hospital Compare outcome measures. The PIHOEM
measures include hospital mortality and readmission rates.

11

e

Hospital Survey: the proposed survey of hospitals by Mathematica, part of the data collection currently under
review by OMB for PRA clearance.

f

OSCAR: Online Survey, Certification and Reporting data are maintained by CMS in cooperation with state longterm care surveying agencies. OSCAR contains data collected by surveyors during inspection surveys conducted at
nursing facilities for the purpose of certification for participation in the Medicare and Medicaid programs.
g
PPR: Program Progress Reports, a set of summary reports on QIO program measures produced by CMS’ QIO data
contractors. These reports are available in the QIO secure network (which is called SDPS/QIONet where SDPS
stands for Standard Data Processing System).
h

CKD Analytic Files and Prevention Disparities Analytic Files are datasets produced from Medicaere claims data by
CMS’ data contractors for the QIO program; they are stored in SDPS/QIONet.

To support these evaluation goals, CMS is requesting OMB clearance for the QIO survey;
the hospital and nursing home surveys; and in-person and telephone discussions with QIO staff,
partner organizations, health care providers, and community health leaders. The QIO survey will
be conducted in summer/fall 2010 and will obtain information about what activities QIOs
undertook to achieve their goals; obtain QIO reports about how clear, feasible, and well-targeted
QIO contractual requirements are; learn about QIO experiences with the contractors, systems,
and tools designed to support their work; identify other important contributors to quality
improvement in the state; obtain QIO views on effective methods of quality improvement (QI)
and remaining barriers to further improvement in the state; and understand the selection process
QIOs used to identify providers with whom they would work (see Table A.2 for a list of key
survey topics).
TABLE A.2
QIO SURVEY KEY TOPICS
Major QIO Activities
QIOs’ experience with contractual requirements
QIOs’ experiences with contractors, systems, and tools designed to support their work
Other important contributors to quality improvement in the state
QIOs’ views on effective methods of QI and remaining barriers to further improvement in the state
The selection process QIOs used to identify providers with whom they work

12

The survey of hospitals and nursing homes will be conducted in late 2010 and early 2011
and will cover descriptive information on level and types of interactions with the QIO, the
perceived value of QIO interactions and materials, hospital and nursing home quality
improvement actions that relate to the measures that the QIO works with them to improve, other
(non-QIO) external quality improvement initiatives the hospitals and nursing homes participate
in, key information sources used by hospitals and nursing homes that say they improved, and
barriers to further improvement (see Table A.3 for a list of key survey topics).
TABLE A.3
HOSPITAL AND NURSING HOME SURVEYS KEY TOPICS
Level and types of interactions with the QIO
Perceived value of QIO interactions and materials
Hospital and nursing home quality improvement actions that relate to the measures that the QIO is
required to work with them to improve
Other (non-QIO) external quality improvement initiatives the hospitals and nursing homes participate in
Key information sources used by hospitals and nursing homes that say they improved
Barriers to further improvement

Discussions with partner organizations will provide information on changes in care observed
by the partnered organizations, the role of the QIO in the partnership and in catalyzing changes
in care, what improvement strategies worked to improve care, what other lessons were learned,
and the partners’ perceptions of the sustainability of gains made and issues in sustainability.
Discussions with partner organizations will take place in November 2010 through February2011.
Case studies of QIOs (and their partners, providers, and other stakeholders in their state) will
provide information on the quality improvement approaches used by the QIO and others that led
to improvements in care; case studies will take place during late fall 2010 through spring 2011.

13

The discussions with the QIO are designed to complement the survey responses (see Table A.4
for a list of key discussion topics by respondent type).
TABLE A.4
CASE STUDY DISCUSSION TOPICS BY TYPE OF RESPONDENT

QIO

Community
Health Leaders

Providers that
Worked with the
QIO

Perceived impact of QIO activities

X

X

X

Understand any difficulties or problems faced by the
QIO with the program contract, infrastructure, and
supports

X

Understand the value of the QIO Support Centers in
facilitating QIO work

X

Identify QIO activities that had greater and lesser
value in leading to operational improvements in care

X

X

X

Actions taken by providers to improve care on the
program-targeted measures, and what prompted them

X

Topic

X

Case study provider’s quality improvement “story”:
reviewing with them their measure trends, timing of
various improvement actions, and what prompted the
actions

X

Providers’ focus on program-targeted measures
versus other measures

X

Lessons learned

X

State quality environment

X

Recruitment of providers and/or beneficiaries

X

Remaining barriers to further improvement

X

X
X

X
X

X

X

The data collected through surveys and in-depth discussions will be used to answer the
following questions:
• Has the quality of care funded by Medicare improved? How much of the observed
quality improvement is attributable to QIOs? What are the costs incurred by CMS in
administering and funding the program relative to the program’s benefits in terms of
improved quality of care?

14

• To what extent can improvements in the quality of care be attributed to the QIO
Program as opposed to other quality initiatives to which providers may be exposed
(such as the IHI, the Joint Commission, the AHRQ, and others)?
• Which technical assistance methods work best, and what evidence supports the use of
certain methods over others?
• Does the current QIO infrastructure adequately support program activities and
organizational learning?

15

B. JUSTIFICATION
1.

Need and Legal Basis
The statutory authority for the QIO Program is found in Part B of Title XI of the Social

Security Act, as amended by the Peer Review Improvement Act of 1982. The Social Security
Act established the Utilization and Quality Control Peer Review Organization Program, now
known as the QIO Program. The statutory mission of the QIO Program, as set forth in Title
XVIII—Health Insurance for the Aged and Disabled, Section 1862(g) of the Social Security
Act—is to improve the effectiveness, efficiency, economy, and quality of services delivered to
Medicare beneficiaries. Based on statutory language and the experience of CMS in administering
the program, CMS identified the following requirements for the QIO Program:
• Improve quality of care for beneficiaries by ensuring that beneficiary care meets
professionally recognized standards of health care
• Protect the integrity of the Medicare Trust Fund by ensuring that Medicare pays only
for services and items that are reasonable and medically necessary and that are
provided in the most economical setting
• Protect beneficiaries by expeditiously addressing individual cases such as beneficiary
quality of care complaints, contested hospital issued notices of non-coverage
(HINNs), alleged Emergency Medical Treatment and Labor Act (EMTALA)
violations, and other beneficiary concerns as required by the statute
The quality strategies of the Medicare QIO Program are carried out by specific QIO
contractors working with health care providers in their state, territory, or the District of
Columbia. Based on statutory language and the experience of CMS in administering the
program, CMS has identified the improvement of quality of care for beneficiaries as one of three
goals of the program.
This QIO contract contains a number of quality improvement initiatives that are authorized
by various provisions in the Act. As a general matter, Section 1862(g) of the Act mandates that

17

the secretary enter into contracts with QIOs for the purpose of determining that Medicare
services are reasonable and medically necessary and for the purposes of promoting the effective,
efficient, and economical delivery of health care services and of promoting the quality of the
type of services for which payment may be made under Medicare. CMS interprets the term
“promoting the quality of services” to involve more than QIOs reviewing care on a case-by-case
basis, but to include a broad range of proactive initiatives that will promote higher quality. CMS
has, for example, included in the SOW tasks in which the QIO will provide technical assistance
to Medicare-participating providers and practitioners in order to help them improve the quality of
the care they furnish to Medicare beneficiaries.
Additional authority for these activities appears in Section 1154(a)(8) of the Act, which
requires that QIOs perform such duties and functions, assume such responsibilities, and comply
with such other requirements as may be required by the Medicare statute. CMS regards survey
activities as appropriate if they will directly benefit Medicare beneficiaries. In addition, Section
1154(a)(10) of the Act specifically requires that the QIOs “coordinate activities, including
information exchanges, which are consistent with economical and efficient operation of
programs among appropriate public and private agencies or organizations, including other public
or private review organizations as may be appropriate.” CMS regards this as specific authority
for QIOs to coordinate and operate a broad range of collaboratives and community activities
among private and public entities, as long as the predicted outcome will directly benefit the
Medicare program.
In addition, Section 1156(c) of the Act states that it is the duty of each QIO to use such
authority or influence as it may possess as a professional organization and to enlist the support of
any other professional or governmental organization having influence or authority over health
care practitioners or entities furnishing services in its area, in assuring that each practitioner or

18

entity shall comply with all obligations imposed on them under Section 1156(a). Under these
obligations, providers and practitioners must assure that they will provide services of a quality
that meets professionally recognized standards of care. (Appendix A contains the relevant
legislation from the Social Security Act cited here.)
An overall evaluation is needed to determine the impact of the Ninth SOW of the QIO
Program on regional and national health outcomes and processes. In keeping with the prior
evaluations and consistent with recommendations of the IOM, an evaluation must also address
not only program impact but also the mechanisms whereby it occurs.
2.

Information Users
Information for the evaluation of the Ninth Scope of Work QIO Program will be collected

and analyzed by Mathematica, under Contract Number HHSM-500-2005-00025I (0010) with
CMS titled “Program Evaluation of the Eighth and Ninth Scope of Work Quality Improvement
Program.” Findings from the evaluation will be included in internal reports to CMS, and papers
will be prepared for publication.
3.

Use of Information Technology
The evaluation will use both a web-based survey instrument (QIO survey) and a computer-

assisted telephone interview (CATI survey of hospitals and nursing homes) as the primary
method of data collection with QIOs, hospitals, and nursing homes. Data collection for all three
surveys will take place in 2010. All participating QIOs will receive an advance letter instructing
them how to log on to the website to complete the QIO survey online. The letter will also
provide a toll-free number to call if the participant has a question. All participating hospitals and
nursing homes will receive an advance letter alerting them that a Mathematica representative will
be calling them soon to conduct a survey interview. The letter will also provide a toll-free

19

number if the participant has a question or would like to call and set up an interview
appointment.
In-person and telephone discussions with QIOs and their partner organizations will take
place in 2010 and 2011. All discussions will be conducted using semi-structured, paper-andpencil discussion guides. Information gathered from the discussions will be entered into
ATLAS.ti software (a commercial tool for analyzing qualitative data) to help identify themes and
illustrative examples.
4.

Duplication of Effort
These information collections do not duplicate any other effort, and the information cannot

be obtained from any other source.
5.

Small Businesses
Some of the hospitals, nursing homes and physician practices targeted for this study may be

small businesses. Participating in the evaluation will impose minimal burden because the
hospital and nursing home surveys are designed to be completed in 20 to 30 minutes, and the onsite interviews with physician practices will be scheduled for 60 minutes. The CATI format for
the hospital and nursing home surveys permits participants to complete the survey efficiently,
and appointments can be made at participants’ convenience.
Twelve case studies of QIOs and selected provider and other health care organizations they
work with will be conducted during November 2010 through May 2011. Some of these
organizations may be small businesses. Each case study will involve a week-long site visit by a
team of two evaluators (a senior/junior team). Each visit will include approximately 30
interviews, spread across 13 to 15 stakeholder organizations. Each interview is expected to last
an average of 48 minutes. We will minimize burden on respondents by scheduling interviews at

20

their convenience whenever possible or by using the telephone if the organizations are unable to
fit into an available time slot while we are on-site.
6.

Less-Frequent Collection
We are collecting only one round of data from most hospitals, nursing homes, QIOs and

QIO partner organizations. Twelve of the 53 QIOs and up to about 36 hospitals and 48 nursing
homes will also participate in case study discussions. However, the survey sample and the case
study providers are unlikely to overlap much, and the case studies are designed to complement
and follow up on the survey information.
7.

Special Circumstances
There are no special circumstances related to the proposed data collection.

8.

Federal Register/Outside Consultation
The notice required by 5 CFR 1320.8 (d), was published in the Federal Register on October

9, 2009. A copy of the notice is in Appendix B.
a.

Public Comment and Responses
No public comments were received.
Outside consultation for the design of the studies and surveys was received from a variety of

experts (see Table B.1 for a list of consultants).

21

TABLE B.1
CONSULTANTS
Individual

Affiliation/Agency/Division

Telephone Number

Within the Department of Health and Human Services
Robert Kambic

Centers for Medicare & Medicaid Services

(410) 786-1515

Outside the Department of Health and Human Services
Therese Moore

Abt Associates

(617) 492-7100

Claudia Schur

Social & Scientific Systems

(301) 628-3001

Kathryn Anne Paez

Social & Scientific Systems

(301) 628-3001

9.

Payments/Gifts to Respondents
No incentive payment will be offered to QIOs, hospitals, or nursing homes for participating

in the surveys. No incentive payment will be offered to QIOs and their partner organizations for
participating in the discussions.
10. Confidentiality
Mathematica will take several steps to assure respondents that the information they provide
will be treated as confidential to the extent permitted by law. Survey respondents will be told that
neither they nor their organization will be identified individually (that is, by name) in any reports
or in any communications to CMS. The assurances and limits of confidentiality will be made
clear in advance material mailed to respondents and will be restated at the beginning of each
telephone interview. For example, participants will be told the following, in verbal introductions
before a data collection or in written advance letters or forms: “Please be assured that responses
to the survey will remain confidential to the extent permitted by law. All data collected for the
purposes of this study will be combined and reported in aggregate form. Neither you nor your
organization will be identified by name in any reports or documents produced from the study

22

findings. Only Mathematica staff that work directly on the evaluation will have access to the
name of your organization and your name.”
Mathematica will restrict access to the QIO web instrument to protect the confidentiality of
respondents. The QIO web instrument will be hosted on Mathematica’s web servers. Data will be
processed and stored on Mathematica’s password-protected local area network (LAN).
Mathematica protects its LAN with several security mechanisms available through the network
operating system. Access to confidential information stored on LAN directories is restricted to
authorized project staff by means of IDs and passwords. In addition, network servers containing
confidential information are kept in a locked area. Finally, Mathematica staff members assigned
to work on the project all sign confidentiality pledges as a term of employment. The
confidentiality pledge requires that staff members maintain the confidentiality of all information
collected.
11. Sensitive Questions
The QIO survey instrument includes questions about activities QIOs undertook to achieve
their goals; how clear, feasible, and well-targeted QIO contractual requirements are; QIO
experiences with contractors, systems, and tools designed to support their work; other important
contributors to quality improvement in the state; effective methods of QI; remaining barriers to
further improvement in the state; and the selection process QIOs used to identify providers with
whom they would work. These questions are not considered sensitive.
The hospital and nursing home instruments include questions about the level and types of
interactions with the QIO, the perceived value of QIO interactions and materials, hospital and
nursing home quality improvement actions that relate to the measures that the QIO is required to
work with them to improve, other (non-QIO) external quality improvement initiatives the
hospitals and nursing homes participate in, key information sources used by hospitals and
23

nursing homes that say they improved, and barriers to further improvement. These questions are
not considered sensitive.
Questions asked during discussions with QIOs cover the perceived impact of QIO activities;
difficulties or problems faced by the QIO with the program contract, infrastructure, and supports;
QIO activities that had greater and lesser value in leading to operational improvements in care;
actions taken by providers to improve care on the program-targeted measures and what prompted
them; lessons learned; and any remaining barriers to improvement. These questions are not
considered sensitive.
Questions asked during discussions with QIO partner organizations cover changes in care at
the partnered organizations, the role of the QIO in the partnership and in catalyzing changes in
care, what improvement strategies worked to improve care, what other lessons were learned, and
the partners’ perceptions of the sustainability of gains made and issues in sustainability. These
questions are not considered sensitive.
12. Burden Estimates (Hours and Wages)
Table B.2 presents estimates of respondent burden for completing the QIO survey. It shows
the expected number of respondents, the hours per response, and the annualized hour and cost
burden. The QIO survey will be administered in summer-fall 2010. Hourly estimates for the QIO
survey are based on pretest interviews completed with less than nine QIO directors and theme
leaders. Interview completion times for QIO directors ranged from 20 to 40 minutes, with an
average length of 30 minutes. Interview completion times for theme leaders ranged from 25 to 50
minutes, with an average length of 45 minutes. The cost per survey was computed using an
estimated annual salary of $152,000 for QIO directors and $100,000 for theme leaders and 2,080
annual work hours as follows: $152,000/2,080*0.5 = $36.53 per response for QIO directors, and
$100,000/2,080*0.75 = $36.05 per response for theme leaders.

24

TABLE B.2
QIO SURVEY RESPONSE BURDEN

Respondent
QIO
Director
Theme
Leader

Number of
Respondents

Frequency of
Response

Hours per
Response

One-Time
Hour
Burden

Cost per
Response

One-Time
Cost
Burden

53

1

0.5

26.5

$36.53

$1,936

342

1

0.75

256.5

$36.05

$12,329

Table B.3 presents estimates of respondent burden for completing the survey of hospitals. It
shows the expected number of respondents, the hours per response, and the annualized hour and
cost burden. The survey of hospitals will take place during summer and fall 2010. Hourly
estimates for the survey are based on pretest interviews with less than nine hospital QI directors.
Interview completion times for the hospital survey ranged from 25 to 45 minutes, with an
average length of 30 minutes. The cost per survey was computed using an estimated annual
salary of $104,000 for hospital QI directors with 2,080 annual work hours as follows:
$104,000/2,080*0.5 = $25.00 per response.

TABLE B.3
HOSPITAL SURVEY RESPONSE BURDEN

Respondent

Number of
Respondents

Frequency of
Response

Hours per
Response

One-Time
Hour
Burden

Cost per
Response

One-Time
Cost
Burden

Hospital QI
Director

1,250

1

0.5

625.0

$25.00

$31,250

Table B.4 presents estimates of respondent burden for completing the survey of nursing
homes. It shows the expected number of respondents, the hours per response, and the annualized
25

hour and cost burden. The survey of nursing homes will take place during summer and fall 2010.
Hourly estimates for the survey are based on pretest interviews with less than nine nursing home
administrators. Interview completion times for the nursing home survey ranged from 17 to 25
minutes, with an average length of 20 minutes. The cost per survey was computed using an
estimated annual salary of $78,000 for nursing home directors, with 2,080 annual work hours as
follows: $78,000/2,080*0.33 = $12.37 per response.
TABLE B.4
NURSING HOME SURVEY RESPONSE BURDEN

Respondent
Nursing home
administrator

Number of
Respondents

Frequency of
Response

Hours per
Response

One-Time
Hour
Burden

Cost per
Response

One-Time
Cost
Burden

1,250

1

0.33

412.5

$12.37

$15,462

Twelve case studies of QIOs, selected health care organizations they work with, and
community health leaders will be conducted during November 2010 through May 2011. Each
case study will involve a week-long site visit by a team of two evaluators (a senior/junior team).
Each visit will include approximately 30 interviews spread across roughly 13–15 stakeholder
organizations. Some interviews may be completed by phone if the organizations are unable to fit
into available time slots on site or a few key respondents are too geographically dispersed to
make an in-person visit feasible within the visit time frame.
Discussions will last roughly 48 minutes per individual and will use semi-structured guides.
Table B.5 presents estimates of respondent burden for completing the discussions with QIOs and
their partner organizations. It shows the expected number of respondents, hours per response,
and the one-time hour and cost burden for the discussions conducted. The cost per discussion is
computed using an estimated annual salary of $80,000 for QIO staff, key staff at selected health
26

care organizations, and community health leaders, with 2,080 annual work hours as follows:
$80,000/2,080*0.8= $30.77 per response.
TABLE B.5
CASE STUDY DISCUSSIONS RESPONSE BURDEN

Respondent
QIO staff, selected
health care
organizations they
work with, and
community health
leaders 1

Number of
Respondents

Frequency
of Response

Hours per
Response

OneTime
Hour
Burden

360

1

0.8

288.0

Cost per
Response

OneTime
Cost
Burden

$30.77

$11,077

QIO partners discussions for the Prevention – CKD and Care Transitions themes will
include up to 200 contacts with health care organizations partnered with the QIO to improve care
on these themes (up to 176 full discussions plus up to 24 additional screener contacts). The
screener contact will take only 5 minutes, and the full interviews will take 45 minutes. Table B.6
presents estimates of respondent burden for completing the QIO partners discussions. It shows
the expected number of respondents, hours per response, and the annualized hour and cost
burden for the discussions conducted. The cost per discussion was computed using an estimated
annual salary of $80,000 for the principal contacts at the partnered organizations, with 2,080
annual work hours as follows: $80,000/2,080*0.8= $30.77 per response.

1

One person at a QIO may lead multiple sub-themes or themes; however, their estimated time is additive
across the themes or sub-themes they lead because each will require separate discussion.

27

TABLE B.6
DISCUSSIONS WITH QIO PARTNER ORGANIZATIONS
RESPONSE BURDEN

Number of
Respondents

Frequency
of Response

Hours per
Response

OneTime
Hour
Burden

Key contacts at
health care
organizations
partnered with the
QIO – full
discussions

176

1

0.8

141

$30.77

$5,416

Key contacts at
partnered
organizations –
screened, no full
discussion

24

1

0.1

2.4

$3.85

$92.40

Respondent

Cost per
Response

OneTime
Cost
Burden

13. Capital Costs
There are no direct costs to respondents other than their time to participate in the study.
14. Cost to Federal Government
The total current value for the QIO Program evaluation contract is $3,500,000 over three
years. The estimated total cost to the government for conducting these data collections is
$911,561 ($65,307 for conducting the QIO survey; $399,123 for conducting the hospital and
nursing home surveys; $447,131 for conducting case study discussions and discussions with QIO
partner organizations). These estimates are based on the contractor’s costs for collecting and
tabulating survey and contact data, including labor and travel; other direct costs for computer,
telephone, postage, reproduction, fax, printing, and survey facilities; and indirect costs for fringe
benefits, general and administrative costs, and fees.

28

15. Changes to Burden
Data collection for QIO Program evaluation is new; therefore, there are no changes to
burden.
16. Publication/Tabulation Dates
The evaluation will produce several reports, including an interim and a final evaluation
report that synthesize findings across states and analytic components. Table B.8 summarizes the
delivery schedule. A summary of each report follows.
TABLE B.7
QIO PROGRAM EVALUATION DELIVERY SCHEDULE OF REPORTS
Report

Month Due

Quarterly and Ad Hoc Interim Reports Pertaining to this Data Collection:
Barriers to improvement in hospitals and nursing homes

November 2010

Site visit results on provider environments and influence on QIO activities

June 2011

Evaluation Design and Methods Report

September 2009

Data Collection and Analysis Report

November 2010

Mid-Course Review Report

December 2010

Final Evaluation Report

October 2011

a.

Quarterly and Ad Hoc Interim Reports
Quarterly and ad hoc interim reports will be due three months after the evaluation begins,

and then every three months thereafter. The reports pertain to specific topics agreed upon with
the Project Officer, with some drawing mostly on quantitative analysis of administrative data,
while others will highlight findings on a particular topic from the data collection efforts
discussed here (listed below).

29

b. Evaluation Design and Methods Report
The evaluation design and methods report, submitted in September 2009, described the
design and methods to be used to collect and analyze all data for the evaluation.
c.

Data Collection and Analysis Report
The data collection and analysis report, due in November 2010, will identify end points and

time frames for any and all qualitative and quantitative data to be used in the final report. The
report will include discussion and review of risk adjustments, confounders, and problems
surfaced to date. The report will also include the chapter headings and dummy tables proposed
for the final report.
d. Mid-Course Review Report
The Mid-Course Review Report, due in December 2010, will present the findings from the
survey of QIOs.
e.

Final Evaluation Report
The final evaluation report, due in October 2011, will synthesize results from all

components of the evaluation. The report will discuss the impact of the QIOs on quality
improvement and the relative impact of different practices employed by QIOs. The report will
employ a useful and tested framing of the potential roles of the QIOs, including such elements as
providing direct services (for example, intervention strategies, data collection); catalyzing
changes; convening partner organizations; engendering commitment; and securing resources.
17. Expiration Date
The OMB expiration date will be displayed on all materials sent to participants, including
the advance letters, and on the initial screen of the computerized QIO web survey instrument.

30

18. Certification Statement
Data collection efforts for the QIO survey, hospital and nursing home surveys, and the
discussions with QIOs and their partner organizations will conform to all provisions of the
Paperwork Reduction Act.

31