(State Gov't) External Quality Review of Medicaid MCOs and Supporting Regulations in 42 CFR 438.360, 438.362, and 438.364

2012 EXTERNAL QUALITY REVIEW (EQR) PROTOCOLS
APPENDIX I: METHODOLOGY AND ORIGIN OF EXTERNAL QUALITY REVIEW (EQR)
PROTOCOL DEVELOPMENT

TABLE OF CONTENTS
PURPOSE OF THE APPENDIX.......................................................................................................................................... 1
PERFORMANCE IMPROVEMENT PROTOCOLS DEVELOPMENT .............................................................. 2
PERFORMANCE MEASUREMENT PROTOCOLS DEVELOPMENT ............................................................. 6
INFORMATION SYSTEM CAPABILITIES ASSESSMENT DEVELOPMENT ................................................ 7
ADDITIONAL REFERENCES ................................................................................................................................. 9

PURPOSE OF THE APPENDIX
The purpose of this Appendix is to describe how the protocols were developed and to document the
references applied to them.
As described in the introduction to the protocols, the original protocols were developed in 2001 by the
Joint Commission on Accreditation of Healthcare Organizations (JCAHO), working with consultants and
representatives of private accrediting organizations, quality measurement experts, State Medicaid
agencies, and advocates for Medicaid beneficiaries, under the direction of the Centers for Medicare &
Medicaid Services (CMS).
In 2010, CMS contracted with Provider Resources, Inc. (PRI) and their subcontractor, the National
Committee for Quality Assurance (NCQA), to work with EQR stakeholders, including States, EQROs,
and MCOs to modernize the protocols by:
1. Aligning them with applicable federal legislation, including:
a. Health Insurance Portability and Accountability Act of 1996 (HIPAA);
b. Children’s Health Insurance Program Reauthorization Act of 2009 (CHIPRA);
c. American Reinvestment and Recovery Act of 2009 (ARRA) / Health Information
Technology for Economic and Clinical Health (HITECH); and
d. Affordable Care Act of 2010;
2. Updating them according to scientific and industry advances in performance improvement and
measurement; and
3. Improving their utility for a broader audience in a more efficient manner.

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid
OMB control number. The valid OMB control number for this information collection is 0938-0786. The time required to complete this information
collection is estimated to average 1,591 hours per response for all activities, including the time to review instructions, search existing data
resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the
time estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports Clearance Officer,
Baltimore, Maryland 21244-1850

EQR Protocol: Appendix I
Methodology and Origin of External Quality Review (EQR) Protocol Development
September 2012

1

The protocol revision process included four components as displayed in the diagram below:

Diagram 1: EQR Protocol 2010 Revision Process

Preliminary Review

•Extensive review of existing documents, CMS responses to public
comments , current regulations, and recent legislation

Recruit Advisors and Conduct Advisory •Based on findings from the preliminary review, conduct in-depth
interviews and online focus groups with EQR stakeholders
Panel Meetings
Draft Revision Recommendations

Final Protocol Revisions

•Develop audience-centric improvements to protocols that
incorporate preliminary review and advisory panel meetings findings

•Incorporate CMS comments about the draft revision
recommendations into a final revision of the protocols

PERFORMANCE IMPROVEMENT PROTOCOLS DEVELOPMENT
The Performance Improvement Project (PIP) Protocols include the following:
•
•
•

Protocol 3: Validating PIPs (mandatory)
Protocol 7: Conducting PIPs (optional)
Protocol 8: Special Studies (optional

The PIP protocols are three of nine EQR protocols developed during 1998-2001 from standards and
guidelines used in the public and private sectors. The 2010 revisions updated the application of the
references in accordance with industry advances, but the methodology remains unchanged.
PIP References
In addition to communications with stakeholders in the industry, the following resources were used in the
development of the original protocols and/or the 2010 revisions:
1. NCQA’s The Essential Guide to Health Care Quality, available at:
http://www.ncqa.org/tabid/460/Default.aspx
2. The Agency for Healthcare Research and Quality’s Child Health Care Quality Toolbox, available
at: http://www.ahrq.gov/chtoolbx/index.htm
3. NCQA Volume 5. HEDIS® Compliance Audit. Standards, Policies and Procedures. 2010
EQR Protocol: Appendix I
Methodology and Origin of External Quality Review (EQR) Protocol Development
September 2012

2

This publication details the standards and guidelines used to evaluate a plan’s ability to
comply with HEDIS® specifications, assess information system capabilities, data collection
programs, analysis and measure results
4. Quality Improvement System for Managed Care (QISMC)
QISMC was a CMS initiative that set forth standards and guidelines of healthcare quality for
Medicaid and Medicare health plans (MCOs, PIHPs, and Medicare+Choice (now Medicare
Advantage) plans) to address MCO and PIHP quality assessment and PIPs.
5. Health Care Quality Improvement Studies in Managed Care Settings: A Guide for State
Medicaid Agencies (NCQA)
Produced under a contract from CMS, this guidebook identifies key concepts related to the
conduct of QI studies and accepted principles of research design and statistical analysis as
they apply to QI studies.
6. A Health Care Quality Improvement System for the Medicaid Managed Care, A Guide for States
(Health Care Financing Administration (HCFA))
This guide for health care QI provides a framework for building QI systems within State
Medicaid managed care initiatives. It offers guidelines for addressing quality assessment and
improvement studies of MCOs and PIHPs. The 1993 guide was the result of the Quality
Assurance Reform Initiative (QARI).
7. Framework for Improving Performance, From Principles to Practice (Joint Commission on
Accreditation of Healthcare Organizations (JCAHO)
This publication describes the Joint Commission’s theory-based, practical methodology for
continuously improving the care practices of healthcare organization and defines the key
characteristics and essential behaviors of healthcare organization striving to achieve high
quality patient care.
8. 1990-2000 Standards for Health Care Networks (SHCN) (JCAHO)
The JCAHO 1990-2000 SHCN provides an evaluation process based on standards to assist
the MCO in measuring, assessing, and improving its network’s performance. It also helps the
MCO focus on conducting performance improvement efforts in a multi disciplinary, systemwide manner. The 1990-2000 SHCN integrates information about the Joint Commission’s
healthcare network accreditation process.
9.

Specifications Manual for Joint Commission National Quality Core Measures (version 2010B2)
This publication provides information describing the specifications of the Joint Commission’s
specifications and rationale for the organization’s core quality measures.

10. NCQA 1997, 1998, and 1999 Standards for Accreditation of Managed Care Organizations and
NCQA 1999 Standards for Accreditation of Managed Behavioral Healthcare Organizations
(MBHO)
These documents include administrative policies and procedures for NCQA’s MCO and
MBHO accreditation programs using 1997, 1998, and 1999 standards, as well as the
rationale for their use.
EQR Protocol: Appendix I
Methodology and Origin of External Quality Review (EQR) Protocol Development
September 2012

3

11. Quality Improvement Organizations (QIO) 9th Statement of Work (SOW) (CMS)
The ninth SOW documents outlined the requirements for PROs to adhere to while
conducting health care quality and improvement activities for Medicare beneficiaries.
12. Peer Review Organizations (PRO) 4th and 5th Scope of Work (SOW) (CMS)
The fourth and fifth SOW documents outlined the requirements for PROs to adhere to
while conducting health care quality and improvement activities for Medicare beneficiaries.
PIP Reference Analysis
An in-depth comparison of these documents identified common activities and features as well as features
unique to individual protocols. The QISMC, JCAHO, and NCQA standards are written as guides for
MCOs to follow in developing, conducting, and evaluating their PIPs. States or their agents (e.g.,
EQROs) may also use these guides to assess compliance with State-mandated guidelines and/or to
facilitate overall plan-to-plan comparisons. QARI targeted States and EQROs to help them assure
compliance with regulations and Medicaid program requirements, and promote consistency in the
manner in which MCOs carry out activities related to PIPs.
PIP Components
The analysis revealed that all the documents identify common characteristics of effective PIPs. These
include:
•

Selection of Topics: All of the reference documents address the need for PIPs to clearly specify
the topic to be addressed. They all acknowledge both clinical and non-clinical health service
delivery issues as appropriate topics.

•

Means of Identifying Topics: Continuous data collection and analysis is stressed throughout all
documents as a means of identifying appropriate topics. Topics should be systematically selected
and prioritized to achieve the greatest practical benefit for enrollees. A minimal set of criteria is
suggested for selecting appropriate topics. These may include: the prevalence of a condition
among, or need for a service by, the MCO’s enrollees; enrollee demographic characteristics and
health risks; the likelihood that the study topic will result in improved health status among the
enrollees; and the interest of consumers in the aspect of care or services to be addressed.

•

Scope of project topics: The QISMC standards specify that PIPs should address the breadth of
the MCO’s or PIHP’s services, such as whether they include physical health and
mental/substance abuse health services. They also identify specific clinical and non-clinical focus
areas that are applicable to all enrollees. The QISMC standards also specify that the scope of the
health plans’ improvement efforts are to include all enrollees.

•

Stating the Study Question(s): The Healthcare Quality Information Systems (HCQIS) Guide
discusses the importance of “stating the study question” after a study topic is identified. It asserts
that stating a study question helps a project team avoid becoming sidetracked by data that is not
central to the issue under study. For example, once a PIP has identified childhood immunizations
as a study topic, it might specify a number of different study questions:
a. Have all children received all scheduled doses of one vaccine in particular?
b. Have all children of all ages received all recommended vaccines appropriate for their age?

EQR Protocol: Appendix I
Methodology and Origin of External Quality Review (EQR) Protocol Development
September 2012

4

c. Have all children of a particular age (e.g., at the age of one, two, six or other years)
received all age-appropriate immunizations?
Alternatively, the purpose of the PIP might require more detailed information and specify the
study questions as:
d. What proportion of Medicaid enrollees who have reached two years of age have received:
•
•
•
•

All four recommended doses of DPT vaccine?
All three recommended doses of the Polio vaccine?
One recommended dose of the MMR vaccine?
At least one dose of Hib vaccine in the second year of life?

A PIP might further specify additional study questions to provide information to assist them in
their QI efforts, such as:
e. In what percent of cases were children not immunized for one of the following reasons?
•
•
•

Refusal by a parent or guardian?
Medical contraindications?
Member non-complaint with the recommended immunization regimen?

Incorporating the process of documenting a study question(s) into the project design can help
ensure that the PIP used a systematic method of identifying appropriate indicators and data to be
collected. In this protocol, “defining the study question(s)” is a key step in designing and
implementing a PIP.
•

Use of Quality Indicators: All reference documents address the need to specify well-defined
indicators for monitoring and evaluation throughout the project. Quality indicators do not need to
be outcome measures. Process measures are also appropriate, especially when there is strong
clinical evidence that the process being measured has a meaningful association with outcomes.
There are various ways to obtain appropriate indicators, such as using those dictated from
outside sources (such as the State or CMS) or by an MCO developing them internally on the
basis of clinical literature or findings of expert panels.

PIP Characteristics
In addition to these features found uniformly in all reference documents, other significant aspects of PIPs
were identified by one or more of the reference documents. These include:
•

“Significant” improvement: NCQA’s document, “Health Care Quality Improvement Studies in
Managed Care Settings,” states that, “When presenting statistical results of any study, it is
important to fully disclose ... the statistical significance of the estimates produced, as well as the
statistical significance of any apparent differences between units of comparison.” Building on this,
CMS’s QISMC document calls for specific amounts of measurable improvement to be
demonstrated by the health plan. QISMC defines “demonstrable” improvement as either 1)
benchmarks established by CMS (for national Medicare projects) or State agencies (for Statewide
Medicaid QI projects) or by the health plans for individual (organizational) projects or 2) a 10
percent reduction in adverse outcomes. This protocol does not call for a specific level of statistical
achievement but, consistent with the NCQA document, calls for disclosure and review of the

EQR Protocol: Appendix I
Methodology and Origin of External Quality Review (EQR) Protocol Development
September 2012

5

statistical significance of any reported improvements in performance as one aspect of reviewing
the overall success of a PIP.
•

Phase-in or time frame requirements: QISMC delineates specific time frame requirements for
MCOs to reach certain phases in a QI cycle. For example:
a. By the end of the first year, an MCO should have initiated at least two PIPs addressing
two different focus areas;
b. By the end of the second review year, at least two additional projects addressing two
different focus areas should be initiated.
c. By the end of the first year, after the 2- year phase-in period, and each subsequent year,
at least two projects are to achieve demonstrable improvement in two of the focus areas.
The PIP protocols do not specify phase-in or time frame requirements, allowing States to specify
any such requirements as they determine appropriate.

•

Evaluation Tools: NCQA’s HCQIS guidebook includes study planning and summary worksheets
to be used in the evaluation of an MCO’s PIP. This feature provides helpful in recording data
during the evaluation process and promotes the collection of consistent information by all
evaluators.

•

Scoring system: NCQA accreditation provides a numerical scoring system to measure
performance against standards and to promote consistency in the process used to evaluate
MCOs. Although the scores do not dictate the final decision with respect to compliance with
standards, they do serve as a guide for NCQA evaluators to recommend non-compliance. This
scoring system includes an opportunity for the MCO to comment on the reviewer’s scores before
a final decision is rendered. It also promotes continuous improvement practices by securing
“customer” input into a final product (i.e., evaluation decisions).

PERFORMANCE MEASUREMENT PROTOCOLS DEVELOPMENT
The Performance Measurement (PM) Protocols include the following:
•
•
•

Protocol 2: Validating Performance Measures (mandatory)
Protocol 4: Validating Encounter Data (optional)
Protocol 6: Calculating Performance Measures (optional)

The PM protocols are three of nine EQR protocols developed during 1998-2001 from standards and
guidelines used in the public and private sectors. They were originally derived from the following
references:
1. HEDIS® Compliance Audit™ Standards and Guidelines, 1999 National Committee for Quality
Assurance (NCQA);
2. Tools used by the Island Peer Review Organization (IPRO) in their audits of HEDIS® measures
for Medicare; and
3. Documents from the MEDSTAT Group, Inc., published in conjunction with work performed in
1997 and 1998 for CMS.

EQR Protocol: Appendix I
Methodology and Origin of External Quality Review (EQR) Protocol Development
September 2012

6

Since the time the PM protocols were originally developed, IPRO has abandoned the tools that were
presented in the initial EQR protocols, and The MEDSTAT Group, Inc., no longer conducts performance
measurement validation work. As of 2011, IPRO and much of the industry use NCQA’s HEDIS®
Compliance Audit – 40 States recognize NCQA MCO accreditation in whole or part in either their
commercial market and/or their Medicaid managed care program. 1 CMS now mandates the use of the
HEDIS® Compliance Audit for the validation of performance measures submitted by Medicare Advantage
Plans. While the HEDIS® Compliance Audit is most often used to validate HEDIS® measures, it is
frequently used to validate other performance measures, including those State-specific measures used in
some Medicaid programs.
The 2010 revisions updated the application of the original references in accordance with industry
advances, which are largely derived from HEDIS®2 2011 Volumes 2 (Technical Specifications) and 5
(HEDIS® Compliance Audit: Standards, Policies and Procedures). Additional sources included:
•
•
•

APCD Council, available at: http://www.apcdcouncil.org;
NCQA’s Medicaid Deeming Toolkit, available at: www.ncqa.org; and
CMS’s Measures Management System Blueprint, available at:
http://www.cms.gov/MMS/19_MeasuresManagementSystemBlueprint.asp

INFORMATION SYSTEM CAPABILITIES ASSESSMENT
DEVELOPMENT
Most of the EQR protocols include an assessment of the structure and integrity of the MCO’s underlying
information system, and of the MCO’s ability to collect valid data from a variety of sources internal and
external to the organization. Furthermore, the regulation at 42 C.F.R. § 438.242 requires the State to
ensure that each MCO maintains a health information system that collects, analyzes, integrates, and
reports data for areas including, but not limited to, utilization, grievances and appeals, and disenrollments
for other than loss of Medicaid eligibility.
The initial EQR protocol package included an Appendix Z, now referred to as Appendix V, for information
system assessment. The 2010 revised information system assessment initiates stepwise modernization
by updating the original references, applying more recent legislation (e.g., HITECH), and aligning it with
current technology standards. Potential expansive revisions of the information system assessment may
be anticipated in the future due to a host of emerging federal information technology initiatives and
industry evolutions of standardized health information systems, their users, and purposes.
A number of public and private sector protocols and tools were examined during the development of both
the original protocols and the 2010 revisions to promote consistency between this assessment and
similar public and private sector activities. These include:
1. The National Committee for Quality Assurance’s (NCQA) Volume 5, HEDIS® Compliance
Audit Standards and Guidelines 2010;
2. Tools used by the Island Peer Review Organization (IPRO) in their audits of HEDIS®
measures;
3. Documents from the MEDSTAT Group, Inc., published in conjunction with work performed
for CMS in 1997 and 1998;
4. Stakeholder interviews with entities that perform external audits; and
1 NCQA website, http://www.ncqa.org/tabid/135/Default.aspx
®
2 HEDIS is a registered trademark of the National Committee for Quality Assurance (NCQA).
EQR Protocol: Appendix I
Methodology and Origin of External Quality Review (EQR) Protocol Development
September 2012

7

5. Review of NCQA HEDIS® Compliance Audit™ Volume 5 for auditing standards, policies,
and procedures related to health information systems as a potential Protocol tool.
Existing regulations at 42 C.F.R. § 438.242 (Health information systems) and 42 C.F.R. § 438.352
(External quality review protocols), require the MCO to ensure its information system processes, utilities,
and functional infrastructure have demonstrable capabilities for collecting valid, accurate data, and then
calculating and reporting quality improvement data. This is important to emerging electronic health record
and health information exchange technologies that States are using as part of quality improvement
initiatives in their care delivery systems.,
However, the HIPAA security audit reporting capabilities assessment supports and focuses on improving
information system integrity. HIPAA includes requirements for the assurance of information systems’
administrative integrity and the integrity of information gathered, processed, and reported by those
systems then used by covered entities, such as health plans. i Previous publications by HHS provide
guidance and tools on key aspects of HIPAA Security Rule compliance. ii

EQR Protocol: Appendix I
Methodology and Origin of External Quality Review (EQR) Protocol Development
September 2012

8

ADDITIONAL REFERENCES
Brown, J. G. (September 1998). Lessons Learned from Medicaid’s Use of External Quality Review
Organizations. Office of the Inspector General, United States Department of Health and Human
Services.
Ireys, H.T., Krissik, T., Verdier, J.M., Faux, M. (May 2005). Using External Quality Review Organizations
to Improve the Quality of Preventive and Developmental Services to Children. Mathematica Policy
Research, Inc., The Commonwealth Fund.
Krissik, T., Ireys, H.T., Markus, A., Rosenbaum, S. (July 2008). Monitoring and Assessing the Use of
External Quality Review Organizations for Young Children: A Toolkit for State Medicaid Agencies.
Mathematica Policy Research, Inc., The Commonwealth Fund, George Washington University.
Levinson, D. R. (June 2008). External Quality Reviews in Medicaid Managed Care. Office of the
Inspector General, United States Department of Health and Human Services.

END OF DOCUMENT

ii

See 45 C.F.R. §164.308 Administrative safeguards.
See Security Rule Educational Paper Series and the sublinks to Security 101, Administrative Safeguards, Physical
Safeguards, Technical Safeguards, Organizational Policies and Procedures and Documentation Requirements,
Basics of Risk Analysis and Risk Management, available at:
http://www.hhs.gov/ocr/privacy/hipaa/administrative/securityrule/securityruleguidance.html
ii

EQR Protocol: Appendix I
Methodology and Origin of External Quality Review (EQR) Protocol Development
September 2012

9