(Private Sector) External Quality Review of Medicaid MCOs and Supporting Regulations in 42 CFR 438.360, 438.362, and 438.364

OMB Approval No.
0938-0786

EQR PROTOCOLS
INTRODUCTION
An Introduction to the External Quality Review (EQR)
Protocols

Department of Health and Human Services (HHS)
Centers for Medicare & Medicaid Services (CMS)

Introduction
Version 1.0
September 2012

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it
displays a valid OMB control number. The valid OMB control number for this information collection is 0938-0786. The time required
to complete this information collection is estimated to average 1,591 hours per response for all activities, including the time to review
instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have
comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: CMS, 7500
Security Boulevard, Attn: PRA Reports Clearance Officer, Baltimore, Maryland 21244-1850

EQR Protocols Introduction
September 2012

2

Table of Contents

EXTERNAL QUALITY REVIEW (EQR) BACKGROUND ............................................................................... 4
PURPOSE OF THE EQR PROTOCOLS ....................................................................................................... 6
Protocol 1 – Assessment of Compliance with Medicaid Managed Care Regulations ...................................8
Protocol 2 – Validation of Measures Reported by the MCO ..............................................................................8
Protocol 3 – Validation of Performance Improvement Projects (PIPs) ............................................................8
Protocol 4 – Validation of Encounter Data Reported by the MCO ...................................................................9
Protocol 5 – Validation and Implementation of Surveys ....................................................................................9
Protocol 6 – Calculation of Measures ...................................................................................................................9
Protocol 7 – Implementation of Performance Improvement Projects (PIPs).................................................10
Protocol 8 – Focused Studies ..............................................................................................................................10
Appendices .............................................................................................................................................................11
CONDUCTING EXTERNAL QUALITY REVIEW.......................................................................................... 11

EQR Protocols Introduction
September 2012

3

EXTERNAL QUALITY REVIEW (EQR) BACKGROUND
The Balanced Budget Act of 1997 (BBA) requires State Medicaid agencies that contract with
Medicaid managed care organizations (MCOs) to develop a State quality assessment and
improvement strategy that is consistent with standards established by the Department of Health
and Human Services (HHS). 1 The BBA also requires HHS to develop protocols for use in the
performance of independent, external reviews of the quality and timeliness of, and access to,
care and services provided to Medicaid beneficiaries by Medicaid MCOs and prepaid inpatient
health plans (PIHPs).
The Children’s Health Insurance Program Reauthorization Act of 2009 (CHIPRA) requires that
Children’s Health Insurance Program (CHIP) managed care plans also participate in external
quality review (EQR). CHIPRA Section 403, adds managed care requirements applicable to
Medicaid under §§1932(a)(4), (a)(5), (b), (c), (d), and (e), to States contracting with MCOs for the
delivery of care under separate CHIP programs. These provisions apply to contract years for
managed care plans beginning on or after July 1, 2009. Section 401(c)(1) of CHIPRA requires
each State to annually report on its child health quality measures and other State-specific
information, including information collected through EQRs. CMS strongly encourages States to
have final EQR Technical Reports available to CMS and the public by April of each year, for data
collected within the prior 15 months. This submission timeframe will align with the collection and
annual reporting on managed care data by the Secretary each September 30th, which is also
required under the Affordable Care Act [Sec. 2701 (d)(2)]. In 2010, the Secretary began an
analysis and publication of information obtained from this annual data.
For the purposes of these protocols, “MCO” will include all managed care organizations subject
to EQR. As of December 31, 2010, “MCO” includes all Medicaid and CHIP MCOs and PIHPs.
A State Medicaid agency may choose an External Quality Review Organization (EQRO) as
defined in Federal regulation or a non-EQRO organization to conduct MCO reviews,
assessments, studies, or validations. For convenience, in these protocols, an “EQRO” refers to
any organization that conducts an EQR of a State Medicaid or CHIP MCO.
The regulations at 42 C.F.R. § 438, subpart E—External Quality Review mandate States to
perform three activities:
1. A review conducted within the previous 3-year period that determines the MCO's
compliance with standards established by the State to comply with the requirements of 42
C.F.R. § 438.204(g), (except standards under 42 C.F.R. §§ 438.240(b)(1) and (b)(2), for
the conduct of performance improvement projects and calculation of performance
measures respectively), as well as applicable elements of the MCO’s State contract
(Protocol 1). It is important to note, however, that the MCO is responsible for addressing
any recommendations by the EQRO for that review in the following year, as part of the
Annual EQRO Technical Report process (per 42 C.F.R. § 438.364);

1

See Section 1932(c)(1)(B) of the Social Security Act.

EQR Protocols Introduction
September 2012

4

2. Validation during the preceding 12 months, to determine the accuracy of the
performance measures reported by the MCO and the MCO’s compliance with rules
outlined by the State as set forth in 42 C.F.R. § 438.240(b)(2) for calculating the
performance measures (Protocol 2); and
3. Validation of performance improvement projects (PIPs) required by the State to comply
with requirements set forth in 42 C.F.R. § 438.240(b)(1) and that were underway during
the preceding 12 months (Protocol 3).
In addition, a State may use five additional EQR activities to assess the performance of its
MCOs:
4. Validation of encounter data reported by the MCO (Protocol 4);
5. Design and administration of a survey or validation of the results of a previously
administered survey (Protocol 5);
6. Calculation of performance measures for a State based on data reported by the MCO
and validation of the accuracy of the calculations of the performance measures made by
the MCO’s information system (Protocol 6);
7. Implementation of PIPs required by the State in addition to those conducted by the MCO
(Protocol 7); and
8. Implementation of focused, one-time studies of the MCO’s clinical and/ or non-clinical
services as directed by the State (Protocol 8).
Under the direction of the Centers for Medicare & Medicaid Services (CMS), The Joint
Commission worked with several consultants and an expert panel, composed of representatives
of private accrediting organizations, quality measurement experts, State Medicaid agencies, and
advocates for Medicaid beneficiaries, to develop an initial set of nine EQR protocols in 2001. The
initial set of protocols included one for each of the eight EQR activities listed above and one for
an information system assessment applicable to the EQR activities. HHS published its final rule
on EQR of Medicaid MCOs on January 24, 2003. The regulations at 42 C.F.R. § 438 require that
State Medicaid programs use Protocols 1, 2, and 3 (mandatory EQR activities) to complete their
review of MCOs.
Significant changes in health care policy occurred in 2009 and 2010 that offer new opportunities
in quality improvement of health care, including CHIPRA, the American Recovery and
Reinvestment Act (ARRA), and the Affordable Care Act. In 2010, CMS contracted Provider
Resources, Inc. (PRI) and their subcontractor, the National Committee for Quality Assurance
(NCQA), to work with EQR stakeholders, including States, EQROs, and MCOs, to modernize the
initial EQR protocols. Substantial updates for both content and structure are included in the
revised protocols provided in this package. Appendix A provides methodology for the
development of the initial EQR protocols and the methodology used for the revisions in 2010. In
addition to the Protocol revisions, CMS is considering changes in regulatory requirements to
address the duplication of effort that occurs when States seek to fulfill both EQR and
accreditation requirements.
EQR Protocols Introduction
September 2012

5

PURPOSE OF THE EQR PROTOCOLS
The purpose of the EQR protocols is to provide States, their Medicaid and CHIP MCOs, and
EQROs with instructions for performing EQR. They fulfill the requirement found in
1932(c)(2)(a)(iii) of the Social Security Act for a governing protocol for EQROs to use for EQR of
MCOs. Nine protocols guide EQR activities. The first three protocols mirror 42 C.F.R. §438,
subpart E specifications for the three mandated EQR activities. Protocols 4 through 8 guide
additional voluntary EQR activities and Appendix V guides an MCO information system
assessment that is necessary for the completion of several other protocols. The protocol map on
the next page illustrates how the EQR protocols and activities relate to one another, as well as
source regulation information.
Each protocol describes why the activity is important to EQR, what the activity accomplishes,
and how to conduct the activity. Specifically, each protocol includes:
•
•
•
•
•

Data to be gathered;
Sources of the data;
Activities and steps to collecting the data to promote its accuracy, validity, and reliability;
Proposed method or methods for validly analyzing and interpreting the data once
obtained; and
Instructions, guidelines, worksheets, and other documents or tools that may be used, but
are not required, for implementing the protocol.

EQR Protocols Introduction
September 2012

6

Figure 1: CMS External Quality Review Protocols

MCO
Compliance

Performance
Measurement

Performance
Improvement

Information
Systems (IS)

Protocol 1
(MANDATORY)

Protocol 2
(MANDATORY)

Protocol 3
(MANDATORY)

Appendix V

[42 CFR 438.358(b)(3)]

[42 CFR 438.358(b) (2)]

[42 CFR 438.358(b) (1)]

Review of MCO
Standards

Validation of Measures
Reported by the MCO

Validation of
Performance
Improvement Projects
(PIPs)

1. Federal Regulations
2. State Regulations
3. MCO Contract

Protocol 4
Validation of
Encounter Data
Reported by MCO

IS Capabilities
Assessment (ISCA)
(applicable to protocols
1, 2, 3, 4, 6)

Protocol 7
Implementation of
PIPs

Protocol 5
Administration and
Validation of
Surveys

(Mandatory in
Protocols 1, 2, 3)

Protocol 8
Implementation of
Special Studies

Protocol 6
Calculation of
Measures

Area of
Review
Mandatory
Protocol
Optional
Protocol

Throughout the protocols, the following definitions apply:
•

•
•

Quality means the degree to which the MCO increases the likelihood of desired health
outcomes of its enrollees through its structural and operational characteristics and
through the provision of health services that are consistent with current professional
knowledge in at least one of the six domains of quality as specified by the Institute of
Medicine (IOM) – efficiency, effectiveness, equity, patient-centeredness, patient safety,
and timeliness. 1
Validation means the review of information, data, and procedures to determine the extent
to which they are accurate, reliable, free from bias, and in accord with standards for data
collection and analysis; and
MCO means all managed care organizations, including PIHPs under a Medicaid and/or
CHIP program.

1

This is the definition of quality in the context of Medicaid/CHIP MCOs, and was adapted from the IOM
definition of quality.
EQR Protocols Introduction
September 2012

7

Protocol 1 – Assessment of Compliance with Medicaid Managed Care Regulations
States are required to perform a compliance review of each MCO once in a 3-year period.
Protocol 1 is mandatory and specifies procedures to determine the extent to which MCOs comply
with Federal regulatory provisions, State standards, and MCO contract requirements. Protocol 1
includes five activities:
1.
2.
3.
4.
5.

Establish Compliance Thresholds;
Perform Preliminary Review;
Conduct MCO Site Visit;
Compile and Analyze Findings; and
Report Results to the State.

In addition, Protocol 1 includes four attachments:
A. Compliance Review Worksheet
B. Compliance Definitions
C. Sample Site Visit Agenda
D. Compliance Interview Questions
While a full compliance review is only required once every 3 years, States must address any
EQR findings in the next reporting year.

Protocol 2 – Validation of Measures Reported by the MCO
States must provide to the EQRO and the MCO the performances measures to be calculated,
the specifications for the measures, and State reporting requirements. Protocol 2 is mandatory
and tells the EQRO how to:
1. Evaluate the accuracy of the Medicaid/ CHIP MCO reported performance measures
based on the measure specifications and State reporting requirements; and
2. Evaluate if the MCO followed the rules outlined by the State Agency for calculating the
measures.
Protocol 2, Attachment A, provides 14 different worksheets for validating performance measures.

Protocol 3 – Validation of Performance Improvement Projects (PIPs)
MCOs are required to implement a performance improvement project (PIP). Protocol 3 is
mandatory and specifies procedures for EQROs to use in assessing the validity and reliability of
a PIP. Protocol 3 includes three activities:
1. Assess the Study Methodology;
2. Verify PIP Study Findings (Optional); and
3. Assess Overall Validity and Reliability of PIP Results.
Protocol 3, Attachment A, provides a PIP Review Worksheet.

EQR Protocols Introduction
September 2012

8

Protocol 4 – Validation of Encounter Data Reported by the MCO
States use encounter data, which are data about a distinct service provided to an enrollee, to
assess and review quality, monitor program integrity, and determine capitation payment rates.
Protocol 4 is voluntary and specifies procedures for assessing the completeness and accuracy of
encounter data submitted by MCOs to the State. It also assists in the improvement of processes
associated with the collection and submission of encounter data. Protocol 4 includes five
activities:
1.
2.
3.
4.
5.

Review State requirements for collecting and submitting encounter data;
Review the MCO’s capability to produce accurate and complete encounter data;
Analyze MCO electronic encounter data for accuracy and completeness;
Review medical records for additional confirmation of findings; and
Submit findings.

Protocol 4, Attachment A, provides Encounter Data Validation Tables and Attachment B provides
a Medical Record Review Worksheet.
While Protocol 4 is voluntary, the significance of encounter data in payment reform continues to
become increasingly important. CMS strongly encourages States to incorporate the validation of
encounter data as part of the responsibilities in the State’s EQR contract.

Protocol 5 – Validation and Implementation of Surveys
Healthcare surveys are a common method of measuring healthcare quality. Protocol 5 is
voluntary and specifies procedures for conducting various types of surveys and validating those
surveys. Protocol 5 includes eight activities:
1.
2.
3.
4.
5.
6.
7.
8.

Identify Survey Purpose(s), Objective(s) and Intended Use;
Select the Survey Instrument;
Develop the Sampling Plan;
Develop a Strategy to Maximize the Response Rate;
Develop a Quality Assurance Plan;
Implement the Survey;
Prepare and Analyze the Data Obtained from the Survey; and
Document the Survey Process and Results.

Protocol 5, Attachment A, provides a Survey Validation Worksheet.

Protocol 6 – Calculation of Measures
States use performance measures to monitor the performance of MCOs over time, to compare
the performance of different MCOs, and to inform the selection and evaluation of quality
improvement activities. Protocol 6 is voluntary, and specifies procedures for calculating MCO
performance measures in accordance with State specifications. It also supplies information to the
State on the extent to which the MCO’s information system provides accurate and complete
information necessary for the calculation of performance measures. Protocol 6 includes three
activities and eleven different steps:
EQR Protocols Introduction
September 2012

9

1. Prepare for Measurement
1.1. Review State Requirements
1.2. Prepare for Data Collection
1.3. Review MCO’s Information System Capabilities Assessment (ISCA)
2. Calculate Measures
2.1. Collect Performance Measure Data
2.2. Clean Data
2.3. Integrate Data into Repository
2.4. Conduct Preliminary Analyses
2.5. Calculate Denominators, Numerators, and Performance Measure Rates
3. Report Results
3.1. Report Preliminary Performance Measurement Rates to MCOs
3.2. Analyze the Data Using Prescribed Benchmarks and Performance Standards
3.3. Submit a Final Report to State
Protocol 6, Attachment A, provides Performance Measure Calculation Tables.

Protocol 7 – Implementation of Performance Improvement Projects (PIPs)
States may request an EQRO conduct a statewide PIP. Protocol 7 is voluntary and specifies
procedures for implementing voluntary PIPs. It includes ten activities:
1. Select the Study Topic(s);
2. Define the Study Question(s);
3. Select the Study Variables(s);
4. Use a Representative and Generalizable Sample;
5. Use Sound Sampling Techniques (if sampling is used);
6. Reliably Collect Data;
7. Implement Intervention and Improvement Strategies;
8. Analyze Data and Interpret Study Results;
9. Plan for “Real” Improvement; and
10. Achieve Sustained Improvement.

Protocol 8 – Focused Studies
States may choose to conduct a focused study of healthcare provided by its MCOs. Protocol 8
specifies procedures to do so and includes seven activities:
1.
2.
3.
4.
5.
6.
7.

Select the Study Topic(s);
Define the Study Question(s);
Select the Study Measures(s);
Study the whole population or use a representative sample;
Use Sound Sampling Techniques;
Reliably Collect Data; and
Analyze Data and Interpret Study Results.

EQR Protocols Introduction
September 2012

10

Appendices
The revised EQR protocol package includes five Appendices that are applicable to all of the
protocols:
I.
II.
III.
IV.
V.

Methodology and Origin of EQR Protocol Development
Sampling Approaches
EQR Glossary
EQR Acronyms
Information Systems Capabilities Assessment

Protocols 1, 2, 3, 4 and 6 mandate that States assess their MCOs’ information system
capabilities. The regulations at 42 C.F.R. § 438.242 also require the State to ensure that each
MCO maintains a health information system that collects, analyzes, integrates, and reports data
for areas including, but not limited to, utilization, grievances and appeals, and disenrollments for
other than loss of Medicaid eligibility. Portions of Appendix V are voluntary; however, there are
components that relate directly to the three mandatory protocols. It defines the desired
capabilities of an MCO’s information system as well as how to assess the strength of the MCO’s
information system capabilities. It includes a basic overview of the processes of collecting,
processing, and reporting data, four activities to performing the assessment, and information
about the future of information system assessments, and four activities:
1.
2.
3.
4.

Completion of the Information System Capabilities Assessment (ISCA);
Review of ISCA and Accompanying Documents;
Interview MCO Staff; and
Analyze ISCA Findings.

Appendix V, Attachment A, provides an ISCA Template and Attachment B provides an
Information System Review Worksheet and Interview Guide.

CONDUCTING EXTERNAL QUALITY REVIEW
The protocols assist States in meeting the requirement to conduct an EQR of its MCOs. The
regulations require a State/ EQRO to use methodologies that are consistent with the Protocols;
therefore, States/ EQROs have the option of using other methods and/or tools than those
provided in these Protocols if the alternative methods are equivalent to the EQR Protocols.
States are encouraged to discuss any potential alternative methods to EQR methods with CMS
prior to implementation to assure they meet regulatory compliance.
While most States hire an EQRO or other appropriately qualified and independent organization
to fulfill the EQR requirements, a State may designate a State department other than the
Medicaid agency to perform the EQR. However, per regulation, an independent organization
(e.g., an EQRO) must draft the final EQR report.
Preparation for an EQR begins with a clear, written understanding of the parameters of the EQR,
including a list of the MCOs for review, selection of applicable protocols (the three mandatory
protocols and any additional that the State chooses), and a review of all applicable Federal
regulations, State regulations or standards, and MCO State contracts. States will determine
EQR Protocols Introduction
September 2012

11

which protocols to apply during established timeframes and for which of their MCOs. States must
also take into consideration the CMS Medicaid and CHIPRA core sets of quality measures and
the annual Secretary’s Reports, published at the end of each September. Prior to conducting
individual protocols, the State and EQRO should confirm with all EQR participants the
understanding of:
1. Each organization’s responsibilities in collecting, reporting, and/ or analyzing data
applicable to selected protocols;
2. Which regulations and/or contracts and/or initiatives will be evaluated;
3. Which reviews will be performed and the tools that will be used; and
4. The scheduled commencement and completion of each protocol.
The State may decide that immediate corrective action is required during any part of an EQR. At
the conclusion of the EQR, the EQRO must prepare a detailed technical report for the State that
includes the following for each activity conducted in accordance with 42 C.F.R. § 438.358:
•
•
•

An assessment of each MCO's strengths and weaknesses with respect to the quality,
timeliness, and access to healthcare services furnished to Medicaid recipients;
Recommendations for improving the quality of healthcare services furnished by each
MCO; and
An assessment of the degree to which each MCO has effectively addressed the
recommendations for quality improvement made by the EQRO during the previous year's
EQR.

The report must also document procedures used by the EQRO to analyze the data collected and
how they reached their conclusions regarding the quality, timeliness, and access to care
furnished by the MCO. This report must be available to the public and must assure the privacy of
patient information. The State may request methodologically appropriate, comparative
information about all MCOs. At the State’s discretion, the State may request the EQRO assess
how each MCO contribute toward the State’s overall Quality Strategy or objectives.
END OF PROTOCOL

EQR Protocols Introduction
September 2012

12