External Quality Review of Medicaid MCOs and Supporting Regulations in 42 CFR 438.360, 438.362, and 438.364

OMB Approval No.
0938-0786

EQR PROTOCOL 6
CALCULATION OF PERFORMANCE
MEASURES
A Voluntary Protocol for External Quality Review (EQR)
Protocol 1: Assessment of Compliance with Medicaid Managed Care Regulations
Protocol 2: Validation of Measures Reported by the MCO
Protocol 3: Validation of Performance Improvement Projects (PIPs)
Protocol 4: Validation of Encounter Data Reported by the MCO
Protocol 5: Validation and Implementation of Surveys
Protocol 6: Calculation of Performance Measures
Protocol 7: Implementation of Performance Improvement Projects (PIPs)
Protocol 8: Focused Studies
Appendix V: Information Systems Capabilities Assessment (ISCA)

Department of Health and Human Services (HHS)
Centers for Medicare & Medicaid Services (CMS)

Protocol 6
Version 2.0
December 1, 2011

PURPOSE AND OVERVIEW OF THE PROTOCOL .............................................................3
ACTIVITY 1: PREPARE FOR MEASUREMENT ...................................................................3
Step 1: Review State Performance Measure Requirements ............................................3
Step 2: Prepare for Data Collection .................................................................................4
Step 3: Review MCO’s Information System Capability Assessment (ISCA) .....................5
ACTIVITY 2: CALCULATE MEASURES ..............................................................................5
Step 1: Collect Performance Measure Data .....................................................................5
Step 2: Clean Data ..........................................................................................................6
Step 3: Integrate Data into Repository .............................................................................7
Step 4: Conduct Preliminary Analysis ..............................................................................8
Step 5: Calculate the Denominators, Numerators, and Performance Measure Rates ...... 9
ACTIVITY 3: REPORT RESULTS ........................................................................................9
Step 1: Report Preliminary Performance Measurement Rates to MCOs ........................ 10
Step 2: Analyze Data Using Prescribed Benchmarks and Performance Standards ....... 10
Step 3: Submit a Final Report to the State.....................................................................10
REFERENCES....................................................................................................................11

Attachments
Attachment A: Performance Measure Calculation Tables

EQR Protocol 6
Calculation of Performance Measures
December 2011

2

PURPOSE AND OVERVIEW OF THE PROTOCOL
This voluntary protocol is used to calculate performance measures for MCOs on dimensions
specified by the State. States use performance measures to monitor the performance of MCO’s
over time, to compare the performance of different MCOs, and to inform the selection and
evaluation of quality improvement activities. Recently, the Centers for Medicare and Medicaid
(CMS) identified a core set of pediatric and adult measures to use in Medicaid and CHIP by the
Children’s Health Insurance Program Reauthorization Act of 2009, (CHIPRA) and the Affordable
Care Act of 2010. These measures are for voluntary use by State agencies; however, CMS
encourages States to adopt and use the core set to work toward nationally standardized
reporting. Many of these measures are part of the HEDIS®1 data set, and have readily available
national and regional benchmarks. In 2009, half of the 20.5 million Medicaid enrollees in full-risk
traditional MCOs were in NCQA accredited plans that already collect HEDIS® data. 2
This protocol specifies how the EQRO is to:
•
•

Calculate measures of MCO performance in accordance with State technical
specifications; and
Report to the State the MCO’s performance compared to State-established
benchmarks or performance standards.

Protocol 6 consists of three activities: preparation, measure calculation, and reporting. For each
phase, the protocol specifies outcomes or objectives and lists the activities to be performed.
Methods of performance are suggested and examples are provided throughout the protocol.

ACTIVITY 1: PREPARE FOR MEASUREMENT
The EQRO will:
•
•
•

Review State performance measure requirements;
Prepare for data collection; and
Review MCO’s Information System Capabilities Assessment (ISCA).

Step 1: Review State Performance Measure Requirements
The State provides the EQRO with a list of performance measures to be calculated along with
technical specifications for their calculation. The EQRO must understand the State’s
specifications (e.g., sampling guidelines, instructions for calculating numerators and

®

1 HEDIS is a registered trademark of the National Committee for Quality Assurance (NCQA).
2
NCQA’s Medicaid Deeming Toolkit available at www.ncqa.org
EQR Protocol 6
Calculation of Performance Measures
December 2011

3

denominators) for each performance measure, as well as the State’s requirements for
benchmarking, analysis, and reporting.
The EQRO must also understand the State’s requirements for the timing and format of the
performance measure report. The EQRO should create a “List of Performance Measures to be
Calculated” (see an example in Attachment A, Table 1) to document the measures required by
the State and the reporting frequency and timeline for each measure.
For each performance measures listed in Table 1, the EQRO should construct a companion
performance measurement worksheet (see an example in Attachment A, Table 2) that contains
the technical specifications for the measure, benchmarks, performance standards, or any other
information needed to analyze the performance measure according to the State’s requirements.
The EQRO may need clarification from the State to specify measures if the State is not using
standardized performance measures or the CMS recommended performance measures. The
EQRO should use this worksheet to document any additional information needed for analysis,
such as benchmarks or percentiles (and their source), prior year performance, or State
averages.

Step 2: Prepare for Data Collection
The EQRO should send an introductory communication to the MCO outlining the purpose,
process, and timeline for its performance measure calculation activities. In addition, the EQRO
should request a contact within the MCO to schedule activities and provide requested
documents and other information. If data will be collected from other sources such as State
public health registries, an all-hospital discharge database, or behavioral service vendor under
contract to the State, the EQRO should establish contact with the organizations responsible for
these data sources.
The EQRO should inform the MCO that it may be necessary to interview MCO or vendor staff
with expertise or responsibility for data collection or performance measurement. Insights of
such staff might improve the EQRO’s ability to collect and integrate the information necessary
for calculating performance measures.
In addition, the EQRO will provide to the MCO:
•
•
•

A list and description of the State-required performance measures it plans to
calculate (the EQRO should send the MCO a copy of Table 1 in Attachment A);
A list of documents that the EQRO will need to review; and
A list of information the EQRO may need about the MCO’s structure and capacity to
supply the data needed to calculate the performance measures.

The information provided by the MCO should inform the EQRO of the location of the required
data, which organization (State, EQRO or MCO) will need to collect and integrate specific data
EQR Protocol 6
Calculation of Performance Measures
December 2011

4

elements, and how to access the data. (See Protocol 2, Attachment A for a comprehensive list
of documents.)

Step 3: Review MCO’s Information System Capability Assessment (ISCA)
The EQRO must review or conduct an ISCA (Appendix V) to:
•
•
•

Understand data sources, flows, and integration processes used by the MCO;
Identify where the EQRO must work with outside data sources to obtain missing
data; and
Determine which data elements are integrated by the MCO and which data elements
that the EQRO must integrate in order to calculate the required performance
measures.

The EQRO should prepare a master worksheet identifying the data elements required for each
performance measure (see Table 3 in Attachment A). Further, the EQRO should prepare a
worksheet (see Table 4 in Attachment A) for each data element identified in Table 3 in
Attachment A, identifying available data sources and noting any completeness or integration
issues for each data element relevant to the denominator and numerator of the measure.
Available data sources include sources internal to the MCO, such as claims or membership
data, which may be available in a data repository. Data sources may also include sources
external to the MCO, such as a State registry, provider medical record, MCO vendor or State
vendor. The EQRO may identify data capture or integration issues within the MCO, such as an
inability to capture individual prenatal care services in parts of the network that the MCO pays
using a global fee, or the EQRO may identify issues external to the MCO. For example, the
EQRO may have difficulty accessing confidential information about mental health services that
the State contracts with another organization to manage, or the data in a voluntary State registry
may be incomplete, or it may face challenges in linking birth records with beneficiaries due to
use of different identifiers.

ACTIVITY 2: CALCULATE MEASURES
The EQRO will:
•
•
•
•
•

Collect performance measure data;
Clean data;
Integrate data into repository;
Conduct preliminary analyses; and
Calculate denominators, numerators and performance measure rates.

Step 1: Collect Performance Measure Data

EQR Protocol 6
Calculation of Performance Measures
December 2011

5

The EQRO must obtain the data needed to calculate the performance measures. For each data
source, the EQRO should:
•
•
•

List the data required;
Specify how the data are to be selected; and
Specify how the data are to be transmitted to the EQRO, with appropriate privacy and
security safeguards.

The EQRO can facilitate consistency by providing an electronic data shell or file format, or a file
structure, such as in Table 5 of Attachment A, along with definitions of all data fields. The EQRO
should construct file formats that are customized to each data supplier. The file format for the
MCO or State encounter data repository will likely include all data elements that originate from
claims/encounter, eligibility and provider transaction systems. A file format used to obtain vital
records or immunization data from a public health department would contain different data fields
that are relevant to the information required from those data sources. Additional relevant
resources, targeted to development of all-payer claims databases, are available from the AllPayers Claim Database (APCD) Council Web site at http://www.apcdcouncil.org/.
The EQRO may need to conduct medical record review to obtain necessary data. If the EQRO
conducts such a medical record abstraction, it should develop the following supports:
•
•
•
•

Abstraction tools;
Training for personnel conducting the medical record abstraction;
Quality assurance procedures to assess the accuracy and reliability of the medical
record abstraction; and
Electronic data entry edits for abstracted medical record information.

If the MCO or other entity is performing medical record review and supplying those data to the
EQRO, the EQRO should refer to Protocol 2 to validate the abstracted medical record
information.

Step 2: Clean Data
The EQRO should clean all data by submitting it to electronic edits. As the EQRO receives data,
it should evaluate parity from each incoming data stream to ensure that the number of bits
received is equal to the number sent. When entering the data into its repository, the EQRO
should screen it to ensure accuracy. Some edits the EQRO can use include the following:
•
•
•

Valid procedure codes (e.g., current code, required number of digits);
Valid diagnosis codes;
Valid diagnosis and procedure codes for the situation (e.g., the member’s age or gender,
practitioner’s specialty);

EQR Protocol 6
Calculation of Performance Measures
December 2011

6

•
•
•
•

Correct field size and type (e.g., alpha, numeric, date, etc.);
Valid date ranges (e.g., “to” date is later than “from” date; dates occur during the
appropriate time-frame for the measure);
Valid practitioners; and
Valid members.

When data fail an edit, the EQRO should contact the supplier and request the data be corrected
and re-submitted. The EQRO should document the nature and extent of failures, including
information about whether it received corrected information. This documentation is necessary
for the EQRO to understand the accuracy and completeness of the data underlying the
performance measures it will calculate.
Data that pass the edit should be integrated in the EQRO’s performance measure repository
(see Step 3).

Step 3: Integrate Data into Repository
The EQRO will likely receive data from multiple MCOs, multiple sources within each MCO, and
from other organizations, such as statewide registry or other State contractors that are
responsible for delivering specific services like prescription drugs or mental health services. To
calculate performance measures, the data must be integrated so that all services provided to a
specific member can be associated with that member. The ISCA, reviewed in Activity 1, Step 3,
will provide information about the adequacy of data integration at the MCO. However, the
EQRO may need to evaluate data integration by other data suppliers, such as the State’s
encounter data repository or other vendors. The EQRO will need to integrate data from these
and other sources to calculate performance measures.
The EQRO must ensure that the appropriate data, including linked data from separate data
sets, are used to identify the entire eligible population for inclusion in the denominator and to
identify qualifying medical events (e.g., diagnoses, procedures, and prescriptions) for inclusion
in the numerator of the performance measure. The “appropriate data” for each measure is
documented in the completed Table 3 of Attachment A, which lists each data element required
for each of the performance measures to be calculated.
During this step, the EQRO will assess the data integration of each non-MCO data supplier.
This may include administering relevant portions of the ISCA to these other suppliers. The
EQRO must determine which portions of the ISCA are relevant depending on the specific data
elements the supplier provides and the degree of data integration the supplier must perform.
The assessment includes assessing the reliability of data transmissions within and from each
data supplier. The EQRO may have different degrees of access to these data suppliers and
must work with them to the degree possible, to understand the data flows and procedures used
to ensure data integrity. For each data supplier, the EQRO may need to:

EQR Protocol 6
Calculation of Performance Measures
December 2011

7

a. Examine the details of the data supplier’s processes to accurately and completely
transfer data from the transaction files (i.e., membership, provider, encounter/claims)
into its data repository, if any;
b. Examine samples of data to assess completeness and accuracy;
c. Investigate the data supplier’s processes to consolidate diversified files, and to extract
required information from its data repository;
d. Compare actual results of file consolidations or extracts to those that should have
resulted according to documented algorithms or specifications;
e. Review procedures for consolidating data from vendors in ways that ensure the
accurate, timely, and complete integration of the data;
f. Review computer program reports or documentation that reflect these vendor
coordination activities, and spot check to verify that no data necessary to performance
measure reporting are lost or inappropriately modified during transfer;
g. Assess the extent to which proper linkage mechanisms have been employed to join data
from all necessary sources (e.g., identifying a member with a given disease/condition);
and
h. If the data supplier uses one, evaluate the structure and format of the data repository,
and examine program flow charts and source code to assess the accuracy of the
process used to extract data for transmission to the EQRO.
To ensure proper data integration within its own data repository, the EQRO must undertake the
following activities:
a. Write program logic or source code for each measure that identifies, tracks, and links
member enrollment within and across product lines (e.g., Medicaid, CHIP), by age and
gender, as well as through possible periods of enrollment and disenrollment, in order to
appropriately comply with the specifications of each performance measure. Conduct
tests of data to assess completeness, integration and integrity, and to ensure there is no
double-counting of services reported through different data systems or suppliers.
b. Assure that all members who were eligible to receive the specified services were
included in the initial population from which the final denominator was produced. This
eligible population will include both members who received the services, as well as those
who did not. This same activity applies to provider groups or other relevant populations
identified in the specifications of each performance measure.

Step 4: Conduct Preliminary Analysis
The EQRO will analyze the data in its repository in order to assess its completeness and
accuracy, and reasonableness, and work with the MCO and other data suppliers until the data
are satisfactory. Referring to the ISCA conducted for the MCO and other data suppliers, the
EQRO will have identified areas of potential weakness. These should be considered in
conducting analyses of missing data, data quality, and supplier data issues.

EQR Protocol 6
Calculation of Performance Measures
December 2011

8

General Magnitude and Types of Missing Data
The EQRO will analyze its repository for evidence of missing data. The EQRO should compare
its data with data from the State, from prior years, and from similar populations, to determine
completeness. Based on findings from the ISCA, the EQRO may pursue specific concerns such
as missing beneficiaries, missing providers, provider locations or provider types, missing
services or service types. Knowledge of the data suppliers’ contractual relationships with
providers from the ISCA, as well as knowledge of the expected magnitude of reporting will help
identify specific areas to investigate for missing data. The EQRO will be aware of instances when
the MCO was unable to submit data, when submitted data failed edits, and when data was not
resubmitted.
Overall Data Quality Issues
The EQRO should analyze the data it has received to identify data quality problems such as
inability to process or retain certain fields. Some MCOs may lack the capacity to capture or
maintain all the data elements that are required for submission, such as secondary diagnosis or
procedure codes or some coding specificity.
Supplier Data Issues
Using the edit checks from Activity 2, Step 2, the EQRO should identify problems in how data
suppliers compiled and submitted their data to the EQRO.

Step 5: Calculate the Denominators, Numerators, and Performance Measure Rates
Following the specifications provided by the State, the EQRO calculates performance measures
from its data repository. To do this, the EQRO must purchase or write and test source code to
properly apply all specifications to identify the denominator population. The EQRO must apply
specified continuous enrollment criteria and exclusions from the denominator.
Once the EQRO has identified all eligible beneficiaries in the denominator for a measure, it must
apply the specifications to identify cases that qualify for inclusion in the numerator. Where
sampling is required such as for medical record review, the EQRO must follow the specifications
for selecting an appropriate sample. The EQRO should follow the medical record review
process outlined in Activity 2, Step 1 regardless of when in the measure calculation process the
medical record review takes place.

ACTIVITY 3: REPORT RESULTS
The EQRO will:
•
•
•

Report preliminary performance measurement rates to the MCOs;
Analyze the data, using prescribed benchmarks and performance standards; and
Submit a final report to the State.

EQR Protocol 6
Calculation of Performance Measures
December 2011

9

Step 1: Report Preliminary Performance Measurement Rates to MCOs
Once the EQRO has calculated the MCO’s performance measures, the EQRO summarizes its
preliminary findings in a report to the MCO for review and comment. The report should include,
at a minimum, the following elements for each performance measure: denominator; sample size
(if relevant); administrative numerator events (if relevant); medical record numerator event (if
relevant); and calculated rate. The report may also include analysis of performance in relation to
the State’s required benchmarks, prior year performance, or other reference sources to enable
the MCO to understand its result. The EQRO should invite the MCO to offer comments and
documentation to support correction of any factual errors or to clarify or explain any issues or
findings identified in the EQRO report. The EQRO should provide a reasonable period of time
for the MCO to return its comments.
Once the EQRO has received the MCO’s comments, it should recalculate measures and revise
its findings where appropriate.

Step 2: Analyze Data Using Prescribed Benchmarks and Performance Standards
Using the final, corrected information following incorporation of MCO comments, the EQRO
should conduct all analyses required by the State, as documented in Table 2. The EQRO
should conduct and review all comparative analyses, and present them, either in a format
prescribed by the State, or if the State has not prescribed a format, in a way that facilitates the
State’s intended use of the performance measures. Examples of formats include bar charts or
box plots, to compare one MCO to another and to an average or benchmark, or individual
reports that present a single MCO’s performance over time.

Step 3: Submit a Final Report to the State
The EQRO will submit its findings to the State in a final report format specified by the State. The
content of the final report will depend on the State’s requirements but is likely to include the
following elements:
•
•
•
•

A summary of the EQR activities, including documentation of the activities performed;
Work papers and detailed results of key steps of the measure calculation process, MCOspecific performance measure rates, and accompanying analyses;
Discussion of areas of MCO strength and opportunities for improvement in both data
management and performance; and
Recommendations for improving MCO performance.

The EQRO must report the performance measure results, analyses and recommendations in
the format prescribed by the State and in the time frame required. If the State requires the
EQRO to submit its report as a draft, it must await the State’s comments and revise the final
report, as required.
EQR Protocol 6
Calculation of Performance Measures
December 2011

10

REFERENCES
APCD Council http://www.apcdcouncil.org last accessed 11/10/10
Measures Management System Blueprint, Centers for Medicare and Medicaid Services
http://www.cms.gov/MMS/19_MeasuresManagementSystemBlueprint.asp last accessed
11/10/10
NCQA’s Medicaid Deeming Toolkit available at www.ncqa.org

END OF PROTOCOL

EQR Protocol 6
Calculation of Performance Measures
December 2011

11