(State Gov't) External Quality Review of Medicaid MCOs and Supporting Regulations in 42 CFR 438.360, 438.362, and 438.364

OMB Approval No.
0938-0786

EQR PROTOCOL 2
VALIDATION OF PERFORMANCE
MEASURES REPORTED BY THE MCO
A Mandatory Protocol for External Quality Review (EQR)
Protocol 1: Assessment of Compliance with Medicaid Managed Care Regulations
Protocol 2: Validation of Measures Reported by the MCO
Protocol 3: Validation of Performance Improvement Projects (PIPs)
Protocol 4: Validation of Encounter Data Reported by the MCO
Protocol 5: Validation and Implementation of Surveys
Protocol 6: Calculation of Performance Measures
Protocol 7: Implementation of Performance Improvement Projects (PIPs)
Protocol 8: Focused Studies
Appendix V: Information Systems Capabilities Assessment (ISCA)

Department of Health and Human Services (HHS)
Centers for Medicare & Medicaid Services (CMS)

Protocol 2
Version 2.0
September 2012

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it
displays a valid OMB control number. The valid OMB control number for this information collection is 0938-0786. The time required
to complete this information collection is estimated to average 1,591 hours per response for all activities, including the time to review
instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you
have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: CMS, 7500
Security Boulevard, Attn: PRA Reports Clearance Officer, Baltimore, Maryland 21244-1850

Table of Contents
PURPOSE AND OVERVIEW OF THE PROTOCOL .............................................................4
ACTIVITY 1: PRE-ONSITE VISIT ACTIVITIES .....................................................................4
Step 1: Define the Scope of the Validation ......................................................................5
Step 2: Assess the Integrity of the MCO’s Information System .......................................5
Step 3: Select Measures for Detailed Review .................................................................8
Step 4: Initiate Review of Medical Record Data Collection..............................................9
Step 5: Prepare the MCO Onsite Visit ..........................................................................10
ACTIVITY 2: ONSITE VISIT ACTIVITIES ............................................................................10
Step 1: Review Information Systems Underlying Performance Measurement............... 11
Step 2: Assess Data Integration and Control for Performance Measure Calculation ... 112
Step 3: Review Performance Measure Production .......................................................13
Step 4: Conduct Detailed Review of Selected Measures ..............................................14
Step 5: Assess the Sampling Process (if applicable) ....................................................16
Step 6: Preliminary Findings and Outstanding Items ....................................................16
ACTIVITY 3: POST-SITE VISIT ACTIVITIES .....................................................................16
Step 1: Determine Preliminary Validation Findings for Each Measure .......................... 17
Step 2: Assess Accuracy of MCO’s Performance Measure Reports to the State .......... 17
Step 3: Submission of Validation Report to State .........................................................18

Attachments
Attachment A: Performance Measure Validation Worksheets

____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

2

____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

3

PURPOSE AND OVERVIEW OF THE PROTOCOL
This mandatory protocol is used to validate Medicaid and CHIP performance measures. The
State may specify the performance measures, as well as requirements for calculating and
reporting them.
Federal programs are aligning efforts more than ever before to utilize health information
technology (HIT) through meaningful use (HITECH) and quality improvement initiatives (e.g.,
the Affordable Care Act) to support States. The Centers for Medicare & Medicaid Services
(CMS) has developed and now strongly encourages States to adopt a core set of performance
measures for child and adult health in CHIP and Medicaid to progress towards nationally
standardized reporting. Many of these measures have already been in widespread use as part
of the HEDIS® data set and have readily available national and regional benchmarks.
The protocol describes a process for the EQRO to use when validating these performance
measures. In general, the EQRO must:
1. Assess the accuracy of performance measures reported by the MCO 1; and
2. Determine the extent to which performance measures calculated by the MCO
follow State specifications and reporting requirements.
The validation process is interactive and concurrent with MCO performance measure
calculation. The protocol activities take place prior to, during, and after an onsite visit. For each
of these three phases, the protocol specifies objectives and lists the activities the EQRO must
perform. This protocol also suggests methods of evaluation and provides tools and worksheets
in an Attachment.
Results of the MCO’s performance measures may be reported in the annual Secretary’s Report
on the Quality of Care for Children in Medicaid and CHIP or the annual Secretary’s Report on
the Quality of Care for Adults in Medicaid. These reports are released every September and
information that is not available from a State’s EQR report may be so noted in the reports. Both
reports will be available on the CMS Medicaid website. States are strongly encouraged to have
EQROs include outcome and trending information of performance measures reported in the
annual EQR technical report.

ACTIVITY 1: PRE-ONSITE VISIT ACTIVITIES
Pre-Onsite visit activities focus on preparing the EQRO and the MCO for the onsite visit. The
EQRO will:
• Define the scope of the validation by confirming:
a. State required technical specifications for each of the performance measures;
b. State requirements for performance measure reporting (e.g., report template,
electronic submission format, etc.);

1 While the protocol is written as if the MCO is calculating performance measures, the MCO may contract
with another entity to calculate and report on its behalf. The protocol applies in either circumstance.
____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

4

•
•
•
•

Assess the integrity of the MCO’s Information System;
Select measures for detailed review;
Initiate review of medical record data collection; and
Prepare for the MCO onsite visit. The EQRO may request documents from the MCO
prior to the site visit.

STEP 1: DEFINE THE SCOPE OF THE VALIDATION
The performance measures each State requires will depend upon the specific needs of the
State. The State will provide the EQRO with a list of its required performance measures along
with requirements for data collection and reporting (e.g., sampling guidelines and instructions for
calculating numerators and denominators). Four data collection sources are used to produce
MCO performance measures include:
1. Administrative data;
2. Medical records (infrequently used due to the level of effort);
3. Administrative data supplemented by medical record review, referred to as “hybrid”
methodology; and
4. Surveys (Administration and validation of surveys are addressed in Protocol 5).
To gather measures and specifications, the EQRO should list the performance measures to be
calculated by the MCO and create a separate performance measure validation worksheet for
each required measure containing the measure specifications and components including:
1. The measure’s eligible population;
2. Data collection methodology;
3. Sampling methodology (if used);
4. Denominator calculations;
5. Numerator calculation; and
6. Calculated and reported rates.
Attachment A provides a template for gathering measures and their specifications (Worksheet
1) and a template for documenting the EQRO’s validation of the measures (Worksheet 2). This
includes placeholders for the components to be validated and the elements to be audited.
Attachment A also includes an example of a completed measure-specific worksheet.
For example, if the measure is Childhood Immunization Status (following HEDIS®
specifications), the EQRO would replace the general “age and continuous enrollment”
categories in the denominator portion of the tool with the particular age and sex specifications
associated with that measure (i.e., children who turn 2 years who were continuously enrolled for
12 months prior to their birthday).

Step 2: Assess the Integrity of the MCO’s Information System
This activity helps focus the onsite validation activities on the areas most likely to be an issue in
the validation process. The EQRO must assess the integrity of the MCO’s information system
and the completeness and accuracy of the data produced before validating individual
performance measures. Prior to conducting the onsite visit, the EQRO should send to the MCO
an Information Systems Capabilities Assessment (ISCA) template (see Appendix V). The ISCA
corresponds to the objectives identified in this protocol and addresses key components of
calculating performance measures including:
____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

5

•
•
•
•
•
•

General information about the MCO;
Membership/enrollment data systems;
Claims/encounter data processing;
Provider data;
Data completeness; and
Integration of data for performance measure calculation

The ISCA provides information about the timing of any other recent, independent, documented
assessment such as a HEDIS® Compliance Audit. If the MCO recently had an independent
assessment of its information systems, the EQRO may review those results. If the MCO has not
had an ISCA within the State’s required timeframe 2, the EQRO will conduct an ISCA as part of
this protocol. The EQRO should document the strengths and weaknesses of the MCO’s
information system relevant to the types of data used in MCO performance measures.
The EQRO should assess every data system/ type the MCO processes to ensure the required
data are current and accurate, particularly at the time it runs its source code/ computer
programs to extract data for its performance measures. The EQRO should assess changes in
the MCO’s data systems that might affect the production of the performance measures. Major
changes, upgrades, or consolidations within the system, or acquisitions/mergers with other
MCOs may impact the accuracy or completeness of required data elements.
At the State’s discretion, it may choose to contract with the EQRO to assess how the MCO
supports the State’s health information technology plan under HITECH for the electronic health
record (EHR) incentive program.
Membership/Enrollment Data
The EQRO should assess:
a. The MCO’s ability to track:
• Members over time;
• Changes in enrollment;
• Name changes; and
• Changes in coverage;
b. The MCO’s processes used to ensure enrollment/membership data are current and
accurate;
c. Changes in the MCO’s membership data systems that might affect the production of
the MCO performance measures; and
d. Whether MCO / State data transactions, such as State eligibility files affect measure
calculation through updating, correcting or overwriting source data (e.g., race or
ethnicity information).

2

Each State must determine the maximum interval between assessments of MCO information
systems, balancing the cost to the State and burden on the MCO with the need to ensure that
changes to the MCO’s information systems are assessed frequently enough to support accurate
performance measurement.

____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

6

The EQRO should determine whether each member is uniquely identifiable and can be linked to
the State’s eligibility file. The membership or enrollment database should capture the following
information:
a. Date of birth;
b. Gender;
c. Race and ethnicity;
d. Enrollment and/or termination dates (including multiple enrollment and termination
dates within and across programs);
e. Primary care provider (e.g., provider name, provider identification number); and
f. Unique Member identifier including State-issued number, CMS-issued Medicare
number.
As States continue to move forward with the Medicaid expansion under the Affordable Care Act,
it will be increasingly important that the EQRO closely assess membership and enrollment data.
This monitoring will also play an essential role in addressing health disparities, a directive
established under Section 4302 of the Affordable Care Act.
Provider Data
The EQRO should assess the MCO’s ability to track providers over time, across multiple office
locations, and through changes in participation.
In addition, the EQRO should assess how many contracted providers utilize electronic health
records (EHRs) and how this interfaces with the collection of performance measure data.
Claims Data and Encounter Data
Claim/encounter data should cover all types of services offered by the MCO and not separately
contracted by the State, such as behavioral health, family planning, home health care, hospital,
laboratory, pharmacy, primary care, radiology, specialty care, and vision care. The EQRO
should note the following for each type of claim/encounter data captured:
a. Total number of diagnosis and procedure codes captured by the system
b. Whether the principal diagnosis, secondary diagnosis, or procedure codes can be
accurately distinguished in the system
c. Maximum number of digits/ characters captured for each data field in each type of
claim/encounter.
The accuracy and validity of measures may be adversely affected if the information system
truncates codes or is unable to collect and/or differentiate among a sufficient number of codes.
The EQRO should understand the various coding systems and forms used by the MCO and its
vendors to capture and process clinical information through its claims and encounter databases;
as well as how the information system translates or maps these codes back to standard codes
for MCO performance measure reporting, and how it ensures the accuracy of these translation
processes.
Medical Record Data
The EQRO should review following:
a. Methods used to retrieve information from medical records;
b. Training and tools that medical record review staff receive; and
c. Processes used to ensure accurate data retrieval, inter-rater reliability, and data
entry into a database used to produce performance measures
____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

7

With increasing adoption of EHRs and State use of Health Information Exchanges, MCOs and
provider practices may use newer methods including reporting programs to extract information
from the medical record.
Pharmacy, Laboratory and Other Ancillary Data
Pharmacy data use standardized codes for prescription drugs such as those promulgated by the
National Council for Prescription Drug Programs (NCPDP). Laboratory services use a similar,
nationally recognized system of coding (e.g., LOINC). Due to the diversity in the size, type, and
ownership of pharmacy, laboratory and other ancillary providers, non-standard codes should be
examined. When found, the EQRO will assess the MCO’s system for cross walking these
different codes to store the necessary information in its performance measure database. The
EQRO should understand the MCO’s mapping system of non-standard codes to standardized
codes and the mechanism used to ensure the accuracy of these translation processes.
If the MCO does not collect pharmacy, laboratory, or other ancillary data through an
administrative or claims database, it may retrieve these data from medical records. Medical
records often are an unreliable source due to non-standard coding and terminology, poor
coordination of records, and record linkages between primary care and specialist providers.
These issues should be addressed during the medical record review, and, if necessary, on any
corrective action plan.
The EQRO must assess the ability of the information system to link these different sources of
data. For example, to identify enrollees with diabetes, a MCO may combine diagnosis code data
from inpatient or ambulatory encounters (not all ongoing conditions are reported at every
encounter) with pharmacy data, lab data, and/or a disease registry if one exists. To determine
whether these diabetic enrollees have received a retinal examination from an ophthalmologist or
optometrist within the previous year, the MCO would have to link procedure code data from
encounter forms, medical records, or claims with information about the specialty of the providers
that performed the examinations for these members.
The EQRO will compare the MCO responses to calculate and report the mandated MCO
performance measures, and will identify any problem areas or items in need of clarification.
Where a response is incomplete or indicates an inadequate process, this issue is noted for
follow-up and further review during the onsite activities.

Step 3: Select Measures for Detailed Review
Based on the State’s reporting requirements and the finding from the ISCA, the EQRO, in
conjunction with the State, will select a set of measures for detailed review. The selected
measures should represent the systems and processes most vulnerable to inaccurate results.
For example, if the MCO uses global billing for maternity care, a maternity measure may be
selected. Similarly, if a previous validation identified problems with a particular measure, that
measure could be selected for detailed review to confirm that the issues were resolved.
Categories for selection may include:
• Claims-dependent denominators;
• Complex continuous enrollment criteria;
• Identification of live births;
• Vendor-supplied data;
____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

8

•
•
•

Ability to link claims and pharmacy data;
Identification of practitioner type; or
Multiple numerator events.

The MCO should be notified of the measure set after the majority of source code programming
has been completed. A detailed review of each measure in the set includes the following:
• Source code for the measure
• Data mapping if applicable
• Measure workflow
• Data review at each stage of the measure
• Record-level numerator and denominator data
If the State requires HEDIS® measures and NCQA-certified software is used to calculate the
measures, the EQRO does not need to review source code for those measures. Only
verification that the measures were calculated as specified by the software is required. The set
of measures for detailed review may be expanded based on findings during the onsite visit. The
EQRO should complete a Performance Measure Validation Worksheet, as shown in Attachment
A, or a similar worksheet for all measures selected for detailed review.

Step 4: Initiate Review of Medical Record Data Collection
The purpose of this review is to verify the accuracy of the medical record review conducted by
each MCO. The medical record review must be validated for MCOs that use medical record
data to report performance measures. To validate the integrity of medical record review
processes, the EQRO will confirm the following about MCO activities:
• The review staff’s experience and credentials;
• Ensure the review tools collect the required information;
• Evaluate the training provided; and
• Verify the process includes statistically sound assessment of reviewer performance.
The EQRO will review a convenience sample of records across measures to identify potential
problems for MCO correction before the MCO completes its medical record abstractions.
NCQA’s HEDIS® Compliance Audit recommends selecting up to ten difficult to review measures
and obtain copies of at least two complete medical record review tools and charts per measure.
If the State requires fewer than ten measures that rely on medical record data, the EQRO
should conduct the sample review for all medical record-dependent measures. Completing this
step early in the process allows the MCO to address identified issues and resolve them during
data collection.
Based on findings from the ISCA, the EQRO will select at least two measures that include
medical record review, targeting statistical validation measures that are new, complex, and
dependent on the medical record data or those with previously identified issues. For each
measure, the EQRO will request a sample of 30 medical records with positive numerator events
and compare the completed abstraction information to the medical record to determine the rate
of agreement. If the agreement rate is less than 100 percent, the EQRO will assess the degree
of bias. Worksheet 12 contains a description of the medical record review process and the
validation tool.

Step 5: Prepare the MCO Onsite Visit
____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

9

Prior to conducting onsite activities, the EQRO will contact the MCO to:
• Explain the procedures and timeline for performance measure validation activities;
• Communicate the EQRO’s policies and procedures for safeguarding confidential
information and signed confidentiality agreements; and
• Organize the onsite visit to ensure the availability of necessary documentation and
staff.
Potential interviewees include MCO or vendor staff involved in performance measurement who
can provide insight to the MCO’s processes to calculate or report performance measures.
Examples include: the Director of Health/Medical Information Systems, Information System
programmers or operators, Director of Member/Patient Services, Director of Utilization
Management, and the Director of Quality Improvement.
The EQRO should request confirmation of the list and description of State-required performance
measures. The EQRO will provide the MCO a list of documents that may be reviewed prior to or
during onsite activities. Worksheet 3 in Attachment A is a checklist that may be used for this
step.

ACTIVITY 2: ONSITE VISIT ACTIVITIES
Onsite visit activities include follow up on findings from the pre-onsite information system
assessment and validation of the production and reporting of performance measures through
document review or direct observation. During the onsite visit, the EQRO will:
• Confirm, observe and query systems used to produce performance measure results
including membership, medical, pharmacy, provider and other ancillary or
supplemental data sources;
• Investigate and follow-up on issues identified from the ISCA;
• Assess data integration and control procedures for accurate production of the
performance measures;
• Assess data completeness; and
• Confirm processes for calculating and reporting the performance measures.
The EQRO will complete the following steps:
• Review information system underlying performance measurement
• Assess data integration and control for performance measure calculation
• Review performance measurement production
• Conduct Detailed Review of Selected Measures
• Assess the sampling process
• Review preliminary findings and outstanding items
At the conclusion of the onsite visit, the EQRO will share its preliminary findings with the MCO
and identify required follow-up activities.

Step 1: Review Information Systems Underlying Performance Measurement
This activity begins with review of the ISCA during the pre-onsite phase and continues during
the onsite. The EQRO will review the information system components that the MCO uses to
____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

10

produce performance measures, including enrollment, medical, pharmacy, provider, lab, and
other ancillary or supplemental data sources.
Staff Interviews
The EQRO will interview key staff involved in the production of performance measures using
questions tailored to the MCO’s processes for producing performance measures based on
findings from the ISCA.
Primary Source Verification
The EQRO will review the primary source documents, including paper forms and other input to
the MCO systems, and confirm that the information from the primary source matches the
information used for performance measurement. In addition, the EQRO will review the
processes used to input, confirm entry, and identify errors and well as those used to transmit
and track the data through systems. Typical forms the EQRO will review include:
a. Member-initiated enrollment data
b. Hospital claims/encounters
c. Ambulatory claims/encounters
d. Prescription data
e. Practitioner demographic forms
f. Practitioner credentialing forms
g. Claims logs
h. Lab results
System and Process Review
The EQRO will review the MCO’s documentation describing the systems and processes used to
calculate performance measures to confirm they conform to State policies and procedures.
These include systems and processes for collecting, storing, and reporting data. All
documentation received and examined must be recorded.
Observation
The EQRO will observe key MCO processes required for performance measure calculation to
assess data entry and other data manipulations. Examples include:
a. Data entry of membership updates, claims/encounter and practitioner data (Confirm
mandatory fields required and that invalid data elements are identified; for example,
invalid birth dates or invalid service dates);
b. Claims operations including overrides or exceptions;
c. Computer operations and security plans to confirm procedures are followed.
The EQRO will directly observe an operator through the process using an observation guide to
confirm the activities, as well as the process where data are incomplete (e.g., a claim without a
provider identification number).
Data File Review
The EQRO will directly examine data files to confirm the data are stored and processed
according to the documentation provided. Examples of files to review include:
a. Transaction files for clinical services, membership and practitioner changes;
b. Intermediate files created by extracts, queries and analysis applications; and
c. Data repository files.
____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

11

Step 2: Assess Data Integration and Control for Performance Measure Calculation
During this activity, the EQRO will assess:
1. The MCO’s ability to link data from multiple sources; and
2. Whether the MCO has created systems and processes to ensure the accuracy of the
calculated performance measures.
In examining the MCO’s documentation, procedures, and data, the EQRO will:
•
Examine the details of the MCO’s processes for accuracy and completeness to
transfer data from membership, provider, encounter/claims, pharmacy, lab and
other data files into a data repository (or use of other mechanism(s) to consolidate
data) to calculate performance measures and to keep the data until the calculations
of the performance measures have been completed and validated;
•
Examine samples of data from the data repository and transaction files to assess
completeness and accuracy;
•
Investigate the MCO’s processes to consolidate diversified files and extract required
information from a performance measure repository or other data consolidation file;
•
Compare actual results of file consolidations or extracts to those which should have
resulted according to documented algorithms or specifications;
•
Review procedures for coordinating the activities of multiple subcontractors to
ensure accurate, timely, and complete integration of the data into the performance
measure database;
•
Review computer program reports or documentation that reflect these vendor
coordination activities and spot check to verify that no data necessary to
performance measure reporting are lost or inappropriately modified during transfer;
• If the MCO uses a data repository (or data warehouse), evaluate its structure and
format and examine program flow charts and source codes to determine the extent
to which the repository/warehouse enables and has enabled analyses and reports;
• Assess the extent to which proper linkage mechanisms have been employed to join
data from all necessary sources (e.g., identifying a member with a given
disease/condition);
• Examine and assess the adequacy of the documentation governing the performance
measures production process, including MCO production activity logs and MCO staff
review of report runs;
• Review documentation that confirms that prescribed data cutoff dates were followed;
• If appropriate, request that the MCO demonstrate it has retained copies of files or
databases used for performance measure reporting, in the event that results need to
be reproduced;
• Review documentation standards that assure that the performance measure
reporting software program is properly documented with respect to every aspect of
the reporting repository, including building, maintaining, managing, testing, and
report production; and
• Review the MCO’s process and documentation to ensure that it complies with the
MCO standards associated with the performance measure reporting program
specifications, code review, and testing.
Worksheet 4 in Attachment A, Data Integration and Control - Documentation Review Checklist,
lists documents, data, and procedures to assess MCO data integration and control. The EQRO
can supplement the direct examination of information system policies, procedures, and data
____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

12

with interviews of MCO personnel such as the Director of Health/Medical Information Systems,
information system programmers or operators, and selected sub-contractors. Worksheet 5 in
Attachment A provides an interview guide for data integration and control MCO personnel and
Worksheet 6 is a template for recording findings.

Step 3: Review Performance Measure Production
The EQRO will review the MCO’s documentation of all steps undertaken in the production of the
performance measures, including:
• Collection of data from various sources (e.g., membership, enrollment, provider, claims,
or encounter files; medical records; laboratory, pharmacy, or other ancillary records);
• Steps taken to integrate the required data into a performance measure data set or
repository; and
• Procedures or programs to query the data set/repository to identify denominators,
generate appropriate samples, determine numerators, and apply proper algorithms to
the data in order to produce valid and reliable performance measures.
Specifically, the EQRO will perform the following tasks:
• Review performance measurement plans and policies to assess the extent to which they
include:
a. Data file and field definitions;
b. Maps to standard coding if standard codes were not used in original data
collection; and
c. Statistical testing of results, corrections or adjustments made after processing;
• Examine documentation (which may be either a schematic diagram or in narrative form)
of programming specifications to ensure that documentation exists for at least the
following information:
a. A project or measurement plan, including workflow;
b. All data sources, including external supplemental data (whether from a vendor,
public registry, or other outside source) and any prior years’ data (if applicable);
c. Documentation of the original universe of data including record-level patient
identifiers that can be used to validate entire programming logic for creating
denominators, numerators, and samples;
d. Detailed medical record review methods and practices, including the
qualifications of medical record review supervisor and staff; reviewer training
materials; audit tools used, including completed copies of each record-level
reviewer determination; processes for extracting data from electronic medical
records; all case-level critical performance measure data elements used to
determine a positive or negative event or exclude a case from same; and interrater reliability testing procedures and results;
e. Detailed computer queries, programming logic, or source codes used to create
all denominators, numerators, and samples (if applicable to the measure)
including the processes for identifying the population or sample for the
denominator and/or numerator for each measure. If sampling is used, include a
description of sampling techniques and documentation that samples used for
baseline and repeat performance measurements were chosen using the same
sampling frame and methodology; and
f. Documentation of calculation for changes in performance from previous periods
(if applicable) including statistical tests of significance.
____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

13

The EQRO should refer to the specifications for each measure. Worksheet 7 in Attachment A
provides a list of the documentation and interview questions used to assess data processes for
calculating and reporting performance measures and Worksheet 8 provides a template for
recording findings.

Step 4: Conduct Detailed Review of Selected Measures
In this step, the EQRO will determine the extent to which the MCO correctly implemented the
technical specifications to produce accurate performance measure results. The EQRO will
validate the performance measurement denominator(s) by examining whether the MCO used
the appropriate data (including linked data from separate data sets) to identify the entire eligible
population. In some cases, the MCO may have to estimate portions of the population, such as
racial or ethnic subgroups. In such cases, the EQRO must confirm that the method used for
such estimations is valid.
The EQRO will validate the numerator(s) by determining whether the MCO has correctly
identified and assessed qualifying medical events (e.g., diagnoses, procedures, and
prescriptions) to include all appropriate events, while excluding events that do not qualify for
inclusion. These “medical events” may be identified through membership/enrollment data,
claim/encounter data, provider data and/or other supplemental data sources, or data extracted
from medical records.
For each performance measure selected for detailed review, the EQRO will assess the
following:
• The MCO included all members eligible to receive the specified services in the initial
population from which the final denominator was produced. This eligible population
includes members who received the services and those who did not. This same
validation activity applies to provider groups, or other relevant populations identified in
the specifications of each performance measure;
• Programming logic or source codes which identify, track, and link member enrollment
within and across product lines (e.g., Medicaid and CHIP), by age and gender, as well
as through possible periods of enrollment and disenrollment, have been appropriately
applied. This is determined by evaluating:
a. Calculations of continuous enrollment criteria were correctly carried out and
applied (if applicable)
b. Use of appropriate mathematical operations to determine patient age
c. Identification of the variable(s) that code the member’s sex in every file or
algorithm, and the MCO can explain what happens if neither of the required
codes is present;
• The MCO correctly calculated member months and member years, if applicable;
• The MCO has properly evaluated the completeness and accuracy of codes used to
identify medical events, such as diagnoses, procedures, or prescriptions, and that these
codes have been appropriately identified and applied;
• Time parameters are correctly followed (e.g., cut-off dates for data collection, counting
30 calendar days after discharge from a hospital);
• Exclusion specifications or definitions were followed in excluding members from a
denominator. For example, if a measure relates to receipt of a specific service, the
____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

14

•

denominator may need to be adjusted to reflect instances in which the service is
contraindicated; and
Validity of systems or methods used by the MCO to estimate populations.

The EQRO will assess whether the MCO adopted and followed procedures to capture data for
those performance measures that could be easily under-reported due to the availability of
services outside the MCO. (For some measures, particularly those focused on women and
children, the member may have received the specified service outside of the MCO provider
base (e.g., children receiving immunizations through public health services or schools, open
access to family planning services). An extra effort must be made to include these events in the
numerator.
• The codes used by the MCO to identify medical events (such as diagnoses, procedures,
prescriptions, etc.) are complete, accurate, and specific. In particular, the MCO correctly
evaluated these codes when classifying members for inclusion or exclusion in the
numerator.
• The MCO avoided or eliminated double-counting of members or numerator events.
• The MCO mapped any non-standard codes used to standard codes in a manner that is
consistent, complete, and reproducible. The EQRO will assess this through a review of
the programming logic or a demonstration of the program.
• The MCO adhered to time parameters required by the specifications of the performance
measure (i.e., that the measured event occurred during the time period specified or
defined in the performance measure).
• The MCO carried out medical record reviews and abstractions in a manner that
facilitates the collection of complete, accurate, and valid data by ensuring that:
a. Record review staff have been properly trained and supervised for the task
b. Record abstraction tools require the appropriate notation that the measured
event occurred
c. Record abstraction tools require notation of the results or findings of the
measured event (if applicable)
d. Medical record data from electronic sources was accurately extracted according
to measure specifications
• Data included in the record extract files are consistent with data found in the medical
records for a sample of medical records for applicable performance measures. The
process of integrating administrative data and medical record data for the purpose of
determining the numerator is consistent and valid.
Worksheet 9 in Attachment A, Policies, Procedures, Data, and Information Used to Produce
Measures – Review Checklist, lists documents, data, and procedures to assess the accuracy of
the MCO’s performance measures calculations. Worksheet 10 in Attachment A provides an
interview guide, and Worksheet 11 is a template that can be used to record findings.
In addition, for at least two of the performance measures requiring medical record review, the
EQRO should validate the results of the medical record review for 30 enrollees who were found
to meet numerator requirements for each of the two or more measures. Procedures and sample
tools for validating medical record review findings are included in Worksheet 12 of
Attachment A.

Step 5: Assess the Sampling Process (if applicable)
____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

15

The EQRO will determine whether the sample represents the entire eligible population in all
relevant dimensions. The MCO’s sampling method should not exclude any population
subgroups to which the topic area and performance measure apply. For example, when
studying well child care, the sample should not exclude children with special health care needs
whose primary care provider is a specialist other than a pediatrician or family practitioner.
For each measure involving a sample, the EQRO will assess the extent to which:
• Each relevant member or provider had an equal chance of being selected; no enrollees
were systematically excluded from the sampling;
• The MCO followed the State’s specifications for the treatment of sample exclusions and
replacements;
• The MCO examined its sample for bias and, if any bias was detected, the MCO is able
to provide documented corrective action;
• The MCO has policies, procedures, and documentation files from which the samples are
maintained in the event a sample must be re-drawn, or replacements made, the original
population is intact;
• Sample sizes meet the requirements of the performance measure specifications;
• The MCO appropriately handled the documentation and reporting of the measure if the
requested sample size exceeds the population size;
• The MCO properly oversampled in order to accommodate potential exclusions; and
• The MCO followed proper substitution methodology in medical record review (for
measures using the hybrid methodology or medical record review):
a. Substitution applied only to those members who met the exclusion criteria
b. Substitutions were made for properly excluded records and the percentage of
substituted records was documented.
Worksheet 13 in Attachment A, Policies, Procedures, Data, and Information Used to Implement
Sampling – Review Checklist, lists documents, data, and procedures to assess the sampling
process, and Worksheet 14 is a template that can be used to record findings.

Step 6: Preliminary Findings and Outstanding Items
At the conclusion of the onsite visit, the EQRO will communicate preliminary findings to the
MCO, including the identification of any outstanding items for follow-up. The information
communicated during the closing conference will appear in the EQRO’s subsequent preliminary
report. In addition, the EQRO should provide a list of outstanding items prior to the completion
of the preliminary report to allow the MCO the maximum time to resolve identified issues.

ACTIVITY 3: POST-SITE VISIT ACTIVITIES
Post-Onsite activities focus on assessing MCO corrective actions and reporting findings to the
State using the format and timeframes established by the State. The EQRO will:
• Analyze all data and submit a preliminary report to the MCO detailing areas of concern,
suggested methods for correction, and a timeline for the MCO to make corrections;
• Re-validate selected performance measures and the measurement processes the MCO
used to make corrections;
• Re-evaluate the corrected information and submit a report of validation findings to the
State;
____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

16

•
•
•

Determine preliminary validation findings for each measure;
Assess accuracy of MCO’s performance measure reports to the State; and
Submit the validation report to State.

Note that throughout this EQR activity or during any part of an EQR, the State may decide that
immediate corrective action is required.
This protocol provides worksheets and tools in its Attachment. They are referenced in the steps
that follow, and include:
1. Measure validation worksheets for each performance measure to be validated
(Worksheet 11);
2. The completed Medical Record Review Validation Tool (Worksheet 12) for measures
selected for statistical validation of medical record review;
3. Findings regarding the MCO’s data integration and control procedures (Worksheet
6); and
4. Sampling validation findings (Worksheet 14).

Step 1: Determine Preliminary Validation Findings for Each Measure
In the preliminary findings report, the EQRO will document its validation findings, identify areas
of concern, and make suggestions for corrective action or long-term improvement for each of
the performance measures that it validated. The report should indicate which elements of the
MCO performance measures were invalid (if any) and provide the MCO with specific targets for
correction and documentation of the changes necessary to improve the measure production
process. In addition to communicating in writing, the EQRO may participate in meetings with key
MCO personnel responsible for the calculation and reporting of performance measures to assist
the MCO in implementing recommended corrective actions.
Once the EQRO has submitted its preliminary findings to the MCO, the MCO may offer
comments and documentation to support correction of factual errors and omissions in the
EQRO’s preliminary report, or at the discretion of the State, the MCO may recalculate
performance measures based on the findings. The EQRO must then revalidate the revised
performance measure(s) and incorporate the MCO’s comments or revised performance
measure validation findings, and submits its findings to the State. If the State chooses not to
allow re-validation of measures, the recommendations will be reviewed in the following year as
part of the MCO assessment of progress towards recommended improvements.

Step 2: Assess Accuracy of MCO’s Performance Measure Reports to the State
The EQRO will assess whether the MCO reported all measures to the State in the prescribed
manner and form, and verify the reporting of each performance measure by reviewing:
1. Procedures for submitting reports that meet State requirements (e.g., specified format,
supporting documentation, timing); and
2. Documentation that the MCO appropriately implemented procedures for properly
submitting required reports to State.
The EQRO will document the extent to which the MCO reported the calculated performance
measures correctly in its final report to the State.
____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

17

Step 3: Submission of Validation Report to State
The EQRO will always use the State’s decision rules for determining the degree to which each
of the MCO’s reported performance measures is sufficiently valid (i.e., accurate and complete).
The decision rules for compliance should be uniform across MCOs within the State. The final
report should follow the State’s required format, and include the following elements:
a. A list of measures for validation;
b. A description of the validation activities including:
• The EQRO’s team members list;
• Description of the validation strategy and considerations;
• Description of the technical methods of data collection and analysis used by the
EQRO;
• List of interviewees; and
• Any other facts relevant to the onsite process;
c. Worksheets and tools as supporting documentation to the report;
d. Details, analysis and conclusions drawn from the validation process for each
performance measure, including actual results of the performance measures (not just
that results were validated) and findings related to the MCO’s information systems
capabilities and data integration; and
e. The validation findings for each performance measure.

END OF PROTOCOL

____________________________________________________________________________________
EQR Protocol 2
Validation of Measures Reported by the MCO
September 2012

18