CMS-R-305 Validating Encounter Data

OMB Approval No.
0938-0786

VALIDATING PERFORMANCE
MEASURES

A protocol for use in Conducting Medicaid External Quality
Review Activities

Department of Health and Human Services
Centers for Medicare & Medicaid Services

Final Protocol
Version 1.0
May 1, 2002

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid
OMB control number. The valid OMB control number for this information collection is 0938-0786. The time required to complete this
information collection is estimated to average 1,591 hours per response for all activities, including the time to review instructions, search existing
data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the
time estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports Clearance Officer,
Baltimore, Maryland 21244-1850.

1

VALIDATING PERFORMANCE MEASURES
I.

PURPOSE OF THE PROTOCOL

This protocol specifies activities to be undertaken by an external quality review organization
(EQRO) 1 in order to validly:
1.

Evaluate the accuracy of Medicaid performance measures reported by, or on behalf of, a
Managed Care Organization (MCO) or a Prepaid Inpatient Health Plan (PIHP); and

2.

Determine the extent to which Medicaid-specific performance measures calculated by an
MCO/PIHP (or by an entity acting on behalf of an MCO or PIHP) followed
specifications established by the State Medicaid agency (the State) for the calculation of
the performance measure(s).

II.

ORIGIN OF THE PROTOCOL

This protocol was derived from protocols and tools commonly used in the public and privatesectors for auditing performance measures. These include:
-

-

the National Committee for Quality Assurance’s (NCQA) 1999 tools used by the
Health Plan Employer Data and Information Set (HEDIS)® publication: Volume
5, HEDIS Compliance Audit™ Standards and Guidelines;
tools use by the Island Peer Review Organization (IPRO) in their audits of
HEDIS measures for Medicare; and
documents from the MEDSTAT Group, Inc., published in conjunction with work
performed for the Centers for Medicare & Medicaid Services (CMS) (formerly
the Health Care Financing Administration (HCFA) in 1997 and 1998.

A review of the tools found that, while there are differences, these documents had much in
common.

1

It is recognized that a State Medicaid agency may choose an organization other than an EQRO as defined
in Federal regulation to validate Medicaid performance measures submitted by or on behalf of an MCO/prepaid
inpatient health plan (PIHP). However, for convenience, in this protocol we use the term, “external quality review
organization” (EQRO) to refer to any organization that validates performance measures.

2

Both NCQA’s and IPRO’s documents address the validation of HEDIS measures only. They
assess:
-

-

the structure and integrity of the MCO’s/PIHP’s underlying information system
(IS);
MCO/PIHP ability to collect valid data from various internal and external
sources;
vendor (or subcontractor) data and processes, and the relationship of these data
sources to those of the MCO/PIHP;
MCO/PIHP ability to integrate different types of information from disparate data
sources into a data repository or set of consolidated files for use in constructing
MCO/PIHP performance measures; and
documentation of the MCO’s/PIHP’s processes to: collect appropriate and
accurate data, manipulate those data through programmed computer queries,
internally validate the results of the operations performed on the data sets, follow
specified procedures for calculating the specified performance measures, and
report the measures appropriately.

The MEDSTAT publications focus primarily on validation of encounter-level data, and the use
of those data in Medicaid MCO performance measures, regardless of whether the performance
measures are based on the NCQA Medicaid HEDIS measures or have been developed by other
groups or organizations. However, the MEDSTAT publications do not provide detailed
instructions or guidelines that an EQRO might use to validate the MCO/PIHP performance
measures once the encounter data are validated.
The protocol presented here is consistent with the approaches used in the IPRO and NCQA
documents, but is designed with a MEDSTAT-like approach in that it describes how to validate
all performance measures - HEDIS measures as well as non-HEDIS measures. It varies from the
IPRO and NCQA protocols in that certain components of performance measure validation may
be performed as a part of this protocol or accomplished through some other mechanism(s) used
by the State. For example, as part of this protocol, an assessment of the MCO’s/PIHP’s IS is
required. This IS assessment may be conducted as a part of this protocol by the EQRO validating
the performance measures, or the EQRO may review an assessment of the MCO=s /PIHP=s IS
conducted by another party.

III.

OVERVIEW OF THE PROTOCOL

The protocol assumes that the State has specified:
-

the performance measures to be calculated by MCOs/PIHPs;
the specifications to be followed in calculating these measures; and
the manner and mechanisms for reporting these measures to the State.
3

Protocol activities address:
1.

Review of the data management processes of the MCO/PIHP;

2.

Evaluation of algorithmic compliance (the translation of captured data into actual
statistics) with specifications defined by the State; and

3.

Verification of either the entire set or a sample of the State-specified performance
measures to confirm that the reported results are based on accurate source information.

The protocol consists of three phases of activities: Pre-Onsite, Onsite, and Post-Onsite activities.
For each of these phases, the protocol specifies outcomes or objectives and lists the activities to
be performed. Methods of evaluation are suggested and tools and worksheets are provided
throughout the protocol and as attachments to the protocol.
Pre-Onsite activities involve:
1.

Communicating with the State to ensure that the EQRO understands:
-

the measures to be validated (i.e., the entire set versus a subset of those calculated
by the MCO/PIHP); and

-

the methodology(ies) the State requires the MCO/PIHP to follow when
calculating and reporting the performance measures;

2.

Preparing MCOs/PIHPs for onsite activities; and

3.

Either conducting an assessment, or reviewing the results of a prior assessment, of the
MCO’s/PIHP’s underlying IS.

Onsite activities focus on: 1) following up on IS findings identified in the Pre-Onsite activities
as being potentially problematic or in need of further review or clarification; and 2) validating
the production and reporting of performance measures through observation of documentation or
procedures. These activities include:
1.

Reviewing and assessing the procedures the MCO/PIHP has in place for collecting and
integrating medical, financial, member and provider information, covering both clinical
and service-related data, from internal and external sources;

2.

Evaluating processes used by the MCO/PIHP to produce performance measures; e.g.,
sampling, calculating denominators and numerators; and

3.

Evaluating the MCO’s/PIHP’s processes for reporting required performance measures to
the State.
4

To accomplish these activities, the EQRO reviews policy and procedure manuals and documents,
observes required activities, and conducts interviews with key MCO/PIHP staff such as the
Director of Health/Medical Information Systems, IS programmers or operators, Director of
Member/Patient Services, Director of Utilization Management, and the Director of Quality
Improvement.
Post-Onsite activities focus on the analysis of the data and information obtained through PreOnsite and Onsite activities, and submission of the validation report and supporting
documentation to the State following the format and time frames established by the State. These
activities include:
1.

Evaluating gathered information and preparing a report of preliminary findings;

2.

Submitting reports of preliminary findings identifying areas of concern to the
MCO/PIHP;

3.

If the State provides the MCO/PIHP with the opportunity to recalculate performance
measures based on EQRO findings, re-reviewing selected performance measurement
processes;

4.

Evaluating gathered information and preparation of findings for the State; and

5.

Submitting reports to the State.

The protocol identifies alternative approaches to determining the extent to which the MCO/PIHP
has complied with requirements for calculating and reporting performance measures. In one
option, the EQRO would submit a summary of its findings along with the completed protocol
assessment tools to the State as supporting documentation, but without a validation designation
for individual performance measures. Based on the information submitted by the EQRO, the
State would make a determination of the extent to which the MCO/PIHP has adequately
calculated and reported the specified performance measures. Alternatively, the EQRO could
apply clearly defined decision rules established by the State and specify a validation finding for
each performance measure.

5

IV.

PROTOCOL ACTIVITIES

PRE-ONSITE ACTIVITIES
Objectives for Pre-Onsite Activities:
The EQRO will:
- understand the technical specifications for each of the performance measures required by the
State;
- understand the State’s requirements for performance measure reporting by the MCO/PIHP to
the State (e.g., report template, electronic submission format, etc.); and
-

conduct and review an assessment (or review the results of a previously
conducted assessment) of the MCO’s/PIHP’s IS.

PRE-ONSITE ACTIVITY 1:

Review the State’s requirements for MCO/PIHP
performance measurement and reporting.

The EQRO will need to obtain from the State a list of all performance measures that the State
requires the MCO/PIHP to produce and ascertain, in consultation with the State, whether the
validation activities are to include all such measures or a subset of those measures. The EQRO
will also need to obtain the State’s instructions (specifications) on how the MCO/PIHP is to
calculate each performance measure.
The specific performance measures that a State requires its Medicaid MCOs/PIHPs to report will
depend on a number of factors unique to each State. If a State chooses to use a set or subset of
established standardized plan-level performance measures, there are a number of options from
which to choose. These include the NCQA’s HEDIS measures, measures identified by the
Foundation for Accountability (FACCT), measures found in the Agency for Healthcare Research
and Quality’s (AHRQ’s) CONQUEST database, or measures suggested by MEDSTAT in its
publication, A Guide for States to Assist in the Collection and Analysis of Medicaid Managed
Care Data 2 . In addition, States with the resources and expertise to develop and test the detailed
specifications necessary for valid and reliable performance measures may establish their own
performance measures. Regardless of the type or number of performance measures chosen by the
State, the EQRO must understand the State’s specifications (e.g., sampling guidelines,
instructions for calculating numerators and denominators) for each performance measure, as well
as the State’s instructions to the MCO/PIHP for reporting the required performance measures to
the State.
2

Prepared under CMS Contract #500-92-0035. December 1998.

6

Four basic data collection methodologies typically are used to produce MCO/PIHP performance
measures: 1) use of administrative data, 2) review of medical records, 3) use of administrative
data together with medical record review (commonly called the “hybrid” methodology), and 4)
use of surveys.
Use of administrative data requires the MCO/PIHP to access data contained in its management
information system(s) to calculate both the denominator and numerator of a given performance
measure. Such data includes encounter or claims data (transaction data) as well as other
automated enrollee and provider information. The rate that is reported is based on information
found solely in these administrative data sources.
Calculating performance measures from medical record review requires the visual inspection of
the medical records of a sample of MCO/PIHP enrollees (denominator) to determine if each
enrollee received the service(s) in question (typically, this is the numerator of the performance
measure). Because medical record reviews are time-consuming and costly, most developers and
users of performance measures are attempting to use, to the extent feasible, performance
measures that can be calculated from administrative data. If medical record review is
unavoidable, the less costly and less burdensome “hybrid” methodology can be used.
The hybrid methodology combines the use of administrative data with a review of medical
records. The denominator of the measure is first identified using administrative data for a sample
of eligible members. The numerator is then determined using data from both administrative and
medical record reviews. Typically, the MCO/PIHP will first query its administrative data for
evidence of the numerator event for all individuals included in the denominator sample. For any
member of the sample who is missing an administrative notation that the numerator service was
received, the medical record is reviewed.
Finally, surveys also are used to produce MCO/PIHP performance measures. Surveys may
include information collected directly from enrollees, relatives, primary caregivers of enrollees,
or providers of healthcare services. Administration and validation of surveys are complex
subjects and are discussed in separate EQR protocols.
States may require or allow MCOs/PIHPs to report performance measures to the State in
different ways. A State may choose to have MCO/PIHP performance measures reported to it in
an electronic format, such as a comma-delimited, ASCII file; or it may establish a set of
electronic reporting “shells” that MCOs/PIHPs fill out and send to the State, with attestations of
the accuracy of the information. States could also allow hardcopy submission of calculated
performance measures.
States will also determine the timing of the submission of the calculated performance measures.
Typically, States require performance measures to be calculated and submitted annually. The
annual submissions may be timed to coincide with the end of the State fiscal year, the calendar
year, or another reporting cycle, such as that used by NCQA for HEDIS submissions. The EQRO
needs to understand the expected dates and format for MCO/PIHP reporting.
7

To facilitate its onsite validation of measures, the EQRO should create a “List of Performance
Measures to be Calculated by the MCO/PIHP” (such as that shown in TABLE 1) in order to
understand the measures required by the State, the possible methods the MCO/PIHP may use to
collect them, and the reporting frequencies and format mandated by the State.

TABLE 1
List of Performance Measures to be Calculated by the MCO/PIHP
(EXAMPLE)
METHOD FOR CALCULATING PERFORMANCE
MEASURE

SAMPLE MEASURES
Administrative
Data

Medical
Record
Review

Hybrid

Survey

Reporting
Frequency and
Format

The table should have a row for
each measure to be calculated
and reported by the MCO/PIHP,
as illustrated below:
Childhood immunization rate
Adolescent immunization rate
Percentage of enrollees with at least
one PCP visit
Lead screening rate
Breast cancer screening rate
Initiation of prenatal care
Comprehensive diabetes care
Availability of language
interpretation services
Follow-up after hospitalization for
mental illnesses
Women’s chlamydia screening rate
Rate of adverse asthma events

8

For each measure in the EQRO-created “List of Performance Measures to be Calculated by the
MCO/PIHP,” the EQRO also should create a separate performance measure validation
worksheet that contains the specifications and components of each performance measure that is
to be validated, including: 1) specifications for the eligible population for the measure; 2) data
collection methodology; 3) sampling methodology (if used); 4) denominator calculations; 5)
numerator calculations; and 6) calculated and reported rates. A generic “Performance Measure
Validation Worksheet” is found below (TABLE 2), containing placeholders for the components
to be validated and the elements to be audited. The EQRO should customized this or a similar
worksheet to include the specifications (defined by the State) for each performance measure to
be reported by the MCO/PIHP. For example, if the measure is Breast Cancer Screening
(following the HEDIS specifications), the EQRO would replace the general “age and sex”
categories in the denominator portion of the tool with the particular age and sex specifications
associated with that measure, i.e., females between the ages of 52-69. Using a performance
measure validation worksheet will improve the efficiency of the validation work performed on
site. An example of a completed Performance Measure Validation Worksheet is included as
ATTACHMENT I.

9

TABLE 2
GENERIC PERFORMANCE MEASURE VALIDATION WORKSHEET
For each performance measure to be validated (as listed in TABLE 1 of this Pre-Onsite Activity),
adapt the generic table shell below to create a validation worksheet for the measure. [An example of a
completed Performance Measure Validation Worksheet is included as ATTACHMENT I].
PERFORMANCE MEASURE { Insert name of performance measure}
Validation
Component
Audit Element
Documentation

Denominator

Numerator

Sampling

Meets Validation
Requirements
Yes No
N/A

Appropriate and complete measurement plans and programming
specifications exist that include data sources, programming logic,
computer source code.
Data sources used to calculate the denominator (e.g., claims files, medical
records, provider files, pharmacy records) were complete and accurate.
Calculation of the performance measure adhered to the specifications for
all components of the denominator of the performance measure (e.g.,
member ID, age, sex, continuous enrollment calculation, clinical codes
such as ICD-9, CPT-4, DSM-IV, member months calculation, member
years calculation, adherence to specified time parameters).
Data sources used to calculate the numerator (e.g., member ID, claims
files, medical records, provider files, pharmacy records, including those for
members who received the services outside the MCO/PIHP=s network) are
complete and accurate.
Calculation of the performance measure adhered to the specifications for all
components of the numerator of the performance measure (e.g., clinical
codes such as ICD-9, CPT-4, DSM-IV, pharmacy data, relevant time
parameters such as admission/discharge dates or treatment start and stop
dates, adherence to specified time parameters, number or type of provider).
If medical record abstraction was used, documentation/tools were adequate.
If hybrid method was used, the integration of administrative and medical
record data was adequate.
If hybrid method or solely medical record review was used, the results of
the medical record review validation substantiate the reported numerator.
Sample was unbiased.
Sample treated all measures independently.
Sample size and replacement methodologies met specifications.

Reporting

State specifications for reporting performance measures were followed.

10

ASSIGNING A VALIDATION FINDING TO THE MEASURE
The validation finding for each measure is determined by the magnitude of the errors detected for the audit elements,
not by the number of audit elements determined to be “NOT MET.” Consequently, it is possible that an error in a
single audit element may result in a designation of “NV” because the impact of the error biased the reported
performance measure by more than “x” percentage points. Conversely, it is also possible that several audit element
errors may have little impact on the reported rate and, thus, the measure could be given a designation of “SC.” The
following is a list of validation findings and their corresponding definitions:
FC

=

Fully Compliant
Measure was fully compliant with State specifications.

SC

=

Substantially Compliant
Measure was substantially compliant with State specifications and had only minor deviations that
did not significantly bias the reported rate.

NV

=

Not Valid
Measure deviated from State specifications such that the reported rate was significantly biased.
This designation is also assigned to measures for which no rate was reported, although reporting of
the rate was required.

NA

=

Not Applicable
Measure was not reported because MCO/PIHP did not have any Medicaid enrollees that qualified
for the denominator.

AUDIT DESIGNATION

- Assigning a validation finding to a measure is discussed in Post-Onsite Activity 1. This material is included
here because it should be part of a performance measure validation worksheet.
PRE-ONSITE ACTIVITY 2:

Prepare the MCO/PIHP for EQRO Onsite Activities.

Prior to conducting onsite activities, the EQRO will contact the MCO/PIHP in order to:
- explain the procedures and time line for performance measure validation
activities;
- request identification of personnel within the MCO/PIHP who will be responsible
for responding to EQRO requests for documentation or information, as well as
scheduling activities and interviews; and
- Communicate the EQRO’s policies and procedures with respect to safeguarding
confidential information.
An introductory letter to the MCO/PIHP should discuss the above issues and explain the
EQRO=s potential need to interview MCO/PIHP personnel, so that interviewees are prepared in
11

terms of time and information. Potential interviewees include any MCO/PIHP or vendor staff
whose areas of expertise or responsibility relate to performance measurement and whose insights
might improve the EQRO’s understanding of MCO/PIHP processes to calculate or report
performance measures. These include, for example: the Director of Health/Medical Information
Systems, IS programmers or operators, Director of Member/Patient Services, Director of
Utilization Management, and the Director of Quality Improvement.
Also, the EQRO will provide to, or request from, the MCO/PIHP four other types of information
in preparation for its onsite activities:
1.
2.
3.
4.

A list and description of all State-required performance measures calculated by or on
behalf of the MCO/PIHP;
A list of all enrollees (or enrollee identifiers) included in the numerators of performance
measures calculated wholly or in part by medical record review;
A list of documents that the EQRO may potentially review during onsite activities; and
Background information on the MCO’s/PIHP’s IS.

1. List of performance measures calculated by the MCO/PIHP. This list of performance
measures calculated by the MCO/PIHP (TABLE 3) is similar to the list completed by the EQRO
during Pre-Onsite Activity 1 (TABLE 1). However, while the TABLE 1 list was prepared by the
EQRO to familiarize itself with the State’s requirements for performance measures, TABLE 3 is
sent to the MCO/PIHP by the EQRO for the MCO/PIHP to complete. The MCO/PIHP is to
insert into the table, next to each performance measure listed in the table, information on the
methods the MCO/PIHP used to calculate the performance measures required by the State. This
is especially important for those measures for which the MCO/PIHP has a choice of methods to
use for their calculation; e.g., administrative, medical record review, or hybrid data collection
methodologies. The EQRO should send to the MCO/PIHP the same list of measures contained in
TABLE 1, but with a modified title and instructions (as illustrated in TABLE 3, below) to reflect
that the MCO/PIHP is to complete the table and return it to the EQRO.

12

TABLE 3
List of Performance Measures Calculated by the MCO/PIHP - Example
Instructions to MCOs/PIHPs: For each measure the State requires you to report (in column 1),
indicate the method(s) your MCO/PIHP used to produce it by checking columns 2 - 5, as
appropriate. In column 6 note the reporting frequencies (e.g., quarterly, annually) and format
(e.g., paper report, electronic medium) your MCO/PIHP has used (or expects to use) to report to
the State. Return this table to (name of EQRO) by (date), so that this information may be
reviewed prior to our site visit to validate your MCO’s/PIHP’s performance measures.
(1)
Measure
{Examples}

(2)
Administrative

(3)
Medical
Record
Review

(4)
Hybrid

(5)
Survey

(6)
Reporting
Frequency
and Format

The table should contain a
row for each measure to be
calculated and reported by the
MCO/PIHP.
Childhood immunization rate
Adolescent immunization rate
Percent of enrollees with at
least one PCP visit
Lead screening rate
Breast cancer screening rate
Initiation of prenatal care
Comprehensive diabetes care
Language interpretation
services - availability
Follow-up after hospitalization
for mental illnesses
Women’s chlamydia screening
rate
Rate of adverse asthma events

13

2. A list of all enrollees (or enrollee identifiers) included in the numerators of all measures
calculated in part or wholly from medical record review. For each of at least three
performance measures which the MCO/PIHP calculated either entirely by medical record review
or by the hybrid methodology, the EQRO will review, onsite, 30 medical records found to meet
numerator requirements. The purpose of this review is to verify the accuracy of the medical
record review conducted by each MCO/PIHP.
To provide sufficient time for each MCO/PIHP to gather the required medical record
documentation, the MCO/PIHP will need to identify to the EQRO, prior to the EQRO=s onsite
visits: 1) all performance measures calculated through medical record review or the hybrid
methodology (obtained by completing TABLE 3), and 2) for measures which used medical
record review or the hybrid methodology and selected by the EQRO, a list of enrollees included
in the numerator for each measure as a result of positive findings through medical record review.
From this list, the EQRO will select 30 members for each performance measure. The MCO/PIHP
will then be asked to make available the medical records or copies of medical records for these
enrollees at the time of the onsite visit. In cases where there are fewer than 30 numerator
positives, the EQRO will review all records for that measure.
3. List of potential validation documents and processes. The List of Potential Validation
Documents and Processes (ATTACHMENT II) identifies documents and information
concerning the MCO’s/PIHP’s data sources and processes that the EQRO may review during the
course of the validation activities. This list is intended to assist the MCO/PIHP in preparing for
the validation audit.
4. Information Systems Capabilities Assessment Tool (ISCA). The EQRO will send an ISCA
to the MCO/PIHP, to be completed and returned to the EQRO prior to the onsite visit. The ISCA
consists of questions and requested documentation to provide the EQRO with background
information on the MCO’s/PIHP’s policies, processes, and data needed for the onsite validation
activities. The ISCA is discussed in detail, in Pre-Onsite Activity 3. A recently conducted ISCA
by another party can be used.
PRE-ONSITE ACTIVITY 3:

Assess the integrity of the MCO’S/PIHP’s information
system.

Complete and accurate data is key to valid and reliable performance measurement. If these two
data characteristics are not maintained, then calculated measures become biased, and their
validity jeopardized. Therefore, prior to validating individual performance measures, the EQRO
must first assess the integrity of the MCO’s/PIHP’s IS and the completeness and accuracy of the
data produced by that system.
Methods of Evaluation
14

Prior to conducting the onsite visit, the EQRO should send to the MCO/PIHP an ISCA such as
that located in Appendix Z. The ISCA asks questions of and requests documentation from the
MCO/PIHP in order to provide information on the MCO’s/PIHP’s IS policies and procedures to
help focus onsite validation activities. The ISCA found in Appendix Z corresponds to the key
objectives identified in this protocol. The first section of the ISCA provides general background
information on the MCO/PIHP. Subsequent sections address the structural components of the IS,
focusing on the collection of administrative, encounter, and clinical data, and the consolidation
or coordination of those data files for use in performance measurement and quality improvement
activities.
The ISCA also requests information from the MCO/PIHP concerning the conduct and timing of
any other recent, independent, documented assessment of its IS. An assessment may already
have been conducted by the State itself or by another entity. IS assessment could have been
performed as a component of validating encounter data or determining compliance with
Medicaid standards pertaining to MCO/PIHP ISs. If the MCO/PIHP has not had an IS capability
assessment completed, or has not had one completed within a time frame that meets State
specifications 3 , the EQRO will conduct an IS assessment as part of this protocol, using an
information systems assessment tool, such as that in Appendix Z. Alternatively, if the
MCO/PIHP recently had an independent assessment of its IS, the EQRO could review the results
of this prior assessment.
The EQRO should assess the MCO’s/PIHP’s IS using questions and approaches such as those
contained in Appendix Z, or review the results of a recent IS assessment consistent with the
content in Appendix Z. This will ensure that auditors are familiar with the strengths and
weaknesses of the MCO’s/PIHP’s IS. As the EQRO reviews the IS assessment report, it should
pay close attention to the strengths and weaknesses of the MCO’s/PIHP’s IS with respect to the
types of data frequently used in MCO/PIHP performance measures, such as data on:
membership/enrollment, providers, claims/encounters, laboratory and pharmacy services, and
medical record data. Some of the characteristics commonly associated with these data elements
that may affect performance measures are:
-

Membership/Enrollment Data. Elements of the membership or enrollment
database will vary by MCO/PIHP. However, for the purposes of MCO/PIHP

3

Each State will determine the frequency with which it wants an MCO’s/PIHP’s IS capability assessment to
take place (thereby determining the length of time such an assessment is valid). On the one hand, the process is timeand resource-intensive, so limiting the burden on the MCO/PIHP should be a factor in the determination. On the
other hand, IS technology changes rapidly, so the State should ensure that changes to an MCO’s/PIHP’s IS are
assessed frequently enough to ensure that the structure and function continue to be adequate for the State-required
tasks.

15

performance measurement, the membership or enrollment database should
capture at least the following information:
age/date of birth.
enrollment and/or termination dates. (Note: The MCO’s/PIHP’s data
system should be able to track multiple enrollment and termination dates).
primary care provider (e.g., name, provider identification number).
member identification number such as the member’s social security
number, MCO- or PIHP-designated number, State-issued Medicaid
number, CMS-issued Medicare number. (Note: Be aware of cases in
which more than one member may exist under the same identification
number within the system; or in which the same member may exist under
more than one identification number within the system; or in which a
member=s identification number may change through re-enrollment, name
change, or switch in product-line coverage).
The EQRO also should be aware of whether the MCO/PIHP has processes in place to
periodically ensure that enrollment/membership data are current and accurate,
particularly at the time it runs its source code/computer programs to identify
denominators for MCO/PIHP performance measures.
Further, the EQRO should be aware of changes in the MCO’s/PIHP’s membership data
systems that might affect the production of the MCO/PIHP performance measures. Major
changes, upgrades or consolidations within the system, or acquisitions/mergers with other
MCOs/PIHPs may impact the accuracy or completeness of any of the data elements,
which, in turn, may impact the validity of the reported measures.
-

Provider Data. Elements of the provider data set should typically include:
Designation as a primary care physician and/or providers’ specialty.
Provider identification number, such as a Tax ID number, or MCO- or
PIHP-designated number. (Note: Though it may be less common to see
duplication of provider numbers within a provider database than
duplication of member identifications within a membership/enrollment
database, the EQRO should be aware of any circumstances in which more
than one provider can exist with the same identification number within the
system, or circumstances in which the same provider may have more than
one identification number within the system).
Providers with more than one office location.
Providers with closed panels (i.e., provider availability).
Provider start and termination dates.
Provider certification data such as licensure, provider
residency/fellowship, date and specialty of Board Certification status.
16

The EQRO should be aware of whether the MCO/PIHP has processes in place to
periodically ensure that provider data are current and accurate for all types of
providers (individual providers, provider groups, provider networks, contracted
vendors). This becomes particularly important at the time the MCO/PIHP runs its
source code/computer programs to identify elements of MCO/PIHP performance
measures.
Further, the EQRO should be aware of changes in the MCO’s/PIHP’s provider
data systems that might affect the production of the performance measures. Major
changes, upgrades or consolidations within the system, or acquisitions/mergers
with other MCOs/PIHPs may impact the accuracy or completeness of any of the
data elements, which, in turn, may impact the validity of the reported measures.
-

Claims Data and Encounter Data. Claim/encounter data should cover all types
of services offered by the MCO/PIHP, such as: behavioral health, family
planning, home health care, hospital, laboratory, pharmacy, primary care,
radiology, specialty care, vision care. These data typically include the following
elements:
- Patient ID
- Sex
- Date of birth
- Last date of service
- Primary diagnosis
- Primary procedure
- Revenue codes
- Provider specialty

- Name
- Age
- First date of service
- Place of service
- Secondary diagnosis
- Secondary procedure
- Provider ID
- Discharge status

For each type of claim/encounter data captured, the EQRO should be aware of: 1)
the total number of diagnosis and procedure codes that can be captured by the
system; 2) whether or not principal or secondary diagnosis or procedure codes can
be accurately distinguished in the system; and 3) the maximum number of digits
or characters the system captures for each type of claim/encounter. For many
MCO/PIHP performance measures, the accuracy and validity of the measure may
be adversely affected if the MCO’s/PIHP’s IS is unable to collect and/or
differentiate among a sufficient number of codes.
The various coding systems and forms used by the MCO/PIHP and its vendors to
capture clinical information through its claims and encounter databases are
17

relevant to validating MCO/PIHP performance systems. Coding systems are
formal, standardized approaches (such as ICD-9, CPT-4, DSM-IV, revenue
codes, or internally developed codes) to categorize types of encounters and
procedures by data elements such as inpatient and ambulatory diagnoses and
procedures for medical, surgical, or mental health/substance abuse
encounters/claims. Note that internally-developed codes may be particularly
problematic. The EQRO should understand how the MCO’s/PIHP’s IS translates
or maps these codes back to standard codes for MCO/PIHP performance measure
reporting, and how it ensures the accuracy of these translation processes.
-

Medical Record Data. In cases where medical records are accessed to obtain
information for calculating MCO/PIHP performance measures, the EQRO should
be aware of how the MCO/PIHP retrieves information from medical records. For
example, the training and tools that medical record review staff receive may affect
the accuracy and completeness of the data retrieval and inter-rater reliability. A
second area of concern is how medical record data is entered into any database
that will be used to produce the performance measures.

-

Pharmacy and Laboratory Data. A key issue commonly encountered with
pharmacy and laboratory data for Medicaid managed care MCOs/PIHPs is that
these services are frequently contracted out to a variety of providers. Ideally,
pharmacy data will use standardized codes for prescription drugs such as those
promulgated by the National Council for Prescription Drug Programs (NCPDP),
and laboratory services will use a similar, nationally recognized system of coding.
However, the diverse nature of the size, type, and ownership of pharmacy and
laboratory providers should lead the EQRO to anticipate wide variations in the
use of standardized coding and a multitude of unique “home grown” codes. These
non-standard coding schemes require that the MCO/PIHP have a system to
develop crosswalks among these different codes in order to store the necessary
information in its performance measure database. As with the assessment of the
claims/encounter data systems, the EQRO should understand not only the
MCO’s/PIHP’s system of mapping non-standard pharmacy and lab codes to
standardized codes, but the mechanism the MCO/PIHP uses to ensure the
accuracy of these translation processes.
If pharmacy or laboratory data are not collected through an administrative or
claims database, pharmacy or lab data may be present in medical records.
However, relying on medical records to supply pharmacy or laboratory data is
problematic because of obstacles such as non-standard coding and terminology
and poor coordination of records and record linkages between primary care and
specialist providers. The EQRO should be aware of these issues and question
providers on the reliability of medical record data and pharmacy data as
appropriate.
18

In addition, for many MCO/PIHP performance measures, the IS will need to be able to link these
different sources of data. For example, in order to identify enrollees with diabetes, an
MCO/PIHP may have to combine diagnosis code data from inpatient or ambulatory encounters
(not all ongoing conditions are reported at every encounter) with pharmacy data, lab data, and/or
a disease registry if one exists. To determine whether these diabetic enrollees have received a
retinal examination from an ophthalmologist or optometrist within the previous year, the
MCO/PIHP would have to link procedure code data from either encounter forms, medical
records, or claims with information about the specialty of the providers that performed the
examinations for these members.
The EQRO will analyze the results of the assessment of the MCO’s/PIHP’s IS and determine the
implications of the findings for the ability of the MCO/PIHP to calculate the performance
measures specified by the State. The EQRO will evaluate MCO/PIHP answers against IS
capabilities necessary to accurately and completely calculate and report the specific MCO/PIHP
performance measures mandated by the State, and will identify any problem areas or items in
need of clarification. Where an answer seems incomplete, or indicates an inadequate process, the
EQRO notes this issue for follow-up and further review during the onsite activities. This will
help the onsite validation activities focus on the areas most likely to be an issue in the validation
process. In addition, knowledge gained from the ISCA provides a knowledge base for effective
interviews with key MCO/PIHP staff.
ONSITE ACTIVITIES
Objectives for Onsite Activities:
The EQRO will evaluate the extent to which the MCO/PIHP has:
-

adequate data integration and control procedures for accurate production of the
State-specified performance measures;

-

complete and accurate documentation of data and processes used to calculate and
report the State-specified performance measures; and

-

correctly implemented appropriate processes for calculating and reporting the
State-specified performance measures.

ONSITE ACTIVITY 1:

Assess data integration and control.

19

In the last activity (Pre-Onsite Activity 3), the EQRO examined background information on the
capability of the MCO’s/PIHP’s IS to collect and integrate valid data from sources internal and
external to the MCO/PIHP. This onsite activity further assesses: 1) the MCO’s/PIHP’s ability to
link data from multiple sources in order to calculate the State-mandated performance measures;
and 2) whether the MCO/PIHP has used these abilities in a manner that ensures the accuracy of
the calculated performance measures. This assessment will be accomplished through:
1.
2.

Review of documentation, procedures, and data pertaining to the MCO’s/PIHP’s IS, and
Interviews of MCO/PIHP personnel with knowledge of the MCO’s/PIHP’s IS and its
application to performance measurement.

ATTACHMENT III, IS Data Integration and Control - Documentation Review Worksheet lists
documents, data, and procedures to be examined to assess MCO/PIHP data integration and
control. EQROs should use a worksheet such as ATTACHMENT III to document their findings.
In examining the MCO’s/PIHP’s documentation, procedures and data, the EQRO should:
1.

2.
3.

4.
5.

6.

7.

8.

Examine for accuracy and completeness the details of the MCO’s/PIHP’s processes to
transfer data from membership, provider, encounter/claims, and other data files into a
data repository (or use of other mechanism(s) to consolidate data) to calculate
performance measures and to keep the data until the calculations of the performance
measures have been completed and validated.
Examine samples of data from the data repository and transaction files to assess
completeness and accuracy.
Investigate the MCO’s/PIHP’s processes to consolidate diversified files and extract
required information from a performance measure repository or other data consolidation
file.
Compare actual results of file consolidations or extracts to those which should have
resulted according to documented algorithms or specifications.
Review procedures for coordinating the activities of multiple subcontractors to ensure
accurate, timely, and complete integration of the data into the performance measure
database.
Review computer program reports or documentation that reflect these vendor
coordination activities and spot check to verify that no data necessary to performance
measure reporting are lost or inappropriately modified during transfer.
If the MCO/PIHP uses a data repository (or data warehouse), evaluate its structure and
format and examine program flow charts and source codes to determine the extent to
which the repository/warehouse enables and has enabled analyses and reports.
Assess the extent to which proper linkage mechanisms have been employed to join data
from all necessary sources (e.g., identifying a member with a given disease/condition).

20

9.

10.
11.

12.

13.

Examine and assess the adequacy of the documentation governing the performance
measures production process, including MCO/PIHP production activity logs, and
MCO/PIHP staff review of report runs.
Review documentation that confirms that prescribed data cutoff dates were followed.
If appropriate, request that the MCO/PIHP demonstrate it has retained copies of files or
databases used for performance measure reporting, in the event that results need to be
reproduced.
Review documentation standards that assure that the performance measure reporting
software program is properly documented with respect to every aspect of the reporting
repository, including building, maintaining, managing, testing, and report production.
Review the MCO’s/PIHP’s process and documentation to ensure that it complies with the
MCO/PIHP standards associated with the performance measure reporting program
specifications, code review, and testing.

In addition, as needed, the EQRO should supplement the direct examination of IS policies,
procedures, and data with interviews of MCO/PIHP personnel. MCO/PIHP personnel who can
potentially provide helpful information include: the Director of Health/Medical Information
Systems, system programmers or operators, and selected sub-contractors. An Interview Guide
and suggested questions to ask during these interviews are located at ATTACHMENT IV, Guide
for Interviews of MCO/PIHP Personnel Concerning Data Integration and Control.
The EQRO should document all findings with respect to the adequacy of the MCO/PIHP data
integration and control procedures on a worksheet such as that found in ATTACHMENT V,
Data Integration and Control Findings - Documentation Worksheet.

ONSITE ACTIVITY 2:
Assess documentation of data and processes used to calculate
and report performance measures.
The MCO/PIHP should have documentation of all steps undertaken in the production of the
State-specified performance measures, including documentation of: 1) the collection of data
from various sources (e.g., membership, enrollment, provider, claims, or encounter files; medical
records; laboratory and/or pharmacy records); 2) steps taken to integrate the required data into a
performance measure data set or repository; and 3) procedures or programs to query the data
set/repository to identify denominators, generate appropriate samples, determine numerators, and
apply proper algorithms to the data in order to produce valid and reliable performance measures.
During this activity, for each measure to be validated, the EQRO will:
1.

Review performance measurement plans and policies to assess the extent to which they
include:
21

-

2.

data file and field definitions;
maps to standard coding if standard codes were not used in original data
collection; and
statistical testing of results, and any corrections or adjustments made after
processing.

Examine documentation (which may be either a schematic diagram or in narrative form)
of programming specifications to ensure that documentation exists for at least the
following information:
-

-

-

-

A project or measurement plan, including work flow.
All data sources, including external data (whether from a vendor, public registry,
or other outside source) and any prior years’ data (if applicable).
Documentation of the original universe of data that includes record-level patient
identifiers that can be used to validate entire programming logic for creating
denominators, numerators, and samples.
detailed medical record review methods and practices, including the qualifications
of medical record review supervisor and staff; reviewer training materials; audit
tools used, including completed copies of each record-level reviewer
determination; all case-level critical performance measure data elements used to
determine a positive or negative event or exclude a case from same; and interrater reliability testing procedures and results.
Detailed computer queries, programming logic, or source codes used to create all
denominators, numerators, and samples (if applicable to the measure). This
Includes the processes for identifying the population or sample for the
denominator and/or numerator for each measure. If sampling is used, this includes
a description of sampling techniques and documentation that samples used for
baseline and repeat performance measurements were chosen using the same
sampling frame and methodology.
Documentation of calculation for changes in performance from previous periods
(if applicable) including statistical tests of significance.

The EQRO will need to refer to the specifications for each measure that were developed by the
EQRO during Pre-Onsite activities (illustrated in ATTACHMENT I). A list of the
documentation to be reviewed is located at ATTACHMENT VI, Data and Processes Used to
Calculate and Report Performance Measures - Documentation Review Worksheet. In addition,
as needed, the EQRO will interview the Director of Health/Medical Information Systems, system
programmers or operators, and the Director of Quality Improvement or other MCO/PIHP
personnel to supplement this information, facilitate demonstrations of performance measurement
processes, and provide the answers to questions such as the following:
22

1.

How are policies governing documentation of data requirements for performance
measurement, (e.g., data file and field definitions, mapping between standard and nonstandard codes) updated and enforced? Who is responsible for this?

2.

How are programming specifications for MCO/PIHP performance measures
documented? Who is responsible for this?

3.

Are the documentation processes up to date?

The results of the EQRO’s review of the MCO’s/PIHP’s documentation of data and processes
used to prepare and submit performance measures should be recorded on a form such as that
found as ATTACHMENT VII: Data and Processes Used to Calculate and Report Performance
Measures - Documentation Worksheet.
ONSITE ACTIVITY 3:

Assess processes used to produce denominators.

The fundamental question to be answered by validating the calculation of the denominator(s) of
performance measures is to what extent the MCO/PIHP used the appropriate data (including
linked data from separate data sets) to identify the entire at-risk population. The “appropriate
data” will vary from measure to measure, depending on criteria such as age, sex, diagnosis, or
procedure, and may be adjusted to exclude certain patients for reasons identified in the
specifications established by the State for calculating the measure. Also, in some cases, the
MCO/PIHP may have to estimate portions of the population, such as newborns, who cannot
always be readily and fully counted. In such cases, the EQRO should confirm that the
methodology used for such estimations is valid. In conducting this activity, the EQRO will need
to refer to the State’s specifications for each measure as noted by the EQRO during Pre-Onsite
activities and as illustrated in ATTACHMENT I.
During this activity, for each performance measure calculated by the MCO/PIHP and chosen to
be included in the validation activity, the EQRO will assess the extent to which:
1.

2.

All members who were eligible to receive the specified services under study were
included in the initial population from which the final denominator was produced. This
“at risk” population will include both members who received the services, as well as
those who did not. This same validation activity applies to provider groups, or other
relevant populations identified in the specifications of each performance measure.
Programming logic or source codes which identify, track, and link member enrollment
within and across product lines (e.g., Medicare and Medicaid), by age and gender, as well
as through possible periods of enrollment and disenrollment, have been appropriately
23

3.
4.

5.

6.

7.

applied according to the specifications of each performance measure. This is determined
by evaluating the extent to which:
- Calculations of continuous enrollment criteria were correctly carried out and
applied to each measure (if applicable).
- The MCO/PIHP used appropriate mathematical operations to determine patient
age or range.
- The MCO/PIHP can identify the variable(s) that code the member’s sex in every
file or algorithm, and that the MCO/PIHP can explain what classification is
carried out if neither of the required codes is present.
The MCO/PIHP has correctly calculated member months and member years, if applicable
to the performance measure.
The MCO/PIHP has properly evaluated the completeness and accuracy of any codes used
to identify medical events, such as diagnoses, procedures, or prescriptions, and that these
codes have been appropriately identified and applied as specified in each performance
measure.
Time parameters required by the performance measure specifications are followed (e.g.,
cut-off dates for data collection, counting 30 calendar days after discharge from a
hospital).
Performance measure specifications or definitions were followed in excluding members
from a denominator. For example, if a measure relates to receipt of a specific service, the
denominator may need to be adjusted to reflect instances in which the patient refuses the
service or the service is contraindicated.
Systems or methods used by the MCO/PIHP to estimate populations when they cannot be
accurately or completely counted (e.g., newborns) are valid.

Policies, procedures, data, and information to be reviewed in conducting these activities are
listed in ATTACHMENT VIII. Information obtained from a review of these policies,
procedures, data, and information should be supplemented and confirmed, as needed, though
interviews with MCO/PIHP personnel, including: the Director of Health/Medical Information
Systems, system programmers or operators, and selected sub-contractors. Suggested questions to
be asked are located in ATTACHMENT IX.
The findings of the EQRO’s documentation review, interviews and any needed demonstrations
of processes should be documented on a Denominator Validation Findings - Reviewer
Worksheet, such as that located at ATTACHMENT X.

ONSITE ACTIVITY 4:

Assess processes used to produce numerators.

The focus of numerator validation is on determining whether the MCO/PIHP has correctly
identified and evaluated qualifying medical events (e.g., diagnoses, procedures, and
24

prescriptions) in order to include appropriate events in the numerator of the performance
measure. These “medical events” may be identified through membership/enrollment data,
claim/encounter data, and/or provider data. They may also be identified through data extracted
from medical records, or through a combination of both administrative data and medical record
abstraction, i.e., the “hybrid” methodology.
As with denominators, accurate and complete data collection is vital to this element of
performance measure calculation. For measures that include sampling in the methodology, the
entire at-risk population must have an equal chance to be included in the numerator. For some
measures, particularly those frequently focused on women and children in the Medicaid
population, the member may have received the specified service outside of the MCO/PIHP
provider base (e.g., children receiving immunizations through public health services or schools),
so an effort must be made to include these events in the numerator.
If either medical record review or the hybrid methodology is used to calculate the performance
measure, the EQRO will need to review a sample of medical records which are identified as
having been included in the sample drawn by the MCO/PIHP. Following specific rules and
guidelines, the EQRO will determine the extent to which data obtained from medical records and
noted as being part of the numerator results can be confirmed during medical record review
validation activities.
During this activity, for each performance measure calculated by the MCO/PIHP and chosen to
be included in the validation activity, the EQRO will assess the extent to which:
1.

2.

3.

4.
5.

6.

The MCO/PIHP has used the appropriate data, including linked data from separate data
sets, to identify the entire at-risk population that meets the specified criteria for inclusion
in the numerator.
The MCO/PIHP has adopted and followed procedures to capture data for those
performance measures which could be easily under-reported due to the availability of
services outside the MCO/PIHP.
The MCO’s/PIHP’s use of codes to identify medical events (such as diagnoses,
procedures, prescriptions, etc.) are complete, accurate, and specific in correctly
describing what has transpired and when. In particular, the EQRO will assess the extent
to which these codes were correctly evaluated when classifying members for inclusion or
exclusion in the numerator.
The MCO/PIHP has avoided or eliminated double-counted members or numerator events.
Any non-standard codes used by the MCO/PIHP are mapped to standard codes in a
manner that is consistent, complete, and reproducible. The EQRO will assess this through
a review of the programming logic or a demonstration of the program.
The MCO/PIHP has adhered to any time parameters required by the specifications of the
performance measure (i.e., that the measured event occurred during the time period
specified or defined in the performance measure).
25

7.

Medical record reviews and abstractions have been carried out in a manner that facilitates
the collection of complete, accurate, and valid data by ensuring that:
- Record review staff have been properly trained and supervised for the task.
- Record abstraction tools require the appropriate notation that the measured event
occurred.
- Record abstraction tools require notation of the results or findings of the
measured event (if applicable).
8.
Data included in the record extract files are consistent with data found in the medical
records for a sample of medical records for applicable performance measures.
9.
The process of integrating administrative data and medical record data for the purpose of
determining the numerator is consistent and valid.
Policies, procedures, data, and information to be reviewed in conducting these activities are
listed in ATTACHMENT XI. These activities will need to be carried out with respect to each
performance measure calculated by the MCO/PIHP and included in the EQRO validation
activities. Because of this, the EQRO will need to refer to the specifications for each measure
that were noted by the EQRO during Pre-Onsite activities as illustrated in ATTACHMENT I. In
addition, for at least three of the performance measures calculated via medical record review or
hybrid methodology, the EQRO will need to validate the results of the medical record review for
30 enrollees who were found to meet numerator requirements for each of the three or more
measures. Procedures and sample tools for validating medical record review findings are
included as ATTACHMENT XII.
Information obtained from a review of policies, procedures, data, and information should be
supplemented or confirmed, as needed, though interviews with MCO/PIHP personnel, including:
the Director of Health/Medical Information Systems, system programmers or operators, and
selected sub-contractors. Suggested questions are the same as those asked with respect to
denominators and are located at ATTACHMENT IX.
The findings of the EQRO’s documentation review, interviews, any needed demonstrations of
processes, and validation of medical record review should be documented on a Numerator
Validation Findings - Reviewer Worksheet such as that located at ATTACHMENT XIII.

ONSITE ACTIVITY 5:

Assess the sampling process (for measures NOT calculated
through administrative data).

The basic task in validating the sampling methodology is determining whether the sample
validly reflects: 1) the performance of all practitioners and providers who serve Medicaid
enrollees and whose activities are the subject of the performance measure; and 2) the care given
to the entire population (including special populations with complex care needs) to which the
performance measure is relevant.
26

As in the previous activity of validating the population included in a denominator, the sampling
methodology employed should not exclude any population subgroups to which the topic area and
performance measure apply. For example, when studying well child care, an MCO’s/PIHP’s
sample should not exclude children with special health care needs whose primary care provider
is a specialist other than a pediatrician or family practitioner.
During this activity, the EQRO will assess the extent to which:
1.
2.
3.

4.
5.
6.

7.

8.

9.
10.
11.
12.
13.

The sampling methodology used by the MCO/PIHP produced an unbiased sample which
is representative of the entire at-risk population.
Each relevant enrollee or provider had an equal chance of being selected; no enrollees
were systematically excluded from the sampling.
The MCO/PIHP followed the specifications set forth by the State for the performance
measure regarding the treatment of sample exclusions and replacements and, if any
activity took place involving replacements of or exclusions from the sample, the
MCO/PIHP kept adequate documentation of that activity.
Each provider serving a given number of enrollees had the same probability of being
selected as any other provider serving the same number of enrollees.
The MCO/PIHP examined its sampled for bias and if any bias was detected, the
MCO/PIHP is able to provide documentation that describes efforts taken to correct it.
The sampling methodology treated all measures independently and there is no correlation
between drawn samples. (This is not intended to be a validation of the prescribed
sampling methodology included in the performance measure specifications, because the
assumption is that it is a valid methodology. The EQRO validation efforts will focus on
the MCO’s/PIHP’s implementation of that sampling methodology to assess the extent to
which it has correctly followed the sampling specifications.)
Relevant members or providers who were not included in the sample for the baseline
measurement have the same chance of being selected for the follow-up measurement as
those who were included in the baseline.
The MCO/PIHP has policies, procedures, and documentation that files from which the
samples were drawn are maintained so that if the sample must be re-drawn, or
replacements made, the original population is intact.
The sample selected conforms to the methodology set forth in the performance measure
specifications.
Sample sizes meet the requirements of the performance measure specifications.
The MCO/PIHP appropriately handled the documentation and reporting of the measure if
the requested sample size exceeds the population size.
The MCO/PIHP properly oversampled in order to accommodate potential exclusions.
The MCO/PIHP followed proper substitution methodology in medical record review (for
measures using the hybrid methodology or medical record review).
27

-

Substitution applied only to those members who met the exclusion criteria
detailed in the performance measure specifications.
Substitutions were made for properly excluded records and the percentage of
substituted records was documented.

Policies, procedures, data, and information to be reviewed in conducting these activities are
listed in ATTACHMENT XIV. These activities need to be carried out with respect to each
performance measure that was calculated using a sample. Because of this, the EQRO will need to
refer to the “List of Performance Measures Calculated by the MCO/PIHP” (TABLE 3) and the
specifications for each measure that were noted by the EQRO during Pre-Onsite activities as
illustrated in ATTACHMENT I.
Information on sampling obtained from a review of policies, procedures, data, and information
should be supplemented and confirmed, as needed, though interviews with MCO/PIHP
personnel, such as: the Director of Health/Medical Information Systems, system programmers or
operators, and selected sub-contractors. Suggested questions to ask are those previously
identified and included as ATTACHMENT IX. Validation findings regarding sampling should
be documented on a worksheet such as that found as ATTACHMENT XV.

ONSITE ACTIVITY 6:

Assess submission of required performance measure reports to
the State.

Once the MCO/PIHP calculates the required performance measures, it must report them to the
State in the manner prescribed by the State. This includes reporting the measures in a proper
format, whether through the use of a hardcopy “shell” report, in an electronic medium and
format, or some combination of both. During the Pre-Onsite phase of the review, the EQRO
familiarized itself with the State’s format and reporting requirements for the MCO’s/PIHP’s
performance measures. During this activity, the EQRO will assess whether measures were
reported to the State in the manner and form prescribed by the State. These activities will need to
be carried out with respect to each performance measure to be calculated by the MCO/PIHP.
Because of this, the EQRO will need to refer to the reporting specifications for all of the
measures that were noted by the EQRO during Pre-Onsite activities as documented in TABLE 1.
To assess the submission of required performance measure reports to the State, the EQRO will
review:
-

Procedures for submitting reports that meet State requirements (e.g., specified
electronic format, supporting documentation, timing); and
Documentation that procedures for properly submitting required reports to State
were implemented appropriately.
28

The extent to which the MCO/PIHP reported the calculated performance measures to the State in
the manner and form prescribed by the State should be documented in the EQRO’s report to the
State.
POST-ONSITE ACTIVITIES
Objectives for Post-Onsite Activities:
The EQRO will evaluate all gathered information and submit a report on its validation findings
to the State following either Option 1 or Option 2 below.
OPTION 1:

The EQRO submits its report of validation findings to the State after review by
the MCO/PIHP for any factual errors or omissions.

OPTION 2:

The EQRO submits a final report to the State after providing the MCO/PIHP with
the opportunity to make corrections to performance measures in response to
preliminary EQRO findings. This would occur as follows:
-

-

The EQRO submits to the MCO/PIHP a preliminary report detailing areas of
concern and suggested methods for correction.
After allowing the MCO/PIHP to correct (as practical) any problems in
calculating or reporting performance measures that were identified in the
preliminary report, the EQRO re-validates selected performance measures and the
measurement processes.
The EQRO again evaluates gathered information and prepares a final report for
the State.
The EQRO submits its report of validation findings to the State.

POST-ONSITE ACTIVITY 1:

Determine preliminary validation findings for each
measure.

Once the EQRO concludes its onsite activities, it aggregates the validation activity findings for
each performance measure. This involves review and analysis of findings and worksheets
produced for each performance measure selected for validation and for the MCO’s/PIHP’s IS as
a result of Pre-Onsite and Onsite activities. In particular, these include:

29

-

-

Completed performance measure validation worksheets for each performance
measure to be validated (as in ATTACHMENT I) in conjunction with the
Denominator Validation Findings (ATTACHMENT X) and Numerator
Validation Findings (ATTACHMENT XIII).
For measures calculated through medical record review, including the hybrid
methodology, the completed Medical Record Review Validation Tool
(ATTACHMENT XII).

-

Findings regarding the MCO’s/PIHP’s data integration and control procedures
(ATTACHMENT V); and

-

Sampling validation findings (ATTACHMENT XV).

The report of preliminary validation findings identifies any areas of concern for each of the
performance measures that were validated by the EQRO and makes suggestions for
improvement. In particular, the report indicates precisely which elements of the MCO/PIHP
performance measures were invalid (if any). This information provides the MCO/PIHP with
specific targets for correction and a tool that can be used to focus MCO/PIHP personnel on the
changes necessary to improve the production process. In addition to communicating in writing,
the EQRO may participate in meetings with key MCO/PIHP personnel responsible for the
calculation and reporting of performance measures.
Once the EQRO has submitted its preliminary findings to the MCO/PIHP, there are two courses
of action that the State may have its EQRO pursue with respect to allowing the MCO/PIHP to
respond to the EQRO’s preliminary findings:
OPTION 1:

The MCO/PIHP may offer comments and documentation to support correction of
factual errors and omissions in the EQRO’s preliminary report; or

OPTION 2:

The MCO/PIHP would be allowed to recalculate performance measures based on
the findings of the EQRO. The EQRO would then revalidate the revised
performance measure(s).
Allowing MCOs/PIHPs to recalculate measures provides States and Medicaid
beneficiaries with a greater amount of accurate information on MCO/PIHP
performance. However, this option requires greater time and financial resources
on the part of the States, EQROs and MCOs/PIHPs. If Option 2 is chosen by the
State, depending on the extent of the corrections necessary or assistance that the
MCO/PIHP needs to improve its performance measure production processes, the
EQRO schedules a time to re-visit the MCO/PIHP as soon as practical, in order to
re-evaluate the performance measures before they are reported to the State. This
30

re-evaluation follows the same format and activities as the initial onsite visit,
except that the EQRO may focus only on those activities that were found to be
problematic during the first validation effort. The EQRO will use worksheets and
tools that are identical to those used in the first onsite visit; any areas not rereviewed should be noted accordingly.
Once Option 1 or Option 2 is completed, and the MCO’s/PIHP’s comments or revised
performance measures validation findings have been appropriately incorporated into the
validation findings, the EQRO will submit its findings to the State.

POST-ONSITE ACTIVITY 2:

Submission of validation report to State.

A State may choose one of two options for determining the validity of each of the MCO's/PIHP's
performance measures:
OPTION 1:

The EQRO submits all working papers and a summary of findings to the State.
The State would make the final decision on the validity of each performance
measure and compliance with reporting requirements.

OPTION 2:

The EQRO references a clearly defined set of decision rules for determining if
each of the MCO’s/PIHP’s reported performance measures were sufficiently
valid; i.e., accurate and complete. In this instance, the State would still receive the
final report and all supporting documentation and would have the final authority
to determine acceptable validity and compliance with State conditions.

Regardless of which option a State chooses, the decision rules for compliance should be uniform
across MCOs/PIHPs within the State. Because States may differ substantially regarding their
requirements for Medicaid MCOs/PIHPs, this protocol provides a framework which the State
can use with its own specific “percentage rules” or requirements for determining validity of
performance measures.
The State will need to specify the level of bias that is permissible or allowable in the calculated
performance measures in order for an MCO’s/PIHP’s performance measure to be considered
“valid measures.” Levels currently in use within the industry range from 5 percent to 10 percent
for commercial and/or Medicare product lines. Bias in reported rates can result from many
factors; e.g., sampling bias, coding errors, and in particular, problems with in complete data. For
example, is a measure calculated using a data set that is known to be only 50 percent complete a
valid measure of performance? What about a measure using data that is 75 or 85 percent
complete? Because there is currently no generally accepted standard for data completeness in the
industry, each State must specify the extent of data incompleteness it allows in measures before
31

the measure is considered to be “not valid.” Data completeness was addressed as part of the
Performance Measure Validation Worksheet for each performance measure as illustrated in
ATTACHMENT I. The EQRO will need to make an estimate about the cumulative affect of all
sources of bias on the validity of the performance measure.
The format for the final report should follow the format specified by the State, but should include
the following elements:
-

-

-

a list of measures for validation. (It is possible that an MCO/PIHP would be
unable to report on all required measures for reasons that would be explained to
the EQRO and the State.)
a description of the onsite validation activities including: 1) a list of the EQRO’s
team members 2) a description of the pre-audit strategy and considerations, 3) a
description of the technical methods of data collection and analysis used by the
EQRO, 4) a list of interviewees, and 5) any other facts relevant to the onsite
process.
details, results, and conclusions drawn of the validation process for each
performance measure, including any medical record abstractions conducted.
as directed by the State, the validation findings for each performance measure
included in the EQRO validation activities.
as directed by the State, analysis and findings with respect to the MCO’s/PIHP’s
data integration and control procedures and performance measure calculation
documentation.

In addition to reporting to the State on the extent to which the MCO/PIHP correctly implemented
processes to calculate and report individual MCO/PIHP performance measures, other aspects of
MCO/PIHP performance measurement that the State may want the EQRO to address in its final
report include the extent to which the MCO/PIHP has:
-

adequate data integration and control necessary for accurate reporting of
performance measures; and

-

complete and accurate documentation of data and processes used to calculate and
report performance measures.

In addition, the EQRO might also be asked to submit all of its worksheets and tools as
supporting documentation to the report.
END OF PROTOCOL
32

ATTACHMENT I

Example of a Completed Performance Measure Validation Worksheet 4
Below is an example of a completed, customized performance measure validation worksheet similar to what an
EQRO would prepare prior to its onsite visit. This worksheet assumes that the State has adopted the HEDIS
methodology for this performance measure. One of the following scoring designations must be checked for each
audit element:
MET: The MCO’s/PIHP’s measurement and reporting process was fully compliant with State
specifications.
NOT MET: The MCO’s/PIHP’s measurement and reporting process was not compliant with State
specifications. This designation should be used for any audit element that deviates from the State
specifications, regardless of the impact of the deviation on the final rate. All audit elements with
this designation must include explanation of the deviation in the comments section.
N/A: The audit element was not applicable to the MCO’s/PIHP's measurement and reporting
process.
PERFORMANCE MEASURE TO BE VALIDATED: BREAST CANCER SCREENING
METHODOLOGY FOR CALCULATING
MEASURE: (Check one)

AUDIT
ELEMENTS
DENOMINATOR
1. Population

AUDIT SPECIFICATIONS
-

-

2. Geographic Area

-

3. Age & Sex

-

4

ADMINISTRATIVE

MET NOT
MET

MEDICAL
RECORD
REVIEW

N/A

HYBRID

COMMENTS

Medicaid population
appropriately segregated from
commercial / Medicare.
Population defined as
effective Medicaid enrollment
as of Dec. 31, 2000.
Dual Medicaid and Medicare
beneficiaries are included.
Includes only those Medicaid
enrollees served in the
MCO’s/PIHP’s reporting area.
Members aged 52-69 as of
12/31/00 (i.e., born between
1/1/31 & 12/31/48)
Only females selected

This worksheet is adapted from the IPRO tools used in the audit of the 1997 Medicare HEDIS data.

33

ATTACHMENT I
AUDIT
ELEMENTS

AUDIT SPECIFICATIONS
- Were members of plan on
4. Enrollment
12/31/00
Calculation
- Were continuously enrolled
from 1/1/99 to 12/31/00 with
one break per year of up to 45
days allowed.
- Switches between populations
(Medicare, Medicaid, and
commercial) were not counted
as breaks.
Based on the IS assessment
5. Data Quality
findings, are any of the data
sources for this denominator
inaccurate?
6. Proper Exclusion
Only members with
Methodology in
contraindications or data
Administrative
errors were excluded.
Data (If no
- Contraindication exclusions
exclusions were
were performed according to
taken, check N/A)
current State specifications.
- Only the codes listed in
specifications as defined by
State were counted as
contraindications.
NUMERATOR
7. Administrative
Data: Counting
Clinical Events

-

-

8. Medical Record
Review
Documentation
Standards

-

-

9. Time Period

-

10.Data Quality

-

MET NOT
MET

N/A

COMMENTS

Standard codes listed in State
specifications or properly
mapped internally developed
codes were used. (Intended to
reference appropriate
specifications as defined by
State.)
Members were counted only
once; double counting of
mammograms was prevented.
Record abstraction tool
required notation of the date
that the mammogram was
performed.
Record abstraction tool
required notation of the
mammogram result or finding.
Mammogram performed on or
between 1/1/99 & 12/31/00.
Properly identified enrollees.

34

ATTACHMENT I
AUDIT
ELEMENTS

AUDIT SPECIFICATIONS
-

SAMPLING
11. Unbiased
Sample

MET NOT
MET

N/A

COMMENTS

Based on the IS assessment
findings, were any of the data
sources used for this
numerator inaccurate?

IF ADMINISTRATIVE METHOD WAS USED, CHECK N/A FOR AUDIT
ELEMENTS 11, 12, AND 13.
- As specified in State
specifications, systematic
sampling method was utilized.

12. Sample Size

-

13. Proper
Substitution
Methodology in
Medical Record
Review (If no
exclusions were
taken, check
N/A)

-

-

After exclusions, sample size
is equal to 1) 411, 2) the
appropriately reduced sample
size, which used the current
year’s administrative rate or
preceding year’s reported
rate, or 3) the total population.
Only excluded members for
whom medical record review
revealed 1) contraindications
that correspond to the codes
listed in appropriate
specifications as defined by
State or 2) data errors.
Substitutions were made for
properly excluded records and
the percentage of substituted
records was documented.

ADDITIONAL QUESTIONS
QUESTIONS

YES

NO

Were members excluded for contraindications found in the administrative data?
Were members excluded for contraindications found during the medical record
review?
Were internally developed codes used?
What range defines the impact of data incompleteness for this measure? (Check one.)
0 - 5 percentage points
>5 - 10 percentage points
>10 - 20 percentage points
>20 - 40 percentage points
>40 percentage points

35

ATTACHMENT I
Unable to Determine

What is the direction of the bias? Check one:

OVER-REPORTING
UNDER-REPORTING
Upon what documentation is the above percentage based? (e.g., internal reports, studies,
comparison to medical records, etc.
VALIDATION FINDING
The validation finding for each measure is determined by the magnitude of the errors detected
for the audit elements, not by the number of audit elements determined to be “NOT MET.”
Consequently, it is possible that an error for a single audit element may result in a designation of
“NV” because the impact of the error biased the reported performance measure by more than
“x” percentage points. Conversely, it is also possible that several audit element errors may have
little impact on the reported rate and, thus the measure could be given a designation of “SC.”
The following is a list of the validation findings and their corresponding definitions:
FC

=

Fully Compliant
Measure was fully compliant with State specifications.

SC

=

Substantially Compliant
Measure was substantially compliant with State specifications and had only minor
deviations that did not significantly bias the reported rate.

NV

=

Not Valid
Measure deviated from State specifications such that the reported rate was
significantly biased. This designation is also assigned to measures for which no
rate was reported, although reporting of the rate was required.

NA

=

Not Applicable
Measure was not reported because MCO/PIHP did not have any Medicaid
enrollees that qualified for the denominator.

AUDIT DESIGNATION

36

ATTACHMENT II

Potential Documents and Processes for Review
In order to assess the MCO’s/PIHP’s IS and the validity of reported performance measures, the
EQRO will need to review a number of data sources and processes. The MCO/PIHP should
ensure that the following documents, data, and procedures are available to the EQRO for
observation; the EQRO will use its discretion in selecting which ones to review.
Integration and Control of Data
-

Procedures and standards for all aspects of the data repository (ies) used in the production
of performance measures, including building, maintaining, managing, testing, and
production of performance measures.

-

Manuals covering application system development methodology, database development,
and design and decision support system utilization.

-

Control system documentation including flow charts and codes for backups, recovery,
archiving, and other control functions.

-

Procedures to consolidate information from disparate transaction files.

-

Record and file formats and descriptions, for entry, intermediate, and repository files.

-

Electronic formats and protocols.

-

Electronic transmission procedures documentation.

-

Processes to extract information from the repository(ies).

-

Source code data entry, data transfer, and data manipulation programs and processes.

-

Descriptive documentation for data entry, transfer, and manipulation programs and
processes.

-

If applicable, procedures for coordinating activities of multiple subcontractors in a way
that safeguards the integrity of the performance measurement data.

-

Samples of data from repository and transaction files to assess accuracy and
completeness of the transfer process.

-

Comparison of actual results from file consolidation and data abstracts to those which
should have resulted according to documented algorithms.

-

Documentation of data flow among vendors to assess the extent to which there has been
proper implementation of procedures for coordinating activities to safeguard the integrity
of the performance measure data.

-

Documentation of data cutoff dates.

-

Documentation of proper run controls and of staff review of report runs.

-

Copies of files and databases used for performance measure calculation and reporting.
37

-

ATTACHMENT II
Procedures governing production process for MCO/PIHP performance measures,
including standards and schedules.

Collection, Calculation, and Documentation of Performance Measurements
-

Policies which stipulate and enforce documentation of data requirements, issues,
validation efforts, and results.

-

A project or measurement plan for each performance measure.

-

Documentation of programming specifications, including work flow, data sources, and
uses which include diagrammatic or narrative descriptions.

-

Documentation of the original universe of data that includes record-level patient
identifiers that can be used to validate entire programming logic for creating
denominators, numerators, and samples.

-

Documentation of computer queries, programming logic, or source code used to create
final denominators, numerators, and interim data files.

-

Documentation that includes dated job log or computer run for denominators and
numerators, with record counts for each programming step and iteration.

-

Documentation of medical record review including: qualifications of medical record
review supervisor and staff; reviewer training materials; audit tools used, including
completed copies of each record-level reviewer determination; all case-level critical
performance measure data elements used to determine a positive or negative event or
exclude a case from same; and inter-rater reliability testing procedures and results.

-

Documentation of results of statistical tests and any corrections or adjustments to data
along with justification for such changes.

-

Documentation of sources of any supporting external data or prior years’ data used in
reporting.

-

Policies to assign unique membership ID that allows all services to be properly related to
the specific appropriate enrollee, despite changes in status, periods of enrollment or
disenrollment, or changes across product lines (e.g., Medicare and Medicaid).

-

Procedures to identify, track, and link member enrollment by product line, product,
geographic area, age, sex, member months, and member years.

-

Procedures to track individual members through enrollment, disenrollment, and possible
re-enrollment.

-

Procedures to track members through changes in family status, changes in benefits or
managed care type (if they switch between Medicaid coverage and another product
within the same MCO/PIHP).

-

Methods to define start and cessation of coverage.
38

ATTACHMENT II
-

Procedures to link member months to member age.

-

Description of software or programming languages used to query each database.

-

Description of software used to execute sampling sort of population files when sampling
(systematic) is used.

-

Member database.

-

Provider data (including facilities, labs, pharmacies, physicians, etc.).

-

Database record layout and data dictionary.

-

Survey data.

-

Policies to maintain files from which the samples are drawn in order to keep population
intact in the event that a sample must be re-drawn, or replacements made.

-

Computer source code or logic identifying specified sampling techniques, and
documentation that the logic matches the specifications set forth for each performance
measure, including sample size and exclusion methodology.

-

Methods used for sampling for measures calling for hybrid data (combination of medical
records and administrative data) or solely medical record review.

-

Documentation assuring that sampling methodology treats all measures independently
and that there is no correlation between drawn samples.

-

Observation or documentation of procedures in which a biased sample was identified and
corrected.

-

Documentation of “frozen” or archived files from which the samples were drawn, and if
applicable, documentation of the MCO’s/PIHP’s process to re-draw a sample or obtain
necessary replacements.

-

For performance measures which are easily under-reported, procedures to capture data
that may reside outside the MCO’s/PIHP’s data sets.

-

Procedures for mapping non-standard codes to standard coding to ensure consistency
completeness, and reproducibility.

-

Policies, procedures, and materials that evidence proper training, supervision, and
adequate tools for medical record abstraction tasks. (May include medical record
abstraction tools, training material, checks of inter-rater reliability, etc.)

-

Procedures for assuring that combinations of record-review data with administratively
determined data are consistent and verifiable.

-

Evidence that MCO’s/PIHP’s use of codes to identify medical events were correctly
evaluated when classifying members for inclusion or exclusion in the numerator.

-

Evidence that MCO/PIHP has counted each member and/or event only once.

-

Programming logic or demonstration that confirms that any non-standard codes used in
determining the numerator have been mapped to a standard coding scheme in a manner
39

ATTACHMENT II
that is consistent, complete, and reproducible.
-

Programming logic or source code that identifies the process for integrating
administrative and medical record data for numerator.

-

Procedures for properly executing complex medical algorithms, such as claim-dependent
events; events that require matching claims and pharmacy data; events that require
matching visit codes; and events that require accurately identifying and computing
multiple numerator events.

-

Procedures for displaying denominator counts, numerator counts, precision levels, sums
and cross-totals.

-

Procedures for reporting small sample sizes (to be consistent with required methodology
established by State).

-

Programming logic and/or source code for arithmetic calculation of each measure.

-

Review of reported measures to assess consistency of common elements (e.g.,
membership counts, number of pregnancies and births, etc.).

-

Programming logic and/or source code for measures with complex algorithms, to ensure
adequate matching and linkage among different types of data.

-

Documentation showing confidence intervals of calculations when sampling
methodology used.

-

Documentation showing calculation of levels of significance of changes.

-

Procedures for submitting reports that meet State requirements (e.g., specified electronic
format, supporting documentation, timing).

-

Documentation that procedures for properly submitting required reports to State were
implemented appropriately.

40

ATTACHMENT III
IS Data Integration and Control - Documentation Review

Worksheet
Documentation

Reviewed

Not
Reviewed

Comments

Procedures and standards for all aspects of
the data repository (ies), including building,
maintaining, managing, testing, and
production of performance measures.
Manuals covering application system
development methodology, database
development and design, and decision
support system utilization.
Control system documentation including
flow charts and codes for backups,
recovery, archiving, and other control
functions.
Procedures to consolidate information from
disparate transaction files to support
performance measurement.
Record and file formats and descriptions,
for entry, intermediate, and repository files.
Electronic formats and protocols.
Electronic transmission procedures
documentation.
Processes to extract information from the
repository to produce intended result.
Source code data entry, data transfer, and
data manipulation programs and processes.
Descriptive documentation for data entry,
data transfer, data manipulation programs
and processes.
If applicable, procedures for coordinating
activities of multiple subcontractors in a
way that safeguards the integrity of the
performance measure data.
Samples of data from repository and
transaction files to assess accuracy and
completeness of the transfer process.
Comparison of actual results from file

41

ATTACHMENT III
consolidation and data abstracts to those
which should have resulted according to
documented algorithms.
Documentation of data flow among vendors
to assess the extent to which there has been
proper implementation of procedures for
coordinating activities to safeguard the
integrity of the performance measure data.
Documentation of data cutoff dates.
Documentation of proper run controls and
of staff review of report runs.
Copies of files and databases used for
performance measure calculation and
reporting.
Procedures governing production process of
plan-level performance measures, including
standards and schedules.

In the comments section, be sure to address the following:
Compare samples of data in the repository to transaction files. Are any members, providers, or
services lost in the process?
Is the required level of coding detail maintained (e.g., all significant digits, primary and
secondary diagnoses remain)?
If the plan uses a performance measure repository, review the repository structure. Does it
contain all the key information necessary for performance measure reporting?
How does the MCO/PIHP test the process used to create the performance measure reports?
Does the MCO/PIHP use any algorithms to check the reasonableness of data integrated to report
the plan-level performance measures?
Examine report production logs and run controls. Is there adequate documentation of the
performance measure report generation process? How are report generation programs
documented? Is there a type of version control in place?

42

ATTACHMENT IV

Guide for Interviews of MCO/PIHP Personnel Concerning
Data Integration and Control
Background Information:
Name of MCO/PIHP:
Date:
Location:
Year of First Medicaid Enrollment:
Year of First MCO/PIHP Performance Report:
Auditors:

Names and Titles of Individuals Interviewed:

Has the MCO/PIHP previously undergone an audit of its State performance measure reporting
process? If so, when did the audit take place and who conducted it?

Other general issues:

Interview Questions:
1.

How is performance measure data collection accomplished:
-

By querying the applicable IS on-line?

-

By using extract files created for analytical purposes? If so, how frequently are the files
updated? How do they account for claim/encounter submission and processing lags?
How is the file creation process checked for accuracy?
43

ATTACHMENT IV
-

2.

By using a separate relational database or data warehouse? If so, is this the same system
all other reporting is produced from? Are reports created from a vendor software
product? If so, how frequently are the files updated? How are reports checked for
accuracy?
Review the procedure(s) for consolidating claims/encounter, member, provider, and other
data necessary for performance reporting (whether it be into a relational database or file
extracts on a measure-by-measure basis).

-

How many different sources of data are merged together to create reports?

-

What control processes are in place to ensure that this merger is accurate and complete?

3.

How does the MCO/PIHP test the process used to create the performance measure
reports?

4.

Does the MCO/PIHP use any algorithms to check the reasonableness of data integrated to
report the MCO/PIHP performance measures

5.

Are performance measurement reporting programs reviewed by supervisory staff?

6.

Is there an internal backup for performance measure programmers - do others know the
programming language and the structure of the actual programs? Is there
documentation?

7.

How does the plan prevent loss of claim and encounter data when systems fail?

8.

What administrative data backup systems are in place?

9.

What types of authorization are required to be able to access claims/encounter, provider,
membership, and performance measure repository data?

Describe Documentation Review and Demonstrations Provided:

44

ATTACHMENT V

Data Integration and Control Findings - Documentation Worksheet
Data Integration and Control Element

Met

Not Met

N/A

Comments

Accuracy of data transfers to assigned performance measure repository.
MCO/PIHP processes accurately and
completely transfer data from the transaction
files (e.g., membership, provider,
encounter/claims) into the repository used to
keep the data until the calculations of the
performance measures have been completed
and validated.
Samples of data from repository are complete
and accurate.

Accuracy of file consolidations, extracts, and derivations.
MCO’s/PIHP’s processes to consolidate
diversified files, and to extract required
information from the performance measure
repository are appropriate.
Actual results of file consolidations or
extracts were consistent with those which
should have resulted according to
documented algorithms or specifications.
Procedures for coordinating the activities of
multiple subcontractors ensure the accurate,
timely, and complete integration of data into
the performance measure database.
Computer program reports or documentation
reflect vendor coordination activities, and no
data necessary to performance measure
reporting are lost or inappropriately modified
during transfer.

If the MCO/PIHP uses one, the structure and format of the performance measure data
repository facilitates any required programming necessary to calculate and report required
performance measures.
The repository’s design, program flow
charts, and source codes enable analyses and
reports.
Proper linkage mechanisms have been
employed to join data from all necessary
sources (e.g., identifying a member with a
given disease/condition).

45

ATTACHMENT V
Assurance of effective management of report production and of the reporting software.
Examine and assess the adequacy of the
documentation governing the production
process, including MCO/PIHP production
activity logs, and MCO/PIHP staff review of
report runs.
Prescribed data cutoff dates were followed.
The MCO/PIHP has retained copies of files
or databases used for performance measure
reporting, in the event that results need to be
reproduced.
Review documentation standards to
determine the extent to which the reporting
software program is properly documented
with respect to every aspect of the
performance measurement reporting
repository, including building, maintaining,
managing, testing, and report production.
Review the MCO’s/PIHP’s processes and
documentation to determine the extent to
which they comply with the MCO/PIHP
standards associated with reporting program
specifications, code review, and testing.

46

ATTACHMENT VI

Data and Processes Used to Calculate and Report Performance Measures Documentation Review Worksheet
Documentation

Reviewed

Not
Reviewed

Comments

Policies which stipulate and enforce
documentation of data requirements, issues,
validation efforts and results.
Procedures for displaying denominator
counts, numerator counts, precision levels,
sums, and cross-totals.
Procedures for reporting small sample sizes
(to be consistent with required methodology
established by State).
Review of reported measures to assess
consistency of common elements (e.g.,
membership counts, number of pregnancies
and births, etc.).

For each measure:
Programming logic and/or source code for
arithmetic calculation.
A project or measurement plan, including
work flow.
Documentation of programming
specifications and data sources.
Documentation of the original universe of
data including record-level patient identifiers
that can be used to validate entire
programming logic for creating
denominators, numerators, and samples.
Documentation of computer queries,
programming logic, or source code used to
create denominators, numerators, and interim
data files.
Documentation that includes dated job log or
computer run for denominators and
numerators, with record counts for each
programming step and iteration.

47

ATTACHMENT VI
Documentation of medical record review for
each measure, as appropriate, including:
qualifications of medical record review
supervisor and staff; reviewer training
materials; audit tools used, including
completed copies of each record-level
reviewer determination; all case-level critical
performance measure data elements used to
determine a positive or negative event or
exclude a case from same; and inter-rater
reliability testing procedures and results.
Documentation of results of statistical tests
and any corrections or adjustments to data
along with justification for such changes for
each measure, as appropriate.
Documentation showing calculation of levels
of significance of changes for each measure.
Documentation (for each performance
measure, as appropriate) showing confidence
intervals of calculations when sampling
methodology used.
Documentation of sources of any supporting
external data or prior years’ data used in
reporting (for each performance measure, as
appropriate).

Describe Documentation Reviewed and Demonstrations Provided:

48

ATTACHMENT VII

Data and Processes Used to Calculate and Report Performance Measures Documentation Worksheet
Audit Element

Met

Not Met

N/A

Comments

Measurement plans and policies which stipulate and enforce documentation of data
requirements, issues, validation efforts and results. These include:
Data file and field definitions used for
each measure.
Maps to standard coding if not used in
original data collection.
Statistical testing of results and any
corrections or adjustments made after
processing.

Documentation of programming specifications (which may be either a schematic diagram or
in narrative form) for each measure includes at least the following:
All data sources, including external
data (whether from a vendor, public
registry, or other outside source), and
any prior years’ data (if applicable).
Detailed medical record review
methods and practices, including the
qualifications of medical record review
supervisor and staff; reviewer training
materials; audit tools used, including
completed copies of each record-level
reviewer determination; all case-level
critical performance measure data
elements used to determine a positive
or negative event or exclude a case
from same; and inter-rater reliability
testing procedures and results.
Detailed computer queries,
programming logic, or source code
used to identify the population or
sample for the denominator and/or
numerator

49

ATTACHMENT VII

Audit Element

Met

Not Met

N/A

Comments

If sampling used, description of
sampling techniques, and
documentation that assures the
reviewer that samples used for baseline
and repeat measurements of the
performance measures were chosen
using the same sampling frame and
methodology.
Documentation of calculation for
changes in performance from previous
periods (if applicable), including
statistical tests of significance.
Data that are related from measure to
measure are consistent (e.g.,
membership counts, provider totals,
number of pregnancies and births).
Appropriate statistical functions are
used to determine confidence intervals
when sampling is used in the measure.
When determining improvement in
performance between measurement
periods, appropriate statistical
methodology is applied to determine
levels of significance of changes.

50

ATTACHMENT VIII

Policies, Procedures, Data and Information Used to Produce Denominators:
Review Worksheet
Policies, Procedures, Data, Information
to be reviewed
Policies to assign unique membership ID that
allows all services to be properly related to
the specific appropriate enrollee, despite
changes in status, periods of enrollment or
disenrollment, or changes across product
lines (e.g., Medicare and Medicaid).
Procedures to identify, track, and link
member enrollment by product line, product,
geographic area, age, gender, member
months, member years.
Procedures to track individual members
through enrollment, disenrollment, and
possible re-enrollment.
Procedures to track members through
changes in family status, changes in
employment or benefits or managed care type
(if they switch between Medicaid coverage
and another product within the same
MCO/PIHP).
Methods to define start and cessation of
coverage.
Procedures to link member months to
member age.
Description of software or programming
languages used to query each database.
Programming logic and/or source code for
arithmetic calculation of each measure.
Programming logic and/or source code for
measures with complex algorithms, to ensure
adequate matching and linkage among
different types of data.
Member database.

Reviewed

Not Reviewed

Comments

Provider data (including facilities, labs,
pharmacies, physicians, etc.).
Database record layout and data dictionary.
Survey data.

51

ATTACHMENT IX

QUESTIONS FOR ASSESSING PROCESSES
USED TO PRODUCE DENOMINATORS AND NUMERATORS
1.

If any part of your network/data/membership was excluded from a performance measure,
how and why did you decide to exclude it?

2.

Why did you select the reporting methodology (e.g., administrative, or hybrid) used to
create each of the measures (where there was an option)?

3.

Did you use the State technical specifications as the specifications for the programmers,
or did your MCO/PIHP write its own instructions/translations for the programmers?

4.

Are there any manual processes used for calculating denominators and/or numerators?
Are manual processes used for sampling?

5.

Are any measures calculated by vendors? If yes, are they checked for accuracy? Please
describe.

6.

Do you have any concerns about the integrity of the information used to create any of the
measures? Please describe.

7.

Do you know of any deviations from performance measure specifications that were
necessary because of data available or because of your MCO’s/PIHP’s IS capabilities?

Other issues.

Names and Titles of Individuals Interviewed:

52

ATTACHMENT X

Denominator Validation Findings - Reviewer Worksheet

Audit Element

Met

Not
Met

N/A

Comments

For each of the performance measures, all members of the relevant populations identified in
the performance measure specifications are included in the population from which the
denominator is produced.
All members who were eligible to
receive the specified services were
included in the initial population from
which the final denominator was
produced. This “at risk” population
included both members who received
the services, as well as those who did
not. This same standard applies to
provider groups or other relevant
populations identified in the
specifications of each performance
measure.

Adequate programming logic or source code exists to appropriately identify all “relevant”
members of the specified denominator population for each of the performance measures.
For each measure, programming logic
or source code which identifies, tracks,
and links member enrollment within
and across product lines (e.g.,
Medicare and Medicaid), by age and
sex, as well as through possible
periods of enrollment and
disenrollment, has been appropriately
applied according to the specifications
of each performance measure.
Calculations of continuous enrollment
criteria were correctly carried out and
applied to each measure (if applicable).
Proper mathematical operations were
used to determine patient age or range.
The MCO/PIHP can identify the
variable(s) that define the member’s
sex in every file or algorithm needed to
calculate the performance measure
denominator, and the MCO/PIHP can
explain what classification is carried
out if neither of the required codes is
present.

53

ATTACHMENT X
Audit Element

Met

Not
Met

N/A

Comments

Correct calculation of member months and member years.
The MCO/PIHP has correctly
calculated member months and
member years, if applicable to the
performance measure.

Completeness and accuracy of the codes used to identify medical events has been identified
and the codes have been appropriately applied.
The MCO/PIHP has properly
evaluated the completeness and
accuracy of any codes used to identify
medical events, such as diagnoses,
procedures, or prescriptions, and these
codes have been appropriately
identified and applied as specified in
each performance measure.

Specified time parameters are followed.
Any time parameters required by the
specifications of the performance
measure are followed (e.g., cut off
dates for data collection, counting 30
calendar days after discharge from a
hospital, etc.).

Exclusion criteria included in the performance measure specifications have been followed.
Performance measure specifications or
definitions that exclude members from
a denominator were followed. For
example, if a measure relates to receipt
of a specific service, the denominator
may need to be adjusted to reflect
instances in which the patient refuses
the service or the service is
contraindicated.

Systems to estimate populations which cannot be accurately counted exist and are utilized
when appropriate.
Systems or methods used by the
MCO/PIHP to estimate populations
when they cannot be accurately or
completely counted (e.g., newborns)
are valid.

54

ATTACHMENT XI

Policies, Procedures, Data, and Information Used to Produce Numerators:
Review Worksheet
Documentation
For performance measures which are
easily under-reported, procedures to
capture data that may reside outside the
MCO/PIHP’s data sets.
Procedures for mapping non-standard
codes to standard coding to ensure
consistency, completeness, and
reproducibility.
Policies, procedures, and materials that
evidence proper training, supervision, and
adequate tools for medical record
abstraction tasks. (May include medical
record abstraction tools, training material,
checks of inter-rater reliability, etc.)
Procedures for assuring that combinations
of record-review data with
administratively determined data are
consistent and verifiable.
MCO’s/PIHP’s use of codes to identify
medical events were correctly evaluated
when classifying members for inclusion or
exclusion in the numerator.
Evidence that MCO/PIHP has counted
each member and/or event only once.
Programming logic or demonstration that
confirms that any non-standard codes
used in determining the numerator have
been mapped to a standard coding scheme
in a manner that is consistent, complete,
and reproducible.
Programming logic or source code that
identifies process for integrating
administrative and medical record data for
numerator.
Programming logic and/or source code for
arithmetic calculation of each measure.
Programming logic and/or source code for
measures with complex algorithms, to
ensure adequate matching and linkage
among different types of data.

Reviewed

Not Reviewed

Comments

Describe documentation review and any demonstrations provided:
55

ATTACHMENT XII

Medical Record Review Validation Tools
The purpose of medical record review (MRR) validation is to verify the accuracy of the MRR
conducted by each MCO/PIHP. For each of at least three measures for which the hybrid method
or solely MRR was used, the EQRO will validate the medical records of 30 enrollees found to
meet numerator requirements. Only those members included in a hybrid or solely MRR sample
will be selected - the EQRO will not be conducting medical record audits to validate
administrative data. Therefore, if an MCO/PIHP used only administrative data for a particular
measure, that measure will not be part of the MRR validation process.
For each measure in which the hybrid method or solely MRR was used, the EQRO will request a
list of all of the members in the MCO’s/PIHP’s MRR sample. From that list the EQRO will
identify a sample of 30 members who meet numerator requirements. MCOs/PIHPs will then be
asked to provide access to or copies of medical records so that the EQRO can verify that each
member was appropriately included in the denominator and received the required numerator
service(s). In cases where there are fewer than 30 numerator positives, the EQRO will review all
records for that measure.
To provide sufficient time for each MCO/PIHP to gather the required medical record
documentation, the EQRO may direct the MCOs/PIHPs to submit their lists of members in their
hybrid sample twice - the first list as a preliminary submission and the second list as a final
submission. Submitting a first list prior to completion of the MRR process would allow an
MCO/PIHP additional time to retrieve medical record documentation. Soon after receipt of the
first list, the EQRO will provide the MCO/PIHP with the list of medical records for which
documentation must be submitted. Only a portion of the 30 medical records for the validation
sample will be included in the EQRO’s first sample request list. The remainder of the 30 records
will be selected from the final list. While the first submission of MRR findings is optional, it is
recommended.
The EQRO would accept the first list submission approximately one month prior to the
scheduled audit. If an MCO/PIHP chooses to submit a first list of medical records, it must still
submit a final listing sufficiently in advance of the scheduled audit as directed by the EQRO. For
each submission, MCOs/PIHPs will need to identify all members for whom MRR has been
conducted and indicate which members have been found to be numerator positives through
MRR. The final list must reflect the MCO’s/PIHP’s final medical record review findings, with
members for whom a medical record was never found identified as not having met the numerator
requirements.

56

ATTACHMENT XII
No predetermined “passing” grade will be set for the medical record audit. Rather, onsite
auditors will use the MRR results to determine if the hybrid rate or solely MRR rate, as a whole,
is biased, and to what extent that bias affects the final reported rate for that measure. The EQRO
will identify to the State what effects bias, as well as incomplete data, will have on the
MCO’s/PIHP’s calculation of the performance measure. For each of the evaluated measures
auditors will determine the impact of the findings from the MRR validation process on the
MCO’s/PIHP’s Final Audit Designation.
Step 1: Calculation of the Medical Record Review Error Rate
The EQRO will review up to 30 records identified by the MCO/PIHP as meeting numerator
requirements (as determined through MRR) for the measures audited. Records are randomly
selected from the entire population of MRR numerator positives identified by the plan, as
indicated on the MRR numerator listings submitted to the EQRO. If fewer than 30 medical
records are found to meet numerator requirements, all records are reviewed. Administrative
numerator positives are not included as part of this validation process. The EQRO will calculate
a MRR error rate for each performance measure calculated by the hybrid method or solely from
MRR as illustrated in TABLE 4, below:
TABLE 4: Summary of Medical Record Review (MRR) Reabstraction Findings:
Column A

Column B

Column C

Column D

Column E

Column F

Performance
Measure

Number of MMR
Positives Selected
for Audit

Number of
Medical Records
Received

Number of Medical
Records Found to
be Compliant

Accuracy Rate
(%) (D/B)

Error Rate (%)
(100% - E)

Column A:
Column B:

Column C:

Column D:

Name of performance measure evaluated.
Total number of MRR numerator positive records reabstracted by EQRO as part
of the medical record review validation process (i.e., 30, or the total population, if
less than 30 MRR numerator positives were reported).
Total number of medical records submitted to EQRO, as part of the medical
record review validation process (i.e., should be equal to Column B or less than
Column B if one or more records were not submitted on time).
Total number of medical records reviewed by EQRO and identified as meeting
numerator requirements.
57

Column E:
Column F:

ATTACHMENT XII
Accuracy rate - percent of records selected for audit that were identified as
meeting numerator requirements (Column D/Column B).
Error rate - percent of records selected for audit that were identified as not
meeting numerator requirements (100% - Column E).

Step 2: Determining the Potential Impact of MRR Reabstraction Findings On Final Audit
Designations
The next step in MRR validation is to determine whether any medical record review errors
significantly biased the final reported rate for a given performance measure. To make this
determination, the EQRO, as directed by the State, should develop and follow decision rules
such as the following:
Sample Decision Rules:
Error Rate of 10 Percent or Less: If the error rate (TABLE 4, column F) is 10 percent or less,
then the measure automatically passes the MRR validation. The Final Audit Designation is then
determined based on the auditors’ findings from the ISCA conducted as Pre-Onsite activity 3 and
Onsite Activity 1. As long as no errors leading to significant bias are discovered during the other
components of the audit process, the final rate is considered as having met the validation
standards.
Error Rate of Greater than 10 Percent: If the error rate (TABLE 4, column F) is greater than 10
percent, then the auditors determine the impact of the MRR validation findings on the final
reported rate for the measure. For each of the measures under review, auditors evaluate the
impact of the MCO’s/PIHP’s MRR processes on its final reported rate by extrapolating the
findings from the audited medical record sample to the universe of all MRR positives. Details on
this process are provided in TABLE 5.
The maximum amount of bias allowed for the final rate to be considered reportable is “x”
percentage points (to be determined by each State).
-

-

If the amount of error in the MCO’s/PIHP’s MRR process (TABLE 5, line 8) does not
cause the final reported rate to be biased by more than x percentage points, then the
measure passes the MRR validation. The compliance designation is then determined
based solely on the auditors’ findings from the ISCA. As long as no errors leading to
significant bias are discovered during the other components of the performance measure
audit process, the final rate is considered valid.
If the amount of error in the MCO’s/PIHP’s medical review process (TABLE 5, line 8)
ultimately causes the final reported rate to be biased by more than x percentage points,
58

ATTACHMENT XII
the rate is automatically considered invalid. The performance measure is then designated
as invalid.

59

ATTACHMENT XII

TABLE 5: Impact of MRR Findings
Line # Description
Final Data Collection Method Used
1
(e.g., MRR, hybrid,)
Error Rate (Percentage of records
2
selected for audit that were identified
as not meeting numerator requirements,
as shown in TABLE 4, column F)
Is error rate < 10%? (Yes or No)
3
--If yes, MCO/PIHP passes MRR
validation; no further MRR
calculations are necessary.
--If no, the rest of the spreadsheet will
be completed to determine the impact
on the final rate.
Denominator
4
(The total number of members
identified for the denominator of this
measure, as identified by the
MCO/PIHP)
Weight of Each Medical Record
5
(Impact of each medical record on the
final overall rate; determined by
dividing 100% by the denominator in
line 4.)
Total Number of MRR Numerator
6
Positives identified by the MCO/PIHP
using MRR.
Expected Number of False Positives
7
(Estimated number of medical records
inappropriately counted as numerator
positives; determined by multiplying
the Error Rate in line 2 by line 6, the
total number of MRR numerator
positives reported.)
Estimated Bias in Final Rate
8
(The amount of bias caused by medical
record review, measured in percentage
points; determined by multiplying the
Expected Number of False Positives in
line 7 by line 5, the Weight of Each
Medical Record.)

Measure A

Measure B

Measure C

If line 8 is x%, then the final rate is considered to be significantly biased. The measure will be considered invalid

60

ATTACHMENT XIII

Numerator Validation Findings - Reviewer Worksheet

Audit Element

Met

Not Met

N/A

Comments

All appropriate data are used to identify the entire at-risk population.
The MCO/PIHP has used the
appropriate data, including linked data
from separate data sets, to identify the
entire at-risk population.
The MCO/PIHP has in place and
utilizes procedures to capture data for
those performance indicators that could
be easily under-reported due to the
availability of services outside the
MCO/PIHP.

Qualifying medical events (such as diagnoses, procedures, prescriptions, etc.) are properly
identified and confirmed for inclusion in terms of time and services
The MCO’s/PIHP’s use of codes used
to identify medical events are
complete, accurate, and specific in
correctly describing what has
transpired and when.
The MCO/PIHP correctly evaluated
medical event codes when classifying
members for inclusion or exclusion in
the numerator.
The MCO/PIHP has avoided or
eliminated all double-counted members
or numerator events.
Any non-standard codes used in
determining the numerator have been
mapped to a standard coding scheme in
a manner that is consistent, complete,
and reproducible as evidenced by a
review of the programming logic or a
demonstration of the program.
Any time parameters required by the
specifications of the performance
measure are adhered to (i.e., that the
measured event occurred during the
time period specified or defined in the
performance measure).

61

ATTACHMENT XIII
Audit Element

Met

Not Met

N/A

Comments

Medical record data extracted for inclusion in the numerator are properly collected.
Medical record reviews and
abstractions have been carried out in a
manner that facilitates the collection of
complete, accurate, and valid data.
Record review staff have been
properly trained and supervised for the
task.
Record abstraction tools require the
appropriate notation that the measured
event occurred.
Record abstraction tools require
notation of the results or findings of
the measured event (if applicable).
Data included in the record extract
files are consistent with data found in
the medical records as evidenced by a
review of a sample of medical record
for applicable performance measures.
(From Medical Record Review
Validation Tools-Table 5,
ATTACHMENT XII)
The process of integrating
administrative data and medical record
data for the purpose of determining the
numerator is consistent and valid.

62

ATTACHMENT XIV

Policies, Procedures, Data, and Information Used to Implement Sampling:
Review Worksheet

Documents

Reviewed Not Reviewed

Comments

Description of software used to execute
sampling sort of population files when
sampling (systematic) is used.
Policies to maintain files from which the
samples are drawn in order to keep
population intact in the event that a
sample must be re-drawn, or replacements
made.
Computer source code or logic identifying
specified sampling techniques, and
documentation that the logic matches the
specifications set forth for each
performance measure, including sample
size and exclusion methodology.
Methods used for sampling for measures
calling for hybrid data or medical record
review.
Documentation assuring that sampling
methodology treats all measures
independently, and that there is no
correlation between drawn samples.
Observation of or documentation of
procedures in which a biased sample was
identified and corrected.
Documentation of “frozen” or archived
files from which the samples were drawn,
and if applicable, documentation of the
MCO’s/PIHP’s process to re-draw a
sample or obtain necessary replacements.

Describe Documentation Review and Demonstrations Provided:

63

ATTACHMENT XV

Sampling Validation Findings - Reviewer Worksheet

Audit Element

Met

Not Met

N/A

Comments

The MCO/PIHP has followed the specified sampling method to produce an unbiased sample
which is representative of the entire at-risk population.
Each relevant member or provider had
an equal chance of being selected; no
one was systematically excluded from
the sampling.
The MCO / PIHP followed the
specifications set forth in the
performance measure regarding the
treatment of sample exclusions and
replacements, and if any activity took
place involving replacements of or
exclusions from the sample, the
MCO/PIHP kept adequate
documentation of that activity.
Each provider serving a given number
of enrollees had the same probability
of being selected as any other provider
serving the same number of enrollees.
The MCO/PIHP examined its sampled
files for bias, and if any bias was
detected, the MCO/PIHP is able to
provide documentation that describes
any efforts taken to correct it.
The sampling methodology employed
treated all measures independently,
and there is no correlation between
drawn samples.
Relevant members or providers who
were not included in the sample for the
baseline measurement had the same
chance of being selected for the
follow-up measurement as providers
who were included in the baseline.

64

ATTACHMENT XV

Audit Element

Met

Not Met

N/A

Comments

The MCO/PIHP maintains its performance measurement population files/ data sets in a
manner which allows a sample to be re-drawn, or used as a source for replacement.
The MCO/PIHP has policies and
procedures to maintain files from
which the samples are drawn in order
to keep the population intact in the
event that a sample must be re-drawn,
or replacements made, and
documentation that the original
population is intact.

Sample sizes collected conform to the methodology set forth in the performance measure
specifications, and the sample is representative of the entire population.
Sample sizes meet the requirements of
the performance measure
specifications.
The MCO/PIHP has appropriately
handled the documentation and
reporting of the measure if the
requested sample size exceeds the
population size.
The MCO/PIHP properly oversampled
in order to accommodate potential
exclusions.

For performance measures which include medical record reviews (e.g., hybrid data
collections methodology), proper substitution methodology was followed.
Substitution applied only to those
members who met the exclusion
criteria specified in the performance
measure definitions or requirements.
Substitutions were made for properly
excluded records and the percentage of
substituted records was documented.

END OF DOCUMENT

65