CMS-R-305 final Calulating PM

External Quality Review of Medicaid MCOs and Supporting Regulations in 42 CFR 438.360, 438.362, and 438.364 (CMS-R-305)

final Calulating PM

External Quality Review of Medicaid MCOs and Supporting Regulations in 42 CFR 438.360, 438.362, and 438.364

OMB: 0938-0786

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 0938-0786 can be found here:

Document [pdf]

Download: pdf | pdf

OMB Approval No.
0938-0786

CALCULATING PERFORMANCE
MEASURES

A Protocol for use in Conducting Medicaid External Quality
Review Activities

Department of Health and Human Services
Centers for Medicare & Medicaid Services

Final Protocol
Version 1.0
May 1, 2002
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid
OMB control number. The valid OMB control number for this information collection is 0938-0786. The time required to complete this
information collection is estimated to average 1,591 hours per response for all activities, including the time to review instructions, search existing
data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the
time estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports Clearance Officer,
Baltimore, Maryland 21244-1850.
Form CMS-R-305

CALCULATING PERFORMANCE MEASURES
I.

PURPOSE OF THE PROTOCOL

This protocol specifies activities to be undertaken by an external quality review organization
(EQRO) 1 in order to:
1.

Calculate measures of Managed Care Organization (MCO) or Prepaid Inpatient Health
Plan (PIHP) performance in accordance with specifications prescribed by the State
Medicaid agency; and

Provide information to the State on the extent to which the MCO’s/PIHP’s Information
Systems (ISs) provided accurate and complete information necessary for the calculation
of performance measures.

II.

ORIGIN OF THE PROTOCOL

This protocol was derived from protocols and tools commonly used in the public and privatesectors for auditing performance measures. These include:
-

the National Committee for Quality Assurance’s (NCQA) 1999 Health Plan
Employers Data and Information Set (HEDIS)® publication: Volume 5, HEDIS
Compliance Audit™ Standards and Guidelines;
tools used by the Island Peer Review Organization (IPRO) in their audits of
HEDIS measures for Medicare; and
documents from the MEDSTAT Group, Inc., published in conjunction with work
performed in 1997 and 1998 for the Centers for Medicare & Medicaid Services
(CMS) (formerly the Health Care Financing Administration (HCFA)).

A review of these tools found that, while there were differences, these documents had much in
common.
Both NCQA’s and IPRO’s documents address the validation of HEDIS measures only. They
assess:
-

the structure and integrity of the MCO’s/PIHP’s underlying information system
(IS);
MCO/PIHP ability to collect valid data from various internal and external
sources;

It is recognized that a State Medicaid agency may choose an organization other than an EQRO as defined
in Federal regulation to calculate Managed Care Organization (MCO) prepaid inpatient health plan (PIHP)
performance measures. However, for convenience, in this protocol we use the term, “EQRO” to refer to any
organization that calculates performance measures.

vendor (or subcontractor) data and processes, and the relationship of these data
sources to those of the MCO/PIHP;
MCO/PIHP ability to integrate different types of information from disparate data
sources into a data repository or set of consolidated files for use in constructing
MCO/PIHP performance measures; and
documentation of the MCO’s/PIHP’s processes to: collect appropriate and
accurate data, manipulate those data through programmed computer queries,
internally validate the results of the operations performed on the data sets, follow
specified procedures for calculating the specified performance measures, and
report the measures appropriately.

The MEDSTAT publications focus primarily on validation of encounter-level data, and the use
of those data in Medicaid MCO performance measures, regardless of whether the performance
measures are based on the NCQA Medicaid HEDIS measures or have been developed by other
groups or organizations. However, the MEDSTAT publications do not provide detailed
instructions or guidelines that an EQRO might use to validate the MCO/PIHP performance
measures once the encounter data are validated.
The protocol presented here is consistent with the approaches used in the IPRO and NCQA
documents, but is designed with a MEDSTAT-like approach in that it describes how to calculate
all performance measures - HEDIS measures as well as non-HEDIS measures. It varies from the
IPRO and NCQA protocols in that one component of performance measure calculation may be
performed as a part of this protocol or accomplished through some other mechanism(s) used by
the State. Specifically, an assessment of the MCO’s/PIHP’s IS is required as part of this
protocol. This IS assessment may be conducted as a part of this protocol by the EQRO
calculating the performance measures, or the EQRO may review an assessment of the MCO’s
/PIHP’s IS conducted by another party.

III.

OVERVIEW OF THE PROTOCOL

The protocol assumes that the State Medicaid agency will prescribe: 1) the performance
measures to be calculated by the EQRO; 2) the specifications and methodology to be followed in
calculating the measures; and 3) the format and mechanisms for reporting these measures to the
State. Protocol activities include:

Determining the extent to which the MCO’s or PIHP’s IS is capable of collecting
and integrating data from all components of its network, in order to enable valid
measurement of its performance on dimensions of care specified by the State;

Validly measuring MCO/PIHP performance on the dimensions specified by the
State through adherence to technical specifications defined by the State;

Timely reporting to the State the specified performance measures in the format
defined by the State; and

Reporting the findings of the EQRO activities in a manner that facilitates
understanding of the MCO’s/PIHP’s performance against any State-established
minimum levels for performance.

The protocol consists of three phases: Pre-Onsite, Onsite and Post-Onsite activities. For each of
the three audit phases, the protocol specifies outcomes or objectives and lists the activities to be
performed. Methods of evaluation are suggested and tools and worksheets are provided
throughout the protocol and as attachments to the protocol.
Pre-Onsite activities are directed to assessing the MCO’s/PIHP’s capabilities to collect and
integrate complete and accurate medical, financial, member, and provider information, covering
both clinical and service-related data, from internal and external sources. Data in these areas are
frequently needed to validly calculate performance measures. In general, these activities include:
1.

Communicating with the State to ensure that the EQRO understands the measures to be
calculated, specifications and any other methodological instructions to be followed when
calculating each measure, and the required format for reporting calculated performance
measures to the State.

Either conducting an assessment of, or reviewing the results of a prior assessment of, the
MCO’s/PIHP’s underlying IS.

Onsite activities focus on: 1) following-up on IS findings identified in the Pre-Onsite activities
as being potentially problematic or in need of further review or clarification; and 2) validly
calculating the State-mandated performance measures according to the State’s specifications.
These activities involve:
1.

Reviewing and assessing the policies and procedures an MCO/PIHP has in place for
collecting and integrating medical, financial, member and provider information, covering
both clinical and service-related data, from internal and external sources; and

Calculating denominators, numerators and performance measurement rates whether using
an administrative, hybrid, or medical record review methodology.

Post-Onsite activities focus on the submission of the performance measure calculations and
supporting documentation to the State. Activities include:
1.

Evaluating gathered information and preparing preliminary findings;

Submitting preliminary findings to the MCO/PIHP for review prior to submission to the
State;
3

Evaluating gathered information and preparation of findings for the State; and

Submitting reports to the State.

IV.

PROTOCOL ACTIVITIES

PRE-ONSITE ACTIVITIES
Objectives for Pre-Onsite Activities:
The EQRO will:
-

Understand the technical specifications for each performance measure required by
the State;

Understand the State’s requirements for performance measure reporting by the
EQRO to the State (e.g., report template, electronic submission format, etc.); and

Conduct and review an assessment (or review the results of a previously
conducted assessment) of the MCO’s/PIHP’s IS.

PRE-ONSITE ACTIVITY 1:

Review the State’s requirements for performance
measurement and reporting.

The EQRO will need to obtain from the State a list of all performance measures that the State
requires the EQRO to produce. The EQRO will also need to obtain the State’s instructions
(specifications) on how the EQRO is to calculate each performance measure.
The specific performance measures that a State requires its EQRO(s) to calculate will depend on
a number of factors unique to each State. If a State chooses to use a set or subset of established
standardized MCO/PIHP performance measures, there are a number of options from which to
choose. These include the NCQA’s HEDIS measures, measures identified by the Foundation for
Accountability (FACCT), measures found in the Agency for Healthcare Research and Quality’s
(AHRQ’s) CONQUEST database, or measures suggested by MEDSTAT in its publication, A
Guide for States to Assist in the Collection and Analysis of Medicaid Managed Care Data 2 . In
addition, States with the resources and expertise to develop and test the detailed specifications
necessary for valid and reliable performance measures may establish their own performance
measures. Regardless of the type or number of performance measures chosen by the State, the
EQRO must understand the State’s specifications (e.g., sampling guidelines, instructions for
calculating numerators and denominators) for each performance measure, as well as the State’s
instructions for reporting the required performance measures to the State.
2

Prepared under CMS Contract #500-92-0035. December 1998.

Four basic data collection methodologies typically are used to produce MCO/PIHP performance
measures: 1) use of administrative data, 2) review of medical records, 3) use of administrative
data together with medical record review (commonly called the “hybrid” methodology), and 4)
use of surveys.
Use of administrative data requires access to data contained in MCO/PIHP management
information system(s) to calculate both the denominator and numerator of a given performance
measure. Such data includes encounter or claims data (transaction data) as well as other
automated enrollee and provider information. The rate that is reported is based on information
found solely in these administrative data sources.
Calculating performance measures from medical record review requires the visual inspection of
the medical records of a sample of MCO/PIHP enrollees (denominator) to determine if each
enrollee received the service(s) in question (typically, this is the numerator of the performance
measure). Because medical record reviews are time-consuming and costly, most developers and
users of performance measures are attempting to use, to the extent feasible, performance
measures that can be calculated from administrative data. If medical record review is
unavoidable, the less costly and less burdensome “hybrid” methodology can be used.
The hybrid methodology combines the use of administrative data with a review of medical
records. The denominator of the measure is first identified using administrative data for a sample
of eligible members. The numerator is then determined using data from both administrative and
medical record reviews. Typically, MCO/PIHP administrative data is first queried for evidence
of the numerator event for all individuals included in the denominator sample. For any member
of the sample who is missing an administrative notation that the numerator service was received,
the medical record is reviewed.
Finally, surveys also are used to produce MCO/PIHP performance measures. Surveys may
include information collected directly from enrollees, relatives, primary caregivers of enrollees,
or providers of healthcare services. Administration and validation of surveys are complex
subjects and are discussed in separate external quality review (EQR) protocols.
States may require or allow EQROs to report performance measures to the State in different
ways. A State may choose to have MCO/PIHP performance measures reported to it in an
electronic format, such as a comma-delimited, ASCII file; or it may establish a set of electronic
reporting “shells” that EQROs fill out and send to the State. States could also allow hardcopy
submission of calculated performance measures.
States will also determine the timing of the submission of the calculated performance measures.
Typically, States require performance measures to be calculated and submitted annually. The
annual submissions may be timed to coincide with the end of the State fiscal year, the calendar
year, or another reporting cycle, such as the reporting cycle used by NCQA for HEDIS
submissions. It is incumbent on the EQRO to understand the expected dates and report format
for performance measure reporting.
5

To facilitate its calculation of performance measures, the EQRO should create a “List of
Performance Measures to be Calculated” (such as that shown in TABLE 1 below) in order to
understand: 1) the measures required by the State, 2) which method or methods the State allows
the EQRO to use to calculate the measures, and 3) the reporting frequencies and format
mandated by the State.

TABLE 1
List of Performance Measures to be Calculated (EXAMPLE)
METHOD FOR CALCULATING PERFORMANCE MEASURES

SAMPLE MEASURES

Administrative
Data

Medical
Record
Review

Hybrid

Survey

Reporting Frequency and
Format

The table should have a row for
each measure to be calculated
and reported by the EQRO, as
illustrated below:

Childhood immunization rate
Adolescent immunization rate
Percentage of enrollees with at
least one PCP visit
Lead screening rate
Breast cancer screening rate
Initiation of prenatal care
Comprehensive diabetes care
Availability of language
interpretation services
Follow-up after hospitalization
for mental illnesses
Women’s chlamydia screening
rate
Rate of adverse asthma events

For each performance measure in the list, the EQRO should construct a companion performance
measurement worksheet that contains the calculation elements and State-mandated specifications
for a given measure. The elements of performance measure calculation include the following:
6

Data collection
methodology:

Measurement plans and programming specifications that
include data sources, programming logic, computer source
code

Sampling
methodology (if used):

Specifications for sample size and replacement
methodologies

Denominator:

Appropriate and complete data sources used (e.g., claims
files, medical records, provider files, pharmacy records)
Denominator components such as member ID, age, gender,
continuous enrollment calculation, clinical codes such as
ICD-9, CPT-4, member months calculation, member years
calculation, and adherence to specified time parameters.

Numerator:

Data sources used (e.g., member ID, claims files, medical
records, provider files, pharmacy records, including those
for members who may have received the services outside
the MCO’s/PIHP’s network)
Numerator components such as clinical codes (such as
ICD-9, CPT-4, DSM-IV), pharmacy data, relevant time
parameters such as admission/discharge dates or treatment
start and stop dates, adherence to specified time
parameters, number or type of provider.
If medical record abstraction included, the
documentation/tools used

Calculated rates

Each of these components should be customized to include appropriate and specified measure
elements, as defined by the State-mandated performance measure. An example of a completed
Performance Measure Calculation Worksheet for a performance measure of Breast Cancer
Screening is contained in TABLE 2, below.

TABLE 2
Completed Example of a Performance Measure Calculation Worksheet
Note: This worksheet assumes that the State has adopted the HEDIS methodology for this
performance measure.
PERFORMANCE MEASURE TO BE CALCULATED: BREAST CANCER SCREENING
METHODOLOGY FOR
CALCULATING MEASURE
(check one):

PERFORMANCE
MEASURE
ELEMENT
DENOMINATOR

ADMINISTRATIVE

MEDICAL RECORD
REVIEW

HYBRID

PERFORMANCE MEASURE SPECIFICATIONS

1. Population

2. Geographic Area

3. Age & Sex

4. Enrollment
Calculation

5.Data Quality

6. Proper Exclusion
Methodology in
Administrative Data
(if no Exclusions were
taken, check NA)

Medicaid population appropriately segregated from
commercial/Medicare
Population defined as effective Medicare enrollment as of Dec. 31, 2000
Dual Medicaid and Medicare beneficiaries are included
Includes only those Medicaid enrollees served in the MCO’s/PIHP’s
Medicaid service and reporting area.
Members aged 52-69 as of 12/31/00 (i.e., born between 1/1/31 &
12/31/48
Only females selected
Was member of plan on 12/31/00
Was continuously enrolled from 1/1/99 to 12/31/00 with one break, per
year, of up to 45 days allowed
Switches between populations (Medicare, Medicaid, and commercial)
are not counted as breaks
Based on the IS process audit findings, are any of the data sources for
this denominator inaccurate?
Only members with contraindications or data errors may be excluded.
Contraindication exclusions are allowed only as per current State
specifications
Only the codes listed in specifications defined by State are counted as
contraindications

NUMERATOR
7. Administrative
Data: Counting
Clinical Events

9. Time Period

Record abstraction tool requires notation of the date that the mammogram was
performed
Record abstraction tool requires notation of the mammogram result or finding
Mammogram performed on or between 1/1/99 & 12/31/00

10. Data Quality

Properly identify enrollees

Based on the IS process audit findings, are any of the data sources used for this
numerator inaccurate?

As specified in State specifications, systematic sampling method is utilized
After exclusions, sample size is equal to (1) 411, (2) the appropriately reduced
sample size, which used the current year’s administrative rate or preceding
year’s reported rate, or (3) the total population
Only exclude members for whom medical record review revealed (1)
contraindications that correspond to the codes listed in Table XX or (2) data
errors. (Intended to reference appropriate specifications as defined by State)

8. Medical Review
Documentation
Standards

Utilize the standard codes listed in State specifications or properly map all
internally developed codes. (Intended to reference appropriate specifications as
defined by State)
Members are counted only once; double counting of mammograms is prevented

SAMPLING
11. Unbiased Sample
12. Sample Size

13. Proper
Substitution
Methodology in
Medical Record
Review (If no
exclusions were taken,
check NA)

Substitutions are made for properly excluded records and the percentage of
substituted records is documented

CALCULATED RATE =

PRE-ONSITE ACTIVITIES 2:

Prepare the MCO/PIHP for EQRO Onsite Activities.

Prior to conducting onsite activities, the EQRO will contact the MCO/PIHP in order to:
-

explain the procedures and time line for performance measure calculation activities;

request identification of personnel within the MCO/PIHP who will be responsible for
responding to EQRO requests for documentation or information, as well as scheduling
activities and interviews; and

communicate the EQRO’s policies and procedures with respect to safeguarding
confidential information.

An introductory letter to the MCO/PIHP should discuss the above issues and explain the
EQRO’s potential need to interview MCO/PIHP personnel, so that interviewees are prepared in
terms of time and information. Potential interviewees include any MCO/PIHP or vendor staff
whose areas of expertise or responsibility relate to performance measurement and whose insights
might improve the EQRO’s understanding of MCO/PIHP processes to collect and integrate the
information necessary for calculating performance measures. These include, for example: the
Director of Health/Medical Information Systems, IS programmers or operators, Director of
Member/Patient Services, Director of Utilization Management, and the Director of Quality
Improvement.
Also, the EQRO will provide three other documents to the MCO/PIHP in preparation for its
onsite activities:
1. a list and description of all State-required performance measures to be calculated by
the EQRO. A completed Table 1 should be sent to the MCO/PIHP.
2. an Information Systems Capabilities Assessment Tool (ISCA)
The EQRO will send an ISCA tool to the MCO/PIHP, to be completed and returned to the
EQRO prior to the onsite visit. The ISCA consists of questions and requested documentation to
provide the EQRO with background information on the MCO’s/PIHP’s policies and processes
pertaining to data collection and integration that are necessary for calculating performance
measures. The ISCA is discussed in detail, in Pre-Onsite Activity 3. A recently conducted ISCA
by another party can be used.
3. a list of documents that the EQRO may potentially review during onsite activities.
The EQRO also will forward to the MCO/PIHP a list of documents that the EQRO might review
during the course of analyzing and understanding ISCA findings. This list is intended to assist
10

the MCO/PIHP in preparing for the calculation of performance measures by the EQRO. This list
is found as Attachment I.
PRE-ONSITE ACTIVITY 3:
system.

Assess the integrity of the MCO’s/PIHP’s information

Complete and accurate data is key to valid and reliable performance measurement. If these two
data characteristics are not maintained, then calculated measures are at risk of being biases, and
their validity jeopardized. Therefore, prior to calculating individual performance measures, the
EQRO must first have knowledge of the integrity of the MCO’s/PIHP’s IS and the completeness
and accuracy of the data contained in that system.
Methods of Evaluation
Prior to conducting the onsite visit, the EQRO should send to the MCO/PIHP an ISCA such as
that located in Appendix Z. The ISCA asks questions of and requests documentation from the
MCO/PIHP in order to provide information on how the MCO/PIHP collects and integrates data.
This will help the EQRO to calculate performance measures. The ISCA found in Appendix Z
corresponds to the key objectives identified in this protocol. The first section of the ISCA
provides general background information on the MCO/PIHP. Subsequent sections address the
structural components of the IS, focusing on the collection of administrative, encounter, and
clinical data, and the consolidation or coordination of those data files for use in performance
measurement and quality improvement activities.
The ISCA also requests information from the MCO/PIHP concerning the conduct and timing of
any other recent, independent, documented assessment of its IS. An assessment may already
have been conducted by the State itself or by another entity. IS assessment could have been
performed as a component of validating encounter data or determining compliance with
Medicaid standards pertaining to MCO/PIHP ISs. If the MCO/PIHP has not had an IS capability
assessment completed, or has not had one completed within a time frame that meets State
specifications 3 , the EQRO will conduct an IS evaluation as part of this protocol, using an IS
assessment tool, such as that in Appendix Z. Alternatively, if the MCO/PIHP recently had an
evaluation of its IS, the EQRO could review the results of this prior assessment.
The EQRO should assess the MCO’s/PIHP’s IS using questions and approaches such as those
contained in Appendix Z, or review the results of a recent IS assessment consistent with the
content in Appendix Z. This will ensure that auditors are familiar with the strengths and
3

Each State will determine the frequency with which it wants an MCO’s/PIHP’s IS capability assessment
to take place (thereby determining the length of time such an assessment is valid). On the one hand, the process is
time- and resource-intensive, so limiting the burden on the MCO/PIHP should be a factor in the determination. On
the other hand, IS technology changes rapidly, so the State should ensure that changes to an MCO’s/PIHP’s IS are
assessed frequently enough to ensure that the structure and function continue to be adequate for the State-required
tasks.

weaknesses of the MCO’s/PIHP’s IS. As the EQRO reviews the IS assessment report, it should
pay close attention to the strengths and weaknesses of the MCO’s/PIHP’s IS with respect to the
types of data frequently used in MCO/PIHP performance measures, such as data on:
membership/enrollment, providers, claims/encounters, laboratory and pharmacy services, and
medical record data. Some of the characteristics commonly associated with these data elements
that may affect the calculation of performance measures are:
-

Membership/Enrollment Data. Elements of the membership or enrollment database
will vary by MCO/PIHP. However, for the purposes of MCO/PIHP performance
measurement, the membership or enrollment database should capture at least the
following information:
age/date of birth.
enrollment and/or termination dates. (Note: The MCO’s/PIHP’s data system
should be able to track multiple enrollment and termination dates).
primary care provider (e.g., name, provider identification number).
member identification number such as the member’s social security number,
MCO- or PIHP-designated number, State-issued Medicaid number, CMSissued Medicare number. (Note: Be aware of cases in which more than one
member may exist under the same identification number within the system; or
in which the same member may exist under more than one identification
number within the system; or in which a member’s identification number may
change through re-enrollment, name change, or switch in product-line
coverage).

The EQRO also should be aware of whether the MCO/PIHP has processes in place to
periodically ensure that enrollment/membership data are current and accurate, particularly at
the time it runs its source code/computer programs to identify denominators for MCO/PIHP
performance measures.
Further, the EQRO should be aware of changes in the MCO’s/PIHP’s membership data
systems that might affect the production of the MCO/PIHP performance measures. Major
changes, upgrades or consolidations within the system, or acquisitions/mergers with other
MCOs/PIHPs may impact the accuracy or completeness of any of the data elements, which,
in turn, may impact the validity of the reported measures.
-

Provider Data. Elements of the provider data set should typically include:
Designation as a primary care physician and/or providers’ specialty.
Provider identification number, such as a Tax ID number, or MCO- or PIHPdesignated number. (Note: Though it may be less common to see duplication
of provider numbers within a provider database than duplication of member
identifications within a membership/enrollment database, the EQRO should
be aware of any circumstances in which more than one provider can exist with
the same identification number within the system, or circumstances in which
the same provider may have more than one identification number within the
system).
12

Providers with more than one office location.
Providers with closed panels (i.e., provider availability).
Provider start and termination dates.
Provider certification data such as licensure, provider residency/fellowship,
date and specialty of Board Certification status.

The EQRO should be aware of whether the MCO/PIHP has processes in place to
periodically ensure that provider data are current and accurate for all types of providers
(individual providers, provider groups, provider networks, contracted vendors). This
becomes particularly important at the time the MCO/PIHP runs its source code/computer
programs to identify elements of MCO/PIHP performance measures.
Further, the EQRO should be aware of changes in the MCO’s/PIHP’s provider data
systems that might affect the production of the performance measures. Major changes,
upgrades or consolidations within the system, or acquisitions/mergers with other
MCOs/PIHPs may impact the accuracy or completeness of any of the data elements,
which, in turn, may impact the validity of the reported measures.
-

Claims Data and Encounter Data. Claim/encounter data should cover all types of
services offered by the MCO/PIHP, such as: behavioral health, family planning, home
health care, hospital, laboratory, pharmacy, primary care, radiology, specialty care,
vision care. These data typically include the following elements:
- Patient ID
- Sex
- Date of birth
- Last date of service
- Primary diagnosis
- Primary procedure
- Revenue codes
- Provider specialty

- Name
- Age
- First date of service
- Place of service
- Secondary diagnosis
- Secondary procedure
- Provider ID
- Discharge status

For each type of claim/encounter data captured, the EQRO should be aware of: 1) the
total number of diagnosis and procedure codes that can be captured by the system; 2)
whether or not principal or secondary diagnosis or procedure codes can be accurately
distinguished in the system; and 3) the maximum number of digits or characters the
system captures for each type of claim/encounter. For many MCO/PIHP performance
measures, the accuracy and validity of the measure may be adversely affected if the
MCO’s/PIHP’s IS is unable to collect and/or differentiate among a sufficient number of
codes.
The various coding systems and forms used by the MCO/PIHP and its vendors to capture
clinical information through its claims and encounter databases are relevant to validating
MCO/PIHP performance systems. Coding systems are formal, standardized approaches
(such as ICD-9, CPT-4, DSM-IV, revenue codes, or internally developed codes) to
13

categorize types of encounters and procedures by data elements such as inpatient and
ambulatory diagnoses and procedures for medical, surgical, or mental health/substance
abuse encounters/claims. Note that internally-developed codes may be particularly
problematic. The EQRO should understand how the MCO’s/PIHP’s IS translates or maps
these codes back to standard codes for MCO/PIHP performance measure reporting, and
how it ensures the accuracy of these translation processes.
-

Medical Record Data. In cases where medical records are accessed to obtain
information for calculating MCO/PIHP performance measures, the EQRO should be
aware of how the MCO/PIHP retrieves information from medical records. For example,
the training and tools that medical record review staff receive may affect the accuracy
and completeness of the data retrieval and inter-rater reliability. A second area of
concern is how medical record data is entered into any database that will be used to
produce the performance measures.

Pharmacy and Laboratory Data. A key issue commonly encountered with pharmacy
and laboratory data for Medicaid managed care MCOs/PIHPs is that these services are
frequently contracted out to a variety of providers. Ideally, pharmacy data will use
standardized codes for prescription drugs such as those promulgated by the National
Council for Prescription Drug Programs (NCPDP), and laboratory services will use a
similar, nationally recognized system of coding. However, the diverse nature of the size,
type, and ownership of pharmacy and laboratory providers should lead the EQRO to
anticipate wide variations in the use of standardized coding and a multitude of unique
“home grown” codes. These non-standard coding schemes require that the MCO/PIHP
have a system to develop crosswalks among these different codes in order to store the
necessary information in its performance measure database. As with the assessment of
the claims/encounter data systems, the EQRO should understand not only the
MCO’s/PIHP’s system of mapping non-standard pharmacy and lab codes to standardized
codes, but the mechanism the MCO/PIHP uses to ensure the accuracy of these translation
processes.
If pharmacy or laboratory data are not collected through an administrative or claims
database, pharmacy or lab data may be present in medical records. However, relying on
medical records to supply pharmacy or laboratory data is problematic because of
obstacles such as non-standard coding and terminology and poor coordination of records
and record linkages between primary care and specialist providers. The EQRO should be
aware of these issues and question providers on the reliability of medical record data and
pharmacy data as appropriate.
In addition, for many MCO/PIHP performance measures, the IS will need to be able to
link these different sources of data. These linked data sets are used to generate
comprehensive reports and information which is capable of being segmented by member
identification and characteristics, site of delivery, primary and secondary diagnoses,
primary and secondary procedures, and provider identification. For example, in order to
identify enrollees with diabetes, an MCO/PIHP may have to combine diagnosis code data
14

from inpatient or ambulatory encounters (not all ongoing conditions are reported at every
encounter) with pharmacy data, lab data, and/or a disease registry if one exists. To
determine whether these diabetic enrollees have received a retinal examination from an
ophthalmologist or optometrist within the previous year, the MCO/PIHP would have to
link procedure code data from either encounter forms, medical records, or claims with
information about the specialty of the providers that performed the examinations for
these members.
The EQRO will analyze the results of the assessment of the MCO’s/PIHP’s IS and
determine the implications of the findings for the calculation of the performance
measures specified by the State. The EQRO will evaluate MCO/PIHP answers against IS
capabilities necessary to accurately and completely calculate and report the specific
MCO/PIHP performance measures mandated by the State, and will identify any problem
areas or items in need of clarification. Where an answer seems incomplete, or indicates
an inadequate process, the EQRO notes this issue for follow-up and further review during
the onsite activities. This will help the onsite activities focus on the areas most likely to
be an issue in calculating performance measures.

ONSITE ACTIVITIES
Key Outcomes and Objectives
-

The EQRO will validate that the MCO/PIHP has adequate data integration and control
necessary for accurate reporting of performance measures.

The EQRO will completely and accurately document data and processes used to collect,
calculate, and report performance measures.

The EQRO will appropriately and correctly implement processes to calculate and report
MCO/PIHP performance measures.

ONSITE ACTIVITY 1: Assess Data Integration and Control Necessary for Accurate
Calculation of Performance Measures
Methods of Evaluation
The emphasis of this activity is not whether the MCO/PIHP is capable of performing the data
integration and control necessary for collecting the performance measures. Rather, the emphasis
is on determining whether the MCO/PIHP has utilized those proven capabilities in a manner that
assures that the MCO/PIHP can reliably and validly capture the entire population without
systematically excluding a subset or subsets of the entire population. In this way, the EQRO can
assure that calculations based on those data sets are also reliable and valid.
15

In Pre-Onsite Activity 1, the EQRO confirms that the MCO/PIHP’s IS has the capacity to collect
valid data from sources internal to the organization as well as those external to the organization.
This first onsite activity assesses the MCO’s/PIHP’s capability of linking the data from multiple
sources in order to proceed with the calculation of the State-mandated performance measures.
During this activity, the EQRO will:
-

Examine the details of the MCO’s/PIHP’s processes to accurately and completely
transfer data from the transaction files (i.e., membership, provider, encounter/claims)
into the repository used to keep the data until the calculations of the performance
measures have been completed and validated.

Examine samples of data to assess completeness and accuracy.

Investigate the MCO’s/PIHP’s processes to consolidate diversified files, and to extract
required information from the performance measure repository.

Compare actual results of file consolidations or extracts to those which should have
resulted according to documented algorithms or specifications.

Review procedures for coordinating the activities of multiple subcontractors in ways that
ensure the accurate, timely, and complete integration of the data into the performance
measure data base.

Review computer program reports or documentation that reflect these vendor
coordination activities, and spot check to verify that no data necessary to performance
measure reporting are lost or inappropriately modified during transfer.

If the MCO/PIHP uses one, evaluate the structure and format of the performance
measure data repository (or data warehouse), and examine program flow charts to
determine the extent to which the repository/warehouse enables analyses and reports.

Assess the extent to which proper linkage mechanisms have been employed to join data
from all necessary sources (e.g., identifying a member with a given disease/condition).

Examine program flow charts and source code to assess the extent to which the data
repository/warehouse has enabled analyses and report preparation.

Potential interviewees in support of this activity might include: the Director of Health/Medical
Information Systems, system programmers or operators, and selected sub-contractors.

Tools and Worksheets
-

Attachment II: MCO/PIHP Documentation for Review Worksheet (Onsite Activity 1)
Attachment III: Interview Guide Background Information and Data Integration and
Control Worksheet (Onsite Activity 1)
Attachment IV: Data Integration Necessary for Accurate Reporting of Performance
Measures Worksheet (Onsite Activity 1)
Performance Measure Calculation Worksheets as designed by EQRO during Pre-Onsite
Activity 1 (Table 2). These will differ for each State, depending on the performance
measures mandated, and the specifications or definitions used by the individual State for
reporting.

ONSITE ACTIVITY 2: Assure complete and accurate documentation of data and
processes used to collect, calculate and report performance
measures
Methods of Evaluation
In the context of this protocol, documentation includes all elements of the production process,
beginning with the data collection from various sources (i.e., membership, enrollment, provider,
claims, or encounter records; medical records; laboratory and/or pharmacy records; consumer
survey results; or MCO/PIHP financial information). It includes the steps taken to integrate the
required data into a performance measure data set or data repository, as well as procedures or
programs that may be implemented to query the data set/data repository to identify
denominators, generate appropriate samples, determine numerators, and apply proper algorithms
to the data in order to produce valid and reliable performance measures.
During this activity the EQRO will:
-

Create or confirm that all measurement plans and policies include:
-

Data file and field definitions used for each measure.

Maps to standard coding if standard codes were not used in original data collection.

Statistical testing of results, and any corrections or adjustments made after
processing.

Develop documentation (which may be either a schematic diagram or in narrative form)
of programming specifications for each measure, to ensure that they include at least the
following information:
-

A project or measurement plan for each performance measure, including workflow.
17

All data sources for each measure, including external data (whether from a vendor, public
registry, or other outside source), and any prior years’ data (if applicable).

Documentation of the original universe of data that includes the original universe of data
that includes record-level patient identifiers that can be used to validate entire
programming logic for creating denominators, numerators, and samples.

Detailed medical record review methods and practices, including the qualifications of
medical record review supervisor and staff; reviewer training materials; completed copies
of each record-level reviewer determination; all case-level critical performance measure
data elements used to determine a positive or negative event or exclude a case from same;
and inter-rater reliability testing procedures and results.

Detailed computer queries, programming logic, or source code used to create all
denominators, numerators, and samples (if applicable to the measure). For example,
depending on the measure specifications, these could include:
-

Process for identifying the population or sample for the denominator and/or
numerator.

If sampling is used, a description of sampling techniques, and documentation that assures
the reviewer that samples used for baseline and repeat measurements of the performance
measures were chosen using the same sampling frame and methodology.

Documentation of calculation for changes in performance from previous periods (if
applicable), including statistical tests of significance.

There is no suggested MCO/PIHP documentation for review during this activity, nor are there
any specified interviews.
Tools and Worksheets
-

Attachment V: Complete and Accurate Documentation of the Data and Processes used to
Prepare and Submit Performance Measures Worksheet (Onsite Activity 2)
Performance Measure Calculation Worksheets as designed by EQRO during Pre-Onsite
Activity 1 (Table 2). These will differ for each State, depending on the performance
measures mandated, and the specifications or definitions used by the individual State for
reporting.

ONSITE ACTIVITY 3: Assure the validity of processes used to identify denominators of
performance measures
Methods of Evaluation
The core task in calculating the denominator(s) of performance measures is ensuring that the
appropriate data, including linked data from separate data sets, is used to identify the entire atrisk population. The “appropriate data” will vary from measure to measure, depending on
criteria such as age, gender, diagnosis or procedure, and may be adjusted to exclude certain
patients for other reasons specified in the measure. In some cases, the EQRO may have to
estimate portions of the population, such as newborns who cannot be uniquely counted.
During this activity, the EQRO will:
-

Assure that all members who were eligible to receive the specified services were
included in the initial population from which the final denominator was produced. This
“at risk” population will include both members who received the services, as well as
those who did not. This same activity applies to provider groups or other relevant
populations identified in the specifications of each performance measure.

Write or collaborate with MCO/PIHP IS staff to program logic or source code for each
measure that identifies, tracks, and links member enrollment within and across product
lines (e.g., Medicare and Medicaid), by age and gender, as well as through possible
periods of enrollment and disenrollment, in order to appropriately comply with the
specifications of each performance measure.

Correctly carry out and apply calculations of continuous enrollment criteria to each
measure (if applicable).

Properly use mathematical operations that determine patient age or range.

Correctly identify the variable(s) that define the member’s gender in every file or
algorithm, and note what classification is carried out if neither of the required codes is
present.

Correctly calculate member months and member years, if applicable to the performance
measure.

Estimate the completeness and accuracy of any codes used to identify medical events,
such as diagnoses, procedures, or prescriptions, and assure that these codes are
appropriately identified and applied as specified in each performance measure.

Adhere to any time parameters required by the specifications of the performance measure
are adhered to (e.g., cut off dates for data collection, counting 30 calendar days after
discharge from a hospital, etc.).
19

Follow performance measure specifications or definitions in excluding members from a
denominator. For example, if a measure relates to receipt of a specific service, the
denominator may need to be adjusted to reflect instances in which the patient refuses the
service or the service is contraindicated.

When appropriate, use valid systems or methods to estimate populations when they
cannot be accurately or completely counted (e.g., newborns).

Potential interviewees in support of this activity might include: the Director of Health/Medical
Information Systems, system programmers or operators, and selected sub-contractors.
Tools and Worksheets
-

Attachment VI: MCO/PIHP Documentation for Review Worksheet (Onsite Activity 3)
Attachment VII: Interview Guide Performance Measures Calculation (Onsite Activities
3-5)
Attachment VIII: Proper Identification of Denominator Worksheet (Onsite Activity 3)
Performance Measure Calculation Worksheets as designed by EQRO during Pre-Onsite
Activity 1 (Table 2). These will differ for each State, depending on the performance
measures mandated, and the specifications or definitions used by the individual State for
reporting.

ONSITE ACTIVITY 4: Assure the validity of processes used to determine numerators of
performance measures (for administrative and hybrid methodologies)
Methods of Evaluation
The primary activity in the calculation of the numerator is correctly identifying and evaluating
qualifying medical events (e.g., diagnoses, procedures, and prescriptions) in order to include the
value in the numerator of the performance measure. These “medical events” may be identified
through membership/enrollment data, claim/encounter data, and/or provider data. They may also
be identified through data extracted from medical records, or through a combination of both
administrative data and medical record abstraction, which is commonly referred to as the
“hybrid” method of data collection.
As with the denominator, appropriate, accurate, and complete data collection is vital to this
element of MCO/PIHP performance calculation. For population-based measures that include
sampling in the methodology, the entire at-risk population must have an equal chance to be
included in the numerator. For some measures, particularly those frequently focused on women
and children in the Medicaid population, the member may have received the specified service
outside of the MCO/PIHP provider base (e.g., children receiving immunizations through public
health services or schools), so an effort must be made to include these events in the numerator.
If a hybrid methodology is chosen for the numerator determination, this component of the
20

protocol may involve the EQRO reviewing a sample of medical records to abstract information
not found in the administrative data set for a given sample of the denominator. Following the
performance measure technical specifications and guidelines, the EQRO will develop a medical
record abstraction tool. It will also develop the mechanisms for assuring that the data are
accurately and completely collected. The actual medical record abstraction activities may vary
from MCO/PIHP to MCO/PIHP, or from State to State. In some cases, the abstraction may be
done using the resources of the MCO/PIHP. In other cases, the EQRO will hire, train, and
oversee the data abstraction activities. However, the MCO/PIHP’s IS staff will be involved in
assuring that the data are properly integrated into the MCO/PIHP’s IS in a way that facilitates
the calculation of performance measures.
During this activity, the EQRO will:
-

Assure the use of appropriate data, including linked data from separate data sets, to
identify the entire at-risk population that meets the specified criteria for inclusion in the
numerator.

Assuring the presence of, or creating, procedures to capture data for those performance
indicators which could be easily under-reported due to the availability of services outside
the MCO/PIHP, and following those procedures.

Confirm that the MCO’s/PIHP’s use of codes used to identify medical events (such as
diagnoses, procedures, prescriptions, etc.) are complete, accurate, and specific in
correctly describing what has transpired and when. More to the purpose of this protocol,
however, is ensuring that these codes are correctly evaluated and applied when
classifying members for inclusion or exclusion in the numerator.

Avoid or eliminate double-counted members or numerator events.

Through a review of the programming logic or a demonstration of the program, confirm
that non-standard codes are mapped to standard codes in a manner that is consistent,
complete, and reproducible.

Adhere to any time parameters required by the specifications of the performance measure
(i.e., that the measured event occurred during the time period specified or defined in the
performance measure).

Confirm that medical record reviews and abstractions are carried out in a manner that
facilitates the collection of complete, accurate, and valid data by assuring that:
-

Record review staff are properly trained and supervised for the task.

Record abstraction tools require the appropriate notation that the measured event
occurred.
21

Record abstraction tools require notation of the results or findings of the measured event
(if applicable).

Ensure that the process of integrating administrative data and medical record data for the
purpose of determining the numerator is consistent and valid.

Attachment VII: Interview Guide Performance Measures Calculation (Onsite
Activities 3-5)
Attachment IX: Proper Determination of Numerator Worksheet (Onsite Activity 4)
Performance Measure Calculation Worksheets as designed by EQRO during PreOnsite Activity 1 (Table 2). These will differ for each State, depending on the
performance measures mandated, and the specifications or definitions used by the
individual State for reporting.

ONSITE ACTIVITY 5: Except for measures calculated through administrative data alone,
assure the validity of processes used to sample the appropriate population for calculation
of performance measures
Methods of Evaluation
The basic task related to the sampling methodology for performance measures is assuring that
the sampled data validly reflect (a) the performance of all practitioners and providers who serve
Medicaid enrollees and whose activities are the subject of the indicator; and (b) the care given to
the entire population (including special populations with complex care needs) to which the
indicator is relevant.
As in the previous activity of validating the population included in a denominator, the sampling
methodology employed should not exclude any population subgroups to which the topic area and
indicators apply. For example, when studying well child care, an MCO’s/PIHP’s sample should
not exclude children with special care needs whose primary care provider is a specialist other
than a pediatrician or family practitioner.
During this activity, the EQRO will:
-

Ensure that the sampling methodology employed produced an unbiased sample which is
representative of the entire at-risk population.
- Assure that each relevant member or provider has an equal chance of being selected;
no one is systematically excluded from the sampling.
- Assure that the specifications set forth in the performance measure regarding the
treatment of sample exclusions and replacements are followed, and that if any activity
22

takes place involving replacements of or exclusions from the sample, that adequate
documentation of that activity is kept.
Assure that each provider serving a given number of enrollees has the same
probability of being selected as any other provider serving the same number of
enrollees.
Examine sample for bias, and if any bias is detected, provide documentation that
describes any efforts taken to correct it.
Assure that the sampling methodology employed treats all measures independently,
and that there is no correlation between drawn samples. (This is not intended to be a
validation of the prescribed sampling methodology included in the performance
measure specifications, since the assumption is that it is a valid methodology. The
EQRO efforts focus on the implementation of that sampling methodology, to assure
that it correctly follows the specifications.)
Confirm that relevant members or providers who were not included in the sample for
the baseline measurement have the same chance of being selected for the follow-up
measurement as providers who were included in the baseline.

With the MCO/PIHP IS staff, develop and implement policies and procedures to maintain
files from which the samples are drawn in order to keep population intact in the event that a
sample must be re-drawn, or replacements made, and documentation that the original
population is intact.

Assure that the sample sizes collected conform to the methodology set forth in the
performance measure specifications.

Assure that sample sizes meet the requirements of the performance measure
specifications.

Appropriately handle the documentation and reporting of the measure if the
requested sample size exceeds the population size.

Assure proper oversampling in order to accommodate potential exclusions.

Follow proper substitution methodology in medical record review (for measures using the
hybrid methodology).
-

Assure that substitution applies only to those members who met the exclusion criteria
specified in the performance measure definitions or requirements.

Assure that substitutions are made for properly excluded records and the percentage
of substituted records is documented.

Potential interviewees in support of this activity might include: the Director of Health/Medical
Information Systems, system programmers or operators, and selected sub-contractors.
23

Tools and Worksheets
-

Attachment VII: Interview Guide Performance Measures Calculation (Onsite
Activities 3-5)
Attachment X: Proper Sampling Techniques Worksheet (Onsite Activity 5)
Performance Measure Calculation Worksheets as designed by EQRO during PreOnsite Activity 1 (Table 2). These will differ for each State, depending on the
performance measures mandated, and the specifications or definitions used by the
individual State for reporting.

ONSITE ACTIVITY 6: Properly submitting required performance measure reports to the
State
Methods of Evaluation
Once the EQRO calculates the required performance measures, it must report them to the State
in the manner prescribed. This includes reporting the measures in a proper format, whether
through the use of a hardcopy “shell” report designed by the State, or in the electronic medium
and format required by the State, or some combination of both. During the pre-onsite phase of
the review, the EQRO familiarizes itself with the State’s requirements of the proper format and
reporting mechanisms for the MCO’s/PIHP’s performance measures.
-

Assure that measures are reported to the State in the manner and form prescribed by
the State

Worksheets and Tools
-

Attachment XI: Proper Submission of Required Reports to State Agency Worksheet
(Onsite Activity 6)
Performance Measure Calculation Worksheets as designed by EQRO during PreOnsite Activity 1 (Table II). These will differ for each State, depending on the
performance measures mandated, and the specifications or definitions used by the
individual State for reporting.

POST-ONSITE ACTIVITIES
Key Outcomes and Objectives
-

The EQRO will evaluate all gathered information and prepare a summary report of
findings for the State

The EQRO will submit its report of the MCO/PIHP’s MCO/PIHP performance
measures to the State

POST-ONSITE ACTIVITY 1: Referring to Pre-Onsite and Onsite information regarding
State processes with the EQRO, complete worksheets and
summarize findings in a report
Methods of Evaluation
Within one month of the onsite activities with respect to calculating the MCO’s/PIHP’s
MCO/PIHP performance measures, the EQRO summarizes its preliminary findings in a report
that will be sent back to the MCO/PIHP for review. The format of the report follows the outline
of the key objectives, and the specific activities associated with each. In its reply, the
MCO/PIHP may offer comments and documentation to support correction of any factual errors
and omissions in the EQRO report.
Once the MCO’s/PIHP’s review is complete, and its comments have been incorporated where
appropriate, the EQRO will submit its findings to the State in a final report that should follow the
format specified by the State.
POST-ONSITE ACTIVITY 2: Submit a final report to the State
This final report format will vary by State, but will probably include the following elements:
General summary of the onsite activities, including a list of the EQRO team members and
performance measurement pre-audit strategy and considerations, a list of measures offered for
calculation (it is possible that an EQRO was unable to calculate on all required measures for
some reason which would be explained to the State), a list of interviewees, and any other facts
relevant to the onsite process.
Details and results of the medical record abstractions conducted as a part of this protocol.
In addition to the final report, the EQRO might also be asked to submit all of its worksheets and
tools as supporting documentation to the report.

END OF PROTOCOL TEXT
25

ATTACHMENT I

Potential Documents For Review
In order to better understand an MCO’s/PIHP’s IS and its implications for calculating
performance measures, the EQRO might need to review a number of data sources and processes.
The MCO/PIHP should ensure that the following documents and data are available for the
EQRO. The EQRO will use its discretion in selecting which ones to review.
Integration and Control of Data for Performance Measurements
-

Procedures and standards for all aspects of data repository, including building,
maintaining, managing, testing, and production of performance measures.
Manuals covering application system development methodology, database
development and design and decision support system utilization.
Control system documentation including flow charts and codes for backups, recovery,
archiving, and other control functions.
Procedures to consolidate information from disparate transaction files to produce
intended result.
Record and file formats and descriptions, for entry, intermediate and repository files.
Electronic formats and protocols.
Electronic transmission procedures documentation.
Processes to extract information from the repository to produce intended result.
Source code data entry, data transfer, and data manipulation programs and processes.
Descriptive documentation for data entry, data transfer, data manipulation programs
and processes.
If applicable, procedures for coordinating activities of multiple subcontractors in a
way that safeguards the integrity of the performance measure data.
Samples of data from repository and transaction files to assess accuracy and
completeness of the transfer process.
Comparison of actual results from file consolidation and data abstracts to those which
should have resulted according to documented algorithms.
Documentation of data flow among vendors to assure proper implementation of
procedures for coordinating activities to safeguard the integrity of the performance
measure data.
Documentation of data cutoff dates.
Documentation of proper run controls and of staff review of report runs.
Copies of files and databases used for performance measure calculation and
reporting.

Collection, Calculation, and Documentation of Performance Measurements
-

Policies to assign unique membership ID that allows all services to be properly
related to the specific appropriate recipient, despite changes in status, periods of
enrollment or disenrollment, or changes across product lines (e.g., Medicare and
26

ATTACHMENT I
-

Medicaid).
Procedures to identify, track and link member enrollment by product line, product,
geographic area, age, gender, member month, and member years.
Procedures to track individual members through enrollment, disenrollment, and
possible re-enrollment.
Procedures to track members through changes in family status, changes in
employment or benefits or managed care type (if they switch between Medicaid
coverage and another product within the same MCO/PIHP).
Methods to define start and cessation of coverage.
Procedures to link member months to member age.
Member database.
Provider data (including facilities, labs, pharmacies, physicians, etc.)
Database record layout and data dictionary.
Survey data.
Procedures for mapping non-standard codes to standard coding to ensure consistency,
completeness, and reproducibility.

ATTACHMENT II
MCO/PIHP Documentation for Review Worksheet (Onsite Activity 1)

Documents
Data Integration and Control
Procedures and standards for all aspects
of data repository, including building,
maintaining, managing, testing, and
production of performance measures.
Manuals covering application system
development methodology, database
development and design and decision
support system utilization.
Control system documentation including
flow charts and codes for backups,
recovery, archiving, and other control
functions.
Procedures to consolidate information
from disparate transaction files to
produce intended result.
Record and file formats and descriptions,
for entry, intermediate and repository
files.
Electronic formats and protocols.

Reviewed

Not Reviewed

Comments

Electronic transmission procedures
documentation.
Processes to extract information from the
repository to produce intended result.
Source code data entry, data transfer,
and data manipulation programs and
processes.
Descriptive documentation for data
entry, data transfer, data manipulation
programs and processes.
If applicable, procedures for
coordinating activities of multiple
subcontractors in a way that safeguards
the integrity of the performance measure
data.
Samples of data from repository and
transaction files to assess accuracy and
completeness of the transfer process.
Comparison of actual results from file
consolidation and data abstracts to those
which should have resulted according to
documented algorithms.
Documentation of data flow among
vendors to assure proper implementation
of procedures for coordinating activities
to safeguard the integrity of the
performance measure data.

ATTACHMENT III
Interview Guide
Background Information

Plan Name:
Date:
Location:
Year of First Medicaid Enrollment:
Year of First Medicare Enrollment:
Year of First MCO/PIHP Performance Report:
Auditors:

Interviewees:

Has the plan ever undergone an audit of its State performance measure reporting process? If so,
when did the audit take place and who conducted it?

Other general issues:

Data Integration and Control (Onsite Activity 1)
1. How is performance measure data collection accomplished:
-

By querying the process system on-line?
By using extract files created for analytical purposes? If so, how frequently are the
files updated? How do they account for claim/encounter submission and processing
lags? How is the file creation process checked for accuracy?
By using a separate relational database or data warehouse? If so, is this the same
system from which all other reporting is produced? Are reports created from a
vendor software product? If so, how frequently are the files updated? How are
reports checked for accuracy?
29

ATTACHMENT III
2. Review the procedure for consolidating claims/encounter, member, and provider data for
performance reporting (whether it be into a relational database or file extracts on a measure
by measure basis).
-

How many different sources of data are merged together to create reports?
What control processes are in place to ensure that this merger is accurate and
complete?
Compare samples of data in the repository to transaction files. Are any members,
providers, or services lost in the process?
Is the required level of coding detail maintained (e.g., all significant digits, primary
and secondary diagnoses remain)?

3. If the plan uses a performance measure repository, review the repository structure. Does it
contain all the key information necessary for performance measure reporting?
4. How does the plan prevent loss of claim and encounter data when systems fail?
5. What administrative data back-up systems are in place?
6. What types of authorization are required to be able to access claims/encounter, provider,
membership, and performance measure repository data?
7. Other issues:

Described Documentation Review and Demonstrations Provided:

ATTACHMENT IV
Data Integration Necessary For Accurate Reporting Of Performance Measures Worksheet
(Onsite Activity 1)
Audit Element

Met

Not Met

N/A

Comments

Accuracy of data transfers to assigned performance measure repository.
MCO’s/PIHP’s processes accurately
and completely transfer data from the
transaction files (i.e., membership,
provider, encounter/claims) into the
repository used to keep the data until
the calculations of the performance
measures have been completed and
validated.
Samples of data to assess completeness
and accuracy.
Accuracy of file consolidations, extracts, and derivations.
Accuracy of MCO’s/PIHP’s processes
to consolidate diversified files, and to
extract required information from the
performance measure repository
Comparison of actual results of file
consolidations or extracts to those
which should have resulted according to
documented algorithms or
specifications.
Procedures for coordinating the
activities of multiple subcontractors
ensure the accurate, timely, and
complete integration of the data into the
performance measure data base.
Computer program reports or
documentation reflect vendor
coordination activities, and no data
necessary to performance measure
reporting are lost or inappropriately
modified during transfer.

Audit Element

Met

Not Met

N/A

Comments

If the MCO/PIHP uses one, the structure and format of the performance measure data repository facilitate any
required programming necessary to calculate and report required performance measures.
Review the repository’s design, and
examine program flow charts to
evaluate the extent to which the
repository enables analyses and reports.
Assess the extent to which proper
linkage mechanisms have been

ATTACHMENT IV
employed to join data from all
necessary sources (e.g., identifying a
member with a given
disease/condition).
Examine program flow charts and
source code to assess the extent to
which the data repository has enabled
analyses and report preparation.
Assurance of effective management of report production, and of the reporting software.
Examine and assess the adequacy of
the documentation governing the
production process, including
MCO/PIHP production activity logs,
and MCO/PIHP staff review of report
runs.
Review documentation that confirms
that prescribed data cutoff dates were
adhered to.
Demonstration that the MCO/PIHP
has retained copies of files or
databases used for performance
measure reporting, in the event that
results need to be reproduced.
Review documentation standards that
assure that the reporting software
program is properly documented with
respect to every aspect of the
performance measurement reporting
repository, including building,
maintaining, managing, testing, and
report production.

Review the MCO’s/PIHP’s process
and documentation to assure that they
comply with the MCO/PIHP standards
associated with reporting program
specifications, code review, and
testing.

ATTACHMENT V
Complete and accurate documentation of the data and processes used
to prepare and submit performance measures (Onsite Activity 2)

Measure Element

Comments

Measurement plans and policies which stipulate and enforce documentation of data requirements, issues,
validation efforts and results. These include:
Data file and field definitions used for
each measure.
Maps to standard coding if not used in
original data collection.
Statistical testing of results, and any
corrections or adjustments made after
processing.
Documentation of programming specifications (which may be either a schematic diagram or in narrative form) for
each measure includes at least the following:
All data sources for each measure,
including external data (whether from a
vendor, public registry, or other outside
source), and any prior years’ data (if
applicable).
Detailed medical record review methods
and practices, including the qualifications
of medical record review supervisor and
staff; reviewer training materials; audit
tools used, including completed copies of
each record-level reviewer
determination; all case-level critical
performance measure data elements used
to determine a positive or negative event
or exclude a case from same; and interrater reliability testing procedures and
results.
Detailed computer queries, programming
logic, or source code used to identify the
population or sample for the denominator
and/or numerator
If sampling used, description of sampling
techniques, and documentation that
assures the reviewer that samples used
for baseline and repeat measurements of
the performance measures were chosen
using the same sampling frame and
methodology.
Measure Element

Comments

Documentation of calculation for
changes in performance from previous
periods (if applicable), including

ATTACHMENT V
statistical tests of significance.
Data that are related from measure to
measure are consistent (e.g., membership
counts, provider totals, number of
pregnancies and births).
Appropriate statistical functions are used
to determine confidence intervals when
sampling is used in the measure
When determining improvement in
performance between measurement
periods, appropriate statistical
methodology is applied to determine
levels of significance of changes.

ATTACHMENT VI

MCO/PIHP Documentation for Review (Onsite Activity 3)

Documents

Reviewed

Not Reviewed

Comments

Denominator
Policies to assign unique membership ID
that allows all services to be properly
related to the specific appropriate
recipient, despite changes in status,
periods of enrollment or disenrollment,
or changes across product lines (e.g.,
Medicare and Medicaid).
Procedures to identify, track, and link
member enrollment by product line,
product, geographic area, age, gender,
member months, member years.
Procedures to track individual member
through enrollment, disenrollment, and
possible re-enrollment.
Procedures to track members through
changes in family status, changes in
employment or benefits or managed care
type (if they switch between Medicaid
coverage and another product within the
same MCO/PIHP).
Methods to define start and cessation of
coverage.
Procedures to link member months to
member age.
Member database.
Provider data (including facilities, labs,
pharmacies, physicians, etc.).
Database record layout and data
dictionary.
Survey data.

ATTACHMENT VII
Interview Guide
Performance Measure Calculation (Onsite Activities 3-5)

1. Do you have any concerns about the integrity of the information used to create any of the
measures? Please describe.

Other issues.

Names and Titles of Individuals Interviewed:

Describe Documentation Review and Demonstrations Provided:

ATTACHMENT VIII
Proper Identification of Denominator Worksheet (Onsite Activity 3)

Measure Element

Comments

All members of the relevant populations identified in the performance measure specifications are included in the
population from which the denominator is identified.
All members who were eligible to
receive the specified services must be
included in the initial population from
which the final denominator was
produced. This “at risk” population
will include both members who
received the services, as well as those
who did not. This same activity
applies to provider groups, or other
relevant populations identified in the
specifications of each performance
measure.
Adequate programming logic or source code exists to appropriately identify all “relevant” members of the
specified population.
Employ appropriate programming
logic or source code for each measure
which identifies, tracks, and links
member enrollment within and across
product lines (e.g., Medicare and
Medicaid), by age and gender, as well
as through possible periods of
enrollment and disenrollment
according to the specifications of each
performance measure.
Correctly carry out and apply
calculations of continuous enrollment
criteria to each measure (if applicable).
Ensure proper use of mathematical
operations that determine patient age
or range.

Measure Element

Comments

Assure identification of the variable(s)
that define the member’s gender in
every file or algorithm, and explain
what classification is carried out if
neither of the required codes is present.
Correct calculation of member months and member years.

ATTACHMENT VIII
Assure correct calculation of member
months and member years, if
applicable to the performance measure.
Completeness and accuracy of the codes used to identify medical events has been identified and, the codes have
been appropriately applied.
Assure proper evaluation of the
completeness and accuracy of any
codes used to identify medical events,
such as diagnoses, procedures, or
prescriptions, and that these codes
have been appropriately identified and
applied as specified in each
performance measure.
Specified time parameters are adhered to.
Assure that any time parameters
required by the specifications of the
performance measure are adhered to
(e.g., cut off dates for data collection,
counting 30 calendar days after
discharge from a hospital, etc.).
Exclusion criteria included in the performance measure specifications have been followed.
Assure that performance measure
specifications or definitions are
followed in excluding members from a
denominator. For example, if a
measure relates to receipt of a specific
service, the denominator may need to
be adjusted to reflect instances in
which the patient refuses the service or
the service is contraindicated.
Measure Element

Comments

Systems to estimate populations which cannot be accurately counted exist and are utilized when appropriate.
Employ valid systems or methods to
estimate populations when they cannot
be accurately or completely counted
(e.g., newborns).

ATTACHMENT IX
Proper Determination of Numerator Worksheet (Onsite Activity 4)

Measure Element

Comments

Appropriate data are used to identify the entire at-risk population.
Use the appropriate data, including
linked data from separate data sets, to
identify the entire at-risk population.
Assure that the there are procedures to
capture data for those performance
indicators which could be easily underreported due to the availability of
services outside the MCO/PIHP, and
those procedures are followed.
Qualifying medical events (such as diagnoses, procedures, prescriptions, etc.) are properly identified and
confirmed for inclusion in terms of time and services
Assure that the MCO’s/PIHP’s use of
codes used to identify medical events
are complete, accurate, and specific in
correctly describing what has
transpired and when.
Assure that medical event codes are
correctly evaluated when classifying
members for inclusion or exclusion in
the numerator.
Avoid or eliminate all double-counted
members or numerator events.
Through a review of the programming
logic or a demonstration of the
program, confirm that any nonstandard codes used in determining the
numerator are mapped to a standard
coding scheme in a manner that is
consistent, complete, and reproducible.
Adhere to any time parameters
required by the specifications of the
performance measure (i.e., that the
measured event occurred during the
time period specified or defined in the
performance measure).

ATTACHMENT IX

Measure Element

Comments

Medical record data extracted for inclusion in the numerator are properly collected.
Assure that medical record reviews and
abstractions are carried out in a manner
that facilitates the collection of
complete, accurate, and valid data.
Assure that record review staff are
properly trained and supervised for the
task.
Assure that record abstraction tools
require the appropriate notation that the
measured event occurred.
Assure that record abstraction tools
require notation of the results or
findings of the measured event (if
applicable).
Assure that process of integrating
administrative data and medical record
data for the purpose of determining the
numerator is consistent and valid.

ATTACHMENT X

Proper Sampling Techniques (If Applicable) Worksheet (Onsite Activity 5)

Measure Element

Comments

Follow the specified sampling method to produce an unbiased sample which is representative of the entire at-risk
population.
Assure that each relevant member or
provider has an equal chance of being
selected; no one is systematically
excluded from the sampling.
Follow the specifications set forth in
the performance measure regarding the
treatment of sample exclusions and
replacements, and that if any activity
takes place involving replacements of
or exclusions from the sample, that
adequate documentation of that activity
is kept.
Assure that each provider serving a
given number of enrollees has the same
probability of being selected as any
other provider serving the same
number of enrollees.
Examine all sampled files for bias and
if any bias is detected, be able to
provide documentation that describes
any efforts taken to correct it.
Assure that the sampling methodology
employed treats all measures
independently, and that there is no
correlation between drawn samples.
Assure that relevant members or
providers who were not included in the
sample for the baseline measurement
have the same chance of being selected
for the follow-up measurement as
providers who were included in the
baseline.

Measure Element

Comments

Assure that the MCO/PIHP has
policies and procedures to maintain
files from which the samples are drawn
in order to keep population intact in
the event that a sample must be redrawn, or replacements made, and

ATTACHMENT X
Measure Element

Comments

documentation that the original
population is intact.
Sample sizes collected conform to the methodology set forth in the performance measure specifications, and the
sample is representative of the entire population.
Assure that sample sizes meet the
requirements of the performance
measure specifications.
Appropriately handle the
documentation and reporting of the
measure if the requested sample size
exceeds the population size.
Assure proper oversampling in order to
accommodate potential exclusions.
For performance measures which include medical record reviews (i.e., hybrid data collection methodology),
proper substitution methodology was followed.
Assure that substitution applies only to
those members who met the exclusion
criteria specified in the performance
measure definitions or requirements.
Assure that substitutions are made for
properly excluded records and the
percentage of substituted records is
documented.

ATTACHMENT XI

Proper Submission of Required Reports to State Agency Worksheet (Onsite Activity 6)

Measure Element

Comments

Measures are reported to the State in
the manner and form prescribed by the
State.

END OF DOCUMENT

File Type	application/pdf
File Title	CALCULATING PERFORMANCE MEASURES
Author	HCFA Software Control
File Modified	2008-12-30
File Created	2008-12-30