(State Gov't) External Quality Review of Medicaid MCOs and Supporting Regulations in 42 CFR 438.360, 438.362, and 438.364

OMB Approval No.
0938-0786

EQR PROTOCOL 7:
IMPLEMENTATION OF
PERFORMANCE IMPROVEMENT
PROJECTS
A Voluntary Protocol for External Quality Review (EQR)
Protocol 1: Assessment of Compliance with Medicaid Managed Care Regulations
Protocol 2: Validation of Measures Reported by the MCO
Protocol 3: Validation of Performance Improvement Projects (PIPs)
Protocol 4: Validation of Encounter Data Reported by the MCO
Protocol 5: Validation and Implementation of Surveys
Protocol 6: Calculation of Performance Measures
Protocol 7: Implementation of Performance Improvement Projects (PIPs)
Protocol 8: Focused Studies
Appendix V: Information Systems Capabilities Assessment (ISCA)

Department of Health and Human Services (HHS)
Centers for Medicare & Medicaid Services (CMS)

Protocol 7
Version 2.0
September 2012

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it
displays a valid OMB control number. The valid OMB control number for this information collection is 0938-0786. The time required
to complete this information collection is estimated to average 1,591 hours per response for all activities, including the time to review
instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you
have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: CMS, 7500
Security Boulevard, Attn: PRA Reports Clearance Officer, Baltimore, Maryland 21244-1850

Table of Contents

PURPOSE AND OVERVIEW OF THE PROTOCOL .................................................................. 3
ACTIVITY 1: SELECT THE STUDY TOPIC(S)........................................................................... 3
ACTIVITY 2: DEFINE THE STUDY QUESTION(S) .................................................................... 4
ACTIVITY 3: USE A REPRESENTATIVE AND GENERALIZABLE STUDY SAMPLE ................ 4
ACTIVITY 4: SELECT THE STUDY VARIABLES ...................................................................... 4
ACTIVITY 5: USE SOUND SAMPLING METHODS ................................................................... 4
ACTIVITY 6: RELIABLY COLLECT DATA ................................................................................. 5
ACTIVITY 7: ANALYZE DATA AND INTERPRET STUDY RESULTS........................................ 5
ACTIVITY 8: IMPLEMENT INTERVENTION AND IMPROVEMENT STRATEGIES ................... 5
ACTIVITY 9: PLAN FOR “REAL” IMPROVEMENT ................................................................... 5
ACTIVITY 10: ACHIEVE SUSTAINED IMPROVEMENT ............................................................ 6

EQR Protocol 7
Implementation of Performance Improvement Projects (PIPs)
September 2012

2

PURPOSE AND OVERVIEW OF THE PROTOCOL
The purpose of this protocol is to provide guidance to EQROs conducting optional Performance
Improvement Projects (PIPs) for the State. Federal regulations at 42 C.F.R. § 438.240(d)
require MCOs to conduct a PIP, which must be validated by an EQR using Protocol 3:
Validating Performance Improvement Projects. States may also chose to have the EQRO
conduct additional PIPs to assess and improve processes and outcomes of care provided by
MCOs in the State. Study topics can align with Federal initiatives such as Partnership for
Patients or the Million Hearts Campaign. States also have the option to have the EQRO provide
technical assistance on study or analytic methodologies to support MCO efforts in this area. It
is also recommended that study questions consider the three aims of the National Quality
Strategy:
•
•
•

Better care for patients and families,
Improved health for communities and populations, and
Affordable health care.

This protocol describes ten steps for implementing optional PIPs. Users of this Protocol should
refer to Protocol 3, including its attachments. Protocol 7 specifies how to conduct the following
ten activities:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.

Select the study topic(s);
Define the study question(s);
Select the study variables(s);
Use a representative and generalizable sample;
Use sound sampling methods (if sampling is used);
Reliably collect data;
Implement intervention and improvement strategies;
Analyze data and interpret study results;
Plan for “real” improvement; and
Achieve sustained improvement.

ACTIVITY 1: SELECT THE STUDY TOPIC(S)
The PIP should target improvement in either clinical or non-clinical services delivered in the
State. Topics selected for study must reflect the Medicaid/CHIP enrollment in terms of
demographic characteristics, prevalence of disease, and the potential consequences of the
disease. Refer to Protocol 3, Activity 1, Step 1 for potential sources of information and
suggested criteria for the selection of a topic. In addition to the data collected by an MCO, the
EQRO may have access to aggregate MCO data maintained by the State.
The CMS suggests that States consider PIPs which address some of the national health
priorities CMS has identified (e.g., in 2011, Partnership for Patients, Million Hearts Campaign,
pediatric oral health, and childhood obesity).

EQR Protocol 7
Implementation of Performance Improvement Projects (PIPs)
September 2012

3

ACTIVITY 2: DEFINE THE STUDY QUESTION(S)
The study question(s) must be clear, concise, and answerable. The study question(s) identifies
the focus of the PIP and sets the framework for data collection, analysis, and interpretation.
See Protocol 3, Activity 1, Step 2 for additional information about developing appropriate study
questions. Potential sources of information to help form the study question include:
•
•
•

State data relevant to the topic being studied;
MCO data relevant to the topic being studied; and
Relevant clinical literature.

ACTIVITY 3: USE A REPRESENTATIVE AND GENERALIZABLE STUDY
SAMPLE
Measurement and improvement efforts must be system-wide. The PIP must clearly identify the
‘system’ or study population, also referred to as the universe. Once the population is identified,
the MCO will determine whether to study data for the entire population or a sample of that
population. A representative sample of the identified population is acceptable. See Protocol 3,
Activity 1, Step 4 for information about how an EQRO validates an appropriate study population.

ACTIVITY 4: SELECT THE STUDY VARIABLES
A study variable is a measurable characteristic, quality, trait, or attribute of a particular
individual, object, or situation being studied. Variables may be quantitative or qualitative and
continuous or discrete. Discrete or categorical variables have a limited number of possible
categories (e.g., an individual has/has not received a flu shot in the last 12 months). In contrast,
continuous variables have unlimited possible values within the limits the variable range, (e.g.,
age, blood pressure, temperature). Data collected on a continuous variable such as blood
pressure can be used for a discrete variable, (e.g., an enrollee’s blood pressure is/is not below a
specified level).
See Protocol 3, Activity 1, Step 3 for more information about how EQROs select the variables to
be measured.

ACTIVITY 5: USE SOUND SAMPLING METHODS
Proper sampling methods are necessary to provide valid and reliable (generalizable) study
results. HEDIS® measures and HEDIS® sampling methodology are generally considered valid
and reliable. If the EQRO is not using HEDIS® measures, a large sample size will be needed to
achieve statistical confidence. See Protocol 3, Activity 1, Step 5, as well as Appendix II, for
information about valid sampling procedures.

EQR Protocol 7
Implementation of Performance Improvement Projects (PIPs)
September 2012

4

ACTIVITY 6: RELIABLY COLLECT DATA
Data collection procedures must ensure that the data used to measure an indicator of
performance are valid and reliable. A valid measure is one that measures what it intends to
measure, while a reliable measure provides consistent results is an indication that the data will
produce consistent, repeatable or reproducible measurements. See Protocol 3, Activity 1, Step
6 for information about how an EQRO validates correct data collection procedures.
Potential sources of data include:
•
•
•
•
•

Administrative data (e.g., membership, enrollment, claims, encounters);
Medical records;
Tracking logs;
Results of any provider interviews; and
Results of any Medicaid beneficiary interviews and surveys.

To ensure sound data collection procedures, consider the criteria outlined in Protocol 3,
Activity 1, Step 6.

ACTIVITY 7: ANALYZE DATA AND INTERPRET STUDY RESULTS
Data analysis begins with examining the performance on the selected clinical or non-clinical
indicators. The examination should be initiated using statistical analysis techniques defined in
the data analysis plan. For detailed guidance, follow the criteria outlined in Protocol 3, Activity 1,
Step 8.

ACTIVITY 8: IMPLEMENT INTERVENTION AND IMPROVEMENT
STRATEGIES
Real, sustained improvements result from a continuous cycle of measuring and analyzing
performance, and developing and implementing system-wide improvements. Actual
improvements depend on thorough analysis and implementation of appropriate solutions. For
detailed guidance, see Protocol 3, Activity 1, Step 7 criteria.

ACTIVITY 9: PLAN FOR “REAL” IMPROVEMENT
It is important to determine if a reported change represents “real” change or is an artifact of a
short-term event unrelated to the intervention, or random chance. See Protocol 3, Activity 1,
Step 9 for information about how an EQRO assesses the probability that reported improvement
is a true improvement.

EQR Protocol 7
Implementation of Performance Improvement Projects (PIPs)
September 2012

5

ACTIVITY 10: ACHIEVE SUSTAINED IMPROVEMENT
Real change is the result of changes in the fundamental processes of health care delivery and is
most valuable when it offers demonstrable sustained improvements. In contrast, a spurious
“one-time” improvement can result from unplanned accidental occurrences or random chance.
See Protocol 3, Activity 1, Step 10 for information about how an EQRO determines if the real
change is sustainable.

END OF PROTOCOL

EQR Protocol 7
Implementation of Performance Improvement Projects (PIPs)
September 2012

6