External Quality Review of Medicaid MCOs and Supporting Regulations in 42 CFR 438.360, 438.362, and 438.364

OMB Approval No.
0938-0786

EQR PROTOCOL 8:
CONDUCTING FOCUSED STUDIES
OF HEALTH CARE QUALITY
A Voluntary Protocol for External Quality Review (EQR)
Protocol 1: Assessment of Compliance with Medicaid Managed Care Regulations
Protocol 2: Validation of Measures Reported by the MCO
Protocol 3: Validation of Performance Improvement Projects (PIPs)
Protocol 4: Validation of Encounter Data Reported by the MCO
Protocol 5: Validation and Implementation of Surveys
Protocol 6: Calculation of Performance Measures
Protocol 7: Implementation of Performance Improvement Projects (PIPs)
Protocol 8: Focused Studies
Appendix V: Information Systems Capabilities Assessment (ISCA)

Department of Health and Human Services (HHS)
Centers for Medicare & Medicaid Services (CMS)

Protocol 8
Version 2.0
December 1, 2011

TABLE OF CONTENTS

ACTIVITY 1: SELECT THE STUDY TOPIC(S)........................................................................... 3
ACTIVITY 2: DEFINE THE STUDY QUESTION(S) .................................................................... 4
ACTIVITY 3: SELECT THE STUDY VARIABLES ...................................................................... 4
ACTIVITY 4: STUDY THE WHOLE POPULATION OR USE A REPRESENTATIVE SAMPLE .. 4
ACTIVITY 5: USE SOUND SAMPLING METHODS ................................................................... 4
ACTIVITY 6: RELIABLY COLLECT DATA ................................................................................. 5
ACTIVITY 7: ANALYZE AND INTERPRET STUDY RESULTS .................................................. 5
ACTIVITY 8: REPORT RESULTS TO THE STATE ................................................................... 5

EQR Protocol 8
Focused Studies
December 2011

2

PURPOSE AND OVERVIEW OF THE PROTOCOL
This voluntary protocol specifies procedures for EQROs to use in conducting Focused Studies
at the request of the State. States may elect to conduct such a study for quality improvement
(QI), administrative, legislative, or other purposes. The assessment must be designed,
conducted and reported in a methodologically sound manner. The process of conducting
Focused Studies mirrors most of the activities of Protocol 3 for conducting PIPs. As each step
for conducting a Focused Study is performed, information should be recorded on a standardized
worksheet such as that provided in Protocol 3, Attachment A.
Protocol 8 describes eight steps for conducting Focused Studies:
1.
2.
3.
4.
5.
6.
7.
8.

Select the study topic(s);
Define the study question(s);
Select the study variable(s);
Study the whole population or use a representative sample;
Use sound sampling methods;
Reliably collect data;
Analyze data and interpret study results; and
Report results to the State.

ACTIVITY 1: SELECT THE STUDY TOPIC(S)
Focused Studies should target relevant areas of MCO clinical care and non-clinical services in
which the State can reasonably assume improvement is needed. Topics selected for study
should reflect MCO enrollment in terms of demographic characteristics, prevalence of disease,
and the potential consequences (risks) of the disease. Refer to Protocol 3, Activity 1, Step 1 for
potential sources of information and suggested criteria for the selection of a topic. In addition to
the data collected by an MCO, the EQRO may have access to aggregate MCO data maintained
by the State.
States and MCOs are encouraged to consider aligning their study topics with priority areas
identified by CMS. As of 2011, such topics included childhood obesity, pediatric asthma, oral
health, and avoidable hospital readmissions. As new revised CMS priorities emerge, those
topics can be found at the CMS Medicaid webpage www.Medicaid.gov.

EQR Protocol 8
Focused Studies
December 2011

3

ACTIVITY 2: DEFINE THE STUDY QUESTION(S)
The study question(s) must be clear, concise, and answerable. The study question(s) identifies
the focus of the study and sets the framework for data collection, analysis, and interpretation.
Potential sources of information to help form the study question include:
• State data relevant to the topic being studied;
• MCO data relevant to the topic being studied; and
• Relevant clinical literature.

ACTIVITY 3: SELECT THE STUDY VARIABLES
A study variable is a measurable characteristic, quality, trait, or attribute of a particular
individual, object, or situation being studied. Variables may be quantitative or qualitative and
continuous or discrete. Discrete or categorical variables have a limited number of possible
categories (e.g., an individual has/has not received a flu shot in the last 12 months). In contrast,
continuous variables have unlimited possible values within the limits the variable range (e.g.,
age, blood pressure, temperature). Data collected on a continuous variable such as blood
pressure can be used for a discrete variable (e.g., an enrollee’s blood pressure is/is not below a
specified level). The study variable in a Focused Study should vary from a variable selected in a
PIP in that a Focused Study is looking to determine the initial state of the population and not
improvement over time.

ACTIVITY 4: STUDY THE WHOLE POPULATION OR USE A
REPRESENTATIVE SAMPLE
Measurement must be system-wide. The Focused Study must clearly identify the ‘system’ or
study population, also referred to as the universe. Once the population is identified, the MCO
will determine whether to study data for the entire population or a sample of that population. A
representative sample of the identified population is acceptable.

ACTIVITY 5: USE SOUND SAMPLING METHODS
Proper sampling methods are necessary to provide valid and reliable (generalizable) study
results. HEDIS® measures and HEDIS® sampling methodology are generally considered valid
and reliable. If the EQRO is not using HEDIS® measures, a large sample size is needed to
achieve statistical confidence.

EQR Protocol 8
Focused Studies
December 2011

4

ACTIVITY 6: RELIABLY COLLECT DATA
Measurement of study variables must be valid and reliable. A valid measure measures what
was intended. A reliable measure is consistent. A valid measure is usually reliable, but a reliable
measure may not be valid. Other Protocols describe components of the data collection plan,
data collection processes, potential sources of data, and methods of implementation. Studies
comparing the performance of multiple entities, such as multiple MCOs, require a greater level
of statistical design and analytical considerations than do studies of a single entity.

ACTIVITY 7: ANALYZE AND INTERPRET STUDY RESULTS
Data analysis begins with examining the performance on the selected clinical or non-clinical
indicators. The examination should be initiated using statistical analysis techniques defined in
the data analysis plan. For detailed guidance, follow the criteria for analysis and reporting
outlined in other Protocols, particularly Protocol 3 for validating PIPs.

ACTIVITY 8: REPORT RESULTS TO THE STATE
The EQRO will draft a report to the State with the results of the study after review of the results.
The EQRO should develop an outline that is approved by the State. Since the State may use
the report to meet its reporting requirements to federal or State agencies, the State legislature,
local advocacy groups, as well as other interested parties, the report may need certain types of
information presented in a specific format. While non-summarized findings might be of interest
to some individuals, the report should include an overall summary of findings. Typically, results
will be shared with the MCOs if they were involved in data collection activities.

END OF PROTOCOL

EQR Protocol 8
Focused Studies
December 2011

5